If an object the size of a brown dwarf passed through the solar system with any regularity (even once in a million years) the orbits of all the planets would be highly eccentric/elliptical. Since our solar system happens to have one of the most stable configurations possible, worrying about such an object is ridiculous.

If you look at data on extrasolar planets their orbits tend to be very elliptical.