What's to stop an intermediate black hole from getting a slingshot boost by good old Sgt A* into our neck of the woods?

Let's just say that if we do get gobbled up, we won't see what hit us.