There is nothing in nature or evolution that would force extraterrestrials of any kind (if they exist) to travel in space to us. Fermi was thinking of pulp fiction and comic book aliens in science fiction civilizations based on empires, values and technology from human history. It is just as likely that the galaxy is teeming with alien life, none of which has traveled here, as it is that aliens are very rare.

And we do not know what an extraterrestrial civilization is because "civilization" is solely a human concept. It is like saying "alien automobiles" or "alien uniforms" - if these seem comical it is because we assume "advanced alien civilizations" can beam themselves around and don't need old-fashioned automobiles, just as we rarely use horse drawn carriages and in the future might develop advanced transport systems. But in reality, all of these "advanced alien" notions are an extrapolation solely of human history applied to hypothetical advanced aliens.

Most people when discussing possible alien life are inferring their existence or behavior based entirely on known human values and human history. This leads to a cul-de-sac.

__________________
"Insignificant molehill sometimes more important than conspicuous mountain." - Charlie Chan