We don't. We just say if we can't observe something it isn't there. Once we do observe it, then we can say it is there. We don't observe the alien-built UFOs, therefore we say they aren't there.