Actually, I'd claim that if you're going to get technical about this, nothing is directly observed. You experience certain sensoria, and infer the existence of external objects. This is just as true of chairs and tables as it is true of electrons and x-rays, though there may be more intermediate steps for the latter. That doesn't change the fact that the experience of more conventional objects remains indirect.

However, if we grant that there do exist external objects which are responsible for your sensory experiences, then the existence of the universe shouldn't be a problem. If we define the universe in the most reasonable fashion, as everything that exists, then if there's anything that exists, there must be a universe. In fact, even if we don't grant that your sensoria correspond to external objects, there's still a universe, since by Decartes reasoning, at least I exist, so something exists whether or not my senses are reliable. The fact that we can't experience the entire universe merely means that our knowledge of it is incomplete, not that it isn't present.