I've long worked on the assumption that the visual POV is not the same as the audio POV. This doesn't account for the whooshing, but I accept it as a cinematic technique to convey speed and direction - or at least I do when it is used well.

One of the most interesting variations of the sound-in-space issue was in Gerry Anderson's series UFO, particularly in the episode "Survival". Although we the audience can hear spacecraft roaring past, it is made clear that the characters cannot hear them. The failure of vacuum to carry sound is put to very dramatic use when one character's suit-mike fails.