Given that Warp 1 is supposed to be c and they were at warp for only a few seconds, the distance from the Earth wouldn't be all that great. The return time quite likely would be pretty long.

From the Bad Astronomer's review of the pilot episode of "Star Trek: Enterprise":

Bad: In the beginning of the show, the captain and one of his crew are talking about the ship. Captain Archer says it can go to Neptune and back in just six minutes at warp 4.5. Later in the show, Archer mentions that warp 4.4 is 30 million kilometers per second.

Good: Sometimes this Good/Bad format fails me. Those numbers work out! Neptune is about 4.4 billion kilometers from the Earth. Getting there and back in six minutes means moving at about 80 times the speed of light, or 24 million kilometers per second. The difference between that number and 30 million is negligible, given that Captain Archer may have been using a rough number. They aren't equal, but neither are they off by much. Again, this is pretty close.

Having said that, there was a misstep later in the show. It was said that the Klingon home world is four days travel away. That comes out to about 8 trillion kilometers (just less than one light year) away from Earth at 80 times lightspeed. The nearest known star to us, Proxima Centauri, is about 40 trillion kilometers away (more than 4 light years), or 20 days travel time. I'm not sure how this inconsistency slipped past the Trek writers, to be honest, but late rewrites are usually the culprit. Again, I'll note that while this is an error, it's not a terribly big one. I would be remiss to ignore it though.

However, I should also note that in several scenes, we see stars streaming past the window as the Enterprise warps past them. If it takes 20 days to get to the nearest star, then it should take roughly that long on average for just one star to flash by. I have heard some fans say that those aren't stars we see going past, but pieces of space dust. That explanation doesn't work; when the ship slow to subwarp, the streaks of light clearly coalesce into stars. So really, we should see the stars something like we do now: not streaking past, but moving verrrry sloooowly, like distant mountains on a long drive.