Indeed. Put yourself in the position of a planet. In this particular case, we're going to ignore rotation, as it makes things easier. Feel free to add the spin later if it makes you happy.

The Sun is exactly to your side (left or right doesn't matter, its just a coordinate thing) and you're moving forward. If you're at EXACTLY the right speed, your forward velocity is exactly right for your pull sideways, and you're in a circular orbit.

A teeny bit more, and you'll start to increase your distance from the Sun as you go round. You're at the closest point of an elliptical orbit, unless you've got enough velocity to escape the gravity well completely, in which case nobody round here will be seeing you any time soon.

A teeny bit less, and you'll start to decrease your distance from the Sun. You're at the furthest point of an elliptical orbit, unless your velocity is so low that you're going to crash into it. Again, no-one is going to notice your existence again after a very short while. By the way, where did you get here from?

So, orbits fall into 5 types.

V < Vboom - you're going to crash
Vmin < V <Vperfect - elliptical orbit
Vperfect - circular orbit
Vperfect < V < Vescape - elliptical orbit
Vescape < V - parabolic flypast.

A planet's orbital velocity being absolutely bang on to have a circular orbit is the sort of thing thats really bloomin' impossible to see in a real-life situation, so everything sits in ellipses after a reasonably short while.