Actually, that is the usual perspective you hear, as in Sagan's remark. I think it is more conceptually useful to say it the opposite way-- different amounts of time would pass for both of you, but you would both age at the same rate. You would be like two movies of different length from start to finish, in which all the action happens at normal speed.
And I don't see anything wrong with that picture, it's just that we move through our own time, not someone else's. They may interpret us as moving through their time, but that is kind of an egocentric perspective.