I agree that there is something special about time, but I think of it a bit differently. I see the fundamental constraint when we start thinking about space and time is the concept of time, which starts with a simple definition. What we call a second is everything that can happen in a second-- that's it. This is just a definition that works for us. There is simply no such thing as "the rate that time passes", any more than there is a "rate that the words in a book appear on the page". A word is a word, and a second is a second, and we can perceive that word or that second any way we like, and it is still one word or one second. The rate of flow of time is a tautology, that's why it sounds like "1 second passes in one second".

However, if we do insist on thinking in terms of a rate of time passing, we attribute a speed to that rate, which is just that tautological speed of 1 second per 1 second. But now we need a concept of distance to explain why different clocks can start and end at the same place having different amounts of time elapsed on them in the process. The concept of distance also made sense of a lot of other things involving causality, and the way more distant things tend to have influences on us that are weaker and come later. So we just introduce the concept of distance to make sense of these facts, because something has to be missing if all we have is time and an invention that it flows at a certain rate.

The punchline is, when we insist that time passes at a rate we call 1 second per second, and we introduce the definition of distance (given by the way we decide to measure it), then all of a sudden reality hands us a connection between time and distance: the speed c. That comes from reality, not from light-- it would be there even if there was no such thing as light. Then physical meaning emerges from several points:
1) everything moves at c through spacetime, which stems from the tautological 1 second per second-- but if they are not stationary relative to the observer, we may imagine the "movement" is partly through space instead of entirely through time, and we need a conversion factor c to make sense of that association.
2) the paths between two points in spacetime do not cover the same spacetime distance, it depends on the path.
3) the way we measure distance and time generate an invariant, called proper time, that can be imagined to be both "real" and objective.
4) everything else we think is going on is just make believe, and stems entirely from our arbitrary way of globally coordinatizing measurements that we can only make locally.