As far as the whole "center" argument is concerned, if you have expansion in infinite direction, then every point becomes the center. This appears to be more or less what is observed.

Also, it is useful to think of distant objects not as far away in just distance or just time, but rather both. This is where spacetime comes in. It is difficult to grasp conceptually since our minds can only conceive of certain sizes. In the end we're just measuring from our own frame of reference, and so spacetime looks as though it's centered on us, when in fact it's centered on everything.