If the Universe were shrinking, we'd see distances between pairs of objects decreasing on average. Instead, we see them increasing on average, so the only conclusion that makes sense is that the Universe iss expanding. The answer to the question "in relation to what" would be "most arbitrarily selected pairs of objects".