Supposing, for a second, that the lunar samples were recovered on earth.
Surely it would take any amount of geologist many, many years to accumulate that many samples of former lunar surface material?
They'd need to plan well in advance to scour the earth for lunar surface material, and to do it in secret.
You could, of course, sidestep this by postulating that more surface material was recovered in the later missions than was admitted by NASA.