It's commonly stated that the liftoff of a Saturn V was the second-loudest man-made sound, second only to a nuclear explosion. The explosion of Krakatoa is the loudest natural sound in recorded history.

The 200 decibel sound pressure was so large it had to be taken into consideration in the mechanical design of the Saturn V.

I believe that every launch caused some damage to the pad, though normally it was minor. The flame deflectors would be the most likely things to suffer damage. Perhaps they were one-use-only, or had to be resurfaced after each launch, but I don't really know. I haven't heard of any major, unexpected damage during an Apollo launch, either.

Could you be thinking of the Soviet Union's N-1 launch attempts? One of them caused devastating damage and hundreds of casualties when it exploded immediately after liftoff.