That is true but they assume that since it is dark when they climb up, then the deer can't see them. I am a hunter and always assumed we risked injury, freezing, and snake bites for that purpose.
IIRC, deer can not see orange or light of greater wavelength, so although hunters now wear orange primarily because they don't want to get shot by other hunters, I think many hunters also think the deer can't see them. Yet, any object the eye can not see will look black, so a deer will see hunters as if they were wearing black, which could stand out like a sore thumb.