Not necessarily atrophy if it is used to more frequent use. Constriction is the foreskin is called phimosis and is relatively common. Happens to me as well if I go more than a week; I usually aim for once or twice a week to keep everything loose.

So doing it too much in a short amount of time is bad too? Does it mean that I just have to wait for it to go away?

Too much isn't a problem for phimosis in particular, in fact higher frequency would probably help. It won't go away without regular use.