I think using a hooded dryer once every 5 days should be fine. I used to dry my rollersets twice a week with a hooded dryer, and never had any problems. I've since switched to air-dryin them since I'm not using any heat.
I stayed away from the hot setting however, because I find it drying to both my hair and scalp... I used either medium or cool air.