I guess often is a relative term. I don't relax often either (maybe 4 times a year in the past, sometimes less, but I'm 19 weeks post-touchup now stretching to 26), but my hair is jet black with the new growth and over time, it gets lighter, browner because every time neutralizing shampoo is used, more peroxide is being put on the hair. Over time, the lightening will become more noticeable.
Also, check out the ingredients of your other products. Some of the alphahydroxy acids (glycolic, lactic and fruit acids) are commonly used in chemical peels which are often times done because people have dark spots that they want to go away. HTH
As far as damaging, I don't know . . .