I rollerset 1-2 times per week and I use a fine-tooth (rattail) comb every time. My hair is relaxed, VERY thick. I don't see any broken hairs in the comb, or on the counter, or my shirt, etc. (If I do, it may be a total of 2 or 3). I deep condition with heat, and I detangle very carefully with a wide-tooth comb and CSI, and I think that helps. I used to have one area on my hair that broke consistently, even that part has grown out nicely.
Maybe you can cut down how frequently you rollerset to limit manipulation...
Back in the day, before I started focusing on healthy hair care, I had broken hairs ALL OVER THE PLACE, every time I combed my hair. It wasn't the combs, it was the health of my hair....I don't use a fine tooth comb if I'm not roller-setting, though. I just use a paddle brush and my fingers.