Okay, definitely don't let him go. I've run into three different guys who told me the same thing, and after a while, they all changed their minds (long-term relationships). A good man instinctively want to please his woman. So, unless he's adamantly against it for some religious or psychological wounds, I wouldn't worry about it. Just be patient and enjoy all his other good qualities.
ITA - I don't go there outside of a LT relationship. But in a relationship, Karrine better watch her back!!!
I wouldn't dump him for it, just see if you can deal with it and see if he evolves as your feelings grow.