Yes, it does matter. As other ladies have said it definitely says something about the guy if he has slept with two dozen women, but only had 1 real relationship, etc. That would NOT make me take a guy seriously. I understand that men see it as a score to hit it and quit it/one night stand it/etc. all that, but to me it's just kind of sleazy past a point. It just shows me how that guy views sex and women, and I would probably feel he has limited potential to be a real quality boyfriend/husband/whatever. I would totally understand if a guy felt the same about a female (and many of them do).
Of course, the older you get the bigger your number will likely get (if you're single) so I take that into consideration too.