The problem I have with the moral stance is that it's not universal. You might think it is, but it isn't. For 99% of human history, marriage was about maintaining the orderly transfer of property. In those periods, mistresses and all assorted varieties of sex workers were necessary to the societal structure. For example, the Middle Ages, a time when the Church ran everything: Do you think they accumulated bad karma? Shoot, them heaux might be useful now.
You might think, 'Well, marriage is different now, it's based on love', but for people born into wealth, the old ways still hold true. And for those who plan to acquire wealth in this generation, who study how others accomplished it, dissolving a union whose primary goal is to siphon wealth to the next generation over what? momentary weakness? would be head-shakingly frivolous. Many women would tell you, 'Let him have his whores', and really mean it. I say all that to say, there are times and circumstances when mistresses provide a valuable service. For some, those times and circumstances exist even today. That's just the way of the world.