The moral fabric of society will be fine. People have been cheating since the beginning of time and before then. If anything, people are more faithful and "virtuous" now than they were back in the day.
Go back a few hundred years and you'd be hard pressed to find a man of a certain status who didn't have a female companion (or companions) outside of his marriage. You had common street walkers, brothel workers, mistresses, middle-class escorts, and high-class escorts (courtesans). In Ancient Greece, high-class escorts were among the only women allowed to debate politics and philosophy with men and become independently wealthy. They were the most educated and influential women in Ancient Greece. This trend continued throughout Western society and in many societies outside of the west until cultures started doing away with dowries and bride price thus making it possible for anyone and everyone to get married.
Women were getting it in, too. Half of our grandfathers probably aren't our real grandfathers. It's not like they were doing paternity tests back then. I grew up hearing stories about the milk man sneaking out the back door while the repair man was hiding under the bed.
I think people have more incentive to stay faithful nowadays and so it is expected. We can choose our own marriage partners and we get married much older than we used to. Women are free to test the goods before marriage so we don't get stuck in a lifelong commitment with bad sex. There really is no excuse for the average Joe or Jane to step outside of the marriage. But alas, most people settle so I get it.
I wouldn't be a mistress but would I turn down the offer to be a female companion to a married wealthy, powerful man? Probably not. In some circles it is still quite common and somewhat acceptable. The wife knows it is going on and she doesn't care. They married for status and not love. I don't want his love, I just want the adventure without the attachment. However, I'd have no desire to get in between a loving relationship between a committed husband and wife because I don't want to get caught up in other people's drama and emotions.