While actual physical issues would be difficult to get through and cause problems, I'd propose that in a lot of situations, rather than sex being the actual problem, it's simply where most of the problems in a relationship become more apparent earlier than they do elsewhere. It's where personality traits are the most difficult to hide or mask, often leading to sex getting labeled as the source, instead of evidence of problems elsewhere in the relationship. For example: a selfish personality is more easily identifiable in a sexual relationship than it would be in the emotional side of a relationship.
“Perhaps the greatest charity comes when we are kind to each other, when we don’t judge or categorize someone else, when we simply give each other the benefit of the doubt or remain quiet. Charity is accepting someone’s differences, weaknesses, and shortcomings; having patience with someone who has let us down; or resisting the impulse to become offended when someone doesn’t handle something the way we might have hoped." Elder Marvin J. Ashton