Let me just say: Don't convince yourself what you're doing is right if you're having an affair with a married man. You have no idea the pain you will cause to yourself and others.
I've been married for 11 years to a man I believed the sun rose and set with. He's always been so sweet and understanding. He'd also been having an affair with another woman for 7 of those years. It started after I had a severe bout of postpartum depression. He was lonely I'm sure, but then, depression is a lonely thing.
The problem is the affair simply continued. Rather than turning to me and working out the problem, he let me suffer alone, and started a problem that eventually left him suffering alone as well.
After about 6 years, she discontinued her birth control and got pregnant in the hopes that he would finally leave me for her. He did not. He even refused to tell me or his family and kept his daughter secret for years.
About a year ago, we were on the brink of divorce (I was finally tired of him being gone all the time - I thought it was his work, but I was lonely and told him so). He finally confessed everything. It hurt me terribly. All the times I had to see the disappointment in my kids faces when I told them Dad was working late, all the things he missed, I now associate with time he spent with her...I feel that still. My kids feel that still. He never left me for her. He tells me he loves me every day. He told me about his daughter because the guilt was eating him alive. I was not and am not perfect. I know that our marriage has not been perfect, but an affair is never the answer. The more she tried to convince him that she was better for him than me, the more he dug his heels in because he knew what he was doing was wrong. While his guilt wouldn't allow him to leave me, his guilt over his child also wouldn't allow him to leave her. He stayed. I know he'll never leave, but I'll never look up to him and trust him the way I did. I love him, but it's a daily struggle. He too feels it. He has lost respect in my eyes, in the eyes of his children and his parents and brothers and in the eyes of my family and still he chooses to stay. Counseling has brought us a long way, but I still hurt a lot and my kids don't trust him or believe him anymore. She is hurting too, I'm sure. She's a single mother. She didn't manage to snag herself a husband. Their daughter has some health problems as well making it all that much harder. His time with his daughter is very limited (a few hours a week) and that hurts him and her.(She limits it because she'd like him to come "hang out" with them alone which is not appropriate or acceptable), so she limits him to once a week when she drops her daughter off. Her daughter is confused, and my boys are hurt because they have a sister that they want to live with them, but can't. I'm hurting because I wanted more kids with my husband (and my kids wanted more siblings) but that's not likely to happen with things in this state. My advice: leave him. You won't win. No one will. Everyone will hurt, everyone will lose.