perhaps it meets other needs - such as a need for uncomplicated physical sex without the trappings of relationship, or for sex with someone of a gender other than one's partner's, or for sex at a time when it is otherwise not available (during travel or a partner's illness, for example). Or it may simply be a natural extension of an emotional and/or physical attraction to someone besides the primary partner.
Many of us learn to think this way in childhood, from parents who have little intimacy or attention for us, so we learn that there is only a limited amount of love in the world and we have to fight for whatever we get - often in cutthroat competition with our brothers and sisters.
Imagine what it would feel like to live in an abundance of sex and love, to feel that you had all of both that you could possibly want, free of any feelings of deprivation or neediness. Imagine how strong you would feel if you got to exercise your "love muscles" that much, and how much love you would have to give!
A good technique for listening is to hear what your partner has to say, and let him know you heard by telling him what you think he just said. Use this clarification technique before you respond with your own thoughts and feelings.