It doesn't matter if you love the other person. It matters if you love the relationship. Sometimes you have to look past the person and evaluate your situation. I have loved very hard just to realize the relationship wasn't sh*t.
In the middle of an argument, pay attention to that thing he blurted out that didn't make sense and didn't seem to fit the conversation. He is confessing to something you aren't even aware of yet.