sorry this whole situation happened to you op, I really think he destroyed the marriage by breaking those vows and stepping outside your marriage with extra marital activities.
he played too many games with your emotions back and forth, i'm moving out, oh I want you back, i'm unsure. it's just too much to put a person through emotionally.
your right not to trust him and don't let your guard down, i'm one for saving a marriage as long as there is no cheating involved but once someone steps outside their marriage that it taking things too far.
he disrespected you and his child by what he did to you, I know your looking for work and the job market sucks right now but by law you have the right to file for legal seperation/divorce would be better and he is responsible to pay the household bills (so you have the right to put him out). adultery is not taken kindly in court and spiritually.
please please please for your own sanity move on, move out, put him out but keep him away from you, he is toxic and will not help in your healing process if he is still actively around playing with your emotions.
something like this happened with my mother in law, she kept her husband around thinking things would change, he went and had a family outside of their marriage, but for some reason she kept the hope that he would come back, (it was obvious because she just would not sell the house). he never came back and now the house is foregoing foreclosure. i'm sure now she wished she would've been strong and moved on before ending up with foreclosure on her credit and in massive debt.
your husband is back because he has nowhere else to go and as others have said he has taken advantage of you and know that you are a loving person and knows how much you care for him.
you are in my prayers and again i'm sooooo sorry this happened to you. but for your own sake please put him out it is really not healthy having someone like him around. it may hurt but in the long run it will be better for you and your heart
.