I am sorry about what your mother said to you. NO babies are hard to love. She is using it as an excuse for her poor behavior as a mother. Don't buy into it.
As for the SO, yes it was small but telling. Listen to it. Of course you could have done it by yourself (and DID) but you should not have had to. His first waking thought should have been, my woman is hurt! I need to see if she needs anything. You should not have even had to ask him or look for him, etc... The fact that you did tells you that you were not close to the top of his list of priorities.
As for the silent treatment on the bday, etc... that just makes it worse. He knows YOU and he knows you are ultra-sensitive and he knew it would hurt you. If he hurts you intentionally NOW, he will hurt you intentionally later and it is NEVER acceptable to me. He sat there, had a thought process and did something to injure you. Doesn't matter what it is, doesn't matter that he didn't snatch you up or shake you or worse. YOU DESERVE BETTER. From BOTH of them - now u stuck with Mom but this guy can go kick rocks.