I think part of the problem is that there is no mutual ground among you three. A conversation needed to happen prior to all of this. A child needs stability and children react to instability differently i.e. change in environment, food, customs etc. Additionally, why allow them to stay up until whenever if you know that there will be an outing the next day at noon. Was there a teen watching over them? If not, there should've been a cut off. It is not unreasonable for a child to go to sleep at 5am, be afraid being up alone at night, and being cranky the next day before noon. Also your boyfriend needs to listen to the mother of his child instead of just shutting her down and getting upset. In fact, it sounds like a relationship red flag to me.
This whole situation sounds confusing and overwhelming. But the adults need to be adults in this situation and talk first. And yes the intricacies of your relationship is "private" however, I don't know how a mother is to perceive a live-in bf and gf situation. I agree that I wouldn't want my child around that type of situation, who knows the permanency of it all. But then you would want your child to have a relationship with his/her father. Again, it sounds confusing and overwhelming.
All the best to you and yours.