No, you didn't scare him off. The schedules just aren't matching up. If he's interested, he has your FB, he could send you a message "Hey, I haven't seen you in the gym lately! How's it going?" At this point just play it cool, if he's interested he'll contact you.