WIBTA if I didn’t invite my “friend” and his group to my bday?

Me (23F) and this guy (22M) started talking after he got rejected by my best friend two and a half months ago.
He had been enamored by her for a year but she wasn’t interested and made it clear when they met to talk in person.
The both of us became "friends" because after this rejection he started to talk to me too, and I talked him through the situation trying to make all parties feel better, so that hopefully he could move on.
After that, his opinion of me grew much more positive and we started talking and hanging out. I guessed it was fine to make new friends.

After just a month from rejection, he started hinting at some kind of interest towards me, which I never fueled in any way and I also mostly ignored.

He says he likes our time together and always walked me home out of concern for my safety at night, I thought it would be fair to comply with his request to watch a film together and I offered to dine together. We got a pizza and watched a film I chose, not romantic at all because I didn’t wish to start anything romantic, AT ALL. But then he starts touching me, making it obvious that the situation was indeed different from what I wanted it to be. I avoided his obvious attempts at kissing, refused his compliments, but he took me into more hugs, threw me back on his lap when I stood up and at a certain point he was on top of me. Nothing happened, as I insisted it was late and he had to leave.

His texting style grows needier and more guilt tripping. I avoided talking to him after this to put more distance between us but then he tells me his grandfather died and I was the only person he wanted to talk with and be with but I was nowhere to be seen. I excuse myself but he still insists on this.

Not knowing what to do I decide to draw a line telling him that there was some kind of "misunderstandable" behaviour on his part (since he never told me he wanted a relationship or sex, out loud) and that I wish to preserve our friendship. I told him it was fine if I misunderstood.

He denies everything, asking me what was on my mind to make such assumptions. I tell him it’s my bad, we can continue being friends as before.

He then tells all of his friends and my friends as well that I hit on him, that I wanted sex and that I had this weird fantasy in my head that he was on board with it, too. But he also said he almost had sex that evening, which is contradictory.

His friends now believe his story and didn’t attempt to ask me for the truth. I know this because he talked about with my male best friend, too, and was (rightfully) ridiculed for it.

It’s my birthday party soon and I don’t want to see any of them, despite the time we spent together. I invited a guy from his friend group that they all don’t like because he invited me to HIS bay party and has been pleasant overall.

I’m sure this will have immense backlash and could ruin this newly formed friend group but I don’t care.