AITA for refusing my in laws friend to come over to our house?

I (27) & my fiancé (30M)
For the last seven years, my fiancé lived with his parents before we moved into our own place. I used to visit every weekend, and I’ve always gotten along well with my in-laws. They do have a family friend let’s call him Richard, who is an older, middle aged man and has always been around ever since I met my fiancé. He is usually there from early morning until the evening, almost every day.

At first it didn’t bother me, but over time it began to feel a bit uncomfortable. Richard comes to every family gathering, holiday, and celebration. Even when my fiancé’s parents went away for the weekend or on holidays and we were looking after the house and pets, Richard would still come over and spend time there. He mainly spends time with my mother-in-law.

He often shares his opinions and comments on things, and sometimes it can feel a little overwhelming. As the years have passed, it has become harder because I don’t get much one on one time with my in-laws, and my fiancé feels the same about wanting more time alone with his parents.

Now that we finally have our own home, we both agreed that when his parents visit, we would really like it to just be them. Richard is mainly their friend, and we simply want some private family time in our own space. We let his parents know that he is, of course, welcome in their home just not at ours. I really dislike conflict and didn’t want to upset anyone, but it seems that Richard took it personally. Now when we visit my fiancé’s parents for family dinners, he leaves before we arrive, which feels a bit unusual. We’ve made it clear he is welcome in their home if they want him there just not at our place. But I do feel terrible and I hate to be the bad guy in this. But now it feels like he hates me and only me as he believes it was my decision.

For some background, my fiancé’s sister’s husband felt similarly a couple of years ago because of Richard’s personality and the way he sometimes speaks to people. He is not allowed to go to their home.