Do they silence you in the sense that they block you? But if they are insulting, that's fine, that is, you have the option to block them and it's as if they did not exist. You can not see what they write nor can they see you. That's very good!
Then, it´s not mandatory to accept the invitations to play. There are people who invite just to learn how to win you in battle. I don´t accept that kind of invitations either. I met a "pro" who invited chat people to play, only to have many wins appear in their history.
I mean, most people in the chat are well intentioned, but you don´t have to accept the invitations to play or respond to private conversations, or anything like that.