If adhominem attacks are logic fallacies, why arent insults?

I trick that a lot of trolls use to get around the logical fallacy of "ad hominem attacks" is by taking personal attacks like "You are stupid/gay/retarded" and simply changing the wording to "Your argument is stupid/gay/retarded" just to insult the other person, without technically breaking the rules. When I first started going on the internet, I didn't understand the difference between an "adhominem" insult, and a "non-adhominen" insult, so when somebody told me "your opinion is full of crap" I would respond "you're full of crap," and I used to wonder why my insult was always considered "trolling" while the other was considered a "valid argument." Now I know that the entire difference between "trolling" and stating a "valid argument" is all just a difference between saying "you" and saying "your opinion."
So here's the question. Why are some types of insults considered logical fallacies, and some are considered valid arguments? Shouldn't all insults be considered logical fallacies?
So here's the question. Why are some types of insults considered logical fallacies, and some are considered valid arguments? Shouldn't all insults be considered logical fallacies?