Nature is brutal. It’s UNNATURAL to limit oneself to eating things with less nutrients. Being one with nature means you kill or be killed, and only the fittest survive.
Now the difference of philosophy here is that the structure humans have created is much more unnatural than that. A druid would agree that going out into the woods with a bow and shooting a deer is fine. But going to a restaurant and ordering chicken that was raised in a barn, for the sole purpose of consumption, is wildly unnatural.
But going to a restaurant and buying a loaf of bread is equally unnatural. It’s not like that would get them pegged as vegetarians specifically. Also, weapons and armor (metal or otherwise) are unnatural. Adventuring is unnatural. Supernatural powers are unnatural.