Agreed that some evaluations of bigotry are simply perception. But bigotry also has real, measurable effects. Bigotry in literature has an historical context, and the author’s intentions don’t always trump the real effect that his/her work has had on his readers.
In Tolkien’s case, much of the evidence against bigotry does not exist within the work he’s best known for (LotR); it exists in peripheral or unpublished sources, like the Silmarillon and his personal notes. Judging him on his published fiction alone, it’s hard not to notice that he does the same thing that many fantasy authors before and since have done, which is to reduce the presence of non-white people to “the enemy” or “the other”, and further ascribe to them qualities like “decadent” and “evil” and “easily duped”. Not altogether different from the way people of non-European cultures have been described by racists for several hundred years.
And further, Tolkien’s influence on the fantasy genre is part of the reason why it’s so heavily focused on medieval (especially northern) Europe and pseudo-medieval Europe; his legacy is not just his own work, but also a generation of Tolkien clones, Tolkien-inspired D&D games full of racial essentialism, and publishing gatekeepers who simply assumed that epic fantasy wouldn’t sell if it wasn’t Eurocentric, like Tolkien’s work. All that’s not really his fault; I’m sure he didn’t intend to have that kind of effect. But for the readers of color who try to engage with epic fantasy and run smack into that essentialism, or those stereotypes, or those gatekeepers, the effect is real and lasting.