Are there any actors of colour in the Lord of the Rings films? And to wade into controversy - homogoneous groups tend to share the same skin colour. As noted across the planet before rapid movement of people began from one location to another. The hobbits - for better or for worse - have been established as very fair-skinned - look at the five principle hobbit actors. No dark faces at Bilbo Baggins farewell birthday party that I could see.
So two points I guess, movie canon had already chosen to establish racially homogenous groupings (elves, rangers, riders of rohan) and they are all fair-skinned. (Does the book canon do the same? I can't remember but I suspect it does.)
And how would racially mixed populations have come about?

I suspect the casting director was acting on an unspoken assumption shared by all the crew and in that sense he has been hung out to dry. They might have done better to stand up for their unspoken position than to simply fire him.