I am medically colorblind. It is a disability that no one seems to care about. Here is an excellent article on the implications. https://blog.prototypr.io/going-colorblind-an-experiment-in-empathy-and-accessibility-98b7003737ca
Why do I point this out? I see colors, I just can't tell many of them apart. A side effect is that because of that, I don't pay much attention to color except when I encounter font/background color combinations that bake it impossible for me to read words on the internet of applications.
What does that have to do with what you wrote? I don't pay much attention to color because I don't see colors at all in the way that you do (colorblindness in women is rare). I doesn't mean that my world is gray, I am aware of colors. With regard to your subject, my colorblindness does not mean that I'm unaware of race as a perceived visual perception, but I don't pay a whole lot of attention to it.
I've been married 50 years and it is only of late that I've given much thought to just how dark my Southeast Asian wife's skin color is, thanks to all the whiteness is evil flying around on Medium. Yes, when we lived in Saudi Arabia she was treated poorly by the Saudis and as someone who has been to 19 countries outside the USA I've seen plenty of racism worse than in the US or UK. It's not just an evil white man thing.
The bottom line, and why I chose to comment, is that the fact that color blindness isn't racism. You really are quite wrong about that. If someone doesn't care about your color, I don't, it has nothing to do with being racist.