Most discussions of racial fetish center on the question of whether it is caused by negative racial stereotypes. In this paper I adopt a different strategy, one that begins with the experiences of those targeted by racial fetish rather than those who possess it; that is, I shift focus away from the origins of racial fetishes to their effects as a social phenomenon in a racially stratified world. I examine the case of preferences for Asian women, also known as ‘yellow fever’, to argue against the claim that racial fetishes are unobjectionable if they are merely based on personal or aesthetic preference rather than racial stereotypes. I contend that even if this were so, yellow fever would still be morally objectionable because of the disproportionate psychological burdens it places on Asian and Asian-American women, along with the role it plays in a pernicious system of racial social meanings.