The biggest question I ask when considering tone color is whether I’m looking for the individual’s sound to blend with other sounds, or rather be a soloistic color within the ensemble.
Of course if you are playing solo literature, it is more a matter of taste. It is important to think holistically in this situation, because the listener takes in everything at once. What may sound good up close on an individual instrument may not sound ideal from the back of a concert hall or the press box of a stadium.
I find that xylophone mallet choice is the most polarizing in terms of it’s role in the ensemble. You can pick a mallet that blends well with other keyboard textures, or something a little more punctuated as a solo color.
This choice is often up to the artists, instructors, and conductors of depending on the context. The wrap material (or lack thereof) plays a small part in this choice, but a lot of it has to do with the core of the mallet. Warmer sounds have a lower density core or a thicker latex wrap (and vise versa for the brighter sounds).