There's an old etiquette stating "no brown in town" referring to the antiquated notion that "country colors" should be reserved for rural settings while a higher degree of formality and decorum is required for the city.
Ironically in this day and age, wearing brown in town (specifically tailored menswear) conveys greater elegance and refinement than traditionally "city" colors.
I, personally, love composing outfits with low-contrast earth tones. There’s something so pleasing and sophisticated about heavily textured brown and gray garments accented by a touch of red! This color palette is reminiscent of an autumnal mountain hike. Yet, it shines brightest at the center of a sprawling metropolis.
This look is such a departure from the typical sea of corporate charcoal and navy you’d expect from the financial district, which serves to magnify its intrigue. It is because earth tones are not "uniform colors" that their use in tailored menswear signals refinement.
Most men who wear tailoring -- suits, sport coats, and the like -- have to because of their jobs. There is an air of obligation and reluctance in the way they wear their clothes. In fact, when a dress code is met with resistance rather than engagement, the clothes may end up wearing the man.
Brown tailoring is free from those connotations; it’s worn for leisure, for comfort, and for celebration of self. It's typically donned by men who have transcended the larval phase of treating clothes as a uniform, and instead wield them as tools for broadcasting poise and persona. Sophistication is defined largely by experience, and brown menswear simply isn't a novice proclivity.
Furthermore, tonal color schemes are in keeping with the theme of sophistication and refinement. To strike a pleasing contrast between garments of similar hue is another challenge that harshly divides the virtuoso from the novice. The latter prefers to muck around with flat, matchy-matchy monochromatic outfits or profanely high-contrast debacles. Elegance lies perfectly balanced between those extremes: in looks that don't cry for attention, but captivates those who take the time to appreciate.
Final verdict: bring the brown downtown, if it so pleases you.