This was touched on in the recent thread on race - but that was much more focused on politics around the calls. This is about having half-orc wizards, or halfling rangers, or gnome monks, and so forth. I can think of a few different approaches:
(A) In Basic D&D, there isn't a separate choice of race or class. Each race (except human) is its own class. (EDITED: removed OD&D, which has restricted combinations)
(B) In AD&D 1st edition, race and class are picked separately -- but many race/class combinations were banned. You couldn't play an elven cleric PC, because that combination wasn't allowed in the level limit chart. Other combinations were allowed but heavily discouraged by level limits, while a few were allowed unlimited.
(C) In D&D 3rd and later, those limits were dispensed with. However, because of ability score modifiers, some combinations result in less effective characters on average. Unless the GM rules it doesn't fit the setting, I can choose to play a half-orc wizard, but my PC is notably less powerful -- though a half-orc paladin works just fine.
(D) Other games don't have the equivalent. For example, in Fantasy HERO (4th) and I think in GURPS (4th), race doesn't change effectiveness much. So if I play a half-orc wizard, I still get my points worth of abilities. A GM can discourage some combinations as not fitting the setting ("I'd prefer you didn't play that"), but if a player does make such a character with GM permission, they're just as effective as other PCs.
Do people have a preference about this? Personally, I definitely preferred (C) to (A) and (B). However, as I think it, I am coming around that (D) makes more sense. Outside of race/class combinations, I've generally felt that using *power* to balance *unusualness* is a poor bargain, because it yields inconsistent results depending on the mindset of the player. Frequently, I'll see a party where half the players are power-gamers with optimized and boring characters, and the other half are experimenters/role-players trying out quirky options who all get overshadowed.
My preferred way to deal with unusualness is on a social level - especially by having a session zero where we come up with who the PCs are as a group, to avoid both too much unusualness overall, and also unlikely matchings of characters (i.e. paladin and assassin).