Yeah, this is pretty much my felling about all of this stuff. One additional issue with racial level caps is that they can get messy if the group actually does reach high level, then you're stuck with a lower level character for perpetuity, cuz non-humans are apparently too retarded to advance after a certain point, which doesn't make a lot of sense conceptually speaking and is just an artificial measure. Plus a lot of groups end up ignoring them anyways, or working around them with increased XP requirements for higher levels.
I prefer XP penalties as a balancing factor, but agree that they feel "cluncky". Though, I'm a strong believer in the idea that you can have (almost) any character ability you want, as long as you "pay" for it somehow in-game. Point buy tends to be better for this. You could just charge races with greater abilities an extra amount of "points" equal to whatever those extra abilities would normally cost.
An XP debt might be better for level-based games. Once you pay off your debt you may advance normally. But till then you're stuck at level 1 or whatever. The two benefits I see in XP Debt vs XP Penalty are that 1) you only pay it once, then never have to think about or keep track of it ever again, and 2) you could work out precise XP costs for different types of abilities and pay for all the extra stuff you're getting exactly what they're worth (presumably).
You're not stuck with a lower level character in perpetuity. You can just pull a non-halfling character out of your binder, and start playing them. Or pick a retainer, and make them your PC. Or even start from scratch with a new 1st level character. Because the way XP works in old school D&D, at least up until name level, is characters of much lower level catch up
fast. Generally speaking, if the party just reached 8th level, and you're first level, then by the time they reach 9th level, you'll be 8th level. That's how doubling XP at each new level works: XP to reach level N = XP to go from level N to level N+1. Sure, those low level party members can be fragile during that compressed period of advancement, but they're easily replaceable. Playing the underdog forever can be tiresome, but playing the underdog for a brief period is often very fun.
And the caps do enforce a certain feel to the setting. No, it doesn't match whatever specific genre you think it's trying to emulate. If you really want to play Tolkien elves, for instance, then they should all start at high levels, and have a level cap at least 3 times the human limit just because. Also there should be common humans, and those with elvish (Numenorean) blood, and the latter are superior to the first group by every racial measure. But if you ignore what you think the rules should do, and look at what they actually do, while the result may not be your taste, that becomes a matter of personal preference not an objective assessment of overall value.
The real thing racial level caps do is enforce scarcity. This operates at the setting level, where all the highest level characters will be human. This creates a human-centric world, since there is more range and power, which encourages putting demihumans in isolated backwaters, and making them more archetypal than varied. This also operates on the player-selection level, where many players will skip demihumans in favor of humans with fewer long-term limitations, even if the demihumans have mechanical advantages out of the gate. If elves are always superior to humans and have no limits, or even if they're equal and perfectly balanced across all levels, then you lose some of that strangeness and exclusivity, that sense of the rare or alien. It also creates a sense of something waning, or diminishing, or ceding the world to more vibrant and diverse races. Which is actually very Tolkienesque, even if the means are not.
Note, I'm not saying that level limits are ideal in many circumstances, or even my preferred way of handling things. Just that GnomeWorks is an insufferable asshole with no redeeming qualities, and that level caps do have real effects both on the world and party composition than some people like, and that the barriers you're describing are less formidable than you're making them out to be.
The idea of an XP deficit doesn't really work. Just look at the B/X elf as an ironic example. Starts with 1d6 hp, armor and attacks of a fighter, and the 1st level spell of a 1st level magic-user. But it takes 4,000 XP to reach 2nd level, by which time the 1d8 hp fighter will have reached second, then third level. That period where everyone else reaches 2nd level while you're stuck at 1st level is a fragile time, when the elf is at their weakest and most likely to die. An XP deficit would just keep them at that stage indefinitely.
It's a little off track, but I think a better approach to elf 1 is to split it into two tiers. Start them as fighters with 1d6 HD, and no spells. Then at the 2,000 half level stage, give them the magic-user spell and maybe a boost of hp to tide them over to 2nd level (even +1 would help). The idea is to smooth out the progression by introducing the two halves of their class abilities in stages. This still doesn't address all the issues with the class, but it is a way to work on that 2nd level problem.