Hey, Keran, do you have any advice or actual play examples to show how you and your group(s) handle "character-driven sprawl" (good term BTW)? What about the extent of the sprawl in your experience? Can you handle situations where the group completely explodes into individual characters having their own, unconnected adventures?
The big thing is setup. If you set up in the sort of typical plotted-adventure fashion, with characters who are only loosely connected and really don't have any good reason to be together, then if the players play in a character-driven fashion, why shouldn't they scatter? The five random strangers who meet in a tavern and get hired by someone they don't know to do something they have no personal reason to care about don't have any strong motives for sticking together that make much in-character sense at the start, unless they all really want the money. They might form bonds as the adventure progresses, but you can't count on that. So you need a different kind of a setup.
At the start of a new campaign, I describe the setting and genre, and pitch a very sketchy idea for a scenario: it'll be some kind of conflict or problem in the setting. Then we do group character creation. Not necessarily at a session, because it takes me longer to think through a character than that; what I do is forward all the emails about the characters back and forth between the players. We figure out how the characters are going to be connected to each other, how they might come to be involved in problems in the setting, and what conflicts they might bring into play themselves, in a process of feedback and adjustment. I do my best to get people to make motivated, connected characters who can work together, in a setting in which there are interesting things to do.
Characters need to have niches that aren't overshadowed. Sometimes it's the profession that's the character's main claim to uniqueness, but it can also be background and connections. I need to balance differently for a focus on roleplayed intrigue and investigation than I would for a game focused on mechanical combat.
I don't want to see characters who can't or won't work with the others, and I don't want to see scoundrels -- hard, dangerous characters with an edge are acceptable; the amoral, the betrayers, the abusers of the innocent are not. I'm not running narrativist games where watching the characters or the group morally self-destruct is the point, so I don't want characters who are likely to bring the group to destruction.
I don't want characters who don't have any drives or goals, passive characters who'll stand there like posts if I don't push a plot at them.
Tension among the PCs is acceptable, and can add a lot of dramatic interest to play, but it's risky and should be constructed with caution. Particularly, it's risky if there's a significant power differential between the PCs and there isn't a pre-existing relationship of OOC trust between the players involved. I've seen tension between the PCs work magnificently when the players were getting along. However, if you don't know the players, particularly if one PC is likely to end up in a position of advantage with respect to the other, walk warily. Real-life control freaks who enjoy pushing people around will gravitate to positions where they can do so under cover -- sometimes the excuse is "I'm just playing my character," but it can also be "I'm just playing by the rules" or "I'm just adding dramatic tension." (Actually, I don't want people like this around at all, because when they're not in a position of power they spend all their time jockeying for one, which is a royal pain in a cooperative game. You can afford to take risks with cooperative players that you can't take with players who think like social-climbing baboons.)
I tell players to make the characters they want to play that make sense in the setting -- they don't have to make a weak starting character unless they specifically want one. The reward for play isn't getting points to try to reach the character you wanted to play all along; it's to explore whatever it is that interested you about the character and the world in the first place, and perhaps to achieve your character's objectives.
(To some extent many systems fight against the construction of some kinds of characters that are perfectly good fictional concepts. And to some extent many systems mildly constrain overarching plot, at least if you use them in the most expected way: the assumption that characters should start weak and then advance has a tendency to channel play into a certain plot pattern -- Claw Your Way to the Top.
If you happen to be interested in characters that the system fights against, or you're interested in the sort of exploration that the Claw Your Way to the Top pattern gets in the way of, then you end up sometimes having to do odd things that don't really fit the character, and which might not really have any good in-world justification, in order to secure the right advantages to try for the play you want. This can create a conflict for the player who really wants to do X with the character, and whose internal model of the character might be built to do X, while he's expected by the system and convention to do Y. The expectation to conform to the system can obscure the desire and intent to do X, without removing it; and if there isn't a lot of fictional-world reason to do Y -- if it's only game-mechanical -- then it becomes more likely that the immersive player will consciously intend Y OOCly while the subconscious character model is pulling toward making a break for X. I don't want my system doing things like this, and my rules-light homebrew is very unconstricting this way.)
After character creation, we play intro scenes for each of the characters, so the players have a chance to get into character and see a bit of the setting before they meet each other, and maybe make some adjustments. And after that we bring the characters together and start play in earnest.
I don't worry about the characters deviating from the plot, because what the characters decide to do
is the plot. Telling the PCs what they ought to be doing isn't my job: they're supposed to figure out what they want to do and then try to do it. They're agents. The NPCs are agents. Natural processes cause events. But preplanned plot has no presence in the fictional world. Neither does theme or game. They can't do anything, can't cause anything: only the characters can. I usually have some idea of what the characters will be undertaking in the next session or so, but it's really up to them.
If the players aren't happy with what their own characters are doing, they can adjust their characters to anything reasonable and not too disconnected from the past, and which their own modelling will permit; we can figure out how to change their character's circumstances; they can make new characters; they can drop out.
Or they could go do something else in the setting. If they interest me enough, I might be willing to run a side campaign for them; but so far I've never had the circumstance come up. Because how many players, given the choice of still connecting to the rest of the PCs in some manner or going it completely alone, are going to pick the latter? What's the payoff for that choice?
Mostly there isn't one.
There are circumstances in which there is a payoff. I think you mentioned one of them earlier -- not having any significant ability to affect the group's actions, being overshadowed or overlooked. The other one is that somebody in the group is playing a jerk character that's making associating with the group a miserable experience.
I may or may not be able to fix the overshadowed or overlooked problem while the player keeps the same character, since I'm not in control of PC social dynamics. But I am the gatekeeper, and I have only myself to blame for a split if I tolerate a jerk character for one minute past the point where I realize that talking to the jerk's player isn't going to change the character's behavior (which is usually the case: most jerk characters are played by jerk players).
I've always played online, and it's fairly easy to set up side sessions online and log them so that other players who didn't watch or play can read them. So we have an approach that part of the pleasure is playing, and part of the pleasure is getting to watch the whole story unfold, wherever it's going. Good players with interesting characters can make this pretty entertaining, and it encourages people to take an interest in other people's subplots.
I'm happy to hand out some authority usually reserved for the GM to players who want it and whose creations mesh well with mine. Often, players will take an interest in developing parts of the setting or cultures that their characters are from, and that's fine with me if they're good at it; and they may also be the ones answering questions about in during play, as well as developing it outside of play. The setting becomes a collaborative effort, not just mine, with more people's knowledge and creativity gone into making it believable and interesting, and more people actively invested in it. I coordinate the whole thing and make sure it hangs together, but I don't necessarily have to be tightly managing everybody else's play -- in fact, at times we might switch off informally in the middle of a session, when one of the players might take over the narrating and resolving if the PCs move into an area he developed.
In my first campaign I hadn't figured out that I needed to do some of this stuff, or how to do it -- I didn't realize at the start that I was going to be running a character-driven open-ended political intrigue instead of a specific adventure with a lot of GM-planned obstacles, and I didn't realize that while the underlying ideas behind typical game-text GMing advice were often still valid, the specific techniques would sometimes be different from standard expectations. So that campaign had a higher dropout rate than my later ones did. In later ones I'd figured out most of the above to some extent, and while I still wasn't great at communicating to players exactly what the structure of the campaign was, I managed to prepare more of them for the idea that it probably wasn't what they were used to.
I guess if someone isn't interested enough to stick with the main campaign, they're not interested enough to stick around at all. So I've had people leave, but I haven't had a campaign threatening to fracture into four separate pieces or something.
Edited to add: Oh, yeah, one thing: Persistent Style Clash != Jerk Player
I've had people who were so strongly grooved into reacting to a GM-prepared plot that they never managed to make characters with the motivations to steer themselves. These people don't actively annoy anybody else; they just wind up passive and bored in my campaigns. Nothing wrong with wanting a plot to react to, it's just that I don't run this way and couldn't manage it longterm. (I find it draining and I'm bad at it besides.)
I've had people whose tastes were so strongly cinematic that they always ended up playing
Pirates of the Caribbean in my
Master and Commander game. They'd do things that made no IC sense because OOC they wanted Action! Now this is actively annoying, but it isn't intended to be so: it's just the player's preference and ingrained habit. And there's nothing wrong with this preference either, but I can't make it work with my campaign worlds.
So both of the above sorts of players really need some other game than mine, but this doesn't imply anything bad about the player.
The jerk player is someone who's pleased only if he's stepping on other people, or who doesn't care even slightly if he does. And I don't mean he wants to beat a challenge or compete in a fictional area where the group agrees that competition is a good thing. He's generally competing in the social area where the players are supposed to be cooperating, and the roleplaying becomes a cover for some of the OOC powergaming, with the characters seen as proxies for the players. It doesn't stay confined to the roleplaying, either. I've seen players try to run out any other player the GM was listening to because they wanted to controlling the game by controlling the GM, and couples where the guy's character is a powerful IC bully and OOCly he behaves in such a domineering manner to his SO that she's afraid to talk to the other players openly and wants to set up secret channels of communication. Behavior that isn't desirable anywhere.