Working up some rules for my OSR sci-fi game and writing up the entry for Astronavigation. Failure puts the ship off course by 1d10 lightyears. Failure with a roll of 95-99% puts the starship in imminent danger (comes out in an asteroid field, or just inside the atmosphere of a planet, etc). I've chosen to make a fail with 100% indicates that the starship comes out of hyper and collides with something, taking serious damage, or they phase back into real space within an object (an asteroid, another vessel, a planet, etc.), killing everyone inside.
Would this be too much for most gamers in the modern world? TPKs through combat with a superior force is one thing, but being due to the failure of a skill check by one player could be devastating for some folks, maybe. Thoughts?
My answer to the general question you pose in the subject line is very different from the specific one you're posing here.
I don't think I would get behind the wheel of an automobile if I thought there was a 1% chance of collision every time I drove somewhere. Let alone if there was a 1% chance of killing me and everyone around me. You might have recognize that what's at issue here is not at all the question you're asking. It's whether or not you're even using an appropriate game mechanic with appropriate outcomes for the activity.
One of the things I really like about old school D&D, and perhaps much of OSR has failed to appreciate it, is you're not just describing and parametizing the character. To a great degree, you're saying something about the world. When AD&D says your character with a 16 STR can force open doors on a 1-3 on d6 and has a 10% chance to bend bars/lift gates, this is not just a reflection of how strong your character is. It's also a reflection of how difficult stuck doors are to open and gates are to lift.
Now also consider the same character with the same strength, what you can break and how much you can lift might vary somewhat from one moment to the next. But not so much that you would be able to lift a gate one day, and fail to lift that same gate the next day. Or that you might bash in a door on the first try one day, but that same door will take 4 tries the next. When you're making these checks, you're simultaneously writing the world itself. You're strong enough to lift 10% of all the gates in the world. We're rolling the dice to determine if you're strong enough to lift this particular gate. In other words, we're not actually checking your strength. We're checking the difficulty of the gate.
I am fully aware this pecker slaps right across the eyes the philosophy that many old school gamers hold, that we're not inventing the world on the fly as we play, how difficult that gate to open is determined in advance and reflected with a sit mod, and your character's attributes are just that--Your. Character's. Attributes. And that's a perfectly fine and intuitive way to play. The problem from where I sit is it just doesn't look to me like most RPG mechanics are oriented towards that philosophy. The designers may hold that philosophy, but they carelessly copied what came before.
As to whether a fumble on a skill roll leading to a TPK is too harsh. Well, I would say this. When I include a small chance of instant death, it's generally intended to be a deterrent. PCs should never do this thing on purpose. For instance, falling damage in my D&D games are less harsh than most (d6 per 10', not cumulative), the idea being to give PCs a reasonably fair chance at surviving a fall without having to give in to hit point inflation. But at the same time, "Falls from Height" (greater than 40') all carry with them a probability of instant death in my campaigns. The idea is no character would jump off a cliff as a short cut thinking they have enough of a hit point cushion to take the fall. Even a small chance of instant death is too much risk. But it still comes up if a character falls accidentally or by trap. It makes these hazards very deadly.
To me, it seems what you're doing is a perfectly fine mechanic for an astronavigator setting in a course to a place he or she is unfamiliar with. The message it would send to players is "You should not do this thing. You don't get to just skip exploring unfamiliar space." This could organically create specialists within the game. Need to get out to a distant space station? Maybe hire a navigator who used to make supply runs there. And then by the end of whatever adventure is happening there, at that point, the PC navigator will be familiar with it and can always get back there in the future without chance for disaster. Now if something crazy happens. Like the PCs fail to stop the villain with his doomsday device that causes the sun of one of the inner worlds to collapse into a blackhole and now the hole is spreading as it swallows up more and more mass of the surrounding systems, hey, maybe risking that jump into the unknown reaches is worth doing. At the end of the day, 1% chance of TPK is better than 100% chance of TPK.