MetaScape
I believe I ordered this from a Dragon Magazine ad or something in the 90s. The company isn’t even around anymore and the guy still kept posting poorly edited updates in a website somewhere (not sure if still around). The game was basically a fantasy sci-fi game that ripped off Star Wars and a dozen space opera tropes. It had a race of cat people with psychic matriarchs, cyborg lizard people, male-only warrior-monk race that looked like Dark Sun Muls on steroids and a mysterious hooded robe-wearing race of magic users who no one had ever seen without their robes (all you could see was darkness if you tried to peek through their hoods), had laser swords and could command magic by harnessing the power of the Source (not to be confused with the Force...but basically the same thing).
I’m not sure if I still have the books somewhere cuz I lost a bunch of stuff to a termite infestation years ago, but the game came in a boxset with a couple of manuals (maybe three or four, can’t recall), a couple of miniatures and its own patented 16-sided dice (I lost to time). They system used some sort of esoteric math were you multiplied the result of the 16-die by another die roll (don’t remember which other dice it used) to determine a potentially infinite result (I believe the system also used some sort of exploting dice or something). It was sort of race/class/skill-based, were your race and your class determined your starting abilities as a sort of package deal, but you could take off from there in a more freeform progression approach.
Don’t really remember details about the setting, but it was basically a far future spacefaring society, predominantly human (or at least a more advanced version of humanity) with a hodgepodge of other allied races, settled around a space sector close to where they made contact with the robe-wearing race of magic users.
As I recall it, the whole thing (setting and system) was a combination of interesting and innovative, with tropey, weird and overly specialized.