I've got a campaign going and the party is investigating a series of murders at the behest of a paranoid scientist. Very accidentally it is similar to The Name of the Rose--isolated, full of monks, other people in control, weird supernatural stuff going on.

Realized I've never done a game like this. System is Traveller. Any suggestions?

Your players will miss clues. You have to take this into account. Either there are multiple ways for the mystery to be solved, or there is a way for the campaign to continue if the mystery is not solved. Or both.

Justin Alexander has an asolute TON on this kind of scenario. You could start with https://thealexandrian.net/wordpress/1118/roleplaying-games/three-clue-rule

First thing is decide what kind of game you are running.  The ways to handle it differ radically depending on what you want:

• Standard adventure that happens to have a mystery to interact with
• A dedicated mystery where the players and characters are engaged in solving the mystery for its own sake
• A patina of mystery around something else

For type 1, failure is always an option because the adventure can go another way.  For type 2, failure is also an option, but you need a way for the players to know that they failed.  Could be a time limit where something changes or something more subtle, but has to be there if the players get stuck.  Using a lot of the classic mystery elements is not a bad plan, here.  For type 3, "failure" is a different animal altogether and is really related to whatever "something else" is.  This type is in the realm of a lot of the advice you will get on running mysteries in RPG sessions.  Type 3 also includes "characters can beat the answer out of someone or magic the answer out of someone or technobabble the answer", though those aren't the only ones.  If you can get the answers without solving the mystery or the answer is handed to you, then it is type 3.

Everything else below is primarily in regards to type 1, as that is what I run.  Bits and pieces may apply to the other types due to some overlap, but many will not.

Three clues are not enough.  You need three findable, understandable clues at a minimum.  Which means in practice you need 5-7 possible clues.  I usually lean closer to 7 because with that many, 1 or 2 can be really subtle (i.e. I can assume that the players won't normally catch on to them).  The mystery should be solvable with a subset of the clues, maybe even 2 if the players are particularly inspired or chase down a great hunch. 

The mystery sits in the context of a larger adventure.  In many cases, you can think of it as a side adventure.  It could be bypassed.  It could be skipped for now and then revisited later.  It could be engaged with on some level but not completed.  The players could go away with an air of mystery unsolved--and that's good!  Maybe not good if they do this every time you throw one out there, but great for the atmosphere of the game if you give them a solid chance on 10 such mysteries and they solve or mostly solve a majority.  (If they are consistently batting less than .500, they probably don't enjoy that kind of scenario much.)

Which brings up how "partially solved" comes across.  It's great when there are layers to the mystery.  In the traditional novel mystery, there may be layers, but the layers are all about confusing the issue and nothing is "right" until the end.  That's hell on an RPG session.  Of course, the downside to partial solving is that the players might get 2 answers out of 3 and decide they are done.  I can live with it.

Never play deliberate red herrings.  Your players will invite enough false leads for themselves.  Some of them they will follow, which will become the red herrings in play. 

It's not the end-all be-all, and I don't consider myself a master of the murder mystery. But a quick and dirty trick that I think is highly effective is, have the murderer frame someone. In doing so, this creates a red herring, places obvious clues to get the PCs moving, by having to account for the murderer's activities in order to both commit the murder and set up the frame that leaves a lot of potential for discovery so players are like to find at least something peculiar without having to paint a bullseye around the clue. And when and if the PCs discover they were misled by the red herring, they will blame the murderer rather than the GM, and the ruse itself becomes a clue to getting the murder.

If you run multiple murder mysteries, you obviously don't want to do that every time. My next pick would be to have the murderer having used unwitting pawns, so the suspicious activities don't point directly back to the murderer, and the pawns don't know enough to spill the beans if questioned.

I found this a pretty good take - https://www.youtube.com/watch?v=2VRy5nNK_So&ab_channel=SethSkorkowsky

Excellent resources, thank you all very much. I saw I had been doing several of the things that would screw the adventure up.