This question is hard to answer without context - genre of the game, target audience, distribution method (not only PDF vs. print, but rather mass market toy chain vs. FLGS vs. Amazon vs. Lulu vs. traditional book store, and sometimes even down to department: SF/fantasy, children books, computer games, manga...).

Size, layout and general presentation should match what is usual in the target market.

Ex:
Super Hero RPG: softcover, trade paperback format (like the Hellboy RPG); free quickstart intro as a 24-32 page regular comic book
(btw: I like the TPB format very much, even out of comic book context)

Definitely no more than two columns.  I love Dungeonslayers, but the 3.5 Edition pdf had three columns.  What a pain.  Thankfully 4E only has two, though I would prefer if it just had one, at least for the pdf.

When publishing, I prefer font size 10 on 8.5 x 11 with two columns, purely because it allows more layout options than digest size - which can seem cluttered if not kept simple.

I like the portability of digest hardbacks but at the gametable I find the larger page size easier to flip and glance through.

I think the Burning Wheel Gold Edition is a good example of something that works, well, if you egnore the knotted border art. Using Bodoni was a somewhat daring choice as a body face, and using cursive for examples is effective. The book was easy to read and reference. On that note, border art is almost always awful, best leave it out completely - it's a waste of good margin space.

I don't have a physical copy yet, but the RQ6 layout is also effective IMO. The side bar text is spot on. Some have disagreed with the use of ligatures (ct, st, etc.) but I got used to them in short time much to my surprise.

The typeface and layout should reflect the themes of the game. So some old-style Venetian face will be out of place in hard SF, and a geometric sans fails in Steampunk (EABA Verne). If your game uses made-up or foreign names and places, even the heading typeface cannot be too unusual because the reader won't be able to piece together the word on their own (Epic RPG). There are a lot of excellent typefaces out there and selecting two or more fonts that work together well and support the theme is part of the art of typography.

As per layout (margins, columns, leading, and so on), there are pretty standard guidelines in typography. Bringhurst's book spells it all out pretty nicely. Centuries of using these rather conservative guidelines has given us expectations for what constitutes comfortable reading. Margins which are too small feel wrong (AD&D), as do page numbers in different places on different pages (Mongoose RQII). Proper margins aren't wasted space, but the monster stat blocks from MRQII certainly are.

Like a lot of others have already said: clean, easy-to-read.  Avoid watermarks, ragged justification, "intense" fonts, etc.  Also not a big fan of glossy paper.

Index: yeah!

Size: 8.5x11 inches or digest-sized if there's not a ton of text.

Make sure the book can lie flat when open.

Quote from: KenHR;561058Make sure the book can lie flat when open.

Exactly, this!

Tying in to what some of us have said above, here's a link to a blog post by a pro:

If You Can Read This... (Mark Evanier's Blog)

Mark Evanier worked professionally in the comics industry for years. He writes for  the Sergio Aragones comic Groo, The Wanderer, for example. And wrote a lot of the Hanna-Barbera comics in the '70s. And worked with Jack Kirby, and is his official biographer. So, his professional opinion means more than my amateur opinion, or the opinion of some other random dweeb on RPG forums. Worth a look.

Black text on white, decent font size and spacing. Necromancer Games were really good at a clean, robust, easy to grok presentation.

Edit: WoTC seem to be really good at layout & presentation failure. The 3e hardback style was too messy and cramped; while 4e's layout is much better but the super-dry 'business briefing' style makes my brain glaze over and renders some of the books (esp the players' books) effectively unreadable. The DMG and DMG2 presentation seem to work reasonably well, though - the content is another matter.

Paizo's usual style is also quite poor, over-stuffed and too close to 3e, but their Pathfinder Beginner Box is much better - it takes a hint from 4e but gets it right. Their Inner Sea World Guide worked for me though; it's the module/adventure style I don't like. Still a million times better than WoTC 4e module presentation of course - I find it's easier to take a 3e D&D module like Forge of Fury and convert it to 4e, than try to actually run a WotC 4e module.

I like any typeface that's chosen deliberately and is legible. Not too many words per line, not too fancy, but, whatever, on purpose. I like trying to figure out the purpose.