This is a site for discussing roleplaying games. Have fun doing so, but there is one major rule: do not discuss political issues that aren't directly and uniquely related to the subject of the thread and about gaming. While this site is dedicated to free speech, the following will not be tolerated: devolving a thread into unrelated political discussion, sockpuppeting (using multiple and/or bogus accounts), disrupting topics without contributing to them, and posting images that could get someone fired in the workplace (an external link is OK, but clearly mark it as Not Safe For Work, or NSFW). If you receive a warning, please take it seriously and either move on to another topic or steer the discussion back to its original RPG-related theme.
NOTICE: Some online security services are reporting that information for a limited number of users from this site is for sale on the "dark web." As of right now, there is no direct evidence of this, but change your password just to be safe.

Author Topic: 'B-but that didn't come up in the playtest'  (Read 792 times)

Drude

  • Newbie
  • *
  • D
  • Posts: 4
'B-but that didn't come up in the playtest'
« on: October 14, 2021, 02:16:08 AM »
Starting to hear this excuse more and more. Whether its ivory tower 'professional' game designers in awe of their own creation or self-professed minimalists who lean on the mantra 'It's just a fun beer 'n' pretzels jam', it feels like these games aren't even useful as compost for better ideas, better games. Sure you can mess about and try to fix them but why bother. Now that everyone can be a published game designer its hard to find games that don't collapse during play unless heavily house-ruled or are wafer-thin ultralite task resolutions that may as well be a coin toss. Games that are just errata-strewn ash-cans but sell in hardback from day one, no thanks.

and don't get me started on supplements and supposed aids to gaming - 'buy my d100 list of words'.

Where's the quality ?

Spinachcat

  • Toxic SocioCat
  • Hero Member
  • *****
  • S
  • Posts: 14279
Re: 'B-but that didn't come up in the playtest'
« Reply #1 on: October 14, 2021, 04:00:14 AM »
90% of everything is crap so it's always been about finding that 10% of good stuff. Of course, "crap" is also somewhat subjective to personal tastes.

I'm part of a playtest community and here's what I've learned. Blind playtests are crucial, but rarely done. If anyone is not familiar with the term, "blind playtest" is when the designer hands the game to a bunch of noobs and watches what they do with it. What happens next is often agony for game designers, much like if Jane Goodall handed her baby to a circle of chimps.

As to "where's the quality?", consider yourself a gold miner. You gotta hack through a lot of not-gold to hopefully find the hidden gold. And once you find that "golden game", play the damn thing with lots of friends and sing its praises for the rest of us.






Chris24601

  • Hero Member
  • *****
  • C
  • Posts: 2081
Re: 'B-but that didn't come up in the playtest'
« Reply #2 on: October 14, 2021, 04:53:47 AM »
Real playtesting is a lot of work; I know because I playtested through every stage of my game’s development and it added years to the process.

Because real playtesting that you pay attention to will also kill a lot of your sacred cows. Whole chunks of the game you thought were going to be amazing are going to ripped apart and gutted. Other sections you threw into a test just because you wanted to prove how bad an idea it was compared to your brilliance… end up blowing your preferred mechanic out of the water.

Here are some real-life personal examples from my own project.

Goblins, Orcs, Ogres and Halflings: In the beginning, I had the idea of making the traditional baddies that were goblins, orcs and ogres into irredeemable soulless monsters. The lore was going to be that, when someone dies before their time, their shadow lingers in the Shadow World until the end of their natural life and, lacking souls, they would always surrender to their baser instincts and become wicked. Ogres were the shadows of kings and heroes who cast proverbial long shadows and so were larger than life, orcs were the shadows of warriors and so quite numerous as many die before their time, goblins were the shadows of common peasants and “small folk.”

In attempt to make my Halflings more interesting and the shadow-concept somewhat playable by PCs; I made them the shadows of dead children. Because they were children their baser instincts were not as wicked (those that were became goblins) and they were called Halflings because they could slip without effort between the mortal and shadow world (something the orcs, ogres and the like could only do through powerful rituals cast by others) making them sort of “half of each.”

So, here’s the thing about these things; probably a great idea to wrap a work of fiction around, but outside of playtests using pre-gens where it was the last choice, my playtest feedback was that no actually wanted to play “the shadow of a murdered child.” There’s edgy “I have shadow powers” and there’s too far for comfort and Halflings crossed that line.

By contrast, I received multiple pieces of feedback asking how they could play an orc or goblin like they had in other games and how they could get a proper half-orc in place of the half-shadow version I had.

So, as cool as I thought the whole shadow idea was, I scrapped it. Halflings went away entirely and I later salvaged some of their mechanics for the genuinely heroic Fetch.

Goblins, Orcs and Ogres got reworked into a variety of beastmen (goblins) and human mutants (orcs, with ogres as orcs who just keep growing). I later developed extensive lore about them… the leader of the orcs claimed descent from the last Praetorian Emperor and so the orcs under him have a very “Roman Empire” theme to them as their goal is to reunify the shattered empire under their banner… beings whose mutations make them hyper-aggressive super-predators who treat all under them as little better than animals; a prime example being the goblins, once a race of free flying bat-men, they were enslaved and so brutalized and malnourished that after a hundred generations (beastmen reach maturity in about two years… 200 years = 100 generations) they are little more than feral beasts with wings too stunted to ever fly (among the few free goblins, many consider killing their slave-kin to be a mercy).

And I salvaged the original concept of the goblin/orc/ogre as a more generic Shade, though without the more famous name they pretty much just ended up being another type of undead… but that’s what happens when you actually listen to your playtester feedback.

Static Bonuses versus “Advantage”: I’ve mentioned this one elsewhere, but my game began as an attempt at a 4E “spiritual successor” (these days you’d only know that because I told you so) and I always had a great animosity for the way 5e threw several parts of 4E I considered to be great innovations like the warlord under the bus.

So, I was pretty determined to not like the main mechanical innovation of 5e; advantage/disadvantage. I spent years futzing with static bonuses and slowly edging them upwards so they’d actually feel meaningful (instead of the +/-2 of 4E).

But, because a big part of my spiel for this project was to take the mantra expressed in the promotional books for 4E (where the developers said they were going to question everything and not be afraid of killing sacred cows) and apply that to 4E in order to find the best mechanics possible… well, I decided to put advantage head-to-head with the static bonuses that I was sure would win out.

Boy, was I wrong.

I knew it was over when in one encounter a desperate player rolled a 2 on the die for an attack to try and avoid a party wipe (he’d needed a 13+). It wasn’t even a cool failure where you could use a natural-1 to riff off as a critical fumble… it was just a whiff when the party really needed a hit.

Then someone remembered the monster was still flat-footed because of a stunt their PC had performed… and that was the condition I’d tied the re-roll and use the best result mechanic to for the test… and you can probably guess what happened on the re-roll.

Yup… Natural Twenty! A critical hit that actually dropped the monster entirely and pulled victory from the jaws of defeat. Absolute euphoria from the players.

No reasonable static bonus would have even gotten that 2 on the die to even a hit, much less turned it into a critical. The feedback I got was overwhelming with a common theme that one tester best summed up as “It was like a saving throw vs. failure.”

The times it didn’t matter and a static bonus might have made a difference? Basically forgotten; people tend to remember outliers.

I still prefer static bonuses, but they’re an optional rule now because there’s no way a competent game designer whose goal is to make a fun and engaging game is going to not use the mechanic that can produce dopamine rushes like that “save vs. failure” accomplished.

I even built it up too… flat-footed let you roll twice, but it could stack to be vulnerable (a successful hit is a critical). Conversely, hindered was roll twice, use worst, but if an attack gets hindered twice the target also takes half damage even if you hit.

A five step ramp - resisted > hindered > normal > flat-footed > vulnerable - provided enough variety to prevent some of 5e’s issues (i.e. three disadvantages + one advantage = normal roll), but was still simple enough for the people who didn’t want tons of fiddly +/-2 modifers. Best of both worlds in the end.

But I never would have figured that out if I hadn’t been willing to actually test alternatives to what I thought was the clearly superior mechanic.

* * * *

Playtesting is a lot of work. Done honestly, it will kill some of your sacred cows. It will increase your development time because when something doesn’t work you need to find a way to fix it and test that too.

If you’re trying to hit a deadline it’s also one of the easiest things to skip or do in only a perfunctory manner to find typos and glitches that are easy to fix without gutting your system.

It also feels like something you can skip if you’re mostly just using someone else’s OGL/SRD because so much of the core mechanics are baked into that that you don’t think your small tweaks will have that big of an effect… until they do (yes, I’m looking at you, “Armor as DR in 3.5e while ignoring hit point and damage scaling”).

Just as with anything else, testing during as well as after development is necessary if you don’t want crap results.

Kyle Aaron

  • high-minded hack
  • Hero Member
  • *****
  • Posts: 9219
  • high-minded hack
    • The Viking Hat GM
Re: 'B-but that didn't come up in the playtest'
« Reply #3 on: October 14, 2021, 05:01:55 AM »
John Kim lists at least 1,101 different published RPGs, and over 500 free RPGs. And of course there'll be another 500 or so, at least, more obscure and unpopular ones he missed (he doesn't have my latest, for example, and seems to have stopped adding to the list after 2016).

https://darkshire.net/jhkim/rpg/encyclopedia/
https://darkshire.net/jhkim/rpg/freerpgs/

Among over 2,000 published RPGs there'll be one which suits your particular exacting requirements. Don't let yourself be overcome by your cynical and jaded adult nature, keep your adolescent geeky enthusiasm.

Steven Mitchell

  • Hero Member
  • *****
  • S
  • Posts: 2752
Re: 'B-but that didn't come up in the playtest'
« Reply #4 on: October 14, 2021, 08:50:46 AM »
The things done to make a game sell and get a profit are not always the same things done to make it a "good game".  Sure, it would be nice to have it all, but the amateur writer is working alone or close to it and the industry writer is on a schedule and needs to make something that will compete.  Do you want a tight, unified vision or great production values?  In practice, it's rare to have both.  Do you want well-designed and production values and fully tested?  You are going to have to wait on that and pay a premium for it.  Or you are buying from the rare organization that can manage egos enough to let a talented designer drive vision while others help meet it.  Never mind all the outside, societal influences.

First, testing is something that both sides need.  It's also difficult, and done fully, expensive.  So if you are buying from an amateur, there's only so much you can expect.  I'm fairly forgiving of things that would have been difficult to find without extensive, blind testing.  I'm not forgiving of things that would have been easily found by the designer actually trying to use the rules at their own tables.  Especially since at least part of the time, the "reason" for such rules surviving into the published product is that the writer didn't use the rule as written.  I don't mean sort of used it.  I mean fudged it entirely or never got around to testing that subsystem.  If you don't use it at the table, why the hell is it in the product! 

Also, very few organizations have the guts to delay but keep the blind play test, even assuming that they were planning on doing one in the first place.  There's no point in doing a blind play test, or even a semi-blind one, when the game has obvious holes evident in less stringent testing.  That's not to say that the blind play test should be skipped, only that my criticism for many products would be "you weren't even ready for the blind test," instead of "you didn't do the blind test."

Second, what a lot of people want is not a "well designed game".  They want a setting that is, for them, evocative, and any old mechanics that they can semi-ignore to use with it.  So if the designer's purpose is a well designed game, then with budget and time, there's a good chance that the setting is going to suffer by comparison.  Or maybe the setting is evocative, but only for a narrow audience that buys the designers vision.  Given that some things that sell apparently well are well-produced fan fiction, barely even relatable to the word "game" at all, there's apparently an audience for that kind of thing.

Third, at the risk of invoking ivory tower theorists, there is something to the simulation versus non-simulation critique.  I say "non-simulation" because that means game play, excitement, handling time, elegance, and everything else that breaks down in theory when the theorist try to put simulation in one category of many instead of sim versus non-sim.  Anyway, among other things, a game is a model.  There's always a line where the model can cross so firmly into sim territory that the rest of the game will suffer.  Maybe not much, maybe even a good trade, but the trade is inevitable.  The subset of the audience that really wants simulation pushed hard is rarely concerned with "well-designed".  When they are, they are usually highly frustrated at not finding the mythical unicorn. 

As for lists of 100 things, those are very much depend on what is on the list and the particular user.  Every GM has strong and weak points.  Sometimes, a particular list is pure gold.  Often, it has a negative value--the GM's brain can supply a useful and better answer faster than consulting the list.  I personally find that the most useful lists are one of two types.  Either the list is highly focused on invoking the flavor of a particular setting to the point that it is almost useless out of it, or the list is a compilation from multiple contributors.  The former belongs in a wider setting product. The latter needs an editor with a light touch.

Bedrockbrendan

  • Hero Member
  • *****
  • Posts: 12534
Re: 'B-but that didn't come up in the playtest'
« Reply #5 on: October 14, 2021, 09:44:05 AM »
I think it really depends on what you are talking about. If the core combat mechanic doesn't work, and that was somehow missed in playtests, or if no one noticed it just examining the system, that is a problem). But there are edge cases that might not come up even after a ton of playtests. Most RPG publishers are small, and limited to testing things in their own groups, perhaps a few groups of fans who are willing to run the system and give feedback, but that is a much more limited pool of playtests than when it is released into the wild and you have hundreds, thousands or more play testing it, and play testing it with different approaches and ideas. That said, play testing is really important. It shouldn't be an afterthought.

I try to be as thorough as I can when I playtest stuff (I often do a ton of isolated combat only, narrow scenario-only, etc; plus full campaigns, plus one-shots, etc). I have also found it useful, just for vetting purposes to do different things like get a group of players together and have them all select monsters from the monster section to fight one another (just so I am getting as many eyes on the entries, the powers, etc). And I usually have more than one group of players to test things with, as well as a few people wiling to run things in their own games. Even then, I feel like if I had more resources I could do much more thorough play testing. And I would love to have the resources to do revised editions because I track what I find and notice, and need to update after a game is released (I am very seriously thinking of doing a second edition of my games soon). We also work at crunching the numbers and the probabilities.

I will say this, play testing definitely improves a game. And there is an art to it as well. When I first started, I ran a lot of playtests but I knew less about the process, was more easily distracted by things that weren't really problems. The longer I did it, the better I got at knowing how to playtest well. One area that really stands out for me is a game where we had four characters paths (basically character classes) and one of them was a scholarly character. I had a lot of ideas for this character initially, and they looked very good on the page, but they didn't translate into anything enjoyable at the table. It literally took playtesting to make this path work in game, in a way that I found satisfying. And the result was one that I think is the best in the game.

But I think a lot of this is less ivory tower and more a resource issue (at least with the majority of publishers, who are not large). I can't speak to WoTC level play testing. Obviously companies like that can playtest at a much larger scale.

HappyDaze

  • Hero Member
  • *****
  • H
  • Posts: 4227
Re: 'B-but that didn't come up in the playtest'
« Reply #6 on: October 14, 2021, 11:13:51 AM »
I'm fairly forgiving of things that would have been difficult to find without extensive, blind testing.  I'm not forgiving of things that would have been easily found by the designer actually trying to use the rules at their own tables.  Especially since at least part of the time, the "reason" for such rules surviving into the published product is that the writer didn't use the rule as written.  I don't mean sort of used it.  I mean fudged it entirely or never got around to testing that subsystem.  If you don't use it at the table, why the hell is it in the product! 
I'm not at all forgiving of issues raised by multiple playtest groups that the designers just ignore feedback on and publish in an untouched form because none of the playtesters obviously undestood what they were going for (but the buyers will...surely...). Later an "expanded" version of whatever it was will be in a suppliment (and playtested just as badly). FFG was a big offender on this one in their Star Wars lines.

GriswaldTerrastone

  • Full Member
  • ***
  • Posts: 183
  • Just here.
Re: 'B-but that didn't come up in the playtest'
« Reply #7 on: October 14, 2021, 01:30:53 PM »
Well, to be fair, there's ALWAYS something unexpected or unaccounted for, no matter how dedicated a designer is to fine-tuning a project. Especially in today's environment.
I'm 55. My profile won't record this. It's only right younger members know how old I am.

Naburimannu

  • Full Member
  • ***
  • N
  • Posts: 204
Re: 'B-but that didn't come up in the playtest'
« Reply #8 on: October 15, 2021, 03:41:56 AM »
I'm not at all forgiving of issues raised by multiple playtest groups that the designers just ignore feedback on and publish in an untouched form because none of the playtesters obviously undestood what they were going for (but the buyers will...surely...). Later an "expanded" version of whatever it was will be in a suppliment (and playtested just as badly). FFG was a big offender on this one in their Star Wars lines.

There's nothing TTRPG-specific about this, sadly, and it's not just computer games, either: even the largest software companies regularly ignore tester feedback with that same sort of rationale.

Omega

  • Hero Member
  • *****
  • O
  • Posts: 15275
Re: 'B-but that didn't come up in the playtest'
« Reply #9 on: October 15, 2021, 05:11:43 AM »
Its usually the little things that get missed even in playtesting. Sometimes its an obscure combo of powers that are only a problem in some weird one off situation. Other times its tied to the core somehow and is not necessarily a problem in the normal sense.

Some do playtest, but not much, or not in useful ways, or all the feedback gets ignored. Pretty common across the board, and in no way limited to RPGs.

And sometimes theres the weird stuff that isnt really a playtest problem. Instead it is a proofreading error that creeped into final print. 5e D&D first print run had alot and those were corrected in later print runs.

And then theres Bruce Baugh and Lizard, the slacker designer team for White Wolf and "oh we didnt write rules for that. The players will do it for us." d20 modern GW where your character can and will die from the environment just from walking outside, even in full enviro gear.

Steven Mitchell

  • Hero Member
  • *****
  • S
  • Posts: 2752
Re: 'B-but that didn't come up in the playtest'
« Reply #10 on: October 15, 2021, 08:41:51 AM »
Yeah, good testing is hard and expensive.  The GM has a brain and should use it, because the game can't cover everything, even if somehow magically designed, developed, and tested perfectly. 

That's not an excuse to skip testing entirely or use bad practices (thus wasting whatever testing time you have) or otherwise do multiple idiotic things (like make multiple rules so bad that a thoughtful editorial pass should catch them long before testing). 

Tester feedback is probably the hardest part of testing.  It's laughable, almost hysterically bad at coming up with solutions to problem, and often completely off the mark on even the diagnosis of the root problem.  But when the feedback is negative, it's nearly always one of two things: 

1. The tester wants the thing to do something it wasn't designed to do.  Which means you need to discover and correct why the tester got the wrong impression.
2. There is a real problem, but it bears no resemblance to what the tester thinks it is.  Which means you need to dig out the real problem, correct, and retest. 

In general*, people are really bad at understanding why something doesn't work, but really good at recognizing what isn't working for them.

* There are, of course, numerous exceptions to this in any specific field, but even people good at testing and analysis are surprisingly likely to revert to the general norm outside their area of expertise.  That is to say that good testing is a combination of understanding the thing being tested and being good at testing processes.

Bedrockbrendan

  • Hero Member
  • *****
  • Posts: 12534
Re: 'B-but that didn't come up in the playtest'
« Reply #11 on: October 15, 2021, 09:39:15 AM »


Tester feedback is probably the hardest part of testing.  It's laughable, almost hysterically bad at coming up with solutions to problem, and often completely off the mark on even the diagnosis of the root problem.  But when the feedback is negative, it's nearly always one of two things: 

1. The tester wants the thing to do something it wasn't designed to do.  Which means you need to discover and correct why the tester got the wrong impression.
2. There is a real problem, but it bears no resemblance to what the tester thinks it is.  Which means you need to dig out the real problem, correct, and retest. 




This is pretty important: you can mess up a game responding to feedback, as much as you can ignoring it. It takes a while to figure out how to sift through feedback and figure out what feedback is relevant and what is going to make the game worse. Some of my biggest mistakes designing games have been responding to well reasoned feedback that just didn't fit what I was trying to do with the game. Usually the feedback itself is perfectly fine, and well intentioned. It just isn't accounting for big picture stuff, and when you are in the weeds of play testing, you can easily get disoriented. Now I have a much better idea of how to keep things focused.

There is also the added problem that full campaigns and individual playtests are different things. So some things that seem like problems during playtest, turn out not to be in the context of a long campaign, and some things that don't seem like problems in short playtest scenarios, become problems in larger campaigns. And larger campaigns are the harder thing to test because those take time and each long campaign only gives you a narrow glimpse into what other groups are going to experience playing your game.

Another issue worth talking about is time for development. Everyone is different, but I shifted to a minimum of two years for most books (excluding things that are more minor or small scale). The one I am working on now, has been in development since 2017.

Chris24601

  • Hero Member
  • *****
  • C
  • Posts: 2081
Re: 'B-but that didn't come up in the playtest'
« Reply #12 on: October 15, 2021, 10:51:40 AM »
One of my solutions to getting better feedback from my testers was to deliberately test alternatives; my previous example where I tested both static modifers and an advantage-like mechanic was just one example where I did that.

A similar thing that can help is to actually reach out to specific playtesters when you notice something. That was how I figured out the Halfling issue. I noted in the feedback from blind tests involving char-gen that no one was making Halflings. So I started asking some of my testers and the pattern of answers indicated the “too disturbing” aspect.

And I’ll definitely agree you need blind tests if you’re going to do any at all. I found several issues in my char-gen system where people dove right past the section on attributes and ultimately had to ask me how they were determined because they couldn’t find them.

Another was that originally I covered backgrounds before classes, but the classes often offered some extra options to the boon selections backgrounds offered and people running straight through the book for character creativity had already picked their boons before they even looked at what extras the class offered.

So even though your background is functionally more important to defining most PCs than class is (backgrounds are all your non-combat options, classes cover only combat) I had to put classes first in the layout because most blind testers would note the options that way and then also check those when they later reached the background section.

That’s not something I would have caught if I’d actively involved myself in those char-gen tests (I was AT the session so I could observe and if someone was stuck I could answer questions… but just the need to ask me a question was an important datapoint).

Hakdov

  • Jr. Member
  • **
  • Posts: 69
    • Iron Spikes
Re: 'B-but that didn't come up in the playtest'
« Reply #13 on: October 15, 2021, 08:16:31 PM »
Steve Jackson wrote long ago in his book on game design on the value of playtesting, especially blind playtesting.  And you can tell that he actually practiced what he preached from the very tight designs of his early games like Ogre and Melee. 

Shawn Driscoll

  • Role-Play Purist
  • Hero Member
  • *****
  • Posts: 2764
Re: 'B-but that didn't come up in the playtest'
« Reply #14 on: October 15, 2021, 10:01:03 PM »
As to "where's the quality?", consider yourself a gold miner. You gotta hack through a lot of not-gold to hopefully find the hidden gold. And once you find that "golden game", play the damn thing with lots of friends and sing its praises for the rest of us.
I recently discovered a fairly new game designer that has been producing game after game each week or two, it seems, for the last couple of months. I plan to do videos once I finish my reviewing.

It's been years since I bought an RPG, I think, other than RPGPundit's books.