Retrospective of A Notebook Prototype

This is cross-posted from my personal blog.

Play A Notebook Prototype Before Reading This!

This week's post will be a mini retrospective for my recent prototype, A Notebook Prototype. I will try to avoid spoilers by being vague, but reading this will likely have some kind of influence on your experience of the game, so I recommend that you play the game before reading any further.

What were the goals?

The goals of this prototype shifted dramatically over the course of development. It originally began as an effort to create a system that could read unstructured text to determine what the player knew (or thought they knew.) While developing the system, I discovered I had no way to measure its fitness, so I couldn't tell if I needed a system capable of Natural Language Processing, or if I could make due with something simpler. I considered how to gather this data with a paper prototype (could I have players play an existing game, and record what they knew as they played it?) but I realized that the way the player is engaging with the work would be important for the accuracy of the data, so my best bet was to build something from scratch that approximated the way I wanted players to engage with the mystery. Since I was already building my own game at this point, I thought I should also include the (already working) simple version of the aforementioned system, so that I could also gather real-world data on its effectiveness.

Once I was deep into the work of authoring the content, I realized that the same prototype could provide other useful data as well. I had no idea how well any of my prior storytelling experience would apply to a branching narrative, so I thought the prototype could help me test if I should be studying writing branching narratives as much as art and music composition (two skills I know I still need to massively improve.) The prototype also seemed to intersect with narrative themes I intend to use in the final project, so I thought this could serve as a sort of canary to detect if players might not readily connect with the theme. Finally, one of the constraints of my writing is that my work tends to have a certain kind of voice (distinct from the individual characters' voices). I usually consider that a strength, but I am aware this bias comes from people who already like me telling me they like it, so the prototype could serve as a way to identify if that opinion is particularly niche or more common.

So, in summary, the final goals were:

1. Record how people use the notebook
2. Test the fitness of the simplest possible notebook implementation
3. Improve my estimate of my storytelling skills
4. Catch if people hate my theme
5. Have a tool I can use later to measure interest in my narrative voice (to better guess at how niche the final product will be)

Did it meet the goals?

I consider the prototype to be a success, though its most interesting piece of data was that I was (in effect) asking the wrong question. Every single play-tester that I observed used the notebook differently, and none of those players (or even myself) ever used it to record insights (though one player came close: that player constantly rearranged the notebook contents so related topics were close to each other.) What this meant, in effect, was that players were recording a variety of different things (what they thought they might need later, what they wanted to ask about, etc) but the one thing they weren't writing was what they knew (which was the one thing I wanted the notebook to detect.)

After reflecting on why that is the case, I think it makes sense: people generally record things they might forget (such as passwords, dates, or clues), while insights (the "a-ha!" moment in the mystery) are generally unforgettable. Even if that isn't the exact reason why, the fact that people don't naturally record insights in their notebook has thrown a wrench in what I've been planning for the larger game, but I am incredibly grateful to have found this with a prototype, as it gives me the chance to revise how I intend to approach engagement with the narrative in the larger game.

Beyond the data about how players used the notebook, the feedback was a lot more straightforward. My storytelling skills seem to transfer better than I expected, but the mechanics underlying how I gated content often interfered with immersion (several play-testers said the story was really good, but several reported feeling like they were redoing content and picking options they didn't want to use because they couldn't figure out how to progress further.) I did not receive any complaints about the theme, and some of the play-testers I observed seemed to emotionally connect most strongly with the sections of the game that aligned most closely with the theme (though I am not sure how much of a role the theme itself played in that.) Lastly, regarding measuring interest in my narrative voice, that mostly remains to be seen, but having it on itch should make it easier to link people to, if I need to ask people to play it for a couple minutes and tell me if they like the writing style (in hindsight, a mobile-friendly browser game would be better for that.)

What other lessons did I learn?

Beyond what I had intended to explore, I also learned a lot of unexpected things through experience. The first one I noticed was how the organization of the branching narrative played a huge role in how maintainable it would be. The entirety of the game's text is stored in a single yarn file, which made it easy to handle interconnectedness and return trips, but it became difficult to find individual segments of dialog, and adding new content often risked overlapping with existing content. Using separate files (possibly with well-defined connections between them) could help avoid creating such a tangled mess.

_{The narrative is actually quite structured, but you definitely can't tell by looking at it in Yarn.}

The second lesson I learned was that the dialog for certain characters was relatively easy to organize, whereas the dialog for other characters (particularly characters with free-flowing conversational style) were much more difficult to organize. The organizational challenges were usually minor inconveniences when going through to fill in all the available choices, but they became quite painful whenever I had to make structural changes to the conversation in response to play-tests. This is useful information, as it will help me know which kinds of conversations may require additional files (or even new tools) just to keep everything maintainable.

A third lesson was that endings are very difficult to do well. Some of that is likely due to inexperience, some of it due to a lack of planning, and (most likely) some of it is due to the fact that endings are genuinely more difficult than other content. Writing the ending last is also a double-edged sword: on the one hand, it can help to have all the details pinned down, so that those details can be referenced accurately in the content of the ending, but on the other hand, content created towards the end of any project is usually subject to more intense pressure to be cut or simplified (to control scope) even though the ending is (most likely) the last thing players will experience (and the last experience is often most influential on one's opinion of something.)

The final lesson (and possibly most important in the short-term) is that getting play-test data is difficult. For my bi-weekly projects, I didn't have time to actually incorporate feedback, so watching people play it was nice, but hardly necessary. For the notebook prototype, though, I knew I needed actual real-world data, as there was the risk that some part of it would distract (or even repel) players to the point that it could fail to provide adequate data about notebook usage. Although the people who agreed to play-test it were generally very helpful, the process was very frustrating and time-consuming (from the start of reaching out until I stopped gathering data took about the same amount of time as it took to create the prototype in the first place.) During this time, I particularly struggled with how I should spend my work days: often I was waiting for someone else to make time to play the game, so I was torn whether to reach out more often (and risk coming across as nagging) or whether to start working on an unrelated project (which would come with a context-switching cost, especially if I was in the middle of a problem when someone provided me with feedback.) Unfortunately, I don't (yet) have any ideas on how to avoid any of the issues related to getting play-test data, but I think knowing about them in advance might help me plan better to deal with them.

What's next?

Since I have the data that I need from it, and the work is effectively done (content is complete, even with multiple endings) I won't be actively working on the notebook prototype any farther (though I will make an effort to fix any bugs and the like.) Whether the notebook will even appear in the larger project I have planned is now unclear, so I won't be putting any effort into polishing the notebook user experience until I figure that out (I've already sunk too much time trying to wrangle Unity's scrollbars to work with that textbox.) Instead, I will be (and have been) looking at the mechanical and aesthetic side of things, to prototype what I can of those aspects of the larger game. In fact, next week I will have a tale of one adventure developing one mechanical prototype, so I hope you'll join me (on my personal blog) then.