ANALYTICS? IN MY INTERACTIVE FICTION?!

Independent producers are not always comfortable using analytics and might link these sorts of metrics to AAA or free-to-play games. But this sort of data can provide valuable feedback about how players experience your interactive media. In this Narrascope 2020 talk Em Lazer-Walker, a Toronto-based artist, engineer, and game designer discusses the benefits, techniques, what to measure, and what not to measure.

My name is Em, I use she her pronouns and I work for Microsoft as a cloud advocate.
I also do a lot of experimental indie game work on my own.

This is a talk about analytics.

It’s in three parts:

1. How free to play games measure things?

2. What is applicable to smaller games and more experimental games?

3. Is there value in adding analytics to your game? How would you actually go about doing that? What are some practical tools you can use?

1. HOW DO COMMERCIAL GAMES USE DATA?

I do not generally endorse the way that commercial metrics analytics works but learning how analytics functions is going to be really useful before we talk about how we can co-opt these techniques for other things. This is an abstracted overview based on my own experience working in free to play games many years ago, as well as talking to some folks I know who have done it more recently.

So if you are a product manager (pm) on a free to play game, you're looking at your game as a product that converts essentially user eyeballs into money. And in order to do that you have the sort of funnel that you narrow in on. You're going to get some new users to download your game, they're going to play it in 2020. This is largely through paying money to acquire these users through channels like video ads. Of all the people who download and install and play your game, some number of them are going to come back the next day, or the next week. Not all of them, but hopefully a lot of them. Of the people who keep coming back and are excited enough about your game and the stories you're telling to keep playing it. Some of them are eventually going to give you some money. And then, of all those players who are giving you money, hopefully some of them will keep giving you more money. A lot of times the vast majority of revenue come from a tiny fraction of players. Again, I am not endorsing the system, but this is the way things are. And so if this is what you're looking at, you can see how a numbers map become important. Say you're losing people at each step of the funnel, if you can stop the bleeding and slightly increase retention, perhaps you can slightly increase the average amount of money that a paying player gives you. That’s gonna make you a lot more money.

So we need some metrics to test that. For the most part, there are really three high level guiding metrics or classes of metrics that you look at for this funnel.

• RETENTION

The first is retention, so are people coming back and playing your game? This is typically measured according to what percentage of players are coming back a certain number of days after installing the game.

So, day one retention is for everyone who downloaded and installed your game, what percentage of them came back the next day to play it again? Day seven is how many players then logged on seven days later to play the game, etc. You don't usually see this measure beyond day 30 which is also really interesting.

So, if long term or short term retention immediately drops after you introduced a new feature that can be a clear sign that you did something wrong.

Retention ends up being used as a proxy for fun, because analysts assume if players are coming back day after day, they probably like the game and are having fun. Although I would argue that's a really tricky assumption. Are they coming back because they're addicted, or they actually enjoy it? To figure out that distinction, analysts typically employ what's often called cohort analysis where instead of just looking at overall retention, they will look at retention among a group of people who start playing on the same day, which makes it easy to look at how things change over time.

• REVENUE

Next is revenue. There are a million different ways to slice and dice this based on are you talking about revenue per user across your entire user base or just for the people who are paying you? Are you looking across the lifetime of a player or just during a single day, etc. So you end up with these really fun acronyms like ARPU versus ARPA, but money is money.

• REACH

And the last one is reach. Reach retention and revenue is what we used to use when I worked at Zynga about a decade ago, in the era of Facebook games. The reach metric measures how new people are coming into your game at the top of the funnel. And the way that new people come into your game is through virality. So you design things into your game that entice players to bring their friends.

That is not how things work in 2020 however. These days, like mobile free to play, which is where most things are, is all about paid user acquisition, primarily through video ads and people spend time crunching these numbers to a flaw. But this is not a thing that game designers worry about anymore. This is like a user acquisition team somewhere else dealing with ads.

HOW DO YOU TRACK THESE THINGS:?

So if we have these sort of three guiding metrics, the question then is what do you actually add into your game to track things? And so typically, a product owner on one of these free to play games will add tracking to the games with an eye towards answering specific questions. Like are people playing your story the way you expected them to? Are people playing it at all? Are they getting all the way to the end?

You also might have a number that represents the overall health of your game in a sense. The number that we really cared about was the total number of turns played per day by all players across the entire game. So, when you have 25 million daily players across all platforms, that's a very large number. But it was still useful to look at how that number fluctuated either over an entire day, or even by noon. That's a really good sign that, you know, maybe we've shipped out some bad code that's breaking the game or some new feature is actively hurting things. Or perhaps there is a real world event that's affecting things that we need to look into and think about. Like, if you look at stats right now, people are playing way more mobile games than they were a couple months ago, because it turns out, when you're sitting at home with nothing to do, you're going to download a lot of free mobile games.

If you know, you really need to get your attention numbers, your revenue numbers up, that is going to influence storytelling and monetization decisions, based on what you know about your audience.

2. HOW MUCH OF THIS IS RELEVANT TO EXPERIMENTAL OR NARRATIVE GAMES?

If you are working as a narrative designer, or a writer or a game designer or a programmer, you are likely isolated from a lot of this. But it's possible to use some of these same things divorced from the larger context, to get a better sense of how players are interacting with your game and engaging with the stories you're trying to tell in ways that are qualitatively different from what you'd get from doing something like in person play-testing or user feedback sessions or reading whether your community hangs out on discord or their steam reviews, or whatever.

We're not looking at individual players. We are really interested in numbers as a way of looking at large groups of people to look at player behaviour in aggregate.

A project that I worked on at my day job, for example, which was a little twine game that's essentially a short escape room. So you're in the spooky old house. And he has all a bunch of puzzles and so I built out this integration between twine in a tool called playfab, so that anytime a player clicked on a link, or loaded a new twine passage, it took all of that information about the event and what the player is doing and where they are in the game world. I had access to this data via a sort of Visual Dashboard where I could write complex queries around that data set to see what players were doing.

We broadly found gathering these analytics to be incredibly useful for answering a pretty wide set of questions and informing different design problems.

Many of you may have used interactive fiction tools that have some sort of automated mechanism for validating that you don't have any impossible states in your game or you don't have any sort of pieces of content that are technically impossible for anyone to get to usually sort of brute forcing their way through the game as an AI.

For us, once we had some number of play testers, analytics, were able to serve a similar goal. So we can go through our web interface and run some queries and figure out if there were any twine passages that had zero visits, given the number of users we had, it was pretty likely that this was not actually accessible in the game itself, which was really useful.

We ran into a situation where all of our play testers that we watched in person play the game had no trouble with the first of two big puzzle bottlenecks in our game. But then once we had way more users many of the players were not people who traditionally played games and were unfamiliar with the puzzle conventions. When we ran the numbers we found that many of them were not getting through that bottleneck. So, it was very easy for us to look at the data and say great cool, we need to more aggressively hit that puzzle, which fixed that problem and we had the tools to know that that problem was fixed by looking at the relative visits of the passage before that puzzle in the passage after it.

This also then lets us look at nonlinear spaces to find the paths that people are playing. W
We could see what people had seen versus not seen of a game during a play session. We also could track the order in which rooms were being visited, which led us to make a few changes about the way different puzzles were previously hinted at in different spaces based on incorrect assumptions about which older players would visit rooms.

The awareness of that discrepancy between your expectation to player behavior, and actual player behavior is going to make you stronger as a designer.

A thing that we did not do, but we really wanted to if you'd had time, was implementing the tell-tale style, actually showing players their choices and context of what other players chose as well. And for me as someone who has some reservations about whether it is ethical to gather this playthrough data from players at all, it helps to be able to concretely, take all this stuff you're gathering and not just use it to inform your own decision making, but to actively give players a gift back that enriches their experience.

At the same time the numbers are not the goal. A good pm is going to know when to ignore the numbers and go on intuition.

3. WHICH ANALYTICS TOOLS TO USE?

If you've ever tried to use analytics tools, it can be overwhelming because there's so many options. I think in broad strokes, though, they fit into a couple of buckets. So, if you are working at a large game studio, you will have in house tools, you might have a team of statisticians writing raw SQL queries, that's not really relevant to us. So you'll probably end up grabbing a tool built for web developers just because that's where the audiences for these sorts of tools is. These can be great, but you have to take these tools that were not meant for games and adapt them to games.

But, I worry about the ethical considerations of what is happening to your data. If you're using something like Google Analytics, Google is a large tech company that essentially runs by selling user data and ads. You may not even be a paying customer of Google Analytics, so what are they doing with all of this data that they’re collecting from your users.

Playfab
Personally, these days I mostly use a tool called play fab, which Microsoft acquired a year or two ago, which I think is great. Because however you might feel about Microsoft as a company, they are not in the business of advertising or user eyeballs or anything like that.

Playfab is this gigantic overwhelming game live ops tool. They basically want to be everything you need in order to run an online game. So large MMOs are using them for everything from matchmaking and party management to voice chat to economy management to cloud saves, and leaderboards and achievements. There's a whole lot in there. But also, crucially, they have a really nice analytics module that works really well because it is a games tool. It understands how to provide analytics for games really well. And because it is this tool that is meant for these super huge games.

If you make narrative games, you're probably going to fit well within the free tier instead of having to pay them. At our peak, we had about 180,000 daily active users and we're using playfab for much more stuff like cloud saves on leaderboards, etc. And we're not paying them a penny, which is awesome.

There are basically five or six lines of JavaScript you need to copy into your twine game, you will need to paste in a magic playfab ID that you'll get when you sign up. And even though I'm doing automatic tracking of passages, I track twine variables but that's more manual. So if you want to track variables, you need to type in the names of those variables so I know what to look for. And then if you do this with your twine game, again, this is free on GitHub, you can grab it yourself. This is going to give you two different things. The first is it's going to send you this daily email, giving you these high level metrics about who's playing your game. Are they new? Are they coming back? What's the retention? If your small team working on a game and wondering if people even playing it, this can feel really, really good.

The other thing that play fab really gives you is an analytics dashboard where all of this data is available. So for that example I gave of a puzzle that not enough people were solving, the way that I was able to tell that was I was able to write this custom query…show me all of the number of passage visits for, you know, the passage before the puzzle in the passage after the puzzle. And we can get really clear visualizations of how many players are reaching each of those relative passages over time.

So if you integrate playfab into your game separately you can get all of that, like the API's for just sending data to play fab are really, really simple.

I can recommend the general approach of just tracking every unit of content as if it were on its own if you don't have much more specific focused ideas of questions you want to answer, but really, anything works.

Lazer-Walker, Em. 2020. Analytics? In MY Interactive Fiction?! edited by Narrascope 2020. U.S.: YouTube.