Two perspectives on sales and luck

Jake Birkett, as is his way, posted a tweet that prompted some reflection. Here it is:

He may be responding to this thread by Momiji studios, though if he intended to make the association he would have done so. I am sympathetic to some parts of the thread, particularly the idea that data on successes are more widely shared than other results and its potential to distort our views of the marketplace. Where the thread loses me is an overemphasis on luck.

The thread ends with some distain for comments or criticism from those who have not made a game, and this is a criticism I need to take seriously. In any article I write I do so with the knowledge that I do not have experience releasing games on Steam. It is possible, however, to take the claims of the thread at face value and see if they lead somewhere useful. Here I will present two perspectives on game sales and luck. Ultimately both perspectives lead to a similar outcome, informed by one of Jake’s older articles about how developers are taking too long on their games.

An odd way of looking at development

I will start with a strange way of looking at game development. I propose that when we are talking about the success of a game with regards to sales we are discussing some function f(x, y, z,…) where x, y, z, and any other inputs are all the elements that go into making a game (f(programming, art), f(hours senior programmer, hours junior programmer, hours senior artist, hours junior artist, dollars spent on Reddit ads, number of hours streamed,…), whatever you can imagine). The function defines weights and relationships between these factors, and the result is the number of units a game sells. Developers act as if this function exists, and when we discuss game development, particularly with regards to what produces certain outcomes, we are trying to gain a better understanding of this function.

The ‘as if’ is important, as is the need to be clear on what is being claimed here. I am not claiming that developers consciously think of this function, or that knowledge of this function is a necessary condition to successfully produce a game. In fact, my own belief is that successful developers understand these relationships, either through intuition, experience, or any of the other ways experts gain their talents, and that analysts, not being endowed with such gifts, need to model this intuition in order to study what developers are up to.

When performing statistical analysis, the parameters for some functions (usually ones less complicated than whatever the ‘true’ f(…) for game development might be) are estimated by specifying an error term. The error term exists because we cannot specify all the factors that determine an outcome (in this case game sales) and so the mathematical concept of randomness represents our ignorance about the world. In order to get useful results from a statistical model certain assumptions must be made about the error, and a common one is that, on average, the factors we haven’t accounted for cancel each other out (for stats mavens: the mean of the error term is zero). This means there are two kinds of unexpected outcome. There are unexpected outcomes that are a natural result of this imperfect understanding, and there are results that are the result of incorrect assumptions about the state of the world. The former leave our understanding of the world intact, even if the specific case observed is unusual, while the latter means we likely have a distorted view of even the parts we can measure.

While this way of looking at development has been driven by statistical concepts, what is really being discussed here are ways of thinking about game development. Both perspectives are defined by their relation to randomness.

What if it really is all luck?

Momiji’s perspective is to say that one of the factors that goes into the function is a random term, which we can simply call luck. Momiji is not claiming that hard work or any of the other factors are irrelevant. The point is that luck is one of the factors of production and that is possible for a game to do everything else right and ultimately fail due to the randomness inherent in game development. Quality, defined as a minimum level of all the other factors under a developer’s control, is a necessary but not sufficient condition for success.

If this is the correct way of looking at development there are many reasons why we would be reluctant to accept it. Successful developers are going to want to focus on the work they put in and not consider the role something outside of their control played. People about to start a project are not going to want to think the resources being committed to it are being gambled. The only comfort this perspective provides is that, past a certain threshold of skill and effort, the outcome is determined by chance and so those who were unsuccessful can be consoled with the thought that there was nothing that could have been done about it. In short, few people will want to believe this, and those who do will not have a big platform.

We may want to test to this hypothesis by considering the number of developers who have had their first successful game and then move on to establish a track record of success. However, a lot of games are released, and with enough time and releases we might expect chance to produce a few track records. I am not aware of a systematic study of Steam incumbents, but a cursory glance seems to suggest that successful developers have more than one success, while pure chance would suggest a larger fraction of successful developers should see failure. The problem is that a cursory glance isn’t seriously engaging with Momiji’s challenge.

A better approach is to accept that games are contingent on luck and see what follows. If, past a certain quality threshold, games are a matter of luck, then the best strategy is to play the odds.

Playing the odds means only spending the minimum required (cash, effort, or time) to be subject to chance and making as many games as possible in order to achieve success, assuming the expected return is equal to or greater than what is being spent. If the expected return is less than what is spent, then we should exit development. Momiji is not advocating giving up and so we assume the expected return is greater than what is spent.

A natural objection to playing the odds is that this is only a concern if the goal of development is the financial return but this approach is, in fact, intended to ensure that development can continue. Most systems for profiting from probabilistic outcomes (think gambling systems or stock market strategies) rely on risking the right amount on a given set of circumstances. We would never see such a system say “when the probability reaches 99% lever up and put everything you can on the outcome” because the 1% event can and does happen. A casino doesn’t stress about individual losses because it operates in such a way that the exceptions get washed away and the underlying probabilities hold. Making as many games as possible is the casino approach in that it both maximizes profit and minimizes the probability of the developer getting wiped out.

Consider the probabilities of success. Momiji offers 10/6000. It is fair to say that not all 6000 games released did (or would) not meet the minimum threshold for quality, but I do not have a ready estimate for this threshold and so we will go with the implied estimate of 0.17% probability of success. Would you bet everything on this? Of course not. Making 10 games at this probability raises the chance of at least one success to 1.69%. 20 games raises the probability to 3.35%. If it takes 3 years to make a game, then at 20 games the next set of probabilities that need to be consulted are actuarial tables. At 100 games the chance of at least one success is about 15.56%, a bit worse than getting a 6 on a die roll (D6 for the tabletop nerds).

Here the role of the publisher is to create a portfolio of games that will cover the losses of the majority to gain the benefits of the few that do succeed. But there aren’t any publishers releasing 100 games a year, and even at 100 games, it is hard to see any investor backing such an enterprise unless the rewards were exceptionally large (why invest in something that will lose your money 5 times out of 6 when you can buy shares in P&G?).

If this model of game development is correct, and the developer is running things as a business, any effort spent on a game past hitting the minimum quality necessary to be eligible for the lucky draw is wasted. Extra effort should be strictly left to hobbyists or projects supported by grants because it comes at the cost of another attempt for success.

This model does not see much external validation. Not only do publishers support fewer games than required to make the probabilities make sense, the acceleration of new games being released is coming from new developers, not incumbents using the profit maximizing strategy. Fortunately, there is another way of thinking about chance.

A measure of our ignorance

Returning to the strange way of thinking about development as a function, we know that we can leave chance to the error term. There will always be factors that affect sales that we can’t account for. If news arrives that a secret lunar colony has been established and has been running successfully for the past year on the day you release a space themed city builder, you are going to be very happy. If some calamity knocks out the power in the United States for a week during your launch, the game will not sell as well it would have. This perspective is distinct from the previous one in that randomness is not a factor of production but rather a way of accounting for these and other contingencies.

Because the error term’s relationship to the outcome can usually be summarized as “cancels each other out over time” it is important to make sure that factors that do systematically affect the outcome are accounted for. The heuristic is a statistical model, but the result of failing to account for all the factors is the same even if we aren’t running a regression: we misjudge the importance of the factors we are measuring, assign too much emphasis to factors outside of our control, and expect the wrong outcome. I think this is what Jake is getting at in his second tweet when he quotes Arthur C. Clarke. Failing to understand the mechanisms for our success or failure may feel like chance, but it does not mean those mechanisms do not exist.

Unfortunately, very few of us are gifted with the kind of insight that allows us to get it right the first time. The benefit of acknowledging assumptions and thinking through things clearly is that it provides us a system for improving. If randomness represents the things we don’t know, as opposed to an inherent factor in the production process, then the next step is to find out about the things we don’t now.

For games experience is almost certainly the best teacher. Indies that seem to come out of nowhere with a breakout hit often have a team that started in established studios. Others make things and learn as they go. And yes, some people just seem to be endowed with a knack for picking things up quickly or having good intuitions. In the model experience provides developers more knowledge about f(.) which is applied to their games, consciously or intuitively. Sometimes you need to burn your hand on the stove to figure out not to do that.

What does this mean for those of us who are not endowed with a natural understanding of game development? It means we need to learn, which means we need to make things. This is general enough to encompass working for someone else to working on one’s own projects. This advice isn’t a secret, as countless aspirants have asked their idols “how do I get into games?” and the answer is some variation on making and finishing things. This perspective offers some elaboration on that advice. You make and finish things because doing so gives you information. Information isn’t the only thing (if the work isn’t a reward then why bother doing any of this?) but it will arm you for larger and more ambitious projects.

The recommendation is similar to the previous section: make things faster. The difference is that the faster pace is not necessarily a permanent state. One of the main reasons to learn is to get to the point where we understand the allocation of resources well enough to know when a large ambitious project is worth it. There will always be exceptions, but an ambitious project too early is a costly way to learn the lessons a smaller project could have taught.

What do we learn?

Framing game development with an abstract concept like the f(.) may create more confusion that it’s worth. Its purpose is to make each perspective’s relationship to randomness explicit. The two perspectives recommend a similar course of action in terms of keeping costs (if only in time) low, but the reasoning is very different. When randomness is a factor of production, spending anything other than what is essential is the kind of waste that only developers who are sustained through other means can afford.

In contrast, considering randomness a measure of our ignorance allows that development can be better understood, and that this knowledge will allow for greater choice in terms of the types of projects and their costs. This perspective has not only more appealing conclusions, but it also has the ability to explain why developers may believe that chance plays a greater role than this perspective would allow. It is an empathetic view. We can’t expect a solo developer who is responsible for everything to have a clear idea of the factors that will affect a game’s success, and so it is understandable why they will be under a misapprehension about chance. Remember, the factors that aren’t accounted for don’t just create a false impression about the outcome, but also the factors that are measured. Learning in such a situation is not easy.

While learning can be difficult, the second perspective remains more hopeful. This article has focused on learning the function through experience, but another way a project can gain experience is through collaboration. Collaboration brings both the knowledge and experience of the collaborator, as well as the knowledge gained from a more focused approach to one’s own work. Hope does not necessarily translate into success, but in this case the more hopeful perspective is aligned with the actions that very well might.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s