Response: Data science will not save you

This article is a critical response to the Gamasutra blog “Steam has a lack of data scientists.” While it is an opinion piece, the article was widely shared and contains a number of contestable statements with regards to Steam’s role, Valve’s capabilities, and the ability of data science to solve structural issues for game developers. This response is an effort to contribute to that discussion and attempt to clear up common misconceptions that appear in the article.

Valve already has data scientists

From the Valve website, four people are identified as data scientists, through its famous workplace culture allows some variability in that number, and it is possible that some of the people listed as other experts are working with the data science team. In addition to their existing resources, Valve has four separate listings under the data science jobs heading (economist, psychologist, statistician, and a general application), communicating an active interest in acquiring more talent. From recollection, Valve has had postings for these kinds of roles for at least a decade, and it seems that every public remark Gabe Newell made that prompted articles asking about Half-Life 3 contained some statement about making sense of the data Steam was generating about its users. While Valve’s efforts may only be fully appreciated now through changes to their algorithms, the company’s use of quantitative data to shape how we buy and play games has been out in the open for at least as long as we’ve been siring bastards in Crusader Kings II.

The title talks about a lack of data scientists, while the body instructs Valve to hire them. It is not clear if the author thinks that Valve does not have any or simply that it needs to hire more, but the tone implies that there are some easy wins if the company would just hire some people who have read James, Witten, Hastie, and Tibshirani (2013). The problem is that the kind of analysis that produces recommendation algorithms does not lend itself to throwing more bodies at the problem. To be clear, a recommendation engine (among Valve’s other data intensive work) is a big job that should take time and collaboration to do well, but the marginal benefits of adding a generic data scientist are not the same as a manufacturing plant adding extra hours to produce more cars. The post is operating on the assumption that Valve does not have people with sufficient knowledge of data science, which is counter to the evidence and uncharitable to the analysts currently working there.

Does Valve have a data problem?

The article’s arguments raise a question as to whether or not Valve’s problem is a data problem in the first place. To put things in perspective, there are 30,000 games on Steam right now and the author’s account contains 355 titles. The fact that an automated process can find anything a user likes is itself an accomplishment, and a success rate above 1% would be better than the author’s own curation of their library (assuming algorithms had nothing to do with these purchases and that all 355 titles were purchased because they were appealing, rather than appearing in bundles or giveaways). If a user only looked at the recommendation queue on weekends (not counting holidays) a success rate of 8% (likely higher than comparable recommendation engines like Amazon’s) would turn them into a “power user” (by the standards of the article) by the end of the year. It is easy to look at a queue filled with games you are not interested in and call it a fair, but this is because of an unrealistic expectation of what success looks like. Even though the hypothetical weekend queue viewer above would become one of the rarest and most valuable Steam users (again, by the standards of the article), they would only do so after passing over 1,000 recommendations they weren’t interested in.

The source of the complaint is a perfectly understandable grievance against changes to the recommendation algorithm that has resulted in reductions in wish lists and other forms of user engagement. Leaving aside the fact that these changes almost certainly came from well informed data scientists (who are supposed to be wearing the white hats in this story), the essence of the complaint is that a group of developers are finding it more difficult to connect with their intended audience after the changes that Steam has made. Changes to the algorithm are the most recent problem, but the article identifies the original sin as the introduction of Steam Direct, which permits any developer with $100 (recoupable) to distribute a game that meets common sense standards for acceptable content. The proposed remedy is to eliminate Steam Direct and instead curate the storefront.

There is nothing inherently wrong with promoting curation since this is how most competing platforms (Origin, Epic, Humble, GOG, among others) work and is the standard outside of PC gaming, though it almost certainly does not align with how Valve sees Steam and wants to strategically position themselves. However, the proposal is at odds with the article’s core message of Steam requiring more data science work. Curation assumes worthy titles will be admitted onto Steam (the threshold presumably falling below whatever the proponent is currently developing), giving most games priority placement on the front page due a fewer titles entering the new releases list. Being able to discern consumer tastes enough to permit some titles on the platform but not others but not being able to connect those judgements to classifications of users seems like an odd system (though classification problems are one of the things machine learning algorithms are good at and so a charitable interpretation would be seeing this as proposing an alternative division of labour).

It is true that the more permissive Steam becomes, the harder it is for a single title to break out simply by showing up. There is at least anecdotal evidence that it is becoming harder to find games because the genres are filled with inappropriately categorized titles (the strategy page list games categorized as ‘indie violent action’, ‘sexual content nudity casual’, ‘early access indie puzzle’, ‘action casual indie gore’, ‘sports simulation indie’ and ‘rpg lovecraftian horror indie’ on the new and trending bar) but this must be weighted against the fact that Valve has been trending towards a more open platform for years and has been able to see exactly how it affects their bottom line in real time. Steam’s adjustments to its algorithm suggest they see room for improvement, but that the tendency towards a more open platform has been successful enough to keep on that path. The author seems to have taken the wrong lesson from Galyonkin’s article: Steam doesn’t have a data problem, developers do (though Galyonkin has been clear he does not think an algorithmic approach is the right one. I touch on that in another article).

Data science alone cannot save developers

When people are referring to data science they are most likely referring to the proliferation of machine learning techniques applied to different problems. These techniques are very well suited to prediction problems, defining prediction as filling in information that you don’t presently have. The applications to storefronts like Steam are clear, since players are unlikely to volunteer a classification like ‘strategy gamer’ but their purchase of Civilization VI and their playtime in XCOM 2 reveals something about their preferences. Steam has a lot of data to work on these problems with and will use it to maximize their objective which is to maximize spending on their platform.

Developers have a different objective in that they want to maximize spending on their specific games. At least some part of the wish for the good old days of Steam is predicated on the assumption that the game an individual is working on will be among those selected in the less competitive environment. The hope for supposedly better applications of data science is similarly founded on the idea that such an algorithm would lead more people to the game the speaker wants them to pay attention to. This is not the only possible outcome. As an algorithm gets better at matching players to games, less efficient matches will give way to more precise matches. The player may have wishlisted both games proposed under the two different systems, but the better match is shown first, meaning that on the margin the less efficient match sees a fall in wish lists. If the rate of wishlising goes up overall, the analyst will deem the change a success, while the developers who see a fall in sales and wishlists will declare it a failure.

With an increase in the number of games being released, a sufficiently powerful prediction model could very will produce an aggregate increase in wishlists with decreases for substantial numbers of developers as the handful of players who will fall in love with an exceptionally niche titles finally find it and move out of the second or third best that was previously recommended. This is an extreme example, but it is intended to illustrate the how Valve’s priorities are going to be embedded in the implementation of any of the proposed changes. An even more extreme application would be where the algorithm is so successful that Steam buys a game for a user in advance and lets them play for 2 hours before charging, secure in the knowledge that the extra sales will pay for the returns of false positives. Developers whose games are not pre-purchased would feel shut out from the platform, even if more games are sold overall. That scenario is good for Valve, arguably good for consumers, and uses data science, but is not something any developer would feel comfortable with.

Developers have a harder data problem in finding people to play their game, since the necessary input of the data itself is much harder to come by. There isn’t even agreement on the best way to gain information on a potential audience in the first place, since Galyonkin’s article puts less weight on the predictive power of a player’s library, while Chris Zukowski’s recent blog does assign at least a type of preferred game. Publishers almost certainly have a bigger role to play in this environment, as they already have an idea of a target audience as well as a better ability to obtain information about a potential audience, though this option may prove unpalatable to some independent developers.

The original article is correct in saying that it is in Valve’s interest that it remain a viable platform for developers, but it does not mean that their interests are completely parallel. The structure of the industry cannot support the number of developers working on the types of games they are currently working on at the current prices. It is just as likely that sophisticated applications of data science will hasten the exit of some developers as it is that it will guarantee some floor of sales. Some of Valve’s changes can be seen as positives, since the tagging system and upcoming releases did seem to be abused by some developers trying to get placement. However, these are better used as calls for transparency and stability on the part of Valve, not an invitation to attend a lecture on convolutional neural networks. Any policy recommendations to Valve should always be viewed with the knowledge that they will only be implemented to the extent that they advance their own interests.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s