As well as being an industry with greater revenue than the global music and film industries combined, gaming is also a domain of profound skill development. Gamers exist in their millions and invest significant time in competing and improving at their favorite games. Automatic tracking shows that the average player of a successful game may play hundreds or thousands of hours, with the most dedicated clocking up more than 10,000 hours1. This is all the more impressive given that such tracking only represents time actively playing the game, not peripheral aspects of practice such as time thinking about or discussing the game, or out-of-game training of components of play. Data from game play has the potential to be very rich, going beyond mere records of match outcomes or scores and including records of every action taken by a player during a game. A key benefit being that every record of play is a record of skill practice for subsequent play
When compared to novices, experts anticipate better, react faster, organize behavioral sequences and strategies differently (Ericsson et al. 2018), and even show different neural responses to domain problems (Bilalić 2017). Comparing individuals of different skills levels has provided us with a good understanding of the underlying cognitive mechanisms which distinguish experts from novices - for example, de Groot’s classic work on chess (De Groot 1965) demonstrated the differences in memory that distinguish expert chess performance. However, such cross-sectional analysis leaves a missing link - a full account of how expert behavior develops from the beginning of initial practice, including which factors maximize final expert-level performance.
Practice is the fundamental factor determining skill and expertise. All studies of skill, regardless of whether they investigate this or alternative factors, must begin with an account of the effect of practice. A lawful relationship exists between practice amount and performance, where learning is initially rapid and slows as it progresses - the learning curve. This is also true for gameplay. Analyzing the longitudinal performance measures of more than 45,000 users of the game “Axon,” a simple game which nonetheless requires core cognitive functions of rapid perceptual decision making and action (Stafford and Dewar 2014), showed this canonical pattern of diminishing returns between practice and performance holds in video games.
. [fig:lcurve]
The relationship of diminishing returns can be characterized mathematically (Figure 1). We show a power-law formulation, which has a long history in the study of skill, but the best function to characterize practice-performance has been contested, with debates over the number of free parameters, and whether exponential or power law functions provide the best fit (Evans et al. 2018; Heathcote, Brown, and Mewhort 2000; Steyvers and Benjamin 2019). In the online material https://osf.io/fvm8s/ we provide code for implementing a simple learning curve and fitting it to observed data. Note that nonlinear curve fitting is an operation of some delicacy. We do not attempt to present comprehensive or best practice in terms of both the definitions of learning curves (which can have multiple forms) or curve fitting. Instead, we wish to illustrate, with a toy example, the in-principle use of a learning curve function. Our claim here is merely that fitting a standard curve can serve as a valuable theoretical anchor: it allows extraction of separate parameters for the learning rate, initial performance and asymptotic performance; this allows us to see the impact of different factors on different aspects of the skill acquisition; and if done across multiple studies will enhance comparability of results. Furthermore, digital games provide exactly the longitudinal, high sample rate, data from a large and diverse sample population that can arbitrate questions about the best form of the learning curve (e.g. analysis was based on data from 54 million plays of a gamified brain training platform).
Our understanding of practice will be enhanced by inspecting the influence of different styles of practice on the learning curve, as mere repetition is not sufficient to develop expertise. The deliberate practice account proposes that experts must engage in extensive practice, while focusing on component decomposition of skill, escalating challenge and using immediate and detailed performance feedback (Ericsson, Krampe, and Tesch-Römer 1993). Deliberate practice explains a large amount of individual variation in measures of performance, but is not the only factor that influences skill acquisition (Macnamara and Maitra 2019). This opens the question of how other practice factors can be related to skill acquisition, a question which games are well positioned to help answer.
For the individual player a major question about factors affecting skill acquisition will be how to maximize gains from practice: how to learn most quickly, how to reach the highest eventual performance level.
The type of practice engaged in has been shown to change the shape of the learning curve. These behaviors range from taking breaks, to exploring the environment, to social or team play. The spacing effect is a robustly established lab phenomenon, showing that spaced practice generates superior retention and/or performance for a given amount of practice, compared to massed practice. Games have afforded the opportunity to confirm the relevance of this phenomenon over longer time scales and with larger sample sizes than most lab studies (Stafford and Dewar 2014; Stafford and Haasnoot 2017; Stafford et al. 2017; Huang et al. 2017).
Analysis of game data does not always confirm experiment reports. Sleep consolidation is the effect whereby performance improves more after a practice-test interval filled with sleep, compared to an equivalent interval awake. couldn’t find any evidence for the sleep consolidation effect, although it is unclear if this is due to the relative simplicity of the game studied, or because observational studies allow participants to self-pace their practice (e.g. resulting in players sleeping only when they have saturated performance gains for the day), or other factors.
This study also showcases another benefit of game data - high density and sample size allows effects to be presented in terms of continuous parameters, rather than as binary comparisons. Figure 2 illustrates.
To maximize learning outcomes players need to focus on actions that have been previously effective, however, to know which strategies and decisions are effective they need to explore the environment. This situates self-directed game play as an example of the explore-exploit trade-off (Mehlhorn et al. 2015). showed that, in their study, players who explored more, on various measures of in-game behavior, may have had higher initial performance, but they did have not faster learning rates. This represents a failure to confirm the prediction that early exploration affects longer term performance. Exploring in social space may be an exception to this pattern; although two studies which suggest this use different operationalizations of in-game social behavior, and find opposite effects. show consistent (low exploration) teammate selection is associated with faster learning, while show that higher assist-rate (a measure of cooperative play within a selected team) was associated with slower learning.
Observational studies allow high sample size analyses of rich behavioral data, covering timescales which are difficult to access in lab studies. They demonstrate the influence of spacing, social play, and exploration phenomena when these choices are learner-determined, in a high motivation, non-arbitrary skill environment. However, because they do not use random assignment to test effects, they leave unanswered the question of whether forcing players to adopt a particular practice style would generate the same changes in skill acquisition.
When we consider a population of players our analysis naturally turns to the wider space of factors that might underlie expertise, including factors which are fixed with respect to the individual but may vary between individuals.
‘Talent’ is commonly attributed to players who who start at a higher level of performance and/or more rapidly progress to high performance. This label obscures the underlying factors. These factors might include basic physiological differences, prior experience, superior between-skill transfer, or greater cognitive capacity to learn or generate insights.
Learning rate and initial ability are not independent factors. Players whose initial performance is higher may learn faster (Stafford and Dewar 2014). analyzed data from 313,184 players of League of Legends, showing that the learning rate on the first 10 games of 2016 predicted final performance a year (and at least 150 games) later.
Progress on understanding components of talent will come from out-of-game measures, such as independent measures of cognitive ability. show that fluid intelligence positively correlates with rank in the game League of Legends. fail to find this relation using data from a similar game. Reasons for divergent results could include different outcome measures, such as categorical rank versus a numerical measure of overall performance in a game, as well how complex analyses account for control variables.
Collecting additional measures, whether demographic or cognitive, has great potential to augment analysis of game data (but with extra effort required and extra concerns with respect to player consent and data storage risk).
An example is analysis of age and skill development. Players’ age has shown to be a consistent predictor of performance in games; older players reach lower levels of performance (Kokkinakis et al. 2017; Röhlcke et al. 2018) and can be more likely to quit when experiencing difficulties in comparison to younger players (Steyvers and Benjamin 2019). Age is not only a factor that influences the interplay between skill and practice but can be used to investigate the development of expertise across the lifespan. In comparison to practice, investigating how skill changes across players’ age allows us to identify ranges of peak performance, declines that occur in later stages of career, and factors that underlie changes in the performance across the lifetime.
For example, looking at the changes in speed-based performance across players’ age using StarCraft 2, a real-time strategy game, show that the peak of performance is identified around the age of 24. This result is close to the prime of careers in speed and power sports, such as basketball (Vaci et al. 2019) and is in contrast to cognitive-based domains, such as chess (Strittmatter, Sunde, and Zegners 2020). After reaching the peak of performance in mid 20ies, players’ skill declines, which is likely influenced by the negative changes in the perception-action speed, yet this decline does not seem to be dependent on the level of their knowledge (Thompson, Blair, and Henrey 2014). However, knowledge and the level of expertise shows alleviating effects on age-related declines in the case of board games, where performance depends more on strategic and tactical thinking (Vaci, Gula, and Bilalić 2015). Focus on the age-related changes in game play performance might unearth other factors relevant for the development of expertise, shining light on complex interactions of inter- and intra-individual factors. For example, when using chess games as a testbed, showed that the interaction between initial ability and practice change throughout players’ careers; whereas the effect of practice is the strongest at the beginning of the career, initial ability predicted a higher level of performance at the peak and later stages of career.
The study of skill in gamers offers promising early results and exciting prospects for future work. However, tantalizing results do not add up to a comprehensive theoretical account. As one review observed, “Cognitive skill acquisition awaits its Newton"(Ohlsson 2008). We gather observations on how learning occurs, but real progress will come with testing theories of the cognitive mechanisms which allow individuals to acquire skills. makes the case that theoretical progress in this area will require computational accounts of complete task performance. While we can’t hope to even sketch such a comprehensive theory here, we believe that learning curve analysis – including formal modeling of individual learning curves – is a necessary step and will allow research on digital games to contribute to the wider topic of skill development. We also wish to highlight some theoretical and methodological challenges which will need to be overcome on the way to such a theory.
Although learning curves are typically portrayed as smooth, this is a simplification. In addition to extraneous noise, endogenous processes within skill development - for example restructuring of skill subcomponents - can interrupt smooth progression. have highlighted the importance of attending to plateaus, dips and leaps in the learning curve, as well as proposing an identification method (Donner and Hardy 2015). Note that this framework puts attention on the progress of the individual learner, rather than utilizing the power of large samples to extract a stable average learning curve.
The restructuring of component skills that occurs as part of skill development means that, at different levels of expertise, the factors which best predict superior performance may vary (Thompson et al. 2013). As expertise establishes consistency between players in a skill component, that consistency will remove the variation which would allow us to identify its importance to performance. This, again, underscores the importance of tracing the learning curve of individuals across the history of skill acquisition.
We suggest that future research into skill acquisition will require attention to the detail of individuals’ learning curves, not just high sample sizes. Fitting a learning curve to an individual’s data creates a simple summary statistic, the learning rate. This allows direct measurement of the rate of skill acquisition and analysis of how different factors affect it. As well as rate of learning, learning curves typically involve a parameter for asymptotic value, which can also be a key statistic for analysis, allowing the prediction of eventual level of skill from early performance.
Since much of this work is inspired by experimental studies, it is perhaps surprising how little of it comprises direct experiments (Johanson et al. 2019; Piller et al. 2020). Part of the challenge of using real games in experiments is that you need to either be, or motivate, a game designer, with those technical and creative abilities. Another challenge is that the decision to design an experimental game immediately raises the question of which game, and which properties it should have. Properties that a game for investigating skill acquisition should have include:
Even carrying out well designed experiments, and taking advantage of the principled foundation provided by fitting learning curves, the analysis of skill development in digital games is presented with numerous complexities and limitations. Players may come to games with different (and unclear) backgrounds, which affect transfer learning as well as creating heterogeneity in terms of both their cognitive abilities and strategic approach to practice. Subsets of players may suffer different motivations, affecting retention, rate of acquisition as well as what aspects of game performance they are trying to maximize (e.g. some players may only want to win, others may play to socialize and care less about winning). Theses hetereogeneities will affect generalization of findings to non-game playing populations and non-game domains.
So far, games-research has taken inspiration from the psychological science of skill acquisition, offering promising confirmation, qualifications and extensions of existing results. We have argued that the potential of games for understanding human skill acquisition cannot be met without more experimental studies, and studies which test multiple factors concurrently. We have suggested the learning curve as an anchor for more theoretically comprehensive studies: it has a mathematical characterization, should be analyzed at the level of the individual and can be a site of integration of different effects. We look forward to the time when the flow of inspiration is both ways, and the study of skill acquisition in games inspires the wider psychological science of skill acquisition in all domains.
show how game data can be used to compare the pattern of influence of multiple factors on performance.
uses move-by-move data and a high-level chess engine to evaluate move quality, thereby creating a de facto optimality analysis of the game, which they use to show a historical increase in move quality.
reflection on a special issue of Topics in Cognitive Science devoted to analysis of game data, evaluating contributions in light of Alan Newell’s program for progress in psychological theory.
present a framework for understanding the importance of discontinuities in the learning curve, showing how they can provide important insights into skill acquisition.
show how individual differences in numerical abilities change the relationship between practice and chess performance over the course of players’ careers.
We thank Lotty Brand and two anonymous reviewers for comments on earlier drafts of this article.