MCV/DEVELOP News, events, research and jobs from the games industry
Quality in its broadest sense is an elusive concept to pin down when applied to games. The graphics can be beautiful, the gameplay revolutionary, but if the whole thing is sitting on a shaky foundation and the details haven’t been refined, then it’s unlikely to engage players for long.
QA teams have always done more than simply find bugs, though that’s still the core of the job. They provide early feedback on design, art, UI and much more. They’re also increasingly specialised and technically capable. And in an industry where ongoing player engagement is now valued above and beyond unit sales, the QA function is arguably more valued too.
More valued maybe, but things are not getting any easier for the QA professionals we spoke to, as Erik Hittenhausen, QA director for games at Testronic, based in Warsaw, explains: “The challenge today is the ever-growing scale and technical complexity of games, combined with the need for ongoing post-release support and updates. Be it an intermittent build or even a full release cycle, it can be challenging if not impossible to cover all features and content in a game on every iteration.”
Meanwhile Gao Wen Xin, QA director at Virtuos in Xi’an, China, tells us of another challenge: rising player expectations.
“Due to continuous advancements in technology, players nowadays have increasingly higher requirements in terms of game quality and professionalism, and are therefore more critical as a result,” he says.
More complex titles and more demanding players are only half the battle though, as others bring up more perennial issues within the industry related to the perceived status of QA.
Michael Bishop, QA tester at Lucid Games, says: “A level of respect needs to be earned in order to express urgency to production and publisher. I’ve seen members of my test team explain an issue to members of production before and it sadly fell on deaf ears… It sometimes needs a push from someone that ‘gets along’ with the higher-ups for the issue to really see the light of day and be made a priority fix.”
That’s a sentiment which is echoed by Gabriel Idowu, senior QA technician at Bossa Studios: “Ultimately the largest issue facing QA remains professional perception. Though the role is highly demanding, requiring large amounts of dedication and technical knowledge, it is still perceived as just ‘playing games’, which results in critical feedback often being ignored in favour of alleviating strain on other departments. Unsurprisingly to most in QA, this often results in the issue re-emerging down the line as a much larger problem.”
QUALITY THAT LASTS
‘Down the line’ these days might stretch well past the initial release. A problem that goes unsolved early on might still be bugging a development team many years later, potentially compounded by numerous updates and content drops in between. Now live service games are nothing new, but QA departments are still refining the best practices for such titles.
Testronic’s Hittenhausen notes the need for greater cooperation on such titles: “The compounding nature of content in live service games prevents you from continuously double-checking all legacy assets and features. Because of this it has become increasingly important for QA and production to work together closely in order to accurately assess risks and identify the areas of a game that can be impacted on every release.”
And as well as working closer with developers, the QA team may also need to take on additional roles, Virtuos’ Gao tells us: “Our QA team, in addition to maintaining the quality of the game throughout successive updates and patches, also undertakes additional responsibilities such as collecting player feedback, analysing popular trends, and providing suggestions on how to increase the game’s fun factor.”
But fun isn’t the biggest concern with such titles, because if gamers can get angry at bugs in a game, imagine the additional rage when microtransactions don’t deliver as expected.
“The ecommerce side can be very tricky and time consuming depending on the level of content being added,” Lucid Games’ Bishop notes. “As an example, a battle pass may just look like one UI screen to the end user most of the time but it needs to be tested to ensure it unlocks and displays the right content at every stage of the product and every route the user can take to access that content.”
Paul Klosowski, QA manager at Sumo Digital, explains that such titles have led testers to working deeper in the software’s systems: “The growth of games-as-a-service means that our testers now work regularly within the telemetry systems of a game to help track, analyse and report findings that arise during alpha, beta and release test phases. This is where QA can really shine as a live and continuous resource to help refine the game experience that our dev teams have worked towards, ensuring that we confirm their overall vision.”
The QA professionals we spoke to are a varied bunch, some working in-house while others are part of outsourced teams working on games made by other studios, as well as co-developed efforts from within.
With ongoing service titles, as opposed to standalone releases, there’s a certain logic that says the developer will need a more persistent QA resource, and Lucid’s Bishop thinks that has led to a shift in QA from outsourced to internal teams: “Due to a lot of products becoming a live service game, it has required a lot more embedded QA testers to be hired. The more content drops a live service game has, the more QA is usually needed to balance it out,” he explains.
Furthermore, the outsourcing of parts of development then logically moves QA back to the developer or publisher to check on that work, Bishop notes: “I think it’s pretty common knowledge that development time on features are being outsourced very regularly but in the same instance QA will need to be hired internally to balance this out.”
Testronic’s Hittenhausen notes that it’s not where the team is based that’s important, but rather that the team is in it for the long-haul: “[Live games] have increased recognition of the immense value in having an experienced QA team, and the importance of retaining a stable team that has deep knowledge about your game. This not only allows the team to understand and effectively verify its quality consistently, but also enables the forming of a close working relationship and mutual understanding that is essential in an agile [development] environment.”
Virtuos’ Gao doesn’t think that such games require “a permanent team for upkeep” but agrees that familiarity with the project is key: “We encourage our QA staff assigned to such projects to be subject matter experts, so whenever a significant update is imminent, or when a spike in demand appears, these same personnel can provide additional support temporarily to maintain quality.”
GOING PUBLIC
The ultimate form of QA outsourcing is getting your players, your community, to test the game for you. Massive public betas and early access both allow for this kind of feedback, but are either really effective as part of the QA process?
Lucid’s Bishop, who has been working on vehicle-based MOBA Switchblade, is a big supporter of such pre-release outings: “Public betas are amazing. Hands down the best move a product can do in its early days. It gathers feedback and throws large numbers of testers towards the product that it hasn’t had before in its life.”
The obvious plus is testing the online aspects, Bishop adds: “Network problems and server stability issues won’t show up in-house most of the time, but when mass numbers are thrown at a product, problems will occur that sometimes you’d never see from a QA perspective.”
“Public betas are amazing. Hands down the best move a product can do in its early days.”
Virtuos’ Gao also believes they are highly useful: “Large-scale public betas can help QA and the entire team to understand more about the ideas and feedback given by players. Some hardcore players would even send us advice or comments, which can prove crucial for the team to control the direction of the product. Also, because there are so many people involved in the test, the chance of issues being found exponentially increases, which can in turn improve the quality of the game dramatically. We believe that this process is vital for large triple-A games, or online games containing huge amounts of content.”
It’s not unanimous, though, with Bossa’s Idowu saying they help the QA process “very little,” and explains: “Ultimately the average mindset of an individual looking to join a beta is not to provide quality feedback. It’s to get an early chance to play a game. So the vast majority of players rarely ever provide feedback. And those who do rarely provide anything actionable.
“What public betas have done is add another massive source of data that QA needs to become familiar with. And it’s the age old problem of the signal-to-noise ratio,” Idowu continues. “Most of the time spent with this data is just filtering it. And I am unconvinced that that time wouldn’t be better spent just testing the game yourself.”
SPRINTING TO THE FINISH
While it’s been a slow and long shift, games development has moved to varying degrees away from traditional development processes to more agile methodologies with shorter, faster iteration loops (commonly known as sprints). But obviously the choice of development model will impact the QA function as well.
“QA work habits have changed a lot in recent years,” Virtuos’ Gao states. “But it has proven to be a good process. The using of agile development in the game industry helps QA to improve its workflow and positively affect quality control as well. We’re always happy for more ways to do so – that’s why we don’t reject any behavior patterns outright that might have the potential to improve the quality of the product.”
Idowu too is positive but cautious: “Agile is like any other tool in the world. When used correctly it can be a massive help. But used incorrectly it can compound issues and increase the pressure to deliver while decreasing the amount of time there is to do so. Ultimately the most important tool that exists is a great producer. No tool can replace a producer that knows what to do when and how to do it.”
“Agile is like any other tool in the world. When used correctly it can be a massive help. But used incorrectly it can compound issues. Ultimately the most important tool that exists is a great producer.”
Sumo’s Klosowski explains that such development methodologies must be matched by new processes for QA as well: “We’ve recently adapted and applied a methodical approach where we plan and test our milestone deliveries to include both Milestone Acceptance Testing (MAT) and User Acceptance Testing (UAT).
“For example, when a game feature goes into a build, we can immediately verify that it has passed pre-defined acceptance criteria by highlighting bugs, all within the same milestone sprint, and collate any problems that occur due to the interactions between multiple systems. This allows us to avoid issues that might otherwise develop in isolation… Following this method has removed bottlenecks and encouraged the integration of QA into the larger game development process.”
THE NEXT STEP
Looking to the future, streaming services such as Stadia will be the next big challenge for QA teams, who will be on the front line in ensuring that titles deliver an acceptable experience within the vagaries of users’ internet connections.
Testronic’s Hittenhausen says: “People often come to us when new technologies that they haven’t had the opportunity to familiarise themselves with arrive on the scene. As such, we often find ourselves on the forefront and have grown accustomed to adopting and rising to the interesting new challenges these developments bring with them. Working with different parties trying to adopt these technologies at the same time puts us in a unique position to quickly learn the do’s and don’ts for testing on new platforms and, outside of the technology, at the end of the day what you are testing is still a game.”
Gao from Virtuos adds: “We are really looking forward to Stadia and the new generation of hardware. As developers, we tend to come into contact with these platforms earlier than the general public, but that doesn’t mean we can rest easy. The new platform requires learning, the new trends in game genres need to be researched, and the test requirements have to be analysed.”
“Automation… There are far too many tasks within the industry that are being done manually and take up a lot of time!”
But the future for QA won’t be defined simply by new platforms, game and genres, as there’s big changes coming in terms of how the job is done too.
For Lucid’s Bishop, there’s one word that defines the future of QA: “Automation.” He explains: “There are far too many tasks within the industry that are being done manually and take up a lot of time! Instead of wasting QA time on such tasks, QA should be trying to destructively test within a game, that’s where you find the major issues.”
While automation should save time for more intuitive QA on a tester’s own initiative, Sumo’s Klosowski strikes a note of caution: “As automation becomes increasingly prominent in the testing process, it’s important to be aware that it can lead to situations where its implementation can outweigh its value, sometimes resulting in a human tester being assigned to verify the work. To counter this, we integrate automated testing in the project as early as possible. This allows us to compare builds, immediately seeing any change, which is useful when validating level playthroughs, performance, smoke tests and so on.”
While manual and automated testing have long been the two sides of the QA coin, could the rise of AI form a valuable middle ground?
Gao replies: “We did hold in-depth discussions about AI internally within the QA department, and the general consensus is that AI can be utilised to replace some basic roles. But at the same time, we also think that some of the job scopes, typically held by QA experts, can’t be replaced by AI.
“Based on our experiences from working on a huge number of different projects, we find that most QA work is actually creative in nature,” Gao states. “The most basic function of QA is to simulate the actual player experience, and players are living human beings. That’s an almost infinite set of individual preferences and expectations to address, so AI can’t replace QA to do most of the work.”
And we’ll let Gao have the last word here too, as he sets out his thoughts on the future of the job: “I personally think that the future of QA will be a very challenging one, becoming a much tougher job with its services valued higher than before. While QA will have more urgent demand for experts, simple repetitive QA work will likely be eliminated, and the development and self-evaluation of QA will become more important. What determines the success or failure of a product? The answer is quality, and QA are the ones that are fully empowered to be arbiters of quality.”
