Feature The Art of Creating Sound in Video Games

Audio engineers sound off on one of the most important elements of game production.

Liana Kerzner

Published

By Liana Kerzner @redlianak

Much ado has been made about video game graphics. A thousand tech monkeys on a thousand keyboards have churned out millions of articles, comments, tweets, and status updates about their opinions of the graphics of various games. However, a study at Cleveland State University in 2010 tried to figure out what mattered more to the player experience, image or sound? Sound won.

So why is there so little noise about sound? Minsc in Baldur's Gate would not have been as loveable without Jim Cummings' yelling "Go for the eyes, Boo!" Halo's Covenant weapons would be far less cool without their distinctive sound effects. And that Halo soundtrack! Of course, great soundtracks are numerous in games: Little Big Planet's eclectic selections, the memorable themes to Super Mario Bros. and Frogger, Saint's Row's use of party anthems for comedy, and Assassin's Creed's classical bravado, just to name a few.

And let's not forget Dragon Age: Origins, a shining example of a game with mediocre graphics and phenomenal audio work: gamers everywhere fell in love with Steve Valentine as Alistair and John Curry as Zevran, even though the animation was so weird that every time those characters smiled, they looked like they'd just soiled themselves.

Reaching out to the video game sound community is easy - industry professionals in this field are approachable, helpful, and open about their craft. Determining what makes great game sound, however... not so much. As Hinterland Studios' Audio Director David Chan puts it, "When I do my job right no one should notice." So great video game sound is something so sublime that you're not really aware of it while you're playing. Easier said than done.

Hunter

Sound is also harder to market than graphics. Chanel Summers, owner of audio production and consulting company Syndicate 17, states:

Unlike graphics, you can't put audio on the back of a box."

Gordon McGladdery of sound design company A Shell in the Pit, points out that sound is harder to quantify than graphics.

Sound reached its technical 'resolution' peak years ago," he says. Humans cannot discern improvements above 48kHz, 24bit, so it is not something quantifiably marketable to consumers."

Sound has one tremendous advantage over visuals," says video game veteran Brian Schmidt. It can grab your attention even when you are paying attention to something else. If you hear that monster behind you in 5.1, no matter what you're looking at on the screen, the right sound will make you turn around to see what's there."

"The unseen monster is always scariest," McGladdery says, echoing Schmidt's monster metaphor. Sound is one of the most intrinsically effective means of getting to the point and can even be more effective when not tied to a visual. A visual without sound is an obvious mistake, while a sound without a visual is often given a pass."

Obvious mistakes in sound include "cluttered" or "muddy" audio. Summers calls the latter phenomenon "Sonic Sludge": too many sounds of a similar audio frequency prevent the player's ears from discerning any details. You've experienced Sonic Sludge if you've played a game and the music overpowers the dialogue you're trying to hear. It's a frustrating experience.

Audio should always be doing one of two things in a game, according to Klei Entertainment's Matthew Marteinsson.

It's either building the world or informing the player," he says. If things get too cluttered and messy the audio landscape will fall apart and the player will just ignore it."

Steve Royea, Game Audio Instructor at the Vancouver Film School's Game Design Program, agrees that it's key to avoid overdoing game sound. But what you do include is obviously also important. He states:

How the script is written, how the dialogue is read, who's the actor, all play a huge role in the emotional impact."

David Chan provides an example of how sound sets tone:

"When I do talks I often show an example of a forest scene with two different soundscapes. It's the same scene, but one version is light and happy with birds and a gentle wind, the other version has a low rumbling drone with hints of screaming and snarling. The video doesn't change, but the sound completely changes the viewer's perception of the scene. It's important for sound designers to remember that they are painting mood so they need to be consistent with the goal of the game."

But how does sound "paint mood?" That's much harder to quantify. Summers explains:

As in any art form, real innovation in game audio design doesn't come from technology, but rather from the creativity and experimentation of the artists who wield it."

Game music and sound design is still the Wild West," Schmidt states. There is a lot of uncharted territory in terms of technology, creative approaches to things and being willing to bend the rules can often result in really great stuff."

To facilitate an exchange of ideas within the game sound community, he runs GamesSoundCon every October in Los Angeles, California.

GameSoundCon
This year's event takes place October 7-8 at the Millennium Biltmore Hotel in Los Angeles.

But there are some starting points to the art of sound. McGladdery stresses that "time and budget for iterative implementation" is critical - iterations being the various versions of a game that evolve with development. Playtest feedback is essential, and that feedback can't happen if the audio is only added at the end of the process.

It takes a lot of experience to know what works and even then, preference can make a huge difference," Chan explains. Which is why you need to be willing to sacrifice sounds that you love for the greater good of the game. A lot of people think you can just take sounds straight out of a library and just slap them into a game."

The industry's past attitude toward audio might be summarized by Chan's humorous description of how he got his start:

I got into BioWare as a sysadmin (System Administrator) and had an opportunity to move into the audio department... and by that I mean become the sole member of said department at the time."

McGladdery agrees that sound is too-often seen as an "afterthought." Marteinsson uses even more evocative language, saying many think that sound is something "you can just cram in... at the end of development."

Anthony Russo of For All To Play, a company making games for the visually impaired, states:

People have told me to my face that sound in games doesn't matter. They tell me that they have proven that sound is unimportant in their favourite games because they mute them and play their own music."

When was the last time a mid-core gamer replaced their audio card?" Marteinsson asks. They probably don't even have one. But there's a good chance they upgraded their graphics card. Trying to discuss audio is very hard because it's ephemeral at best. You can't stop a sound and discuss how it should be changed to work better. When it's stopped it's gone."

But sound does matter, as proven by numerous games such as Halo, The Last of Us, Dead Space, and Bioshock. The sounds of Rapture aim for a deliberately distorted quality to make the experience more immersive.

Russo explains:

The grain on the pre-recorded record players mixed with the excellent voice acting really leaves the player feeling like they are in a livable, breathable forgotten city with these '50s era characters guiding you along. If the game had flat voices with no mixing at modern MP3 quality, a lot of the charm and appeal of the game would be totally lost."

Amnesia: A Machine For Pigs
Amnesia: A Machine For Pigs

McGladdery similarly praises Amnesia: A Machine For Pigs as a game that does sound well.

Samuel Justice obviously chose every single sound very carefully to maximize putting the player on edge," he says. A mechanic in the game is to raise your lantern to illuminate darkened areas. The sound of raising the lantern is a necessary sound for player feedback. That could be referred to as a 'scientific' necessity, but the sound he chose for that mechanic scares me every time I do it, no matter how many times I hear it. That is art."

Meanwhile, Marteinsson's game, Mark of the Ninja, was nominated alongside Assassin's Creed III in the best audio category of the Canadian Video Game Awards.

Making great game sound is one thing. Making a game where sound is the primary driver of gameplay is a different story. Brian Schmidt and Anthony Russo both tackled the challenge of making a game that's accessible and fun for the blind.

Why did you decide to do this?

Anthony Russo:

At Worcester Polytechnic Institute, you have to do a junior year project called an Interactive Qualifying Project (IQ) that benefits a certain demographic or identifies a proposed solution to a problem or need. So, I selfishly wanted to do one about games, because I love games. I had been dealing with sound a lot, and I always thought it would be cool to make games that were completely based on sound. This turned into blind accessible games, and my IQP Creating Video Games for the Visually Impaired was born."

As part of this assignment, I went to Perkin's School for the Blind in Massachusetts and asked a group of students of varied ages and genders what they wanted in games. It was a great experience, and I learned that screen readers and synthesized voice are such an integral part of their everyday life that hearing it in games really breaks the immersion and takes away from the fun. I also learned that many of the games that are available to the blind are older titles that have been around since the '90s. To oversimplify, they wanted new, fun, high quality games with no synthesized voices."

Brian Schmidt:

I've been studying 3D hearing on and off since the '80s, and was always intrigued at the prospect of sound-driven games. I used to give talks about using surround sound when I was at Xbox, and would often describe a hypothetical point in a game where a player would enter a cave and if they had no torch they'd have to play by sound alone. Fast forward a dozen years and I figured I'd try and see how that would work in practice first hand. I like the casual-game/arcade gameplay so I wanted to make a 'bite sized' game. I also wanted to see what it was like to design, develop, program, do audio for and release and market a game, getting my hands dirty and getting experience on every step of the process. That was the genesis of EarMonsters."

Perkins School for the Blind Xbox

What are the unique challenges in creating a game with just sounds?

Brian Schmidt:

One of the biggest is player expectations. Video games are very... visual. And players are accustomed to relying on visuals for everything from learning gameplay mechanics to receiving feedback and status on how well they're doing. Another huge challenge for my particular game is that my gameplay mechanic relies on 3D sound via HRTF (Head Related Transfer Function). But even in the real world, our ability to precisely locate a sound in 3D space is not very good. So having a gameplay mechanic that relies on 3D sound alone is very challenging."

This is in stark contrast to visual information, which of course we as a species are able to do with exquisite precision. I definitely had a couple false starts, and ended up modifying my gameplay mechanic, and in fact added limited visuals specifically to help the player learn what they needed to do in the game. A corollary to that is that, I also needed something to help blind gamers learn the game as well. Since I couldn't rely on visuals for them, I ended up adding specific aural cues and gameplay elements to compensate."

Anthony Russo:

The main challenge is creating a world that you can imagine clearly using only audio to build it. I was heavily inspired by radio drama - I was really into the BBC Hitchhikers Guide to the Galaxy one when I was a kid - and draw from its use of sound effects, musical stings, theatre-like acting, and overall tone."

Also, the game relies heavily on its story and acting, so in addition to ensuring the written material was funny and entertaining, I had to make sure the voice actors themselves were delivering the lines the way they were written. Don't get me wrong, my voice actors are awesome, but getting them to record what I hear in my head is not always easy."

The Hitchhikers Guide to the Galaxy

A lot of people are very uninformed about ableism (discrimination against people with disabilities) in video games. Based on your own work, what barriers are there for people with disabilities?

Anthony Russo:

A lot of games are just not designed with people with disabilities in mind, and it sucks. There are actually a lot of quick and easy ways a game company can make their games accessible too, be it a colorblind mode or support for screen readers. Organizations such as The AbleGamers Foundation and their Includification game accessibility guidelines are a great resource for easy simple ways to make your game more accessible. Not only does this really help out the people that would like to play your game, but, from a business side, you're expanding your target market. More players means more people buying your game which means more money. Everyone wins."

Brian Schmidt:

I was really stunned to discover that the iPhone was such a great device for the visually impaired! I'd originally thought 'well, such a visually oriented device as a touch screen must be incredibly difficult to use if you have trouble seeing.' Turns out Apple did a stellar job making the iOS devices accessible via a technology called 'voiceover,' which lets a user use their fingers as sort of surrogate eyes."

That said, the largest barrier is probably games that unknowingly are unnecessarily inaccessible. Game developers don't necessarily think, 'hmm, what minor tweak could I make that would open up my game for a range of players with disabilities.' Even something as simple as accounting for colour-blindness when creating navigation graphics can help make a game more playable to more people."

There's a great story about how a blind gamer contacted the maker of (the game) Injustice with a minor suggestion to add an audio cue to what was a visual only cue during gameplay - that one change suddenly made the game fully playable just by sound alone. The IGDA has a great special interest group specifically on accessibility; I think as more game developers realize that minor changes can make their games more broadly accessibly, things will definitely pick up!"

The AbleGamers Foundation

More players means more people buying your game which means more money. Everyone wins."

Some might think that sound design is less exciting than working with images, but all the sound specialists were enthusiastic and inspired by the continuing possibilities and challenges offered by interactive sound design. Summers insists that game audio can and should aspire to deliver "more impactful experiences" than any other medium that's come before.

Schmidt applies a similar lofty scope to video game audio:

A game sound designer/composer has to use finite resources to create sounds for an almost unlimited set of circumstances. Great game audio has a tremendous amount of subtlety, which is easy to overlook, but gives games that extra 'something' that you can't always put your finger on right away."

And that unlimited set of circumstances includes multiple variables: music, sound effects, ambient sounds, and voices. So how does each sound designer decide what type of sound to use, and when? Chan uses his upcoming survival game, The Long Dark, as an example of how he approaches the challenge of getting the balance right, while emphasizing that there's no single right way to do it.

The Long Dark's art style is very minimal and simple yet striking," he states. That's the way we came at the sound design and music as well. We wanted to use music sparingly to allow the environment to become a character. Music is used for important events and to set mood in conjunction with ambient sounds... we try to keep 'fake' sounds to a minimum. In other words if you hear something it's because there is a reason to hear it. Wolves howling, crows cawing, wind howling are all important to gameplay and give you feedback you need to be successful in the game. In other games like Skyrim, music is going to be more frequent and more fitting to the elaborate gameplay and art style."

Skyrim screenshot
Skyrim is one of the most articulate games ever.

McGladdery advises:

When you want to 100% accurately enforce an emotion, that's when you employ music, which we are culturally programmed to absorb as accurately as the composer wishes. We are programmed to feel melancholic with a lonely piano refrain, excited with full orchestral hits and timpanis, afraid when the shrill, dissonant violins kick in. Sometimes this is necessary, but sometimes a subtler approach is desirable. For instance, an environment's sounds don't always tell a player how to feel; it will be different for everyone and makes the experience personal. This causes the player to reflect without knowing it, to be afraid of things they maybe shouldn't have been afraid of, and to investigate their surroundings more closely. I like to equate it to Steve Martin's style of standup comedy, which was never providing the audience with a punchline; let them decide when to laugh."

Schmidt says that it's fun to blur the lines between music and sound effects in "casual" games, citing Peggle 2 as an example of this blur.

As a more generic answer, music is excellent for keeping, maintaining and manipulating emotions on a macro level," he declares. Sound effects are ideal when you want to convey more precise information, and the micro level. Think of tense battle music, which keeps your adrenaline flowing, vs. specific enemy or weapon sounds during the battle, which help convey specific information about what you or your opponents are doing.

Russo shares: "My workflow prioritizes important sound/dialogue, then added sound effects that usually emphasize the action at hand, and then mood setting music or ambient sound. It's a delicate balance and easy to mess up, but you know when something sounds right."

Royea, meanwhile, has a near-opposite process, stating that "You need to start with the basic soundscape of the game, usually a music track and background ambiances that match the environments of the game world. As other elements are added in, they should be balanced with this bed track."

Much like his free-form workplace, Marteinsson doesn't have a personal blueprint to follow.

Generally everything that is world building or informing the player of something will get a sound," he says. Music is more varied but still feeds a lot into those things as well. Making the music interactive with the game becomes incredibly important to create an immersive world. And then what parts feed that interactivity. Do you want to tell the player there's danger? That it's now safe? That they're doing something? That some other state change is about to happen? All these cases are different game to game."

We are constantly listening in our day to day lives, living in three dimensional sound," Royea adds. The closer to reality sound is, the more we believe it. Also, because sound and memory are so tightly linked, when we hear something new that we haven't heard before it lights up in our brain. In my opinion the real reason for Star Wars' success is the whole new world of sounds that Ben Burtt created for George Lucas' universe."

But the unknown can also cause a less pleasant response. According to Summers, "Known sounds can force the audience to recall memories and experiences... unknown sounds can create confusion and possibly fear." She says, citing the examples of the mother in Psycho and the Wizard in The Wizard of Oz.

The final point should be obvious: players are going to hear key game sounds a lot. Consider the coin collection sound in Mario, or the title swap sound in Candy Crush Saga. Chan cautions, "In a two hour movie you might hear a door opening sound a handful of times. In ten hours of gameplay you might hear it hundreds of times so it better not suck."

So what do the people behind you favorite game's sound want you to know? Schmidt offers, "Probably that a whole lot more goes into sound for games than you probably think."

Summers advises would-be designers: "The tools should be there to enable your vision, but ultimately, you need to be a storyteller and/or provide an experience."

I'd like people to think about how sound effects them every day," Chan says. How many times a day they rely on sound for important information? That's what we are trying to create in a virtual world and it's challenging, but also a lot of fun."

Don't be afraid to make things and mess up doing it," Russo adds. If you want to make a sound game and people tell you it's stupid, do it anyway because its what you want to do. If you mess up and the game isn't what you hoped, at least you created something, and I bet if you try again, it'll be even better."

Most importantly, many of the designers sincerely want people to experience the stories told by sound in video games with good quality, properly configured headphones or speakers. If you're not really hearing the games you play, you're missing out on multiple levels of the entertainment you paid for.

Star Wars: The Empire Strikes Back

Making the music interactive with the game becomes incredibly important to create an immersive world."

Who's Who?

David Chan

David Chan

Current position: Audio Director, Hinterland Games

Previous work: Baldur's Gate II, Neverwinter Nights, Knight of the Old Republic, Prey, Splinter Cell: Double Agent and more recently mobile titles such as Office Jerk and Grumpy Bears.

Upcoming project: The Long Dark

The best part of working at his company: The people hands down. It's a great group of experienced developers and eager newcomers. I am really enjoying working with the Hinterland team.

What he's currently playing: A fair amount of Minecraft, Titanfall, Diablo and I've been trying out a lot of other survival games to get a feel for what works and what doesn't.

Brian Schmidt

Brian Schmidt

Current position: Brian Schmidt Studios, Executive Director of GamesSoundCon, Founder of Ear Games.

Previous work: 140+ arcade and video games , including Madden, NARC, Zynga games, and pinball machines, 10 years as head of audio technology for Xbox and Xbox 360, designed the Xbox One "Shape"

Upcoming projects: GamesSoundCon 2014 - October 7-8, 2014 in Los Angeles

The best part of working at his company: Working for myself means I can get involved in many different peoples' projects. One day I'm composing orchestral scores and hiring musicians and another day I'm writing some code to show the game programmers exactly what I want them to do.

The games he's currently playing: I finally just finished Papa Sangre II. I'm also trying to get out and play more pinball since I'm back to working on those again. And I'm waiting for Destiny, like most of the rest of the world.

Gordon McGladdery

Gordon McGladdery

Current position: Owner, A Shell in the Pit, collaborates frequently with Power Up Audio

Previous work: Rogue Legacy

Upcoming projects: Duelyst, Okhlos, Parkitect, Bunker Punks, Viking Squad (Preview audio)

The best part of working at his company: Having work! I'm reaching a comfortable place now, but starting my own business in such an insanely competitive environment was probably the biggest challenge I've undertaken. I overcame it with extreme persistence, pride-swallowing, carefully considering criticism, hard work and luck. The sound community is also extremely supportive of one another. I wouldn't be anywhere without it.

The games he's currently playing: Brothers, replayed Dear Esther, and I regularly dig into Nidhogg and Towerfall with friends.

Anthony Russo

Anthony Russo

Current position: Sound Designer / Writer / Game Designer / Artist at For All To Play, a Worcester, MA startup.

Previous work: My senior year thesis at Worcester Polytechnic Institute was a game for the blind. We had a psychological sciences major, Greg Moore, run the testing so that tests were as accurate as possible and feedback would hold weight. He was invaluable and I owe him for life.

Upcoming projects: Grail to the Thief: An Interactive Audio Adventure, which is a blind accessible adventure game inspired by old text adventures and point-and-clicks, like Zork and Day of the Tentacle.

The best part of working at his company: I love creating something that people legitimately enjoy. It's an indescribable feeling when someone, just for a brief moment, is being entertained because of something that I was part of. It's surreal.

The games he's currently playing: Dota 2, which I have to stop doing because it's eating my life; Bro Force and Crawl with my girlfriend's siblings.

Chanel Summers

Chanel Summers

Current position: Owner, Syndicate 17; adjunct faculty member, University of Southern California's School of Cinematic Arts, Interactive Media & Games Division; Chief Product Officer, Score Music Interactive; international lecturer.

Previous work: Fighter Ace, Original Xbox Development Team

Upcoming projects: A3E Audio Conference, emergent technology, large-scale corporate installations, global academic work.

The best part of working at her company: Combining consulting, academic and production work.

The games she's currently playing: 140 (by Jeppe Carlsen, the designer of Limbo), Fract, Blindside, and Papa Sangre II.

Matthew Marteinsson

Matthew Marteinsson

Current position: I work for Klei Entertainment. We don't actually have job titles anymore at Klei.

Previous work: With Klei I've done Shank 2, Mark of The Ninja, Eets Munchies, and Don't Starve

Upcoming projects: Invisible Inc.

The best part of working at his company: Overall the company culture is just really great here. It feels like a strong emphasis was put on building a team that works really well together. And really pushing to do away with crunch culture, so that you can work in video games and still have a family and life outside work.

The games he's currently playing: Road Not Taken, FRACT OSC, Banner Saga, Ironclad Tactics, Transistor, was in on the Destiny beta, a bunch of other indie stuff on PS3 (Thomas Was Alone, Rogue Legacy, The Swapper), Hearthstone on my iPad and Threes on my phone.

Steve Royea

Steve Royea

Current position: Game Audio Instructor, Vancouver Film School Game Design Program

Previous work: EA Canada, Blackbox Games, Next Level Games.

Upcoming projects: I'm currently involved in the Indy scene in Vancouver, British Columbia, but NDAs mean I can't say much.

The best part of working at his company: I'm always amazed with the creative range of the game design students, and feel honoured to be a part of their process.

The games he's currently playing: I do most of my gaming with my 14 year old son. We just finished up both the Mass Effect and Uncharted trilogies on the PS3. I bought a PS4 to play the Alpha and Beta of Destiny, we can't wait until September 9 when it gets released!

Images courtesy of David Chan, Brian Schmidt, Gordon McGladdery, Jeremy Lim Photography, Chanel Summers, Anthony Russo, Matthew Marteinsson, Steve Royea, Microsoft, Lucasfilm, Frictional Games, Bethesda Softworks, The AbleGamers Foundation, Infocom and Perkin's School for the Blind.