The future of next-gen sound blah blah blah

March 9th, 2013

Sorry, I couldn’t resist being a little snarky as I typed that title out. Every time there’s a new generation of consoles on the horizon, words begin to flow about what “next-gen” means in relation to (insert your discipline here).  For me, there are two interrelated aspects that we can look at to push envelopes further: technological and structural. Technological advances are those made possible by the capabilities of the hardware and how that interacts with software. CD-ROMS meant we could start streaming redbook audio, tons of voiceover, and video. The Xbox’s DSP chip gave us a low pass filter and reverbs built into the system. The PS3’s SPU core architecture gave us an entire core (or more I suppose if you sufficiently bribed your programmers) to do with whatever we wanted: create custom DSP or utilize FMOD or Wwise and go crazy with the delays and reverbs and eqs, etc. The PS4’s 8gb of memory means, given the time to load data into RAM, we have a near limitless reservoir for game resources. Ok, so “near limitless” is probably an over-exaggeration, but we’re talking a 16x increase over the last generation!

By structural, I mean how does the technology create new ways for us to deliver a sonic experience. The sub-industry of mobile and web development have democratized game development significantly, and with them and the rise of Unity as a viable engine has audio middleware solutions like FMOD and Wwise along for the ride as well. Even Wwise, which started out as a PC, 360, PS3 only platform 6 years ago, now has support for iOS, Android, Windows Phone, and direct integration into Unity. With the democratization of tools comes the possibility to use these tools in novel ways. One such example is adaptive mixing. While in console land we’ve been doing this for years (for a great example of this see Rod Bridgett’s discussions of mixing Scarface: The World is Yours for PS2 back in 2006), this is only now being possible across all platforms. And with the potential for the Ouya, Green Throttle, Steam Box, Apple TV and other small Android, iOS and PC-based home consoles in the coming years we should see “next-gen” meaning what can we do to push content to be more impactful no matter the platform.

While I think the structural aspect has far more implications for sound design as a whole, much of what becomes possible is through technology. I want to touch on one specific technology in this post: procedural synthesis and design. Procedural synthesis is nothing new. Guys like Nicolas Fournel and Dylan Menzies have been doing it for years. Audiokinetic have had wind and impact generation tools in Wwise for several years now. Audiogaming’s wind and rain tools are integrated into the new version of FMOD Studio and will be making their way into Wwise soon (not to mention their latest plug-ins for procedural generation of footsteps, vehicle engine models, and fire).  And there have been countless papers and demonstration videos showing off better and better sounding procedural algorithms from the aforementioned to elements like fabric and full physics simulations.

When developing a game, we often take a cross-platform approach because it’s the easiest way to maximize profits for minimal cost: put your product out on every possible platform and you have a multiplier effect on how many people may play/buy your game, ideally with minimal additional effort on your part. Hopefully in the next year, if all these new hardware devices do come out, we’ll be at a point where we have enough processor to utilize procedural synthesis in games across all platforms, and not just minimal use on 360, PS3, and PC. Having these effects not just available, but possible, across all platforms maybe the shot in the arm procedural synthesis needs to finally bridge the gap from “talked about incessantly” to “the here and now.”

These two elements: realtime, dynamic mixing and procedural synthesis, while nothing new, may be the holy grail of audio development for games in the near-future. I am eagerly looking forward to how things shape up over the next few years, to see what others are doing, and to further explore these waters myself.

 

 


…introducing what I’ve been doing for the past year!

March 1st, 2013

As most people know, Sony introduced the Playstation 4 last week which means the veils of silence and secrecy have been lifted and I can FINALLY divulge the project and platform I’m working on. I’m really excited about the project and am quite happy with how it’s been shaping up. Below is our debut trailer, which I worked with our team down in Foster City to do the sound design and mix. I hope to share some interesting tidbits about the audio systems I’ve designed for inFamous: Second Son over the coming months, so check back here occasionally or follow me on twitter or shoot me an email, give me a call, send me a pigeon, etc.  But for now, here’s the worlds first peek at inFamous: Second Son captured in-engine (and using lots of in-game sounds although we did the sound pass in post just because it was a lot easier to do so).


An homage to the best boss I’ve ever had

January 27th, 2013

Let’s face it, if you’ve been in the industry for a while, you’ll know how tumultuous it can be. Studio upheavals, shutdowns, re-organizations, corrections of vision, layoffs…call them what you will, but chances are you’ve seen a few. It’s an unfortunate reality in this industry. In my now 15(!) years in game audio production, I’ve been through at least 10 rounds of layoffs, and two studio shutdowns. (The only studio I’ve worked for that’s still around today besides Sucker Punch is Free Range Games.  Keep it up guys! Otherwise people might think the problem is me!)

The hardest thing for me when I see news stories about studio layoffs and shutdowns is thinking of the talented people affected. Most folks probably land on their feet at another studio, but often at the cost of uprooting their families and lives across the country. And sometimes, the new jobs just don’t show up. I’ve seen a lot of exceedingly talented people get booted from the industry as a result of no jobs for them.

I guess this is a story about one such man, who did so much for me and my career and was also the greatest boss I’ve ever had. (And I’ve had a couple good ones since, but Jun Funahashi takes the cake). Jun was a Konami man. He started there in Japan back in the late 80s and worked as a composer and sound designer on such classic NES games as TMNT and Castlevania III. (In fact when we’d interview people at the studio and they found out some of his credits, they would often start fawning over Jun, not in an effort to get hired, but because they were fanboy struck). Back then Konami had their own band, The Konami All Stars, or Kukeiha Club, who would travel around and play Konami music at various events, and Jun was their keyboardist.  As time went on, he climbed the ranks and eventually moved to Konami’s studio in Chicago. He loved it and worked there for several years until they asked him to move out to Redwood City in California and join the team as Audio Lead. In the late 90s, Konami decided to open a new studio in Honolulu, Hawaii, and picked Jun to become the Audio Manager and build up their audio team from scratch. When I got there in 2001, they had been open for less than two  years and had worked on a few aborted projects, as well as a Major League Soccer game. They had one other sound designer, Jaren Tolman, who had been at Single Trac (makers of Jet Moto), was immensely talented and friendly and would go on to be a great audio director in his own right (No stranger to layoffs himself, I believe he’s now doing freelance for games and film). In addition to the new soccer game, we were working on a Jurassic Park III title for Game Boy Advance and had two other GBA titles that would be coming into production over the next few months.

I remember my interview with Jun in late 2000 or early 2001 at the studio. We were seated in the big conference room surrounded by windows overlooking Waikiki and the Pacific Ocean. It was unreal, the best way to sell a company for sure! About halfway through the interview Jun says to me, “You do realize this is a Japanese company, right?’ I tell him, ‘Yes, but what does that mean actually?” I was totally surprised and taken aback by his answer, “Um…it means we’re really disorganized.”

That’s the first thing that struck me about Jun was his honesty. He spoke from the hip and didn’t try to keep us in the dark about what was going on, like some managers I’ve known or had. We were always kept abreast of what was going on in the studio, and Jun made sure we were well informed (at least as much as he knew). With that said, he also did his job phenomenally, which, in essence, was to take care of all the bureaucratic bullshit and let us do our jobs. Now that I’m a manager and have to spend so much of my time doing those tasks, I realize how important that was for our team (and how thankless it must have been for Jun, and things were WAY more bureaucratic and nutty at that studio than any other I’ve worked at).

Another place Jun excelled at was understanding the value of a good team. Jun took his time assembling his team and while we missed out on some talented people who couldn’t get over the fact that we lived on an island just 45 miles across, there were other people we passed over because the fit wasn’t there with us culturally. This was the first time I’d ever seen personality become such an important factor in the hiring process, and the results were overwhelmingly apparent. While the studio as a whole may have been disorganized, the audio team was lean, efficient, and somehow able to handle 3-5 simultaneous projects with just 2-3 designers and a programmer. This was due to the people Jun hired, (and on a more macro scale to Jun’s intuitive good sense in hiring). We worked really well together and had a functional relationship whereby one of us would be the lead on each project and figure out the direction and audio needs and the others would pitch in as required with design. We worked a lot, but it was such a great team that it made the crunches quite a bit more bearable.

Which leads me to another of Jun’s great legacies that I try to extend into my own career: he never overmanaged. As I said, Jun did a great job of taking the brunt of the bureaucracy out of our jobs so we could focus on being creative, but he also trusted us enough to let us each take responsibility for our projects. We checked in frequently, but it was very, very rare that Jun would question our direction, and if so it was often with good reason. And he always provided sound, concise feedback.

Jun is also a man with golden ears. Not literally, or else Glen Beck may have sliced them off years ago. But we used Jun as the final arbiter for whether our mixes were good enough. Mixing at Konami was always a group experience, which was really a great culmination of all of our separate work coming together into the final product. We would all nestle into our tracking studio in the back of the office and play through the game taking notes of what needed to be quieter, louder, eq’d differently, etc. We would all provide feedback, but ultimately we always put it upon Jun to give the final yay or nay on the mix. His golden ears were invaluable during all the music games we worked on as well. (5 DDR titles while I was there and another 2 afterwards I believe). At one point early in my time at Konami when I was still a bit fresh faced in the industry (I’d been a sound designer for about 3 years at that point, but only for PC and online titles). We were working on a Game Boy Advance collection of classic Konami arcade games. I was the lead, and so it was my job to recreate all the music and sfx from these classic games. (No small feat considering they were all old chip sounds, made from square waves and noise generators, so I had to recreate everything from scratch). While I love writing and playing music, transcribing music is not a strength of mine. Jun realized this and helped me resequence the music for nearly every one of the six games in the collection. He taught me all those necessary tricks of the NES and pre-sampled audio days like using noise generators with different envelopes to make everything from snares and high hats to explosions. Jun thought nothing of coming in and saving the project effortlessly and selflessly, all the while maintaining his own duties at the same time.

I could go on for days with Jun anecdotes: about how he quit a 40-cup-a-day coffee habit cold turkey or saved my ass from getting fired for trying to put together a collection to buy a team a copy of their newly released game (it was studio policy at the time that we didn’t get a free copy of our own games we just finished!), or how understanding he was when we began to plan my departure from the studio, but this isn’t a eulogy; the guys alive, kicking and doing great!  It’s more of a remembrance of his time in the industry.

Unfortunately, Konami Hawaii studio was shut down in 2006. There were really no other studios going in Hawaii at the time other than Henk Rogers’ Blue Planet, (which was sold off to Jamdat, which in turn became EA, which in turn got shut down.  He and a bunch of former colleagues are still out there with Tetris Online). Jun definitely didn’t want to go back to Japan, and his job hunt wasn’t going so well. How many studios need Audio Managers at any given time, and once someone has that position, it’s pretty rare that they’re going to leave. So he decided to head back to where it all began: Chicago. Jun got some freelance work at Midway working on Stranglehold (the John Woo/Chow Yun Fat game), but after that there was nothing. No leads, no jobs, but a family to feed, rent to pay, etc. So Jun had to leave the industry (though he’s still available for freelance work if you’re interested!). He’s doing great with a sales job, traveling the country, and his kids are growing up. But I also know what Jun did for me as a person, for my career as an audio director, and besides being eternally indebted to him and his wisdom, spirit, humor, friendship, and mentorship, I am remiss at the other designers out there who have missed out on the amazing experience of working for Jun Funahashi.

You can see a partial list of Jun’s credits here, but what you miss until you meet the man are the intangible qualities of a great human being.

 


Adventures in the Field, Volume 2: Snowboarding

November 14th, 2012

While we were working on the skateboarding game back at Free Range Games, we were hoping it would take off and they’d ask us to do a snowboarding game (it didn’t, but as a stopgap we ended up making SummitX Snowboarding on our own).  Since the project would have likely happened in the summertime, I opted to make a huge sacrifice for the team and spend a lot of time up in Lake Tahoe recording snowboarding sounds during the winter.  I needed to get a gamut of various terrain types from the corduroy and packed powder of ski run groomers to spring corn to the neck deep powder of the backcountry and even loathesome sheets of ice and rocks.

Fortunately, I’m a much more competent snowboarder than I am a skateboarder, so I would be able to capture the sounds myself.  The challenge here was how to get quality sounds of the board carving through various snow types with minimal (and ideally no) wind. I decided to try two methods simultaneously and see what worked.  First off, I bought some little windscreens for my Core sound binaural mic.  I attached this to an Edirol R-09 stuffed into my pocket.  Over the course of several sessions, I experimented with several mic placements: taping them to the back of the board on either side (facing backwards to minimize wind), strapping them to the top of my boots, and taping them to the middle of the board on the left and right. I coupled this with a Zoom H2 with a windscreen stretched over the mics held in my hand as low to the ground as I could get it. Again, not the best quality recorders, but with a high impact sport like snowboarding I was only willing to risk my equipment so much!

 

Between the two, over a course of many days, I was able to get some decent sounds across multiple types of terrains.  (The schedule was a scant 2 months, so we ended up forgoing terrain types.  I experimented with changing the terrain based on altitude from powder to packed powder to ice, but it didn’t work very well without a visual or physical change to the terrain). I found the best results actually came from the mics on the inner (wind-protected) side of the binding and holding a recorder low behind me with the mic facing up the mountain (and thus away from the wind direction).

The bulk of what ended up in SummitX Snowboarding was from a bluebird day doing backcountry with my friends Sati and Melody Shah on Rose Knob Peak, near Mt. Rose in North Lake Tahoe.  It was perfect powdery spring conditions with very little wind.  We did see some bear tracks, but fortunately, no bears! To get the sound of your board riding over rock, rather than sacrifice my board, I ran the edges over a stone mortar and pestle and the results worked pretty well. The turns, carves, and powerslides were taken from some of my other recordings, and I made soundsets for more of the planned terrains, which have yet to see the light of day. For the snowboarder movement including clothing, boot sqeuaks and binding creaks, I recorded those in the comfort and relative quiet of my apartment, using Mackie Onyx pres and a Neumann TLM 193.

Here’s a composite from the game:

 

You can also buy it on Android or iOS


Psuedo-Occlusion in Unity

September 30th, 2012

Unity is an awesome engine for quickly iterating and building game content. The audio features have definitely improved over the years, but it’s still rather limited in many ways.  Randomization of pitch, volume, lpf, or even sounds can only be done with a little bit of scripting savvy.  Someday, either Unity will fix this or I’ll publish my scripts for these common practices on the asset store :-).   One of the features we added at Free Game Games which I’m most proud of was a psuedo-occlusion scheme utilizing trigger boxes and an enum (and enumerated static variable) to attenuate and apply a low pass filter on certain sounds. This feature was used prominently throughout Free Range Games’ canceled skateboarding game, as well as Freefall Tournament, which is playable on Kongregate.com.  It was one of the more advanced features we added to Unity on the audio side.  I relied heavily on the scripting wizardry of Jeff Wood, a fantastic designer whom I worked with both at Free Range Games and Shaba Games before that, to handle most of the technical scripting. Our solution was not necessarily the best method to occlude sounds, but it was functional, so I’d like to outline the system so that people can ideally glean some information from it and possibly improve upon it themselves.

The core script components of our system were an AudioOcclusionTrigger and an AudioOcclusionObject.  All objects the we wanted to occlude would have an AudioOcclusionObject script attached to it.  To trigger the occlusion we created trigger boxes and attached an AudioOcclusionTrigger to that trigger box.  Since we may want an object to be occluded in multiple boxes, we created an enum containing a list of occlusion “categories.”  This list was rather arbitrary and dependent on the level design.  I believe we had things like “Hallway,” “ExtAmbience”,  and “IntAmbience” So for example in our skateboarding game, we had a warehouse level in which two cavernous rooms were connected by a small hallway. Each room had an ambient emitter which played a looping ambient drone and  occasional one shots and a PA loudspeaker which was pumping out our licensed music soundtrack. The hallway had a trigger box around it with an AudioOcclusionTrigger script labeled as “hallway.” The ambience and music emitters in each large room where tagged with the hallway enum with their respective AudioOcclusionObject scripts, and whenever the player entered that trigger, all the sounds which contained an AudioOcclusionObject script with the category set to “hallway” would  attenuate and get filtered over time.  And when the player exited that box, the reverse wold happen (the objects volume and lpf would be restored to where they were prior to occlusion).

Here is a short video demonstrating the effect on the music and ambience:

We did a lot of safety checks to make sure the sounds aren’t already occluded before attempting attenuation, creating filters if they don’t exist, etc. but it’s still a far from perfect solution.  It really only works well with fairly simple geometry and the more occlusion categories you add, the crazier it can get to keep track of your objects and make sure everything is set up properly.

But there you go.  If you’d like to download the scripts and check it out yourself, they’re available here.