I’ve written before that at SIGGRAPH you starve while at GDC they stuff you like a foie gras goose.  No more.  This year there were no breakfast pastries before the lectures started.  There were no cookies at the coffee breaks.  Some lines ran out of lunches and sodas.  Perhaps CMP took a look at the girth of the attendees and decided that the average game developer would benefit from a diet.

The lean times extended to the recruiting floor.  It was obvious looking at the career expo that, despite rumors to the contrary, our industry is far from recession proof.  There looked to be about half as many booths as there were last year.

Recruiting aside, GDC looked as crowded as ever–which is pretty impressive considering that attendance last year was 18,000 developers.  Unfortunately, this whole paragraph from my write-up last year is still accurate:

GDC hasn’t managed to scale to handle its growth.  Internet connectivity was poor throughout the convention center…. proceedings didn’t become available at all until after the conference, and those in the form of a slim handful of PowerPoint decks.  The clever people have taken to bringing cameras to the talks and snapping pictures of each slide that goes up, which is distracting but understandable.  The state of the GDC proceedings is an embarrassment to the industry, but it’s one that I don’t expect to change unless CMP stops handing out Gigapasses to speakers who show up without their slides.

This year, even AT&T’s 3G network was overwhelmed.  I had to switch my phone back to the EDGE network to get any bandwidth.   18,000 geeks use a lot of bandwidth.

What’s hot:  SPU task profiles.  Screen-space postprocessing.  iPhones.  AI.  Larrabee.  Producers explicitly defending crunch.  Margaret Robertson.

What’s not:  Spherical harmonics (still).  Every mobile phone except the iPhone.  Scrum (it’s lost its New Process smell… crunch is the new scrum).  The 7 to 9 scale.  OnLive.

In no particular order, these are the talks I found noteworthy:

The Game Design Challenge:  My First Time

This year’s Game Design Challenge was supposed to be about “sex and autobiography,” an assignment that the designers took literally.  The scheduled participants were Kim Swift of Valve, Sulka Haro of Habbo and industry veteran Steve Meretzky.  Management at Valve apparently decided that the topic was too explicit for them to be associated with, though, because Kim Swift withdrew on short notice to be replaced by the two-woman team of Heather Kelley and Erin Robinson.

Kelley and Robinson pitched a series of minigames that explicitly (in every sense) reenacted their first sexual experiences.  They proposed an arcadey Wii-style game, like a cross between Barbie Fashion Designer and Cooking Mama, with challenges such as:

• Pick the least complicated/easiest to remove outfit
• Shave your legs
• Choose the sexiest LP
• Unbuckle partner’s clothing
• Etc…

Sulka Haro pitched “Your First Time”, a 4-6 player co-op anonymous story game.  One player would pick an image from Flickr.  Other players would respond with a brief fiction based on the image, and the original player would judge the results like you would in a game of Apples to Apples.  I think Haro’s submission was the most mature design, but it adhered least to the assigned subject.  The game sounds like it would work as well with stories about Star Trek characters as sexual experiences, and it’s in no way autobiographical.

Steve Meretzky looked back to his Infocom days, proposing a text adventure, “Wait.  Time Passes…” in which the player tried to pick up women at different phases in his life, starting as a loser high school student doomed to failure but eventually growing up into a successful game designer who has all the sex he wants.  The gameplay seemed somewhat questionable, though, since the player’s only real option at any point is to wait until he outgrows his adolescent awkwardness.

Kelley and Robinson won.  I find it notable that the women’s autobiographical stories have no possibility that they’ll fail to find sex if they want to have it, but also make it clear that they’re the ones who initiate the act.  Which I guess means that (a) they’re women and (b) they’re dating geek men, poor things.

Stop Wasting My Time, Margaret Robinson

Margaret Robinson gave the most enjoyable talk of the show.  She analyzed what kind of storytelling does and doesn’t work in games, offering less high-concept theory than a discussion of what storytelling mechanisms did and didn’t work in a variety of specific games.

Robinson cites Call of Duty 4, for example, not for its high-level plot (there’s a terrorist… and Russians… and a nuclear weapon… and, um, I forget the rest) but for the relationship between the SAS operator inhabited by the player and Captain Price, his gruff commanding officer.  At the beginning of the game, the player is the new guy, and Price and the rest of his squad treat him with casual disrespect.  Over the course of the game, the player gets better at the game and the other characters treat him as more of an equal.  Then, in a key chapter, the game does something that no movie or book could do:  it shifts point of view and lets the player become Captain Price in a flashback mission.  When he returns to being an SAS squaddie, his relationship with Captain Price has fundamentally changed.

In the notes for his last novel, F. Scott Fitzgerald wrote, “Action is character.”  And the equality of action and character is very useful to us, Robertson observed, because action is what games are good at.

There are lots of different ways of telling a story in a game, and they form a hierarchy:

• HUD
• art
• animation
• sound
• text
• voice-over
• video

As you move down that list, cost increases while player engagement decreases.  Robinson’s point is that the thing that is easiest is the thing that is best for us!  Implicit storytelling is superior to explicit storytelling in games.  Story in games is important for setting tone and for putting the player in an emotional context.  Exposition is expensive–you need to hire voice actors and make cinematic animations–and it takes the player out of the game.  The best way to tell a story in a game is through the systems with which the player is already interacting.

So if you’re creating a game, ask yourself if you need a story at all.  If so, what’s it for?  What’s it about?  What’s it really about?  What systems does it depict?

“I read a load of game pitches that start with the inner psychology of the character, and that’s not a good place to start,” Robinson said.  ”Think about your game idea, and if it would look really good on the silver screen–dump it!”

Margaret Robertson talks about games with a rare mix of enthusiasm and insight.  If you get a chance to hear her speak, don’t miss it.

Valve’s Approach to Playtesting

I’ve listened to a talk or two about Valve’s playtesting process before, so I wasn’t expecting to learn much from this talk by Mike Ambinder, but it ended up being pretty remarkable.  Where Valve’s earlier playtest talks have focused on why they do user testing, how they recruit testers, and how they integrate testing into their content creation process, this talk focused on the lengths to which Valve goes in order to extract accurate data from their testers.

There are a lot of problems with getting honest responses from experimental participants.  Humans have strong cognitive biases toward consensus, toward confirmation of preconceptions, and toward self-serving interpretations.  Ambinder said, “If you ask a group if a level is hard, the first person to respond wins.”  Even individually, they tend to give the answers they think questioners want to hear.

Valve tries to work around these issues to some extent by using multiple choice surveys so that players can’t equivocate and by asking questions designed to give specific limited factual answers.

A more direct path to honest answers, though, is to use biometic data to measure the player’s emotional state directly instead of quizzing him about how he feels.  Valve tries to gather data by measuring players:

• heart rate
• skin conductance
• eye tracking (blink rate, pupil dilation)
• facial expression
• EEG readings
• EMG readings (electomyography, which infers emotional state by measuring facial muscle contractions)

They caution that all of these measures are noisy and sometimes hard to interpret.  But they’re refreshingly free of the biases that make focus test results so frustrating.

Valve is clearly pushing the edge of the envelope when it comes to gathering data from user testing.

Game Critics Rant

I went to the Game Critics Rant because I wanted to see Tom Chick, who moderates the Quarter to Three forums on which I spend more time lurking than I should.  In the event, Chick didn’t make it–really, SciFi?  You couldn’t afford to buy the guy a lousy plane ticket from LA?–but the rant session ended up being so much fun that I stuck around anyway.

N’Gai Croal, formerly of Newsweek, opened the event by vivisecting the idea of a “hardcore gamer.”  ”Hardcore,” he points out, describes a genre rather than how games fit into people’s lives.  Why is a person who plays Bejeweled sixteen hours a day considered a casual gamer while a person who plays Halo once a week is considered hardcore?  ”Please Google the phrase ‘a new taxonomy of gamers‘,” Croal said.

Stephen Totilo continued with another critic’s rant against his own.  He pointed out that game journalists are often criticized for not being reporters.  ”Our reporting is fine… it’s our writing that sucks,” he observed.  ”We abandon what should be the mandate to write in clear crisp ways.”  He then called on game reviewers to abandon:

• “Compelling”
• “Visceral”
• “Very”
• Adverbs

I thought his rant was very viscerally compelling.  Also, very is an adverb.

Leigh Alexander of Gamasutra gave a rant that was a little incoherent, opening with the complaint that Gamasutra is “dealing with an unpleasable audience that sends the message that they don’t care about quality” and going on to say that there’s a “three-way ecosystem of negativity” between readers, game developers and writers.  She suggested that she wanted to establish trust–but then immediately blamed game studios for making her talk to PR drones instead of letting her talk to developers who’d tell the unvarnished truth.

Still, if her message was a little mixed, she made some good points.  Journalists in the game industry don’t have any contact with the people who actually make games.  Most of them don’t come from game development themselves and don’t have a detailed understanding of the development process.  No matter how talented they are as writers and cultural critics, these are challenges that make it hard for them to do their jobs well.  So if any writers out there ever want analysis from an actual developer, feel free to drop me an e-mail.

Jason Della Rocca talked about why he’s stepping down as chair of the IGDA:  ”In IGDA, 1% were doing the work.  The rest of you were really fucking lazy and didn’t do shit…  Sorry about not doing better when you guys bitched of getting you to do stuff.”  Geez.  All I said was that I missed the cookies.

Author Heather Chaplin continued the transition from critics criticizing their own industry to criticizing mine.  Specifically, she complained that we’re still making games about space marines and big-boobed chicks in metal bikinis.  Some people say that the industry is still immature because games are an art form that’s only 35 years old, she said.  But by the time rock and roll was 35 years old, it had created The Beatles.  By the time movies were 35 years old, they’d produced Fritz Lang and film noir.  The problem, she claimed, isn’t that games development is in its adolescence, it’s that game developers are permanently stuck in adolescent “guy culture”.  Guys, Chaplin said, fear four things:

• Responsibility
• Introspection
• Intimacy
• Intellectual discovery

When you’re talking about culture makers, she said, this is a problem.  ”What, if you sure that you’re really sensitive people might pick up on that you’re basically a bunch of kids masquerading as men?”

I’ve got a lot of respect for Chaplin for having the guts to get up in front of a room full of developers and give that rant, but I think her ire is misplaced.  Here are the ten best-selling games from 2008:

1. Wii Play
2. Mario Kart (Wii)
3. Wii Fit
4. Super Smash Bros:  Brawl
5. Grand Theft Auto IV (Xbox 360)
6. Call of Duty:  World at War (Xbox 360)
7. Gears of War 2
8. Grand Theft Auto IV (PS3)
9. Madden NFL 09
10. Mario Kart (Nintendo DS)

Here are the ten best-selling movies from 2008:

1. The Dark Knight
2. Iron Man
3. Indiana Jones and the Kingdom of the Crystal Skull
4. Hancock
5. WALL-E
6. Kung Fu Panda
7. Twilight
8. Madagascar:  Escape 2 Africa
9. Quantum of Solace
10. Dr. Seuss’ Horton Hears a Who!

I’m supposed to believe that game developers are adolescent because they make Mario Kart instead of Horton Hears a Who?  Big hit movies are every bit as childish as big hit games.  Sure, you can find movies that actually say something meaningful about the human condition, if you watch indie films that don’t sell very well.  But you can also find games that challenge players preconceptions and aspire to be art if you look at indie games instead of the big-budget lowest-common-denominator stuff that clogs the shelves at Gamestop.

Chris Hecker was invited up to present this year’s “Duct Tape Award” to Ubisoft’s Clint Hocking for the best rant from last year.  Hecker then gave a rant of his own, starting from three axioms:

• Journalism is an important job.  (Probably even more important than game development.)
• Journalism affects people.
• With great power comes great responsibility.

Hecker then proceeded to eviscerate two particular stories.  In one, 1up took two posts in different on-line forums and turned them into an article titled “Chris Hecker Apparently Responsible For Simplified Sport Gameplay”–despite the fact that one of the two forum posts consisted of Hecker calling the accusation “nonsense”.  No one at 1up ever actually tried to talk to Hecker about the story.  In his other citation, Hecker quoted as an example of game journalism:

“There’s a tendency among the press to attribute the creation of a game to a single person,” says Warren Spector, creator of Thief and Deus Ex.

Ouch.  Coming on the heels of so many liberal arts majors, Hecker rants like a programmer:  his points are like mathematical proofs.  There’s just not really any counterargument to make.

The final rant, from Adam Sessler of G4, was titled “Fuck Metacritic”.  Sessler said that he had heard from an angry developer because he had given a game a 40%.  But he’d given the game two stars out of five.  When Sessler complained to Metacritic that they shouldn’t map that to a 40% score, they told him that they’d map it however they pleased.  Sessler said he didn’t understand how a hundred-point scale worked, anyway:  ”Somebody in the room tell me the difference between a 73 and a 74!”

Sessler lamented that game publishers are now actually using Metacritic scores to determine how much developers get paid.  ”Shame on you ever single publisher that’s pulling this stunt…. There’s a really good tool for determining if a game as good or not:  the market!”

Robot Testing to the Rescue

Paul Du Bois of Doublefine described how the company uses a robot “player” scripted in Lua to drive functional tests of Brütal Legend, their upcoming open world rock-and-roll fantasy game (which looks like Savage Skies).  Their Lua scripts simulate controller input to progress through combat, skip cinematics, and generally cheat a lot.

This is all administered by a TestRunner app running on a host PC that looks for idle dev kits and starts test sessions of the game on any that haven’t been used in the last five minutes.  TestRunner monitors each test for errors, warnings and crashes.

Their Lua implementation emulates cooperative multitasking with sleep calls.  A player test script might look something like:

function Class:waitForActiveLine(self, ent)
while true do
  self:sleep(0)
    if ent.CoVoice.HasActiveVoiceLine then
      return
    end
  end
end

As a debugging tool, this has several useful characteristics.  ”Failed tests tend to be that the game has crashed or the bot has gotten stuck,” Du Bois said.  They also catch “desyncs”–determinism failures when they’re running two networked copies of the game in lockstep during multiplayer.  And they generate graphs of performance/memory usage that can be analyzed either over the course of a single game session or from one build to another.

Debug builds are usually so slow that humans don’t want to play them.  But the test robot is infinitely patient.  It will play at any framerate, however slow, and crashes with debug symbols.

Best quotes from GDC

“Entertainment is supposed to be enjoyable.  If it can’t be enjoyed, it’s not the consumer’s fault.  The fault belongs to us.”  – Satoru Iwata

“There’s nothing more humbling than looking at regular people trying to play your game without assistance.”  – Gordon Walton

“One of the danger signs [on Star Wars Galaxies] was that people didn’t play the game.  In a AAA game company, everybody plays the game.”  – Rich Vogel

The Race For a New Game Machine, by David Shippy and Mickie Phipps, is a clumsily-written and frustratingly vague book, but game developers should read it anyway.

Shippy led the team that developed the PowerPC core common to the PS3’s Cell processor and the Xbox 360 CPU.  Although his subordinate Mickie Phipps is given equal authorial credit, Race is written entirely in Shippy’s voice.  Shippy was middle management, two levels up from the engineers designing the logic and circuitry of the processor and two levels down from the corporate VPs negotiating with Microsoft and Sony.  As a result the book mixes corporate politics and technical detail.  Unfortunately it rarely gives a satisfying dose of either.

“When the [Halo 3] player swings the joystick and levels a weapon at a charging alien beast,” Shippy writes in his introduction, “then presses the button and showers it with lead, splattering it straight back to hell, the quality of the experience depends less on the fancy code written by the people at Microsoft than on the processor brains in the chip inside the box.”  I’m a software developer, so my biases are obvious, but the primacy of hardware strikes me as a difficult argument to make.  There are plenty of games available for the Xbox 360 that most players would agree don’t come close to the “quality of the experience” delivered by Halo, though they run on the same platform.  And consumers voting with their money seem to think that Wii Fitness delivers a higher-quality experience than Halo does, on hardware with a fraction of the processing power.  I’d argue that hardware has been a commodity for some time, that game code is in the process of becoming a commodity, and that for a game to differentiate itself in the market today, it has to do so in the quality of its creative attributes:  art direction, story and game design.

Unfortunately Shippy has a habit of  resorting to purple prose at the cost of clarity and accuracy.  Consider his description of the history of electronic gaming:

Throughout the 1970s and 1980s the PC industry skyrocketed, too, providing the perfect low-cost platform for game developers.  They capitalized on each step up in the PC’s speed or bandwidth, each increase in the size of its memory or access time, and every decrease in power consumption.  Graphics engines (separate chips that could handle the graphics creation for the games) allowed the PC’s microprocessor to focus on the calculations to create characters that move and behave more like real people and environments that are more intense…. The rise in the PC games market dealt a near death blow to the game console industry.  By the early 1980s, even with new models still on the shelf, consumers were rapidly losing interest in the aging Atari game machine, and no heir-apparent was in sight…. It took nearly a decade, but the game console market suddenly bounced back from the dead with the launch of Sony’s Playstation in 1994…

In the space of a page, Shippy gives the impression that

• Consumer PC 3D graphics cards existed in the eighties
• PC game developers care about power consumption
• Gaming PCs with graphics and sound acceleration killed the Atari 2600
• The Nintendo NES never existed

Coming from a pair of engineers, such technical imprecision is astonishing.  The clumsiness extends to the authors’ use of language as well.  They “hone in on” solutions and “flush out ideas.”  Shippy interrupts a discussion of logic changes with the tortuously mixed metaphor, “In other words, there would be road construction to add new highways, and the traffic cop would have a lot more balls up in the air to juggle.”

But if a good copy editor could have helped the writers with their wording, he would have been powerless to correct their technical sloppiness.  For example, Shippy provides the oddest description of a game’s update-render loop that I’ve ever heard:

Although it may seem like magic to most people, game code is really quite simple.  It can fundamentally be broken down into three major options.  First it requires incoming streams of large amounts of data generated from a peripheral such as a joystick, controller, or keyboard.  This is called “read streaming.”  Second, that incoming data stream has to be processed by one or more numeric engines.  Finally, the data processed in the numeric engines has to be streamed back out to the video display to create the colorful graphics.  This is called “write streaming.”

I’ve worked as a software engineer on games for more than a decade, most of that in core technology, and I’ve never heard of “read streaming” or “write streaming” before.  Controller inputs provide a trivially small amount of data.  (That’s why networked real-time strategy games tend to sync inputs and run deterministic simulations instead of syncing all game state.)  Write streaming, or “rendering” as we call it in the biz, is more memory-bandwidth intensive, but Shippy confuses matters by blurring the line between operations performed on the CPU and operations performed on the graphics processor.

Whether Shippy intends “write streaming” to refer to the transfer of data from CPU to GPU or from the GPU to the framebuffer, it accounts for a small fraction of a frame.  Most of the CPU processing time for any game is spent marshaling data and performing general-purpose logic, with smaller portions of the frame devoted to number-crunching for physics, animation or graphics.  Or as Chris Hecker puts it, “graphics and physics grind on large homogenous floating point data structures in a very straight-line structured way.  Then we have AI and gameplay code.  Lots of exceptions, tunable parameters, indirections and often messy.  We hate this code, it’s a mess, but this is the code that makes the game DIFFERENT.”

You have to wonder if the difference between Hecker’s understanding of how a frame loop operates and Shippy’s is part of the reason why the PS3 contains a CPU optimized for stream processing rather than one optimized for messy game logic.  And puzzles like this are the reason why The Race for a New Game Machine is worth reading despite its flaws.  The book tells a story, and if it’s awkwardly written and frustratingly vague, it also offers occasional fascinating insights into how the consoles on which we develop were created.

As the book relates, the Power core used in the Xbox 360 and the PS3 was originally developed in a joint venture between Sony, Toshiba and IBM.  While development was still ongoing, IBM–which retained the rights to use the chip in products for other clients–contracted with Microsoft to use the new Power core in their console.  This arrangement left Sony engineers in an IBM facility unknowingly working on features to support Sony’s biggest competitor, and left Shippy and other IBM engineers feeling conflicted in their loyalties.

Race paints a striking picture of the different corporate cultures at Sony and Microsoft.  Sony comes across as a vast impersonal hierarchy ruled by the impulses of Ken Kutaragi, then head of Sony Computer Entertainment.  Most of the high-level innovations of the Cell processor are attributed to Kutaragi.  So are other  strange and whimsical mandates.  At one point Kutaragi demands that the Cell must have eight Synergistic Processing Units, rather than the originally agreed-upon six, because “eight is beautiful.”  IBM complies and produces a chip with eight SPUs.  (The book doesn’t mention that, as Wikipedia points out, the PS3 only has “six accessible [SPUs] … to improve production yields.”  It also never mentions that the 4 GHz core Shippy set out to design became a 3.2 GHz core in the shipping consoles, presumably also to improve yields.)

Microsoft is the agile hare to Sony’s tortoise.  Individual Microsoft engineers come across as having more initiative–and a certain cockiness to go with it–but their desires are less grandiose than Kutaragi’s and they seem to have a better idea what they want.  Microsoft is focused on software and services.  They want simpler hardware than Sony’s Cell processor and they’re flexible in cutting features as long as they make their intended ship date.  Because they’re developing a processor more similar to an off-the-shelf PowerPC chip, Microsoft is able to put development kits in the hands of game developers a year and a half before Sony.  Because they control their manufacturing risks, they’re able to put final hardware on shelves a year before Sony.

For readers who are more familiar with the design of software than hardware, The Race for a New Game Machine offers a window into a strange parallel world where much is intriguingly similar and some things are very different.  In the early days of development on the new Power core, Shippy describes junior engineers “chomping at the bit to dive into the detailed implementation” before the high-level design is done, an experience familiar to most experienced software developers.  But a few pages later, Shippy tells how a “junior engineer’s face lit up with enthusiasm” at the idea of getting a new idea patented.  Software patents are anathema to most game developers!  The technical work the book describes is a mix of logic design, which sounds familiar to anyone used to writing code, and physical layout, which sounds more like playing SimCity as the chip designers map that logic onto actual circuits on the smallest chip possible.  Finally, Shippy’s description of the brutal crunch engineers experienced in finishing the chip’s design will be all too familiar to anyone who’s shipped a game.

One of the unnecessary risks Sony took on in developing the PS3 was an attempt to build their own graphics processor.  Toward the end of the Cell’s development, Shippy writes, “the graphics chip designed by the Sony engineers was slipping schedule.  Sony had designed simple graphics chips in the past, but it was clearly not their area of expertise.  After many months of schedule slips, it became obvious that Sony could not deliver the promised graphics chip.”  This story raises as many questions as it answers.  Was Sony trying to develop a full GPU, or merely developing a rasterizer in the assumption that the Cell’s SPUs could handle vertex transformation?  If the latter, did the graphics processor fail to perform, did the SPUs, or could the platform not transfer data from SPUs to processor fast enough?

The story raises as many questions as it answers.  But if The Race for a New Game Machine hadn’t been written, we wouldn’t know to ask those questions at all.  We’re better off for having read the book.  It’s just a shame the book’s not better.

Love is patient, love is kind.  It does not envy, it does not boast, it is not proud.  It is not rude, it is not self-seeking, it is not easily angered, it keeps no record of wrongs.  Love does not delight in evil but rejoices with the truth.  It always protects, always trusts, always hopes, always perseveres.

1 Corinthians 13

Love is easy.  Middleware is hard.

We used middleware on Fracture for physics, tree rendering, audio, animation, facial animation, network transport, and various other systems.  Now that the game is finished, we’re taking stock of the lessons that we learned and deciding what packages we want to keep and what we want to replace as we move forward.

Middleware offers two main benefits, each of which is balanced by an associated cost:

1.  Middleware provides you with more code than you could write yourself for a fraction of what it would cost you to try.  No matter how clever you are, that’s how the economics of the situation work.  The costs of developing a piece of software are largely fixed.  But once written, any piece of software can be sold over and over again.  Because they can spread their costs over a large number of customers, middleware vendors can afford to keep larger, more experienced teams working on a given piece of functionality than independent game developers can.  During Fracture, we watched a couple of pieces of immature middleware spring up after we’d written similar tools ourselves.  We watched those pieces of middleware grow and surpass our tools, because we couldn’t afford to devote the resources to our tools that the middleware vendors devoted to theirs.

The corresponding cost is that none of the functionality you get is exactly what you would have written yourself, and much of it will be entirely useless to you.  Middleware is written to support the most common cases.  If you have specific needs that differ from the norm, you may be better off implementing a solution yourself.

2.  Middleware offers structure.  Middleware draws a line between the things that you have to worry about and the things you don’t.  As long as it’s reasonably well documented and stable, you don’t need to waste mental bandwidth worrying about the things that go on underneath your middleware’s public API.  As games grow ever-larger and more complex it’s become incredibly valuable to be able to draw a line and say, The stuff on the other side of that line isn’t my responsibility, and I don’t have to worry about it.

The associated cost is that you can’t change what’s on the other side of that line.  If you’re going to use middleware, you have to be willing to accept a certain amount of inflexibility in dealing with the problems that the middleware solves.  You have to be willing to shape your own technology to suit the third-party libraries that you’re buying.  Trying to do otherwise is a recipe for misery.

Given those benefits and costs, it makes sense to use middleware wherever you can–as long as you don’t try to license technology in the areas where you want your game to be unique.  Every game has certain unique selling propositions:  things that make it distinct from every other game on the market.  Likewise, every game has certain characteristics that it shares with many others.  When it comes to the latter, you should buy off-the-shelf technology, accept that technology’s inflexibility, and modify your game design to suit it.  When it comes to the former, you should write the code in-house.  As Joel Spolsky says, don’t outsource your core competency.

So now that you’ve figured out which game functions you should be implementing through middleware, how do you decide which of the scads of available middleware packages is best for you?  There are a number of issues to keep in mind.

Good middleware lets you hook your own memory allocator.  If your approach to memory is to partition the entire free space up front and minimize runtime allocations, then you’ll want the ability to reserve a block for your middleware and allocate out of that.  Even if you allow dynamic allocations at any time, you’ll want to track how much memory is used by each system and instrument allocations to detect memory leaks.  Any piece of middleware that goes behind your back and allocates memory directly just isn’t worth buying.

Good middleware lets you hook your own I/O functions.  Most games store resources in package files like the old Doom WAD files.  Middleware that doesn’t let you hook file I/O doesn’t let you put its resources in packfiles.  Many modern games stream resources off DVD as the player progresses through a level.  If you can’t control middleware I/O operations, then you can’t sort file accesses to minimize DVD seeks.  You’ll waste read bandwidth and your game will be subject to unpredictable hitches.  Again, any piece of middleware that does file I/O directly just isn’t worth buying.

Good middleware has extensible functionality.  No middleware package will do everything you want out of the box.  But you shouldn’t have to modify any piece of middleware to make it do what you need.  Good middleware offers abstract interfaces that you can implement and callbacks that you can hook where you need to do something unique to your game.  An animation package may let you implement custom animation controllers.  A physics package may let you write your own collision primitives.  Your objects should be first-class citizens of your middleware’s world.

Good middleware avoids symbol conflicts.  Beware of middleware that uses the Standard Template Library carelessly.  It’ll work fine until you try to switch to STLPort or upgrade to a new development environment, and then you’ll suddenly find that your engine has multiple conflicting definitions of std::string and other common classes.  To avoid symbol clashes, every class in a middleware library should start with a custom prefix or be scoped inside a library namespace.  And if a piece of middleware is going to use the STL, it should do so carefully, making sure that every STL class instantiated uses a custom allocator.  That allows you to hook your own memory allocator and avoids symbol conflicts.

Good middleware is explicit about its thread safety.  We live in an ever-more-multithreaded world, but one where most game engines are still bound by main thread operations most of the time.  For best performance, you want to offload any operations you can onto other threads.  To do that with a piece of middleware, you need to know which operations can be performed concurrently and which have to happen sequentially.  Ideally a piece of middleware will let you create resources asynchronously so you can construct objects in a loader thread before handing them off to the game.

Good middleware fits into your data pipeline.  Most companies export data in an inefficient platform-independent format, then cook it into optimized platform-specific formats and build package files out of those as part of a resource build.  Any piece of middleware should allow content creators to export their assets in a platform-independent format.  Ideally, that format should still be directly loadable.  The build process should be able to generate platform-specific versions of those assets using command-line tools.

Good middleware is stable.  One of the main benefits of middleware is that it frees your mind to focus on more critical things.  In this respect, middleware is like your compiler:  It frees you from having to think about low-level implementation details–but only as long as you trust the compiler!  A buggy piece of middleware is a double curse, because instead of freeing your attention from a piece of functionality, it forces you to focus your attention there, on code that was written by a stranger and that no one in your company understands.  Worse still, if you’re forced to make bug fixes yourself, then you need to carry them forward with each new code drop you get of your middleware libraries.  You shouldn’t need to concern yourself with the implementation details of your middleware.  Middleware is only a benefit to the extent that its API remains inviolate.

Good middleware gives you source.  Despite the previous point, having access to the source for any middleware package is a must.  Sometimes you’ll suspect that there’s a bug in the middleware.  Sometimes you need to see how a particular input led to a particular result before you can understand why the input was wrong.  Sometimes you’ll have to fix a bug no matter how good the middleware is.  And frequently you’ll need to recompile to handle a new platform SDK release or to link with some esoteric build configuration.

There are other questions to ask about any piece of middleware:  How much memory does it use?  How much CPU time does it require?  What’s the upgrade path for your current code and data?  How does it interact with your other middleware?  How good is the vendor’s support?  How much does it cost?

But those questions have vaguer answers.  Acceptable performance or cost will vary depending on the nature of your product.  A cell phone game probably can’t afford to license the Unreal Engine–and probably couldn’t fit it in available memory if it did!  Data upgrade paths are less of a concern if you’re writing a new engine from scratch than if you’re making the umpteenth version of an annual football game.

The rules that don’t vary are:  You should buy middleware wherever you can do so without outsourcing your core competency.  And to be worth buying, any piece of middleware should behave itself with respect to resource management and concurrency.

Many systems naturally lend themselves to a design that gathers the high-level functionality of the system behind a single interface:  A FileManager, for example, might expose functions for building a file database from packfiles and loose files.  An AudioManager might expose functions for loading and playing sound cues.  A SceneManager might expose functions for loading, moving and rendering models.

Once upon a time, these objects would all have been global variables.  There are a number of problems with using global objects as managers, though, the most critical of which is the uncertainty of static initialization and destruction order.  Your managers start out in a pristine garden of Eden, free of all knowledge of good, evil, and (most importantly) one another.  But as your game gets more and more complicated, your managers are going to develop more dependencies on one another.  To add streaming of sounds to AudioManager, for example, you may need to make it reference FileManager.  To stream asynchronously, you may also need to reference AsyncJobManager.  But at construction time, you can’t be sure that any of those managers exist yet.

This prompts clever people who’ve read Design Patterns to think, Aha!  I’ll make all my managers Meyers singletons!  And they go and write something like

FileManager& GetFileManager()
{ 
    static FileManager theFileManager; 
    return theFileManager;
}

This function will create a FileManager object the first time it’s called and register it for destruction at program exit.  There are still a couple of problems with that, though:

• First, like all function-local static variables in C++98, theFileManager is inherently unthreadsafe.  If two threads happen to call GetFileManager simultaneously, and theFileManager hasn’t been constructed yet, they’ll both try to construct it.  Wackiness ensues.  This quibble is due to be fixed in C++0x.
• Imagine what happens if the constructor for AudioManager calls GetFileManager and the constructor for FileManager calls GetAudioManager.  To quote chapter and verse of the ISO C++ Standard, “If control re-enters the declaration (recursively) while the [static] object is being initialized, the behavior is undefined.” (ISO Standard 6.7.4) The compiler can do whatever it wants, and whatever it does is unlikely to be what you want.
• Although Meyers singletons give you a JIT safe order of construction, they make no promises about their order of destruction.  If AudioManager is destroyed at program exit time, but FileManager wants to call one last function in AudioManager from its destructor, then again you have undefined behavior.

We dealt with those issues in MechAssault 2 by eschewing automatic singletons in favor of explicitly constructed and destroyed singletons.  Every manager-type object had an interface that looked something like 

class FileManager
{
public:  
    static bool CreateInstance();
    static void DestroyInstance();
    static FileManager* GetInstance();
};

This works better than automatic singletons.  It worked well enough that we were able to ship a couple of quite successful games with this approach.  We wrote a high-level function in the game that constructed all the managers in some order and another high-level function that shut them all down again.  This was thread-safe, explicitly ordered and deterministic in its order of destruction.

But as we added more and more library managers, cracks started to show.  The main problem was that although we constructed managers in an explicit order, dependencies between managers were still implicit.  Suppose, for example, that AudioManager is created before FileManager, but that you’re tasked with scanning the filesystem for audio files at start-up time.  That makes AudioManager dependent on FileManager for the first time.  Now AudioManager needs to be created after FileManager.

Changing the order in which managers were constructed was always fraught with peril.  Finding a new working order was a time-consuming process because ordering failures weren’t apparent at compile time.  To catch them, you needed to actually run the game and see what code would crash dereferencing null pointers.  And once a new ordering was found, it needed to be propagated to every application that used the reordered managers.  Day 1 has always placed a strong emphasis on its content creation tools, and most of those tools link with some subset of the game’s core libraries, so a change might need to be duplicated in six or eight tools, all of which would always compile–but possibly fail at runtime.

With Despair, one of our governing principles has been to make implicit relationships explicit.  As applied to system manager construction, that meant that managers would no longer have static GetInstance functions that could be called whether the manager existed or not.  Instead, each manager takes pointers to other managers as constructor parameters.  As long as you don’t try to pass a manager’s constructor a null pointer (which will assert), any order-of-initialization errors will be compile-time failures.  To make sure that managers are also destroyed in a valid order, we use ref-counted smart pointers to refer from one manager to another.  As long any manager exists, the managers that it requires will also exist.

Our current code looks something like  

class IAudioManager
{
public:
   IAudioManagerPtr Create(const IFileManagerPtr& fileMgr);
};

class AudioManager
{
public:
   explicit AudioManager(const IFileManagerPtr& fileMgr);
};


One problem remains.  Although AudioManager exposes the public interface of the audio library to the rest of the application, there are also private support and utility functions that the audio system should only expose to its own resources.  Furthermore, many of these utility functions will probably need access to lower-level library managers.

This could be solved by having global utility functions that take manager references as parameters and by making every audio resource hold onto pointers to every lower-level manager that it might use.  But that would bloat our objects and add reference-counting overhead.  A more efficient and better-encapsulated solution was to give each library a context object.  As Allan Kelly describes in his paper on the Encapsulated Context pattern:  

A system contains data that must be generally available to divergent parts of the system but we wish to avoid using long parameter lists to functions or globaldata.  Therefore, we place the necessary data in a Context Object and pass this object from function to function.    

In our case, that meant wrapping up all smart pointers to lower-level libraries in a single library context, which would in turn be owned by the library manager and (for maximum safety) by all other objects in the library that need to access a lower-level manager.  Over time, other private library shared state has crept into our library contexts as well.  To maintain a strict ordering, the library manager generally references all the objects that it creates, and they reference the library context, but know nothing about the library manager.

This architecture has generally worked well for us, and has been entirely successful in avoiding the manager order-of-creation and order-of-shutdown issues that plagued earlier games we worked on.  It does have definite costs, however:

•  A lot of typing.  Library managers know about library resource objects which know about library contexts.  But frequently there’s some functionality that you’d like to be accessible both to objects outside the library and objects inside the library.  In this architecture, there’s nowhere to put that functionality except in the library context, with pass-through functions in the library manager.  Since the library manager is usually hidden behind an abstract interface, you can end up adding function declarations to three headers and implementations to two cpp files before you’re done.
• Even more typing.  All the lower-level managers held by a library also get passed through a creation function to the library manager’s constructor to the library context’s constructor.  That was a mild annoyance when the engine was small and managers were few, but Despair now comprises over fifty libraries.  Most of those libraries have managers, and there are a handful of high-level library managers that take almost all lower-level library managers as constructor parameters.  Managers that take forty or fifty smart pointers as parameters are hard to create and slow to compile.
• Reference counting woes.  In principle, every object should strongly reference its library context to maintain a strict hierarchy of ownership and to make sure that nothing is left accessing freed memory at program shutdown time.  In practice, though, this doesn’t work well when objects can be created in multiple threads.  Without interlocked operations, your reference counts will get out of sync and eventually your program will probably crash.  But with interlocked operations, adding references to library contexts becomes much more expensive, and can become a significant part of your object creation costs.  In practice, we’ve ended up using raw pointers to contexts in several libraries where the extra safety wasn’t worth the additional reference cost.

So managers and contexts are far from perfect.  They’re just the best way we’ve found so far to stay safe in an ever-more complex virtual world.

Nonetheless, every game engine we’d worked on before Despair had a similar deep inheritance hierarchy of the sort that was in vogue in the mid-nineties:  a player class might inherit from some kind of combatant class, which would inherit from a mover class, which would inherit from a physical object class, which would inherit from a base game object class.

This architecture has a lot of shortcomings.  Let me enumerate a few of them:

First, it’s inflexible.  If you want to create a new AI enemy that has some of the capabilities of enemy A and some of the capabilities of enemy B, that’s not a task that fits naturally into a hierarchical object classification.  Ideally, you’d like the designers to be able to create a new enemy type without involving you, the programmer, at all.  But with a deep object hierarchy, you have to get involved and you have to try to pick the best implementation from several bad options:  to have your new enemy class inherit from one object and cut-and-paste the functions you need from the other; to not inherit from either, and to cut-and-paste the functions you need from both; or to tiptoe down the treacherous slippery slope of multiple inheritance and hope that it doesn’t lead to a diamond of death.

Second, a handful of classes in your hierarchy tend to grow without bound over a game’s development.  If the player class is part of the object hierarchy, then you can expect this class to include input and control systems, custom animation controls, pickup and inventory systems, targeting assistance, network synchronization–plus any special systems required by the particular game that you’re making.  One previous game that we worked on features a 13,000 line player class implementation, and the player class inherited from a 12,000 physical object class.  It’s hard to find anything in files that size, and they’re frequent spots for merge conflicts since everyone’s trying to add new stuff to them all the time.

Third, deep inheritance is poor physical structure.  If class A inherits from class B which inherits from class C, then the header file for A–A.h–has to #include B.h and C.h.  As your hierarchy gets deeper, you’ll find that all of your leaf classes have to include four or five extra headers for their base classes at different levels.  For most modern games, as much compile time is spent opening and reading header files as is spent actually compiling code.  The more loosely your code is coupled, the faster you can compile.  (See Lakos for more details.)

Therefore we resolved to make Despair as component-based as we could, and to keep our inheritance hierarchies as flat as possible.  A game object in Despair, for example, is basically a thin wrapper around a UID and a standard vector of game components.  Components can be queried for type information and dynamically cast.  The game object provides only lifetime management and identifier scoping.  It knows nothing about component implementations.  It contains no traditional game object state like position, bounds, or visual representation.

This approach has informed other systems as well.  Our scene object implementation is similar to the game object implementation, with a single object representing each model that provides lifetime management for a vector of scene nodes.  Scene nodes manage their own hierarchies for render state or skeletal transforms.

Another family of systems is built on a flow-graph library for visual editing.  Game logic, animation systems, and materials can all be built by non-programmers wiring together graph components in the appropriate tools.

Using composition instead of inheritance has worked very well for us.  Our primary concern when we set out in this new direction was that we’d end up with something that had horrible runtime performance.  With Fracture almost complete, though, there’s no evidence that our performance is worse than it would have been with a deep inheritance hierarchy.  If anything, I’m inclined to suspect that it’s better, since well-encapsulated components have better cache locality than large objects and since the fact that we only update dirty components each frame means that we can decide what does and doesn’t need to be updated at a finer granularity.

If I were starting over again, the only change I’d make with respect to object composition is to make scene objects more opaque and less like game objects.  Scene objects have a different problem to solve.  We have several hundred game components now, with more going in all the time, and the flexibility of having a thin game object interface that allows querying components for type has paid big dividends.  I think that our game object system is close to ideal for iterating rapidly on gameplay.  Scene objects are a different kind of problem, though.  We haven’t added any new scene node types since the scene library was written, and all scene nodes are implemented in the scene library instead of in higher-level code.  At the same time, it would be nice to be able to experiment with different optimizations of updating skeletal hierarchies without breaking higher-level code.  All of this argues that following the Law of Demeter and hiding scene object implementation details would have been appropriate for scene objects even if it wasn’t appropriate in the rapid-prototyping environment of game objects.

Beyond the perfect abstract world of software architecture, component based design also created a couple of surprises in the messier world of development process and human interaction.  One lesson of working with a composition-based engine is that the learning curve for new programmers is steeper.  For programmers who are used to being able to step through a few big nested functions and see the whole game, it can be disorienting to step through the game and discover that there’s just no there there.  For example, Despair contains over 500 classes with the word “component” in their names and 100 classes with the word “object” in their names.  Our games aren’t defined by C++ objects, they’re defined by relationships between them.  To understand those relationships, you need good documentation and communication more than ever.

Another composition lesson is that component-based design takes a lot of the complexity that used to exist in code and pushes it into data.  Designers aren’t used to designing objects and constructing inheritance hierarchies.  Working with components requires new processes and good people.  Cross-pollination is important.  Your programmers need to work building objects in your tools, as well as writing code for components, and they need to work hand-in-hand with good technical designers who can provide tool feedback and build on the object primitives available to them.  Like a game, your team isn’t defined by its individual components but by the relationships between them.

The engine that we used at Cyan, for example, was a pure scene graph.  Every part of the game was represented by one or more nodes in a hierarchy.  But the hierarchy represented logical relationships in some places and kinematic relationships in others.  Throughout the graph, ownership was conflated with update order:  children would be deleted when their parents were deleted and parents always updated before their children.  Kinematic attachment was performed by pruning and grafting trees in the graph, which had the effect of tying the lifetimes of attached objects to the lifetimes of their parents.

The engine used for MechAssault 2 was similar, except that it had separate game object and scene hierarchies.  In both hierarchies, graph relationships defined ownership and update order:  children would be deleted when their parents were deleted and parents were always updated before their children.  In the game object graph, hierarchy also defined logical relationships that varied based on the objects involved.  Triggers, for example, would treat certain of their child objects as inputs to be checked and others as objects to be acted on.  In principle, the entire game object hierarchy updated, then the entire scene hierarchy updated.  In practice, we needed to know the final animated positions of some objects so that we knew where to create effects, so some trees in the scene hierarchy got explicitly updated during the game object update.

All of this was very confusing.  It also meant that changes to the hierarchy motivated by one concern tended to have unforeseen side-effects in other areas.  And it necessitated odd hacks like the MechAssault’s explicit updating of sub-trees in the scene graph. In developing Despair, therefore, we decided to have one data-structure per purpose.  As mentioned previously, we have a well-defined ownership hierarchy among our objects.  For example,

• The streaming manager owns worlds
• Worlds own game objects
• Game objects own game components
• Game components own scene objects (and audio resources, and HUD elements, etc…)
• Scene objects own mesh nodes (and transform nodes, and light nodes, etc…)
• Mesh nodes own vertex groups and index lists

In some cases ownership is shared and in some cases it isn’t, but each object has its level of the hierarchy.  But in addition to the hierarchy that we use for ownership, we also have parallel data structures for

• the object registry used for object look-up
• the dependency graph used for object update
• the physical representation of the world (stored in Havok’s internal hierarchy)
• the renderable representation of the world (stored in our own sphere tree of renderable objects)
• etc.

This has been a big win in terms of clarity and flexibility.  It’s clear when looking at a component what its lifetime will be.  It’s easy to make a component update later in the frame, or not update at all, without having to worry that your change will affect when the object is deleted or whether it’s rendered.  Scene objects and game components are part of the same dependency hierarchy.  Therefore, the component that creates a fire effect can make itself dependent on the scene object for the player’s weapon.  That way the fire effect won’t be created until the weapon has determined its final position for the frame, and the fire effect will be created at the correct location.

The performance implications of orthogonal views are unclear.  One of the frustrations of building a game engine is that you never have the leisure of changing one architectural variable while keeping all others constant.  That means that you can never be sure you’ve made the best decision.  You can only be sure–sometimes–when you’ve made a very bad one.Having orthogonal views of our data was clearly not a decision worthy of regret.  But it’s difficult to be certain whether it’s a net win in terms of performance.  On the negative side, all these different hierarchies are scattered at widely different locations in memory, and on modern hardware memory locality drives performance more than the number of instructions executed does.  But on the other hand, having independent views of the data means that each only needs to process what matters.  Our update pass only updates objects that are actually changing.  Our render pass doesn’t waste any overhead on objects that aren’t renderable.  And when those systems operate, they perform similar operations on sorted lists of similar objects, which is the sort of thing that modern hardware has gotten very good at.  So I think this idiom is a performance win, but the only hard data I have to back that up is that we’re spending less time waiting on cache misses than we were in MechAssault 2.  Given the other differences between Despair and the MechAssault 2 engine and between the Xbox 360 and the original Xbox, that evidence is suggestive but not conclusive.

The one area in which orthogonal views have a clear cost is memory.  As objects add themselves to all our different systems, they may have representations in the registry, in the dependency graph, the scene hierarchy, the physics hierarchy, the render hierarchy, and probably one or two other systems that I’m forgetting.  I’d estimate that this overhead adds up to somewhere between two and three megabytes for a typical level in Fracture.  On a console with 512 megabytes of memory that cost seems justifiable, but it would have been less practical on an Xbox or PlayStation 2 game.

We wanted to improve stability and leak less memory in Despair, and this led us to consider exception safety.  Exception safety is the idea that a program should always remain in a valid state in the presence of exceptions.  Operations should succeed or fail, but if they fail, they should have no side effects and program data should remain unchanged.  Memory leaks frequently occur when exception unsafe code encounters an exceptional situation.  A function may throw an exception to signal bad data, for example, and thereby skip clean-up of temporary objects done at the end of the function.

Despair doesn’t use exceptions, because they introduce significant overhead, but we do use error codes and FAIL_RETURN macros that may early exit from a function.  These introduce similar problems.  Therefore we try to be exception-safe wherever possible without hurting performance.  In practice, performance concerns generally keep us from using the PIMPL idiom to implement atomic transactions.  But we do use the Resource Acquisition Is Initialization idiom extensively.  Our XML writer uses scoped EnterElement/LeaveElement objects.  Our critical sections use scoped locks.  And we use smart pointers extensively.

The Despair Engine contains about ten smart pointer types, depending on how broadly you define a smart pointer.  I’ve had very smart, experienced engineers roll their eyes and say, “One smart pointer type is too many!” when I tell them that, but in actual practice they’ve worked so well for us that I couldn’t imagine working again on an engine that didn’t use smart pointers.

There are some things that must be borne in mind when using smart pointers:

• Smart pointers don’t remove any of the decisions about object ownership that have to be made in designing an engine.  You still need to decide what objects are shared, if any, and you need to define a strict ordering of which objects own which other objects.  All that smart pointers do is help document that ordering (you know that the object with a scoped_ptr owns the one without, even if there are pointers going both ways) and help enforce that ownership across the space of possible execution paths.
• Smart pointers are not the same as raw pointers.  They may use pointer syntax, but it’s impossible to use them efficiently without understanding what they are and understanding when they’re going to free memory or call AddRef/Release functions.  Smart pointers passed as function parameters should almost always be passed by const reference.

Our smart pointer menagerie includes

• a pointer types for internally-refcounted objects
• a pointer type for externally-refcounted objects (rarely used)
• scoped pointers (a std::auto_ptr replacement that’s sane)
• scoped arrays
• resource pointers that start NULL and are filled in as resources load asynchronously
• intrusive weak pointers
• and a few other specific pointers for managing internally-refcounted objects in third-party APIs

Programmers are encouraged to establish strong references at load time and use them at runtime.  Because we don’t generally create or pass around smart pointers during a frame, the runtime cost of smart pointers in Despair is minimal.

I credit our use of smart pointers, our rules for ownership, and our use of the RAII idiom for giving us an engine more stable than any I’ve ever worked on before, even as the number of engineers writing code in it has grown from two to thirty.  A new memory leak shows up about every month or two, and is almost always the result of someone’s not using smart pointers and then forgetting to call delete in all code paths.  In our engine, if you find yourself having to type out the keyword “delete”, then you’re probably doing something wrong.

Another, less frequent, error that sometimes occurs is for someone to introduce a cyclic reference that causes approximately every object in the game to leak.  In Despair, this happens about once every six months.  This is not a consequence of our use of smart pointers.  Most of the other game engines that I’ve worked on had manually-refcounted objects, and had similar issues on occasion.  Despair’s smart-pointer classes do help when we’re trying to debug this situation, though.  A debug macro enables capturing the callstack on each AddRef and associating each callstack with a particular smart pointer.  We can thus report the callstack of all leaked AddRef calls for each refcounted object.  Like I said, we don’t need this capability very often, but it’s nice to have.

The name started as a joke.  Adrian and I were out to lunch with our lead one day.  Somebody made a crack about naming the engine “Despair.”  One thing led to another, and by the end of the meal we’d plotted out a whole suite of despair-themed content creation tools, most of which never got made.  It was 2004.  We were starting from scratch on core technology for big-budget AAA games running on consoles that didn’t even exist yet.  We figured that “Despair” would work one way or the other, either as an imperative to our competitors if we succeeded or as a sadly accurate description of our own feelings if we failed.

There are two broad opposing ideologies when it comes to engine design for games.

Call the first the Low-Level Faction.  This school of design traces its roots to embedded devices and to early consoles like the Dreamcast and the first PlayStation.  This school is characterized by a preference for coding close to the metal, tends to produce games rather than game engines, and eschews abstraction in favor of raw performance.  Many of the games produced by this school of thought are raw C rather than C++, or if they are C++, they tend to minimize use of C++ features like polymorphism, virtual functions and templates.  When Insomniac’s Mike Action says at GDC that “Software is not a platform!  Hardware is a platform!” he’s voicing this faction’s frustrations.

Call the other group the Abstraction Faction.  This school of game engine development grew out of PC game development, where audio, graphics and I/O devices are hidden behind a driver layer.  Programming straight to the hardware simply isn’t an option.  The programmers of this school learned their trade on a platform with virtual memory and highly variable capabilities, so they learned to use levels of indirection to abstract away platform differences.  When the Epic guys say they’d sacrifice 10% of their runtime performance for a 50% improvement in content creation time, they’re justifying their higher level of abstraction.

Both ideologies of engine development have produced great games.  Halo, for example, is a straight-C game of the Low-Level Faction.  Gears of War is an example of a game with a lot of abstraction that uses lots of script and the advanced features of C++.

The lesson that I’ve learned in a decade of software development is that the specific decisions you make are less important than that decisions are made at all.  Software requires structure.  Every decision has tradeoffs, but once made, even seemingly-arbitrary decisions have to be enforced, or entropy will consume your codebase with a vengeance.  Every-man-for-himself design will get you a million lines of code, any one of which may reference any other, and any one of which may have undefined side effects.

The Despair Engine is an unapologetic representative of the Abstraction Faction of game engine design.  This is not because I think that’s the only way to create games.  It’s because that’s the way that I know how to create games.  We use the advanced features of C++ very aggressively.  If our history of uncovering compiler errors is any indication, the Despair Engine is unique among game engines in the degree to which it pushes the infrequently-explored edges of the language.  We use STL containers and algorithms.  We use the Boost library of C++ template extensions.  We use template metaprogramming to create our own embedded domain-specific reflection language.  And we use abstraction and polymorphism throughout our code.

Over the coming weeks, I’d like to discuss some of the Despair’s more unusual idioms and axioms.  For each, I’ll describe the problem that our approach was intended to solve, and I’ll talk about the continuing challenges of the solution that we implemented.

Like a career in game development, GDC is a mix of small rewards and great frustrations–and yet the two somehow balance each other out.  Back in 1998, the attendees were a small brash crowd, excited that gaming had finally arrived and enthusiastic about the future.  I think that was the first time somebody announced the oft-repeated canard about how we’re bigger than the movie industry.  My friends and I all made the crawl from room to room on Suite Night, availing ourselves of the free drinks and the catered food from the warming trays.  There was a party on the Queen Mary.  I got a thick pile of t-shirts and an Intel graphics card, and thought, 3dfx better watch out now that Intel’s getting into the market.

Now GDC has become like SIGGRAPH.  The crowds are enormous–nearly 15,000 in attendance this year, I’m told.  But it’s not the size of the crowds that makes it less intimate.  It’s the fact that so many of the people there don’t really have much to say to each other:  there are indie game developers, console game developers, serious game developers, mobile phone game developers, people selling middleware and hardware and outsourcing to all of the above, recruiters, wannabes, publishers trying to sign developers, developers looking for a publisher, HR folks looking to hire artists and programmers and musicians, and press trying to cover the whole spectacle.  The death of E3 meant that this year there were more sessions than ever that could be summed up as, ”look at my game and how awesome it is.”  I tried to avoid those.  I spent my three days at GDC looking for quiet places to talk to the people who do the same thing I do.  I didn’t go to any of the parties.

GDC hasn’t managed to scale to handle its growth.  Internet connectivity was poor throughout the convention center–even worse than last year.  The food is as identically terrible as ever.  And this year, for the first time, proceedings didn’t become available at all until after the conference, and those in the form of a slim handful of PowerPoint decks.  The clever people have taken to bringing cameras to the talks and snapping pictures of each slide that goes up, which is distracting but understandable.  The state of the GDC proceedings is an embarrassment to the industry, but it’s one that I don’t expect to change unless CMP stops handing out Gigapasses to speakers who show up without their slides.

My main interest this year was learning how other developers are dealing with problems of scale.  The team I’m working on at Day 1 has gone from three people to eighty people in the last three years.  Our engine just passed a million lines of code.  We’ve had to turn our resource installers into multi-volume installers because they crossed the two-gig threshold that’s the largest file size Inno Setup supports.  I think that we’re dealing with our scale better than GDC is, but it’s always good to gather additional perspectives.  I’ll talk more about the things that we’re doing and what we can learn from the presenters at GDC as I go.

Running Halo 3 Without a Hard Drive

Part of the problem of scale in next-generation console development is that console memory is getting larger and content sizes are growing to match, but DVDs and memory aren’t getting a whole lot faster.  That means that we developers have to work harder and harder to keep players fed with fresh content.

Mat Noguchi’s talk on streaming content in Halo 3 was interesting mostly because it showed how Bungie approached the problem of streaming from the opposite end from what we’ve done at Day 1, but we seem to be ending up in similar places.  They took existing level technology and shoehorned streaming into BSP regions.  We built levels up out of smaller streamable primitives.  Bungie has always emphasized separation of file I/O (deciding what gets loaded) and resource management (managing data once it’s been loaded), and they do lots of off-line computation to determine what resources actually need to be loaded for each level.  We allow any asynchronous resource load to trigger other asynchronous resources loads, and don’t return requested resources to the main game thread until the entire requested resource tree is complete.  In short, their system was designed for the most efficient player experience and ours was designed to be as flexible as possible while we’re iterating on content creation.

In the end, though, we’re doing similar optimizations to get to where we need to be to ship.  We’re sorting resources in the order they’ll be read to minimize DVD seeks, we’re grouping resources that have similar level ownership, and we’re compressing groups that will all be read together.  Like Bungie, we’re having problems with audio resources interacting badly with everything else that needs to be streamed–they solved that problem by just not playing any streamed AI voices if the player doesn’t have a hard drive to cache to.

Bungie seemed to give half the talks at GDC this year, and they were even kind enough to make their slides available–though not on the GDC Proceedings site.

Automating Regression Discovery: Finding the Wrenches in the GEARS OF WAR

I’m the senior engineer at Day 1 most interested in build infrastructure and automated testing, so I always like to hear what other teams are doing in that department.  Epic’s test and metrics infrastructure isn’t quite up to Microsoft standards, but they do as well as any independent developer I’ve heard of.  They use CruiseControl as a continuous integration framework, building all configurations of the Unreal Engine across ten dedicated build machines every time a developer checks in.  They also do continuous integration for content, building fresh cooked resources every time an artist or designer checks in.

The resource build process collects performance and automated test data for each level.  Any crashes get logged to a database.  Mesh and texture size information gets dumped to spreadsheets covering the resource usage of each level.  The system collects performance data from fly-through camera paths, bot matches, human playthroughs, and render sample points scattered around each level.  Their performance gathering for UT3 definitely benefited from the fact that it was a bot-based multiplayer game, but their performance gathering across the board sounds second-to-none.

One of the gems I took away from this talk was the oh-yeah-that’s-obvious point that collecting performance data about average framerate really isn’t very useful.  On the consoles, we’re always V-synced at 60, 30, 20, 15, 10 or 5 Hz anyway–usually 30 or 20.  A more useful statistic to collect is the percent of frames, during a playthrough of any level, that are below 30 Hz.  It’s also useful to collect the percent below 5 or 10 Hz to catch dramatic stalls.  If you have more than a couple of percent below 30 Hz, and if you have any frames below 10 Hz, then something needs to be done to improve performance.

Sharing Code Roundtable

I attended the sharing code roundtable partly because we’re sharing the same engine across two projects at Day 1, and partly because I interviewed with the moderator, Brian Sharp, at Ion Storm, and I think he’s a smart guy and I was interested in what he had to say.  In the end, though, I found the session frustrating because no two people in the room even had problems similar enough to be able to have a “this worked for me, what worked for you?” conversation.  We mostly ended up talking past each other.  A few of the forty-or-so attendees:

• Me:  Working on one big-budget console game, sharing an engine with another game being developed in another Perforce branch, and integrating back and forth every week.
• Brian Sharp:  Organizing an effort to share code across all Midway studios… his challenge was trying to improve communication between different studios.
• A guy from EA Canada:  EAC is like a middleware provider to the other EA studios, and as a dedicated middleware provider, he wanted to know how, after giving another studio a drop of some technology, EAC could merge back fixes and new features with a minimum of pain.
• A guy from academia:  He was working on a distributed development project with undergraduate students in different countries, and wanted to know what the rest of us thought of distributed source control systems like Mercurial and Git.  I’m pretty sure most of us had never heard of them.

I think Brian did a good job moderating, but it just seemed like the wrong group of forty people to have a conversation.  Or maybe it was just the wrong conversation for forty people to have.

I did have one startled and happy moment when the conversation tangentially touched on automated testing.  I had a similar moment in the Gears of War infrastructure talk when Martin Sweitzer talked about Epic’s use of PC-Lint.  They said:  We have all this existing legacy code and it doesn’t have unit tests and Lint generates a million warnings for it.  How can we test the new code we write without having to touch the ugly legacy stuff that we know works and that we’re afraid to change?  Day 1’s in-house Despair Engine is three years old, and it was written with automated unit tests running for every library and a nightly Linting of the entire engine from the very beginning.  We pass all our tests.  We don’t generate any Lint warnings–at least not the ones we care about.  Modern console development is really, really hard.  It’s nice, sometimes, to hear about the problems you don’t have.

Ray Kurzweil

I missed the Microsoft keynote on Wednesday for a meeting, but I was happy to be able to make Ray Kurzweil’s keynote on Thursday.  Kurzweil’s an inventor and futurist, and the author of The Age of Spiritual Machines and The Singularity is Near:  When Humans Transcend Biology.  He’s what we used to call an extropian back while I was in college.  His talk covered much of the same material that he covered in The Age of Spiritual Machines (and presumably The Singularity is Near, though I haven’t read that one):  innovation and intelligence are increasing at an exponential pace, and in the near-future we’re going to cross a super-evolutionary threshold so sudden and so steep that we have no ability to predict the future of humanity beyond it.

All of this has a shrill ring of millenarianism to it that is easily dismissed as the “rapture of the nerds”, but I’ve always been willing to forgive the extropians their evident wackiness because they’re probably right:  human science has advanced more in the last hundred years than in the thousand before that, and while Kurzweil sounds a little optimistic to me–I don’t expect to ascend to nerdvanna in the next twenty years–it’s hard for me to believe that in a hundred years people will still be dying of old age because we still haven’t figured out how the genes for aging work.

So here’s the techno-optimism:  In the last 40 years, Kurzweil says, computing power has increased a billion times and computer size has decreased 100,000 times.  The same will happen again over the next 25 years to produce injectable computers the size of blood cells that can enter your brain and feed your senses a full 3D virtual reality.

Or those injectable computers can rewire your body chemistry.  The software running our bodies is obsolete, evolved for a hunter-gatherer lifestyle tens of thousands of years ago.  Your body stores fat because our genes evolved in a time of scarcity.  But now we live in a time of abundance.  RNA interference is on the verge of being able to turn off genes for storing fat.  Current diet drugs work by suppressing appetite, which, Kurzweil notes, “is like birth control that works by inhibiting interest in sex.”

Our ability to predict the future also evolved in a different time, when exponential progress was so slow that it was indistinguishable from linear progress because the exponential curve was so flat.  Now people make stupid predictions that Social Security will go bankrupt and that greenhouse gasses will melt the ice caps even though technologies are coming that will soon render all those assumptions obsolete.  Within five years, Kurzweil says, nano-engineered solar panels will be more cost-effective than fossil fuels.  Within twenty years, we’ll have largely replaced fossil fuels as a means of power production.

And so on.  Slides here.  If you’re skeptical about his claim that by 2010 desktop computers will disappear and wearable ubiquitous computing devices will project high-definition augmented-reality images directly into our retinas, so am I.  But I want to believe.  Kurzweil has a way of making you feel like we’re living in the future right now.

Life on the Bungie Farm

For Halo 3, Bungie had a build farm of 180 computers with 300 processors.  These ran in-house client/server software written in C# that communicated through atomic transactions in a SQL database to manage

• code builds and automated tests
• lightmap rendering
• content builds
• other tasks (cubemap rendering, shader compilation, builds of bungie.net, and maintenance)

It’s a neat set-up.  I wish that we had a half-million dollars to spend on build hardware, but honestly I’m not sure what we’d do with that many computers.  The majority of Bungie’s workload was lightmap rendering.  I’ve read elsewhere that they were cooking spherical harmonic coefficients for reflection functions into their lightmaps, similar to Half Life 2’s orthogonal lightmap basis, which I suppose helps explain why their lightmaps were so much more expensive than, say lightmap generation in Quake I.

But Day 1’s games are all about environmental destruction, which means that we really can’t get away with static lighting.  All lighting in both our current games is dynamic.  I wouldn’t mind having, say, twenty build machines to throw at code and resource builds, but I think that would take care of our needs just fine.

That said, there are definitely some features of Bungie’s set-up that make me envious.  A C# client/server model communicating through SQL would be a step up from our PHP build scripts communicating through text files.  And Bungie uploads all symbols from their automated builds to a symbol server, as we do, but they also copy all source to a fixed location on the network and link the executable to the location of the source with which it was built using the undocumented-and-not-mentioned-anywhere-else /SOURCEMAP linker option.  We’ll probably make those changes to our build system in the next few months.

Internal & Outsourcer Management of Tools & Pipelines

I’m not really interested in the process of outsourcing, which is more of an art and production function, but I’m very interested in how tools get built and distributed to the people who use them.  Day 1 has about 140 developers working on two different games now.  We build Inno Setup installers for all our tools every night as part of our automated build, they get tested by our QA department every morning, and if there aren’t any showstopper bugs then we update the latest release version in a SQL database.  A little system-tray app that we run turns yellow on everyone’s desktop and prompts them to right-click and select “Update” to install latest.

Volition has even larger teams than we do:  100 internal developers and 20 outsourcers on Saints Row; 101 internal developers and 44 outsourcers on Saints Row 2; and 80 internal developers and 52 outsourcers on Red Faction.  Like us, they’ve invested in some automatic distribution infrastructure.  They have an app called vInstaller that runs MSBuild scripts to do tool installs.  I think that I prefer having a proper installer, since it means that our tools can be uninstalled through the Add/Remove Programs option in the Windows Control Panel.  Volition devs have to uninstall through vInstaller, which apparently makes Windows Vista unhappy.  Their IT guys apparently had to disable User Account Control on all their Vista machines.

They did say that on one occasion they’d had the misfortune of auto-deploying a virus to all developers.  We’ve been lucky enough to avoid that one so far.

They integrated an art outsourcing company in Shanghai into their existing tool distribution framework, which created new challenges.  The outsourcers didn’t use Perforce.  Users weren’t admins on their own machines.  And they tended to install tools into many different locations, which broke non-robust tools.

We generate a slightly trimmed-down installer for outsourcers of just the tools that they need to preview assets in our engine.  Here again, I think that we benefit from having a traditional Windows installer instead of a standalone installation app.  I’m don’t think a user even needs to be an admin to run the installer and I’m pretty confident that our tools will work whatever path you install them to.  We usually install to C:\Program Files, and we don’t expect to access any other tools or data through relative paths.

Slides for Volition’s talk here.

Unreal Tournament 3 Postmortem

I went to the UT3 postmortem with some hesitation, because Epic has a bit of a rep for turning every talk into a sales pitch for the Unreal Engine.  In the event, I was very pleasantly surprised by the candor with which Jeff Morris and Mike Capps talked about UT3’s successes and shortcomings.

The talk followed the now-traditional What Went Right/What Went Wrong format of every post-mortem in Game Developer magazine, but like most developers, Jeff and Mike seemed to find more passion in talking about what went wrong than what went right:

• They wasted a lot of effort on a character customization system that had no effect on actual gameplay.  With characters moving at 30 MPH in front of often-animated backgrounds, you couldn’t tell what they looked like.
• The single-player campaign gameplay came together too late to be polished or balanced.  “Head and shoulders above any UT game we had made in the past, but it wasn’t Bioshock,” Mike said.  Of course, Bioshock is an Unreal Engine game.
• The game didn’t have enough polish time overall, and releases were scheduled badly.  They diluted their marketing efforts by releasing at different times on different platforms and in different regions.  And they walked into the most competitive season ever for first-person shooters, going up against Bioshock, Call of Duty 4 and Halo 3.  “We’re very big in Russia,” Mike observed sardonically.

The Best Talk I Missed

I like to listen to GDC Audio proceedings while I take long road trips, and Ubisoft Creative Director Clint Hocking showed up on my radar when I heard his excellent talk on Narrative in the Splinter Cell Trilogy and his robust showing at the Game Design Challenge at GDC 2005.  I’d already noticed as I played the Splinter Cell games that all the ones that came out of Ubisoft Montreal were fantastic, and the ones that came out of Ubisoft Shanghai were… not so much.  After hearing Clint’s GDC 2005 talks, I suspected that he might be one of the big reasons for that.

Unfortunately, I had conflicts that kept me from being able to make it to Clint’s talk at GDC 2008, I-fi:  Immersive Fidelity in Game Design, but he’s been generous enough to make his slides and an actual paper available on his blog.  He discusses player immersion in terms both critical and creative.  I’m a proponent of more scientific game design:  I believe that designers should form credible hypotheses regarding what is and isn’t fun and create experiments that test those hypotheses.  I think that application of the scientific method is the only way to consistently create more involving games.  I’m not sure that I-fi is even quite a hypothesis yet, but it’s close, and it suggests avenues of advance that can only make the games we create more compelling.

Looking Ahead

All in all, there was some really good material at GDC this year.  I felt like the overall tone of presentation was somewhat muted, though.  Game developers are a typically cocky bunch, but scaling up to handle hundred-man teams and multi-core development has been a challenge, and I think it’s left a lot of us feeling less like we know The Way to solve these issues.  Everybody’s trying, and the companies that are still standing are clearly doing better than the many companies that aren’t, but even the survivors are a little shell-shocked.

And we have two years until the next console generation comes along and shakes everything up again, multiplying memory and core counts by a factor of eight and team sizes by a factor of two.  “If you’re planning a project today that’s going to take more than six months,” Ray Kurzweil said, “then the pace of change is so great that you really need to take exponential growth into account.”  Our industry has been living with that reality for a while.

CityDesk hasn’t had a lot of work over the last couple of years, though, and I never have managed to figure out how to make it generate a proper RSS feed.  So I’m switching the site over to WordPress, about which I’ve heard good things.  All old content is still available at http://www.gamearchitect.net/citydesk.html.  Pardon the dust and plaster.