Musings on Game Engine Design

Good Middleware

with 12 comments

Love is patient, love is kind.  It does not envy, it does not boast, it is not proud.  It is not rude, it is not self-seeking, it is not easily angered, it keeps no record of wrongs.  Love does not delight in evil but rejoices with the truth.  It always protects, always trusts, always hopes, always perseveres.

1 Corinthians 13

Love is easy.  Middleware is hard.

We used middleware on Fracture for physics, tree rendering, audio, animation, facial animation, network transport, and various other systems.  Now that the game is finished, we’re taking stock of the lessons that we learned and deciding what packages we want to keep and what we want to replace as we move forward.

Middleware offers two main benefits, each of which is balanced by an associated cost:

1.  Middleware provides you with more code than you could write yourself for a fraction of what it would cost you to try.  No matter how clever you are, that’s how the economics of the situation work.  The costs of developing a piece of software are largely fixed.  But once written, any piece of software can be sold over and over again.  Because they can spread their costs over a large number of customers, middleware vendors can afford to keep larger, more experienced teams working on a given piece of functionality than independent game developers can.  During Fracture, we watched a couple of pieces of immature middleware spring up after we’d written similar tools ourselves.  We watched those pieces of middleware grow and surpass our tools, because we couldn’t afford to devote the resources to our tools that the middleware vendors devoted to theirs.

The corresponding cost is that none of the functionality you get is exactly what you would have written yourself, and much of it will be entirely useless to you.  Middleware is written to support the most common cases.  If you have specific needs that differ from the norm, you may be better off implementing a solution yourself.

2.  Middleware offers structure.  Middleware draws a line between the things that you have to worry about and the things you don’t.  As long as it’s reasonably well documented and stable, you don’t need to waste mental bandwidth worrying about the things that go on underneath your middleware’s public API.  As games grow ever-larger and more complex it’s become incredibly valuable to be able to draw a line and say, The stuff on the other side of that line isn’t my responsibility, and I don’t have to worry about it.

The associated cost is that you can’t change what’s on the other side of that line.  If you’re going to use middleware, you have to be willing to accept a certain amount of inflexibility in dealing with the problems that the middleware solves.  You have to be willing to shape your own technology to suit the third-party libraries that you’re buying.  Trying to do otherwise is a recipe for misery.

Given those benefits and costs, it makes sense to use middleware wherever you can–as long as you don’t try to license technology in the areas where you want your game to be unique.  Every game has certain unique selling propositions:  things that make it distinct from every other game on the market.  Likewise, every game has certain characteristics that it shares with many others.  When it comes to the latter, you should buy off-the-shelf technology, accept that technology’s inflexibility, and modify your game design to suit it.  When it comes to the former, you should write the code in-house.  As Joel Spolsky says, don’t outsource your core competency.

So now that you’ve figured out which game functions you should be implementing through middleware, how do you decide which of the scads of available middleware packages is best for you?  There are a number of issues to keep in mind.

Good middleware lets you hook your own memory allocator.  If your approach to memory is to partition the entire free space up front and minimize runtime allocations, then you’ll want the ability to reserve a block for your middleware and allocate out of that.  Even if you allow dynamic allocations at any time, you’ll want to track how much memory is used by each system and instrument allocations to detect memory leaks.  Any piece of middleware that goes behind your back and allocates memory directly just isn’t worth buying.

Good middleware lets you hook your own I/O functions.  Most games store resources in package files like the old Doom WAD files.  Middleware that doesn’t let you hook file I/O doesn’t let you put its resources in packfiles.  Many modern games stream resources off DVD as the player progresses through a level.  If you can’t control middleware I/O operations, then you can’t sort file accesses to minimize DVD seeks.  You’ll waste read bandwidth and your game will be subject to unpredictable hitches.  Again, any piece of middleware that does file I/O directly just isn’t worth buying.

Good middleware has extensible functionality.  No middleware package will do everything you want out of the box.  But you shouldn’t have to modify any piece of middleware to make it do what you need.  Good middleware offers abstract interfaces that you can implement and callbacks that you can hook where you need to do something unique to your game.  An animation package may let you implement custom animation controllers.  A physics package may let you write your own collision primitives.  Your objects should be first-class citizens of your middleware’s world.

Good middleware avoids symbol conflicts.  Beware of middleware that uses the Standard Template Library carelessly.  It’ll work fine until you try to switch to STLPort or upgrade to a new development environment, and then you’ll suddenly find that your engine has multiple conflicting definitions of std::string and other common classes.  To avoid symbol clashes, every class in a middleware library should start with a custom prefix or be scoped inside a library namespace.  And if a piece of middleware is going to use the STL, it should do so carefully, making sure that every STL class instantiated uses a custom allocator.  That allows you to hook your own memory allocator and avoids symbol conflicts.

Good middleware is explicit about its thread safety.  We live in an ever-more-multithreaded world, but one where most game engines are still bound by main thread operations most of the time.  For best performance, you want to offload any operations you can onto other threads.  To do that with a piece of middleware, you need to know which operations can be performed concurrently and which have to happen sequentially.  Ideally a piece of middleware will let you create resources asynchronously so you can construct objects in a loader thread before handing them off to the game.

Good middleware fits into your data pipeline.  Most companies export data in an inefficient platform-independent format, then cook it into optimized platform-specific formats and build package files out of those as part of a resource build.  Any piece of middleware should allow content creators to export their assets in a platform-independent format.  Ideally, that format should still be directly loadable.  The build process should be able to generate platform-specific versions of those assets using command-line tools.

Good middleware is stable.  One of the main benefits of middleware is that it frees your mind to focus on more critical things.  In this respect, middleware is like your compiler:  It frees you from having to think about low-level implementation details–but only as long as you trust the compiler!  A buggy piece of middleware is a double curse, because instead of freeing your attention from a piece of functionality, it forces you to focus your attention there, on code that was written by a stranger and that no one in your company understands.  Worse still, if you’re forced to make bug fixes yourself, then you need to carry them forward with each new code drop you get of your middleware libraries.  You shouldn’t need to concern yourself with the implementation details of your middleware.  Middleware is only a benefit to the extent that its API remains inviolate.

Good middleware gives you source.  Despite the previous point, having access to the source for any middleware package is a must.  Sometimes you’ll suspect that there’s a bug in the middleware.  Sometimes you need to see how a particular input led to a particular result before you can understand why the input was wrong.  Sometimes you’ll have to fix a bug no matter how good the middleware is.  And frequently you’ll need to recompile to handle a new platform SDK release or to link with some esoteric build configuration.

There are other questions to ask about any piece of middleware:  How much memory does it use?  How much CPU time does it require?  What’s the upgrade path for your current code and data?  How does it interact with your other middleware?  How good is the vendor’s support?  How much does it cost?

But those questions have vaguer answers.  Acceptable performance or cost will vary depending on the nature of your product.  A cell phone game probably can’t afford to license the Unreal Engine–and probably couldn’t fit it in available memory if it did!  Data upgrade paths are less of a concern if you’re writing a new engine from scratch than if you’re making the umpteenth version of an annual football game.

The rules that don’t vary are:  You should buy middleware wherever you can do so without outsourcing your core competency.  And to be worth buying, any piece of middleware should behave itself with respect to resource management and concurrency.

Written by Kyle

September 19th, 2008 at 10:17 pm

12 Responses to 'Good Middleware'

Subscribe to comments with RSS or TrackBack to 'Good Middleware'.

  1. Hey there, nice post, but I think most folks take a different meaning for middleware:

    Sounds like you’re talking about 3rd party or off-the-shelf libraries or frameworks.


    20 Sep 08 at 8:11 pm

  2. I guess this might be a better fit:

    I’ve always seen middleware as libraries/frameworks/engines that are in the “middle” between the game and insert_functionality_here with the benefit of (sometimes) being portable.

    Yes that is a very broad definition that I use…


    22 Sep 08 at 4:22 pm

  3. Enjoyed the article; it hits on the most important aspects when considering a 3rd party solution.

    You said in the beginning that you used various middleware for the development of Fracture. Can you elaborate and tell us which packages you used and how those met (or didn’t meet) the requirements you gave?


    23 Sep 08 at 7:28 am

  4. Nice article, it reminded me of a presentation I listened to by Casey Muratori. Were he draws knowledge from lessons learnt creating tools entitled Designing and Evaluating Reusable Components. The audio and slides of which are available at the following url


    24 Sep 08 at 12:41 pm

  5. I’m happy to list the middleware packages that we used, but I’d rather not review them here. I hope that my co-workers who are most familiar with specific systems will review them for Game Developer at greater length.

    We used:
    Bink for video compression
    FaceFX for facial animation
    FMOD for audio
    Havok for physics and animation
    RakNet for our low-level network layer
    SpeedTree for foliage

    And Boost, Zlib, Flex++ and Bison++, Sony’s EDGE libraries, and dlmalloc. We used a couple of other third-party packages in our tools that didn’t ship in the game itself.


    24 Sep 08 at 4:55 pm

  6. Casey Muratori is a bright guy. We used his Granny animation library in MechAssault and MechAssault 2, and it worked well. We switched away from it for Fracture because we were adopting Havok anyway and thought it would be more convenient to get our animation and physics from the same source.


    24 Sep 08 at 5:00 pm

  7. Is Gamebryo Good Middleware?…

    Kyle Wilson writes nicely thought out articles on game development. He just wrote a piece, Defining Good Middleware.

    I’ve been asked how Gamebryo stacks up to his definition. So, I’ll discuss that, plus throw in some additional thoughts. …

  8. Great article! I’m curious in regards to your thoughts on multi-threading and concurrency. Suppose you have your own job/task scheduling system to handle multiple cores etc. How should middleware handle this? I don’t think most have tackled this. Would it be beneficial that the middleware hands back “tasks/jobs” that I can run? i.e. “Good Middleware lets you hook into the job/task/concurrency scheduler”?


    29 Dec 08 at 7:42 am

  9. I recently had a conversation with a friend about this entry (good entry btw), and a point he made was in the not too distant future, good middleware is going to allow you to hook their job scheduling just like you can hook memory allocation. I developed these thoughts further over at my blog:


    3 Jan 09 at 12:31 pm

  10. IMHO, one more important issue is missed:
    Good middleware has “realtime” support (or at least one which answers in a few days). A fast answer from author of a code can save a lot of time spent in debugger.


    2 Feb 09 at 7:49 am

  11. Hey Kyle,

    We’d love to hear your particular feedback about RakNet. I believe the version you were using at the time didn’t support redefining the memory allocator, so that was probably directed to us 🙂

    Kevin Jenkins

    5 May 09 at 7:43 am

  12. Hi, Kevin. My comments weren’t directed at any middleware package in particular. I’m not sure how much our branch of RakNet even fits the definition of middleware anymore, since we’ve been making our own modifications to it since the day we forked and we’ve never looked back. I’m not aware of any big architectural complaints, but our network guys could probably give you more specific feedback than I can.


    7 May 09 at 7:27 pm

Leave a Reply