me == DeanoC

Deano Calver's place on the InterWeb

e-mail@DeanoC

Games and the engines they run on are extremely complex, effectively creating an entire world on which the game runs in. Everything from the visuals, physics, sound and of course the interaction with the player or players need to be developed.

Even the definition of what a game is, is complex and argued about, so making an engine that is all things to all people is an almost impossible goal.

The current solution has been for the engine part to be made by a third party like Unreal Engine or Unity. The idea being that by using their engine and tools you can concentrate on making the game portion.

The commercial game engines available employ thousands of people and are used by millions of potential game developers. Whilst at first it appears to be something a solo or small team couldn’t compete with, it also brings with it a major issue.

  • Complexity

In trying to be a universal engine for any game, they often have to be general in a way smaller more bespoke engines don’t have to be. And therein lies an achilles heel, indeed so complex are generic engines trying to be both AAA quality but simple for beginners, it can be hard to work with/on if it doesn’t align completely with how it does things.

Obviously as commercial interests they don’t point out any downsides, only the upsides and of course many people do use them to produce great games.

They tend to designed around the idea that visual tools are better for most developers and they might be right, but you might be someone who is happier writing some code than dragging boxes.

They come with prebuilt ideas how games worlds should be built (entity systems etc.), how a renderer, etc. should be designed and whilst often designed by knowledgable people they might not fit your view or your particular game. Whilst they are generic they tend to have focus on a few genres, which tend to be for 1st and 3rd person 3D action games, they can and are used for other types but the further you go from these the more you tend to not use (for example if you have completely procedural levels, the level editors might have to be used in ‘odd’ ways).

So the obvious solution is not to use one of the commercial engines and write a bespoke one!

But as we said, its difficult road to travel, if you aim it to make a better totally generic clone of one of the big engines, you will probably fail. They are too big, too well resourced and ultimately you would end up with the same issues they have.

But if you keep to just what you need, particularly if your game doesn’t fit into the mainstream of what the generic engines are designed for, you can make something decent. There are also many freely useable libraries now available, in fact many of the advantages of generic engines can be achieved just be sticking together various modules you can download. This allows a middle approach, bespoke develop the things you want and use high quality libraries from the internet for parts you either don’t need or want to customise.

How can a solo or small team make an engine that can produce something that looks and feels like it was made with Unreal or Unity given their budgets and people power?

The answer lies in not being general, make the engine fit the game. Only write the bits you need and not try to create something completely generic. Try and make it reusable for a similar game in the future but don’t worry too much . As long as it works enough for the game you’re making today that’s good enough.

This simple constraint can save a huge amounts of time and money, whereas a generic engine might need an entire compiler team to turn some code into a faster equivalent, you on the other hand might not even need faster code and if you do, its probable some guidelines and spot optimising will do the job.

The magnitude of their problem is massively larger than yours, because you don’t have to expect anyone outside a few people (your team) to use it, whereas a general engine team has to expect anybody to use it and so handle all those cases.

Another consideration is porting to different platforms, a console generation ago this was a difficult proposition with radically different architectures requiring specialist knowledge so it made sense for smaller teams to use a generic engine to enable them to ‘easily’ port to another platform but today most desktop platforms are based on PC technology (PC, Mac, Linux, XBox 1 and PS4) with varying performance. The porting effort for each is mostly performance considerations which is true on a generic engine or not. If you’re targeting mobile, it’s a bit more complex but Apple, Android and Switch are largely the same at various price points. There is again performance deltas but the issues remains again with the generic engine.

In some ways its easier to port with a bespoke engine, you don’t have to worry about it breaking anybody else's game just as longs it works for yours and if your see a gain for a platform you can do it, you have not only the code but the understanding of the how’s and why’s.

Tooling is where the generic engines have a hard to beat feature. Realistically at the moment, you’re probably going to have to accept your tools will be simpler and more specialised than some of the tools provided or made easy to develop on the generic engines. You will likely require fewer tools, as generic engines tend to need game specific tools to avoid errors allowed by the generic system. A generic engine requires a lot of filtering to only allow things valid to your game out of the almost limitless possibilities a generic engine provides. You’re probably going to rely more on text definitions than elaborate visual tools. Whether this is a downside likely depends on you team and their skills. Its not even an art versus programmer issue, I’ve worked with an art director who would happily write things as complex as an ambient occlusion script for an art package and games designers who can write entire games. But if your team is heavily opposed to text based definitions and scripting, its likely any bespoke engine will have to devote a lot of time to tooling it which is an area the generic engines have all the aces.

The seemingly obvious downside of writing a bespoke engine is the extra time it will take. Even if it’s possible to write the code required for a bespoke engine, it still has to be written and that takes time.

And there is no argument against it, with a generic engine you have a vast array, possibly too many features you can start using from day 1. The only delays is learning to use a feature and morph it into the form that fits your game.

But that time to learn and adapt the generic engine version of a feature can range with tiny to longer than you’d expect. Especially when it is still small, using something can be literally seconds from first using it to doing something useful. The issues tend to arise when you get to more complex usage. A classic case is animation systems, no matter what the project or engine (bespoke or general) animation system never quite work as wanted, either its not useable by the animators or doesn’t have the events requires by game play etc. For the bespoke system this usually means tweaks and additions throughout the game, but for generics its often a harder problem, even with source, tweaks can be difficult as learning the system inside out can sometimes takes longer than someone experienced writing it from scratch.

Another time sink for generic engines over bespoke is the issues related to late development, something only spoken about in in hushed dark places but generic engines are generally hard to ship medium to large games on. Its a practical result of the conflicting desires of easy to use for beginners with AAA quality, many of the things that AAA quality games do aren’t user friendly not because the developers were difficult but because the bigger the game the more processing that data requires. What works for a game with a few gigabytes of assets fails badly with a few terabytes!

Whereas it’s sold as a feature that someone can submit a new art asset into a level almost instantly, its a disaster if that asset stops a daily build that took all night to cook. A lot of these issues are organisational but often are hard to work around in generic engines. Bespoke will suffer from the same issues but its usually either designed in early on if reaching this scale or at least easier to retrofit.

The big question is does the downside of a generic engine generality latter on a project out way the cost of the initial start up time of a bespoke engine? There is no one answer, I believe there is a range where if you can afford the start up cost of a bespoke engine and tooling, it may actually be quicker time wise than fighting the generality of the general engine but clearly this is both game and team specific. Often commercial reality is that you need something up front quick (to sell to kickstarter or a publisher) whereas slipping a release date is relatively easy.

Time wise, reusing modules is a clear winner but also know it brings some of the issues related to using a generic engine, though usually at a finer grain.

A successful bespoke engine is probably tiny compared the the big generic engines, it probably only has the tools needed for your team. It will have a large portion of code from free third party source and will likely have lots of cases not covered.

But it will have enough for your game, it will do that well and probably more efficient and better than a generic engine. It should be usable for a sequel and/or modular enough to make another game in a similar vein a bit quicker to develop due to some reuse.

You will understand it better, like a custom suit it will fit you better than an off the shelf one. Its likely you will be able to make prototypes and modifications quicker as you know it inside and out.

Making a bespoke engine isn’t for everyone, its quite possible that for you, using one of the mainstream generic engines is the right way of getting your game released. In fact I’d argue that it is the first thing you should try even for us oldies who grew up on custom engines.

But if you don’t get on with it or you want to learn more about engines or you just can’t seem to map your game idea to the generic mainstream engine, a bespoke engine is both possible and efficient if you don’t get off track and remember you are making a engine for a game (or game genre), as soon as you try and solve all the problems for all the games, you will end up in the rabbit hole.

Where it is not vital, reuse freely available code, it doesn’t have to be perfect just as long as it does the job. The amount of tooling you require will be a huge factor and thats not something you can easily get at the moment from open sources.

In many ways a bespoke engine is easier for the very small and the very large games and teams. A team of a few people often don’t need the generic tools or hiring people who know how to do X in engine Y and can support the needs of the various people making the game as there are only a few in total. For large games and teams the upsides to the generic engines is usually not worth the problems. Some may take a generic engine source drop and then customise it like a bespoke one, never really bothered by rewriting major systems as its developed as a custom. The truly large publisher engines are massive and often suffer the same issue the publicly available generic engines do. There aren’t bespoke by my definition but are custom, they are trying to be all things to all people.

The mid sized teams and games are the hardest to use a bespoke, they need to tools and like the ability to hire people using engine X (especially it sizing up from a smaller team). The support of a bespoke engine is likely too expensive versus working around the issues they have with generic engine until a certain team size and tooling stage is hit. To do it probably requires a multi game commitment to a bespoke engine, slowly building up more and more tools and modules that make it cheaper and quicker to develop each game, it likely also means keeping the games produces to a genre or type. It might make sense to develop a bespoke engine at first purely for internal tools, whilst releasing on a generic engine for the first game in a series even.

One of the simplest techniques for maintaining large code bases is to enforce a strict hierarchy of libraries.

I first remember reading about it well over a decade ago in “Large Scale C++ Software Design” by John Lakos. Its grown in my own usage until now my own code is organised in a literal levelled folder system.

The point is that a library should only use functions lower down than itself, so there are clear path from higher level functionality toward code with less dependencies. It also avoids cycles where X depends on Y which depends on X, it can’t happen as Y can’t depend on X.

So my libs structure is divided into levels (except my level zero libraries which live in the root of the libs structure).

Then level1 contains

And so on, a library in level 1 can only access core and binify, a level 2 library has access binny, input and math as well as core and binify.

I use cmake’s parent scope feature to inject a levels set of libraries without the levelX part into the include paths further down the chain.

This results in my app directory (where the actual program source code live) being able to access any of the libraries via library/foo.h even though it might be physically located in level2/library/foo.h. The same is true for the libraries themselves, though obviously they can only access libraries of a lower level. This makes it easy to adjust a library’s level as needed (its simple a physical move and change of 2 lines in cmakelists.txt files, I’m tempted to automate the last step so it automatically picks up movements of libraries).

I have a strict policy of include files via a #include “library/include.h” syntax even when the library part isn’t strictly necessary, this make it more visible and to me means I can treat the include path as a form of name spacing. I don’t feel bad about having the same named file in separate folders, as it obvious which is being used by the library prefix.

So for example my low level render system is a level 3 and lives in level3/render. My Vulkan implementation of that system lives in level 4 at level4/vulkan. My mid level render system lives in level4/midrender as it derives off render, the fact that behind the scenes potentially uses level4/vulkan doesn’t matter. It depends only on level 3 functionality, so its a level 4 library. My namespace follows the include path fairly closely (some differences due to case choices). The namespace are Render, Vulkan and MidRender (I choose camel case to separate from the standard library all lower case name spaces). There are texture.h files in both render and vulkan but are always referred to as “render/texture.h” or “vulkan/texture.h” so easy to tell which was intended.

Its not much work to implement and really helps as things grow, its usually fairly easy to tell where new functionality should go and stops any accidental breaking of the architecture.

Push versus Pull Renderers

The biggest choice a renderer has to make is whether its a push or pull type, however in practise most choose push and so never really think about the choice.

In these dicussions we split the HW into two types, one where the world simulation (physics, inputs, game code, etc.) which we call the CPU and another which performs computation related to rendering, the GPU. In some HW topologies its possible we use a physical CPU core as a GPU slave, and as such in this text is considered part of the GPU.

A push renderer is almost certainly what you’ve seen in 99% of places. Its defined as a system where a world database and simulation is held on the CPU and the CPU selects what to render and push the data required to render what its selected to the GPU. Its know as a ‘Push’ renderer as the CPU pushs data to the GPU.

A pull renderer is a very different beast. Here the renderer looks directly in the world database and uses that to render things to the user. A full pull renderer would have the CPU doing nothing for the scene to be rendered. The GPU decides what and how to render without the CPU doing anything, hence the GPU ‘Pulls’ render data directly.

Pull renderers are rare due to many issues, from graphics api having CPU only bindings, to the CPU hardware being the only device which has the flexibility to render something. Render APIs are almost always push biased, you have a object that you call Change State and Draw calls directly. Pull require a higher level API than currently in favour, it has to know about your chosen bounding volumes, understand your material and geometry structures potentially. It it also require choices about what to render to be in the GPU space.

A first glance this sounds like a classic render thread architecture, where a thread receives commands from the CPU and turns them into renders. However this is still a push renderer, as the CPU still has to push some data to the render thread. A pull renderer might use a render thread but if so it goes a fetchs what to render rather than just follow a command list.

First I’ll describe a PS3 pull renderer, I’ve written in the pass (should be safe from NDAs) and then some ideas for a future PC pull renderer.

The key to a PS3 pull renderer, is slaving at least one SPU to the GPU. Effectively using a SPU as a fancy front end for the GPU. Apart from writing the SPU pull code, getting the CPU synchronisation is the first part needed. A key point of any pull renderer is ensuring no data races when accessing the shared world database, the simplest method a copy inside a critical section, ensuring that accept during the lock period whilst it duplicates the world database.

Once per frame required, the CPU will issue a ‘kickoff’ signal, telling the GPU to start pulling (in theory it possible to have a free running pull renderer, but in practise it helps to have a kick off signal from the CPU side).

When signalled the GPU slave will start reading the world database and cull objects using there bounding volume and world space matrix. If visible, it decodes the render mesh data structure and writes a portion of a command list to render that object. As the GPU front end the entire render pipeline will be here, with the CPU getting back to its simulation, its entire render overhead is just the time for the signal and a lock.

I’m in the last parts of closing up my life in the UK and moving to Stockholm, Sweden to join EA SEED. Its not the first time I’ve lived abroad and to be honest, I’ve learnt a lot, so feel more confident about it.

The job itself is very exciting, SEED is a research lab looking into the future of games, which is pretty much the description of my ideal job! What I’ll actually be doing, I don’t know yet but its such an awesome remit, finding things to do won’t be an issue. More likely too many ideas and things to investigate than not having something I want to do.

The team Johann has pulled together is also a pretty awesome dream team, its full of real industry heavy hitters and many names I know. The saying that “work with people smarter than you” will be very true i suspect! I’ll also be working with Mopp (Paul Greveson) again, we worked together on Brink a few years ago at Splash Damage. Mopp is one of those (rare!) talented art/tech combo people, which makes him an amazing person to work with.

New BLOG!

I’ve got out of the habit of updating my blog… Its been a crazy few years, the last post I made with last year!

So its time I get back it on a proper schedule etc., the old blog is on blogger but I’ve decided to use a thegrid.io as the engine this time, though I’ve linked the old one in for prosperity.

I've also starting to use me@deanoc.com as my permanent personal email.

Will be interesting to see how thegrid.io works… the idea is a AI designer will smarten the website up.