Planning Aether3D Rewrite For 2015

In 2015 spring I'll rewrite my engine again.

It's that time again, the moment when I look at my engine, started in 2013, and an accumulated list of architecture improvements that are easier to put into a completely new engine instead of shoehorning them into the current engine. I know that rewriting is a thing you should never do, but I am quite satisfied with the current engine and can take many things into the new engine. Refactoring could get my current engine into the right direction, but some of the things I want are just too big for current architecture.

Rationale:

Static linkage/getting rid of virtual methods

My current engine's whole API is exposed as virtual methods, but I want to use them only where it really makes sense. Static linkage also makes configuration easier, not having to worry about DLL's location or format.

Singletons/global state

Current engine has some, but I want none.

Entity-Component System

I want to compose game objects by combining components like in Unity. Separation of Concerns FTW.

Memory labeling/profiling

I want to have a breakdown of memory used by a game, asset by asset. That means custom allocators, which will come handy because...

Minimizing dynamic allocation

I want to have a data-oriented cache-friendly design. When I create a texture, I want to get an index into a texture pool that was allocated on startup. I will analyze my main loop and minimize dynamic allocations. I will also check cache-misses.

API inside a namespace

My current engine's public API doesn't use namespaces, so collisions are possible. Not so in my next engine.

More const-correct design

While my current engine is mostly const-correct, more objects and state could be made immutable, making reasoning about program's behaviour easier.

Compute shaders aren't exposed

My current engine uses them internally for light culling, but I want to give the power to you, the user.

Post-Processing Effects can be pulled out

Separation of Concerns, baby. Unity has them as a separate package, I want to have them out of engine core, too.

AZDO

Maybe not full-on, but definitely something to strive for.

Updated Renderer APIs

Exact versions can vary, but I aim for OpenGL 4.5, GLES 3.1, D3D11.1 and Metal. Need to figure out something for OS X, maybe 4.1.

Editor

Many open-source engines have editors that contain basic usability problems that are easy to fix. I have been prototyping a custom scene editor using Qt 5 for several months and believe I can make an editor that's not painful to use. The editor will be in sync with the engine and support new features as soon as they are implemented. It is by no means necessary to use the editor, the engine works without it, too.

GitHub

The whole building of the new engine will happen on GitHub, right from the start. It gives the engine discoverability and easier management of issues/documentation/wiki etc.

Dogfooding

I want to have a real use-case for every feature I add to the engine. A simple game, a demo, whatever, just something real, released stuff.

Schedule

I will do research until GDC 2015 (March 2-6) and a few weeks after that I plan to make the first commit to GitHub.

Software Development Terms

Here's a quick description of random software development terms. Most developers probably know them but someone could find something new.

Broken Windows Theory
If there is a low-quality part of a codebase, people maintaining the codebase tend to produce more low-quality code.

Technical Debt
Taking shortcuts by making intentionally suboptimal design or implementation decisions. Not always bad, not all debt have to be paid.

Dogfooding
Using your own unfinished product in day-to-day work while developing it.

Code Smell
Doing something that could lead to problems.

Heisenbug
A bug that doesn't repro when debugging.

Cargo-Cult
Applying a method that works in one context in a different context without knowing why it worked in the original context.

Use That Profiler

I was on a long bus trip and was bored, so I launched Xcode and worked on my game prototype. I wanted to see its memory usage so I opened Instruments Allocator tool. I noticed something strange, multiple OpenGL buffer creation calls on every frame. I figured out it must be in RenderQueue class because the screen was only showing 2D stuff:

But my render queue only creates GL buffers when it's full! Turns out I emptied the queue element container always after drawing, causing the next draw to generate it again. The solution was to let the container grow and reuse old elements. This bug has been in my engine since the beginning and only after actually starting to make a game I noticed it even though I have profiled the engine every now and then. Runtime buffer creation should be avoided as much as possible.

Takeaway: profile your code early and often. Don't make engines or games, make both.

2048 Point Lights @ 60 FPS

Today I released an update to my engine. The most prominent feature
is Forward+ aka Tiled Forward rendering. To my knowledge, no other
open source engine implements it.

Forward+ dates back to 2012 and is used in several AAA games like DiRT Showdown, Ryse (partially) and The Order: 1886. Its main advantages are support for MSAA, transparency and multiple BRDFs. The algorithm works by culling lights against a screen-space grid where cells are for example 16x16 pixels. It stores a per-tile list of light indices that are then used in shading. Culling is done with a compute shader and it uses depth texture to get each tile's extents. My engine does a depth-pass in any case, because it's needed for SSAO etc.

The implementation I did is based on AMD's ForwardPlus11 sample. However, the sample is D3D11 and uses deprecated D3DX libs, but my implementation also supports OpenGL 4.3. I was able to render 2048 point lights at 60 FPS in Sponza scene on my year-old MacBook Pro (GeForce 750M). All the surfaces use Cook-Torrance with normal maps.

If you want to implement it yourself or just test it, you can download my engine. Here's some API mappings from HLSL to GLSL:
gl_WorkGroupID: SV_GroupID
gl_LocalInvocationID: SV_GroupThreadID
gl_GlobalInvocationID: SV_DispatchThreadID
uintBitsToFloat(): asfloat()
floatBitsToUint(): asuint()


Here's some useful links:
http://www.cse.chalmers.se/~olaolss/main_frame.php?contents=publications
http://aras-p.info/blog/2012/03/27/tiled-forward-shading-links/
http://diaryofagraphicsprogrammer.blogspot.fi/2012/04/tile-based-deferred-and-forward.html
http://articles.pjblewis.com/
http://focus.gscept.com/2014ip19/

So, what's next for Aether3D? I'll add spot lights soon and for the next release my focus will be in fixing open bugs and adding unit tests and support for CMake. I'm also planning to dogfood my engine this year by making a simple first-person game. Also, I'm working on an editor in Qt and C++. I'll blog more about them when they are more mature.

6 Steps to Better Commits

While working on my multi-platform game engine, here's some things I have learned:

1. Run all unit tests or functionality tests before submitting.
Should be a no-brainer, but laziness can lead to less testing and more bugs. Especially bad if you make many submits without testing because by then your search-space has grown needlessly. If you don't have any tests, stop reading this and write them now.

2. Submit only related changes.
For example, if you are hunting down a bug and inspect a changelist with a message "Fixed texture loading" and the changelist also contains modifications to Audio.cpp, finding the bug takes more time.

3. Fix bugs before adding new features.
Worst-case scenario would be when a new feature depends on a bug in an older feature and when the bug is finally fixed, the new feature stops working.

4. Describe all the changes you did in the commit message.
If you browse through a commit log of a source file and find a commit with a message that has nothing to do with that file, you have not documented your commit well and it could be hard to reason about why the file was changed.

5. Diff all the changes to be submitted for sanity-check.
Sometimes you have been making ad-hoc changes to test/debug a feature but forgot to revert them before submitting.

6. Try to submit a whole, working new feature in one submit instead of many non-working submits.
If the feature is really big, you can split it into smaller submits if each of the smaller submits represents a logical unit that can work without other submits.

GDC 2014

Game Developers Conference 2014 was held on March 17-21, 2014 at Moscone Center, San Francisco. It was my fourth GDC so far, and in this blog post I’ll share some details.

There were several big engine announcements. Unity unveiled Unity 5 which features real-time GI using Enlighten, full deferred shading, 64-bit editor, SpeedTree, physically-based built-in shader and a frame debugger among other features. PBS is great, but there are no area light sources. However, real-time GI helps a lot to that effect. Unity 4.6 will also have a new GUI system, but I avoid the topic because of an NDA.

Epic launched much anticipated Unreal Engine 4 and surprised everyone by making the source code available on GitHub for developers. The licence cost is very indie-friendly at $20/mo, 5 % of sales excluding consoles. Monthly subscription (EaaS) seems to be the new trend among engines, Unity has had an option for it for a while and Crytek also announced EaaS model for CRYENGINE at GDC. I’ll definitely try all the new engines in the near future and maybe write up blog posts about them.

I attended several game/engine rendering talks and PBS using Schlick Fresnel, GGX and Cook-Torrance seems to be pretty standard stuff nowadays. Khronos announced OpenGL ES 3.1 which finally features compute shaders. I’m currently evaluating which algorithm I’ll use for my engine’s lighting and Forward+ is a strong candidate so CS support in ES 3.1 would make it more feasible.

Overall, this year’s GDC was great and I met many cool people and got a lot of takeaways from the sessions that I’ll start applying into my engine when the jet lag wears off.

Rebooted Aether3D Game Engine First Release

I released my new game engine today. This first release contains many of the features I planned on doing in my previous blog post.

I see no reason to support deprecated systems when developing a new engine. The oldest OpenGL context I'm supporting is 3.2 and I only support it to get the engine running on a Mac. Mavericks supports 4.1 but I started developing before it was released.

So, what's next for Aether3D? I try to add a performance test suite for tracking performance differences between releases, SSAO, instancing, shadows and support for vertex color to the next release.