Debugging Graphics

Intro

I develop a lot of graphics code using Vulkan, OpenGL, D3D12 and Metal and have found the following methods to make my life easier when something doesn't render right:

Enable the debug layer

Enable your API's debug output and check for runtime errors and warnings and fix them all.

 

D3D12

ID3D12Debug* debugController;
const HRESULT dhr = D3D12GetDebugInterface( IID_PPV_ARGS( &debugController ) );
if (dhr == S_OK)
{
debugController->EnableDebugLayer();
debugController->Release();
}
... CreateDevice()...
hr = device->QueryInterface( IID_PPV_ARGS( &infoQueue ) );
infoQueue->SetBreakOnSeverity( D3D12_MESSAGE_SEVERITY_ERROR, TRUE );

OpenGL

See https://www.opengl.org/wiki/Debug_Output

Vulkan

Sidenote: On my system RenderDoc crashes if I try to attach a program that has debug layer enabled.
First you need to install LunarG Vulkan SDK from http://lunarg.com/vulkan-sdk/
I contain my debug layer code inside a namespace like this:

namespace debug
{
    PFN_vkCreateDebugReportCallbackEXT CreateDebugReportCallback = nullptr;
    PFN_vkDestroyDebugReportCallbackEXT DestroyDebugReportCallback = nullptr;
    PFN_vkDebugReportMessageEXT dbgBreakCallback = nullptr;

    VkDebugReportCallbackEXT debugReportCallback = nullptr;
    const int validationLayerCount = 9;
    const char *validationLayerNames[] =
    {
        "VK_LAYER_GOOGLE_threading",
        "VK_LAYER_LUNARG_mem_tracker",
        "VK_LAYER_LUNARG_object_tracker",
        "VK_LAYER_LUNARG_draw_state",
        "VK_LAYER_LUNARG_param_checker",
        "VK_LAYER_LUNARG_swapchain",
        "VK_LAYER_LUNARG_device_limits",
        "VK_LAYER_LUNARG_image",
        "VK_LAYER_GOOGLE_unique_objects",
    };

    VkBool32 messageCallback(
        VkDebugReportFlagsEXT flags,
        VkDebugReportObjectTypeEXT,
        uint64_t, size_t, int32_t msgCode,
        const char* pLayerPrefix, const char* pMsg,
        void* )
    {
        if (flags & VK_DEBUG_REPORT_ERROR_BIT_EXT)
        {
            ae3d::System::Print( "Vulkan error: [%s], code: %d: %s\n", pLayerPrefix, msgCode, pMsg );
        }
        else if (flags & VK_DEBUG_REPORT_WARNING_BIT_EXT)
        {
            ae3d::System::Print( "Vulkan warning: [%s], code: %d: %s\n", pLayerPrefix, msgCode, pMsg );
        }

        return VK_FALSE;
    }

    void Setup( VkInstance instance )
    {
        CreateDebugReportCallback = (PFN_vkCreateDebugReportCallbackEXT)vkGetInstanceProcAddr( instance, "vkCreateDebugReportCallbackEXT" );
        DestroyDebugReportCallback = (PFN_vkDestroyDebugReportCallbackEXT)vkGetInstanceProcAddr( instance, "vkDestroyDebugReportCallbackEXT" );
        dbgBreakCallback = (PFN_vkDebugReportMessageEXT)vkGetInstanceProcAddr( instance, "vkDebugReportMessageEXT" );

        VkDebugReportCallbackCreateInfoEXT dbgCreateInfo;
        dbgCreateInfo.sType = VK_STRUCTURE_TYPE_DEBUG_REPORT_CREATE_INFO_EXT;
        dbgCreateInfo.pNext = nullptr;
        dbgCreateInfo.pfnCallback = (PFN_vkDebugReportCallbackEXT)messageCallback;
        dbgCreateInfo.pUserData = nullptr;
        dbgCreateInfo.flags = VK_DEBUG_REPORT_ERROR_BIT_EXT | VK_DEBUG_REPORT_WARNING_BIT_EXT | VK_DEBUG_REPORT_PERFORMANCE_WARNING_BIT_EXT;
        VkResult err = CreateDebugReportCallback( instance, &dbgCreateInfo, nullptr, &debugReportCallback );
    }

When creating Vulkan instance, I append VK_EXT_DEBUG_REPORT_EXTENSION_NAME into instanceCreateInfo.ppEnabledExtensionNames.
When creating Vulkan device, I pass validation layers like this: 
deviceCreateInfo.enabledLayerCount = debug::validationLayerCount;
deviceCreateInfo.ppEnabledLayerNames = debug::validationLayerNames;

Use debug names

Debug names appear in graphics debugger tools and validation layer messages so they help you find the object.

OpenGL

You'll need to make sure extension KHR_debug is available before using these functions:
glObjectLabel( GL_TEXTURE, textureHandle, nameLength, name );
glObjectLabel( GL_PROGRAM, shaderHandle, nameLength, name );
glObjectLabel( GL_FRAMEBUFFER, fboHandle, nameLength, name );
etc.

D3D12

ID3D12Resource* texture = ...;
texture->SetName( L"texture" );

If you need to convert a const char* into an LPCWSTR you can do it like this: 
wchar_t wstr[ 128 ];
std::mbstowcs( wstr, my_string.c_str(), 128 );
texture->SetName( wstr );

Metal

Many objects have a .label property:
metalTexture.label = @"texture";

Use tools

These tools can be used to verify the rendering process by inspecting textures, render targets, buffers, rasterizer state etc.
RenderDoc is a good debugger for D3D11, OpenGL and Vulkan.
For D3D12 Visual Studio's own debugger is good. You can get it by installing "graphics tools" in Windows 10: Settings->System->Apps & Features -> Manage optional features
OpenGL ES and Metal users on Mac will probably use Xcode's debugger.
AMD and NVIDIA also have tools for this.

Shader debugging

GLSL shaders can be compiled and checked for errors by glslangValidator. You can make it a part of your build process for extra credit. You can also use general static analysis tools like PVS-Studio or CppCheck. Using these tools I have found uninitialized variables in rendering code etc. Writing a shader hot-reloading system is not a big task but it pays off: Imagine debugging a video blitting shader that shows wrong colors. You can pause your game on a frame, modify the shader and see the results in that video frame instantly. You can also make the system take a screenshot before and after recompilation to more easily compare the results in an external program like Photoshop.

Test on multiple GPUs, even from the same vendor

There are differences in how textures are initialized (garbage or white/black etc.), resource transitions are done, flags are handled etc.

Conclusion

Many things can and will go wrong when rendering but there are features like validation layers and graphics debuggers that make finding the problem easier.