Coherent UI in CryEngine 3 (Redux)

After the article on CryDev.net about Coherent UI, a lot of people read our previous blog post about our experiment with CryEngine3 and suggested that we should use Hendrik Polczynski’s Plugin SDK to get low-level access to the CryEngine 3 renderer. The problem we previously had was that we couldn’t find a proper way of creating a texture and “inserting” it into the engine, having an ID and whatnot. I talked with Hendrik, he gave me some pointers about the features of his SDK and how it solves our problem. As a result I present you an improved version of Coherent UI in CryEngine 3! You’ll find the link for the source code at the end of this post.

This is a video of the end result with more of the same as last time. Except this time it’s much more fluid.

First I’ll start with some notes about the usage of the code:

• We used CryEngine 3.4.0 (3696) Free SDK for our tests. If you want to use 3.4.3, you need to change the version string in Code/Plugin_SDK/inc/IPluginBase.h to “3.4.3”. The code in the Coherent UI plugin doesn’t have any version specific parts so it should be ready to use with the new SDK, although we haven’t tested it.
• The 64-bit build hasn’t been tested neither because we updated our developer machines with Windows 8 and the CryEngine executables do not start. We had an issue with the 32-bit executables as well – the login dialog just froze when either the Launcher or Editor were started but that turned out to be caused by the AV – after excluding the executables we could finally start using CryEngine again.
• CryEngine 3 is one of the top engines on the market – as such it squeezes every bit of resources the machine has. On the other hand, Coherent UI is designed as an out of process interface framework that needs some CPU/GPU as well. There’s a chance that CryEngine will take most of the system resources and the UI can appear laggy. The best way to alleviate the problem in the free SDK is to check your FPS (use  r_DisplayInfo 1 to show dev info if not already shown) and set a maximum framerate value (using  sys_MaxFPS value) to something a little lower than what you’re getting from the FPS counter.
• When in-game, you can hide the default CryEngine interface using   g_ShowHUD 0
• The Editor isn’t fully supported; while in the Launcher we create and destroy Coherent UI Views in  OnLoadingStart/OnUnloadComplete callbacks, these are unsuitable for the sandbox because  OnLoadingStart is called even if you aren’t editing a level and OnUnloadComplete isn’t called at all. To work around this limitation, we’re creating the views in the  OnActionEvent callback when the user enters game mode. This has some drawbacks since we’re registering the player event listener in the same callback, but in another listener that is called before the one in the Coherent UI DLL. This means that player notifications will not work in the editor (e.g. the health change notification).

Plugin installation

Add the following action event handler as well (it’s used for registering a player event listener provided by the Coherent UI plugin that does the binding between C++ and JavaScript methods):

You’ll also need to setup include and library paths for the CryGame project.

 Plugin limitations

When using a DirectX9 renderer, Coherent UI Views can only use shared memory (and not shared textures), since CryEngine uses a  IDirect3DDevice9 , while shared textures require a  IDirect3DDevice9Ex. Creating a shared texture without an Ex device will just fail.

When using a DirectX11 renderer, HUD Views cannot be drawn, because the D3D plugin doesn’t hook to the  IDXGISwapChain::Present method. This is because there is no specific swap chain associated with the device which means you cannot use the device to get the swap chain. I thought of hooking to  IDXGIFactory::CreateSwapChain and using the device and HWND that I have to determine the instance of the swap chain so I can hook to the present method, but that’s left for the future. Shared textures can be used, though. Generally, it’s recommended to use the DX9 renderer so you can experience all the features.

Plugin architecture

The Coherent UI plugin follows the Plugin SDK architecture and exposes just a tiny bit of functions that are needed by CryGame. These functions are:

•  Initialize
•  Shutdown
•  GetPlayerEventListener

Class list

•  CoherentGeometry – Loads geometry in .obj format and intersects rays with the loaded geometry, returning distance to the hit point and barycentric coordinates. This class is used because I couldn’t find a way to get the texture coordinates out of the CryEngine provided methods (the physics query did not return texture coordinates)
•  CoherentInputEventListener – IInputEventListener implementation that converts CryEngine input events to Coherent input events.
•  CoherentPlayerEventListener – Used for demonstrating binding of engine events to UI events.
•  CoherentSystemEventListener – UI system events listener, required by Coherent UI.
•  CoherentUISystem – Deals with the communication with the UI system, managing Coherent UI views, creation of rendering resources and all UI logic.
•  CoherentViewListener – Reports events for Coherent UI Views and performs low-level drawing; it can also own a  CoherentGeometry object that is considered the collision geometry for the View.
•  CPluginCoherentUI – Implements the plugin interface.
•  FullscreenTriangleDrawer – Does exactly what the name implies. Used for drawing the HUD.

Code walkthrough

The classes that do most of the work are CoherentUISystem and CoherentViewListener, so the focus will be on them. We’ll just skim through the CoherentInputEventListener briefly, so you know what’s happening when you press buttons. The input listener does nothing fancy – it just converts CryEngine input events to Coherent UI input events. Examine the code if you’re interested how to do that.

It maps the following keys to some useful functions:

• Numpad0 stops player input and allows forwarding to Coherent UI Views
• Numpad2 toggles drawing the Coherent UI HUD
• Numpad3 hides and shows the mouse cursor while playing.

The input is forwarded to the Coherent UI View that you last moused over (it helps showing the mouse so you know where the cursor is).

To summarize, when you want to send input to a Coherent UI View, press Numpad0 to stop player input and enable forwarding to Coherent UI, press Numpad3 to show the cursor, and mouse over the view that you want to send input to.

On to the CoherentViewListener. This class has a pointer to a DirectX texture that it draws to when it receives the OnDraw notification. It’s important that we draw in the Render Thread or otherwise two threads could simultaneously access the DirectX device and cause a corruption. The OnDraw notification is called in the same call stack that Coherent::UI::UISystem::FetchSurfaces is – that means FetchSurfacesmust be called from the render thread. We do that by using the D3D plugin which hooks to functions that are always called in the needed context. The drawing itself consists of copying one surface onto another, nothing fancy there.

The listener is also notified when Coherent UI needs to create or destroy textures. These notifications are in the Coherent::UI::UISystem::Update call stack, which is in the main thread. This means we can’t use the device immediately and we need to ask the CoherentUISystem object to do the job in the rendering thread – that’s no problem because Coherenet UI supports asynchronous surface creation. The last thing the listener does is providing a convenient method for ray casting through the listener’s collision geometry, returning the texture coordinates of the hit point (if any).

Last, the CoherentUISystem class – it’s pretty much the alpha and omega of the plugin. It receives DirectX notifications from the D3D plugin, executes tasks scheduled for the render thread (such as creating and destroying textures), manages the Coherent UI Views and encapsulates all the UI logic. It can also make coffee if you ask nicely :).

I tried to keep the UI logic as realistic as possible and this is what I came up with: the UI system itself is initialized in the plugin when the game starts and is active throughout the its lifetime. When a new level starts loading, view listeners and their corresponding Views are created for the HUD and objects. The object names are hard coded for the purposes of the example and are relevant for the level provided. Also during loading the input and player listeners are registered. Everything that is created when loading is destroyed when unloading a level.

The management of Coherent UI Views mostly consists of creating and destroying textures, requested by the views and providing help with the drawing of HUD Views. The CoherentUISystem maintains a collection of requests for texture creation that need to be executed on the render thread. These requests are processed in the CCoherentUISystem::OnPostBeginScene every frame.

Views that are placed on objects are updated in the CCoherentViewListener::OnDraw method – a texture is rendered to and later used by CryEngine, no further work required. When displaying a HUD, there’s a bit more to it – it’s a separate texture that needs to be drawn after all post-processing and is not associated with any object. For this purpose we’re doing that in the CCoherentUISystem::OnPrePresent callback – we just render a full screen triangle with the HUD texture mapped onto it. The textures from Coherent UI come with premultiplied alpha so before drawing the triangle we need to set the blending modes to One/Inverse Source Alpha for the source/destination targets respectively. Note that as we mentioned this callback is not available in the D3D plugin for the DX11 renderer, so no HUD there. Creating the textures used by CryEngine is done through the D3D plugin. Examine the CCoherentUISystem::SetTexturesForListeners method to see exactly how.

Check out the full source code for the plugin at github. Bear in mind that the limited version supports only one Coherent UI View, so you’ll have to choose which one you want. There’s a friendly error message during compilation that will direct you to the code that you need to change.