Coherent UI provides a very easy way to create UI by embedding HTML5 into the game engine. But what about the other way around? There are many games that display 3D game content in their UI. Just imagine how static would strategy games UI look without the animated unit portraits.

Many of our users have asked for similar features and as we discussed with our community, we have found many interesting uses of this technology: animated character portraits, 3D models of the weapons in the player inventory, isometric top-down camera to render realistic minimap. The ideas kept coming.

So we have researched how to implement it. Our goal was to be very easy to use, and on the other hand to be very efficient. Here is the design we have created and our process, we will be very happy to hear your opinion or additional ideas about the feature and how do you imagine using it.

The Live game view as part of the HUD

With modern game user interfaces, it is increasingly common to mix 3D content rendered by the game engine itself with the UI. There are many compelling examples:

• Unit portraits. In many strategy games, when you select a unit, its animated 3D portrait is visible in the UI. This adds a nice touch of immersion and makes the interface much more interesting and compelling.

• Mini-maps. Most mini-maps are just simple 2D images with some icons on them. A more creative decision would be to have the map rendered in 3D with the game engine. In this way it can be zoomed more freely and effects added more easily.

• Additional scene views. Imagine a car simulator – the rear view mirror is a good example of a second “eye” in the same 3D scene.

• Character pictures. In the inventory/equipment dialog of an RPG you could see your hero drawn in 3D in full armor instead of some pictorial representation of what he is wearing.

A naive way to accomplish such effects is to splat any of those secondary game scenes on-screen somewhere over your interface. That will work but more interesting results can be achieved if the 3D image effectively becomes part of the UI itself. That is because an interface system that knows about these images and can access them, could apply effects, draw text on them, blend seamlessly with other elements – this is the UI of the future, it blurs the line between the 3D world and the user interface.

We’ve tried a solution even with the current functionality.

The easiest way now is to get the bytes that make up the image you have rendered and want to use in the UI and send it to JavaScript with the Binding API. Then in JS you can use those bytes to create an ImageData object from a canvas and paint it. Once you have the ImageData you can also apply filters to it, draw text, animate it etc..

This simple solution works but is not fast enough for per-frame changing content on high resolutions.

We decided to add the feature as core to Coherent UI with a high-performance API. We believe it will add a whole new palette of exciting usage scenarios to our clients.

It will work something like this:

1. You render the image you want to display in the UI

2. Use a special Coherent UI object that represents a link between the game and the UI to update the representation in the HTML5/JS world

3. Done. Now via an object in the JS DOM you have the updated image that you can draw in a canvas, add to other even CSS 3D animated elements etc.

Initial tests are very promising and we notice no performance impact even with full HD images passed from the engine to the UI.

We’ll appreciate your feedback on the new feature and what you think you will use it for.

Follow Stoyan on Twitter: @stoyannk