Overview

The user interface or GUI is one of the most important aspects of any game or application. When Epic Games released the Unreal Engine 4 subscription a few months ago, we have posted a comparison between the different ways to create a user interface (UI) for Unreal Engine 4: Slate, Canvas, and using HTML with Coherent UI HTML5 rendering engine.
Some of the benefits of using HTML for the game user interface are: easy to use and widespread technology, a lot of existing libraries and templates, saving considerable time, more convenient workflow, visual tools support and much more.

Open-source solutions

CEF or Chromium Embedded Framework is one of the easiest ways to integrate HTML renderer in any application. Many developers are choosing to roll their own CEF-based Unreal Engine 4 integration and there are some commercial products based on CEF framework.
We will describe the process of creating your own integration of CEF framework in UE4 and list some specifics that should be considered.

Integrating CEF

Chromium Embedded Framework is a free, open-source framework that is based on Chromium web browser and provides the infrastructure developers need to quickly add HTML renderer and JavaScript to a C++ project. Implementation is very straightforward and usually can be achieved in a couple of days. This might be a good starting point if you are interested in using HTML within your UE4 game. More information about CEF can be found at https://code.google.com/p/chromiumembedded/.

The best way to start is to get a pre-built version of CEF for the needed platform, in order to save time in building an own version of Chromium and CEF, which is a very time-consuming task. For basic web browser functionality or simplistic UI in the game, this should be enough.

1. Download the latest CEF binaries
2. Create a UE4 plugin cloned from BlankPlugin sample (available in UE4) and add references to the CEF library
3. Edit the BlankPlugin code and initialize CEF on plugin initialization with your HTML content – e.g your game main menu (there are many examples on CEF website)
4. On each frame call appropriate render function in the CEF library, which returns rendered HTML view as a bitmap, your plugin should have a UTexture where to copy the bytes from the CEF surface
5. Compose the UTexture to a desired quad or mesh
6. Wire the UE4 keyboard/mouse events to the CEF ones, performing coordinate conversion, so the web surfaces can be interacted with
7. Voila – now you have a simple web browser within your UE4 game!

Beyond simple integration

Of course, this is only the beginning, there are more things to consider for a fully-fledged UI solution.

One of the first things to think about is that CEF was not designed for games (or any demanding real-time application). Games have higher requirements in terms of performance and are quite different than normal desktop apps like Chrome itself. For games, GPU acceleration and drawing everything in hardware is mandatory for achieving 60 FPS experience.
Chromium and CEF offer hardware (GPU) accelerated rendering, but use a separate operating system process for rendering the page in this case. Getting image bytes to put in UE4 texture requires at least two copy operations on each UI frame between the GPU VRAM and system RAM, transfer it across the game and rendering processes, and then in the UE4 engine upload it back from system RAM to VRAM. These copies are expensive and can significantly impact the smoothness of the game UI, especially if you use animation or play video.

One way to solve this is to render everything on the CPU (which is how CEF does it) but then many modern HTML features are not available. There are alternative solutions, which require significant implementation and modifications to Chromium code, but those require a deep understanding of both Chromium and the game engine and are very hard to maintain because Chromium and CEF code is updated frequently.

Another performance consideration is the lag or the time it takes for an update to populate to the UI, and the game engine to receive back the rendered frame. The system that CEF uses to transfer the data between rendering and game processes, known as IPC (Inter-Process Communication) has been developed with the needs of a desktop browser, and for games may introduce several millisecond lags which may be undesirable.

Chromium and CEF are designed to take advantage of multithreading capabilities and many of its functions are asynchronous. There is distinct separation of logic in several operating system processes (for more information check out http://www.chromium.org/developers/design-documents/multi-process-architecture ). The good thing about that is that it maximizes the use of the system resources and provides memory isolation. But the major drawback is that the rendering is never in sync with the 3D engine and usually lags several frames. This is ok for many types of UI like menus and in-game browsers, but for specific scenario of 3D positioned views, like overhead name-plates that have to move with the player, lag is not acceptable.

As with every complex system, it is more important to test things with real content. Our recommendation is: if you are building your own CEF implementation to test with real UI elements as early as possible and if you are using a commercial product to require a trial version.

Having an up-to-date version of the underlying code is also a major consideration point. Chromium as a platform is evolving very fast with hundreds of fixes and new features each week. There are 2 major rendering paths in Chromium – software and hardware. To integrate accelerated rendering with a DirectX engine, only hardware can be used with high performance. CEF used the software path to implement off-screen rendering, but unfortunately, it is being removed in Chromium. This will probably lead CEF to use Chromium workarounds which may not be supported and will limit the upgrade path of the solution. As before, our advice is to test and make sure that all the HTML5 features that you need, like animations and CSS 3D transforms are available, and perform fast enough, before committing to a solution.

The CEF API and commercial solutions, that are thin wrappers based on CEF, expose an interface that is missing many features needed for modern game UI and are not well suited for games. For example, one common task in game UI is to determine “Is the mouse on the UI”. When the user clicks on the HUD, the developer must know if an element on the UI was clicked or the mouse should be forwarded down to the game world. Only basic browser or square UIs are easily supported with CEF, but developers have to invest significant time to create such functionality if the game needs irregular UI with transparent elements (like HUDs).

Another important aspect of UI development is the scripting of the interface and its interaction with the game. CEF supports low-level C++ to and from JavaScript communication. Exporting basic types is several lines of code, but creating a cohesive system for more complex custom types, object hierarchies and containers is a serious task. The trivial way in which the data is duplicated in the engine and in the UI JavaScript DOM may lead to excessive memory usage, memory leaks, and de-synchronization. Having a good binding system can be a huge time-saver and helps minimize potential bugs, but is a serious investment.

Because CEF and Chromium are open source projects, that integrate many additional libraries with a diverse set of licenses, it is very important to review the licensing requirements with your legal counsel.
An additional complication is that some of the libraries that make Chromium and are included in CEF have multiple licenses and depend on compilation options of the libraries. If you want to be sure you may need to compile the libraries yourself, instead of relying on pre-compiled versions. Imagine a situation where you have to take down your released project and replace the UI! If you are acquiring a commercial library, make sure that the vendor has included information in the license agreement and complies with the licenses.

Summary

To sum up, CEF has many advantages for implementing game UI and is very easy to embed and use quickly but also has several points that have to be carefully considered.

We have not touched many advanced uses like controlling picture-in-picture (live views) support, time control and stopping time (e.g when game is being debugged), multi-character text entry or IME (needed to supporting different languages), audio integration, and more. We may cover some of these in future postings if there is interest.

Our recommendation is to try out your own CEF implementation or other HTML solutions available with realistic content, specific performance requirements, and hardware and software compatibility in mind.

Our product, Coherent UI, is based on Chromium (but not CEF) and is a very good way to start using HTML5 for your game UI. We have addressed all of the above-mentioned points and we have plenty of additional features. Our free fully-functional trial will allow you to quickly see and experiment how you can create HTML5-powered UI quickly. Getting started with Coherent UI is quite easy thanks to our extensive documentation, many samples, and video tutorials. The product’s high performance has been proven in many games on the market and being deployed to millions of end users allows us to have extensive hardware and software configuration support.

Try the high-performance HTML5-based UI SDK Coherent UI 2.0 for free HERE.