2.9.16.0
Coherent GT for UE4
Coherent GT for Unreal Engine 4 plugin

Getting started

This tutorial will show you how to add the Coherent GT plugin into a project and create two types of UI elements - a HUD and an in-world UI view . Here are the steps to achieve this:

1. Install the Coherent GT plugin

First, you need to install the Coherent GT plugin using the installer wizard. The installer auto-detects the UE4 engine versions that are available on this computer and gives you the options to install UI kits and a sample game

InstallWizard.png
Installing Coherent GT plugin

2. Enable the plugin in your project

Then start the Unreal Editor, and choose either an existing project or create a new one via New Project -> Blueprint -> Blank, and save it in a folder of your choice.

The first thing you have to do is to enable the Coherent GT plugin in the editor. From the main menu click Edit -> Plugins, then navigate to the User Interface section and find the Coherent GT Plugin and click the checkbox Enable. You'll have to restart the editor and when you do a new drop-down menu will appear - Coherent GT.

Then create a new level and open the world settings. From the Game Mode section, click the + button to create a new GameMode override, let's name it MyGameMode.

OverrideDefaultGameMode.png
Override the default game mode
Now, from the main menu choose Edit -> Project Settings, and navigate to Maps & Modes in the Project section, then set the Default GameMode to MyGameMode.

3. Add a HUD

From the Coherent GT menu, choose Add HUD. This will change the current GameMode's HUD class to GTGameHUDBP and will initialize it. Also, it will add a hud.html page in YourProjectDir/Content/UIResources/ directory, if not already present; then this page will be opened in your default browser.

GameModeOverrideWithGTHUD.png
GTGameHUDBP for GameMode's HUD
For the HUD, we'll add some UI resources in YourProjectDir/Content/uiresources folder, which can be downloaded from here: UIResources.zip. As we previously mentioned, the HUD has been pre-setup for you and is ready to use, so you don't need to do any modifications to it, except for changing the view Page URL to coui://uiresources/HUD/MainUI.html.

To see its content, open the GTGameHUDBP blueprint in the Blueprint Editor. If you experience difficulties locating GTGameHUDBP blueprint, you need to set Show Plugin Content and Show Engine Content checkboxes in the Content Browser View Options and search for CoherentUIGTPlugin Content. Here's a brief explanation of what happens:

  • setting up the View - page, transparency, alpha threshold
ViewSetup.png
Setup the View
  • the UI input is initialized - this involves spawning and initializing a CoherentUIGTInputActor, and then setting “Accept mouse input” to true.
InputSetup.png
UI Input initialization
  • setting up UI scripting (the two-way binding between JavaScript and Blueprints)
UIScripting.png
Setup UI Scripting

The only part that we cover in this tutorial, is the View setup. The UI Scripting and the input will be covered in upcoming tutorials.

The page is set by default to coui://uiresources/hud.html, so the newly created page by Add HUD will be used if you skip changing it. However, you can change it whenever you want to a local page, placed in the uiresources folder and use the coui:// scheme, or to a page hosted on the web and use a standard scheme like http://.

4. Add in-world UI

Choose Coherent GT -> Add In-world UI. This will add a GTPlane to your scene, just in front of the player. You can change its transform via the World Transform in the Details tab of the GTPlane.

Click Edit GTPlane from the same tab and its blueprint will open. As you can see, it has a CoherentUIGT component and the UIScripting already setup for this view.

Follow these steps to setup the view::

  1. Get a reference to the CoherentUIGT component in the EventGraph of the GTPlane blueprint.
  2. Drag a pin from the reference and create SetURL node and set the url to coui://uiresources/animation-demo.html
  3. Add a Begin play and connect it with the url setter.
  4. Now go to the editor and click Play.
InWorldUISetup.png
Setup in-world view

Here is the final result:

GameScreenshot.png
Final Result

Programmer's Quickstart guide

Engine plugin or a game plugin? Which one do I need?

CoherentUIGT can be installed as either engine plugin or a game plugin. Each of them having their own merit.

Engine plugin

This is the default option of istallation. When you have CoherentUIGT installed as an engine plugin, the plugin will be available for all of the games that are created with that particular instance of the UE4 engine. This is useful when you are working on more than one game at the same tyme, or you are prototyping.

Note: The engine plugin option is only available when using a source built engine. This is due to limitations of the binary engine not being to compile engine / plugins code.

Game plugin

Usually the game plugin option is used when using a binary engine. This is because this is the only way to build a plugin when using a binary engine. Another reason why you would like to use CoherentUIGT as a game plugin would be if you don't want the plugin to be visible to other games using the same engine.

Plugin core classes

Coherent GT consists of several core classes:

  • UCoherentUIGTBaseComponent is an abstract class, actor component, which contains the necessary code to hook Coherent GT up with UE4.
  • UCoherentUIGTComponent is derived from the BaseComponent and can be used to add UI in the 3D world. It's one of the core Coherent GT classes and represents a single HTML page that is rendered. Also holds methods for manipulating the view.
  • UCoherentUIGTHUD is derived from the base component and can be added to a AHUD actor to add HUD.
  • ACoherentUIGTGameHUD is a HUD class that is derived from AHUD and is the easiest way to use GT - you can just replace your AHUD class with this one. It's a wrapper around UCoherentUIGTHUD with fairly simple implementation. This should be your main starting point if you want to have a Coherent GT View attached on your view.
  • ACoherentUIGTInputActor is the input actor is responsible for capturing and forwarding mouse and keyboard input to the currently focused Coherent GT View. It must be spawned and initialized in the game world.
  • SCoherentUIGTWidget is a slate widget capable of running Coherent GT.
  • UCoherentUIGTWidget is an UMG widget capable of running Coherent GT.
  • SCoherentUIGTLoadingScreenWidget is a slate widget for creating loading screen with Coherent GT. It's derived from SCoherentUIGTWidget and requires GT running in async mode.

Coherent GT Views in the 3D world

Attaching a Coherent GT View to a 3D object in the game world is actually a very easy thing to do - all you have to do is:

  1. Add a CoherentUIGTComponent to the object
  2. In the Static Mesh component of the object, change the material used to "GTMaterial".

C++ and Blueprint binding with HTML/JavaScript

The CoherentUIGT plugin provides a two way binding between the C++ or Blueprints and the HTML/JavaScript. Bofore any events are called, you must first register for receiving them.

  • To register for an event received on the C++ side you have to use one of these methods:
    CoherentUIGTComponent->GetView()->RegisterForEvent(...)
    CoherentUIGTComponent->GetView()->BindCall(...)
    
  • To register for an event received by the JavaScript your to call the engine.on function:
    engine.on('Click', function()
    {
        console.log('Click called');
    });
    

In both cases, the view you must wait until the view is ready for binding and then attach your event handlers.

For a detailed guide on binding you can check the UI Scripting section.

Introduction

This page will introduce you on how to set-up and use the Coherent GT middleware library with Unreal Engine 4.

Coherent GT is a modern HTML5-based user interface middleware library for real-time applications. It has been pre-integrated with Unreal Engine 4 and all the source of the integration plugin is available to Coherent GT and UE4 licensees.

Coherent GT is based on browser-like technology and can also be used as an in-game browser. It can load and render any page from the web and supports all modern authentication methods and protocols.

This guide requires basic knowledge of both Coherent GT and Unreal Engine 4. For more info on the discussed topics, please refer to the Coherent GT Documentation. and the Unreal Engine Documentation.

Sample game

To quickly try out Coherent GT with the provided sample - try the provided CoUIGTTestFPS game.

The first thing you have to do is to install the Coherent GT plugin using the installer wizard. The installer auto-detects the UE4 engine versions that are available on this computer and gives you the options to install UI kits and the Coherent GT sample game.

  1. Build the UE4 Engine.
  2. Build the CoUIGTTestFPS game.
  3. Run the UE4 Engine.
  4. Click Projects->Browse and browse to the CoUIGTTestFPS.uproject file.
  5. Open the game.

If you try to directly run the game from VS before having it previously opened, you might receive an error *"Failed to open descriptor file"*.

The game shows a simple menu, HUD, and when you press I, an in-game inventory in 3D will pop-up. If you press Tab, you'll be presented with a simple in-game menu.

UE4_sample_ui.jpg
Sample UI

Plugin structure

Coherent GT's plugin uses three modules:

  • one for the actual GT C++ shared libraries (.dll, .dylib, etc.);
  • an Engine module that sets up the dependency to the core Coherent GT library. It exposes several classes and Components specific for the development with UE4 and takes care of all the initialization and management of Coherent GT.;
  • an Editor module which eases your workflow with Coherent GT in the Unreal Editor.

And has one plugin as a dependency - CoherentRenderingPlugin, which in turn consists of two modules:

  • one for the actual Renoir C++ shared libraries (.dll, .dylib, etc.);
  • an Engine module that sets up the dependency to the core Renoir library. It takes care of all the initialization and management of the Renoir rendering library and contains the rendering backend which serves as the glue between the library and UE4's RHI.

Setting up the ThirdParty dependency

Coherent GT has to be added as a ThirdParty dependency to UE4. This will make sure that all shared libraries (.dll, .dylib, etc.) are loaded when the game starts and they will be referenced by the engine correctly.

The plugin can be installed either as an 'Engine' or as a 'Game' plugin. In UE4's terminology, an 'Engine' plugin resides in the EngineDir/Plugins directory, whereas a game plugin is installed in GameDir/Plugins. Our installer will choose the 'Engine' option for compiled from source distributions of UE4 and the 'Game' option for binary distributions.

Using an engine plugin has some advantages over the game plugin (for example you only need to install it once for all your projects) which is why we recommend it.

The Coherent GT installer will make sure that all the modules are installed correctly.

Install steps:

  1. Run the provided installer.
  2. When asked what to install, check UnrealEngine Plugin.
  3. Select which UE4 version you'd like to install the plugin to.
    1. If you've selected a binary distribution, Coherent GT will be installed as a game plugin and you will be asked for the path to the game.
    2. If you've selected an compiled from source distribution, Coherent GT will be installed as an engine plugin.
  4. Click install.

The installer provides some other useful items such as a sample game and a collection of ready-to-use HTML5 kits.

Add Coherent GT to your game

After you've installed the plugin, you need to enable it from the UE4 editor:

  1. Open the Unreal Engine Editor.
  2. Click on the Window menu.
  3. Click Plugins.
  4. Find the Coherent GT in the list of plugins.

If you are only going to use Coherent GT through the UE4 Editor / Blueprints, you are done. If you going to to use Coherent GT through C++ code, make sure your game references our plugin in its Build.cs file. Doing so is trivial - open YourGame/Source/YourGame/YourGame.Build.cs. Add the following lines to the constructor of the class YourGame:

PrivateDependencyModuleNames.Add("CoherentUIGTPlugin");
PrivateDependencyModuleNames.Add("CoherentUIGT");
Note
If you've installed the plugin as a 'Game' plugin to an existing game, the previous step has already been done for your by the installer.
In case than you can not activate the plugin from the editor, you can also activate it by adding the following lines in the .uproject file of your game.
"Plugins": [
    {
        "Name": "CoherentUIGTPlugin",
        "Enabled": true
    }
]

This will tell the engine that it should load the compiled CoherentUIGTPlugin.dll, if your project is using Coherent GT.

Note
Coherent GT will look for resources loaded through custom protocols (such as coui://) in *<YourGame>/Content/UIResources* folder.

Adding the Coherent GT plugin to your blueprint game

Using the installer

  1. Run the provided installer.
  2. When asked what to install, check Add Coherent GT to an existing game and uncheck Generate Project Files.
  3. Browse to the root directory of your game.
  4. Click install.

These are almost the same steps as in the previous section but generating project files will fail if your game has no C++ code.

One last step

Open the editor and enable the Coherent GT plugin for your game using the Window -> Plugins dialog. Coherent GT plugin is in the Installed / User Interface section.

Note
Without C++ code, you won't be able to distribute your game. See the next section for details.

Distributing (Packaging) Coherent GT games

UE4 currently doesn't build plugins in blueprint-only projects, therefore you can't use Coherent GT (or any other runtime plugin) without any C++ code. You can fix this by adding a simple dummy C++ class to your project, which is described in the next paragraph. If your game already has C++ code, skip the next section.

Fix for blueprint-only games

  1. Open the UE editor and go to File -> Add Code to Project
  2. Select empty class.
  3. Click "Next" until you're asked Would you like to edit the code now?. Click yes. (or click no and open the solution in Visual Studio manually)
  4. Build the solution from Visual Studio (CTRL + SHIFT + B). Don't worry if you get a linker error for UE4Editor-CoherentUIGTPlugin.dll. This is caused if your editor is still running while building.

Required steps for distribution

  1. Make sure your UIResources folder is located in YourGameDir/Content.
  2. Go to File -> Package Project -> Packaging settings and expand the Packaging options.
  3. Add UIResources under Additional Non-Asset Directories to Package.
  4. Package your game from the editor.
  5. Copy the CoherenUIGT and Renoir binaries for your platform (where YourPlatform should be Win64/Win32/Linux/MacOSX/etc.)
    • For Win64/Win32:
      • From: YourUnrealEngineDir/Engine/Binaries/ThirdParty/CoherentUIGT/YourPlatform and YourUnrealEngineDir/Engine/Binaries/ThirdParty/Renoir/YourPlatform
      • To: YourPackageDir/WindowsNoEditor/Engine/Binaries/ThirdParty/CoherentUIGT/YourPlatform and YourPackageDir/WindowsNoEditor/Engine/Binaries/ThirdParty/Renoir/YourPlatform
    • For Linux put our native shared libraries next to the game's executable:
      • From: YourUnrealEngineDir/Engine/Binaries/ThirdParty/CoherentUIGT/YourPlatform and YourUnrealEngineDir/Engine/Binaries/ThirdParty/Renoir/YourPlatform
      • To: YourPackageDir/YourPlatfromNoEditor/YourGame/Binaries/YourPlatform
    • For MacOSX, put our native libraries next to the game's executable:
      • From: YourUnrealEngineDir/Engine/Source/ThirdParty/CoherentUIGT/lib/YourPlatform and YourUnrealEngineDir/Engine/Source/ThirdParty/Renoir/lib/YourPlatform
      • To: YourPackageDir/YourPlatfromNoEditor/YourGame/Binaries/YourPlatform
    • For MacOSX also copy the libraries to the plugins' Binaries folder:
      • From: YourUnrealEngineDir/Engine/Source/ThirdParty/CoherentUIGT/lib/YourPlatform and YourUnrealEngineDir/Engine/Source/ThirdParty/Renoir/lib/YourPlatform
      • To: YourUnrealEngineDir/Engine/Plugins/Runtime/Coherent/CoherentUIGTPlugin/Binaries/YourPlatform and YourUnrealEngineDir/Engine/Plugins/Runtime/Coherent/CoherentRenderingPlugin/Binaries/YourPlatform

Note: Step 5 is done automatically by the plugin for all platforms. The information above can be used as a guide in case the packaged game does not run out of the box.

Note: Previous Coherent GT versions loaded UIResources from the root of your game directory (as opposed to YourGameDir/Content). That behaviour is still supported but it is deprecated, and will be removed in future versions.

If you are still using that behaviour, you are required to manually copy the UIResources folder to YourPackageDir/WindowsNoEditor/YourGameDir, next to the Content and Binary folders.

Your package directory structure should resemble this:

|-YourPackageDir
  |-WindowsNoEditor
    |-Engine
      |-Build
      |-Binaries
        |-ThirdParty
          |-CoherentUIGT
            |-Win64
              |-CoherentUIGT.dll
              |-CoherentGTCore.dll
              |-...
            |-Mac
              |-CoherentUIGT.dylib
              |-CoherentGTCore.dylib
              |-...
          |-Renoir
            |-Win64
              |-RenoirCore.WindowsDesktop.dll
              |-RenoirCore.WindowsDesktop.pdb
            |-Mac
              |-RenoirCore.Mac.dylib
    |-YourGameDir
      |-Saved
      |-Content
        |-UIResources
      |-Binaries
        |-Win64
          |-YourGame.exe
        |-<any_other_platform>
          |-YourGame.exe

Optional steps

You may want to automatically load a certain set of fonts (instead of using the CSS property @font-face). You can do so by adding them in the folder YourGameDir/Content/Fonts. Please note that Coherent GT requires the raw *.ttf* and *.otf* files and not the *.uasset* files UE4 will generated which also means that in order to distribute them you'll need to add the Fonts directory under Additional Non-Asset Directories to Package just as we did above for the UIResources directory.

Coherent GT Usage

The Coherent GT plugin allows to easily add Coherent GT Views to UE4 HUDs as well as surfaces in the world - this means having UI Views as textures on objects.

All exported Components inherit the CoherentUIGTViewListener class. It takes care of managing the creation/destruction of Coherent GT View resources and textures, and also listens for View events.

Some of the View events are broadcasted through Unreal Engine Delegates, too. Almost all interactions can be achieved through the Unreal Editor and the Blueprint Editor, and all View properties are exposed to them.

Coherent GT Editor Module

Coherent GT comes to Unreal with an editor module that eases your workflow. This section describes the editor module. For detailed information about each feature scroll past the section.

Coherent GT's Editor Module adds a pull-down menu to the main toolbar in your Unreal Editor, to the right of the Window menu.

coherent_uigt_pulldown_menu.png
Coherent GT's Editor Menu
Here's an overview of the functionality provided by each item in the menu:

Add HUD

Changes the GameMode's HUD class to the one provided with Coherent GT - CoherentUIGTGameHUD and adds a HTML file named hud.html to YourGame/Content/UIResources.

This item is only available if the current default GameMode is a Blueprint class. To see or change your current default GameMode, go to Edit -> Project Settings -> Maps and Modes -> Selected GameMode. GameMode classes written in C++ can't be changed from the editor which is why to use this option you would need to create a new blueprint that inherits from AGameMode. For further information, please refer to the UE4 docs.

By default, CoherentUIGTGameHUD will try to load hud.html (and use several other sensible default settings). If you'd like to customize the HUD's settings, you can do so with the following Blueprint:

bphud1_CoherentUIGTGameHUD_bp.jpg
Blueprint HUD with CoherentUIGTGameHUD HUD class

It's important to call SetupUIGTView as shown in the image above during Event Begin Play or otherwise the HUD will fall-back to its default settings.

Add Plane

Adds a 3D Plane with a CoherentUIGTComponent to the scene, facing current camera's position. The plane is an instance of GTPlane - a Coherent GT asset, which will be copied to YourGame/Content in case it's missing.

Add Page

Adds a HTML page to YourGame/Content/UIResources and launches it in your default browser.

Launch Inspector

Launches Coherent GT's debugger. You can change the path to the inspector in the Settings menu (see below). For more information about the debugger, see the Coherent GT Debugger section.

This item is only available if the game is not currently running in PIE.

Open Documentation

Opens Coherent GT's documentation on http://coherent-labs.com.

Open Tutorials

Opens Coherent GT's tutorials on http://coherent-labs.com.

Settings

Opens Coherent GT's settings (you can also access them from Edit -> Project Settings -> Plugins -> Coherent GT. This allows you to change the Inspector's port.

Coherent GT Views

At the core of Coherent GT stays the View. A Coherent GT View is basically an HTML5 page and the JavaScript context for it. Coherent GT will render the View, execute the JavaScript and provide UE4 with a texture with what it has drawn.

Views provide methods to change the current page, use local resources and interact with JavaScript. In UE4 you'll seldom have to use them directly since they are encapsulated in Components that can be manipulated in C++ code and in Blueprints.

Coherent GT Components

The Coherent GT UE4 plugin has two main Components - UCoherentUIGTHUD and UCoherentUIGTComponent and an actor - ACoherentUIGTSystem. This section, together with Coherent GT System, covers in detail all of them.

All Coherent GT UE4 Components inherit from the UCoherentUIGTBaseComponent Component. It exports the most important Coherent GT properties and events to the UE4 Engine and Blueprint Editor.

Note
The UCoherentUIGTBaseComponent should NOT be used directly by itself.

The system actor takes care of the properties that are global for all Components. If you don't create one, a default will be created in the game for you.

Coherent GT Views for HUDs (Blueprints)

You'll need a game mode override, which uses the CoherentUIGTGameHUD HUD class, in the Editor's World Settings.

worldsettings.png
World Settings custom HUD class
Then, you need to initialize the HUD using a blueprint similar to this one:

bphud1_CoherentUIGTGameHUD_bp.jpg
Blueprint HUD with CoherentUIGTGameHUD HUD class

If you don't initialize the HUD, Coherent GT will try to load your HUD from coui://uiresources/hud.html and use default view settings.

Using your own game mode/HUD class

In case you already have your own HUD class, you can just add a Coherent GT component in your HUD blueprint.

Next, make sure that you select the Manual texture option in the Coherent GT component's properties.

Last, draw the texture given by the Coherent GT component in the "Receive Draw HUD" event. Here's a sample blueprint for doing that:

bphud1_CoherentUIGT_component_bp.jpg
Blueprint HUD with CoherentUIGT component

Usually you'd be able to use the "Draw Simple Texture" function for doing simple drawing like this, but as of UE 4.7 there is a bug in the engine.

Coherent GT uses a render target texture and UE4 doesn't override the methods FTextureRenderTarget2DResource::GetSizeX() / GetSizeY() for it and instead return 0. These methods are used by "Draw Simple Texture" for obtaining the size so you cannot use that function. Instead, you'll have to use the more complex "Draw Texture".

You can get the size of the texture using the properties SizeX/SizeY and set them to the Screen W/H input pins of "Draw Texture".

Next, you need to set the the texture coordinates that will be used for sampling. If you want to draw the whole texture, use the following values for U/V/UWidth/VWidth: 0 / 0 / 1 / 1.

You can also do some simple effects like flipping the texture by modifying these values. For example, if you want to flip the texture vertically, use the following values for U/V/UWidth/VWidth: 0 / 1 / 1 / -1.

If you need to achieve some special effect using a material, you can also use the "Draw material" node. You just need to create a dynamic material instance that has a TextureSampleParameter2D which will be updated using the "Set Texture Parameter Value" function in the HUD blueprint:

bphud1_CoherentUIGT_component_bp_material2.png
Blueprint HUD with CoherentUIGT component
bphud1_CoherentUIGT_component_bp_material1.png
Blueprint HUD with CoherentUIGT using material

Coherent GT Views for HUDs (C++)

The CoherentUIGTHUD component is used to easily add Coherent GT Views to a game HUD. A sample game accompanies this guide and details can be seen there.

The Component must be added to the HUD Actor of the game, and then the HUD is in charge of creating its View, and must call the CoherentUIGTHUD::DrawHUD method inside it's own DrawHUD method (see CoUIGTTestFPSHUD.h and CoUIGTTestFPSHUD.cpp files for details).

The UCoherentUIGTHUD class is a Component that takes care of loading the HTML page with the HUD interface and draw it in the Canvas of the HUD Actor.

In the sample the CoherentUIGTHUD Component is created and added to the ACoUIGTTestFPSHUD Actor with the following code:

CoherentUIGTHUD = PCIP.CreateDefaultSubobject<UCoherentUIGTHUD>(this, TEXT("CoherentUIGTHUD"));

Then on every DrawHUD event the Coherent HUD is checked for readiness and eventually draw the HUD.

if (CoherentUIGTHUD)
{
    auto view = CoherentUIGTHUD->GetView();

    if (!view && !CoherentUIGTHUD->HasRequestedView() && CoherentUIGTHUD->IsReadyToCreateView())
    {
        CreateView();
    }
    // check that the size matches
    if (view)
    {
        if (view->GetWidth() != Canvas->ClipX
            || view->GetHeight() != Canvas->ClipY)
        {
            CoherentUIGTHUD->Resize(Canvas->ClipX, Canvas->ClipY);
        }
    }

    CoherentUIGTHUD->DrawHUD(Canvas, 0, 0);
}

In the snippet above you can see that the Component is checked for readiness and if it has already requested the creation of a View for itself. It it hasn't, it'll call the CreateView method.

If we already have a View we check if it has to be resized (for instance after we've resized the game window), and then call the DrawHUD method with the current Canvas and the screen coordinates where we want our View to draw itself.

The following code requests the creation of a View:

void ACoUIGTTestFPSHUD::CreateView()
{
    Coherent::UIGT::ViewInfo info;
    info.Width = Canvas->ClipX;
    info.Height = Canvas->ClipY;
    info.IsTransparent = true;

    CoherentUIGTHUD->ReadyForBindings.AddDynamic(this, &ACoUIGTTestFPSHUD::BindUI);

    CoherentUIGTHUD->CreateHUDView(info, TEXT("coui://UIResources/MainUI.html"));
}

All properties are self-explanatory, but in case you need more detail on any of them please consult with the Coherent GT C++ documentation.

The line

CoherentUIGTHUD->CreateHUDView(info, TEXT("coui://UIResources/MainUI.html"));

is the one that actually initiates the creation of the View. The coui://UIResources/MainUI.html page is the initial page that will be loaded. Note that coui is a special protocol that denotes local resources in the game. coui is read through a FCoherentUIGTFileHandler that for UE4 can be found in the CoherentUIGTBaseComponent.cpp file.

It will take care to read all files relative to the current game's content directory, so you can put your UI resources in the folder of your game. Coherent GT supports all standard protocols too, so you can load pages from the Internet.

The more interesting part from the code snippet above is this one:

CoherentUIGTHUD->ReadyForBindings.AddDynamic(this, &ACoUIGTTestFPSHUD::BindUI);

It binds a method that will be called when the View is ready to accept its bindings. Binding is the facility that allows the communication between C++ and JavaScript code in the page. For more information please refer to the Scripting section of this document.

Coherent GT Views for in-game Surfaces

Coherent GT Views can be used as textures on in-game objects to create interactive displays, call-outs and many other interesting effects. The component used is called CoherentUIGTComponent.

The steps to add a View to an object in the world are straight-forward.

  1. Create a Material in the Editor and connect a Texture2D object to it's Base Color input (or any field you need).
  2. Make the Texture a Parameter by right-clicking on it and selecting Convertto Parameter.
  3. Name the Parameter UITexture. The Coherent GT Component will later enumerate the materials and search for a parameter named so and it will dynamically update it with the View texture.
  4. Create a Material Instance from said material.
  5. Find the object you want to apply the View to and create a Blueprint for it.
  6. Set its material to the just created Material Instance.
  7. Add the CoherentUIGTComponent to the Blueprint and set all the parameters you need for it - size, URL etc.

Now you can use the Blueprint and add it in the world. The UI GT Component will automatically update the UITexture parameter with the texture of the View.

A material ready to use by Coherent GT can be found at Engine/Plugins/CoherentUIGTPlugin/Content/GTMaterial.uasset. In the sample game please check the GTPlane (which can also be found under Engine/Plugins/CoherentUIGTPlugin/Content/) objects that has the material setup and a "CoherentUIGTComponent" and can show pages on the in-world.

The properties of each View can be edited in the Blueprint editor in the Components menu under the CoherentUIGTComponent View section.

Anti Aliasing for in-game Surfaces

In UE4 it is possible to see bluriness in the in-game UI when moving. This is caused by the anti aliasing algorithm used by the engine. When Temporal AA is used (Unreal's default choice) in-game UI will be blurry.

How to fix it: Change AA algorithm to FXAA or disable AA.

If Temporal AA is needed:

  1. Open GTMaterial from Coherent GT Plugin contents.
  2. Check Responsive AA property in Translucency tab in the Details of the material.

Note: If the in-game surface is transparent the objects behind the surface will be aliased. Unfortunately, this is an issue with the way UE4 handles in-game UI.

Input overview

The input in Coherent GT is managed through a Slate widget (SCoherentUIGTInputForward). This widget is an empty widget that covers the whole screen and forwards input to a Coherent GT View. For simplicity we have created the ACoherentUIGTInputActor actor, which encapsulates the logic behind the widget and is accessible through both C++ and blueprints. All you have to do is spawn the actor and call its initialize method.

Keyboard, Mouse and Joystick focus in UE4

Keyboard focus, in UE4 terms, means that the Coherent GT Input Forward widget has keyboard focus. When the widget has keyboard focus, it is able to receive keyboard events, otherwise it is not.

Mouse events don't require focus and they are propagated to the widgets under the cursor until one of the widgets handles the event. The Coherent GT Input Forward widget is generally on the top of the viewport so it can process mouse events first. The processing in the widget works as follows:

  • If the widget does not have keyboard focus and the widget isn't set to always accept mouse events anyway, the event is unhandled. By default, the widget does not always allow mouse events so you need to have keyboard focus to receive them.
  • Check the pixel under the cursor for every HUD view. If a solid pixel is found (solid meaning having an alpha value greater than the click-through alpha threshold specified in the view component) the event is handled.
  • If none of the HUD views should handle the mouse event, a raycast through the mouse position is made. Depending on the widget settings it can trace either multiple objects, or just the first object the ray hits. If the hit object has a Coherent GT component the event is handled.
  • If neither a HUD view, nor a 3D object view should receive the mouse event, it's left unhandled and it's propagated to the next widgets.

Joystick input works by default when the viewport widget has focus. Keyboard focus implies joystick focus, which means that if the Coherent GT widget has keyboard focus, the joystick events will stop working. If that's not intended, the input actor has a method for keeping joystick focus to the viewport after setting keyboard focus.

View focus

Each view can be focused, regardless of the widget focus in UE4. This is core functionality and it's independent of any other focus mechanics. The view focus controls the color of selected text, whether you have a blinking caret in input fields and the like.

Here are a few examples for views that have focus (left) and ones that don't (right):

input_view_focus_1.jpg
Selected text
input_view_focus_2.jpg
Blinking caret

The current implementation of the input actor/widget allow for only one focused view at a time. If you need some custom behaviour you'll have to modify the plugin's code.

Each view component has a property (Receive Input) that controls whether a specific view takes part in the mouse event forwarding.

Input (Blueprints)

To add input in a blueprint only game, you need to spawn an actor that will set up the SCoherentInputForward widget. This is an empty widget that covers the whole screen and forwards input to a Coherent GT View.

The widget should be added in the front of the viewport so it can process mouse and keyboard messages first. Due to Unreal Engine 4's architecture, the widget can process keyboard messages only if it's focused.

Here's a sample blueprint that enables the input:

bp_input.jpg
Blueprint Input

The Coherent GT Input actor setup part shows how to spawn and initialize the input actor, which in turn adds the SCoherentUIGTInputForward widget to the viewport.

Note
At any time there should be only one input actor spawned, otherwise the input might not work correctly.

The Game to UI Focus part is for managing input focus - when you press Tab, the character input will stop and the UI will take it. This would seem to be enough, but currently UE4 stops firing key events when the focus is not on the viewport so we need other means to get focus back to the game. This is shown in the UI to game focus part. The Coherent GT input actor provides events for key/mouse up/down - in the sample we're using the key down event. When it's fired there's a check for the key enum to see if it's the Tab key, and if it is, the focus is toggled.

The "Toggle Coherent GT Input Focus" function provides means for toggling input without tracking it. If you want to, there are also functions for querying and setting the focus explicitly.

Input (C++)

The input in Coherent GT is managed through a Slate widget (SCoherentUIGTInputForward). This widget is an empty widget that covers the whole screen and forwards input to a Coherent GT View. The widget should be added in the front of the viewport so it can process mouse and keyboard messages first. Due to Unreal Engine 4's architecture, the widget can process keyboard messages only if it's focused. To focus the widget, you can use the FSlateApplication::Get().SetKeyboardFocus(Widget) method. Focus management must be implemented by the client.

You can also show the system mouse cursor using FSlateApplication::Get().ResetToDefaultInputSettings();. If you want to set focus back to the game, FSlateApplication::Get().SetFocusToGameViewport(); will do that.

If the SCoherentUIGTInputForward widget has focus, then keyboard events are sent to the focused Coherent GT View. To focus a view, you should click anywhere on it.

Mouse events are sent only to the Coherent GT View that is below the cursor. The code is inside the SCoherentUIGTInputForward::OnMouseEvent method. The method does roughly the following:

  • Checks if the mouse is over a solid pixel of any of the HUD Views, using the View::IssueMouseOnUIQuery, View::FetchMouseOnUIQuery and View::IsMouseOnView APIs. If any of the HUD Views is below the mouse, the appropriate mouse event is sent to it.
  • If none of the HUD Views should receive the event, a raycast through the world's geometry is done. Note that the raycast is configured to check only bodies of the ECC_WorldDynamic physics type. You can change that by using the SCoherentUIGTInputForward::SetRaycastCollisionGroup API. Currently, the collision group is set for all objects. If an object is hit and it has a UCoherentUIGTComponent, the mouse coordinates re converted in Coherent GT View space and sent to the View. The conversion is done using the UV coordinates of the static mesh. By default, the first UV channel is used, but that can be changed with SCoherentUIGTInputForward::SetRaycastUVChannel.

To setup the widget in the CoUIGTTestFPSSample the following code is added in CoUIGTTestFPSCharacter.h:

TSharedPtr<class SCoherentUIGTInputForward> CoherentUIGTInputForwardWidget;

Then in ACoUITestFPSCharacter::SetupCoherentUIGTInput in CoUITestFPSCharaceter.cpp the widget is instanced and initialized:

CoherentInputForwardWidget = SNew(SCoherentInputForward).Owner(this);
if (GEngine->IsValidLowLevel())
{
    GEngine->GameViewport->AddViewportWidgetContent(
        SNew(SWeakWidget).PossiblyNullContent(
            CoherentInputForwardWidget.ToSharedRef()));
}
if (CoherentInputForwardWidget.IsValid())
{
    CoherentInputForwardWidget->SetVisibility(EVisibility::Visible);
}

The code above creates and adds the SCoherentUIGTInputForward widget to the front of the viewport. After that the widget takes care for sending input to Coherent GT as explained above.

Of course, you can also use the Coherent GT View API to create your own input events and send them to Coherent GT. The View provides the MouseEvent, KeyEvent and TouchEvent methods that send input to the it. Each Coherent GT input event has to be populated with properties that can be trivially translated from the UE4 events.

Input use cases

Here we outline some common usages of the input system. The examples are given using blueprints and the Coherent GT Input actor as these maps directly to C++ code. We'll assume that you already have a spawned and initialized Coherent GT Input actor:

initialize_input_actor.png
Initialize Input Actor

Toggling input focus between the UI and the Game

Toggling input focus can be done 2 ways:

  • Clicking with the mouse on a Coherent GT view gives focus to the view since the Unreal's Slate system will focus the widget that handles the click. If you click back somewhere in the game viewport, the game will take focus.
  • The "click to focus" method works, but in a more realistic use case you'd want to be able to change the focus programatically. You can do that using the Coherent GT input actor's ToggleCoherentUIGTInputFocus method. The actor keeps track of the current focus internally so it can toggle between game and UI at any time. If you want to set it explicitly, you can use SetCoherentUIGTInputFocus, or query the state with IsCoherentUIGTFocused.
input_toggle_focus.jpg
Toggle focus

After the Coherent GT Input Actor has focused the input forwarding widget, you need to set which view should receive the keyboard input. This can be done with the SetCoherentUIGTViewFocus method. An alternative is to simply let the user click on the input field that she wants to type in.

input_set_view_focus.jpg
Set view focus

Propagating keyboard and joystick events when UI is focused

When the UI is focused, keyboard events are consumed by the Coherent GT Input forward widget by default. You can change that using the SetInputPropagationBehaviour method. The available options are to forward keyboard events, joystick events, both or none to other widgets.

This can be useful if you need to have your action handlers (e.g. "Jump", "Fire", etc.) executed even if the focused widget isn't the game viewport.

Note that joystick forwarding is only taken into account when using the Coherent GT Input actor's methods for setting (or toggling) focus.

input_propagation.jpg
Input propagation

Forwarding mouse input events to Coherent GT regardless of the focused widget

If you want your mouse input to be forwarded to Coherent GT first for hover events, but still receive keyboard events in your game, then you can use the AlwaysAcceptMouseInput method. When set to true, mouse events will be received by Coherent GT, regardless of whether the input forward widget is focused or not.

input_always_accept_mouse.jpg
Always accept mouse input

Including or removing specific Coherent GT components in the input forwarding

To control whether a Coherent GT view receives mouse events you can change the View component's Receive Input property. Only when this property is set to true will the component be considered for receiving input events.

input_set_receive_input_bp.jpg
Set Receive input property dynamically with BP
input_set_receive_input_components.jpg
Set receive input property from components tab

Input on multiple Coherent GT components in the 3D world

When you have multiple views on 3D objects and they are overlapping when viewed with the current camera, the front one will receive the mouse input events, and the back one will get nothing. If, for example, you have disabled input on the front one, the back one will still get nothing, because 3D objects under the cursor are gathered with a raycast from UE4, which is independent from the input forward widget's logic.

For performance reasons, by default the raycast returns only the first hit object. In a scenario like above, when you have 2 objects and you want to forward input to the back one when the front one is marked as not receiving input, you have to change the raycast type from returning single object to returning multiple objects.

input_focused_back_view_bp.jpg
Back view takes mouse input
input_focused_back_view.jpg
Back view takes mouse input

Capturing mouse input on transparent parts of the HUD

It's common in some cases (e.g. in-game menu) that you'd pause your game and show a menu that doesn't cover the whole screen.

With the default implementation of the input widget, mouse events will be passed on to the viewport (or other 3D views) when they happen in a transparent area of the HUD. This may not be desired with in-game menus, since the designer would likely want to have the menu focused until the player explicitly resumes the game.

There's a simple solution for that - setting the click-through alpha threshold. This value determines the maximum alpha value of a pixel so it's considered transparent and input is passed on the next handler. If you set that to 0, only completely transparent pixels pass input. If you set that to a negative value, e.g. -1, every pixel is assumed to be solid since its alpha is in the range [0, 1], thus never passing input to other handlers.

You can also do this with Blueprints as the CoherentUIGTComponent has the whole Coherent::UIGT::View API exposed.

AlphaThresholdBP.jpg
Set alpha threshold

Blueprint integration

Almost all View methods have been exposed to the Blueprint visual editing system. Coherent GT Components also trigger dynamic multicast delegates on a variety of events that users can subscribe to and use either from C++ or Blueprint.

A sample of the Blueprint usage is given if you open the Blueprint editor on the StaticActorCoherent object in the sample game.

blueprint_load.png
Blueprint load

In this case the FinishLoad event (a dynamic multicast delegate) of the Coherent GT Component will trigger a Print String with the text "Page on cube loaded!". FinishLoad is triggered by the Component when its page has been successfully loaded.

For a full list of the events exposed by the Coherent GT Components please refer to the CoherentUIGTBaseComponent.h file or review them in the Blueprint editor.

component_events.jpg
Component events
The Components also expose most of the functions of the View to Blueprints. They allow you to change the current page of the view, resize it, get all its current properties, etc. All functions are available under the View category.

For a full list of the provided methods please refer to the CoherentUIGTBaseComponent.h file or review them in the Blueprint editor.

component_functions.png
Component functions

UMG integration

Coherent GT is also integrated into Unreal Motion Graphics using the provided UCoherentUIGTWidget which can be found under Coherent GT category in the UMG editor.

umg_widget_editor.png
Coherent GT UMG Widget

An example usage of the UMG widget can be seen in the provided UMG_Map found in CoUIGTTestFps/Content/Maps. The map uses the UMGHUD widget blueprint found in CoUIGTTestFPS/Content which contains a single CoherentUIGTWidget stretched across the screen.

The widget behaves akin to the components we talked above but due to limitations of Unreal Engine there are some differences. The following properties and methods are present in UCoherentUIGTComponent but not in UCoherentUIGTWidget:

  • bool ManualTexture
  • int32 Width
  • int32 Height
  • virtual void Resize(uint32 width, uint32 height)
  • FUIGTOnLiveViewSizeRequest LiveViewSizeRequested

The rest of the public interface is the same and you can use the UMG widget wherever you are using the actor components.

Note that to have sound with the UMG widget you must assign an actor to UCoherentUIGTWidget::Owner after the game has started. This can be done like this:

umg_widget_assign_owner.png
Assigning an Owner Actor

Using UMG widgets on in-game objects

You can use Coherent's UMG Widget (UCoherentUIGTWidget) within UMG widgets attached as components on objects in the game world. To get optimal visual quality in this case it is required to:

  1. Go to the Rendering properties of the widget and set Blend Mode to Transparent.
  2. Uncheck the Gamma Corrected Material property of the UCoherentUIGTWidget

UI Scripting

Binding is the Coherent GT name for the communication between the game and the JavaScript pages. In this document we'll use the terms binding and UI scripting interchangeably. This section covers UE4-specifics for communication between the game and the UI. For a more detailed look into the workings of our binding system, see the binding section in our native docs

Note
UI Scripting can be made with either Blueprints or C++.

There are two major ways to transfer data between your game and the UI. One is via message passing (e.g. calling a UI function from the game) and the other one is data-binding - the ability to expose game objects in the UI and directly access them there.

The following sections cover how to use message passing and data-binding, from both C++ and Blueprints to do your UI Scripting.

Data-binding from Blueprints / C++

The data-binding feature allows you to populate your HTML elements based on the current state of the game. For example, if you want to display the player's health somewhere on the page this can be achieved with a one-liner:

<div data-bind-value="{{player.health}}"></div>

Our system will make sure that anytime the player's health changes, the UI is updated. On the game side, this feature requires you to do 3 things:

  • Tell GT what game objects you'd like to expose. Any such exposed object we call a data model.

From Blueprints call the UCoherentUIGTBaseComponent::CreateDataModelFromStruct / UCoherentUIGTBaseComponent::CreateDataModelFromObject depending on whether you want to expose a UObject instance or a USTRUCT. From C++ the same can be achieved from the Coherent::UIGT::View::CreateModel method:

GTComponent->GetView()->CreateModel("player", Player);
  • Next, whenever something changes, tell GT that it changed

From Blueprints use UCoherentUIGTBaseComponent::UpdateWholeDataModelFromStruct / UCoherentUIGTBaseComponent::UpdateWholeDataModelFromObject. From C++ use Coherent::UIGT::View::UpdateWholeModel:

GTComponent->GetView()->UpdateWholeModel(Player);

C++ code can do more than Blueprints and you can also mark only certain parts of your data model for update instead of the entire model. This can improve performance when you are data-binding complex objects. See the <Coherent/UIGT/View.h> header and our native docs for more info.

  • Calling the update methods from the previous step does not actually update the UI - these methods only mark certain pieces of it as dirty. To avoid constant UI updates, this is done in another method which walks through all elements that need updates and syncs them with the game.

From Blueprints use UCoherentUIGTBaseComponent::SynchronizeModels From C++ use Coherent::UIGT::View::SynchronizeModels`:

GTComponent->GetView()->SynchronizeModels();

A complete Blueprint script may look like this:

databinding_with_blueprints.png
Working with databinding from Blueprints

This was only a very brief introduction to data-binding. For full information about the data-binding feature - e.g. what the other bindings are available (aside from data-bind-value), how does it work under the hood, our complimentary component system and advanced usage - see our native docs at http://coherent-labs.com/Documentation/cpp-gt/.

UI Scripting (message passing) via Blueprints

Coherent GT supports scripting the UI via Blueprints. This allows developers to expose game objects to the UI to populate the interface and trigger actions from the UI like starting a new game when the player clicks "Start game" in a menu.

With Coherent GT you can script the whole UI via Blueprints without any C++ code.

Developers can trigger events in the UI JavaScript code via Blueprints. They can also listen to events triggered in JavaScript and implement game logic on such events. Developers can expose primitive types or any UObject to JavaScript to populate the UI.

In this section we'll introduce a small example that shows two-way communication between Blueprints and the UI JavaScript. The example is available in the Coherent GT sample game.

Triggering Custom Events in Blueprints

Coherent GT supports triggering Custom Events defined in Blueprints. The event must be declared in the Blueprint that contains the UCoherentUIGTComponent component. To use this functionallity, simply right click on the Event Graph and choose Add Event -> Custom Event from the menu. Name the event and add the necessary input arguments, if any. Let's have the event CreateCharacter which takes the character's name as only argument. Here is how the custom event looks in the Blueprint editor:

CustomEvent.jpg
Custom event for character creation

Now you can trigger this event from your UI like this:

engine.trigger('CreateCharacter', 'Joe Monster');

Coherent GT will automatically trigger the custom event with its argument from JavaScript. Please note that currently only primitive types are supported as arguments - bool, byte, int, string, float.

Triggering Events from Blueprints

In this section we'll show an example showing how to send data from a Blueprint to the UI. The first part of the example, will show you how to trigger a simple event from a Blueprint. The second one expands it with exporting whole game objects to the UI.

Create and send a simple event from Blueprint

There are two ways of triggering JavaScript events from a blueprint.

We'll start with the easier one first. It encapsulates all the required logic that you would otherwise have to do yourself with multiple blueprint nodes.

  1. Open a blueprint where you want the event to be triggered.
  2. Right-click and search for "Trigger event". Spawn a "Trigger JavaScript Event" node from the "Coherent UIGT" category.
    bp_script_compact_triggerevent_node.jpg
    Trigger event node in a blueprint
  3. Set the Coherent GT Component Pin. You need to connect the component that will trigger the event.
  4. Set the Event Name Pin. This will be the name of the event that's going to be triggered in JavaScript.
  5. You can dynamically add or remove input argument pins from the details pane, and also give names to the arguments. If you select a currently unsupported type, you'll get a compilation error. At the time of writing, the supported types for binding are primitive types, arrays and UCLASS objects.
    bp_script_compact_triggerevent_detail.jpg
    Trigger event node details pane
  6. Set the argument values either by directly typing the value, or by making links from other nodes.
  7. Make sure that you invoke the "Trigger JavaScript Event" node after you have received the UIGTScriptingReady event.

Here's how the final blueprint looks like:

bp_script_map_jsevent_compact.jpg
Sending an event from JS from Blueprints

The second ways is more verbose and explicit. We'll examine it in the context of the sample game:

  1. Open the Example_Map map of the sample game
  2. Open the Level Blueprint
  3. You'll see the Blueprint already created. You can study it or directly delete it and re-create it better understand what it does.
  4. Add a Begin Play Event
  5. Add a Get Player Controller node.
  6. Add a Get HUD Node from the PlayerController
  7. From the Get HUD node, drag a pin and add a Cast to CoUIGTTestFPSHUD Node.
  8. From the CoUIGTTestFPSHUD node, drag a pin and get its Coherent GT HUD property.
  9. Drag a pin from the Coherent GT HUD object and select Assign UIGT Scripting Ready. The UIGTScriptingReady_Event event will be triggered when the page is ready to receive events. Note that you MUST add call to this event manually to your JavaScript code in the UI AFTER all your initial engine.on event subscriptions.
  10. To create the event: Drag the Coherent GT HUD pin and create a Create JSEvent node. This node produces the object that will represent our event.
  11. Connect Assign to UIGTScriptingReady with Create JS Event. This means that as soon as the page is ready to receive events, we'll send this one.
  12. Drag the Return value pin of the Create JS Event node and create a Sequence node.
  13. Drag a pin from Then 0 of the Sequence node, and create Add String node. The JS event object that we just created can carry an arbitrary amount of arguments that will be available to the JavaScript of the page (each parameter must be added to the event with the appropriate Add XXXX node before the event is triggered; the order by which arguments are added to the event will be the order the JavaScript code receives them).
  14. Connect the return value of Create JSEvent with the target of the Add String node.
  15. Drag the Coherent GT HUD pin and create a Trigger JSEvent node. This is the node that will effectively execute the event and send it to JavaScript. It must happen after all arguments have been added.
  16. From Then 0 in the Sequence node, drag a pin and create Add String node.
  17. Connect Then 1 to the EventData pin of Trigger JSEvent.
bp_script_map_jsevent.png
Sending an event from JS from Blueprints

Automatic export of unreal types to UI scripts

This guide assumes you have completed the first part of the sample, creating and sending a simple JS event, steps 1 to 13.

We'll make available the whole PlayerCharacter object to the UI and we'll show on-screen the player name, the max health she can have and the max ammo she can carry.

In the "hud.html" page you can see that there is a "playerInfo" element that contains the player data we want to show. We'll populate this data from the game through Blueprints.

  1. Add a Get Player Character node.
  2. Drag the Return value pin of the Create JS Event node and create a Add Object node. The JS event object that we just created can carry an arbitrary amount of arguments that will be available to the JavaScript of the page. Each parameter must be added to the event with the appropriate Add XXXX node before the event is triggered. The order by which arguments are added to the event will be the order the JavaScript code receives them.
  3. Connect the Get Player Character return value to the Object pin of the Add Object. This means that our event will send the whole player character object to JavaScript and we'll be able to use it's properties in the UI.
  4. Drag the Coherent GT HUD pin and create a Trigger JS Event node. This is the node that will effectively execute the event and send it to JavaScript. It must happen after all arguments have been added.
  5. Connect the JS event to the EventData pin.
  6. Set the event name (the Name property) to SetPlayerState.
  7. Connect the Then 0 pin of the sequence to the Add Object node and the Then 1 to the Trigger JS event node.
  8. Press Play.

The screenshot below shows the complete Blueprint.

bp_script_map1.png
Sending whole objects to JS from Blueprints

As soon as the game starts and the HUD loads, the SetPlayerStats event will be triggered and the corresponding JavaScript code that populates the Statistics fields in the UI will execute.

In JavaScript we've added a handler for the SetPlayerStats event (see CoUIGTTestFPS/Content/Resources/uiresources/MainUI.html.

engine.on('SetPlayerStats', function (character) {
    $("#playerName").html(character.PlayerName);
    $("#playerMaxHealth").html(character.MaxHealth);
    $("#playerMaxAmmo").html(character.MaxAmmo);

    $("#playerInfo").css("display", "initial");
});
// IMPORTANT: Signal the game we are redy to receive events
// We have setup all JS event listeners (the 'engine.on' calls)
engine.call("UIGTScriptingReady");

The character variable contains all the properties of the APlayerCharacter actor in the game. Now we can use them to populate the "Statistics" of our UI.

Note that when exporting UObjects, only their primitive type UPROPERTY-tagged fields are exported. UObjects contained in exported UObjects are not recursively exported because they might contain circular dependencies and cause memory outages.

The object in JavaScript is a copy of the original one, so changing directly its properties won't affect the object in the game. You can however call events back in the game with parameters and use this mechanism to update logic in the game from the UI. This is explained in the next section.

Calling Blueprints from UI scripts

This section show the other way of communication - from the UI to the game. We'll use the Menu level from the Coherent GT sample game. When the player clicks on the "Start game" button we'll send an event from JavaScript to the game and change the level.

First, let's see what happens in JavaScript when the player clicks Start game:

document.getElementById('play').onclick = function () {
    engine.call('onStartGame', 'Example_Map');
};

The code is pretty self-explanatory - when the 'play' button is clicked call an event in the game called onStartGame. Example_Map is a parameter for this event and is the name of the level to load.

Now lets create the Blueprint that will handle this event and actually load the new level.

  1. Create a new level, called Menu_Map, in the Sample game.
  2. Open the Level Blueprint.
  3. Add a Begin Play Event.
  4. Add a Get Player Controller node.
  5. Add a Get HUD Node.
  6. Add a Cast to CoUIGTTestFPSHUD Node.
  7. Connect the nodes. Now you have the HUD object for this scene.
  8. From the CoUIGTTestFPSHUD object get the Coherent GT HUD View so as to get the Coherent GT HUD property.
  9. Drag a pin from the Coherent GT HUD object and select Assign to UIGTScriptingReady. The UIGTScriptingReady event will be triggered when the page sends an event to the game.
  10. Drag a pin from the JavaScriptEvent_Event node and assign it a Switch on String. The JavaScriptEvent_Event will fire each time a call is made from JavaScript. However, we are interested in only one type of event - onStartGame.
  11. Drag the pin from the JavaScriptEvent_Event Payload property. This is an object of type CoherentUIGTJSPayload that contains the name of the event fired as well as all the parameters passed by JavaScript. Get the Event Name property and connect it to the Switch on String node.
  12. Add a pin to the Switch on String node and set its name to onStartGame.
  13. Create a Sequence node and connect it to the Switch on String. Our sequence will be getting the name of the level passed from JS through the Payload object and then loading a new level with that name.
  14. Drag a pin from the Payload object and create a Get String node. This will be the node that retrieves the name of our level. Coherent GT supports all primitive UE4 types as well as UObjects. Note the index property of this node. It tells the node which parameter to select from the ones passed by JavaScript. You can pass multiple parameters, their order is the same from left to right as passed in JS starting from 0. If we had a second parameter of type "int", we'd have to add a second Get Int 32 node with index = 1.
  15. Add an Open Level node. Connect the pin Return value from the Get String node to the Level Name.
  16. Play.

This is how the final blueprint looks like:

bp_script_menu1.png
JavaScript to BP menu

Remember that Coherent GT supports as many arguments as you need in both directions of scripting and you can send UObjects both ways, too - they work automatically.

UObject scripting

Coherent GT supports automatic export of instances of USTRUCT and UCLASS (e.g. UObject) from the game to the UI. Just send a UObject parameter and all its UProperties will be available in the UI page. Only first level properties are sent, UObjects inside other UObjects are not recursively expanded.

You can also update values in UObjects from JavaScript. To do this create a JavaScript object with the properties you want to change named the same way as in the game. The object must also contain a special property __Type with the name of the C++ type. For instance:

var myCharacter = {
    __Type: 'PlayerCharacter',
    PlayerName: "MyJSPlayer",
    MaxAmmo: 800
};

Now you can send an event to the game with this object:

engine.call("UpdatePlayer", myCharacter);

In the game Blueprint, when you receive the event, you can create a Read Object node from the CoherentUIGTJSPayload object and pass it to the APlayerCharacter of your game. All properties that have the same types and names as the ones passed from JavaScript will be updated. You can do this with any UObject. As long as the names and types of the objects coincide, they'll be updated by the "Read Object" node.

Remarks

Currently returning values from the game to JavaScript events is not supported for Blueprints (it is possible with the C++ interface) but will soon be available.

Objects sent to JavaScript are copies of the ones in the game. Therefore, sending smaller objects less often is preferable as it incurs less resource usage.

As a general note the best way to structure the scripting is to create blueprint functions for every interaction and call them when needed from the master Blueprint. This makes the master Event Graph much less cluttered and more readable.

UI Scripting (message passing) with C++

Coherent GT provides a very powerful binding framework that can be used to connect the C++ logic of the game with the UI JavaScript code and vice-versa.

A detailed guide on Binding is also given in the C++ documentation of Coherent GT.

To use the Coherent GT Binding feature, your page MUST include the coherent.js file.

To trigger events from C++ to JavaScript you can use the TriggerEvent method of the View.

void ACoUIGTTestFPSCharacter::OnTabReleased()
{
    Coherent::UIGT::View* View = GetCoherentUIGTView();
    if (View)
    {
        View->TriggerEvent("ToggleMenu");
    }
}

Any parameters you pass to it will be also sent to the JavaScript code. In JavaScript you must define which function should be called when an event is triggered from C++. The coherent.js file exposes a special JavaScript object named engine that provides the glue and connections between the events in C++ and JS. You must use that object to define all your event handlers. The on method will bind an event to a JS handler.

engine.on('ToggleMenu', toggleMenu);

In this case as soon as C++ calls TriggerEvent("ToggleMenu"), the toggleMenu function will be executed in JavaScript.

Triggering events from JavaScript to C++ is analogous. First, in C++ you need to define which method (or UE4 delegate) will handle a specific event triggered from JS.

Defining C++ handlers must happen after the View has triggered the ReadyForBindings event. If you want to handle this event, you must subscribe to it and in its handler you should declare all your bindings. In the ACoUIGTTestFPSHUD Actor in the sample game this is done with the following line:

CoherentUIGTHUD->ReadyForBindings.AddDynamic(this, &ACoUIGTTestFPSHUD::BindUI);

Now, as soon as the View has been loaded, it will invoke the BindUI method where you can declare all the bindings for the View.

The BindUI method is straight-forward:

void ACoUIGTTestFPSHUD::BindUI(int32 frameid, const FString& path, bool isMain)
{
    CoherentUIGTHUD->GetView()->BindCall("CallFromJavaScript",
        Coherent::UI::MakeHandler(&CalledFromJSSampleDelegate,
        &(FCalledFromJSSample::ExecuteIfBound)));
    CoherentUIGTHUD->GetView()->BindCall("CalledFromJSString",
        Coherent::UI::MakeHandler(this,
        &ACoUIGTTestFPSHUD::CalledFromJSStringHandler));
    CoherentUIGTHUD->GetView()->RegisterForEvent(
        "CalledFromJSUStruct",
        Coherent::UIGT::MakeHandler(this, &ACoUIGTTestFPSHUD::CalledFromJSUStructHandler));
}

Essentially we say: "When JavaScript fires the **CallFromJavaScript** event, in C++ you must execute **CalledFromJSSampleDelegate** delegate". The same applies for the second binding but in that case it'll call the CalledFromJSStringHandler method of the class.

In JavaScript you just have to use:

engine.call("CallFromJavaScript", 123);
engine.call("CalledFromJSString", "Hello from JavaScript!");
var ustructData = { ... };
engine.trigger("CalledFromJSUStruct", ustructData);

Please note that you can pass arguments from JS to C++ again but the handler signatures must coincide with the arguments passed, otherwise an error is generated.

In the sample game the handlers for those methods are very simple:

void ACoUIGTTestFPSHUD::CalledFromJSHandler(int32 number)
{
    UE_LOG(LogScript, Log, TEXT("UE4 Delegate called from JavaScript"));
}

void ACoUIGTTestFPSHUD::CalledFromJSStringHandler(const FString& str)
{
    UE_LOG(LogScript, Log, TEXT("String received from JS: %s"), *str);
}

We just log that we were called from JS, so that we know everything works fine. Coherent GT supports passing parameters to/from JS for all primitive types, STL containers as well as UE4's strings (FString, FText, FName), containers (TMap, TArray) and colors & vectors. In addition, any type that UE4 knows about i.e. USTRUCTs and UCLASSs can be automatically bound. In order to use these types you must include the relevant headers available in the Coherent/UIGT/Binding and CoherentUIGTPlugin/Public folders.

Automatic binding for UCLASS and USTRUCT types

To enable the automatic binding support for Unreal Engine types you have to:

#include "CoherentUIGTUTypeBinder.h"

Please note that nested USTRUCT types are going to be recursively exposed as well:

USTRUCT()
struct FBar
{
    int32 Value;
};
USTRUCT()
struct FFoo
{
    FString String; // Will be exposed; strings are considered primitive types
    TArray<int32> SomeValues; // Will be exposed; containers of primitive types are exposed.
    // Will NOT export nested types unless they are of type USTRUCT.
    FBar Bar; // Will be exposed as it is USTRUCT.
};
Warning
Your property names can end up being automatically exposed to JavaScript with a different casing than the original UPROPERTY. To work around this, you can force lowercase exposure for all properties.

Nested USTRUCT types will be exposed recursively by default. Note that nested UObjects are currently not bound recursively as their sheer size makes recursive binding unusable for performance reasons (a recursive binding to an AActor for example would also need to bind the entire ULevel as the actor holds a pointer to it).

Coherent GT also has support for binding user defined types that don't fit the categories above. For a detailed guide on this topic, please consult the native Coherent GT C++ documentation.

Casing in automatically exposed UCLASS and USTRUCT types

For instance, if we have a class Foo and a property called Bar, like the example above, JavaScript might receive a value such as Foo.bar or Foo.bAr. For this to occur, there needs to be an FName in the Engine/Game that shares its content with the name of your property, but has a different casing. This usually occurs only in shipping builds and happens because in Unreal Engine properties are stored as FNames, which are case-insensitive. There is no supported way in UE4 to know the casing in advance, but you can work around this by turning on the Use Lower Case Names For Auto Exposed Properties in Coherent GT Plugin's miscellaneous settings, which will guarantee that property names will be exposed as lowercase, regardless of their casing in the engine.

gt_automatic_binding_setting.png
Lowercase names in automatically exposed properties

Preserving the State of the Coherent GT View between levels

If you want to keep the HUD or UMG Widget, Player Character and any other actor alive while changing levels, we strongly recommend using Seamless Travel.

If however, that is not an option for you for whatever reason, yet you want to keep the Coherent GT ViewState inside the HUD or UMG Widget, you can achieve that by binding functions to the delegates listed below.

HUD's UCoherentUIGTBaseComponent ViewState delegates:

UCoherentUIGTWidget ViewState delegates:

These delegates are part of the public API of the HUD's UCoherentUIGTBaseComponent and UCoherentUIGTWidget, and they will let you preserve and reuse the already created Coherent GT View and System, while recreating the HUD or UMG Widget.

Some of these delegates are going to be executed early on in the initialization of the component or in the recreation of the widget, and there might not be an easy way of getting the HUD or Widget that early to bind them. We suggest creating your own HUD actor or UMG widget class, inherited from ACoherentUIGTGameHUD or UCoherentUIGTWidget respectively and bind delegates in the constructor. This will also help in reusing the ViewState only on the same type of HUD actor or UMG Widget.

Warning
This should be used with extreme caution - if you are using Coherent binding or databinding to expose methods or models (BindCall, RegisterForEvent, RegisterModel), you should also make sure they are setup in such a way that they will outlive the Level change, or at least Unregister and Register them again. If this is not done correctly, crashes are bound to happen!

You can check the implementation of the ACoUIGTKeepViewStateHUD and UCoUIGTKeepViewStateWidget, that are included in the CoUIGTTestFPS sample game as an example of how this can be setup.

Async mode

If you are running into performance problems, you can make use of Coherent UI GT's asynchronous mode. Enabling this mode, moves most of the computations into a separate thread and effectively reduces the overhead of your UI almost to zero. Multithreading is an advanced feature and using it requires understanding of the possible consequences such as race conditions. Coherent UI GT provides some solutions for these problems and does its best to hide any differences with the synchronous mode.

To enable the asynchronous mode go to Coherent GT -> Options and check Run Asynchronous. Note that you must restart the editor in order for the effect to take change. This is because of a limitation of WebKit that cannot be avoided.

To see the async mode in action, load the AsyncSample_Map provided in the CoUIGTTestFPS sample game. The map shows how to properly setup the communication between JavaScript and the game by featuring a simple transition from a main menu to in-game menu and ability for the UI to ask the game for new data. As noted above, you must restart the editor in order to try out the map with both modes.

See Asynchronous mode usage for further details.

In async mode calls to Coherent GT will not complete immediately. Consider the following code:

CoherentHUD->TriggerEvent("ShowEULA"); // Show the EULA in a modal window
// ..
CoherentHUD->DrawHUD();

The TriggerEvent call will put a task on underlying task queue and will get executed whenever the UI thread can do so. This means that there's a chance that the EULA will not actually show before the DrawHUD. Most of the time this will not be a problem as Coherent GT makes sure to synchronize the main and UI threads during UCoherentUIGTBaseComponent::Tick and ACoherentUIGTSystem::Tick but if you absolutely need to manually synchronize the threads we provide a fencing API akin to the UE4's rendering fences.

The relevant methods are ACoherentUIGTSystem::Fence and ACoherentUIGTSystem::WaitFor and can be used like so:

CoherentHUD->TriggerEvent("ShowEULA"); // Show the EULA in a modal window
auto FenceId = CoherentSystemActor->Fence(); // Issue a fence immediatly after the trigger event
// ..
CoherentSystemActor->WaitFor(FenceId); // Make sure the trigger event has been completed
CoherentHUD->DrawHUD();

ACoherentUIGTSystem::Fence() and ACoherentUIGTSystem::WaitFor(unsigned) do nothing when invoked on a synchronous system so feel safe to use them without changing any of your code.

Communication with JavaScript

Asynchronicty implies that the JavaScript execution is also deferred on the UI thread. This may require some changes if your existing code depends on the fact that up to now, callbacks were executed in the same stack.

Consider the following communication:

C++:

void Bar()
{
    // Code
}

// During initialization
CoherentComponent->GetView()->BindCall("bar", Bar);

// During your main loop
CoherentComponent->TriggerJSEvent("foo", nullptr);

JS:

engine.on("foo", function fooHandler() {
    // Code
    engine.trigger("bar");
});

Calling view->TriggerEvent("foo") in the synchronous mode will execute the methods in the following simplified callstack:

Callstack
Bar
JavaScript::fooHandler
CoherentUIGTBaseComponent::TriggerJSEvent
GameLoop

In the async mode you get no such guarantee. fooHandler will be executed whenever the UI thread gets to it and Bar will be executed during the first View::Layout or View::ExecuteJSTimers after fooHandler has completed. The following diagram demonstrates that:

async_vs_sync_ue4.png
Async vs Sync execution

Remove implicit dependencies on synchronous mode

Let's look at a more concrete example

C++:

void StartGame()
{
    // Initialize the game
}

// During initialization
CoherentUIGTHUD->GetView()->BindCall("StartGame", StartGame);

or the same code in Blueprints:

async_incorrect.png
Faulty BP asynchronous code

JS:

startButton.addEventListener("click", function () {
    engine.trigger("StartGame");
    doSomethingAssumingTheGameHasStarted();
});

If you are running in a synchronous mode, this code will work fine. When the button is clicked, the engine will call StartGame immediately, which will in turn initialize the game. Thus, by the time engine.call("StartGame") returns you can assume that everything is ready.

This assumption does not hold in asynchronous mode as the execution of StartGame will be delayed. To cope with this, use another event to notify the JS that everything is ready:

C++:

void StartGame()
{
    // Initialize the game
    CoherentHUD->TriggerJSEvent("GameStarted", nullptr);
}

// During initialization
CoherentUIGTHUD->GetView()->BindCall("StartGame", StartGame);

or the same code in Blueprints:

async_correct.png
Correct BP asynchronous code

JS:

startButton.addEventListener("click", function () {
    engine.call("StartGame");
});

engine.on("GameStarted", function () {
    doSomethingAssumingTheGameHasStarted();
});

Usage Summary

  • If your code depends on the assumption that Coherent GT works synchronously, small changes to it will be required.
  • Working with the asynchronous mode resembles the model that most web applications use. You might think about the game as the server and your UI as its client
  • Multithreading requires thorough understanding of the threading model in order to minimize the errors and maximize the profits.

Coherent GT System

To use Coherent GT during gameplay a UISystem object has to be created and live somewhere. The plugin automatically takes care of the creation and management of the UISystem, using a default set of options. This is done by spawning a ACoherentUIGTSystem Actor in the world that will set everything up.

Using custom UI System

If you would like to customize the UISystem properties, you can spawn a ACoherentUIGTSystem Actor of your own and edit its properties in the Defaults tab of the Blueprint Editor. The UISystem properties define many aspects of the runtime such as to whether support cookies, the UI Debugger port etc. If you don't add a ACoherentUIGTSystem Actor, then a default one will be created as soon as you try to create a View.

Note
You can have only one UISystem at a time.

The easiest way to spawn a custom UISystem is to drag the CoUIGTSystemBlueprint blueprint into the world in editor mode and then edit its settings by clicking the Edit CoUIGTSystemBlueprint in the Scene Outliner window.

Coherent GT behaviour when the UE4 engine is paused

Sometimes it may be the case that the UI should be paused when the UE4 engine is paused. There is an option for that, which requires using a custom Coherent UISystem (Refer to the Using custom UI System section for details).

Note: The default value of the UpdateWhenPaused option is set to true. To change it:

  1. Create an instance of the CoUIGTSystemBlueprint in the world.
  2. Locate the Scene Outliner tab in the UE4 editor and find the instance of CoUIGTSystemBlueprint.
  3. Click Edit CoUIGTSystemBlueprint.
  4. Uncheck the UpdateWhenPaused option in System Settings section.

Console Commands

Coherent GT provides several console commands that can help during development. The following commands are supported:

  • cohgt.list - lists the id and document of all views
  • cohgt.reload [<id>] - reload the view with the specified <id> or all views if no <id> is specified
  • cohgt.redirect [<id>] <url> - change the page of the view with the specified <id> or all views if no <id> is specified. The <url> must be surrounded by double quotes (e.g. "http://coherent-labs.com")
  • cohgt.debug [<id>] - toggles paint rectangles on the view with the specified <id> or all views if no <id> is specified
  • cohgt.track [<id>] - toggles whether the view with the specified <id> is tracked during stat CoherentGT. Only tracked views are profiled with stat CoherentGT. If no views are explicitly profiled, we profile all views. Don't pass <id> to reset all tracking information.
console_commands_uigt.png
Console commands and profiling

Coherent GT Debugger

Coherent GT comes with a powerful tool for live-debugging and editing your UI. To start the debugger, go to the Coherent GT menu in the editor, and choose Launch Debugger.

Inspector in detail

It is advisable to use the debugging functionality only during development. Leaving it active on a shipped product leaves the UI accessible to users. This might be a desired feature as it aids UI modding but should be something to keep in mind when deploying a final product. The debugger is implemented in the CoherentUIGTDevelopment library. During development you should put the library in the same folder as the main CoherentUIGT module. If the library is missing, the debugging functionality will not be enabled. You can safely remove CoherentUIGTDevelopment from your shipped product.

The Inspector works over the network, so you can attach it either to you local machine or to a remote one. The debugging port is selected in the Coherent::UIGT::SystemSettings::DebuggerPort field. If you pass -1, debugging will be disabled.

Debugging is done live, so start your application and have a Coherent GT running.

Use the menu "Coherent GT -> Launch Inspector" to run the Inspector in the Editor. The Inspector can only be started if your game is not currently running. You can also keep it running and debug other running games, provided that the ports don't collide.

You'll be presented with a list of Coherent GT Views that can be explored, debugged and edited.

Using the debugger

The debugger allows live viewing/editing of CSS styles, JavaScript debugging (setting breakpoints, watches, editing variables), resource load inspection, profiling.

The tool uses the WebKit Inspector UI, so for more information please refer to: Apple's Introduction to the Inspector

Changing elements live

With the Inspector you can manually change all aspects of the elements within the page as well as adding/removing whole sections of it. This makes prototyping and iterating on the UI very easy. Go to the Elements tab. The page with all its current state will be shown. You can hover elements and they will be highlighted in the View in the application. All properties of the elements are editable and changes will be immediately mirrored in the live application.

Debugging JavaScript

A complete JavaScript debugger is included in the Inspector. To use it go to the Sources tab, click Enable Debugging. On the left side a list of the loaded JS sources is given. Users can add breakpoints, watches, step through expressions and catch exceptions.

Performance considerations

A key feature of Coherent GT that makes it generally faster than alternatives like the UE4 bundled UMG, is that it re-draws just parts of the screen and UI layers that have changed. Users are encouraged to follow our guidelines to get the best from the toolkit.

A detailed performance guide comes bundled in the package alongside this one. Please check it out when you start UI development to get useful tips and tricks for developing optimal UI.

Performance warnings

By default, in the UE4 Editor, Coherent GT will generate performance warnings when it detects issues. The warnings will not be fired in shipped games.

Warnings help developers notice eventual inefficiencies as early as possible. They will be printed both in the Output Log and on-screen. The warning thresholds are arbitrary and can be modified any specific scenarios. Warnings can be disabled altogether form the menu "Coherent GT -> Options -> Allow Performance Warnings In Editor". You can disable on-screen warnings, but leave them in the log by un-checking the "Show Warnings On Screen" option in the same menu.

Warning can be customized per-View. You can disable warnings and change thresholds for a View in the "Defaults" section of the View component, using the "View warnings" category of options.

For HUD Views you can modify the thresholds or disable the warnings by modifying the properties in the component via C++ code or Blueprints.

When you receive a performance warning, you can use the Inspector to profile eventual inefficiencies in the interface. To use the Inspector just click "Coherent GT -> Launch Inspector". Please consult also the "Performance guide" that gives plenty of information about the characteristics of different workloads and tips on how to optimize the UI. The guide is accessible by clicking "Coherent GT -> Open Performance Guide".

Performance statistics

Coherent GT's heaviest computations are tracked using UE4's stat subsystem. You can use the usual Unreal's toolset to track down problems:

  • stat CoherentGT will display information about the time spent in each major function:
    • Total CPU usage
    • CPU usage for system advance
    • CPU usage for JS execution
    • CPU usage for page layout
    • CPU usage for rendering commands recording
    • CPU usage for painting
  • stat CoherentGTCounters will display information about GT's total CPU memory usage
  • stat CoherentRenderingCounters will display the number of draw calls and rendering commands. It will also display information about the total GPU memory usage:
    • Number of draw calls
    • Number of rendering commands
    • Total CPU memory used
    • Total GPU memory used
    • Total GPU memory for textures
    • Total GPU memory for depth-stencil textures
    • Total GPU memory for vertex buffers
    • Total GPU memory for index buffers
  • If you want to profile a single view, use the command cohgt.track. See Console Commands section for more information.
  • You can also record data with the Sessiont frontend. Coherent GT's computations will appear there as well.
  • To see rendering stats run profilegpu or stat GPU (the latter being available only on UE4 4.14+)

Performance auditing

In addition to the warnings, Coherent GT can also audit HTML pages and check for suboptimal CSS and HTML usage. To launch the performance auditor, go to Coherent GT -> Launch Performance Auditor.

performance_auditor.png
Coherent GT's Performance Auditor

You'll be presented with all views currently loaded in the level (be it components or UMG widgets). Clicking any of the Audit buttons will trigger a performance audit for the selected view. After the audit is completed, a messagebox will display the results. Closing the messagebox will also trigger a pulsating animation for several seconds on all faulty HTML elements so that they can be easily seen.

performance_audit_complete.png
Audit Completed Report

The auditor has some limitations:

  1. The list of views is not refreshed if any of the views is destroyed or a new one is created.
  2. Attempting to audit a destroyed view will result in an error message.
  3. Attempting to audit a view that has does not include coherent.js will result in an error message
  4. To audit a newly created view, you have to relaunch the auditor.

Live Views (Picture-in-picture)

Coherent GT supports a powerful feature called Live Views. With it developers can embed content rendered by Unreal Engine 4 in their UI. The feature is ideal for 3D player portraits, equipment screens, rear view mirrors in racing games but also advanced effects like adding particle effects or complex 3D content inside the interface.

A complete Live View sample can be seen in the "LiveView_Map" world that comes in the CoUIGTTestFPS game distributed in the package.

Setting up and using Live Views in Unreal Engine 4 is very easy. The Live View is represented by a specially set "img" tag inside the HTML page of the UI.

<img id="myLiveView" src="TextureRenderTarget2D\'/Game/LiveViews/LiveViewRT.LiveViewRT\'">

The image source is a reference to an Unreal Render Target asset. To create such an asset:

  • In the Editor, right click and select Materials & Textures->Render Target
  • Set the texture's size
  • Disable HDR
  • Set the gamma value of the texture to 2.2(GT does no gamma correction)
live_view_rt.png
Set the render target properties
  • Save the asset
  • Right click on it and select Copy Reference
  • Paste the link as source for the <img> tag

There are many ways to draw in the Render Target, the simplest one is to use an Unreal Engine 4 Camera with a SceneCaptureComponent2D component. The camera will render what it captures in the world in the Render Target that will then be rendered in the UI by Coherent GT.

In the Content Browser add a new Blueprint and set its parent to "CameraActor". Edit the Blueprint and add a "SceneCaptureComponent2D". Then set the "Texture Target" to the one that will be used by the Live View and set the "Capture Source" to "Final Color".

live_view_camera.png
Setup a camera for the Live View

Now when you Play your level, everything that is rendered by the camera will be available in the HTML UI page. The Live View inside the interface will work the same as a normal image (but will be automatically updated) and content can be drawn over it, it can be animated, scaled etc.

Preloaded textures

Usually Coherent GT would read from disk, decode and upload to the GPU all textures it is using whenever a texture is needed. There are use cases when this would not be the optimal thing to do. For example:

  • The game and the UI share textures (this will result in double processing)
  • You don't want to keep the raw images around, just the imported .uasset files.

The loading of textures happens when Coherent GT requests them which invokes the asynchronous loading of the uassets. This will happen when the whole HTML page is parsed and its resources are requested, which includes the preloaded images/live views present in the page declaration. UE4 will not load them into memory in advance when Coherent GT is initialized.

Once the HTML page is parsed and the preloaded resources are requested, their loading will be handled by the FStreamableManager class. The FStreamableManager implementation in Unreal Engine could load the texture instantly in case the .uasset is already loaded into memory, otherwise it will be streamed from disk.

Note
FStreamableManager will be used in versions of UE 4.16 and above. Prior to UE 4.16, uassets loading is done via LoadObject, because part of the API is not available in these earlier versions.

To use preloaded images, use UE4 object paths to reference your images from HTML / CSS instead of standard urls e.g. instead of this:

<img src="coui://uiresources/foo.png"></img>

use this

<img src="coui://Texture2D'/Game/uiresources/foo.foo'"></img>

To get the object path for your texture you can right-click it in the editor's content browser and click Copy Reference, then paste the result in your HTML.

Note that system cannot freely preload all images because there may be visual differences between using images the usual way and having them preloaded. This is due to the fact that Coherent GT has some expectations on the way images are treated. To make sure that there are no visual differences follow these guidelines:

  • Coherent GT expects images to have premultiplied alpha, UE4 does not. Transparent images will cause the transparent areas to receive a non-transparent color unless premultiplied.
  • By default UE4 stores images in sRGB (i.e. it applies gamma-correction). This will result in textures showing darker in some parts. To prevent this issue, make sure that the sRGB property of the texture asset is checked off.
  • Make sure you select the proper texture compression for your texture. For example, DXT is a lossy format and this may cause artefacts which may or may not be acceptable for your use case.

If an image cannot be preloaded, the system will attempt to decode it from the raw file. Check the PreloadedImages_Map level to see this feature in action.