 |
Coherent UI for Unreal Engine 4
2.5.5.0
A modern user interface library for games
|
|
Coherent UI for Unreal Engine 4
2.5.5.0
A modern user interface library for games
|
This page will introduce you on how to set-up and use the Coherent UI middleware library with Unreal Engine 4.
Coherent UI is a modern HTML5-based user interface middleware library for realtime applications. It has been pre-integrated with Unreal Engine 4 and all the source of the integration plugin is available to Coherent UI and UE4 licensees.
Coherent UI is based on browser-like technology and can also be used as an in-game browser. It can load and render any page from the web and supports all modern authentication methods and protocols.
This guide requires basic knowledge of both Coherent UI and Unreal Engine 4. For more info on the discussed topics, please refer to the Coherent UI Documentation and the Unreal Engine Documentation.
To quickly try out Coherent UI with the provided sample - try the provided CoUITestFPS game.
After the Installer has completed, it'll add the game to the UE4 solution in Visual Studio.
If you try to directly run the game from VS before having it previously opened, you might receive an error Failed to open descriptor file.
The game shows a simple menu, HUD and if you press "I", it'll pop-up an in-game inventory in 3D. If you press "Tab" you'll be presented with a simple in-game menu.
The Coherent UI integration is divided in two main parts: a "ThirdParty" dependency for UE4 that is the Coherent UI C++ DLL itself along with it's dependencies and an Engine Plugin that provides the "glue" between UE4 and Coherent UI. This plugin exposes several classes and Components specific for the development with UE4 and takes care of all the initialization and management of Coherent UI.
Coherent UI has to be added as a "ThirdParty" dependency to UE4. This will make sure that all DLL are loaded when the game starts and they will be referenced by the engine correctly. Currently, Coherent UI uses 2 plugins, one for the actual gameplay code, and another Engine plugin that sets up the dependency to the core Coherent UI library.
The Coherent UI installator will make sure that both the Coherent UI core plugin and the gameplay plugins are installed correctly.
#There are two ways to add the Coherent UI plugin to your game. Automatic and Manual:
option and then activate the plugin from the editor with the following steps:
PrivateDependencyModuleNames.AddRange(
new string[] { "CoherentUIPlugin", "CoherentUI" }
);
Note: In case than you can not activate the plugin from the editor, you can also activate it by addiding the following lines in the .uproject file of your game.
"Plugins": [
{
"Name": "CoherentUIPlugin",
"Enabled": true
}
]
This will tell the engine that it should load the compiled CoherentUIPlugin dll if your project is using Coherent UI.
Note: CoherentUI will look for resources loaded through custom protocols (such as coui://) in "YourGame/Content/UIResources" folder.
These are almost the same steps as in the previous section but generating project files will fail if your game has no C++ code.
Open the editor and enable the Coherent UI plugin for your game using the Window -> Plugins dialog. Coherent UI plugin is in the Installed / User Interface section.
Note: Without C++ code, you won't be able to distribute your game. See the next section for details.
UE4 currently doesn't build plugins in blueprint-only projects, thus you can't use CoherentUI (or any other runtime plugin) without any C++ code. You can fix this by adding a simple dummy C++ class to your project which is described in the next paragraph. If your game already has C++ code, skip the next section.
Note: Previous CoherentUI versions loaded UIResources from the root of your game directory (as opposed to "YourGameDir\Content"). That behaviour is still supported but it is deprecated and will be removed in future versions. If you are still using that behaviour, you are required to manually copy the "UIResources" folder to "YourPackageDir/WindowsNoEditor/YourGameDir" (next to "Content" and "Binary").
Your package directory structure should resemble this:
|-YourPackageDir
|-WindowsNoEditor
|-Engine
|-Build
|-Binaries
|-ThirdParty
|-CoherentUI
|-Win64
|-host
|-<any_other_platform>
|-host
|-YourGameDir
|-Saved
|-Content
|-UIResources
|-Binaries
|-Win64
|-CoherentUI64.dll
|-YourGame.exe
|-Win32
|-CoherentUI.dll
|-YourGame.exe
|-Mac
|-libCoherentUI.dylib
|-YourGame.exe
|-<any_other_platform>
|-<CoherentUI client library>
|-YourGame.exe
The current plugin allows to easily add Coherent UI Views to UE4 HUDs as well as surface on the world - that is having UI Views as textures on objects.
All exported Components inherit the "CoherentUIViewListener" class. It takes care to listen manage the creation/destruction of Coherent UI View resources and textures and listens to View events. Some of the View events are also broad-casted through Unreal Engine Delegates. Almost all interactions can be achieved through the Unreal Editor and the Blueprint Editor, and all View properties are exposed to them.
Coherent UI comes to Unreal with an editor plugin that eases your workflow. The next section describes the editor plugin. For detailed information about each feature scroll past the section.
Coherent UI's Editor Plugin adds pulldown on the main toolbar in your Unreal Editor, to the right of the Window menu.
Changes the GameMode's HUD class to the one provided with Coherent UI - CoherentUIGameHUD and adds a HTML file named hud.html to YourGame/Content/UIResources.
This item is only available if the current default GameMode is a Blueprint class. To see or change your current default GameMode, go to Edit -> Project Settings -> Maps and Modes -> Selected GameMode. GameMode classes written in C++ can't be changed from the editor which is why to use this option you would need to create a new blueprint that inherits from AGameMode. For further information, please refer to the UE4 docs.
By default, CoherentUIGameHUD will try to load hud.html (and use several others sensible default settings). If you'd like to customize the HUD's settings, you can do so with the following Blueprint:
It's important to call SetupUIView as shown in the image above during Event Begin Play or otherwise the HUD will fallback to its default settings.
Adds a 3D Plane with a CoherentUIComponent to the scene, facing current camera's position. The plane is an instance of CoherentUIPlane - an asset of ours that will be copied to YourGame/Content if it's missing.
Adds a HTML page to YourGame/Content/UIResources and launches it in your browser.
Launches Coherent UI GT's Inspector. You can change the path to the debugger in the settings (see below).
This item is only available if the game is currently running in PIE.
Opens Coherent UI's documentation on http://coherent-labs.com.
Opens Coherent UI's tutorials on http://coherent-labs.com.
Opens Coherent UI's settings (you can also access them from Edit -> Project Settings -> Plugins -> Coherent UI. This enables you to change the Debugger's port and path.
At the core of Coherent UI stays the so called View. A Coherent UI View is basically an HTML5 page and the JavaScript context for it. Coherent UI will render the View, execute the JavaScript and provide UE4 with a texture with what it has drawn.
Views provide methods to change the current page, use local resources, interact with JavaScript. In UE4 you'll seldom have to use them directly as they are encapsulated in Components that can be manipulated in code or Blueprint.
The Coherent UI UE4 plugin has two main Components - UCoherentUIHUD and UCoherentUIComponent and an actor - ACoherentUISystem. This document introduces in detail all of them. There's also the UCoherentUILiveViewComponent component which allows easy transfer of image data from UE4 into your HTML page.
All Coherent UI UE4 Components inherit from the UCoherentBaseComponent Component. It should not be used directly by itself. It exports all the most important Coherent UI properties and events to the UE4 Engine and Blueprint Editor.
Coherent UI renders all pages out-of-process and hence needs some time to start-up. All Components provide an important IsReadyToCreateView method. You shouldn't create the Components View before it is ready to do so.
The system actor takes care of properties that are global for all the Components. If you don't create one, a default will be created in the game for you.
The "CoherentUIHUD" component can be used to easily add Coherent UI Views to a game HUD. A sample Game accompanies this code and details can be seen there. The Component must be added to the HUD Actor of the game and than the HUD is in charge of creating it's View and must call the "CoherentUIHUD::DrawHUD" method when inside it's own "DrawHUD" method. Details can be seen in the CoUITestFPSHUD.h and CoUITestFPSHUD.cpp files.
The UCoherentUIHUD class is a Component that takes care of loading the HTML page with the HUD interface and can draw it in the Canvas of the HUD Actor. In the sample the CoherentUIHUD Component is created and added to the ACoUITestFPSHUD Actor with the following code:
CoherentUIHUD = PCIP.CreateDefaultSubobject<UCoherentUIHUD>(this, TEXT("CoherentUIHUD"));
Then on every DrawHUD event the Coherent HUD is checked for readiness and eventually draw the HUD.
if (CoherentUIHUD)
{
if (CoherentUIHUD->IsReadyToCreateView() && !CoherentUIHUD->HasRequestedView())
{
CreateView();
}
// check that the size matches
auto view = CoherentUIHUD->GetView();
if (view)
{
if (view->GetWidth() != Canvas->ClipX
|| view->GetHeight() != Canvas->ClipY)
{
CoherentUIHUD->Resize(Canvas->ClipX, Canvas->ClipY);
}
}
CoherentUIHUD->DrawHUD(Canvas, 0, 0);
}
In the snippet above you can see that the Component is checked for readiness and if it has already requested the creation of a View for itself. It it hasn't, it'll call the CreateView method. If we already have a View we check if it has to be resized (for instance after we've resized the game window) and calls the DrawHUD method with the current Canvas and the coordinates on the screen where we want our View to draw itself.
The following code will request the creation of a View:
void ACoUITestFPSHUD::CreateView()
{
Coherent::UI::ViewInfo info;
info.Width = Canvas->ClipX;
info.Height = Canvas->ClipY;
info.TargetFrameRate = 60;
info.UsesSharedMemory = true;
info.IsTransparent = true;
info.SupportClickThrough = true;
CoherentUIHUD->ReadyForBindings.AddDynamic(this, &ACoUITestFPSHUD::BindUI);
CoherentUIHUD->CreateHUDView(info, TEXT("coui://UIResources/HUD/hud.html"));
}
All properties are self-explanatory - for more detail on each of them please consult the Coherent UI C++ documentation.
The line
CoherentUIHUD->CreateHUDView(info, TEXT("coui://UIResources/HUD/hud.html"));
is the one that actually initiates the creation of the View. The "coui://UIResources/HUD/hud.html" page is the initial page that will be loaded. Note that coui is a special protocol that denotes local resources in the game. coui is read through a FCoherentFileHandler that for UE4 can be found in the CoUISystemHolder.cpp file. It will take care to read all files relative to the game's current content directory. Coherent UI supports all standard protocols too, so you can load pages from the Internet.
The more interesting snippet above is the line:
CoherentUIHUD->ReadyForBindings.AddDynamic(this, &ACoUITestFPSHUD::BindUI);
It binds a method that will be called when the View is ready to accept its bindings. Binding is the facility that allows the communication between C++ and JavaScript code in the page. For more information please refer to the Scripting section of this document.
There are 2 ways to add a HUD View using blueprints.
You'll need a game mode override in the Editor's World Settings which uses the CoherentUIGameHUD HUD class.
If you don't initialize the HUD, Coherent UI will try to load your hud from coui://uiresources/hud.html and use default view settings.
If you have a custom HUD class that you want to use instead of the CoherentUIGameHUD one, you can use this option.
Start by creating a new game mode (e.g. "MyGameMode"), then create a new blueprint whose parent class is the HUD class you want to use. In the following example we use the generic HUD as a parent for blueprint:
Finally, just draw the texture of the Coherent UI view in the Draw HUD event:
Coherent UI Views can be used as textures on in-game objects to create interactive displays, call-outs and many other interesting effects. The component used is called "CoherentUIComponent". The steps to add a View to an object in the world are straightforward.
Now you can use the Blueprint and add it in the world. The UI Component will automatically update the "UITexture" parameter with the texture of the View.
A material setup for use by Coherent UI can be found in the sample game and is named CoherentUIMaterial. In the sample game please check the CoUIPlane and the StaticActorCoherent objects that have the material setup and a "CoherentUIComponent" and can show pages on the in-world. The StaticActorCoherent is a cube that shows web pages on all its faces.
The properties of each View can be edited in the Bluperint editor in the Components menu under the "CoherentUIComponent" View section.
Live Game Views are Coherent UI's high performance way to transfer image data from your game engine into a HTML canvas in your UI.
To use them in an actor/blueprint, you need to do the following:
The Live Game View has the following properties:
TickComponent function. If set to false, you'll need to call UCoherentUILiveViewComponent::QueueUpdateLiveViewOnRenderThread yourself whenever you want to queue an update.In the HTML there's one requirement: you need to have coherent.js in the page. Whenever UE4 sends texture data, you'll receive the onEngineImageDataUpdate callback on your <canvas> objects with two arguments - the name of the link and the ImageData object. The following sample demonstrates placing the received data in the canvas:
<!DOCTYPE html>
<html lang="en">
<head>
<script type="text/javascript" src="js/jquery-2.0.3.min.js"></script>
<script type="text/javascript" src="js/coherent.js"></script>
</head>
<body>
<div id="live-view-container">
<canvas id="live-view-canvas" width="384" height="384"
style="border:1px solid #d3d3d3; ">
Your browser does not support the HTML5 canvas tag.
</canvas>
</div>
<script type="text/javascript">
$(document).ready(function() {
var c = document.getElementById("live-view-canvas");
c.onEngineImageDataUpdated = function (linkName, image) {
// If you have more than 1 live view in your page,
// you'll need to check the name of the live view link
var ctx = c.getContext("2d");
// It's important to clear the canvas before drawing
ctx.clearRect(0, 0, c.width, c.height);
ctx.putImageData(image, 0, 0);
}
});
</script>
</body>
</html>
After you have added a Live Game View to an actor, you can specify the behaviour of the View during a pause. There is an option called UpdateWhenPaused in the properties of the LiveViewComponent which, depending on whether it is set or not, will update the Live View.
To use Coherent UI during gameplay a UISystem object has to be created and live somewhere. The plugin automatically takes care of the creation and management of the UISystem, using a default set of options. This is done by spawning a ACoherentUISystem Actor in the world that will set everything up.
If you would like to customize the UI System properties, you can spawn a ACoherentUISystem Actor of your own and edit its properties in the Defaults tab of the Blueprint Editor. The System properties define many aspects of the runtime such as to whether support cookies, the UI Debugger port etc. If you don't add a ACoherentUISystem Actor, then a default one will be created as soon as you try to create a View.
Note: The easiest way to spawn a Custom UI System is to drag the "CoUISystemBlueprint" blueprint into the world in editor mode and then edit its settings by clicking the "Edit CoUISystemBlueprint" in the Scene Outliner window.
Sometimes it may be the case that the UI should be paused when the UE4 engine is paused. There is an option for that, which requires using a custom Coherent UISystem (Refer to the Using custom UI System section for details).
Note: The default value of the UpdateWhenPaused option is set true. To change it:
The input in Coherent UI is managed through a Slate widget (SCoherentUIInputForward). This widget is an empty widget that covers the whole screen and forwards input to a Coherent UI View. For simplicity we have created the ACoherentUIInputActor actor, which encapsulates the logic behind the widget and is accessible through both C++ and blueprints. All you have to do is spawn the actor and call its initialize method.
Keyboard focus, in UE4 terms, means that the Coherent UI Input Forward widget has keyboard focus. When the widget has keyboard focus it is able to receive keyboard events, otherwise it is not.
Mouse events don't require focus and they are propagated to the widgets under the cursor until one of the widgets handles the event. The Coherent UI Input Forward widget is generally on the top of the viewport so it can process mouse events first. The processing in the widget works as follows:
Joystick input works by default when the viewport widget has focus. Keyboard focus implies joystick focus, which means that if the Coherent UI widget has keyboard focus, the joystick events will stop working. If that's not intended, the input actor has a method for keeping joystick focus to the viewport after setting keyboard focus.
Each view can be focused, regardless of the widget focus in UE4. This is core functionality and it's independent of any other focus mechanics. The view focus controls the color of selected text, whether you have a blinking caret in input fields and the like.
Here are a few examples for views that have focus (left) and ones that don't (right):
The current implementation of the input actor/widget allow for only one focused view at a time. If you need some custom behaviour you'll have to modify the plugin's code.
Each view component has a property (Receive Input) that controls whether a specific view takes part in the mouse event forwarding.
The input in Coherent UI is managed through a Slate widget (SCoherentInputForward). This widget is an empty widget that covers the whole screen and forwards input to a Coherent UI View. The widget should be added in the front of the viewport so it can process mouse and keyboard messages first. Due to Unreal Engine 4's architecture, the widget can process keyboard messages only if it's focused. To focus the widget, you can use the FSlateApplication::Get().SetKeyboardFocus(Widget) method. Focus management must be implemented by the client.
You can also show the system mouse cursor using FSlateApplication::Get().ResetToDefaultInputSettings();. If you want to set focus back to the game, FSlateApplication::Get().SetFocusToGameViewport(); will do that.
If the SCoherentInputForward widget has focus, then keyboard events are sent to the focused Coherent UI View. To focus a view, you should click anywhere on it.
Mouse events are sent only to the Coherent UI View that is below the cursor. The code is inside the SCoherentInputForward::OnMouseEvent method. The method does roughly the following:
View::IssueMouseOnUIQuery, View::FetchMouseOnUIQuery and View::IsMouseOnView APIs. If any of the HUD Views is below the mouse, the appropriate mouse event is sent to it.SCoherentInputForward::SetRaycastCollisionGroup API. Currently, the collision group is set for all objects. If an object is hit and it has a UCoherentUIComponent, the mouse coordinates re converted in Coherent UI View space and sent to the View. The conversion is done using the UV coordinates of the static mesh. By default, the first UV channel is used, but that can be changed with SCoherentInputForward::SetRaycastUVChannel.To setup the widget in the CoUITestFPSSample the following code is added in CoUITestFPSCharacter.h:
TSharedPtr<class SCoherentInputForward> CoherentInputForwardWidget;
Then, in the end of the ACoUITestFPSCharacter::PostInitializeComponents in CoUITestFPSCharaceter.cpp:
CoherentInputForwardWidget = SNew(SCoherentInputForward).Owner(this);
if (GEngine->IsValidLowLevel())
{
GEngine->GameViewport->AddViewportWidgetContent(
SNew(SWeakWidget).PossiblyNullContent(
CoherentInputForwardWidget.ToSharedRef()));
}
if (CoherentInputForwardWidget.IsValid())
{
CoherentInputForwardWidget->SetVisibility(EVisibility::Visible);
}
The code above creates and adds the SCoherentInputForward widget to the front of the viewport. After that the widget takes care for sending input to Coherent UI as explained above.
Of course, you can also use the Coherent UI View API to create your own input events and send them to Coherent UI. The View provides the MouseEvent, KeyEvent and TouchEvent methods that send input to the it. Each Coherent UI input event has to be populated with properties that can be trivially translated from the UE4 events.
To add input in a blueprint only game, you need to spawn an actor that will set up the SCoherentInputForward widget. This widget is an empty widget that covers the whole screen and forwards input to a Coherent UI View. The widget should be added in the front of the viewport so it can process mouse and keyboard messages first. In fact, due to Unreal Engine 4's architecture, the widget can process keyboard messages only if it's focused.
Here's a sample blueprint that enables the input:
The actor setup part spawns and initializes the actor, which in turn adds the SCoherentInputForward widget to the viewport.
The "Game to UI Focus" part is for managing input focus - when you press Tab, the character input will stop and the UI will take it. This would seem to be enough, but currently UE4 stops firing key events when the focus is not on the viewport so we need other means to get focus back to the game. This is shown in the "UI to game focus" part. The Coherent UI input actor provides events for key/mouse up/down - in the sample we're using the key down event. When it's fired there's a check for the key enum to see if it's the Tab key, and if it is, the focus is toggled.
The "Toggle Coherent UI Input Focus" function provides means for toggling input without tracking it. If you want to, there are also functions for querying and setting the focus explicitly.
Here we outline some common usages of the input system. The examples are given using blueprints and the Coherent UI Input actor as that maps directly to C++ code. We'll assume that you already have a spawned and initialized Coherent UI Input actor:
Toggling input focus can be done 2 ways:
ToggleCoherentUIInputFocus method. The actor keeps track of the current focus internally so it can toggle between game and UI at any time. If you want to set it explicitly, you can use SetCoherentUIInputFocus, or query the state with IsCoherentUIFocused.
SetCoherentUIViewFocus method. An alternative is to simply let the user click on the input field that she wants to type in.
When the UI is focused, keyboard events are consumed by the Coherent UI GT Input forward widget by default. You can change that using the SetInputPropagationBehaviour method. The available options are to forward keyboard events, joystick events, both or none to other widgets.
This can be useful if you need to have your action handlers (e.g. "Jump", "Fire", etc.) executed even if the focused widget isn't the game viewport.
Note that joystick forwarding is only taken into account when using the Coherent UI GT Input actor's methods for setting (or toggling) focus.
If you want your mouse input to be forwarded to Coherent UI first for hover events, but still receive keyboard events in your game, then you can use the AlwaysAcceptMouseInput method. When set to true, mouse events will be received by Coherent UI, regardless of whether the input forward widget is focused or not.
To control whether a Coherent UI view receives mouse events you can change the View component's Receive Input property. Only when this property is set to true will the component be considered for receiving input events.
When you have multiple views on 3D objects and they are overlapping when viewed with the current camera, the front one will receive the mouse input events, and the back one will get nothing. If, for example, you have disabled input on the front one, the back one will still get nothing, because 3D objects under the cursor are gathered with a raycast from UE4 and which is independent from the input forward widget's logic. For performance reasons, by default the raycast returns only the first hit object. In a scenario like above, when you have 2 objects and you want to forward input to the back one when the front one is marked as not receiving input, you have to change the raycast type from returning single object to returning multiple objects.
It's common in some cases (e.g. in-game menu) that you'd pause your game and show a menu that doesn't cover the whole screen.
With the default implementation of the input widget, mouse events will be passed on the viewport (or other 3D views) when they happen on a transparent area of the HUD. This may not be desired with in-game menus, since the designer would likely want to have the menu focused until the player explicitly resumes the game.
There's a simple solution for that - setting the click-through alpha threshold. This value determines the maximum alpha value of a pixel so it's considered transparent and input is passed on the next handler. If you set that to 0, only completely transparent pixels pass input. If you set that to a negative value, e.g. -1, every pixel is assumed to be solid since its alpha is in the range [0, 1], thus never passing input to other handlers.
In our sample game we have the following code for capturing input when the in-game menu is shown by pressing the Tab key (see ACoUITestFPSCharacter::OnTabReleased):
Coherent::UI::View* View = GetCoherentUIView(); // Set the alpha threshold to a negative value when the in-game menu // is shown so if you click on a transparent area, you don't return // to the game. View->SetClickThroughAlphaThreshold(-1.0f);
When the in-game menu is closed in ACoUITestFPSCharacter::OnCloseIGMenu, the alpha threshold is set back to the default. If you show the mouse cursor again using the *=* key, which toggles game/UI focus, you'll be able to interact with other 3D surface Coherent UI views, unlike when you open the in-game menu.
Coherent::UI::View* View = GetCoherentUIView(); View->SetClickThroughAlphaThreshold(0.0f);
You can also do this with Blueprints as the CoherentUIComponent has the whole Coherent::UI::View API exposed.
Note: UI Scripting can be made with Blueprints too. Refer to the UI Scripting with Blueprints section for details.
Binding is the Coherent UI name for the communication between the game and the pages JavaScript. In this document we'll use the terms binding and UI scripting interchancably.
Coherent UI provides a very powerful binding framework that can be used to connect the C++ logic of the game with the UI JavaScript code and vice-versa.
A detailed guide on Binding is also given in the C++ documentation of Coherent UI.
To use the Coherent UI Binding feature, your page MUST include the coherent.js file.
To trigger events from C++ to JavaScript you can use the TriggerEvent method of the View.
void ACoUITestFPSCharacter::OnToggleGameMenu()
{
Coherent::UI::View* View = GetCoherentUIView();
if (View)
{
View->TriggerEvent("ToggleMenu");
}
}
Any parameters you pass to it will also go to the JavaScript code. In JavaScript you must define which function will be called when an event is triggered from C++. The coherent.js file exposes a special JavaScript object named engine that provides the glue and connections between events in C++ and JS. You must use that object to define all your event handlers. The on method will bind an event to a JS handler.
engine.on('ToggleMenu', toggleMenu);
In this case as soon as C++ calls TriggerEvent("ToggleMenu"), the toggleMenu function will be executed in JavaScript.
Triggering events from JavaScript to C++ is analogous. First in C++ you need to define what method (or UE4 delegate) will handle a specific event triggered from JS.
Defining C++ handlers must happed after the View has triggered the ReadyForBindings event. If you want to handle events from JS, you must subscribe to it and in its handler declare all your bindings. In the ACoUITestFPSHUD Actor in the sample game this is done with the following line:
CoherentUIHUD->ReadyForBindings.AddDynamic(this, &ACoUITestFPSHUD::BindUI);
Now as soon as the View has been loaded it will invoke the BindUI method and there you can declare all the bindings for the View.
The BindUI method is straightforward:
void ACoUITestFPSHUD::BindUI(int32 frameid, const FString& path, bool isMain)
{
CoherentUIHUD->GetView()->BindCall("CallFromJavaScript",
Coherent::UI::MakeHandler(&CalledFromJSSampleDelegate,
&(FCalledFromJSSample::ExecuteIfBound)));
CoherentUIHUD->GetView()->BindCall("CalledFromJSString",
Coherent::UI::MakeHandler(this,
&ACoUITestFPSHUD::CalledFromJSStringHandler));
}
Essentially we say: "When JavaScript fires the CallFromJavaScript event, in C++ you must execute the CalledFromJSSampleDelegate delegate". The same applies for the second binding but in that case it'll call the CalledFromJSStringHandler method of the class.
In JavaScript you just have to use:
engine.call("CallFromJavaScript", 123);
engine.call("CalledFromJSString", "Hello from JavaScript!");
Note that you can again pass arguments from JS to C++ but the handler signatures must coincide with the arguments passed, otherwise an error is generated.
In the sample game the handlers for those methods are very simple:
void ACoUITestFPSHUD::CalledFromJSHandler(int32 number)
{
UE_LOG(LogScript, Log, TEXT("UE4 Delegate called from JavaScript"));
}
void ACoUITestFPSHUD::CalledFromJSStringHandler(const FString& str)
{
UE_LOG(LogScript, Log, TEXT("String received from JS: %s"), *str);
}
We just log that we were called from JS, so that we know everything works fine. Coherent UI support passing parameters to/from JS for all primitive types, STL containers as well as the UE4 FString type. To use those types you must include the relevant headers available in the Coherent/UI/Binding folder.
Coherent UI also support binding user defined types. For a detailed guide on that topic please consult the Coherent UI C++ documentation.
Almost all View methods have been exposed to the Blueprint visual editing system. Coherent UI Components also trigger dynamic multicast delegates on a variety of events that users can subscribe to and use either from C++ or Blueprint.
A sample of the Blueprint usage is given if you open the Blueprint editor on the StaticActorCoherent object in the sample game.
In this case the FinishLoad event (a dynamic multicast delegate) of the Coherent UI Component will trigger a Print String with the text "Page on cube loaded!". The FinishLoad is triggered by the Component when the page it is loading has been successfully loaded. Another event we've already used in the previous section is the ReadyForBindings event that we bound through C++.
For a full list of the event exposed by the Coherent UI Components please refer to the CoherentUIBaseComponent.h file or review them in the Blueprint editor.
For a full list of the provided methods please refer to the CoherentUIBaseComponent.h file or review them in the Blueprint editor.
Binding is the Coherent UI name for the communication between the game and the pages JavaScript. In this document we'll use the terms binding and UI scripting interchancably.
Coherent UI supports scripting the UI via Blueprints. This allows developers to expose game objects to the UI to populate the interface and trigger actions from the UI like starting a new game when the player clicks "Start game" in a menu.
With Coherent UI you can script the whole UI via Blueprints without C++ code.
Developers can trigger events in the UI JavaScript code via Blueprints. They can also listen to events triggered in JavaScript and implement game logic on such events. Developers can expose primitive types and any UObject to JavaScript to populate the UI.
In this section we'll introduce a small example that shows two-way communication between Blueprints and the UI JavaScript. The example is available in the Coherent UI Sample game.
The first example show sending game data to the UI. We'll make available the whole "PlayerCharacter" object to the UI and we'll show on-screen the player name, the max health she can have and the max ammo she can carry. In the "hud.html" page you can see that there is a "playerInfo" element that contains the player data we want to show. We'll populate this data from the game through Blueprints.
The screenshot below shows the complete Blueprint.
As soon as the game starts and the HUD loads, the "SetPlayerState" event will be triggered and the corresponding JavaScript code that populates the "Statistics" fields in the UI will execute.
In JavaScript we've added a handler for the "SetPlayerState" event.
engine.on('SetPlayerStats', function (character) {
$("#playerName").html(character.PlayerName);
$("#playerMaxHealth").html(character.MaxHealth);
$("#playerMaxAmmo").html(character.MaxAmmo);
$("#playerInfo").css("display", "initial");
});
// IMPORTANT: Signal the game we are redy to receive events
// We have setup all JS event listeners (the 'engine.on' calls)
engine.call("UIScriptingReady");
The "character" variable contains all the properties of the APlayerCharacter actor in the game. Now we can use them to populate the "Statistics" of our UI.
This is how some of the properties of the "character" parameter look in the Coherent UI JavaScript debugger. Their names are the same as in the game.
Note that when exporting UObjects, only their primitive type UPROPERTY tagged fields are exported. UObjects contained in exported UObjects are not recursively exported because they might contain circular dependencies and cause memory outages.
The object in JavaScript is a copy of the original one, so changing directly its properties won't affect the object in the game. You can however call events back in the game passing parameters and use that mechanism to update logic in the game from the UI. This is explained in the next section.
This section show the other way of communication - from the UI to the game. We'll use the Menu level from the Coherent UI Sample game. When the player clicks on the "Start game" button we'll send an event from JavaScript to the game and change the level.
First let's see what happens in JavaScript when the player clicks "Start game":
document.getElementById('play').onclick = function () {
engine.call('onStartGame', 'Example_Map');
};
The code is pretty self-explanatory - when the 'play' button is clicked call an event in the game called "onStartGame". "Example_Map" is a parameter for this event and is the name of the level to load.
Now lets create the Blueprint that will handle this event and actually load the new level.
This is how the final blueprint looks like:
Remember that Coherent UI supports as many arguments as you need in both directions of scripting and you can send also UObjects both ways - they work automatically.
Coherent UI supports triggering Custom Events defined in Blueprints. The event must be declared in the Blueprint that contains the UCoherentUIComponent component. To use this functionallity, simply right click on the Event Graph and choose Add Event -> Custom Event from the menu. Name the event and add any input arguments. Say, the event is CreateCharacter and the argument is its name. Here is how the custom event looks in the Blueprint editor:
Now you can trigger this event from your UI using:
engine.trigger('CreateCharacter', 'Joe Monster');
Coherent UI will automatically trigger the custom event with all arguments from JavaScript. Please note that currently only primitive types are supported as arguments - bool, byte, int, string, float.
Coherent UI supports auto-magic export of UObjects from the game to the UI. Just send a UObject parameter and all its UProperties will be available in the UI page. Only first level properties are sent, UObjects inside other UObjects are not recursively expanded.
You can also updates values in UObjects from JavaScript. To do this create a JavaScript object with the properties you want to change named the same way as in the game. For instance:
var mycharacter = {
PlayerName: "MyJSPlayer",
MaxAmmo: 800
};
Now you can send an event to the game with this object:
engine.call("UpdatePlayer", mycharacter);
In the game Blueprint when you receive the event you can create a "Read Object" node from the "CoherentUIJSPaylod" object and pass it the APlayerCharacter of your game. All properties that have the same types and names as the ones passed from JavaScript will be updated. You can do this with any UObject. As long as the names and types of the objects coincide, they'll be updated by the "Read Object" node.
Currently returning values from the game to JavaScript events is not supported for Blueprints (it is possible with the C++ interface) but will soon be available.
Objects sent to JavaScript are copies of the game versions. Sending smaller objects less often is preferable as it incurs less resource usage.
As a general note the best way to structure the scripting is to create blueprint functions for every interaction and call them when needed from the master Blueprint. This makes the master Event Graph much less cluttered and more readable.
Coherent UI comes bundled with the powerful Coherent UI Debugger. It is modeled after the Chrome dev tools and provides an easy way to inspect elements in the page, debug JavaScript, check the performance of the page. The Debugger allows live editing and debugging of the UI.
You can find the Debugger in the CouiTestFPS/Tools/Debugger folder.
To use the Debugger start it while your UI is running in the UE4 Editor or in your game. By default the port 9999 is used for the Debugger to connect to but you can change it by modifying the DebuggerPort property of the ACoherentUISystem Actor. When connected, the Debugger will offer a list of currently active Views, you have to select the one you want to work on.
The Elements tab provides an overview of the page along with the properties and styles of all the elements in the page. When you hover over the elements in the Debugger, they are highlighted in the game as well.
The Resource tab shows all the loaded sources and images in the page.
The Sources tab can be used to debug JavaScript code. You can put breakpoints and inspect JavaScript variables and code flow.
The Timeline tab allows profiling the page. It shows detailed stats on how much time different operations have taken.
The Profile tab can be used to profile detailed aspects of the page as the JavaScript execution or the CSS selector. It also allows checking for JavaScript memory wastages.
The Console tab shows a JavaScript console that you can use to easily execute code in the page, debug and check properties of objects.
Coherent UI Menu allows you to start immediately using Coherent UI for the menus in your game. It also allows to create the menu and implement its logic without leaving the Unreal Engine 4 editor. This menu can be used for a quick prototype, as a substitute for the final menus during development and even for the final game after some styling. You can see the menu in action in the UE4Editor by loading the MenuBP_Map level.
To create a menu, you create CoherentUIMenuInfo and call Setup UI Menu with it and the view that should show the menu. You can create the CoherentUIMenuInfo either using the Make CoherentUIMenuInfo function or as a varible in the Blueprint. Setup UI Menu will return the CoherentUIMenu instance you can use to control the menu.
The properties of CoherentUIMenuInfo are:
Menu ID - the unique id of this menu in its view. This id is used internally by CoherentUIMenu, but can be used from your JavaScript code to manipulate the menu or to listen for its events. See the JavaScript API for more details.Parent Element ID - the id of the element in which to show the menu. You can control the position of the menu by changing the position of the parent element.Visible - whether the menu is visible by default or not. You can later change the visiblity of the menu using the Show and Hide methods.Buttons - the list of buttons in the menu. Please note, that currently the list of the buttons can not be changed after the menu has been created.Here is how a quit game menu might look like:
The CoherentUIMenu fires the MenuButtonClicked event button when a button is clicked passing the label of the click button as an argument. So to handle a click on a menu, you can attach a delegate to the MenuButtonClicked event, switch on the button label and call any code you need.
Here is how the final menu looks in the Blueprint editor:
Using the Coherent UI Menu requires some JavaScript and CSS styles to be present in the view.
The required stylesheets are:
<link rel="stylesheet" type="text/css" href="components/flat-ui-official/bootstrap/css/bootstrap.css" /> <link rel="stylesheet" type="text/css" href="components/flat-ui-official/css/flat-ui.css" /> <link rel="stylesheet" type="text/css" href="coherent/css/menu_style.css" />
And JavaScript resources:
<script type="text/javascript" src="coherent.js"></script> <script type="text/javascript" src="coherent/js/game_menu.js"></script>
Please note that game_menu.js depends on coherent.js, so it is included after coherent.js.
The JavaScript API of the menu kit consist of the following events:
cui.MenuButtonClicked - triggered on button clicked with the Id of the menu and the Label of the buttoncui.ShowMenu - triggered to show the menu with the menu Idcui.HideMenu - triggered to hide the menu with the menu IdYou can attach to and trigger these events using the engine.on and engine.trigger functions.
Coherent UI supports running in the UE4Editor on Mac OS X as well as in packaged games.
Currently there is no installator for Mac OS X, but all necessary files are installed by the Windows installator. The easiest way to install Coherent UI on Mac OS X is to transfer the installed files from Windows either by your source control system or by simply copying the files.
You need to generate the sample project first. To do so, make sure you add a file named UE4Games.uprojectdirs in the root UE4 folder and add a line with the path to the folder, containing the CoUITestFPS folder. After that, run the GenerateProjects.command script and after it's finished, open the UE4.xcodeproject file. You should have a scheme named CoUITestFPSEditor. Select that scheme and build it.
The sample project is a modification of the First Person C++ template. To run it, start the UE4Editor.app and locate the CoUITestFPS.uproject file in the "Open Project" selection. You can also do that by right clicking on the CoUITestFPS.uproject file in the Finder and opening it with the UE4 version that has Coherent UI installed.
After you open the project, you need to enable the Coherent UI plugin. To do so, go to Window->*Plugins*, find the entry for Coherent UI plugin and enable it. You'll need to restart the Editor and then you can run the sample.
Coherent UI supports cross compilation to Linux. After setting up your Unreal Engine 4 for Linux, you can cross compile our sample game for Linux. Due to limitations in the current plugin support of Unreal Engine 4, some additional steps are required to make the game work after being packaged. The steps are:
Coherent UI provides a very fast out-of-process rendering scheme. Each View is drawn in it's own process and communication with the game is performed via high-performance IPC. As Coherent UI can be used as an in-game browser, the external processes provide better encapsulation and security for the game. Memory spaces are also cleanly divided.
The asynchronous model guarantees that the UI or web pages will never tax the game too much, so in a situation where computational resources are scarce, the UI might run slower but it will never "steal" resources from the game slowing it down.
By default Coherent UI renders GPU-accelerated. This means that UE4 and Coherent UI have to divide the GPU time between themselves. As UE4 is the main process, it WILL always get the rendering resources that it needs and might cut down the share that Coherent UI requires if GPU resources are scarce. Event though Coherent UI needs very little GPU time, if UE4 consumes all of it, the UI might "starve" and the frame-rate of the UI might decrease. This is perfectly fine in extraordinary situations when UE4 might need all the GPU - for instance when an exceptionally heavy scene is loaded or an effect shown. The UI might slow down a little but the game will run perfectly.
If you notice slow-downs in the UI it might be a case when UE4 is starving Coherent UI on GPU time. Usually this happens when VSync is off. Enabling Vsync solves any hiccups 90% of time. Enabling VSync is usually ok for most games as there is no visually perceivable difference if the game runs on more than 60 FPS (monitors usually refresh at that rate). VSync also saves power and hence battery life on laptops. One case when a disabled VSync might be required is in competitive multiplayer games where some player prefer to have it off in order to diminish the lag from the input.
Coherent UI provides also a very fast Software renderer. If your game is very heavily GPU bound you can try setting your Views to software rendering. This is done through the "Force Software Rendering" property. Note that when in software mode, not all rendering features are available in the View. For more information please refer to the C++ documentation.
To force Coherent UI to not let UE4 starve it on GPU resources you can also enable the "Is on Demand" property. In that case UE4 will wait on every frame for Coherent UI to render it's frame.
