Meet Coherent Labs on Game Connection in Paris

There are only 6 days left till the start of Game Connection 2012 in Paris. And we can hardly wait because next week our team will be in the French capital among with more than 500 game studios, publishers, distributors and investors from all around the world.

If you're not familiar with Game Connection, this is one of the best game conferences and networking events in Europe. The reason  for this is that you are actually allowed to meet and get to know the key players in the gaming industry through something similar to "speed dating". The end goal, of course, is to find the best possible partners for your business and this game conference might be the perfect place to do so.

The Game Connection meeting app allows you to schedule up to 10 meetings per day, well in advance of the event by sending and approving requests to/from other attendants. In case you have already decided to attend the event you might pass through our exhibitor table T350. Even better is to request a meeting in advance because we might be quite busy during these 4 days.

We will keep you posted on Game Connection 2012 and we will share with you our impressions of the event.

Unity3D integration with Coherent UI - Public Beta

Coherent UI is really happy to announce today the official public beta of our integration with the Unity3D game engine!

In other words, with Coherent UI you can build your game user interface in modern HTML5 & CSS3 and trivially integrate it in your Unity3D game.

If you were reading our blog, you might have known already that this was coming. Early this month in Coherent UI in Unity3D - First look we released a short teaser and later on we talked in another post about the upcoming official integration with the game engine.

For the curious ones we have prepared a video tutorial explaining how Coherent UI works inside the editor:





If this gets your curiosity and you'd like to get your hands on the public beta, don't hesitate to request your version on our download page.

We will highly appreciate your feedback about our integration with Unity3D. Feel free to contact us with any problem you might encounter or suggestions you have at 'info at coherent-labs dot com'.

Coherent UI in the Unity3D editor - Introduction

We would like to share a video showing the first version of the Coherent UI integration in the Unity3D editor. In this video we show how to:

• Import the Coherent UI package
• Create an in-game object with a web page on it
• Interact with a web page in the game
• Add a HUD powered by Coherent UI
• Build the game

We plan to release a Beta version of the product in a couple of weeks tops so stay tuned! For any suggestions or thoughts about the Unity integration please leave a comment or visit this thread.

How it works

The integration is very powerful yet very simple to use and requires little or no programming at all as we have wrapped everything in components with editor-accessible properties. However, we also expose to script all the features available in the .NET and C++ version of the library so more custom behavior can also be achieved.

What happens in the video:

1. We load an empty scene and add a floor, a light and a cube on which we'll project what in Coherent UI terms we call a 'View'. A view is something that Coherent UI renders - that could be a HUD element, a projected web page, an animation - anything wrapped in an HTML page.

Simple scene

2. We add a character controller so that we can move around

3.We import the Coherent UI package. The component we are interested in this tutorial is 'CoherentUIView'. We drag it on one of the faces of the cube. All the various properties of the View are editable in the Inspector. By default it will load google.com. Let's hit 'Play'.

A Coherent UI View on an object

4. We can see the web page on the game object in Unity!

5. We change the web page and the resolution.

6. Now we have a nicer page in a better resolution.

An interactable web-page projected on a game object

7. The next thing is to make the page interactable. To do this we add a Mesh Collider component to the mesh. Then a UserScipt component that just handles the raycast and mouse button translation to the Coherent UI View. We also add some code to the camera to stop it from receiving when we hit 'L', otherwise it's very hard to click on anything with the mouse-look on. That's it - the View is now fully interactable.

8. Now for the HUD - we just drag the 'CoherentUIView' component on our Main Camera. We have made the HUD resources and copy them in our project and set the View's Page property to coui://TestPages/demo/demo.html.

'coui' is a special protocol we use to signal that the resource is local and subject to loading through the file handlers supplied by the application. In this way you can use a custom resource manager as well as for instance encrypt your UI data. The HUD looks pixelated because the resolution we set does not coincide with the one of the Editor pane. It will however be OK in the game when we build it. In your games the resolution will always be tied to the actual resolution of the Camera (the back-buffer).

A sample HUD made rendered through Coherent UI

9. The Coherent UI integration handles all cases and detects if the component is attached to and object in the world or a camera. It is compatible with post effects. We can add an effect on the camera. By default that effect will be applied to the View also. However this might not be desirable so by clicking 'Apply After Post-Effects' you can disable them on the View.

Post effects can be applied or skipped on Coherent UI Views

10. We now just have to build the game. Coherent UI resources will automatically be copied and made available at runtime.

The built game

 All Coherent UI rendering happens in the native C++ plugin so it's impact on the performance of your game should be minimal. 

Porting Brackets to a new platform

Brackets is a code editor for HTML, CSS and JavaScript that is built in HTML, CSS and JavaScript. That makes it a fully featured desktop application written using Web technologies. Brackets embeds a browser to show and run the entire editor and extends it with additional functionality used by JavaScript. There are more and more applications using this kind of mixed platform - HTML/JavaScript becomes first class User Interface (UI) (and not only) for Window 8 applications, other tools from Adobe such as Edge Animate, Icenium - a new cloud IDE, recently announced by Telerik. Sencha Animator also is an HTML5 application, running inside QT WebKit. Even the UI of most browsers is written in HTML.
This approach has several advantages:

• fully cross-platform UI - as long as the browser is running on a platform, the UI is going to be running on that platform too
• full UI customization using CSS lots of available application frameworks MVC, MVVM - Backbone.js, knockout.js, you name it
• lots of UI libraries - Kendo UI, jQuery UI, mochaui
• easy development of the UI using Chrome Dev Tools, Firebug, etc.
• Live reload of the UI, without restarting the whole application!
• native access to filesystem and everything without sandboxing
• use existing native APIs
• native performance where necessary
• only single browser to support - this is a huge relief for the HTML/JavaScript developers

Given the advantages, I am sure that more and more mixed native and HTML applications are going to appear.

Brackets Architecture

Brackets consists of two applications running together - an HTML/JavaScript application and a native C++ shell that runs the HTML application. The native shell consists of two processes - one running the HTML application inside a browser, and one running the heavy native part like filesystem operations and number crunching. The communication between the two processes is asynchronous, using JavaScript callbacks and request / response for the native code.

• The shell extends the standard HTML DOM with the following functions:
• Open file or directory using the OS dialog
• Read file
• Write file
• Create directory
• Rename file or directory
• Get file last modification time
• List directory contents
• Open and close browser for Live Preview
• Open extensions directory in OS file browser
• Open developer tools for brackets itself
• Get the system default language
• Time since the application start
• Get the application support directory

Brackets Shell implements the asynchronous communication between the HTML render process and the browser in a very simplistic way: JavaScript executes a function appshell.fs.readdir(path, callback), the render process stores the callback in a mapping from a request id to callback and calls the native process with the name of the function. When the native process is ready it sends back any result of the call together with the request id. The HTML process finds the callback by the request id and executes it with any result.

This architecture is the same as for any good GUI application - the UI never executes any expensive functions and is always responsive and all data manipulation is in a separate thread.

Running Brackets in Coherent UI

Making Brackets run inside Coherent UI is really easy. We start by creating a view that loads www/index.html relatively to the executable. To support Coherent UI we have to include some JavaScript files in the index.html of Brackets

These scripts are Coherent UI dependencies, Coherent UI itself and the abstraction layer between all JavaScript code of Brackets and Coherent UI. Then we have to register our native callbacks for the asynchronous calls:

Brackets native functions always return an error code as first argument to the JavaScript callback. This mechanism can be implemented in Coherent UI, but then the callbacks always have to do two things - handle the correct result and handle the error, which is kind of annoying. engine.Call might take two callbacks - one for the successful result and one for error. Therefore we have to wrap the normal callback in an object with separate handlers for success and error.
gist: javascript callback wrapper
All that is left now is to wrap the callbacks and use engine.Call instead of native function.

and to write the native function

Handling synchronous calls  

Brackets has and some synchronous methods that return to JavaScript immediately. These methods are:

• Get the application support directory
• Get the system default language
• Time since the application start

Coherent UI does not support providing synchronous JavaScript functions by design, so we will have to work around that.

The application support directory remains constant through a single run of Brackets, so we can set it once and for all during application initialization:

Getting the system default language might be implemented in the same way. The last method left is time since the application start and is used only in Show Performance Data menu. This method might be implemented entirely in JavaScript, assuming Brackets is not going to be reloaded during the performance test run.

Porting Bracket to Linux

Since Coherent UI already runs on Linux and we have implemented most of the native functions using the cross-platform boost::filesystem library, all we have to do to get is showing an open file dialog, creating and closing a chrome instance opening an URL and folder in the default OS HTML browser and file manager.

We use the GtkFileChooserDialog and the xdg-open tool. Unfortunately, the Live Preview doesn't work under Linux. Live Preview in Brackets creates a new Google Chrome instance with enabled remote debugger, attaches to the debugger using XmlHttpRequest and WebSockets and controls the instance via the debugger. What happens on Linux is that one of the XmlHttpRequests fails with "DOM Exception" and the debugger is unable to attach to the instance.

Another Linux related issue is that I couldn't find a way to close a Google Chrome  tab on Linux gracefully, so when the developer tools are closed you get the "Ow, Snap" page. In a future version we will stop using Google Chrome for Live Preview and for showing the developer tools, which will fix this problem.

Here is a short video of Brackets running on Linux:

Get a prebuild package or get Coherent UI and start hacking!

Coherent UI in Unity3D - First look

We want to share with you the first video of the integration of Coherent UI with Unity3D:

Coherent UI already support HUD-type views as well as views mapped to objects with input. The integration is designed to be very user-friendly yet powerful. The demo shows the awesome CSS3 periodic table by Ricardo Cabello aka Mr. Doob and in Unity3D it looks even better.

Update: Check out the introduction to the integration of Coherent UI in the Unity3D editor.

Passing a struct from C# to C++ gone wrong

Have you ever tried calling a C# method returning a structure from C++? This is not a common use of interoperability so you probably haven't, but let me tell you a story about shooting yourself in the foot.

Imagine the following scenario: you have a C++ library that exports an interface of callback methods that will be called by the library during the lifetime of the application. One of the interface's methods returns a struct (let's call them GetStruct() and SimpleStruct, respectively). So far so good. Now you want to port your library for .NET by making a wrapper of the C++ library. You make a managed counterpart of the C++ interface and SimpleStruct(with all the marshaling needed, if any) and you're done. Except that it doesn't (always) work.

Here's an example of a case when it doesn't. Let this be our C++ library (the interface part I was talking about is omitted for brevity)

Nothing special about this code, except some unnecessary typedefs and weird processing of the returned value in FireCallback but I'll get to that in a second. This will be the C# part of the program:

Again,some standard interop use. If you compile the library and executable you'd expect a 3 written in the console, but instead you get this:

Wait, what, stack corruption? Everything's fine before the call of g_MyCallback(x, y) but it somehow corrupts the stack. If you add a breakpoint in MakeResult in the C# code you'll notice something interesting.

The value of x is something funny and y is 1 instead of 2. It seems like the parameters are offset by one. And yet all the other methods of our imaginary interface returning primitive types work? This calls for some disassembly. Let's see what happens in g_MyCallback(x, y).

5C6C13B7 8B F4                mov         esi,esp 
5C6C13B9 8B 45 0C             mov         eax,dword ptr [y] 
5C6C13BC 50                   push        eax 
5C6C13BD 8B 4D 08             mov         ecx,dword ptr [x] 
5C6C13C0 51                   push        ecx 
5C6C13C1 8D 95 20 FF FF FF    lea         edx,[ebp-0E0h] 
5C6C13C7 52                   push        edx 
5C6C13C8 FF 15 30 71 6C 5C    call        dword ptr [g_MyCallback (5C6C7130h)]  

Ok, we push the x and y parameters and then push something else. The theory for offsetting the parameters by one seems correct. But why is the compiler doing this? Well, our method returns a struct by value and copying it isn't very effective, so the (Named) Return Value Optimization kicks in (it's applied even when compiling with /Od). In short, the last pushed parameter is the address where the returned value will be stored and no copying will occur. The C++ compiler is aware of this fact and works its magic. When we cross the language boundary to C#, however, the stack is broken. You'd expect that the CLR would know these things, and it does, but we hit a corner case. The rules for function return values can be found here. More specifically:

• POD return values 32 bits or smaller will be returned in the EAX register.
• POD return values 33-64 bits in size will be returned via the EAX:EDX registers.
• Non-POD return values or values larger than 64-bits, the calling code will allocate space and passes a pointer to this space via a hidden parameter on the stack. The called function writes the return value to this address.

  (The bullets are points 12,13 and 14)

The C# compiler simply couldn't know if the C++ structure is a POD or not so it applies the rules for non-PODs and it doesn't expect the hidden parameter. With the mystery unveiled, we have the following options for making our scenario work:

• Make the structure a POD. In the example we can do this by removing the constructor. It's the only thing breaking the POD-ness.
• Change the signature of the callback in the C++ code so it returns an integral type of the same size. In other words, change "typedef SimpleStruct ReturnType" to "typedef int ReturnType". This way the compiler won't emit code for RVO. If you have a 64-bit structure, you can use long long.
• Instead of return value, make the structure an output parameter.
• Add bogus fields in the structure to make it larger than 64-bits.

The last option is the least desirable one and I added it for completeness. Since having a constructor is useful in some cases, I opted for the signature change in our project which lets us keep the non-POD parts and is hidden from the user. It's not the prettiest solution and you have to keep it in mind if you ever change the size of the structure but it works :).

Note that for x64 builds the stack won't be corrupted (when compiling with Visual Studio) because the first four integral or pointer parameters are saved in the RCX, RDX, R8 and R9 registers. In the example function we only have 2 parameters, so the hidden RVO parameter will go to a register (if the function had 4 or more arguments, then maybe we will corrupt the stack, if the compiler decides to push the additional arguments and not preallocate memory by modifying the stack pointer at the beginning of the function). The return value will be wrong though, because the compiler will generate code that interprets the returned value in RAX as an address, and not a value, so it will read the memory at that address. This can be fixed using the same solutions as the ones for a 32-bit build.

Client application multi-threaded rendering support

Our on-going integration effort in Unity3D prompted us to accelerate a feature we've been planning for a long time but didn't have the time to finish until now - support for client applications with a multi-threaded rendering architecture.

Although it is even now possible to incorporate Coherent UI in an application with multi-threaded rendering, it is inherently difficult because the rendering-related events must be performed in the thread that updates the system.

For the sake of simplicity I'll call the thread that performs the logic in the client app the 'update' thread and the one that renders the 'rendering' thread.

There were two major challenges we needed to beat in order to support a separate 'rendering' thread in the client - rendering resource management and draw callbacks.

When Coherent UI needs a rendering resource - usually a texture, it calls the appropriate callback provided by the user. This happens when a new View has to be created or a View gets re-sized. The API expected the result of the operation to be immediately available.

We changed it so that it now uses a 'SurfaceResponse' object that must be signaled with the result of the operation. This signaling can happen at any later time, so the resource could be created in a separate thread. This is analogous to the resource requests (usually file-reads or file-writes) we already support in the same manner.

All new surfaces are fetched while calling 'UISystem::FetchSurfaces' for buffered Views and 'View::FetchSurface' for on-demand Views. This results in calls to the 'ViewListener::OnDraw' callback provided by the user. Usually a copy is made of the received surface for rendering and up until now those methods had to be called in the 'update' thread. Now it is perfectly safe to call them from your 'rendering' thread. This not only allows for easy integration with multi-threaded rendering pipelines but could be used as a performance optimization, as it voids the need to make an eventual copy of the surface to be used later for rendering.

Note that the 'ViewListener::DestroySurface' could now be called from both the 'update' and the 'rendering' thread but Coherent UI has already relinquished any ownership on the surface when it calls the method so it's trivial to dispose it even when the callback happens in the 'update' thread.

The API changes will be available in the next version of Coherent UI.

Announcing Coherent UI for .Net

We are proud to announce the official release of Coherent UI for .Net.

With Coherent UI game developers and UI artists can use standard modern HTML5, CSS3, and JavaScript to create user interface and interaction for their XNA, SlimDX or SharpDX games. Features like secure micro-transactions and in-game store, social networks integration are easy to implement using the full in-game browser that Coherent UI provides. In addition to incredible HUDs for XNA and SlimDX based games, Coherent UI gives you fully integrated browser controls for Windows Forms, WPF and Gtk#, so that you can integrate your desktop application with any social network, show YouTube videos or access a web service.
The major highlights are:

• WinForms browser control - full HTML5 and CSS3 support with 3D transformations, HTML5 video and Flash support



YouTube running in a Windows Forms application

• WPF browser control - integrated with XAML, without the complexity of using the WebBrowser control from Windows Forms

  

  Editing the WPF browser control XAML in Visual Studio design mode

• Gtk# browser control - create truly cross-platform managed applications with embedded browser
• integrates with SlimDX and XNA
• connecting arbitrary .Net delegates to JavaScript events
• exposing arbitrary .Net types to JavaScript
• supports both Windows and Linux via Mono

For complete list of Coherent UI features visit our website.

Our next milestone is fully integrating Coherent UI with Unity and now is the time to share your thoughts and ideas about Coherent UI for Unity!