There are only 6 days left till the start of Game Connection 2012 in Paris. And we can hardly wait because next week our team will be in the French capital among with more than 500 game studios, publishers, distributors and investors from all around the world.
If you're not familiar with Game Connection, this is one of the best game conferences and networking events in Europe. The reason for this is that you are actually allowed to meet and get to know the key players in the gaming industry through something similar to "speed dating". The end goal, of course, is to find the best possible partners for your business and this game conference might be the perfect place to do so.
The Game Connection meeting app allows you to schedule up to 10 meetings per day, well in advance of the event by sending and approving requests to/from other attendants. In case you have already decided to attend the event you might pass through our exhibitor table T350. Even better is to request a meeting in advance because we might be quite busy during these 4 days.
We will keep you posted on Game Connection 2012 and we will share with you our impressions of the event.
Coherent UI is really happy to announce today the official public beta of our integration with the Unity3D game engine!
In other words, with Coherent UI you can build your game user interface in modern HTML5 & CSS3 and trivially integrate it in your Unity3D game.
If you were reading our blog, you might have known already that this was coming. Early this month in Coherent UI in Unity3D - First look we released a short teaser and later on we talked in another post about the upcoming official integration with the game engine.
For the curious ones we have prepared a video tutorial explaining how Coherent UI works inside the editor:
If this gets your curiosity and you'd like to get your hands on the public beta, don't hesitate to request your version on our download page.
We will highly appreciate your feedback about our integration with Unity3D. Feel free to contact us with any problem you might encounter or suggestions you have at 'info at coherent-labs dot com'.
We would like to share a video showing the first version of the Coherent UI integration in the Unity3D editor.
In this video we show how to:
Import the Coherent UI package
Create an in-game object with a web page on it
Interact with a web page in the game
Add a HUD powered by Coherent UI
Build the game
We plan to release a Beta version of the product in a couple of weeks tops so stay tuned! For any suggestions or thoughts about the Unity integration please leave a comment or visit this thread.
How it works
The integration is very powerful yet very simple to use and requires little or no programming at all as we have wrapped everything in components with editor-accessible properties. However, we also expose to script all the features available in the .NET and C++ version of the library so more custom behavior can also be achieved.
What happens in the video:
1. We load an empty scene and add a floor, a light and a cube on which we'll project what in Coherent UI terms we call a 'View'. A view is something that Coherent UI renders - that could be a HUD element, a projected web page, an animation - anything wrapped in an HTML page.
Simple scene
2. We add a character controller so that we can move around
3.We import the Coherent UI package. The component we are interested in this tutorial is 'CoherentUIView'. We drag it on one of the faces of the cube. All the various properties of the View are editable in the Inspector. By default it will load google.com. Let's hit 'Play'.
A Coherent UI View on an object
4. We can see the web page on the game object in Unity!
5. We change the web page and the resolution.
6. Now we have a nicer page in a better resolution.
An interactable web-page projected on a game object
7. The next thing is to make the page interactable. To do this we add a Mesh Collider component to the mesh. Then a UserScipt component that just handles the raycast and mouse button translation to the Coherent UI View. We also add some code to the camera to stop it from receiving when we hit 'L', otherwise it's very hard to click on anything with the mouse-look on. That's it - the View is now fully interactable.
8. Now for the HUD - we just drag the 'CoherentUIView' component on our Main Camera. We have made the HUD resources and copy them in our project and set the View's Page property to coui://TestPages/demo/demo.html.
'coui' is a special protocol we use to signal that the resource is local and subject to loading through the file handlers supplied by the application. In this way you can use a custom resource manager as well as for instance encrypt your UI data. The HUD looks pixelated because the resolution we set does not coincide with the one of the Editor pane. It will however be OK in the game when we build it. In your games the resolution will always be tied to the actual resolution of the Camera (the back-buffer).
A sample HUD made rendered through Coherent UI
9. The Coherent UI integration handles all cases and detects if the component is attached to and object in the world or a camera. It is compatible with post effects. We can add an effect on the camera. By default that effect will be applied to the View also. However this might not be desirable so by clicking 'Apply After Post-Effects' you can disable them on the View.
Post effects can be applied or skipped on Coherent UI Views
10. We now just have to build the game. Coherent UI resources will automatically be copied and made available at runtime.
The built game
All Coherent UI rendering happens in the native C++ plugin so it's impact on the performance of your game should be minimal.
Brackets is a code editor for HTML, CSS and JavaScript that is built in
HTML, CSS and JavaScript. That makes it a fully featured desktop
application written using Web technologies. Brackets embeds a browser
to show and run the entire editor and extends it with
additional functionality used by JavaScript. There are more and more
applications using this kind of mixed platform - HTML/JavaScript becomes
first class User Interface (UI) (and not only) for Window 8 applications, other tools from Adobe such as
Edge Animate, Icenium - a new cloud IDE, recently announced by Telerik.
Sencha Animator also is an HTML5 application, running inside QT WebKit. Even the UI of most browsers is written in HTML.
This approach has several advantages:
fully cross-platform UI - as long as the browser is running on a platform,
the UI is going to be running on that platform too
full UI customization using CSS
lots of available application frameworks MVC, MVVM - Backbone.js,
knockout.js, you name it
lots of UI libraries - Kendo UI, jQuery UI, mochaui
easy development of the UI using Chrome Dev Tools, Firebug, etc.
Live reload of the UI, without restarting the whole application!
native access to filesystem and everything without sandboxing
use existing native APIs
native performance where necessary
only single browser to support - this is a huge relief for the HTML/JavaScript developers
Given the advantages, I am sure that more and more mixed native and HTML applications are going to appear.
Brackets Architecture
Brackets consists of two applications running together - an
HTML/JavaScript application and a native C++ shell
that runs the HTML application. The native shell consists of two
processes - one running the HTML application inside a browser, and one
running the heavy native part like filesystem operations and number
crunching. The communication between the two processes is asynchronous,
using JavaScript callbacks and request / response for the native code.
The shell extends the standard HTML DOM with the following functions:
Open file or directory using the OS dialog
Read file
Write file
Create directory
Rename file or directory
Get file last modification time
List directory contents
Open and close browser for Live Preview
Open extensions directory in OS file browser
Open developer tools for brackets itself
Get the system default language
Time since the application start
Get the application support directory
Brackets Shell implements the asynchronous communication between the
HTML render process and the browser in a very simplistic way: JavaScript
executes a function appshell.fs.readdir(path, callback), the
render process stores the callback in a mapping from a request id to
callback and calls the native process with the name of the function.
When the native process is ready it sends back any result of the call
together with the request id. The HTML process finds the callback by the
request id and executes it with any result.
This architecture is the same as for any good GUI application - the UI never executes any expensive functions and is always responsive and all data manipulation is in a separate thread.
Running Brackets in Coherent UI
Making Brackets run inside Coherent UI is really easy. We start by
creating a view that loads www/index.html relatively to the executable.
To support Coherent UI we have to include some JavaScript files in the index.html
of Brackets
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
These scripts are Coherent UI dependencies, Coherent UI itself and the abstraction layer between all JavaScript code of Brackets and Coherent UI.
Then we have to register our native callbacks for the asynchronous
calls:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Brackets native functions always return an error code as first argument
to the JavaScript callback. This mechanism can be implemented in
Coherent UI, but then the callbacks always have to do two things -
handle the correct result and handle the error, which is kind of
annoying. engine.Call might take two callbacks
- one for the successful result and one for error. Therefore we have to
wrap the normal callback in an object with separate handlers for
success and error.
gist: javascript callback wrapper
All that is left now is to wrap the callbacks and use engine.Call
instead of native function.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Brackets has and some synchronous methods that return to JavaScript
immediately. These methods are:
Get the application support directory
Get the system default language
Time since the application start
Coherent UI does not support providing synchronous JavaScript
functions by design, so we will have to work around that.
The application support directory remains constant through a single run of Brackets, so we can set it once and for all during application initialization:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Getting the system default language might be implemented in the same way.
The last method left is time since the application start and is used only in Show Performance Data menu. This method might be implemented entirely in JavaScript, assuming Brackets is not going to be reloaded during the performance test run.
Porting Bracket to Linux
Since Coherent UI already runs on Linux and we have implemented most of the native
functions using the cross-platform boost::filesystem library, all we
have to do to get is
showing an open file dialog, creating and closing a chrome instance
opening an URL and folder in the default OS HTML browser and file manager.
We use the GtkFileChooserDialog and the xdg-open tool.
Unfortunately, the Live Preview doesn't work under Linux. Live Preview in Brackets creates a new Google Chrome instance with
enabled remote debugger, attaches to the debugger using XmlHttpRequest
and WebSockets and controls the instance via the debugger. What happens on Linux
is that one of the XmlHttpRequests fails with "DOM Exception" and the
debugger is unable to attach to the instance.
Another Linux related issue is that I couldn't find a way to close a Google Chrome tab on Linux gracefully, so when the developer tools are closed
you get the "Ow, Snap" page. In a future version we will stop using
Google Chrome for Live Preview and for showing the developer tools, which will fix this problem.
Here is a short video of Brackets running on Linux:
We want to share with you the first video of the integration of Coherent UI with Unity3D:
Coherent UI already support HUD-type views as well as views mapped to objects with input. The integration is designed to be very user-friendly yet powerful.
The demo shows the awesome CSS3 periodic table by Ricardo Cabello aka Mr. Doob and in Unity3D it looks even better.
Have you ever tried calling a C# method returning a structure from C++? This is not a common use of interoperability so you probably haven't, but let me tell you a story about shooting yourself in the foot.
Imagine the following scenario: you have a C++ library that exports an interface of callback methods that will be called by the library during the lifetime of the application. One of the interface's methods returns a struct (let's call them GetStruct() and SimpleStruct, respectively). So far so good. Now you want to port your library for .NET by making a wrapper of the C++ library. You make a managed counterpart of the C++ interface and SimpleStruct(with all the marshaling needed, if any) and you're done. Except that it doesn't (always) work.
Here's an example of a case when it doesn't. Let this be our C++ library (the interface part I was talking about is omitted for brevity)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Nothing special about this code, except some unnecessary typedefs and weird processing of the returned value in FireCallback but I'll get to that in a second. This will be the C# part of the program:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Again,some standard interop use. If you compile the library and executable you'd expect a 3 written in the console, but instead you get this:
Wait, what, stack corruption? Everything's fine before the call of g_MyCallback(x, y) but it somehow corrupts the stack. If you add a breakpoint in MakeResult in the C# code you'll notice something interesting.
The value of x is something funny and y is 1 instead of 2. It seems like the parameters are offset by one. And yet all the other methods of our imaginary interface returning primitive types work? This calls for some disassembly. Let's see what happens in g_MyCallback(x, y). 5C6C13B7 8B F4 mov esi,esp 5C6C13B9 8B 45 0C mov eax,dword ptr [y] 5C6C13BC 50 push eax 5C6C13BD 8B 4D 08 mov ecx,dword ptr [x] 5C6C13C0 51 push ecx 5C6C13C1 8D 95 20 FF FF FF lea edx,[ebp-0E0h] 5C6C13C7 52 push edx 5C6C13C8 FF 15 30 71 6C 5C call dword ptr [g_MyCallback (5C6C7130h)]
Ok, we push the x and y parameters and then push something else. The theory for offsetting the parameters by one seems correct. But why is the compiler doing this? Well, our method returns a struct by value and copying it isn't very effective, so the (Named) Return Value Optimization kicks in (it's applied even when compiling with /Od). In short, the last pushed parameter is the address where the returned value will be stored and no copying will occur. The C++ compiler is aware of this fact and works its magic. When we cross the language boundary to C#, however, the stack is broken. You'd expect that the CLR would know these things, and it does, but we hit a corner case. The rules for function return values can be found here. More specifically:
POD return values 32 bits or smaller will be returned in the EAX register.
POD return values 33-64 bits in size will be returned via the EAX:EDX registers.
Non-POD return values or values larger than 64-bits, the calling
code will allocate space and passes a pointer to this space via a hidden
parameter on the stack. The called function writes the return value to
this address.
(The bullets are points 12,13 and 14)
The C# compiler simply couldn't know if the C++ structure is a POD or not so it applies the rules for non-PODsand it doesn't expect the hidden parameter. With the mystery unveiled, we have the following options for making our scenario work:
Make the structure a POD. In the example we can do this by removing the constructor. It's the only thing breaking the POD-ness.
Change the signature of the callback in the C++ code so it returns an integral type of the same size. In other words, change "typedef SimpleStruct ReturnType" to "typedef int ReturnType". This way the compiler won't emit code for RVO. If you have a 64-bit structure, you can use long long.
Instead of return value, make the structure an output parameter.
Add bogus fields in the structure to make it larger than 64-bits.
The last option is the least desirable one and I added it for completeness. Since having a constructor is useful in some cases, I opted for the signature change in our project which lets us keep the non-POD parts and is hidden from the user. It's not the prettiest solution and you have to keep it in mind if you ever change the size of the structure but it works :).
Note that for x64 builds the stack won't be corrupted (when compiling with Visual Studio) because the first four integral or pointer parameters are saved in the RCX, RDX, R8 and R9 registers. In the example function we only have 2 parameters, so the hidden RVO parameter will go to a register (if the function had 4 or more arguments, then maybe we will corrupt the stack, if the compiler decides to push the additional arguments and not preallocate memory by modifying the stack pointer at the beginning of the function). The return value will be wrong though, because the compiler will generate code that interprets the returned value in RAX as an address, and not a value, so it will read the memory at that address. This can be fixed using the same solutions as the ones for a 32-bit build.
Our on-going integration effort in Unity3D prompted us to accelerate a feature we've been planning for a long time but didn't have the time to finish until now - support for client applications with a multi-threaded rendering architecture.
Although it is even now possible to incorporate Coherent UI in an application with multi-threaded rendering, it is inherently difficult because the rendering-related events must be performed in the thread that updates the system.
For the sake of simplicity I'll call the thread that performs the logic in the client app the 'update' thread and the one that renders the 'rendering' thread.
There were two major challenges we needed to beat in order to support a separate 'rendering' thread in the client - rendering resource management and draw callbacks.
When Coherent UI needs a rendering resource - usually a texture, it calls the appropriate callback provided by the user. This happens when a new View has to be created or a View gets re-sized. The API expected the result of the operation to be immediately available.
We changed it so that it now uses a 'SurfaceResponse' object that must be signaled with the result of the operation. This signaling can happen at any later time, so the resource could be created in a separate thread. This is analogous to the resource requests (usually file-reads or file-writes) we already support in the same manner.
All new surfaces are fetched while calling 'UISystem::FetchSurfaces' for buffered Views and 'View::FetchSurface' for on-demand Views. This results in calls to the 'ViewListener::OnDraw' callback provided by the user. Usually a copy is made of the received surface for rendering and up until now those methods had to be called in the 'update' thread. Now it is perfectly safe to call them from your 'rendering' thread. This not only allows for easy integration with multi-threaded rendering pipelines but could be used as a performance optimization, as it voids the need to make an eventual copy of the surface to be used later for rendering.
Note that the 'ViewListener::DestroySurface' could now be called from both the 'update' and the 'rendering' thread but Coherent UI has already relinquished any ownership on the surface when it calls the method so it's trivial to dispose it even when the callback happens in the 'update' thread.
The API changes will be available in the next version of Coherent UI.
We are proud to announce the official release of Coherent UI for .Net.
With Coherent UI game developers and UI artists can use standard modern HTML5, CSS3, and JavaScript to create user interface and interaction for their XNA, SlimDX or SharpDX games. Features like secure micro-transactions and in-game store, social networks integration are easy to implement using the full in-game browser that Coherent UI provides. In addition to incredible HUDs for XNA and SlimDX based games, Coherent UI gives you fully integrated browser controls for Windows Forms, WPF and Gtk#, so that you can integrate your desktop application with any social network, show YouTube videos or access a web service.
The major highlights are:
WinForms browser control - full HTML5 and CSS3 support with 3D transformations, HTML5 video and Flash support
YouTube running in a Windows Forms application
WPF browser control - integrated with XAML, without the complexity of using the WebBrowser control from Windows Forms
Editing the WPF browser control XAML in Visual Studio design mode
Gtk# browser control - create truly cross-platform managed applications with embedded browser
