Webcast: Realtime HMI-Design with VRED 2024

Autodesk VRED offers a convenient way of streaming almost any HTML content onto a geometry in your 3D-Scene. In VREDs Media Editor one can create a WebEngine that connects a web resource, like a website or a web app, with a material that can be places in the scene. This mechanism allows to integrate rich user interfaces or other dynamic information into a VRED scene. Most importantly it is possible to control the HTML content with a mouse, VR-controller or with hand gestures. This makes them very versatile and perfect for integrating HMIs in virtual cars.

Figma is a cloud-based design and prototyping tool that has revolutionized collaborative design workflows. With its intuitive interface, designers can create, share, and iterate on designs seamlessly. It is already widely adopted in large companies and used for developing interactive desig mockups and prototypes.

Figma includes a web preview that can be configured to directly stream in one of VREDs Webengines. The web preview uses a static URL that can be used over multiple sessions. Whenever a designer performs changes on a linked design, the changes are immediatelly reflected in the web preview and therefore VRED. This allows for editing HMI designs in real-time during VR review session, enabling designers, engineers and decision makers to discuss designs and quickly apply changes.

As any material and texture is projected onto geometry it is neccessary to prepare the 3D model to show HTML content correctly. This is done by creating clean and un-distored UV maps of your display geometry. When this is done, the outlines of this UV map can be exported and used in Figma as a basic template that shows where to place the display elements. For rectengular displays this is a straight forward process. When dealing with curved displays, or displays with a free form outline this becomes more difficult.

The next step is to create a webview in VREDs Media Editor, link the Figma Preview and link the webview with a material. Don't forget to use "UV-Mapping" as texture mapping.

As we already know, HTML interfaces in VRED can be controlled with a mouse, a VR-controller or with hand gestures when using for example a Varjio headset. Therefore we can navigate an interactive Figma prototype and control any aspect that the designers intended to. It would be nice to send information between VRED and Figma to update the state of an HMI with other information, for example a vehicle simulation that provides speed and navigation information, or a status about open doors.

Unfortunatelly this is currently quite limited, because Figma can not continuesly poll data from a server. Only when a user is interacting with a custom component Figma is able to update information and therefore get information from VRED.

On the other hand, sending commands or information from Figma to VRED can be achieved with simple external server that just relays commands. When pressing a button in Figma, a custom component can issue a command to a server, VRED polls this commands and acts on it.

There are other tools however that bridge the gap between real-time design tools and interactive prototyping platforms. ProtoPie enables teams to develop complex HMIs that share data and communicate with each other. They allow the integration of tools like Figma and enable real-time interactions with visualization tools like Autodesk VRED or Unreal Engine.

We will cover these topics more in-depth in our following insights.