From TLDR newsletter about Runway’s Solaris:
Solaris is an Interface World Model. It can generate real-time interactive interfaces frame by frame as users interact with it. Solaris handles rendering and interactions jointly, generating every frame and every response to user input, eliminating the need for an intermediate representation. The model opens up new ways of building websites, apps, and other online interfaces. It can also be used to train agents in much more dynamic environments.
Very cool demos!
The basic idea is to use a MLLM to generate prompts from user inputs across multiple modalities (including texts, direct manipulation, etc.) and the current image frame content, and use the prompts to generate the next frame via a fast video diffusion model.
The article heavily emphasizes the contrast with approaches that generate an intermediate representation (like code) which can be too lossy for more complex images (like natural photographs rather than simple layouts).
Because the “interface” can be generated entirely on the fly at the OS level, it can replace the traditional app-based approach and allow for more dynamic and personalized interfaces that orchestrate underlying modules/tools to meet the user’s needs and preferences in real-time.
How to connect this interface world model with real computation substrate can be an interesting next step (e.g., via an agentic orchestration layer that can translate the interface world model into actual computation and data retrieval).