Confessions of a researchaholic

From TLDR newsletter about Runway’s Solaris:

Solaris is an Interface World Model. It can generate real-time interactive interfaces frame by frame as users interact with it. Solaris handles rendering and interactions jointly, generating every frame and every response to user input, eliminating the need for an intermediate representation. The model opens up new ways of building websites, apps, and other online interfaces. It can also be used to train agents in much more dynamic environments.

Very cool demos!
The basic idea is to use a MLLM to generate prompts from user inputs across multiple modalities (including texts, direct manipulation, etc.) and the current image frame content, and use the prompts to generate the next frame via a fast video diffusion model.
The article heavily emphasizes the contrast with approaches that generate an intermediate representation (like code) which can be too lossy for more complex images (like natural photographs rather than simple layouts).

Because the “interface” can be generated entirely on the fly at the OS level, it can replace the traditional app-based approach and allow for more dynamic and personalized interfaces that orchestrate underlying modules/tools to meet the user’s needs and preferences in real-time.
How to connect this interface world model with real computation substrate can be an interesting next step (e.g., via an agentic orchestration layer that can translate the interface world model into actual computation and data retrieval).

Nowadays it is entirely possible to let these AI agents write a research paper and code from ground up based on some initial human instructions. But I think it is important to think about which parts to automate and which parts to keep in human hands.

Personally, these are the parts that I would be happy to delegate to AI:

• improve writing style and grammar from an initial paper draft
• help with literature survey, such as finding papers related to a topic
• critic my writing for clarity, especially on the algorithm descriptions

I would not want to delegate the parts that I need to do manually, at least the first few drafts, to help me think through the research:

• the ideation parts, such as initial paper title, abstract, and introduction outlining the problem and motivation
• the core approach, including algorithm description, pseudo code, architecture/pipeline illustration
• the limitations and future works

This post was inspired by PaperBanana for generating paper figures, which I think would be more suitable for results figures (like statistical plots) than conceptual figures (e.g., algorithm/pipeline illustration).