Dreamcrafter (In Progress)
Overview
For my masters thesis, I am trying to create a system based on the proposal I made in 2022 here. In the first semester (Fall 2023) of my masters program, I led a group in a VR/AR course that implemented a system prototype which leveraged NeRFs, 3DGS, and Stable Diffusion to create a VR interface for 3D photo-realistic content creation. We call our system Dreamcrafter, a Virtual Reality 3D content generation and editing system assisted by generative AI.
Integration of NeRF/3DGS and diffusion models into user-friendly platforms for 3D environment creation is still in its infancy. Editing of radiance fields and generative 3D objects is currently limited to text prompts or limited 2D interfaces. Current research in NeRFs and diffusion models is primarily focussed on enhancing image/reconstruction quality, and we aim to address the noticeable lack of exploration in the application of user interfaces designed for editing and controllability of these models and novel 3D representations.
The core of our approach is a VR-based system that allows users to interact with and manipulate 3D objects and environments in real-time to enhance the process of 3D creation and editing. Dreamcrafter involves two subsystems which leverage novel 3D representations and stable diffusion. The stable diffusion powered system assigns semantically mapped spatial tags to 3D primitive objects to generate stable diffusion previews of scenes. Our second subsystem leverages NeRFs and 3D Gaussian Splatting for rendering and editing of 3D photo realistic scenes. Dreamcrafter is designed to be simple to use, lowering the barrier to entry for users without extensive experience in 3D modeling, while still providing realistic output results.
Our current system is a proof-of-concept prototype which we developed within a month.
Prototype Demos
Here are some demo videos of our prototype systems that demonstrate a basic version of some of the key features/interactions.
Here is our in-class paper write-up on the system
Here is a NeRF/GS capture of our class poster demo:
Next steps
We are working towards a publication for UIST 24 and I am currently exploring incorporating other systems such as room based gpt LLM interactions for more intelligent selection and automatic object segmentation from an input NeRF/GS to edit specific objects.