We propose a method for editing 3DGS scenes with text-instructions in a method similar to Instruct-NeRF2NeRF. Given a 3DGS scene of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to accomplish more realistic, targeted edits than prior work.

We present a pipeline for integrating NeRFs into traditional compositing VFX pipelines using Nerfstudio, an open-source framework for training and rendering NeRFs. Our approach involves using Blender to align camera paths and composite NeRF renders with meshes and other NeRFs, allowing for seamless integration of NeRFs into traditional VFX pipelines. Shown in the CVPR 2023 Art Gallery. View more NeRF VFX renders (here)

In this work, we identify problems associated with interaction carried out with large groups of users. To address this, we introduce an additional user role: the co-host. They mediate communications between the VR user and many spectators. To facilitate this mediation, we present StreamFunnel, which allows the co-host to be part of the VR application's space and interact with it.

We propose WebTransceiVR, an asymmetric collaboration toolkit which when integrated into a VR application, allows multiple non-VR users to share the virtual space of the VR user. It allows external users to enter and be part of the VR application's space through standard web browsers on mobile and computers. WebTransceiVR also includes a cloud-based streaming solution that enables many passive spectators to view the scene through any of the active cameras. We conduct informal user testing to gain additional insights for future work.

Unity toolkit for on-device 3D Gaussian Splatting rendering and interactions design for AR and VR applications compatible with standalone devices including Meta Quest 3 and Apple Vision Pro. We introduce mesh-augmented splats to enable physics and lighting interactions with splats by auto-generating a mesh from the input splat.

Datasets for radiance field training (3DGS and NeRFs) of dozens of captures primarily of buildings of UC Berkeley and Princeton University, including exteriors and interiors. These datasets are composed of captures of videos (iPhone), images (captured via Sony a7iii), and fisheye images (Insta360 X4 and Samsung Gear360). Online VR splat gallery coming soon.

Since Jan 2023, I have been contributing features to the Nerfstudio system including the Blender VFX add-on and VR180/Omnidirectional (VR 360) video/image render outputs.

Various projects and explorations from over the years in VR/AR, 3D, and more.