Virtual Toy Interaction (Intel RealSense)

Introduction

I participated in the Intel RealSense App Challenge 2014 where I created a Unity game using the Intel RealSense depth sensing camera that allows the player to use physical toys as a controller in the game.

Original Concept

The original idea was to build a system where a user would interact real Lego toys with virtual ones in an Augmented Reality environment, but this was in 2014-2015, so the tools were not ready yet to build the original vision. I initially had the idea to have trackable lego minifigures and the user could move them around in a physical space, but with AR glasses, there would be an environment. The idea was essentially combining real legos with the digital environment. The user could virtually build objects or combine real lego objects in the digital world.

This is a concept video I made in July 2014 before the challenge where I made a mockup in Autodesk Maya 2014 of my original plan.

Background

I originally got 5 of my ideas accepted in the contest, and this was the original idea that I wanted to build. I learned of this contest after attending a meetup and asking one of the speakers about how to build this and he recommended applying for the competition. Some of my other project ideas were a virtual video communication hand tracking table tennis game (with AR), an invention educational experience, and a virtual conductor demo. I started building the table tennis demo and got some parts working, but after the project got corrupted a few times and I had problems with physics, I decided to just focus on the virtual toy interaction one.

Developing the Project

I first tried using 2D image trackers on the Lego spaceship, but the image tracking was not reliable enough, so I used handtracking instead. I created multiple versions, and decided to use the spaceship demo since it demonstrated the basic functionality and idea.

Final Submission

Here is my final video submission:

Development Tools

The app is built with Unity and the Intel RealSense depth camera. I used Autodesk Maya to 3D model the assets. This was the first project where I incorporated custom 3D models into my Unity project.

(Desk Easter Egg: The 3D models for the blue spaceship and the lego tie fighter are on my desk render in the bottom right corner. A 3D scan of my Intel RealSense camera is on the front left part of the desk.)