Figma Mirai Meets Augmented Reality

If you're sadden by the fact that Figma Mirai Suenaga is nothing more than an illusion, or if you've heard about the release of Dennou Figure ARis about a couple of months ago but couldn't bear to part with 9,800 yen (like me), here's your chance to have some fun with a virtual figure.



Based on the same Augmented Reality technology of ARis, I've implemented a virtual Mirai Suenaga - mascot character of dannychoo.com. This virtual Mirai is a lower polycount version of the Figma Mirai I've done previously. The application is called ARmirai.

The downside of ARmirai, however, is that you can't pose or molest... I mean... interact with the virtual figure other than move the marker around and rotate her using the left/right arrow keys.

Video taken with a Genius Look 110 webcam. Recorded using FRAPS on my sister's faster laptop. I apologize for the poor camera quality. I guess this is the kind of result I should expect from a budgeted webcam.




Download

• ARmirai (Build 20081223) - 4.49MB
• ARmirai (Build 20081226) - 4.49MB
  (Modified the video plug-in to read XML configuration file.)
• ARmirai (Build 20090105) - 4.48MB
  (Removed duplicate polygons in one of the model files.)
• ARmirai (Build 20090107) - 4.56MB
  (Included MSVCP71.DLL required for some systems.)
  

Webcam, marker boxes and Caramelldansen song not included.




License Agreement (EULA)

ARmirai is free for personal and non-commercial use, meaning users cannot sell, profit from, or commercialize the software. All 3D models that come with this package are copyrighted and cannot be redistributed without written consent of the author. By using the software, you accept the above terms. If you do not accept them, do not use the software.




Requirements

• Microsoft Windows 2000/XP/Vista
  
• Graphics card with shader support
  (Might be able to run without shader but program console will be flooded with error messages)
• 20MB of harddisk space
• USB Webcam using VFW or WDM drivers
  (Let me know if the Xbox LIVE Vision Camera works here)
• Printed markers (4x4 cm)

How to use ARmirai

1. Extract the ARmirai folder to any where in your PC.
2. Markers are located in the ../ARmirai/patterns subfolder. Open and print the GIF files using Microsoft Paint program at the default resolution, i.e. 96 dpi.
   
   Note: Height of the 3D figure is dependent on the marker size, i.e. 4x4 cm marker will display an approximately 13 cm tall figure.
   
   
3. Run ARmirai.exe file.
   
4. Configure webcam properties.
   
   
   
   If you do not see this setup, that means your webcam is not using VFW or WDM drivers and therefore not supported. Sorry.
   
5. Place the printed marker in front of your webcam and ta-daa~!
   
   
   
   Virtual Figma Mirai in comparison with my Figuremate Asuna ^^
   
6. Press "h" to toggle the help screen and "Esc" to quit.

Additional Programs

• osgviewer.exe
  
  ARmirai supports only IVE (OpenSceneGraph native binary) 3D format in order to reduce package size.
  
  The osgviewer program is used to view the IVE model files. See the user guide for more information.
  
  
  
  
  
• mk_patt.exe
  
  I've also included ARToolKit's mk_patt program in the package if you want to create your own custom marker. Please refer to the ARToolKit documentation on how to use it. You can also read the documentation by Team Bear3Media which is easier to understand.
  
  To register your custom marker with ARmirai, you'll need to edit the ARmirai.cfg file using a text editor. Append a line for the new marker, e.g:
  
  
  marker ID single;data/patt.mymarker;80;0;0
  
  
  • ID is an identifier.
  • patt.mymarker is the custom marker filename.
  
  
  The new marker line must be followed with a model entry, e.g.:
  
  
  model ID path/model.ive
  
  
  • path is the path where the 3D model resides.
  • model.ive is the 3D model file.
  
  
  Translate, scale and rotate settings are optional.
  
  Normally, you do not need to calibrate the webcam to use this application but if you have problem getting your camera to recognize markers, you might need to run the calib_camera2 program from ARToolKit (not included in ARmirai). You'll also require glut32.dll to run the ARToolKit program.
  

Troubleshooting

Problem: I can only see a blank screen after the camera property sheet setup.

Cause: When two or more video input devices (USB camera, software camera, capture card, and TV tuner, etc.) were connected with the PC previously, the ARToolKit video plug-in does not know which device to use.

Solution: Edit the camera configuration file to explicitly read the device name of the webcam.

1. Make sure the webcam is plugged in.
2. Open Windows Explorer and go to "My Computer".
3. Note down the device name of the webcam (excluding the # and number that follow).
   
   
   
   You can also use AMCAP, which comes with most webcam installations, to confirm the device name.
   
4. Navigate to the ../ARmirai/data/ subfolder.
5. Open WDM_camera_flipV.xml with a text editor.
6. Find and replace "PGR" with the device name of your webcam.
   
   
7. Save the XML file and give ARmirai another try.

Author's Notes

• 3D Modeling
  
  The model of Figma Mirai in ARmirai is recreated using 3ds Max 9 and has about 28,000 polygons, low enough to be used in real-time applications like this, and maybe even games.
  
  
  
  For some reasons, I had great difficulties exporting bone and skin animation in 3ds Max to 3D file formats other than Autodesk's. Most of the time, the animation is either missing or doesn't look right. Fortunately, I made Figma Mirai so that she only consists of rigid body parts.
  
  The 3ds Max exporter I used is a third-party program called OSGExp (version 0.9.4). Works great with the settings shown below.
  
  
  
  FK animation translates nicely but not IK. I had to do extra work to clone and rebind the legs in order to mimic IK animation. I actually made quite a few animation sequences but due to the above limitations, I ended up with only the Caramelldansen/Uma uma dance.
  
  
  
• Programming
  
  The application is created using the latest osgART 2.0 Release Candidate 3. It is a combination of OpenSceneGraph 2.4.0 and ARToolKit 2.72.1. OpenSceneGraph is not the latest version though. I could not compile the newer scene graph binaries on Microsoft Visual Studio 2005 due to missing dependencies. Luckily, changes in OpenSceneGraph 2.6.1 does not affect the workings of the application.
  
  I was not working with any 3D scene graph libraries initially since I thought I could make use of OpenVRML renderer supported by ARToolKit. However, the SimpleVRML example produced really weird clipping problems with the exported WRL files on my laptop.
  
  
  
  The WRL files looked fine though when I viewed them using a browser with the appropriate plug-in.
  
  
  
  View the VRML 2.0/97 model (3.25MB) HERE if your have a VRML plug-in installed in your browser. The weirdness is probably due to the outdated OpenVRML binary package that comes with ARToolKit. Not to mention VRML is ancient technology.
  
  Next, I tried the MQO loader for OpenGL/GLUT which is popular among the Hatsune Miku + ARToolKit demos on Nico Nico Douga. The MQO loader is straight forward and very easy to use for simple models. There is also a MQO importer/exporter for 3ds Max 9 that works like a charm.
  
  
  
  Sadly, the MQO format does not contain 3D animation data and 3D rendering looks flat when used with the loader in OpenGL. See example. That means you need to texture and shade the entire 3D model if you want it to look presentable. There is a newer(?) MqoLoader_080221 which renders gouraud-like facets but it's slow and crashes my system most of the time.
  
  Then I stumbled upon osgART 2.0. (I wonder why they have different web sites for version 1.0.) Although osgART lacks proper documentation as of this writing, it does a pretty good job rendering my exported models. So naturally, I started using osgART for this project.
  
  Too bad there is no OgreART. I personally prefer Ogre to OpenSceneGraph in terms of programming struture.
  

Other References

• Augmented reality - Wikipedia, the free encyclopedia
• 「攻殻機動隊」「電脳コイル」の世界を実現！ - ARToolKitを使った拡張現実感プログラミング