Tuesday, December 23, 2008

Figma Mirai Meets Augmented Reality

If you're sadden by the fact that Figma Mirai Suenaga is nothing more than an illusion, or if you've heard about the release of Dennou Figure ARis about a couple of months ago but couldn't bear to part with 9,800 yen (like me), here's your chance to have some fun with a virtual figure.



Based on the same Augmented Reality technology of ARis, I've implemented a virtual Mirai Suenaga - mascot character of dannychoo.com. This virtual Mirai is a lower polycount version of the Figma Mirai I've done previously. The application is called ARmirai.

The downside of ARmirai, however, is that you can't pose or molest... I mean... interact with the virtual figure other than move the marker around and rotate her using the left/right arrow keys.


Video taken with a Genius Look 110 webcam. Recorded using FRAPS on my sister's faster laptop. I apologize for the poor camera quality. I guess this is the kind of result I should expect from a budgeted webcam.




Download
  • ARmirai (Build 20081223) - 4.49MB
  • ARmirai (Build 20081226) - 4.49MB
    (Modified the video plug-in to read XML configuration file.)
  • ARmirai (Build 20090105) - 4.48MB
    (Removed duplicate polygons in one of the model files.)
  • ARmirai (Build 20090107) - 4.56MB
    (Included MSVCP71.DLL required for some systems.)
Webcam, marker boxes and Caramelldansen song not included.




License Agreement (EULA)

ARmirai is free for personal and non-commercial use, meaning users cannot sell, profit from, or commercialize the software. All 3D models that come with this package are copyrighted and cannot be redistributed without written consent of the author. By using the software, you accept the above terms. If you do not accept them, do not use the software.




Requirements
  • Microsoft Windows 2000/XP/Vista
  • Graphics card with shader support
    (Might be able to run without shader but program console will be flooded with error messages)
  • 20MB of harddisk space
  • USB Webcam using VFW or WDM drivers
    (Let me know if the Xbox LIVE Vision Camera works here)
  • Printed markers (4x4 cm)



How to use ARmirai
  1. Extract the ARmirai folder to any where in your PC.
  2. Markers are located in the ../ARmirai/patterns subfolder. Open and print the GIF files using Microsoft Paint program at the default resolution, i.e. 96 dpi.

    Note: Height of the 3D figure is dependent on the marker size, i.e. 4x4 cm marker will display an approximately 13 cm tall figure.

  3. Run ARmirai.exe file.
  4. Configure webcam properties.



    If you do not see this setup, that means your webcam is not using VFW or WDM drivers and therefore not supported. Sorry.
  5. Place the printed marker in front of your webcam and ta-daa~!



    Virtual Figma Mirai in comparison with my Figuremate Asuna ^^
  6. Press "h" to toggle the help screen and "Esc" to quit.



Additional Programs
  • osgviewer.exe

    ARmirai supports only IVE (OpenSceneGraph native binary) 3D format in order to reduce package size.

    The osgviewer program is used to view the IVE model files. See the user guide for more information.




  • mk_patt.exe

    I've also included ARToolKit's mk_patt program in the package if you want to create your own custom marker. Please refer to the ARToolKit documentation on how to use it. You can also read the documentation by Team Bear3Media which is easier to understand.

    To register your custom marker with ARmirai, you'll need to edit the ARmirai.cfg file using a text editor. Append a line for the new marker, e.g:

    marker ID single;data/patt.mymarker;80;0;0

    • ID is an identifier.
    • patt.mymarker is the custom marker filename.


    The new marker line must be followed with a model entry, e.g.:

    model ID path/model.ive

    • path is the path where the 3D model resides.
    • model.ive is the 3D model file.


    Translate, scale and rotate settings are optional.

    Normally, you do not need to calibrate the webcam to use this application but if you have problem getting your camera to recognize markers, you might need to run the calib_camera2 program from ARToolKit (not included in ARmirai). You'll also require glut32.dll to run the ARToolKit program.



Troubleshooting

Problem: I can only see a blank screen after the camera property sheet setup.

Cause: When two or more video input devices (USB camera, software camera, capture card, and TV tuner, etc.) were connected with the PC previously, the ARToolKit video plug-in does not know which device to use.

Solution: Edit the camera configuration file to explicitly read the device name of the webcam.
  1. Make sure the webcam is plugged in.
  2. Open Windows Explorer and go to "My Computer".
  3. Note down the device name of the webcam (excluding the # and number that follow).

    My Computer

    You can also use AMCAP, which comes with most webcam installations, to confirm the device name.
  4. Navigate to the ../ARmirai/data/ subfolder.
  5. Open WDM_camera_flipV.xml with a text editor.
  6. Find and replace "PGR" with the device name of your webcam.

    Editor
  7. Save the XML file and give ARmirai another try.



Author's Notes
  • 3D Modeling

    The model of Figma Mirai in ARmirai is recreated using 3ds Max 9 and has about 28,000 polygons, low enough to be used in real-time applications like this, and maybe even games.



    For some reasons, I had great difficulties exporting bone and skin animation in 3ds Max to 3D file formats other than Autodesk's. Most of the time, the animation is either missing or doesn't look right. Fortunately, I made Figma Mirai so that she only consists of rigid body parts.

    The 3ds Max exporter I used is a third-party program called OSGExp (version 0.9.4). Works great with the settings shown below.



    FK animation translates nicely but not IK. I had to do extra work to clone and rebind the legs in order to mimic IK animation. I actually made quite a few animation sequences but due to the above limitations, I ended up with only the Caramelldansen/Uma uma dance.


  • Programming

    The application is created using the latest osgART 2.0 Release Candidate 3. It is a combination of OpenSceneGraph 2.4.0 and ARToolKit 2.72.1. OpenSceneGraph is not the latest version though. I could not compile the newer scene graph binaries on Microsoft Visual Studio 2005 due to missing dependencies. Luckily, changes in OpenSceneGraph 2.6.1 does not affect the workings of the application.

    I was not working with any 3D scene graph libraries initially since I thought I could make use of OpenVRML renderer supported by ARToolKit. However, the SimpleVRML example produced really weird clipping problems with the exported WRL files on my laptop.



    The WRL files looked fine though when I viewed them using a browser with the appropriate plug-in.



    View the VRML 2.0/97 model (3.25MB) HERE if your have a VRML plug-in installed in your browser. The weirdness is probably due to the outdated OpenVRML binary package that comes with ARToolKit. Not to mention VRML is ancient technology.

    Next, I tried the MQO loader for OpenGL/GLUT which is popular among the Hatsune Miku + ARToolKit demos on Nico Nico Douga. The MQO loader is straight forward and very easy to use for simple models. There is also a MQO importer/exporter for 3ds Max 9 that works like a charm.



    Sadly, the MQO format does not contain 3D animation data and 3D rendering looks flat when used with the loader in OpenGL. See example. That means you need to texture and shade the entire 3D model if you want it to look presentable. There is a newer(?) MqoLoader_080221 which renders gouraud-like facets but it's slow and crashes my system most of the time.

    Then I stumbled upon osgART 2.0. (I wonder why they have different web sites for version 1.0.) Although osgART lacks proper documentation as of this writing, it does a pretty good job rendering my exported models. So naturally, I started using osgART for this project.

    Too bad there is no OgreART. I personally prefer Ogre to OpenSceneGraph in terms of programming struture.



Other References

49 comments:

  1. sooo cute! ^^ but I dont have a webcam oh..

    ReplyDelete
  2. That's really cool! You've done a really impressive work.

    ReplyDelete
  3. Hmmm it doesn`t work for me.
    I have Gembird pc cam and there seems to be no problem with the Camera settings "test". so I don`t get it. if the problem isn`t in my camera than please tell me where it is. And please specify what do you mean by "in foront of the camera" Thank you in advance.

    ReplyDelete
  4. ARmirai can only work with webcams that use VFW or WDM drivers. Diagnostic tools will test with whatever drivers that come with the webcam installation. To check if your webcam uses VFW or WDM drivers, try running Willing Webcam.

    http://www.willingsoftware.com/

    If the required drivers are present, you should see a fullscreen camera output window once you clicked OK on the property sheet in step 4 of "How to use ARmirai". Otherwise, you see only a blank screen.

    "Place the printed marker in front of your webcam" as if you're a photographer holding the webcam to shoot a picture of the square marker, just like what I was doing in the Youtube video.

    Hope this clarify your doubts.

    ReplyDelete
  5. willing connects my webcam when I select webcam (VFW or WDM)
    but after settings page on ARmirai I saw only a black blue screen. I tried calib_camera2 and it reported that I missed glut32.dll
    so, do I actually need to have Visual Studio .NET installed to run ARmirai?
    I don't think I did anything wrong, though... (I borrow this webcam from my family. should be a VFW lenovo webcam)

    ReplyDelete
  6. Visual Studio .NET is not required to run ARmirai. If you see a dark blue blank screen after the camera property sheet, it means that ARmirai could not detect the webcam properly. Both calib_camera2.exe and glut32.dll are of no use if there is no video signal.

    After some research, it seems like you are facing the same problem discussed on the Japanese site in my "Other References".

    As explained on the Japanese site, when two or more video input devices (USB camera, software camera, capture card, and TV tuner, etc.) were connected with the PC previously, you might get a blank screen because the video plugin does not know which device to use.

    I did not write the video plugin so I'm not exactly sure what went wrong. I now modified the video plugin to read the device name manually.

    Please help me test if this new plugin works:

    1. Download and overwrite your existing ARmirai folder.
    http://project.ionized.googlepages.com/ARmirai_manual_video_plugin.rar

    2. Make sure your webcam is plugged in.

    3. Open Windows Explorer and go to "My Computer".

    4. Note down the device name of your webcam (excluding the #2 or whatever number).
    See http://www1.bbiq.jp/kougaku/image/my_comp_device_check.png
    You can also use Willing or AMCAP to confirm the device name.

    5. Navigate to ../ARmirai/data/ folder.

    6. Open WDM_camera_flipV.xml with a text editor.

    7. Replace "PGR" with the device name of your webcam.

    8. Save the XML file and try ARmirai again.

    ReplyDelete
  7. I'm not really sure how to test that plugin, but isn't that the same dll file as the one in ARmirai\osgplugins-2.4.0 folder?

    Anyway yes I have a TVTuner installed and after setting the xml file ARmirai works :)
    thanks

    ReplyDelete
  8. Yes, the plugin I referred to is the new DLL file.

    Glad to hear it works! Thanks for all the feedback.

    ARmirai now repackaged to include the modified plugin.

    ReplyDelete
  9. thenk you verry much for the quick reply (I am the first annoymous) And now a stupid question if you would forgive me: Where am I supposed to see the mirai figma. on my ps screen or as a hollogram or something like that

    ReplyDelete
  10. You'll see the virtual figure on your monitor screen or any visual display unit you can connect to your computer. See the Dennou Figure ARis example.

    Augmented reality is not exactly the same as the holograms you'd see in the Star Wars movies. You'll probably need special equipment such as this or this to view 3D holographic images.

    If you like anime, a show called Dennou Coil envisions the use of augmented reality technology in the future very nicely.

    ReplyDelete
  11. Could you please tell me how you downloaded the source code of osgart 2.0 because I have some problems doing so?

    ReplyDelete
  12. Hi,

    To the poster above: you can download the SDK (including source code) for osgART 2.0 here: http://www.osgart.org/downloads

    It's cool to see people using osgART. We are currently adding documentation and examples to the osgART website to show how you can create applications like the one shown in this blog.

    Good work!

    ReplyDelete
  13. Thank you Julian!

    I visited this link yesterday, but I couldn't find the source code..

    It seems that there is a problem with firefox, since you can't see the attachment..
    When using IE there is no such problem..

    Keep up the good work,

    Dimitris

    ReplyDelete
  14. Hi, downloaded the zip file you provided and tried to run ARmirai.exe but it needed a MSVCP71.dll file that was missing. So I had to download that from someother place in order to be able to run the software. Same problem for others aswell?

    ReplyDelete
  15. Thanks for the feedback Donut. I'll include the DLL file in the next build.

    I ran ARmirai on 3 machines and all worked fine but it's possible the DLL file was installed on them from other applications previously.

    ReplyDelete
  16. Or my computer trying to be "Special" as it sometimes behave weird.

    Btw tried to use your figure here, my crappy cam had problem with the other figures so I ran the one with neko pad on

    http://www.youtube.com/watch?v=KPzmEpjpPgs

    There's my try. I wonder how to cam it inside, as you see I use a secondary cam to get it :O.

    Another question that I might just have to try to find out. Can you use a DV Camera? And why'sthere no godzilla yet :D hehe
    Cheers!

    ReplyDelete
  17. Nice big monitors Donut :)

    To record a video without a secondary cam, use FRAPS or any video capture software for games.

    I've seen people using FireWire DV cameras with ARToolKit (ARmirai uses the same ARToolKit video library). Unfortunately, I do not know the details on how to get such cameras to work with ARmirai. My guess is you'll probably need DirectShow and the appropriate camera drivers.

    Errr.... what Godzilla?

    ReplyDelete
  18. Thanks for FRAPS!

    Hahaha wouldn't it be intresting with like a giant monster or something else coming out from the small paper piece :P

    Is there anyone who have assembled a program where one figur does something? Saw a Miku Hatsune on youtube that reacted to a leek and started to run after it, wonder if that source code is out or something.

    ReplyDelete
  19. Well, you can change some settings in ARmirai.cfg to get a life-size Mirai if you want to. lol

    I've seen the video you've mentioned. Somehow, I suspected her actions are scripted unless someone programmed ARToolKit to recognize a leek?

    ReplyDelete
  20. They(the Japanese authors) use a tool called Metasequoia(ummm it's commercial ware) to input/edit/pack and output their 3D models, ARtoolKit Miku is the example.

    my uma dancing test vid:
    http://www.youtube.com/watch?v=B93nVrHDAqE

    It seems the animation data was a bit big and difficult to edit cos it wasn't only a mecro but frames of action like 3D pre-rendering anime does, not just script or any programming works, it's more like something in MikuMikuDancing, but here was (per frame) per 3d model file(include bones)?

    ReplyDelete
  21. MQO I mentioned in the blog above is, in fact, short for Metasequoia. There is also a freeware version of this modeling software with limited functionality though.

    I understand what you're trying to say. MQO files do no contain animation data. Thus, if you use the MQO loader, separate model files are required for each frame of animation. MQO does not have support for bones too. You'll need to write functions to take care of loading and rendering of the model files in ARToolKit, as explained in the last YouTube video in this blog. Not the best method but at least it works.

    I would suggest rewriting the rendering function if you are to use the MQO loader because the one shown has poor control over the video frame rate.

    BTW, nice Miku uma dance.

    ReplyDelete
  22. RokDeBone2, this is what they use, that's how *.quo files come from.

    Example: uma/1/miku_xx_RokDeBone2用モーションファイル_ウマウマ.qua

    open and unpack it with RokDeBone2, you will get tons of *.mqo files, that was the source animation data.

    So, does it mean we can edit them in RokDeBone2? I didn't try this yet.

    And yes, the program may sucked when the big mount of MQOs rushing in, as the author said in his vid description and readme file.

    While that wasn't the only way(ARfigure ARis has its way) to make miku or anything move and dancing, but I didn't find any source code(script only) from some vids on nico(keywords: ARToolKit). I think you have already seen that vid, multi-miku dancing animation, not the same author there. If that was some mount of MQOs, god!

    ReplyDelete
  23. I thought there's a question, why the script can't render QUA directly? If the program can only render each frame which was already existed in the queue, that may be the answer, cos QUA doesn't contain any frames, it just include nude model and position data. So, RokDeBone2's job is a frame maker, the rest thing is just like making a traditional 2D anime.

    After watching some MikuMikuDance works, I thought "well, is that the way they make the source data?", and soon someone said he had tried input MMD data into ARTool, but I didn't find any results on it.

    ReplyDelete
  24. It would definitely be interesting if someone has written software libraries in C/C++ to read RokDeBone2(QUA) or MikuMikuDance(MMD) animation data files and use that data to animate 3D models in OpenGL directly. Not an easy task I must add.

    ReplyDelete
  25. I hope someone is able to answer this question. I current have a vista and a webcam that uses a VFW or WDM driver I'm not sure which (The webcam is a LifeCam VX-3000) In any case I am able to get the program to work and see everything just fine on the screen except I am not able to get the figure to appear. Is there something that I am doing wrong?

    ReplyDelete
  26. Other factors may prevent the virtual figure from appearing as well. I would suggest checking ARmirai's console window for error messages. (Press F to switch to window mode when the webcam is running and you should see a DOS console window behind.) If the console window is clean, make sure the printed marker is not obscured and can be seen clearly on your computer display. Also note that light and camera angle can affect image quality.

    If that doesn't help, do a screen capture with FRAPS, share the screenshot on ImageShack, and I'll try to figure out your problem.

    ReplyDelete
  27. Thanks for the tips :D But it seems that all it took was to reinstall my webcam... so ARmirai does work on vista :D thanks again!

    ReplyDelete
  28. Hey, thanks for the feedback. Good to know ARmirai works with Vista. :)

    ReplyDelete
  29. At me it is impossible to use Mobiola Web Camera (Nokia N81). It is possible to connect it somehow?

    ReplyDelete
  30. I do not know what webcam driver is Mobiola running but chances of getting your Nokia N81 to work with ARmirai is low. Just connect your mobile to your PC with your Nokia software as you would have done normally when using Mobiola Web Camera. Run ARmirai and give it a try.

    I have a Windows Mobile phone with a web camera software as well. Unfortunately, the ARToolKit video library is not written to recognize the video stream from my phone.

    ReplyDelete
  31. Hello!

    Yours AR dools are sooo cute *-*
    I was astonished with the video.

    Could you teach me how I make a hair like yours?

    Thank you for any help...

    xoxo

    ReplyDelete
  32. Like the rest of the body, I used the box modeling technique to create the hair (including the pigtails). Which means, I create a primitive box object, convert it to a poly, subdivide the faces, and slowly shape the box by moving the vertices/edges/faces.

    I won't be writing a tutorial on this because there are already tons of them on the Internet. Googled and found some for your reference:
    http://www.evermotion.org/tutorials/modeling/trex/index.htm
    http://www.3dsmaxtutorials.net/modelling-tutorials/box-modelling-a-cat
    http://www.cgarena.com/freestuff/tutorials/harshvideos/
    http://www.youtube.com/watch?v=PNfT6UfX4hw
    http://www.youtube.com/watch?v=ztJ0nYPB_rI

    ReplyDelete
  33. Hi..is possible add interactivity..maybe to shake the girl dance in other way or make something?...thanks

    ReplyDelete
  34. hey, sorry to bother but the download link seems not working rite for me.. its always stop downloading at 1.17 Mb.. any solutions?

    ReplyDelete
  35. You could try clearing your browsing cache or getting it from MediaFire.

    ReplyDelete
  36. tarotaro did implement some on is blog just check it:
    http://flash.tarotaro.org/blog/2010/02/02/flar-mmd-dancing-ar/#more-42

    yhe great about tarotaro

    ReplyDelete
  37. wait a second? that character, that pose, that program, do you make that character dance the Caramelldancen? and can't believe it, may you be the creator of some of the videos in YouTube?

    ReplyDelete
  38. Hai I am not able to view my webcam name in my laptop what should i do that to get name

    help me out

    thanks

    ReplyDelete
  39. I do not know if laptop integrated webcam will work with this program or not. Sorry, can't help.

    ReplyDelete
  40. hi

    Is there any way to make this application to work in laptop with integrated webcam

    ReplyDelete
  41. Unless you know how to rewrite drivers for integrated webcam to work with VFW or WDM.

    ReplyDelete
  42. hi, think its the late hours for me i'm kinda lost halfway thru ur post.. did u use vrml as ur model's file format? i tried using bones to animate using 3ds max but the bones are detached from the model, moving on its own, after exporting it.. how did u manage to animate your model?

    ReplyDelete
  43. I did try to export my 3D model to VRML format initially but I quickly dropped the idea due to many problems I encountered - one problem being what you mentioned - bone animations going haywire in VRML. Eventually, I removed all the bones, FK linked all body parts and exported to osgART's model format for this project.

    ReplyDelete
  44. pardon my noobie-ness, what's FK linked? and does changing to osgART affect the ARToolkit codes? bcos i've alrdy had most of the codes out and lacking models that can work..
    using osgART, is it exportable from 3ds max? and what format will that b?

    ReplyDelete
  45. FK stands for forward kinematics. Animating a 3D model, say a humanoid arm or leg linked by FK chains requires a lot more effort as opposed to IK (inverse kinematic) chains commonly used in biped rigs.

    osgART is just a graphics library that utilizes the open source OSG toolkit to work with the ARToolKit library. The core algorithm in ARToolKit remains the same but higher level codes like the 3D model manipulation/translation codes are slightly different in osgART if I recall correctly.

    The 3ds Max -> OSG exporter is mentioned in my write-up above.

    I like to re-emphasize that the 3D model in this project consists of rigid body parts - think industrial robotic arms. If you're working on a single-poly deformable organic model, you may still experience problems when exporting animations using my solution.

    By the way, I'm afraid osgART is getting old. There has been no updates on osgART since I last visited their official site. You may want to look for more updated graphics libraries that work better with ARToolKit.

    ReplyDelete
  46. Hai

    How to run this on my 64 bit system

    ReplyDelete
  47. can you make a tutorial how to make this program.
    I have try,,, but I can't add interactivity, like adjust scale,transalation, or rotate by keyboard...

    ReplyDelete