Voodoo Camera Tracker: A tool for the integration of virtual and real scenes

This non-commercial software tool is developed for research purpose at the Laboratorium für Informationstechnologie, University of Hannover. Permission is granted to any individual or institution to use, copy, and distribute this software, provided that this complete copyright and permission notice is maintained, intact, in all copies and supporting documentation. University of Hannover provides this software "as is" without express or implied warranty.

Overview

The voodoo camera tracker estimates camera parameters and reconstructs a 3D scene from image sequences. The estimation algorithm offers a full automatic and robust solution to estimate camera parameters for long video sequences. The results are useful for many applications like film production, 3D reconstruction or video coding. The estimated parameters can be exported to the 3D animation packages: 3D Studio Max, Blender, Lightwave, Maya, and Softimage.

The voodoo camera tracker works very alike to commercial available camera trackers (also called match movers). (e.g. Science D Visions 3D Equalizer, 2d3 boujou, RealViz Matchmover,...).

The estimation method consist of four processing steps:

• Automatic detection of feature points
• Automatic correspondence analysis
• Outlier elimination
• Robust estimation of the camera parameters

Automatic detection of feature points

The feature points are detected with sub-pixel accuracy using a corner detector. The detected feature points in the current image are marked with green crosses. Feature points that are detected as outliers during the processing are marked as red crosses.

Automatic correspondence analysis

The points can be matched from one image to the next by choosing matches which have the highest cross-correlation of image intensity for regions surrounding the points. The paths of the feature points are drawn in yellow here.

Outlier elimination

Due to misalignments some of the correspondences may be incorrect. To achieve a robust matching, a random sampling algorithm is employed to detect bad correspondences called outliers.

Robust estimation of camera parameters

Finally the parameters are estimated by using optimization technique applied to the good correspondences called inlier. Virtual objects are integrated into the real image.

Download

Usage

A typical camera tracking session includes the following steps

• Load an Image Sequence
• Set the Initial Camera parameter manually or load a camera file.
• Change some Controls concerning the tracking process.
• Start the tracking by pressing the Track button in the lower right corner.
• Check the Camera Parameter by using the Modelling Tools to insert 3D test objects, or examine the 3D scene with the 3D Scene Viewer.
• Export the estimated camera parameters to the 3D animation package 3D Studio Max, Blender, Lightwave, Maya, or Softimage.

Main Tracking Window

After typing the command >voodoo the Main Tracking window is shown (Fig. 1).

Figure 1. Main Tracking Window

The menu bar of this window has the following structure:

• File
  • Load
    • Sequence
    • Initial Camera
  • Save
    • 3D MAX Script
    • Blender Python Script
    • Lightwave 3D
    • Maya Script
    • Softimage 3.9 | XSI
    • CAHV Cameras
  • Exit
• View
  • Controls
  • Draw Options
  • Camera Parameter
  • 3D Scene Viewer
  • Modelling Tools
• Help
  • Help
  • Notify Level

Sequence

By choosing this item from the menu the Sequence Select window opens (Fig. 2).

Figure 2. Sequence Select window

Load a image sequence that is stored in several image files. The voodoo camera tracker is able to load every image format supported by the ImageMagick library

• Click Browse, a file browser opens.
• Select the file name of the image sequence you want to track.
  For example, if you want to open the sequence: "./sample_image_sequence/uni.##.jpg", you first choose the file folder named "sample_image_sequence", and then click on any file from the sequence, e.g. "uni.00.jpg". At last click Open.
  The voodoo camera tracker tries to detects all other images from the sequence automatically by replacing any number in the file name by a "#". Then it replaces the "#" by any possible number and checks if there is such a file in the directory. The voodoo camera tracker can handle numbering with and without leading zeros. If you use numbering without leading zeros, please choose a file with the highest number of digits in the file browser.
• The voodoo camera tracker fills the detected start and end sequence number as well as step number in the Start, End and Step text field. You can modify these numbers according to your needs. With step number 2 for example every second image will be skipped.
• In the field Deinterlace you can check one of three options. With not interlaced checked the image sequence will not be deinterlaced. The original data from the file is used for tracking. With the selection maintain even lines or maintain odd lines the image sequence will be deinterlaced accordingly.
• In the field Motion Type you can select two options. With the selection rotation the camera motion model takes only rotation and zoom operations into account. This motion model is valid for sequence taken with a camera on a tripod. If you select free move the camera must include a translational motion.

• Click OK and the selected sequence is loaded.

Initial Camera

By choosing this item from the menu the Initial Camera Parameters window opens (Fig. 3).

Figure 3. Initial Camera Parameter window

Set the initial camera parameters manually or load a camera file.

• Set the parameters in the text fields manually
  • The desired camera parameters can be filled in the text fields. Only the text field with white background color can be edited, the others with gray are read only.
  • In general it is not necessary to fill in all text fields. The two most important parameters are the Focal Length and the size of the Film Back, which are located in the widget called General. If you do not know the Focal Length it does not matter, the voodoo camera tracker is configured to estimate it by default - please see Controls.
  • In the widget called Advanced some additional parameters like Principal Point and Radial Distortion can be set. Leave these parameters to default values if you are not familiar with their meaning.
  • The widget called Position hold the translation and rotation of the camera according the the origin. The rotation is given by the angles pan, tilt and roll, which correspond to three successive rotations around the Y-, X- and Z-axes, respectively.
  • For each image of the sequence the voodoo camera tracker holds one camera parameter set that is linked to this image. By clicking OK all camera parameter sets of the image sequence are set according to those parameters.
• Load a CAHV camera file
  • The voodoo camera tracker uses the CAHV model introduced by Yakimovski and Cunningham. The parameter of these model are read from files which have the extension ".cam". For more information on the CAHV model see http://www.tnt.uni-hannover.de/project/3dmod/camera/central/cahv.html
  • To read a CAHV camera file first click Load in this window.
  • Select the file folder for example here named sample_image_sequence
  • Click the file folder calibrate in the next opened window.
  • Click on the file calibrate.cam and then the button Open.
  • The loaded parameters are shown in the widget.
  • For each image of the sequence the voodoo camera tracker holds one camera parameter set that is linked to this image. By clicking OK all camera parameter sets of the image sequence are set according to those parameters.

CAHV Cameras

By choosing this item from the menu the Saving Camera Files window opens (Fig. 4).

Figure 4. Saving Camera Files window

For each image of the sequence the voodoo camera tracker holds one camera parameter set that is linked to this image. These parameters can be stored in a CAHV camera files.

• The voodoo camera tracker suggests a file template.
• If you want to change this template you can use the Browse button. The voodoo camera tracker automatically replaces any number in the file name you choose by a "#".
• The voodoo camera tracker fills the start and end sequence number as well as step number in the Start, End and Step text field with the values used for tracking.
• By clicking OK the estimated camera parameter of the image sequence are saved.

3D Studio Max (MAX Script)

By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package 3D Studio Max using a Max Script file. This file has the extension *.ms. To run the script go to [MAXScripts->Run Script] in 3D Studio Max. The image sequence can be displayed as an environment map in 3D Studio Max:

• in the perspective viewport right-click "Perspective" and select [Views->voodoo_cam]
• select [Rendering->Environment]
• click the "Environment Map" button and select "bitmap" from the list
• enter the file name of the image sequence
• activate the "Sequence" checkbox
• select [Views->Viewport Background]
• activate the "Use Environment Background" and the "Display Background" checkbox and select "voodoo_cam" in the "Viewport" popup list

Figure 5. Export camera parameters to 3D Studio Max

Blender Python Script

By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Blender using a Python Script file. This file has the extension *.py. Load the python script into Blender's text editor and execute the script with ALT-P. Select the "voodoo_render_cam" camera and press Ctrl-Numpad 0 to activate it. To display the image sequence as a backbuffer in Blender, go to [Buttons Windows->Scene->Render Buttons->Output] and enter the file name of the image sequence in the "backbuf" field. A '#' at the end of the file name is replaced by the current (four-digit) frame number. We tested the import with Blender Version 2.42a.

LightWave 3D

By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Lightwave 3D using a LightWave Scene file. This file has the extension *.lws. To open the file, go to [File->Load->Load Scene] in Lightwave. The image sequence can be displayed as a background image in Lightwave:

• select [Image Editor->Load]
• enter the file name of the image sequence and activate "Sequence" in the "Image Type" popup list
• open [Scene->Backdrop->Compositing] and select the image sequence in the "Background Image" popup list

Maya Script

By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Maya using a Maya Script file. This file has the extension *.ma. To run the script go to [File->Open Scene] in Maya. The image sequence can be displayed as an environment map in Maya:

• select the "VoodooCam_Shape" object
• open the "Environment" field and create an image plane
• the "Image Plane Attributes" window opens and you can enter the file name of the image sequence
• activate "Use Image Sequence" checkbox and adjust the "Frame Offset"
• in the "Placement" window select fit "To Size"
• select [Panels->Perspective->VoodooCam]

Softimage 3.9 | XSI (3.0)

By choosing this item from the menu the Voodoo Camera Tracker exports the estimated camera parameters to the 3D animation package Softimage 3D using XSI File Format V3.0. This file has the extension *.xsi and can be imported to Softimage 3.9 or Softimage XSI. To import the file go to [Tools->Import->Objects->XSI Import] in Softimage 3.9 and in XSI go to [File->Import->Input .xsi].

Figure 6. Export camera parameters to Softimage 3.9

Exit

By choosing this item from the menu the Voodoo Camera Tracker will be closed.

All data, that is not saved, will be lost.

Controls

By choosing this item from the menu the Main Control Panel window opens (Fig. 7).

Figure 7. Main Control Panel window with selected Harris detector

The camera parameters estimation (camera tracking) consists of four basic steps: Detection, Correlation, Outlier Elimination and Estimation. For every step the Voodoo Camera Tracker offers different algorithms. You are free to choose different combination of algorithms for every step. For example you can use the Harris corner detector, Susan or our own detector called DLineCorner in the detection step. Some algorithms are in development state and therefore disabled in this beta version of the Voodoo Camera Tracker. If you are not familiar with the algorithms you can leave everything to default configuration or play around with some parameters. In case of trouble feel free to contact us for support.

With the Main Control Panel you can choose the combination and parameter of the algorithms you want.

• By clicking on the box in front of the yellow square a list of algorithms will be shown.
• An algorithm is selected if a dot is putted in the box in front of the respective algorithm name and the Apply or OK button is pressed.
• If you click once on the algorithm name the parameter widget of the algorithm is shown on the left side.

Draw Options

By choosing this item from the menu the Draw Options Deck window opens (Fig. 8).

Figure 8. Draw Options Deck window

There are four button groups in this window

• 2D Flags

  The 2D Flags influence the 2D drawings which overlay the actual image in the Main Tracking Window. In the following some important terms are described:

  • Inlier

    Feature points or correspondences which are found reliably in the outlier elimination step. These feature points or correspondences are used in the final estimation step.

  • Outlier

    Feature points or correspondences which are found unreliably in the outlier elimination step. These feature points or correspondences are neglected in the final estimation step.

  • FPoints

    Abbreviation for feature points.

  • FCorrs

    Abbreviation for (feature) correspondences.

  • Search Win

    Search window for the establishment of the correspondences between two successive image use in the correspondence analysis step

• 3D Flags

  Enables/disables the drawing of three dimensional virtual objects into the image sequence.

• Change Color

  The color shown on the color button is the current color of the respective object. Click on the color button and select the color you want in the color dialog.

• Polar Raster

  Click on the button Enable and a Polar Raster appears in the Main Tracking Window.

Fig. 9 shows an example of tracking results. Inlier are marked as green crosses and outlier as red crosses. The paths of feature points are drawn in yellow. One feature point is selected, marked as blue cross and the belonging search window is drawn in orange.

Figure 9. Detected feature points and their paths

Camera Parameter

By choosing this item from the menu the Camera Parameter window opens (Fig. 10.).

Figure 10.Camera Parameter window

This window has the same items as the Initial Camera window expect the Load button. For each image of the sequence the voodoo camera tracker holds one camera parameter set that is linked to this image. By changing the displayed image in the Main Tracking Window the parameter shown in the Camera Parameter window change accordingly. Hence you have access to every parameter set of the sequence and can change the parameters according to your needs. Changes are applied to the stored camera parameter set by pressing the OK or Apply button.

3D Scene Viewer

By choosing this item from the menu the 3D Scene Viewer window opens (Fig. 11.).

Figure 11. 3D Scene Viewer window

In this viewer a 3D scene is displayed with a global 3D coordinate system. One unit of the coordinate system raster equals the focal length given by the actual camera parameter set. By pressing a key and a mouse button you can perform to following movements to explore the scene.

The Options menu of the 3D Scene Viewer window contains a Draw Options window where you can choose whether a 3D object is drawn or not and what color is used for drawing.

Modelling Tools

By choosing this item from the menu the Modelling Tools toolbox is displayed (Fig. 12.).

Figure 12. Modelling Tools toolbox

With these modelling tools you can quickly place simple primitives into the scene and test your estimated camera parameters.

First select a primitive from the "Create" toolbox. You can place a primitve easily by using the "2d anchor" tool button. If the "2d anchor" tool button is selected and you move the mouse over a vertex of the primitive, it will be highlighted (see Fig. 12.). You can now move the vertex to the correct position in the image by clicking and dragging with the left mouse button. The translation, rotation and scale of the primitive is changed automatically as the vertex is moved. With the (T)-, (R)-, and (S)- buttons, which will appear wenn you select the "2d anchor" tool, you can (de)activate the estimation of the primitive's (T)ranslation, (R)otation, and (S)cale parameters. To place the vertex more exactly you can use the "zoom" tool button. If you want to delete an object press the DELETE key.

In Fig. 13. you can see the result of two fitted cubes. You can download this sequence from here, and try yourself.

Figure 13. Fitted Cubes

Help

By choosing this item from the menu this manual opens.

Notify Level

By choosing this item from the menu the Notify Level window opens (Fig. 14).

Figure 14. Notify Level window

By choosing the notify level you can change the amount of information printed to the console, while the program is running.

FPoint Editor

Figure 15. FPoint Editor window

The FPoint Editor window displays the parameters of a feature point. This window is displayed, if you select a feature point by using the "selection" tool button of the Modelling Tools

Bugs and Hints

Please send bugs and hints to Thorsten Thormählen (http://www.tnt.uni-hannover.de/~thormae) or Hellward Broszio (http://www.tnt.uni-hannover.de/~broszio). Your experience report is important to us in order to improve the software according to your needs. Do not hesitate to ask for support in case of any trouble with this software.

	Page maintained by: Webmaster Last updated: Feb/13/2007