---

Tracking algorithm

The aim of CaraPost’s automatic tracking algorithm is to track each 3D marker in the marker set from one frame to the next, tracking the 3D positions of as many marker points as possible through the whole take. It does this by matching blob appearance in each camera view from one frame to the next.

The algorithm is summarized below. For each marker point in the initial frame:

1.    Estimate the 3D position of the marker in the next frame in the take, and calculate the corresponding 2D position of the marker in each camera view.

2.    For each camera view, find the blob detection in the next frame which is the best match, based upon its position and appearance. If no matching blob detection is found, try and find a possible match by tracking a gray-level image patch.

3.    Recalculate the 3D marker positions in the next frame based upon the matched blobs, and perform bundle adjustment to re-estimate the camera positions in the next frame.

4.    Based upon the new 3D marker positions, reject any blob match candidates (and corresponding 3D points) which we believe to be incorrectly matched. Blob matches may be rejected either if the 2D re-projection error(s) for the 3D point are too high, or if the new 3D marker position is too inconsistent its neighbors.

Note the following important points:

❙       Fast facial motions are more difficult to track than sequences where the face does not move very much. This is because the marker may move further from one frame to the next, and the algorithm must consider many more blob match-candidates, which may result in more incorrect matches.

❙       Dense marker sets are in general harder to track than sparse marker sets. Again, for a dense marker set, the algorithm must consider many more blob match-candidates than for a sparse marker set, which may result in more incorrect matches.