Another type of extension to multi-frame is a recursive one where the multiple images are not available in batch form but rather are streaming in serially. One way of formulating this extension is by repeating the classic 2-frame estimate such as the Longuet-Higgens technique [36] or the algorithm described above [30]. Oliensis and Thomas [42] and Soatto et al. [50] sequentially compute such 2-frame estimates and post-process the output with a smoothing Kalman filter (KF). Here, the measurement vectors and the state vectors are the same so the KF is linear, completely observable and hence does not have any linearization problems. In fact, the KF is only acting as a smoothing filter. It is not really being used in its full capacity as a state estimator where the measurements are nonlinearly inverted to obtain state information while keeping track of the state's internal complex dynamics.
Extended Kalman Filters (EKFs) deal with nonlinearity explicitly and can be applied to nonlinearly uncover motion and structure instead of smooth the output of 2-frame techniques. EKF frameworks were utilized on image sequences by Ayache and Faugeras [1], Broida and Chellappa [11], Dickmanns and Graefe [15], Faugeras et al. [20], Heel [28], Matthies et al. [38] and Young and Chellappa [62]. A seminal paper by Broida, Chandrashekhar and Chellappa features a nonlinear EKF for recovering state information [10] . It does not rely on 2-frame techniques but rather folds the estimation into the Kalman filtering equations. The filter is used to nonlinearly invert the measurements to gather state information. One important deficiency of these techniques is that camera internal geometry is not always estimated. This is acceptable for some camera parameters such as skew, etc. which can be less significant and constant in modern cameras. However, the focal length (which is related to the zoom) readily changes in many different video situations. An additional problem is the perceived unreliability of these techniques due to the linearization at each time step in EKF calculations. We now outline our method which estimates focal length and has stronger stability properties due to parameterization changes.