Lens Correction in PanoTools

This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see Lens correction model

Lens Correction in PanoTools

The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch.

Why Lens Correction?

To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the alignment of the images), and a radial projection function that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- optimization. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.

Mapping View Angle <=> Radius

The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image.

Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are

Rectilinear: $\textstyle {\frac {R}{F}}=\tan(A)$

Equal-angle: $\textstyle {\frac {R}{F}}=A$

Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection.

In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write $\textstyle R=FA$ or $\textstyle R=F\tan(A)$, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections:

Equal-Area: $\textstyle {\frac {R}{F}}=2\sin \left({\frac {A}{2}}\right)$

Stereographic: $\textstyle {\frac {R}{F}}=2\tan \left({\frac {A}{2}}\right)$.

The dimensionless quantity $\textstyle N={\frac {R}{F}}$ is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.

Generic Correction Scheme

The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,

$\textstyle N={\frac {R}{F}}$,

because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:

$\textstyle r=Fn$.

Many calibration packages use a polynomial with only even order terms beyond the first:

$\textstyle n=N+aN^{2}+bN^{4}+cN^{6}$.

Equivalently

$\textstyle n=N(1+aN+bN^{3}+cN^{5})$

The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.

PanoTools Correction Scheme

Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.

Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images.

Third, the correction is normalized to hold a certain radius, $\textstyle r_{0}$, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is $\textstyle {\frac {R}{r_{0}}}$, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when $\textstyle R=r_{0}$. With

$\textstyle X={\frac {R}{r_{0}}}$

The correction factor is

$\textstyle x=(1-a-b-c)+aX+bX^{2}+cX^{3}$,

and the observed radius is given by

$\textstyle r=Rx$.

The observed radius is thus formally a 4th order polynomial in R:

$\textstyle r=sR+tR^{2}+uR^{3}+vR^{4}$,

where $\textstyle s=(1-a-b-c),\ t={\frac {a}{r_{0}}},\ u={\frac {b}{{r_{0}}^{2}}},\ v={\frac {c}{{r_{0}}^{3}}}$.

The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is

$\textstyle R=FN$

where F is the ideal focal length in pixels, so we can write the adjusted radius as

$\textstyle r=FN\ \operatorname {poly} \left({\frac {FN}{r_{0}}}\right)$,

If $r_{0}$ is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when $r_{0}$ is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.

The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then $\textstyle R={\sqrt {h^{2}+v^{2}}}$, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.

$\textstyle d={\frac {half\ width\ of\ pano}{\operatorname {A2Npano} \ {(half\ hfov\ of\ pano)}}}$,

$\textstyle e={\frac {half\ width\ of\ source}{\operatorname {A2Nsource} \ {(half\ hfov\ of\ source)}}}$,

where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections:

$\textstyle {\operatorname {A2Nsource} (a)}=tan(a)$ for rectilinear lenses

$\textstyle {\operatorname {A2Nsource} (a)}=a$ for fisheye lenses.

The scale factor from panorama to source coordinates is

$\textstyle s={\frac {e}{d}}$.

Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to $\textstyle {\frac {R}{F}}$. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.

For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.

Portable Lens Parameters

Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.

The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:

$\textstyle F={(FL\ in\ mm)}{\frac {image\ width\ in\ pixels}{sensor\ width\ in\ mm}}$.

In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.

In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format.

In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,

$\textstyle N={\frac {R_{pano}}{F_{pano}}}$

and the fitted coefficient values would be portable.

The current non-portable coefficients can be converted to a portable form using data available inside libpano. With

$\textstyle k={\frac {d}{r_{0}}}$,

$\textstyle w'=wk$,

$\textstyle a'=ak^{2}$,

$\textstyle b'=bk^{3}$,

$\textstyle c'=ck^{4}$

are the coefficients of a polynomial in $\textstyle N={\frac {R}{d}}$ that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are

$\textstyle W=1$

$\textstyle A=a{\frac {k}{w}}$

$\textstyle B=b{\frac {k^{2}}{w}}$

$\textstyle C=c{\frac {k^{3}}{w}}$

So the portable radius mapping is

$\textstyle r=R(1+AN+BN^{2}+CN^{3})$

Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.

To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is

$\textstyle F_{mm}=hF_{pixels}=he$, scale factor e defined above.

The center shifts can be converted to mm the same way.

As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously.

-- 23 Jan 2010 T K Sharpless