Lens Correction in PanoTools

From PanoTools.org Wiki
(Difference between revisions)
Jump to: navigation, search
(Portable Lens Parameters)
m (Reverted edits by Koae (Talk) to last version by Tksharpless)
 
(31 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable.  For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]
 +
 
=Lens Correction in PanoTools=
 
=Lens Correction in PanoTools=
  
Line 7: Line 9:
 
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo.  Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens.  Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama.  Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.
 
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo.  Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens.  Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama.  Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.
  
==Mapping View Angle <=> Radius==
+
==Ideal Lens Models==
  
 
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens.  A calibration is a parametrized mathematical model, fitted to experimental data.  The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image.   
 
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens.  A calibration is a parametrized mathematical model, fitted to experimental data.  The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image.   
  
 
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others.  The rectilinear projection has radius proportional to the tangent of the view angle.  PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself.  The constant of proportionality is the lens focal length, F.  With angle A in radians, and R the ideal radius, the formulas are
 
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others.  The rectilinear projection has radius proportional to the tangent of the view angle.  PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself.  The constant of proportionality is the lens focal length, F.  With angle A in radians, and R the ideal radius, the formulas are
: Rectilinear: R / F  =  tan( A )
+
: Rectilinear: <math>\textstyle \frac R F  =  \tan(A)</math>
: Equal-angle: R / F  =  A
+
: Equal-angle: <math>\textstyle \frac R F  =  A</math>
Of course R and F have to be measured in the same units.  If we have F in mm, then R is in mm also.  If we want to measure R in pixels, then we need F in pixels; for example
+
Of course R and F have to be measured in the same units.  If we have F in mm, then R is in mm also.  If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection.   
: F (pixels) =  (FL in mm) * (image width in pixels) / (sensor width in mm).
+
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection.   
+
  
In physical optics, focal length is defined as the first derivative of R by A, at A = 0.  That is easy to see if we write  R = F A  or  R = F tan(A), because the slopes of A and tan(A) are both 1 at A = 0.  This is also true of other trigonometric functions commonly used as ideal lens projections:  
+
In physical optics, focal length is defined as the first derivative of R by A, at A = 0.  That is easy to see if we write  <math>\textstyle R = F A</math> or  <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0.  This is also true of other trigonometric functions commonly used as ideal lens projections:  
: Equal-Area: R / F  = 2 sin( A / 2 )
+
: Equal-Area: <math>\textstyle \frac R F  = 2\sin\left(\frac A 2 \right)</math>
: Stereographic: R / F  =  2 tan( A / 2 ).
+
: Stereographic: <math>\textstyle \frac R F  =  2\tan\left(\frac A 2 \right)</math>.
  
The dimensionless quantity N = R / F is the normalized ideal radius.  Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.
+
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius.  Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.
  
 
==Generic Correction Scheme==
 
==Generic Correction Scheme==
  
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius.  The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data.  The argument to the polynomial is usually the normalized ideal radius,
+
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius.  The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data.  The argument to the polynomial should be the normalized ideal radius,
:      N = R / F,  
+
:      <math>\textstyle N = \frac R F</math>,  
 
because that makes the polynomial coefficients independent of how image size is measured.  The constant term is 0 because both radii are zero at the same point.  If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F.  Multiplying n by the focal length, in any units, gives the observed image radius in the same units:
 
because that makes the polynomial coefficients independent of how image size is measured.  The constant term is 0 because both radii are zero at the same point.  If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F.  Multiplying n by the focal length, in any units, gives the observed image radius in the same units:
: r = F n.
+
: <math>\textstyle r = F n</math>.
  
 
Many calibration packages use a  polynomial with only even order terms beyond the first:
 
Many calibration packages use a  polynomial with only even order terms beyond the first:
: n =  N  +  a N ^2  +  b N ^4  +  c N ^6.
+
: <math>\textstyle n =  N  +  a N^2  +  b N^4  +  c N^6</math>.
 
Equivalently
 
Equivalently
: n  =  N ( 1 + a N  +  b N^3  +  c N^5 )
+
: <math>\textstyle n  =  N ( 1 + a N  +  b N^3  +  c N^5 )</math>
 
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.
 
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.
 
  
 
==PanoTools Correction Scheme==
 
==PanoTools Correction Scheme==
Line 42: Line 41:
 
Lens correction in PanoTools is unusual in several respects.  First, it ignores the physical parameters of the lens (focal length) and camera (pixel size).  Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below.  All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above.  However, normalized radii are evaluated implicitly.
 
Lens correction in PanoTools is unusual in several respects.  First, it ignores the physical parameters of the lens (focal length) and camera (pixel size).  Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below.  All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above.  However, normalized radii are evaluated implicitly.
  
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates.  Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images.   
+
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates.  Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images.   
  
Third, the correction is normalized to hold a certain radius, r0, constant.  It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius.  The argument to this polynomial is R / r0, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when R = r0.  With
+
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant.  It is a cubic polynomial, that computes the ratio of observed to ideal radius.  Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>.  With
: X  =  R / r0
+
: <math>\textstyle X  =  \frac R {r_0}</math>
 
The correction factor is
 
The correction factor is
: x  =  (1 - a - b - c)  +  a X  +  b X^2  +  c X^3,
+
: <math>\textstyle x  =  (1 - a - b - c)  +  a X  +  b X^2  +  c X^3</math>,
 
and the observed radius is given by
 
and the observed radius is given by
: r  =  R x.
+
: <math>\textstyle r  =  R x</math>.
 
The observed radius is thus formally a 4th order polynomial in R:
 
The observed radius is thus formally a 4th order polynomial in R:
: r  =  s R  +  t R^2  +  u R^3  +  v R^4,
+
: <math>\textstyle r  =  s R  +  t R^2  +  u R^3  +  v R^4</math>,
where  s = (1-a-b-c),  t = a / r0,  u = b / r0^2,  v = c / r0^3.
+
where  <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.
  
The normalization makes this relation stable under optimization of a, b, and c.  It is also essential to the correctness of the result, which can be seen as follows.  The ideal radius is
+
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients.  This can be seen as follows.  The ideal radius is
: R  =  F N
+
: <math>\textstyle R  =  F N</math>
where F is the ideal focal length in pixels.  We can thus write the adjusted radius as
+
where F is the ideal focal length in pixels, so we can write the adjusted radius as
: r  =   F A poly( F A / r0 ),
+
: <math>\textstyle r  = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,
The normalized observed radius n  =  r / F, so
+
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definitionBut the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.
: n  =  A poly( F A / r0 ).
+
Since n is a dimensionless number, this can only be true if the argument to poly() is also a dimensionless numberF has the dimension of pixels, therefore it must be divided by another parameter dimensioned in pixels.
+
 
 
The overall computation proceeds as follows.  PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection.  Then  R =   sqrt( h^2 + v^2 ),  where h and v are the pixel coordinates relative to the center of the equal-angle projection.  Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ).  If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed.  In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image.  The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.
+
The overall computation proceeds as follows.  PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection.  Then  <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>,  where h and v are the pixel coordinates relative to the center of the equal-angle projection.  Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ).  If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed.  In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image.  The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.
: d = (half width of pano) / A2Npano( half hfov of pano ),
+
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br>
: e  = (half width of source) / A2Nsource( half hfov of source ),
+
: <math>\textstyle e  = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,
where A2Npano and A2Nsource are the ideal functions for panorama and lens.  Then
+
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens.  There are many panorama projections but only two lens projections:
: Rsource Rpano e  / d.
+
:       <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math>  for rectilinear lenses
 +
:      <math>\textstyle {\operatorname{A2Nsource}( a )} a </math>  for fisheye lenses.
 +
The scale factor from panorama to source coordinates is
 +
: <math>\textstyle s = \frac e d</math>.
  
The scale factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to R / F.   For the panorama, which follows an ideal projection, d is identical to F.  In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.
+
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>.  In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.
  
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters.  Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.
+
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters.  Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.
  
==Portable Lens Parameters==
+
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>.  However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length.  The only real defect in the PanoTools scheme is that its parameters work for just one image format.
  
The focal length in pixels must be known in order to calibrate a lens.  This quantity clearly depends on both lens and camera properties.  In most cases today, equipment manufacturers' specifications can provide the needed data:
+
==Portable Correction Coefficients==
: F  = (FL in mm) * (image width in pixels) / (sensor width in mm).
+
Computing F this way makes it possible for the fitted correction coefficients to be independent of camera format, as explained above.  In any practical calibration scheme F is actually an adjustable parameter.  However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.
+
  
Focal length and projection function are separable lens properties.  In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values.
+
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.
  
The normalization of the radial polynomial makes it possible to equate the correction computed by PanoTools to the generic one.  As discussed above, that is true for any choice of r0 that is proportional to image size.  The best choice would be
+
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,
: r0  =  Fpano  =  “distance parameter” d, defined above.
+
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math>
That would make the argument of the radius scaling polynomial  
+
and the fitted coefficient values would be portable.
: Rpano / Fpano
+
which is equal to the ideal normalized radius, N.  Then the polynomial coefficients fitted by PanoTools would be independent of image dimensions, and could be used directly in a generic function, to compute the radius correction in camera coordinates.  The value actually used is merely proportional to Fpano; the proportionality factor is a function of image dimensions, so the correction coefficients depend on the image format and are not portable.
+
  
To convert these PT lens parameters to a fully portable form requires two things:  computing portable radial coefficients, and defining F in a way that is not tied to an image format.  The latter requires access to at least one physical parameter, not in the PT parameter set.  Ideally that would be the physical size of a pixel.  With w the pixel width in mm,
+
Alternatively the current non-portable coefficients can be converted using data available inside libpano.  With
: F(mm)  =  w F(pixels)  =  w (scale factor e, defined above)
+
: <math>\textstyle k = \frac d {r_0}</math>,
This could be calculated by front-end software using the value of e from libpano.
+
: <math>\textstyle w' =  w k </math>,
 
+
: <math>\textstyle a' =  a k^2</math>,
The coefficients can be converted using data available inside libpano.  With
+
: <math>\textstyle b' =  b k^3</math>,
: k = d / r0,
+
: <math>\textstyle c' =  c k^4</math>
: w' =  w k =  (1 – a – b – c) k,
+
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial.  The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'.  The reduced coefficients are
: a' =  a k^2,
+
:      <math>\textstyle W  =  1</math>
: b' =  b k^3,
+
:      <math>\textstyle A  =  a \frac k w</math>
: c' =  c k^4
+
:      <math>\textstyle B  =  b \frac {k^2} w</math>
are the coefficients of a polynomial in ''N'' = R / d that computes the same radius correction factor as the PT polynomial.  The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'.  The reduced coefficients are
+
:      <math>\textstyle C  =  c \frac {k^3} w</math>
:      W  =  1
+
:      A  =  a k / w  
+
:      B  =  b k^2 / w
+
:      C  =  c k^3 / w
+
 
So the portable radius mapping is
 
So the portable radius mapping is
:      r = R ( 1 + A ''N'' + B ''N''^2 + C ''N''^3 )
+
:      <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math>
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens projection function.
+
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.
 +
 
 +
==Fully Portable Corrections==
 +
 
 +
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. 
 +
 
 +
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not.  Physically, this quantity depends on lens focal length and camera properties.  Today, equipment manufacturers' specifications usually provide the needed data:
 +
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.
 +
Alternatively, the EXIF data from most high-end cameras includes the "focal plane resolution" field, which gives the physical pixel size directly.
 +
 
 +
In any practical calibration scheme <math>\textstyle F_{pixels} </math> is actually an adjustable parameter.  However the fitted value is expected to be quite close to the one given by the physical specifications, which would of course be used as the initial value.  The main uncertainty is how accurately the nominal lens focal length reflects the true one, because normally the focal plane resolution is precisely known.
 +
 
 +
With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is
 +
: <math>\textstyle F_{mm}  =  h F_{pixels} =  h e</math>, scale factor e defined above.
 +
To adapt a portable correction to a given image it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the pixel size associated with the image.
 +
 +
Unfortunately PanoTools does not now use any physical parameters, so fully portable corrections would have to be calculated, saved and restored by front-end software that has access to focal length and pixel size.  But if those were added to the PanoTools parameter set, libpano could handle fully portable corrections autonomously.
 +
 
  
This function could be very useful even without a portable version of PT's focal length estimate. Ultimately, focal length is just a linear scale factor, that can be fitted or estimated in many different ways according to the problem at hand, while the radial function represents the essential 'lens curve' that is independent of image scale.
+
  -- 24 Jan 2010 T K Sharpless
  
-- 19 Jan 2010 T K Sharpless
+
[[Category:Community:Project]]

Latest revision as of 13:01, 10 November 2011

This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see Lens correction model

Contents

[edit] Lens Correction in PanoTools

The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch.

[edit] Why Lens Correction?

To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the alignment of the images), and a radial projection function that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- optimization. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.

[edit] Ideal Lens Models

The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image.

Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are

Rectilinear: \textstyle \frac R F  =  \tan(A)
Equal-angle: \textstyle \frac R F  =  A

Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection.

In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write \textstyle R = F A or \textstyle R = F \tan(A), because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections:

Equal-Area: \textstyle \frac R F  = 2\sin\left(\frac A 2 \right)
Stereographic: \textstyle \frac R F  =  2\tan\left(\frac A 2 \right).

The dimensionless quantity \textstyle N = \frac R F is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.

[edit] Generic Correction Scheme

The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,

\textstyle N = \frac R F,

because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:

\textstyle r = F n.

Many calibration packages use a polynomial with only even order terms beyond the first:

\textstyle n =  N  +  a N^2  +  b N^4  +  c N^6.

Equivalently

\textstyle n  =  N ( 1 + a N  +  b N^3  +  c N^5 )

The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.

[edit] PanoTools Correction Scheme

Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.

Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images.

Third, the correction polynomial is normalized to hold a certain radius, \textstyle r_0, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is \textstyle \frac R {r_0}, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when \textstyle R = r_0. With

\textstyle X  =  \frac R {r_0}

The correction factor is

\textstyle x  =  (1 - a - b - c)  +  a X  +  b X^2  +  c X^3,

and the observed radius is given by

\textstyle r   =  R x.

The observed radius is thus formally a 4th order polynomial in R:

\textstyle r  =  s R  +  t R^2  +  u R^3  +  v R^4,

where \textstyle s = (1-a-b-c),\  t = \frac a {r_0},\  u = \frac b {{r_0}^2},\  v = \frac c {{r_0}^3}.

The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is

\textstyle R  =  F N

where F is the ideal focal length in pixels, so we can write the adjusted radius as

\textstyle r  =  F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right),

If  r_0 is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when  r_0 is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.

The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then \textstyle R = \sqrt{ h^2 + v^2 }, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.

\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}},
\textstyle e  = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}},

where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections:

\textstyle {\operatorname{A2Nsource}( a )} = tan( a ) for rectilinear lenses
\textstyle {\operatorname{A2Nsource}( a )} =  a for fisheye lenses.

The scale factor from panorama to source coordinates is

\textstyle s = \frac e d.

Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to \textstyle \frac R F. For the panorama, which follows an ideal projection, d is identical to F_{pano}. In fact, under the name “distance factor”, d is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.

The true source projection is unknown, so e is an estimate of F_{source} according to the fitted correction parameters. Since hfov is one of those parameters, e will be proportional to the true F_{source}; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.

In other words, e is a biased estimate of F_{source}. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.

[edit] Portable Correction Coefficients

In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.

In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,

\textstyle N = \frac {R_{pano}} {F_{pano}}

and the fitted coefficient values would be portable.

Alternatively the current non-portable coefficients can be converted using data available inside libpano. With

\textstyle k = \frac d {r_0},
\textstyle w' =  w k ,
\textstyle a' =  a k^2,
\textstyle b' =  b k^3,
\textstyle c' =  c k^4

are the coefficients of a polynomial in \textstyle N = \frac R d that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are

\textstyle W  =  1
\textstyle A  =  a \frac k w
\textstyle B  =  b \frac {k^2} w
\textstyle C  =  c \frac {k^3} w

So the portable radius mapping is

\textstyle r = R ( 1 + A N + B N^2 + C N^3 )

Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.

[edit] Fully Portable Corrections

To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels.

The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. Today, equipment manufacturers' specifications usually provide the needed data:

\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}.

Alternatively, the EXIF data from most high-end cameras includes the "focal plane resolution" field, which gives the physical pixel size directly.

In any practical calibration scheme \textstyle F_{pixels} is actually an adjustable parameter. However the fitted value is expected to be quite close to the one given by the physical specifications, which would of course be used as the initial value. The main uncertainty is how accurately the nominal lens focal length reflects the true one, because normally the focal plane resolution is precisely known.

With  h the width of a pixel in mm, the portable form of the fitted lens focal length is

\textstyle F_{mm}  =  h F_{pixels} =  h e, scale factor e defined above.

To adapt a portable correction to a given image it is only necessary to calculate \textstyle F_{pixels} from the calibrated \textstyle F_{mm} and the the pixel size associated with the image.

Unfortunately PanoTools does not now use any physical parameters, so fully portable corrections would have to be calculated, saved and restored by front-end software that has access to focal length and pixel size. But if those were added to the PanoTools parameter set, libpano could handle fully portable corrections autonomously.


-- 24 Jan 2010 T K Sharpless
Personal tools
Namespaces

Variants
Actions
Navigation
tools
Tools