https://wiki.panotools.org/api.php?action=feedcontributions&user=Tksharpless&feedformat=atomPanoTools.org Wiki - User contributions [en]2021-02-26T16:27:59ZUser contributionsMediaWiki 1.31.12https://wiki.panotools.org/index.php?title=The_General_Panini_Projection&diff=16015The General Panini Projection2018-01-06T23:37:00Z<p>Tksharpless: </p>
<hr />
<div>The general Panini projection is {{Glossary|an adjustable projection that can render convincing simulated perspective views of scenes up to 150 degrees wide, or even wider in some cases|1}}. It was first implemented in early 2009 in the panorama viewer, Panini, and in limited form in the panotools library. This page describes the improved version that was added to the panotools library in January 2010, and to PTStitcherNG in February 2010. <br />
<br />
The Pannini projection is a cylindrical projection, so it keeps vertical straight lines straight and vertical. Unlike other cylindrical projections, it keeps radial lines through image center straight as well. Those two kinds of straight lines are the most important perspective cues in many scenes, so a Pannini view often resembles a normal rectilinear perspective. But the horizontal field of view can be very wide, without the peripheral distortion (stretching) that is so noticeable when a rectilinear perspective is pushed too far.<br />
<br />
The general Panini projection is suitable for architectural subjects, especially interiors. It works best for views with a single central vanishing point -- straight down a city street, or the aisle of a church, for example. However it is good for oblique interior and street views too. It can make nice city-scapes, but is not suitable for close-up exterior views of buildings, especially when looking toward a corner of the building.<br />
<br />
For best results the general Panini projection should be used interactively: you adjust the control parameters until the image looks just right. The 'fast' preview window in the 2010 version of Hugin works well for that most of the time but has a few problems. However, it is possible to get equally good results with the 'slow' preview window, and with experience, even with scripts for the command line tools, PTStitcherNG or PTmender.<br />
<br />
== Geometrical Description ==<br />
[[File:220x120deg-panini-grid.png|thumb|400px|Standard Panini projection, 10 degree intervals]]<br />
<br />
The basic Panini projection -- the cylindrical stereographic projection -- renders vertical, horizontal and radial straight lines as shown in the diagram at right. Verticals are straight, a prerequisite for any perspective view. Radial lines through the view center are also straight. That creates a convincing perspective illusion when there is a vanishing point at or near view center. But horizontal straight lines are curves, with the strongest curvature in the middle of the image. That makes many images seem to 'bulge' at top and bottom center. <br />
<br />
The general Panini adds two adjustable parameters to the basic projection. One sets the horizontal angle compression, so the projection can vary smoothly from rectilinear to a highly condensed form. The other reduces the curvature of horizontal lines. This can make many central perspectives look more natural, and can also improve the perspective illusion when the vanishing point is off-center. We call this a 'squeeze' parameter because early implementations actually compressed the middle of the image; current versions stretch the sides instead. The adjustment is purely vertical so that verticals remain straight. There are actually separate squeeze parameters for the top and bottom halves of the image.<br />
<br />
There are two kinds of squeeze, 'hard' and 'soft'. A hard squeeze can fully straighten horizontal lines, but only works on fields of view well under 180 degrees. A soft squeeze provides less complete straightening but works for wider fields of view. Of course nothing is free: reducing the curvature of horizontal lines comes at the cost of increasing the curvature of radial lines. Fortunately that curvature is strongest far from image center, and typically unnoticeable on fields of view of 130 degrees or less. <br />
<br />
For more detailed technical and historical information, visit http://vedutismo.net/Pannini/.<br />
<br />
{{clr}}<br />
<br />
== Parameters ==<br />
The general Panini projection is controlled by three parameters. <br />
<br />
The main parameter, called compression and labelled 'Cmpr' in the hugin previews, has a range of 0 to 150 (you can think of that as percent). It adjusts the form of the projection between the completely uncompressed rectilinear projection (Cmpr = 0) and the maximally compressed cylindrical orthographic projection (Cmpr = 150) with the standard Pannini (cylindrical stereographic) projection at Cmpr = 100. <br />
<br />
The maximum horizontal field of view varies with compression, from 160 degrees at Cmpr = 0 to 320 degrees at Cmpr = 100, then back to 180 degrees at Cmpr = 150. Hugin's FOV sliders automatically respect these limits. If you enter an infeasible hfov value in a script (for PTmender, for example) the actual field of view will be the largest feasible value.<br />
<br />
Two secondary parameters, labeled 'Tops' and 'Bots', control vertical squeeze adjustments applied to the top and bottom halves of the image, for the purpose of reducing the curvature of horizontal lines. These have a range of -100 to 100. There is no squeeze when the parameter is zero. Positive values invoke a 'hard' squeeze, that can exactly straighten horizontal lines, but only over a limited range of about 160 degrees. Negative values invoke a 'soft' squeeze that works for wider fields of view but cannot eliminate all curvature.<br />
<br />
== Applications ==<br />
=== Distortionless Wide-Angle Photos ===<br />
[[image:TKS-Mmm-obliquev120-recti.png|thumb|right|250px|Rectilinear, hfov 120 deg.]]<br />
[[image:TKS-Mmm-obliquev120-gP70_50_50.png|thumb|left|250px|Panini(70,50,50), hfov 120 deg.]]<br />
One of the best uses of the general Panini projection is simulating rectilinear perspective, without peripheral distortion, on fields of view typical of wide angle to super-wide angle camera lenses, say in the range of 75 to 120 degrees. A compression well under 100 is usually enough, plus a hard squeeze to straighten prominent transverse lines. The appropriate squeeze strength depends on the inclination of those lines: 100% if they are horizontal, less as the slope increases. <br />
<br />
Compare the shapes of the rose window (and other features) in the mild Panini perspective at left and the true rectilinear one at right.<br />
<br />
{{clr}}<br />
<br />
=== Perspective Views from Fish-Eye Photos ===<br />
[[File:LagardePont-defish.jpg|left|thumb|250px|Panini view]]<br />
[[File:LagardePont-fish.jpg|right|thumb|180px|Fisheye snapshot]]<br />
Hugin with the general Panini projection is an excellent 'de-fishing' tool. Load a fisheye photo, adjust pitch and roll angles to align the true vertical with the vertical axis of the view, then adjust the general Panini parameters for a pleasing perspective -- anything from rectilinear to highly compressed and squeezed. Note that you can adjust pitch and roll by dragging in the fast preview window.<br />
<br />
If vertical lines look curved, you should adjust the lens focal length or FOV until they are straight. Temporarily switching to the rectilinear projection may make it easier to do this. The curvature results when the panotools lens model does not match the actual projection function of your lens. Adjusting the fov provides a first order correction that is usually 'close enough'; but for even better results, you could set the lens fov and a,b,c distortion parameters to values that were optimized on a well-aligned spherical panorama made with the same lens.<br />
<br />
{{clr}}<br />
<br />
=== Very Wide and Ultra-Wide Views from Panoramas ===<br />
<br />
[[File:Pantheon-150x100-rect-panini.jpg|right|360px]]<br />
<br />
With a panorama as the source image, the general Panini projection can be used like any other projection to render views in various directions. However it is best for views where the line of sight is not tilted up or down. It is important to align the vertical direction carefully in "roll" as well. Both pitch and roll alignment can be done by dragging the image in Hugin's fast preview window. To get a perfectly framed view, it may be necessary to render oversize and crop later.<br />
<br />
The image at right compares rectilinear (top) and general Pannini (bottom) projections of a panoramic view 150 degrees wide by 100 degrees high in the Pantheon (Rome), a large, perfectly circular space. The point of view here is well away from the middle of the room. Notice how the Pannini perspective both magnifies the center, so that people in the middle look closer, and condenses the periphery, so that people near the edges have more natural shapes -- for example, the man taking a picture at far right. A 50 percent "soft squeeze" has been applied at both top and bottom to improve the perspective of the floor and dome.<br />
<br />
It is often possible to render a convincing view more than 180 degrees wide, if the subject has a strong central perspective. The spectacular image of the main concourse at New York's Grand Central Station, below, is 220 degrees wide. The slanted walls at the sides are actually the ends of a transverse wall that stands well behind the point of view. This is a standard Pannini projection without any "squeeze".<br />
<br />
{{clr}}<br />
<br />
[[File:C Marchi-GrandCentral-adj.jpg|thumb|center|800px|Grand Central Terminal by Cristian Marchi; Panini projection, hFOV 220 degrees]]<br />
<br />
== Software Versions ==<br />
Your libpano13 and hugin must have been built after 20 January 2010, from panotools source version SVN 1237 or later and hugin source version SVN 4920 or later. You have to build pre-release versions, or download them from a test builder's site, as the first official 2010 release of Hugin does not include the general Panini projection. You can get a self installing Win32 binary at http://tksharpless.net [click 'Panoramic Software', then double-click 'hugin-2010.1.0.4920.exe'. Running this creates a new self contained installation that won't interfere with any existing installation.]<br />
<br />
In the first versions of Hugin that support the general Panini projection, the 'fast' preview window sometimes has trouble displaying general Panini views, usually parts go black, occasionally it shows an apparently complete but incorrect view. The 'slow' preview window shows the true view in all cases; use it if the fast preview display seems garbled or unreliable. Since 2015, the fast preview display is generally stable apart from sometimes showing crop bars after a field-of-view change. These normally disappear when the view is refreshed.<br />
<br />
Helmut Dersch's PTStitcherNG implements the general Panini projection just like libpano13. It only runs from scripts and provides no interactive preview; however it is so fast that an iterative adjustment of the script parameters is feasible.<br />
<br />
The Panini projection is available in several panorama stitchers besides Hugin, notably PTGui (as "vedutismo") and PTAssembler (as "Recti-Perspective"). Those implementations provide adjustable compression but not the "squeeze" controls of the General Panini.<br />
<br />
-- T. K. Sharpless, 18 January 2010 - 05 Jan 2018 --</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14500Remove tripod shadows2012-12-01T18:04:57Z<p>Tksharpless: </p>
<hr />
<div>Most panographers know how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer. However, this can be done by applying exactly the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. When the camera shadow is sufficiently far away, it is possible to get a completely clean patch by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Shoot from a spot that gives you enough room to maneuver yourself and your equipment, and if possible puts the camera shadow either far away, or in a dark place, perhaps inside some natural shadow. <br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using manual shutter release -- take the view in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and handle it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take at least two shadow patches. It is easier to align two small patches than one big one; but more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, you are far more likely to get patches that cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then clear the masks only where needed to fill the gaps.<br />
<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|700px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. <br />
<br />
You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees; 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14499Remove tripod shadows2012-12-01T18:02:43Z<p>Tksharpless: </p>
<hr />
<div>Most panographers know how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer. However, this can be done by applying exactly the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. When the camera shadow is sufficiently far away, it is possible to get a completely clean patch by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Shoot from a spot that gives you enough room to maneuver yourself and your equipment, and if possible puts the camera shadow either far away, or in a dark place, perhaps inside some natural shadow. <br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using manual shutter release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and handle it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take at least two shadow patches. It is easier to align two small patches than one big one; but more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, you are far more likely to get patches that cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then clear the masks only where needed to fill the gaps.<br />
<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|700px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. <br />
<br />
You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees; 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14498Remove tripod shadows2012-12-01T17:55:35Z<p>Tksharpless: </p>
<hr />
<div>Most panographers know how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer. However, this can be done by applying exactly the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. When the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a good spot, that gives you enough room to maneuver yourself and your equipment, and if possible puts the camera shadow either far away, or in a dark place, perhaps inside some natural shadow that will be part of the picture. <br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using manual shutter release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take at least two shadow patches. It is easier to align two small patches than one big one; but more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, you are far more likely to get patches that cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then clear the masks just where needed to fill the gaps in the fixed images.<br />
<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|700px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees; 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14497Remove tripod shadows2012-12-01T17:33:08Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|700px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees; 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14496Remove tripod shadows2012-12-01T17:29:45Z<p>Tksharpless: /* Example 1: monopod supported by hand */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|700px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching all the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14495Remove tripod shadows2012-12-01T17:28:30Z<p>Tksharpless: /* Example 1: monopod supported by hand */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|600px|center]]<br />
<br />
The basic shooting pattern was just 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (photos cropped and contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching all the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14494Remove tripod shadows2012-12-01T17:26:15Z<p>Tksharpless: /* Example 1: monopod supported by hand */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadows of me, the pole and camera extend a long way into the water. <br />
<br />
[[File: Rocks-shadows.jpg|600px|center]]<br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved to the left and held the pole at a flat slant to to shoot a combined shadow and nadir patch (images contrast enhanced to make the shadows easier to see). <br />
<br />
[[File: Rocks-stitched.jpg|350px|center]]<br />
<br />
Before stitching I masked out the shadows and the top of the monopod in all images, and masked all but the relevant part of the patch image. The last picture shows that part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching all the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Rocks-shadows.jpg&diff=14493File:Rocks-shadows.jpg2012-12-01T17:16:05Z<p>Tksharpless: views with shadows before stitching</p>
<hr />
<div>views with shadows before stitching</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Rocks-stitched.jpg&diff=14492File:Rocks-stitched.jpg2012-12-01T17:12:42Z<p>Tksharpless: detail of nadir area after stitching out shadows</p>
<hr />
<div>detail of nadir area after stitching out shadows</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14491Remove tripod shadows2012-12-01T16:18:34Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. <br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views also show the shadow. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken. <br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Before stitching, I painted masks over all shadows and images of myself and the support. After stitching all the shadows were gone, and I just had to clone out two tiny unpatched areas at the nadir and on the bottom fence rail.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14490Remove tripod shadows2012-12-01T16:07:21Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. <br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Part of the stitched panorama, after cloning out small unpatched areas at the nadir and the camera shadow on the bottom fence rail. All shadows were removed during stitching.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Patch_0.jpg&diff=14489File:Patch 0.jpg2012-12-01T16:05:41Z<p>Tksharpless: Tksharpless uploaded a new version of &quot;File:Patch 0.jpg&quot;: increase brightness & contrast</p>
<hr />
<div>example nadir area after shadow removal by stitching</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14488Remove tripod shadows2012-12-01T16:00:15Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. <br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a pole that rotates on its own ball-jointed foot. For the fixed shots (6 around down 55 degrees, 8 around level; 6 around up 55 degrees) the pole was supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull.<br />
<br />
[[File:Shadows.jpg|600px|center]]<br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
[[File:patch_0.jpg|340px|center]]<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow on the bottom fence rail. All other shadows were patched in stitching.<br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Shadows.jpg&diff=14487File:Shadows.jpg2012-12-01T15:52:33Z<p>Tksharpless: Tksharpless uploaded a new version of &quot;File:Shadows.jpg&quot;: smaller</p>
<hr />
<div>panorama view showing shadows, and two shadow patch shots</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Shadows.jpg&diff=14486File:Shadows.jpg2012-12-01T15:49:34Z<p>Tksharpless: panorama view showing shadows, and two shadow patch shots</p>
<hr />
<div>panorama view showing shadows, and two shadow patch shots</div>Tksharplesshttps://wiki.panotools.org/index.php?title=File:Patch_0.jpg&diff=14485File:Patch 0.jpg2012-12-01T15:47:53Z<p>Tksharpless: example nadir area after shadow removal by stitching</p>
<hr />
<div>example nadir area after shadow removal by stitching</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14484Remove tripod shadows2012-12-01T15:38:10Z<p>Tksharpless: /* Example 1: hand-supported monopod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: monopod supported by hand ==<br />
180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. <br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. <br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow on the bottom fence rail. All other shadows were patched in stitching. <br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless<br />
[[Category:Tutorial:Nice to know]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14472Remove tripod shadows2012-12-01T04:18:17Z<p>Tksharpless: /* Example 1: hand-supported monopod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. <br />
<br />
The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version [http://www.360cities.net/image/bright-rocks-narragansett-bay#352.60,0.50,77.5 here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. <br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow on the bottom fence rail. All other shadows were patched in stitching. <br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14471Remove tripod shadows2012-12-01T04:13:50Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. <br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow on the bottom fence rail. All other shadows were patched in stitching. <br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14470Remove tripod shadows2012-12-01T04:10:08Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. <br />
<br />
Left, the down view in the direction of the shadow. Several other down and two level views have shadows that had to be masked out. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow. All the shadows were removed in stitching. <br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14469Remove tripod shadows2012-12-01T04:04:17Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. The shadows of the tripod, pole, and me cover a lot of the ground and wall as shown in the first 3 pictures.<br />
<br />
Left, one of the 'down' views. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
Part of the stitched panorama, extracted for cloning out small unpatched (black) areas at the nadir and the camera shadow. All the shadows were removed in stitching. <br />
<br />
The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html [here]].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14468Remove tripod shadows2012-12-01T04:00:17Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole, supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. The shadows of the tripod, pole, and me cover a lot of the ground and wall as shown in the first 3 pictures.<br />
<br />
Left, one of the 'down' views. Center, shadow patch shot from the left. Right, shadow patch shot from the right. A separate nadir patch was also taken.<br />
<br />
This picture was extracted from the stitched panorama for cloning out small unpatched (black) areas at the nadir and the camera shadow. All the shadows were removed in stitching. The full size interactive panorama is [http://tksharpless.net/FlashPanos/30thStreet-6AM.html [here]].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14467Remove tripod shadows2012-12-01T03:22:58Z<p>Tksharpless: /* Example 2: pole supported by tripod */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
15mm fisheye lens on a 1.6 crop DSLR mounted on a telescoping pole supported by a medium-weight tripod via a plastic spring clip that lets me separate the pole from the tripod with a firm pull. The pole rotates on its own ball-jointed foot. Shooting pattern 6 around down 55 degrees, 8 around level; 6 around up 55 degrees. The shadows of the tripod, pole, and me cover a lot of the ground and wall as shown in the first 2 pictures, which are 'standing left' and 'standing right' versions of one of the 'down' views.<br />
<br />
The next two pictures are shadow patches shot from the left with the pole at different slants. It would have been better to shoot one of these from the right. I also took a separate nadir shot.<br />
<br />
The last picture was extracted from the stitched panorama for cloning out small unpatched areas at the nadir and the camera shadow. All the shadows are gone. Interactive panorama [here].<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14466Remove tripod shadows2012-12-01T03:06:02Z<p>Tksharpless: /* Example 1: hand-held pole */</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-supported monopod ==<br />
Shot with a 180 degree fisheye lens on a small camera mounted on a light monopod, equipped with a bubble level, standing on a rock. It was early morning, so the shadow of the pole and camera extend a long way into the water. The basic shooting pattern was simply 4 shots around, but I took the view in the direction of the shadow twice, once standing to the left of the camera and once to the right. Then I moved well to the right and held the pole at a flat slant to to take a combined shadow and nadir patch shot. <br />
<br />
To stitch I just masked the unwanted shadows and the top of the monopod, and blended all the images. The last picture shows part of the stitched panorama. You can see the interactive version of it [here].<br />
<br />
== Example 2: pole supported by tripod ==<br />
<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14465Remove tripod shadows2012-12-01T02:41:58Z<p>Tksharpless: </p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.<br />
<br />
== Example 1: hand-held pole ==<br />
<br />
== Example 2: pole supported by tripod ==<br />
<br />
<br />
-- 30 November 2012 Thomas K Sharpless</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Remove_tripod_shadows&diff=14464Remove tripod shadows2012-12-01T02:37:13Z<p>Tksharpless: Equipment shadows can usually be removed by taking patch images and masking.</p>
<hr />
<div>Many panographers like to present a pure spherical image, with no sign of the photographer or his/her apparatus. It is widely known how to use an additional "nadir shot" to replace the image of a tripod, but somehow it seems more difficult to eliminate the shadows of the tripod, pole, and photographer from a pano taken at a strongly lighted spot. However, this can be done by applying the same principle: combine several shots in which the shadows are in different places.<br />
<br />
If all images are taken from a fixed lens position, the shadow of the camera itself must be at the same place in all of them. But we can move the shadow of the support, simply by moving the support. This is easiest if the camera is mounted on a telescoping monopod or pole, but can be done with a conventional tripod as well. And when the camera shadow is sufficiently far away, it is possible to get a completely clean 'patch shot' by shifting the camera slightly to one side of the panocenter.<br />
<br />
As in all other aspects of photography, the key to success is visualizing problems before shooting, and planning ways around them. First, you must be aware of the direction of the light that is making the shadows. You will need to move things at right angles to that direction in order to to shift the shadows. Second, choose a spot where the shadow of the camera itself is either far away or in a dark place, perhaps inside some natural shadow that will be part of the picture. Make sure the chosen spot gives you enough room to maneuver yourself and your equipment. If using a tripod, try to place it so the shadows of two legs coincide.<br />
<br />
When taking the fixed shots from panocenter, take care that your own shadow is not in any of them; or when that is not possible -- for example when shooting from a hand-supported pole or using a cable release -- take each shot in which your shadow appears twice, once standing to the left of the camera and once standing to the right.<br />
<br />
To take your 'shadow patch' shots, slant the support off to the side (at right angles to the light) while keeping the lens as nearly as possible at panocenter and pointed at the shadow area. This is easiest with a pole, especially one that rests on the ground. In that case you can get a pretty stable shooting position by extending the pole enough to compensate for the slant. for example:<br />
slant 15 degrees: no compensation needed;<br />
slant 30 degrees: 1/6 of the distance you move the foot;<br />
slant 45 degrees: 2/5 of the distance you move the foot.<br />
<br />
If using a tripod, extend its center column as much as possible for the fixed shooting. Then fold the legs and use it like a pole. You will need more slant than with a pole, to let the camera see past the tripod's shoulders. It may be feasible to extend the legs, but I usually just hold it up in the air. <br />
<br />
Before moving the support, sight on a couple of reference points to help you judge lens position, and tighten the lock screw on your rotator if using one.<br />
<br />
Sometimes the same shot can serve as both nadir patch and shadow patch, but it is best to take them separately, optimizing each for its particular function. In fact it is a good plan to take several shadow patches, for two reasons. It is much easier to align a small patch well than a big one. But more important, in a single shot you can easily have part of your shadow where part of the tripod shadow was -- it isn't there anymore, so it's hard to tell. If you take two, standing on opposite sides of the shadow area, with the pole slanted in opposite directions, it is much more likely you will cover everything.<br />
<br />
When stitching, handle your shadow patches like a nadir patch. Enable viewpoint optimization. Make sure all control points are near the shadow region, and more or less on one plane. That does not have to be a horizontal plane, or even a flat surface, in fact just picking points roughly equidistant from the camera also works well. Mask out all shadows in the fixed images, mask the patch images completely, then erase the masks just where needed to fill the gaps.<br />
<br />
In difficult cases you can try advanced patching tricks such as using the same patch twice with different control points and mask. But if you plan and execute your shooting carefully that should not be necessary.</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13062Lens Correction in PanoTools2011-01-24T17:16:34Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. Today, equipment manufacturers' specifications usually provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
Alternatively, the EXIF data from most high-end cameras includes the "focal plane resolution" field, which gives the physical pixel size directly.<br />
<br />
In any practical calibration scheme <math>\textstyle F_{pixels} </math> is actually an adjustable parameter. However the fitted value is expected to be quite close to the one given by the physical specifications, which would of course be used as the initial value. The main uncertainty is how accurately the nominal lens focal length reflects the true one, because normally the focal plane resolution is precisely known.<br />
<br />
With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
To adapt a portable correction to a given image it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the pixel size associated with the image.<br />
<br />
Unfortunately PanoTools does not now use any physical parameters, so fully portable corrections would have to be calculated, saved and restored by front-end software that has access to focal length and pixel size. But if those were added to the PanoTools parameter set, libpano could handle fully portable corrections autonomously.<br />
<br />
<br />
-- 24 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13061Lens Correction in PanoTools2011-01-24T17:09:03Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. Today, equipment manufacturers' specifications usually provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
Alternatively, the EXIF data from most modern high-end cameras includes the "focal plane resolution" field, which gives the physical pixel size directly.<br />
<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one given by the physical specifications, which would of course be used as the initial value. The main uncertainty is how accurately the nominal lens focal length reflects the true one, because for digital cameras the focal plane resolution is known exactly. With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
To adapt a portable correction to a given image it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the pixel size associated with the image.<br />
<br />
Unfortunately PanoTools does not now use any physical parameters, so fully portable corrections would have to be calculated, saved and restored by front-end software that has access to focal length and pixel size. But if those were added to the PanoTools parameter set, libpano could handle fully portable corrections autonomously.<br />
<br />
<br />
-- 24 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13060Lens Correction in PanoTools2011-01-24T16:46:10Z<p>Tksharpless: /* Fully Portable Calibrations */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. Today, equipment manufacturers' specifications usually provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
Alternatively, the EXIF data from most modern high-end cameras includes the "focal plane resolution" field, which gives the physical pixel size directly.<br />
<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one given by the physical specifications, which would of course be used as the initial value. The main uncertainty is how accurately the nominal lens focal length reflects the true one, because for digital cameras the focal plane resolution is known exactly. With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the pixel size associated with the image. As a fall-back when no pixel size is specified, <math>\textstyle F_{pixels} </math> can be calculated from hfov and image width as libpano does now. If hfov was fitted on the same image format, this will be the correct value; so portable corrections would not invalidate existing scripts in presently correct use cases.<br />
<br />
Unfortunately PanoTools does not now use any physical parameters. As it stands now, fully portable corrections would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size and focal length were added to the PanoTools parameter set, libpano could handle fully portable corrections autonomously. <br />
<br />
<br />
-- 24 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13057Lens Correction in PanoTools2011-01-24T02:37:46Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Calibrations==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.<br />
<br />
With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image format it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the format's pixel size.<br />
<br />
Unfortunately PanoTools does not use any physical parameters. So as it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size were added to the PanoTools parameter set, libpano could handle portable lens calibrations autonomously. <br />
<br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13056Lens Correction in PanoTools2011-01-24T02:36:38Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.<br />
<br />
With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image format it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the format's pixel size.<br />
<br />
Unfortunately PanoTools does not use any physical parameters. So as it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size were added to the PanoTools parameter set, libpano could handle portable lens calibrations autonomously. <br />
<br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13055Lens Correction in PanoTools2011-01-24T02:34:20Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on lens focal length and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F_{pixels} = F_{mm} \frac {sensor\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.<br />
<br />
With <math> h </math> the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image format it is only necessary to calculate <math>\textstyle F_{pixels} </math> from the calibrated <math>\textstyle F_{mm} </math> and the the given sensor dimensions and pixel counts.<br />
<br />
Unfortunately PanoTools does not use any physical parameters. So as it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size were added to the PanoTools parameter set, libpano could handle portable lens calibrations autonomously. <br />
<br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13054Lens Correction in PanoTools2011-01-24T02:27:29Z<p>Tksharpless: /* Portable Correction Coefficients */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Alternatively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.<br />
<br />
If h is the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image format it is only necessary to calculate <math>\textstyle F_{pixels} </math> from <math>\textstyle F_{mm} </math> and the the given sensor dimensions and pixel counts.<br />
<br />
Unfortunately PanoTools does not use any physical parameters. So as it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size were added to the PanoTools parameter set, libpano could handle portable lens calibrations autonomously. <br />
<br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13053Lens Correction in PanoTools2011-01-24T02:24:59Z<p>Tksharpless: /* Fully Portable Corrections */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Altenratively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make a lens correction fully portable also requires expressing the fitted focal length in physical units rather than in pixels. <br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. Physically, this quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one.<br />
<br />
If h is the width of a pixel in mm, the portable form of the fitted lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
<br />
To adapt a portable correction to a given image format it is only necessary to calculate <math>\textstyle F_{pixels} </math> from <math>\textstyle F_{mm} </math> and the the given sensor dimensions and pixel counts.<br />
<br />
Unfortunately PanoTools does not use any physical parameters. So as it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if physical pixel size were added to the PanoTools parameter set, libpano could handle portable lens calibrations autonomously. <br />
<br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13052Lens Correction in PanoTools2011-01-24T02:15:58Z<p>Tksharpless: /* Portable Correction Coefficients */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Altenratively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make the lens correction parameters fully portable also requires expressing the fitted focal length (and optical center shifts) in physical units rather than in pixels. That depends on the physical pixel size, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the portable lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can also be converted to mm by multiplying them by h. Or they could be made dimensionless by dividing them by F.<br />
<br />
All that is needed to adapt the portable correction to a given image format is to calculate the working <math>\textstyle F_{pixels} </math> from <math>\textstyle F_{mm} </math> and the the given sensor dimensions and pixel counts.<br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13051Lens Correction in PanoTools2011-01-24T02:12:00Z<p>Tksharpless: /* Portable Lens Parameters */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Correction Coefficients==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, any lens calibration, portable or not. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values.<br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
Altenratively the current non-portable coefficients can be converted using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
==Fully Portable Corrections==<br />
<br />
To make the lens correction parameters fully portable also requires expressing the fitted focal length (and optical center shifts) in physical units rather than in pixels. That depends on the physical pixel size, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the portable lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can also be converted to mm by multiplying them by h. Or they could be made dimensionless by dividing them by F.<br />
<br />
All that is needed to adapt the portable correction to a given image format is to calculate the working <math>\textstyle F_{pixels} </math> from <math>\textstyle F_{mm} </math> and the the given sensor dimensions and pixel counts.<br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13050Lens Correction in PanoTools2011-01-24T01:47:31Z<p>Tksharpless: /* Mapping View Angle Radius in Image */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Ideal Lens Models==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13049Lens Correction in PanoTools2011-01-24T01:46:27Z<p>Tksharpless: /* Mapping View Angle Radius */</p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius in Image==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13048Lens Correction in PanoTools2011-01-24T01:44:08Z<p>Tksharpless: </p>
<hr />
<div>This article is a mathematical analysis of how the panotools library computes lens correction parameters, why those parameters are not portable, and how they could be made portable. For a more general and use-oriented description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13047Lens Correction in PanoTools2011-01-24T01:31:41Z<p>Tksharpless: /* Portable Lens Parameters */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications provide the needed data:<br />
: <math>\textstyle F = {(focal\ length\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be quite close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13046Lens Correction in PanoTools2011-01-24T01:29:46Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
The true source projection is unknown, so <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13045Lens Correction in PanoTools2011-01-24T01:25:31Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors <math>d</math> and <math>e</math> are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, <math>d</math> is identical to <math>F_{pano}</math>. In fact, under the name “distance factor”, <math>d</math> is used by many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, <math>e</math> is an estimate of <math>F_{source}</math> according to the fitted correction parameters. Since hfov is one of those parameters, <math>e</math> will be proportional to the true <math>F_{source}</math>; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0. <br />
<br />
In other words, PT's <math>e</math> is a biased estimate of <math>F_{source}</math>. However, the overall correction is equivalent to the generic one because the bias in the correction polynomial cancels the bias in the focal length. The only real defect in the PanoTools scheme is that its parameters work for just one image format.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13044Lens Correction in PanoTools2011-01-24T01:08:14Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction polynomial is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It is a cubic polynomial, that computes the ratio of observed to ideal radius. Its argument is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13043Lens Correction in PanoTools2011-01-24T01:02:15Z<p>Tksharpless: /* Mapping View Angle Radius */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels. In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13042Lens Correction in PanoTools2011-01-24T00:58:20Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels; for example <br />
: <math>\textstyle F\ (pixels) = FL\ in\ mm * \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes the PanoTools polynomial equivalent to the generic one, but with different coefficients. This can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to source image size, which is proportional to F by definition. But the proportionality factor varies with source image format, so the PanoTools coefficients also depend on source format.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13041Lens Correction in PanoTools2011-01-24T00:48:34Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels; for example <br />
: <math>\textstyle F\ (pixels) = FL\ in\ mm * \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization is essential to the correctness of the result, which can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to image size, which is proportional to F by definition.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano() and A2Nsource() are the ideal projection functions for panorama and lens. There are many panorama projections but only two lens projections: <br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = tan( a )</math> for rectilinear lenses<br />
: <math>\textstyle {\operatorname{A2Nsource}( a )} = a </math> for fisheye lenses.<br />
The scale factor from panorama to source coordinates is<br />
: <math>\textstyle s = \frac e d</math>.<br />
<br />
Factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13040Lens Correction in PanoTools2011-01-24T00:28:35Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels; for example <br />
: <math>\textstyle F\ (pixels) = FL\ in\ mm * \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes this relation stable under optimization of a, b, and c. It is also essential to the correctness of the result, which can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to image size, which is proportional to F by definition.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano and A2Nsource are the ideal functions for panorama and lens. Then<br />
: <math>\textstyle R_{source} = R_{pano} \frac e d</math>.<br />
<br />
The scale factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13039Lens Correction in PanoTools2011-01-24T00:25:18Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels; for example <br />
: <math>\textstyle F\ (pixels) = FL\ in\ mm * \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes this relation stable under optimization of a, b, and c. It is also essential to the correctness of the result, which can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels, so we can write the adjusted radius as<br />
: <math>\textstyle r = F N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>,<br />
The normalized observed radius <math>\textstyle n = \frac r F</math>, so<br />
: <math>\textstyle n = N\ \operatorname{poly}\left(\frac {F N} {r_0} \right)</math>.<br />
If <math> r_0 </math> is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. That is the case when <math> r_0 </math> is proportional to image size, which is proportional to F by definition.<br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano and A2Nsource are the ideal functions for panorama and lens. Then<br />
: <math>\textstyle R_{source} = R_{pano} \frac e d</math>.<br />
<br />
The scale factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharplesshttps://wiki.panotools.org/index.php?title=Lens_Correction_in_PanoTools&diff=13038Lens Correction in PanoTools2011-01-24T00:11:40Z<p>Tksharpless: /* PanoTools Correction Scheme */</p>
<hr />
<div>This article is about a possible new way to correct for lens distortions in panotools library. If you look for a general description of the current way panotools deals with lens distortion see [[Lens correction model]]<br />
<br />
=Lens Correction in PanoTools=<br />
<br />
The PanoTools library implements an effective, but rather idiosyncratic method for correcting lens projections, that causes a good deal of puzzlement. Lens parameters optimized for one image format generally do not work for a different format; even rotating a set of images 90 degrees before aligning them produces different and incompatible lens parameters. One would expect that there must be a way to convert either of those parameter sets to a common form, that would apply equally well to both formats, or indeed to any image taken with the same lens. To see how that might be done, I have made a detailed analysis of PanoTools lens correction computations, based on the code in historic as well as current versions of libpano and helpful discussions with Helmut Dersch. <br />
<br />
==Why Lens Correction?==<br />
<br />
To make a panoramic image from photographs, it is essential to be able to calculate the direction in space corresponding to any given position in a given photo. Specifically, we need to know the angles between the view directions of the photos (the ''alignment'' of the images), and a ''radial projection function'' that relates the distance of a point from image center to the true angle of view, measured from the optical axis of the lens. Given a set of control points linking the images, PanoTools estimates both the alignment and the lens projection by a nonlinear least squares fitting procedure -- ''optimization''. Using the fitted lens parameters, the stitcher can correct each image to match the ideal geometry of the scene, according to whatever projection is chosen for the panorama. Done right, that makes all the images fit together perfectly; moreover, it yields a panoramic image that seems to have been made with a perfect lens.<br />
<br />
==Mapping View Angle <=> Radius==<br />
<br />
The radial projection curve of a real lens may approximate some known mathematical function, but in practice it must be determined experimentally, a process known as calibrating the lens. A calibration is a parametrized mathematical model, fitted to experimental data. The typical model consists of an ideal angle-to-radius function, and a polynomial that converts the ideal radius to the actual radius measured on the image. <br />
<br />
Like many lens calibration programs, libpano uses just two ideal functions to model lenses: rectilinear, for 'normal' lenses, and 'fisheye', for all others. The rectilinear projection has radius proportional to the tangent of the view angle. PT's 'fisheye', better known as the equal-angle spherical projection, has radius proportional to the angle itself. The constant of proportionality is the lens focal length, F. With angle A in radians, and R the ideal radius, the formulas are<br />
: Rectilinear: <math>\textstyle \frac R F = \tan(A)</math><br />
: Equal-angle: <math>\textstyle \frac R F = A</math><br />
Of course R and F have to be measured in the same units. If we have F in mm, then R is in mm also. If we want to measure R in pixels, then we need F in pixels; for example <br />
: <math>\textstyle F\ (pixels) = FL\ in\ mm * \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any case, F is the constant of proportionality between the actual radius and the value of a trigonometric function that defines the basic shape of the projection. <br />
<br />
In physical optics, focal length is defined as the first derivative of R by A, at A = 0. That is easy to see if we write <math>\textstyle R = F A</math> or <math>\textstyle R = F \tan(A)</math>, because the slopes of A and tan(A) are both 1 at A = 0. This is also true of other trigonometric functions commonly used as ideal lens projections: <br />
: Equal-Area: <math>\textstyle \frac R F = 2\sin\left(\frac A 2 \right)</math><br />
: Stereographic: <math>\textstyle \frac R F = 2\tan\left(\frac A 2 \right)</math>.<br />
<br />
The dimensionless quantity <math>\textstyle N = \frac R F</math> is the normalized ideal radius. Multiplying N by the focal length, in any units, gives the ideal image radius in the same units.<br />
<br />
==Generic Correction Scheme==<br />
<br />
The difference between the real lens projection and the ideal one is modeled by an adjustable correction function that gives the observed radius as a function of the ideal radius. The adjustable part is almost always a polynomial, because it it easy to fit polynomials to experimental data. The argument to the polynomial should be the normalized ideal radius,<br />
: <math>\textstyle N = \frac R F</math>, <br />
because that makes the polynomial coefficients independent of how image size is measured. The constant term is 0 because both radii are zero at the same point. If the coefficient of the linear term is 1, so that the first derivative at 0 is 1, then the value of the polynomial will be the normalized observed radius, n = r / F. Multiplying n by the focal length, in any units, gives the observed image radius in the same units:<br />
: <math>\textstyle r = F n</math>.<br />
<br />
Many calibration packages use a polynomial with only even order terms beyond the first:<br />
: <math>\textstyle n = N + a N^2 + b N^4 + c N^6</math>.<br />
Equivalently<br />
: <math>\textstyle n = N ( 1 + a N + b N^3 + c N^5 )</math><br />
The expression in parentheses is the ratio of observed to ideal radius, which is expected to be close to 1 everywhere if the ideal model function is well chosen.<br />
<br />
==PanoTools Correction Scheme==<br />
<br />
Lens correction in PanoTools is unusual in several respects. First, it ignores the physical parameters of the lens (focal length) and camera (pixel size). Instead, it computes angle-to-radius scale factors from image dimensions and fields of view, as described below. All correction computations are in terms of image radii, measured in pixels, rather than the normalized radii described above. However, normalized radii are evaluated implicitly.<br />
<br />
Second, the correction is computed in equal-angle spherical coordinates, rather than camera coordinates. Observed image points are found by remapping those coordinates according to the ideal lens projection, and rescaling them according to the ratio of pixel sizes in the source and ideal images. <br />
<br />
Third, the correction is normalized to hold a certain radius, <math>\textstyle r_0</math>, constant. It essentially consists of a cubic polynomial that computes the ratio of observed to ideal radius. The argument to this polynomial is <math>\textstyle \frac R {r_0}</math>, and its constant term is set so that the result is exactly 1 when the argument is 1, that is, when <math>\textstyle R = r_0</math>. With<br />
: <math>\textstyle X = \frac R {r_0}</math><br />
The correction factor is<br />
: <math>\textstyle x = (1 - a - b - c) + a X + b X^2 + c X^3</math>,<br />
and the observed radius is given by<br />
: <math>\textstyle r = R x</math>.<br />
The observed radius is thus formally a 4th order polynomial in R:<br />
: <math>\textstyle r = s R + t R^2 + u R^3 + v R^4</math>,<br />
where <math>\textstyle s = (1-a-b-c),\ t = \frac a {r_0},\ u = \frac b {{r_0}^2},\ v = \frac c {{r_0}^3}</math>.<br />
<br />
The normalization makes this relation stable under optimization of a, b, and c. It is also essential to the correctness of the result, which can be seen as follows. The ideal radius is<br />
: <math>\textstyle R = F N</math><br />
where F is the ideal focal length in pixels. In equal-angle coordinates N = A, so we can write the adjusted radius as<br />
: <math>\textstyle r = F A\ \operatorname{poly}\left(\frac {F A} {r_0} \right)</math>,<br />
The normalized observed radius <math>\textstyle n = \frac r F</math>, so<br />
: <math>\textstyle n = A\ \operatorname{poly}\left(\frac {F A} {r_0} \right)</math>.<br />
Since n is a dimensionless number, this can only be true if the argument to poly() is also a dimensionless number. F has the dimension of pixels, therefore it must be divided by another parameter dimensioned in pixels. If that parameter is proportional to F, then the quotient is proportional to N, and the polynomial is equivalent to one whose argument is N. <br />
<br />
The overall computation proceeds as follows. PanoTools computes the ideal radius R by mapping a point in the panorama (which plays the role of the ideal image) to equal angle spherical projection. Then <math>\textstyle R = \sqrt{ h^2 + v^2 }</math>, where h and v are the pixel coordinates relative to the center of the equal-angle projection. Then PT's radius() function computes x as described, and returns scaled coordinates ( h x, v x ). If the lens is rectilinear, PT next remaps those coordinates to rectilinear; if it is a fisheye, no remapping is needed. In either case the coordinates are finally rescaled to account for any difference in resolution between the panorama and the source image. The scale factor is computed from the dimensions and angular fields of view of the panorama and the source image, as follows.<br />
: <math>\textstyle d = \frac {half\ width\ of\ pano} {\operatorname{A2Npano}\ {(half\ hfov\ of\ pano)}}</math>,<br><br />
: <math>\textstyle e = \frac {half\ width\ of\ source} {\operatorname{A2Nsource}\ {(half\ hfov\ of\ source)}}</math>,<br />
where A2Npano and A2Nsource are the ideal functions for panorama and lens. Then<br />
: <math>\textstyle R_{source} = R_{pano} \frac e d</math>.<br />
<br />
The scale factors d and e are focal lengths in pixels, because A2N() yields the normalized radius, equal to <math>\textstyle \frac R F</math>. For the panorama, which follows an ideal projection, d is identical to F. In fact d, under the name “distance factor”, is used in many of libpano's coordinate transformation functions to convert radius in pixels to the ideal normalized radius in trigonometric units.<br />
<br />
For the source image, whose true projection is only approximately known, e is an estimate of F according to the fitted correction parameters. Since hfov is one of those parameters, the fitted value of e will be proportional to the true F; the constant of proportionality will approach 1 as the fitted polynomial coefficients approach 0.<br />
<br />
==Portable Lens Parameters==<br />
<br />
Focal length and projection function are separable lens properties. In fact many schemes determine F from different data than those used to fit the lens curve. PanoTools is somewhat unique in fitting all lens parameters to one set of experimental values. However there is no reason why it should not compute portable correction parameters.<br />
<br />
The focal length in pixels must be known in order to compute, or to apply, a portable lens calibration. This quantity depends on both lens and camera properties. In most cases today, equipment manufacturers' specifications can provide the needed data:<br />
: <math>\textstyle F = {(FL\ in\ mm)} \frac {image\ width\ in\ pixels} {sensor\ width\ in\ mm}</math>.<br />
In any practical calibration scheme F is actually an adjustable parameter. However the fitted value is expected to be close to the one implied by these physical specifications, the main uncertainty being how accurately the nominal lens focal length reflects the true one. <br />
<br />
In the generic calibration scheme, dividing image coordinates by F makes it possible for the fitted correction parameters (apart from F) to be independent of both image format and physical pixel size, so that they apply to any image made with the given lens. As explained above, dividing image coordinates by any factor proportional to F is logically sufficient; however values other than F itself lead to non-portable parameter values that depend on image format. <br />
<br />
In the PanoTools scheme, the "distance parameter" d, which is the focal length in panorama pixels, would be the appropriate divisor. That would make the argument of the radius scaling polynomial the ideal normalized radius,<br />
: <math>\textstyle N = \frac {R_{pano}} {F_{pano}}</math><br />
and the fitted coefficient values would be portable.<br />
<br />
The current non-portable coefficients can be converted to a portable form using data available inside libpano. With<br />
: <math>\textstyle k = \frac d {r_0}</math>,<br />
: <math>\textstyle w' = w k </math>,<br />
: <math>\textstyle a' = a k^2</math>,<br />
: <math>\textstyle b' = b k^3</math>,<br />
: <math>\textstyle c' = c k^4</math><br />
are the coefficients of a polynomial in <math>\textstyle N = \frac R d</math> that computes the same radius correction factor as the PT polynomial. The constant term w' is no longer a simple function of the other three, however it can be reduced to 1 by dividing all coefficients by w'. The reduced coefficients are<br />
: <math>\textstyle W = 1</math><br />
: <math>\textstyle A = a \frac k w</math><br />
: <math>\textstyle B = b \frac {k^2} w</math><br />
: <math>\textstyle C = c \frac {k^3} w</math><br />
So the portable radius mapping is<br />
: <math>\textstyle r = R ( 1 + A N + B N^2 + C N^3 )</math><br />
Along with the ideal function A2Nsource(), which gives N as a function of angle, this constitutes a portable lens correction function.<br />
<br />
To convert the PT lens parameters to a fully portable form also requires expressing the fitted focal length (and the optical center shifts d, e) in physical units rather than in pixels. That depends on the physical pixel width, which unfortunately PanoTools does not use. If h is the width of a pixel in mm, the calibrated lens focal length is<br />
: <math>\textstyle F_{mm} = h F_{pixels} = h e</math>, scale factor e defined above.<br />
The center shifts can be converted to mm the same way. <br />
<br />
As it stands now, portable lens calibrations would have to be calculated, saved and restored by front-end software that has access to the camera's sensor size. But if pixel size were added to the PanoTools parameter set, libpano could provide portable lens calibrations autonomously. <br />
<br />
-- 23 Jan 2010 T K Sharpless<br />
<br />
[[Category:Community:Project]]</div>Tksharpless