Panotools internals - Revision history

Erik Krause: /* Details */ typos

2009-01-15T21:52:52Z

‎Details: typos

Erik Krause: /* Line control points */ typo

2008-07-13T19:15:40Z

‎Line control points: typo

Erik Krause: /* Image formats */ mentioned greyscale images

2007-12-10T22:20:59Z

‎Image formats: mentioned greyscale images

Erik Krause: New page on some panotools internals

2007-11-06T22:50:00Z

New page on some panotools internals

New page

== Intro ==
With the [[GUI front-ends|GUIs]] panorama creation has become fairly easy. However, if you want to achieve special things or need to do something the GUIs don't support it should be necessary to know how panotools works internally.

== Image formats ==
Since the [[Panorama Tools Plugins]] work from within an imaging application (f.e. Photoshop, The Gimp, Paint Shop Pro) they support any image file format the respective application supports.

The [[GUI front-ends|GUIs]] all try to support the major formats. [[PTStitcher]] used to support some other formats like [[BMP]] on Windows or [[PICT]] on Mac. These formats are a bit outdated nowadays. Since the PTStitcher source code was not available there are several replacements, like [[nona]], [[PTMender]], [[PTGui]] internal stitcher.

=== Read ===
* [[JPEG|JPG]]: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 8 bit: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 16 bit: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 32 bit: nona, PTGui
* [[PNG]] 8 bit: PTStitcher, PTMender, nona, PTGui
* [[PNG]] 16 bit: PTGui
* [[EXR]]: nona, PTGui
* [[HDR]]: nona, PTGui

=== Write ===
* [[JPEG|JPG]]: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 8 bit: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 16 bit: PTStitcher, PTMender, nona, PTGui
* [[TIFF]] 32 bit: nona, PTGui
* [[PNG]] 8 bit: PTStitcher, PTMender, nona, PTGui
* [[PNG]] 16 bit: PTGui
* [[PSD]] 8 bit single or multi layer: PTStitcher, PTGui
* [[PSD]] 16 bit single layer: PTStitcher, PTGui
* [[PSD]] 16 bit multi layer: PTGui
* [[PSB]] 8, 16 or 32 bit single or multi layer: PTGui
* [[EXR]]: nona, PTGui
* [[HDR]]: nona, PTGui
* cylindrical [[QTVR]]: PTStitcher
* cubic [[QTVR]]: PTGui

TIFF files can contain an [[Alpha channel| Alpha Mask]] which is either the shape mask of the remapped image or the clip mask with feather at the overlap center. PSD or PSB files can be single or multi layer, the layered ones contain either one remapped image per layer with shape mask or optionally an additional feathered clip mask per layer.

For details see [[Full 16 bit workflow]].

Besides the major image formats PTStitcher used to write a number of specialised formats for panorama players. For details see the sample [[PTStitcher#Example_Script|PTStitcher script]]. Most of this formats are obsolete now, since the players aren't available any more.

== Image [[projections]] ==
Both the [[Panorama Tools Plugins|plugins]] and [[PTStitcher]] can remap from almost any projection a camera can deliver to a growing number of different output projections.

Both input and output can be
* [[Equirectangular Projection|equirectangular]] (scanning roundshot camera with fisheye)
* [[Cylindrical Projection|cylindrical]] (scanning roundshot camera with rectilinear lens)
* circular and full frame [[Fisheye Projection|fisheye]]
* [[Rectilinear Projection|rectilinear]]
* [[Cubic Projection|cubic]] can be input or output using a script

Most recently there where some interesting projections added for output only. See [[Projections#Stereographic projection|Stereographic projection]] and below and [[Hugin Stitcher tab#Panorama|Hugin Stitcher tab]].

== Workflow ==
Originally the Panotools consisted of some Java based helper applications (details see [[Panorama tools applications|PT applications]], the [[Panorama Tools Plugins|plugins]] and two command line programs: [[PTOptimizer]] and [[PTStitcher]], which are the basis of panoramic image assembly. They are controlled by [[#Scripts|scripts]] as follows:

# A script containing information about the input images, control points linking the images and the intended output format is passed to PTOptimizer.
# The optimizer tries to determine the images positions (without actually looking at the images) such that the mean error between corresponding control points is minimal.
# The image positions are written back in the script file.
# The script can now be passed to PTStitcher.
# The images are remapped by PTStitcher.
# Eventually they are assembled into a layered document.
# The result is written to disk.

The [[GUI front-ends|GUIs]] follow this workflow but try to hide most of the hardships from the user.

=== Scripts ===
The scripts are read on a line by line basis. Each line starts with a control character which determines what the following values mean. The syntax for a PTOptimizer and a PTStitcher script is slightly different. See [[PTOptimizer]] and [[PTStitcher]] for details.

Some of the [[GUI front-ends|GUIs]] use and extend the script syntax to store project data.

=== Optimization ===
[[PTOptimizer]] is a mere wrapper for the optimizer function in the panotools library. This function consists of highly specialised algorithms that are able to optimize all of the parameters together - if there are enough control points and they are well distributed.

=== Remapping ===
Panotools does all necessary remappings in one step, thus avoiding several [[interpolation]] steps which always decrease image quality. Remapping is done directly to the output projection, including all chosen corrections like
* image center [[#Shift_and_Shear|shift]]
* image [[#Shift_and_Shear|shear]]
* fisheye correction
* lens distortion
* perspective

=== Blending ===
Panotools itself does only a very simple blending. A seam line is calculated through the middle of the overlap region and a feather (a transparency gradient) with adjustable width is applied to that seam. This feather can be stored as a part of the [[Alpha channel| Alpha Mask]] in a TIFF file or as a layer mask in a PSD file.

More advanced blending using multiresolution splines is available by using external programs like [[enblend]], [[smartblend]] or internally in PTGui. Both enblend (version 3 and higher) and smartblend do a seam line optimisation in order to avoid [[ghosts]].

=== HDR ===
Original panotools was limited to 16 bit (integer) per channel images. Newer developments enhanced panotools in order to be able to process 16 and 32 bit floating point images. Both [[hugin]] and [[PTGui]] frontends as well as the [[nona]] command line application have the ability to assemble images with different exposures into a single High Dynamic Range panorama.

== Details ==
=== Remapping ===
Originally panotools do a very exact remapping where for any result pixel the source pixel position is calculated individually. This is highly accurate but relatively slow. In August 2004 [[Fulvio Senore]] published a version of panotools with '''Fast Transform''', which speeds up remapping greatly but is slightly less accurate.

For normal and even for high quality panorama work you won't see any difference. However, for any purpose that involves no-overlapping images, like cubeface extraction for a [[Cubic Projection|cubic viewer]] it is not recommended to use '''Fast Transform''' since it might cause tiny empty areas along the border, especially in the corners. This empty areas f.e. show up as small black dots in the corner of cubefaces.

=== Shift and Shear ===
In addition to the correction the common geometric lens errors - for barrel-, pincushion or wavy distortion - panotools can optimize and correct for two additional mapping errors during stitching: Shift and shear. Although both sound somehow similar they refer to very different effects.

* '''Shift''' corrects for a misplacement of the lens relative to the sensor. In this case the center of the image - which is where the [[optical axis]] hits the sensor - is not in the center of the sensor. This can also be cause by a mispositioned [[#Crop|C-type crop]].

* '''Shear''' corrects for a geometric error caused by a scanner or scanning camera, where one side of the image is moved parallel to the oposite side, such that the image forms a parallelogram. This is never the case for a digitally shot image.

=== Line control points ===
In addition to normal control points that mark identical features, panotools know three types of line control points. These are [[horizontal control points]], [[vertical control points]] and [[straight line control points]].

As the names say these control point types can be used to mark features which panotools optimizer should try to make either horizontal, vertical or straight '''in the result image'''.

Horizontal and vertical control points are used pairwise just as normal control points, but must be placed on a feature apart from each other (in order to define a line).

[[Straight line control points]] are used with at minimum two pairs. More pairs are possible. Those are very useful to calibrate lenses for lens distortion. For a tutorial see [http://www.erik-krause.de/verzeichnung/distort_en.htm]

Depending on the output [[projection]] horizontal, vertical or straight lines in reality or in the source images are bent in the result image and hence not can be used

* '''[[Rectilinear Projection|Rectilinear]]''' As the name suggests, all kinds of straight lines keep their properties. Horizontal and vertical ones that run parallel to the sensor plane stay horizontal and vertical and straight ones stay straight no matter what their direction is. Lines that run away from or towards the camera suffer from perspective distortion. This can be corrected using vertical and/or horizontal control points. See [[Perspective correction]] for details.

* '''[[Cylindrical]]''' and '''[[Equirectangular]]''' All vertical lines stay vertical in these projections, but only the horizon itself stays horizontal. Same as a rectilinear image a cylindrical pano can be shot with the camera tilted up or down (with persective distortion in the source image). However, in the resulting cylindrical or equirect if the horizon is straight, all verticals will be vertical (perspective corrected) again. Hence both vertical and horizontal control points can be used to level a panorama. Careful consideration is recommended, since there are cases where they won't yield the desired effect because of the three-dimensional nature of these projections. Since all lines but the horizon and vertical lines get bent in a cylindrical or equirect, straight line control points can only be used for respective features.

* '''[[Fisheye]]''' Only lines going through the image center stay straight in this (rarely used) output projection.

=== Cropping ===
Cropping here refers to a definition of a rectangular or circular region in source images which is actually used. Panotools supports two cropping modes, called '''Selection''' and '''Crop'''. Both croppings are defined by the left, right, top and bottom coordinates in pixel values with the top left corner as coordinate origin. Cropping is defined in the o-line of the [[PTStitcher]] script.

==== Selection ====
The character S preceds the actual coordinates. Only pixels from the respective area are taken, however, all other image parameters, especially [[Lens correction model|Lens correction]] and [[Field of View|FoV]] refer to the complete image. This implies:
* any source image can use a different sized crop area without the need to allow FoV and Lens parameters to be optimized for each image individually.
* any source image needs to be optimized for individual [[#Shift_and_Shear|shift]] values, if the crop area is not equal sized and positioned for all source images.

[[hugin]] uses this crop type by default.

==== Crop ====
The character C preceds the actual coordinates. Only pixels from the respective area are taken and all other image parameters, especially [[Lens correction model|Lens correction]] and [[Field of View|FoV]] refer to the cropped area. This implies:
* all source image must use the same sized (but not positioned) crop area if FoV and Lens parameters should not be optimized for each image individually.
* it is not absolutely necessary to optimize all source images for individual [[#Shift_and_Shear|shift]] values. However, it is recommended to do it, since manual cropping might introduce some minor positional errors.

--[[User:Erik Krause|Erik Krause]] 23:50, 6 November 2007 (CET)

[[Category:Tutorial:Specialised]]

@@ Line 116: / Line 116: @@
 * '''[[Rectilinear Projection|Rectilinear]]''' As the name suggests, all kinds of straight lines keep their properties. Horizontal and vertical ones that run parallel to the sensor plane stay horizontal and vertical and straight ones stay straight no matter what their direction is. Lines that run away from or towards the camera suffer from perspective distortion. This can be corrected using vertical and/or horizontal control points. See [[Perspective correction]] for details.
-* '''[[Cylindrical]]''' and '''[[Equirectangular]]''' All vertical lines stay vertical in these projections, but only the horizon itself stays horizontal. Same as a rectilinear image a cylindrical pano can be shot with the camera tilted up or down (with persective distortion in the source image). However, in the resulting cylindrical or equirect if the horizon is straight, all verticals will be vertical (perspective corrected) again. <br><br>Hence both vertical and horizontal control points can be used to level a panorama. Careful consideration is recommended, since there are cases where they won't yield the desired effect because of the three-dimensional nature of these projections. <br><br>Since all lines but the horizon and vertical lines get bent in a cylindrical or equirect, straight line control points can only be used for respective features.
+* '''[[Cylindrical]]''' and '''[[Equirectangular]]''' All vertical lines stay vertical in these projections, but only the horizon itself stays horizontal. Same as a rectilinear image a cylindrical pano can be shot with the camera tilted up or down (with perspective distortion in the source image). However, in the resulting cylindrical or equirect if the horizon is straight, all verticals will be vertical (perspective corrected) again. <br><br>Hence both vertical and horizontal control points can be used to level a panorama. Careful consideration is recommended, since there are cases where they won't yield the desired effect because of the three-dimensional nature of these projections. <br><br>Since all lines but the horizon and vertical lines get bent in a cylindrical or equirect, straight line control points can only be used for respective features.
 * '''[[Fisheye]]''' Only lines going through the image center stay straight in this (rarely used) output projection.

@@ Line 94: / Line 94: @@
 Originally panotools do a very exact remapping where for any result pixel the source pixel position is calculated individually. This is highly accurate but relatively slow. In August 2004 [[Fulvio Senore]] published a version of panotools with '''Fast Transform''', which speeds up remapping greatly but is slightly less accurate.
-For normal and even for high quality panorama work you won't see any difference. However, for any purpose that involves no-overlapping images, like cubeface extraction for a [[Cubic Projection|cubic viewer]] it is not recommended to use '''Fast Transform''' since it might cause tiny empty areas along the border, especially in the corners. This empty areas f.e. show up as small black dots in the corner of cubefaces.
+For normal and even for high quality panorama work you won't see any difference. However, for any purpose that involves non-overlapping images, like cubeface extraction for a [[Cubic Projection|cubic viewer]] it is not recommended to use '''Fast Transform''' since it might cause tiny empty areas along the border, especially in the corners. This empty areas f.e. show up as small black dots in the corner of cubefaces.
 === Shift and Shear ===
-In addition to the correction the common geometric lens errors - for barrel-, pincushion or wavy distortion - panotools can optimize and correct for two additional mapping errors during stitching: Shift and shear. Although both sound somehow similar they refer to very different effects.
+In addition to the correction the common geometric lens errors - for [[barrel distortion|barrel]]-, [[pincushion distortion|pincushion]] or [[wavy distortion]] - panotools can optimize and correct for two additional mapping errors during stitching: Shift and shear. Although both sound somehow similar they refer to very different effects.
 * '''Shift''' corrects for a misplacement of the lens relative to the sensor. In this case the center of the image - which is where the [[optical axis]] hits the sensor - is not in the center of the sensor. This can also be cause by a mispositioned [[#Crop|C-type crop]].
-* '''Shear''' corrects for a geometric error caused by a scanner or scanning camera, where one side of the image is moved parallel to the oposite side, such that the image forms a parallelogram. This is never the case for a digitally shot image.
+* '''Shear''' corrects for a geometric error caused by a scanner or scanning camera, where one side of the image is moved parallel to the opposite side, such that the image forms a parallelogram. This is never the case for a digitally shot image.
 === Line control points ===
-In addition to normal control points that mark identical features, panotools know three types of line control points. These are [[horizontal control points]], [[vertical control points]] and [[straight line control points]].
+In addition to normal control points that mark identical features, panotools knows three types of line control points. These are [[horizontal control points]], [[vertical control points]] and [[straight line control points]].
 As the names say these control point types can be used to mark features which panotools optimizer should try to make either horizontal, vertical or straight '''in the result image'''.
@@ Line 135: / Line 135: @@
 * it is not absolutely necessary to optimize all source images for individual [[#Shift_and_Shear|shift]] values. However, it is recommended to do it, since manual cropping might introduce some minor positional errors.
 <small>--[[User:Erik Krause|Erik Krause]] 23:50, 6 November 2007 (CET)</small>
 [[Category:Tutorial:Specialised]]

@@ Line 3: / Line 3: @@
 == Image formats ==
 Since the [[Panorama Tools Plugins]] work from within an imaging application (f.e. Photoshop, The Gimp, Paint Shop Pro) they support any image file format the respective application supports.
@@ Line 37: / Line 39: @@
 For details see [[Full 16 bit workflow]].
 Besides the major image formats PTStitcher used to write a number of specialised formats for panorama players. For details see the sample [[PTStitcher#Example_Script|PTStitcher script]]. Most of this formats are obsolete now, since the players aren't available any more.
 == Image [[projections]] ==