Panotools internals

Intro

With the GUIs panorama creation has become fairly easy. However, if you want to achieve special things or need to do something the GUIs don't support it should be necessary to know how panotools works internally.

Image formats

Panorama tools are designed to work with RGB (plus an optional alpha channel) image data. Neither greyscale nor images with other color channels are supported.

Since the Panorama Tools Plugins work from within an imaging application (f.e. Photoshop, The Gimp, Paint Shop Pro) they support any image file format the respective application supports.

The GUIs all try to support the major formats. PTStitcher used to support some other formats like BMP on Windows or PICT on Mac. These formats are a bit outdated nowadays. Since the PTStitcher source code was not available there are several replacements, like nona, PTMender, PTGui internal stitcher.

Read

• JPG: PTStitcher, PTMender, nona, PTGui
• TIFF 8 bit: PTStitcher, PTMender, nona, PTGui
• TIFF 16 bit: PTStitcher, PTMender, nona, PTGui
• TIFF 32 bit: nona, PTGui
• PNG 8 bit: PTStitcher, PTMender, nona, PTGui
• PNG 16 bit: PTGui
• EXR: nona, PTGui
• HDR: nona, PTGui

Write

• JPG: PTStitcher, PTMender, nona, PTGui
• TIFF 8 bit: PTStitcher, PTMender, nona, PTGui
• TIFF 16 bit: PTStitcher, PTMender, nona, PTGui
• TIFF 32 bit: nona, PTGui
• PNG 8 bit: PTStitcher, PTMender, nona, PTGui
• PNG 16 bit: PTGui
• PSD 8 bit single or multi layer: PTStitcher, PTGui
• PSD 16 bit single layer: PTStitcher, PTGui
• PSD 16 bit multi layer: PTGui
• PSB 8, 16 or 32 bit single or multi layer: PTGui
• EXR: nona, PTGui
• HDR: nona, PTGui
• cylindrical QTVR: PTStitcher
• cubic QTVR: PTGui

TIFF files can contain an Alpha Mask which is either the shape mask of the remapped image or the clip mask with feather at the overlap center. PSD or PSB files can be single or multi layer, the layered ones contain either one remapped image per layer with shape mask or optionally an additional feathered clip mask per layer.

For details see Full 16 bit workflow.

Besides the major image formats PTStitcher used to write a number of specialised formats for panorama players. For details see the sample PTStitcher script. Most of this formats are obsolete now, since the players aren't available any more.

Image projections

Both the plugins and PTStitcher can remap from almost any projection a camera can deliver to a growing number of different output projections.

Both input and output can be

• equirectangular (scanning roundshot camera with fisheye)
• cylindrical (scanning roundshot camera with rectilinear lens)
• circular and full frame fisheye
• rectilinear
• cubic can be input or output using a script

Most recently there where some interesting projections added for output only. See Stereographic projection and below and Hugin Stitcher tab.

Workflow

Originally the Panotools consisted of some Java based helper applications (details see PT applications, the plugins and two command line programs: PTOptimizer and PTStitcher, which are the basis of panoramic image assembly. They are controlled by scripts as follows:

1. A script containing information about the input images, control points linking the images and the intended output format is passed to PTOptimizer.
2. The optimizer tries to determine the images positions (without actually looking at the images) such that the mean error between corresponding control points is minimal.
3. The image positions are written back in the script file.
4. The script can now be passed to PTStitcher.
5. The images are remapped by PTStitcher.
6. Eventually they are assembled into a layered document.
7. The result is written to disk.

The GUIs follow this workflow but try to hide most of the hardships from the user.

Scripts

The scripts are read on a line by line basis. Each line starts with a control character which determines what the following values mean. The syntax for a PTOptimizer and a PTStitcher script is slightly different. See PTOptimizer and PTStitcher for details.

Some of the GUIs use and extend the script syntax to store project data.

Optimization

PTOptimizer is a mere wrapper for the optimizer function in the panotools library. This function consists of highly specialised algorithms that are able to optimize all of the parameters together - if there are enough control points and they are well distributed.

Remapping

Panotools does all necessary remappings in one step, thus avoiding several interpolation steps which always decrease image quality. Remapping is done directly to the output projection, including all chosen corrections like

• image center shift
• image shear
• fisheye correction
• lens distortion
• perspective

Blending

Panotools itself does only a very simple blending. A seam line is calculated through the middle of the overlap region and a feather (a transparency gradient) with adjustable width is applied to that seam. This feather can be stored as a part of the Alpha Mask in a TIFF file or as a layer mask in a PSD file.

More advanced blending using multiresolution splines is available by using external programs like enblend, smartblend or internally in PTGui. Both enblend (version 3 and higher) and smartblend do a seam line optimisation in order to avoid ghosts.

HDR

Original panotools was limited to 16 bit (integer) per channel images. Newer developments enhanced panotools in order to be able to process 16 and 32 bit floating point images. Both hugin and PTGui frontends as well as the nona command line application have the ability to assemble images with different exposures into a single High Dynamic Range panorama.

Details

Remapping

Originally panotools do a very exact remapping where for any result pixel the source pixel position is calculated individually. This is highly accurate but relatively slow. In August 2004 Fulvio Senore published a version of panotools with Fast Transform, which speeds up remapping greatly but is slightly less accurate.

For normal and even for high quality panorama work you won't see any difference. However, for any purpose that involves non-overlapping images, like cubeface extraction for a cubic viewer it is not recommended to use Fast Transform since it might cause tiny empty areas along the border, especially in the corners. This empty areas f.e. show up as small black dots in the corner of cubefaces.

Shift and Shear

In addition to the correction the common geometric lens errors - for barrel-, pincushion or wavy distortion - panotools can optimize and correct for two additional mapping errors during stitching: Shift and shear. Although both sound somehow similar they refer to very different effects.

• Shift corrects for a misplacement of the lens relative to the sensor. In this case the center of the image - which is where the optical axis hits the sensor - is not in the center of the sensor. This can also be caused by a mispositioned C-type crop.

• Shear corrects for a geometric error caused by a scanner or scanning camera, where one side of the image is moved parallel to the opposite side, such that the image forms a parallelogram. This is never the case for a digitally shot image.

Line control points

In addition to normal control points that mark identical features, panotools knows three types of line control points. These are horizontal control points, vertical control points and straight line control points.

As the names say these control point types can be used to mark features which panotools optimizer should try to make either horizontal, vertical or straight in the result image.

Horizontal and vertical control points are used pairwise just as normal control points, but must be placed on a feature apart from each other (in order to define a line).

Straight line control points are used with at minimum two pairs. More pairs are possible. Those are very useful to calibrate lenses for lens distortion. For a tutorial see [1]

Depending on the output projection horizontal, vertical or straight lines in reality or in the source images are bent in the result image and hence not can be used

• Rectilinear As the name suggests, all kinds of straight lines keep their properties. Horizontal and vertical ones that run parallel to the sensor plane stay horizontal and vertical and straight ones stay straight no matter what their direction is. Lines that run away from or towards the camera suffer from perspective distortion. This can be corrected using vertical and/or horizontal control points. See Perspective correction for details.

• Cylindrical and Equirectangular All vertical lines stay vertical in these projections, but only the horizon itself stays horizontal. Same as a rectilinear image a cylindrical pano can be shot with the camera tilted up or down (with perspective distortion in the source image). However, in the resulting cylindrical or equirect if the horizon is straight, all verticals will be vertical (perspective corrected) again.
  
  Hence both vertical and horizontal control points can be used to level a panorama. Careful consideration is recommended, since there are cases where they won't yield the desired effect because of the three-dimensional nature of these projections.
  
  Since all lines but the horizon and vertical lines get bent in a cylindrical or equirect, straight line control points can only be used for respective features.

• Fisheye Only lines going through the image center stay straight in this (rarely used) output projection.

Cropping

Cropping here refers to a definition of a rectangular or circular region in source images which is actually used. Panotools supports two cropping modes, called Selection and Crop. Both croppings are defined by the left, right, top and bottom coordinates in pixel values with the top left corner as coordinate origin. Cropping is defined in the o-line of the PTStitcher script.

Selection

The character S preceds the actual coordinates. Only pixels from the respective area are taken, however, all other image parameters, especially Lens correction and FoV refer to the complete image. This implies:

• any source image can use a different sized crop area without the need to allow FoV and Lens parameters to be optimized for each image individually.
• any source image needs to be optimized for individual shift values, if the crop area is not equal sized and positioned for all source images.

hugin uses this crop type by default.

Crop

The character C preceds the actual coordinates. Only pixels from the respective area are taken and all other image parameters, especially Lens correction and FoV refer to the cropped area. This implies:

• all source image must use the same sized (but not positioned) crop area if FoV and Lens parameters should not be optimized for each image individually.
• it is not absolutely necessary to optimize all source images for individual shift values. However, it is recommended to do it, since manual cropping might introduce some minor positional errors.

PTGui uses this crop type by default.

--Erik Krause 23:50, 6 November 2007 (CET)