SoC2007 projects

From PanoTools.org Wiki
(Difference between revisions)
Jump to: navigation, search
(Anti-ghosting HDR panorama blending and merging algorithm)
(Automatic feature matching for panoramic images)
 
(10 intermediate revisions by 7 users not shown)
Line 3: Line 3:
 
= Introduction =
 
= Introduction =
  
This is the work in progress list of possible projects for the [[Google_SoC_2007]]
+
The Google Summer of Code 2007 is officially over.
 +
 
 +
If you land here as a student looking for an internship opportunity, get ready in case there is a 2008 edition of this initiative:
 +
* join our community at [http://groups.google.com/group/hugin-ptx/ hugin-ptx].
 +
* learn and understand panorama photography. Concepts like the pinhole camera, 3D projections, etc. You are on the right Wiki for that!
 +
* learn about our code. [http://sourceforge.net/projects/hugin/ hugin] [http://sourceforge.net/projects/panotools/ panotools] [http://sourceforge.net/projects/freepv/ freepv]
 +
* one of the best ways to join the community as a coder is to submit a patch - download the latest SVN code, make an improvement / bugfix and submit a patch to the maintainers.
 +
 
 +
If you land here as a person with software development skills and time to spare, we'd love to have you in our community as well. The steps are similar as for the students.
  
 
Panoramic imaging is a very broad field and touches many different areas of expertise, such as photography, computer vision, art and programming.
 
Panoramic imaging is a very broad field and touches many different areas of expertise, such as photography, computer vision, art and programming.
 
There is a thriving community with experience from arts to science that provides many interesting ideas and explores new territory in panoramic imaging. In addition to the mentors, this open community will provide good support and innovative ideas.
 
There is a thriving community with experience from arts to science that provides many interesting ideas and explores new territory in panoramic imaging. In addition to the mentors, this open community will provide good support and innovative ideas.
  
Generally, all development should done with multiple platforms in mind (at least Windows, OSX and Linux/Unix). We have an open communication culture via mailing-lists and mostly develop using C and C++.
+
Generally, all development should be done with multiple platforms in mind (at least Windows, OSX and Linux/Unix). We have an open communication culture via mailing-lists and mostly develop using C and C++.
  
The project below are just suggestions. If you are an interested student and have questions or new ideas, please let us know on the relevant [[Discussion_lists]], for example [https://lists.sourceforge.net/lists/listinfo/panotools-devel panotools-devel].
+
The projects below are just suggestions. If you are an interested student and have questions or new ideas, please let us know on the developers discussion list [https://lists.sourceforge.net/lists/listinfo/panotools-devel panotools-devel].
  
 
= Development style =
 
= Development style =
Line 22: Line 30:
 
== Intuitive yet powerful GUI for panorama creation ==
 
== Intuitive yet powerful GUI for panorama creation ==
  
Goal: Redesign/Reimplement the graphical user interface of the premiere open source panoramic imaging suite, [[Hugin]], to increase ease of use, and provide better access to its unmatched capabilities. Currently the GUI is written using [http://www.wxwidgets.org wxWidgets], which has proven to have slightly different behaviours on different platforms, especially with a complex GUI such as the one of Hugin. This is very annoying and a lot of time has been spent on minor issues such as making the GUI look good on all platforms with different font sizes etc. Therefore two options are available:
+
''Update: this was implemented as a major refactoring of the codebase, separating wxwidgets GUI from the backend code and available since Hugin 0.7.0''
  
# Rewrite the whole GUI using QT (which is very consistent even across platforms, and has (IMHO) a much nicer API). This is a lot more work, and not all functionality of the current GUI can be recreated during the Summer of Code, but it will provide a better platform to build onto in the future. It is also possible to implement the GUI logic in a scripting language such as Ruby or Python. This project should only be tackled by a student who has experience in developing non-trivial GUI applications with QT. The GUI and core panorama code are already separated into different libraries.
+
Project successfully completed. The new GUI framework is there and it is waiting for coders to write widgets (in Qt) and bring the functionality to the surface. [http://google-summer-of-code-2007-pano.googlecode.com/files/Ippei_Ukai.tar.gz]
#* '''Details to be discussed on [[SoC_2007_project_New_GUI_Framework|the separate page]]'''.
+
# Extend and enhance the existing wxWidgets based GUI.
+
 
+
General goals for the improved GUI include:
+
* Providing a simple, yet helpful user interface that suggests or highlights potentially useful next steps.
+
* Enhancing and integrating manual and automated control point placement and management.
+
* Improving lens parameter management.
+
* Providing a batch processing interface.
+
* Expert mode with access to all features and internals.
+
 
+
Recommended knowledge or interest in:
+
* Workflow analysis and UI design skills
+
* Experience with building cross platform GUI programs (Windows/Linux/OSX), either using wxWidgets or QT
+
* Creative use of panoramic imaging
+
 
+
Mentor: Pablo d'Angelo, ?
+
 
+
License: GPL
+
  
 
== Automatic feature detection for panoramic images ==
 
== Automatic feature detection for panoramic images ==
  
Goal: Robust extraction of local image features using a Hessian-based detector and a suitable descriptor.
+
''update: this has been successfully implemented as [[cpfind]] in the current [[Hugin]] codebase''
A detector and descriptor that takes into account the approximately known distortions will have a much higher matching rate, especially when fisheye or wide angle images are used.
+
  
Tasks:
+
Project successfully completed. Code is being further improved. Integration requires feature matching.
* Implementation of the feature detector and descriptor, and a suitable test suite to verify the correctness of the implementation.
+
  
'''Further details are being discussed in the separate [[SoC2007_project_Feature_Descriptor]] page'''.
+
== Automatic feature matching for panoramic images ==
  
A desired result of the projects would be:
+
''update: this has been successfully implemented as [[cpfind]] in the current [[Hugin]] codebase''
* C or C++ library for the matching step.
+
''update: this was completed in GSoC 2008, control point generators still need work''
* Standalone program to extract the matches from images.
+
  
Required knowledge or interest in:
 
* signal or image processing background
 
* C or C++ development skills.
 
* Matlab or octave
 
 
Mentor: Pablo d'Angelo, Herbert Bay, ?
 
 
== Automatic feature matching for panoramic images ==
 
 
Goal: Robust and efficient matching of local image features.
 
Goal: Robust and efficient matching of local image features.
  
Line 71: Line 50:
 
* Implementation of the matching step, including geometry based outlier pruning (for example using RANSAC) and nearest neighbour matching, possibly using a fast algorithm such as cover trees. For the panoramic imaging use case, several heuristics could be used to improve the matching behaviour, including using the EXIF timestamps, or previously known approximate orientation of the images.
 
* Implementation of the matching step, including geometry based outlier pruning (for example using RANSAC) and nearest neighbour matching, possibly using a fast algorithm such as cover trees. For the panoramic imaging use case, several heuristics could be used to improve the matching behaviour, including using the EXIF timestamps, or previously known approximate orientation of the images.
  
'''Further details are being discussed in the separate [[SoC2007_project_Feature_Matching| SoC Feature Matching project]] page'''.
+
'''Further details are being discussed in the separate [[SoC2007 project Feature Matching|SoC Feature Matching project]] page'''.
  
 
A desired result of the projects would be:
 
A desired result of the projects would be:
Line 90: Line 69:
 
== Interactive panoramic viewer ==
 
== Interactive panoramic viewer ==
  
Goal: The [[Freepv]] panoramic viewer aims to provide a superior viewing experience
+
''Update: the freepv panorama viewer was integrated into VLC in GSoC 2008''
for panoramas on all major platforms (Windows, Mac and Linux/Unix), based on
+
exploiting powerful graphics hardware using OpenGL. Currently it provides
+
basic but solid viewing capabilities for Quicktime VR, cylindrical, cubic and equi-rectangular panoramas. Plugins for Mozilla/Firefox and a standalone viewer are available. Several important features are still missing from the viewer include:
+
  
 +
Project successfully completed. The code now supports also SPi-V panoramas. There are always advanced features that are not yet implemented and could be. If you have ideas of what you want to implement, and you have time to do it, contact us.
 +
 +
One idea is to turn freepv into a library and write a wrapper around it so that existing multi-media plugin can pass on to freepv qtvr and spi-v panoramas to display. this will avoid the current race condition, that either freepv or one of the existing multi-media plugins take over the content and so either VR or movies but not both can be displayed in the browser.
 +
 +
Improvement ideas:
 
* Support for hotspots
 
* Support for hotspots
 
* Optimisation for panoramas larger than the Video RAM
 
* Optimisation for panoramas larger than the Video RAM
 
* Display of high dynamic range panoramas with adaptive exposure
 
* Display of high dynamic range panoramas with adaptive exposure
* Support for reading a SPi-V compatible .xml file, for platforms where SPi-V is not available (Linux/Unix).
 
 
* Fallback software renderer
 
* Fallback software renderer
  
Line 107: Line 87:
 
Mentor: Pablo d'Angelo, Fulvio Senore ?
 
Mentor: Pablo d'Angelo, Fulvio Senore ?
  
Student: [http://wiki.panotools.org/User:Leonox Leon Moctezuma]
+
Student: [[User:Leonox|Leon Moctezuma]]
 +
 
 +
[[Interactive Panoramic Viewer|Project detail page]]
  
 
License: LGPL
 
License: LGPL
Line 113: Line 95:
 
== Anti-ghosting HDR panorama blending and merging algorithm ==
 
== Anti-ghosting HDR panorama blending and merging algorithm ==
  
Goal: Most HDR creation algorithms are designed to work only with very small variations in camera viewing direction. Assume that registration and response curve estimation has already happened. An improved blending method for HDR images together that have not been shot using the traditional exposure stack method. It should avoid ghosting and be insensitive to small misregistrations.  
+
''Update: anti-ghosting for HDR was implemented in GSoC 2008 and anti-ghosting for enfuse was implemented in GSoC 2009. Deghosting for enfuse will be available in Hugin 2010.0.0''
  
'''Further details are being discussed in the separate [[SoC2007_project_Anti_Ghosting | SoC HDR Anti-ghosting]] project page.'''
+
Project was successful and is being integrated in the next hugin release.
  
Interesting research papers:
+
== Processing of very large images ==
* [http://www.cs.berkeley.edu/~eden/737_eden_a.pdf Seamless Image Stitching of Scenes with Large Motions and Exposure Differences]
+
* [http://graphics.cs.ucf.edu/ekhan/project_ghost.htm Ghost Removal in High Dynamic Range Images]
+
* [http://research.microsoft.com/IVM/HDR/hdr_sg2003.pdf High Dynamic Range Video]
+
  
Required knowledge or interest in:
+
Project open.
* Strong background signal/image processing
+
* Creative mind with ideas beyond the state of the art in computer vision/graphics research.
+
 
+
Mentor: Pablo d'Angelo, ?
+
 
+
License: GPL
+
 
+
== Processing of very large images ==
+
  
 
Goal: Allow the creation of arbitrary large panoramas.
 
Goal: Allow the creation of arbitrary large panoramas.
Line 151: Line 122:
  
 
== Architectural Overhaul of Panotools ==
 
== Architectural Overhaul of Panotools ==
 +
 +
Some of this has been done in the GUI framework project.
  
 
Panotools is a very monolithic application. The goal of this project is to refactor the functionality of panotools into 4 main parts, as independent of each other as possible.
 
Panotools is a very monolithic application. The goal of this project is to refactor the functionality of panotools into 4 main parts, as independent of each other as possible.
Line 163: Line 136:
 
the creation and parsing of a script). It will  make it easier to replace one component of panotools with another one; and it will improve the future maintenance of panotools.
 
the creation and parsing of a script). It will  make it easier to replace one component of panotools with another one; and it will improve the future maintenance of panotools.
  
'''Details to be discussed on [[SoC2007_project_Panotools_Architecture|the separate page]]'''.
+
'''Details to be discussed on [[SoC2007 project Panotools Architecture|the separate page]]'''.
  
 
A desired result of the projects would be:
 
A desired result of the projects would be:
Line 179: Line 152:
  
 
== PTButcher ==
 
== PTButcher ==
 +
 +
''Update: the Hugin Batch Processor was implemented in GSoC 2008 and has been available since Hugin 0.8.0''
  
 
A batch creator-project manager for batch creating projects, optimizing, managing groups of projects, names outputs, indipendently invoke the software parts and feed scripts.
 
A batch creator-project manager for batch creating projects, optimizing, managing groups of projects, names outputs, indipendently invoke the software parts and feed scripts.
Line 225: Line 200:
 
Proposal: Luca Vascon
 
Proposal: Luca Vascon
  
Mentor: needed
+
Mentor: Yuval Levy
  
 
Licence: GPL
 
Licence: GPL
 +
 +
Student: [[User:Stereo sl|Zoran Mesec]]
 +
 +
[[PTButcher proposal]]
  
 
== PtPatcher, module for Interactive panoramic viewer ==
 
== PtPatcher, module for Interactive panoramic viewer ==
 
Was: PTeditor2.
 
Was: PTeditor2.
 +
 +
''Note: This functionality is available via [[panoglview]] and [http://www.flickr.com/photos/36383814@N00/2845671569/ pafextract]''
  
 
The basic need would be to have it integrated in photoshop and Gimp, with a simple “paint on this side” interface. 16Bit workflow needed, as multiple input-output filetype.
 
The basic need would be to have it integrated in photoshop and Gimp, with a simple “paint on this side” interface. 16Bit workflow needed, as multiple input-output filetype.
 
See also skypaint for inspiration interface and capabilities.
 
See also skypaint for inspiration interface and capabilities.
  
There is a lot of overlap here with [[SoC2007_projects#Interactive_panoramic_viewer|SoC2007 Interactive Panorama Viewer]].  Note
+
There is a lot of overlap here with [[SoC2007 projects#Interactive panoramic viewer|SoC2007 Interactive Panorama Viewer]].  Note
 
that the viewer simply needs to fork and pass the current filename, pitch, yaw and Field of View to an external script which can handle
 
that the viewer simply needs to fork and pass the current filename, pitch, yaw and Field of View to an external script which can handle
 
the extraction, editing and reinsertion - The viewer itself doesn't need to be involved with any image processing.
 
the extraction, editing and reinsertion - The viewer itself doesn't need to be involved with any image processing.

Latest revision as of 01:25, 31 March 2011

See SoC 2007 overview for usage hints and a page list.

Contents

[edit] Introduction

The Google Summer of Code 2007 is officially over.

If you land here as a student looking for an internship opportunity, get ready in case there is a 2008 edition of this initiative:

  • join our community at hugin-ptx.
  • learn and understand panorama photography. Concepts like the pinhole camera, 3D projections, etc. You are on the right Wiki for that!
  • learn about our code. hugin panotools freepv
  • one of the best ways to join the community as a coder is to submit a patch - download the latest SVN code, make an improvement / bugfix and submit a patch to the maintainers.

If you land here as a person with software development skills and time to spare, we'd love to have you in our community as well. The steps are similar as for the students.

Panoramic imaging is a very broad field and touches many different areas of expertise, such as photography, computer vision, art and programming. There is a thriving community with experience from arts to science that provides many interesting ideas and explores new territory in panoramic imaging. In addition to the mentors, this open community will provide good support and innovative ideas.

Generally, all development should be done with multiple platforms in mind (at least Windows, OSX and Linux/Unix). We have an open communication culture via mailing-lists and mostly develop using C and C++.

The projects below are just suggestions. If you are an interested student and have questions or new ideas, please let us know on the developers discussion list panotools-devel.

[edit] Development style

Most of the projects below are related to Hugin, and some also relate to Panotools or tlalli. Hugin is mostly written in C++, and uses the VIGRA image processing library to support different types of images (for example, 8bit, 16bit and float (HDR) images). The core functionality is implemented in a platform independent C++ library, which is used by the GUI based on wxWidgets toolkit, and the command line programs (nona, fulla).

The development of the projects should take place in a separate branch of the projects CVS (or SVN) repository. Communication with the mentors should usually happen through the appropriate development mailing list. All code should work on the major platforms supported (Linux, OSX, Windows).

[edit] Possible Projects

[edit] Intuitive yet powerful GUI for panorama creation

Update: this was implemented as a major refactoring of the codebase, separating wxwidgets GUI from the backend code and available since Hugin 0.7.0

Project successfully completed. The new GUI framework is there and it is waiting for coders to write widgets (in Qt) and bring the functionality to the surface. [1]

[edit] Automatic feature detection for panoramic images

update: this has been successfully implemented as cpfind in the current Hugin codebase

Project successfully completed. Code is being further improved. Integration requires feature matching.

[edit] Automatic feature matching for panoramic images

update: this has been successfully implemented as cpfind in the current Hugin codebase update: this was completed in GSoC 2008, control point generators still need work

Goal: Robust and efficient matching of local image features.

Tasks:

  • Implementation of the matching step, including geometry based outlier pruning (for example using RANSAC) and nearest neighbour matching, possibly using a fast algorithm such as cover trees. For the panoramic imaging use case, several heuristics could be used to improve the matching behaviour, including using the EXIF timestamps, or previously known approximate orientation of the images.

Further details are being discussed in the separate SoC Feature Matching project page.

A desired result of the projects would be:

  • Standalone program for the feature matching part, which at the end should accept the features found by the automatic feature detection task. Preliminary studies can be done using the existing SIFT and SURF detector/descriptors.
  • Integration into hugin and a standalone executable similar to Autopano-sift or Autopano

Required knowledge or interest in:

  • signal or image processing background
  • C or C++ development skills.

Possibly useful resources and libraries:

Mentor: Pablo d'Angelo, ?

License: GPL

[edit] Interactive panoramic viewer

Update: the freepv panorama viewer was integrated into VLC in GSoC 2008

Project successfully completed. The code now supports also SPi-V panoramas. There are always advanced features that are not yet implemented and could be. If you have ideas of what you want to implement, and you have time to do it, contact us.

One idea is to turn freepv into a library and write a wrapper around it so that existing multi-media plugin can pass on to freepv qtvr and spi-v panoramas to display. this will avoid the current race condition, that either freepv or one of the existing multi-media plugins take over the content and so either VR or movies but not both can be displayed in the browser.

Improvement ideas:

  • Support for hotspots
  • Optimisation for panoramas larger than the Video RAM
  • Display of high dynamic range panoramas with adaptive exposure
  • Fallback software renderer

Required knowledge or interest in:

  • OpenGL or other 3D programming experience.
  • Creating cool and nice looking interactive experiences.

Mentor: Pablo d'Angelo, Fulvio Senore ?

Student: Leon Moctezuma

Project detail page

License: LGPL

[edit] Anti-ghosting HDR panorama blending and merging algorithm

Update: anti-ghosting for HDR was implemented in GSoC 2008 and anti-ghosting for enfuse was implemented in GSoC 2009. Deghosting for enfuse will be available in Hugin 2010.0.0

Project was successful and is being integrated in the next hugin release.

[edit] Processing of very large images

Project open.

Goal: Allow the creation of arbitrary large panoramas. [2]

Currently panotools, as well as hugin/nona require memory to hold the complete input and the remapped output image in memory, which consumes a lot of RAM, especially for spherical panoramas. VIPS is a powerful and modular image processing library that supports very large images and multiple processors natively. By porting the core remapping routines of panotools/hugin to VIPS, panoramas of almost arbitrary size can be computed.

A desired result of the projects would be:

  • VIPS operations that support the geometric and photometric (vignetting correction) transformations.
  • Standalone command line program that remaps images using these routines.
  • Program/script to convert panotools scripts to nip2 projects.

Required knowledge or interest in:

  • C/C++ programming
  • image processing

Mentor: John Cupitt, Pablo d'Angelo

License: GPL

[edit] Architectural Overhaul of Panotools

Some of this has been done in the GUI framework project.

Panotools is a very monolithic application. The goal of this project is to refactor the functionality of panotools into 4 main parts, as independent of each other as possible. These parts are (at least):

  • Calculation of position of images (optimization)
  • Mapping from input images to output images
  • Projection related computations
  • Parsing of input scripts/Generation of input scripts

Were this project successful the functionality of panotools will be available as a collection of routines that be called directly (as opposed to the current model that requires the creation and parsing of a script). It will make it easier to replace one component of panotools with another one; and it will improve the future maintenance of panotools.

Details to be discussed on the separate page.

A desired result of the projects would be:

  • Set of functions/classes that are available to hugin to perform optimization and remapping of images
  • Test suite to verify that the functionality before and after is equivalent.

Required knowledge or interest in:

  • C/C++ programming
  • Knowledge of Numerical methods
  • Basic knowledge of image processing

Mentor: D.M. German

License: GPL

[edit] PTButcher

Update: the Hugin Batch Processor was implemented in GSoC 2008 and has been available since Hugin 0.8.0

A batch creator-project manager for batch creating projects, optimizing, managing groups of projects, names outputs, indipendently invoke the software parts and feed scripts.

All the software is based on advanced find-replace in text files, generation of script files, but needs an user-friendly GUI, the main investment in the coding I suppose.

General

  • ability to translate projects between the various gui.
  • To search projects in subfolders
  • to keep history of all operations, with undos ability.

batch building projects

  • creates projects basing on the subfolder of a main folders, creates CPs and optimizes.
  • Default name schemes customizable for result images and projects. As well as project folder-locations
  • Template application (selection in base of image number, or exif)
  • Customizable CP finder and refinery
  • Customizable optimizer
  • report with result and-or creation of a little test image
  • capable of hugin-ptgui-whatever panorama project format.

project manager, you have a table of all your project, with ability to change, one by one or in group, and without opening them

  • relative and absolute paths for images, changing the full path or a part of it, allowing HD migrations even if projects are foldered in creative ways.
  • Path for output image
  • output image type, resolution, layers.
  • Involved software and modificators.
  • 16 bit workflow warnings

Batch stitching

  • ability of batch processing stitch operations, or to invoke the specific software and feed its batch

Ability to stitch panovideos: Given a template with the (n) of images involved and (n) folders, containing (m) png images extracted from the (n) streams to stitch together. Invoke itself the softwares, and cycles the batch to build (m) images in a new folder.

Support for HDR or ADR workflows, always template based. i.e. Given 3 set of 4 fisheye images and the corresponding template, it generates the main project, optimizes images together, and THEN, render 3images (a,b,c) involving images 0,3,6,9 (a) 1,4,7,10) (b) (2,5,8,11) (c)

See Panotools::Script for a perl approach to this. Note that the PanoTools script format is is a bit hard to understand, there is a proposal in the SoC2007 project Panotools Architecture to replace it with a more workable XML file format, so work should concentrate on the workflow aspects instead of another panotools script parser.

Proposal: Luca Vascon

Mentor: Yuval Levy

Licence: GPL

Student: Zoran Mesec

PTButcher proposal

[edit] PtPatcher, module for Interactive panoramic viewer

Was: PTeditor2.

Note: This functionality is available via panoglview and pafextract

The basic need would be to have it integrated in photoshop and Gimp, with a simple “paint on this side” interface. 16Bit workflow needed, as multiple input-output filetype. See also skypaint for inspiration interface and capabilities.

There is a lot of overlap here with SoC2007 Interactive Panorama Viewer. Note that the viewer simply needs to fork and pass the current filename, pitch, yaw and Field of View to an external script which can handle the extraction, editing and reinsertion - The viewer itself doesn't need to be involved with any image processing.

- If this should be a photoshop plugin it would be a possibility to interact with the Adjust (or PTAdjust) plugin, which can do the extraction and insertion within photoshop (might be possible for the Gimp, too. See Panorama Tools Plugins#Adjust for details.--Erik Krause 23:55, 23 March 2007 (CET)

The viewer should allow to save a fileproject in order to allow:

  • unlimited close and reopen of the viewer
  • multiple interventions and extractions with only one final reinsertion, in order not to loose quality
  • working in parallel with multiple images
  • batchable alone or with PTButcher, in order to extraxct all nadirs and/or zeniths, with given view angle 'XxY' of panos in a folder and then reinsert them all.

Proposal: Luca Vascon

Mentor:

Do we incorporate this with the viewer?

Licence: GPL

Personal tools
Namespaces

Variants
Actions
Navigation
tools
Tools