SoC 2008 student proposals
- 1 Template
- 2 Add your project here
- 3 James Legg, OpenGL Accelerated Hugin Preview
- 4 Marko Kuder, Utility for Creating A Philosphere
- 5 Michael Ploujnikov, Automatic Test Suite
- 6 Onur Kucuktunc, Automatic Feature Matching for Panoramic Images
- name / university / current enrollment
- short bio / overview of your educational background
- did you ever code in C or C++, yes/no? please provide examples of code.
- do you photograph panoramas? please provide examples.
- do you make other use of hugin/panotools than for stitching panoramas? please describe and show examples.
- were you involved in hugin/panotools development in the past? what was your contribution?
- were you involved in other OpenSource development projects in the past? which, when and in what role?
- why have you chosen your development idea and what do you expect from your implementation?
- how much time you plan to invest in the project? (we expected full time 40h/week but better make this explicit)
- please provide a schedule of how this time will be spent on subtasks of the project. While this is only preliminary, be aware that at the beginning of the project you will be required to provide a detailed plan, and during the project you will issue weekly progress reports against that plan.
Add your project here
James Legg, OpenGL Accelerated Hugin Preview
- I am James Legg, a 2nd year undergraduate in Computer Science and Mathematics at The University of York (UK).
- At university I have taken modules in human computer interaction, artificial intelligence, computer graphics and visualisation, theory of computation, algorithms and data structures, group & ring theory, linear algebra, vector calculus, and analysis among others. I have A-Levels in Physics, Maths, and Further Maths, and a Critical Thinking AS-Level.
- I have experience in C++ and OpenGL, for example I created a program that displays a procedural random terrain, including multiple layers of textures, using OpenGL, SDL (for platform independent event handling), and libnoise (for coherent noise generation). The source is available from http://lankyleggy.members.beeb.net (Download megamap.tar.gz)
- I do photograph panoramas and stitch them with Hugin, there are a few examples at: http://lankyleggy.members.beeb.net/panoramas/
- I also use Hugin for aligning bracketed photos to create HDR images, and I use enblend (and the Gimp) to create seamless textures from photographs. An HDR example and a tonemap are on the same page as linked above.
- I have only recently started contributing to Hugin, see http://groups.google.com/group/hugin-ptx/browse_thread/thread/f94c584095fcb9d0.
- I have not been involved in other open source development projects beyond use, evangelising and bug reporting.
- After completing this project, I expect to have a new optional preview display, acting largely like the existing one. This should instead use OpenGL, and a grid-like mesh to remap the images. It will only be approximate, but still functional. It will not require OpenGL shaders, as the hardware for this is not available to most users, although I would like to make it easy for someone to add this functionality at a later date. The main purpose is to speed up the preview process. Unfortunately, it will not have the accuracy and logarithmic HDR display of the current preview.
- I would like to do this project since I have an interest in computer graphics and human interface design.
- I will invest at least 40 hours a week after the first 3 weeks of the project. Unfortunately the 3rd week of Coding Phase 1 is an exam week, However I will attempt to make up the time before the official coding start date.
- I plan to split the project into the following stages:
- Beginnings: I will derive a new panorama preview from Hugin's existing one, set up an OpenGL context for it, and add extra preferences into Hugin to control selection of preview types, and the amount of texture memory and mesh resolution to use. I plan to have this done before the official "Coding starts" date.
- Texture creation: After this each image would be scaled appropriately and placed in texture memory.
- Photometric Alignment: OpenGL (without shaders) only directly supports linear colour transformations, however after this stage, white balance correction and (approximate) exposure correction will be used to display the images.
- Mesh creation: A grid-like mesh will be created for each displayed image. This will then be transformed under distortion correction, input lens projection and output projection, to get the coordinates used to make the display on the preview window. This will require using and extending the current remapping code. I will find a way to track duplicates (e.g. fisheye projections with a FOV > 360 degrees), and to split the mesh across discontinuities in the projection. The final output would then be a linear approximation between the mesh points (along continuous mappings), which is then handled by OpenGL. The results of the mesh transformations will be stored in memory so that changing parameters does not cause unnecessary costly calculations. I expect to have started this by the end of coding phase 1, but I expect this will be the most time consuming stage.
- Usability improvements: This will add extra features, e.g. a feature show the outline of an image in the preview, which will hopefully provide quicker identification of images.
- Performance enhancing and testing: I will try to improve the performance of my code, and fix any bugs I may have. I will call for testers on the mailing list during this stage, and I may add additional requested features if I have the time. I will also make sure that the documentation is completely up-to-date.
If this feature is not wanted, or mentors are not available, I have an alternative proposal: I would like to make a cube map projection, since cube maps are the most common way of handling static reflections in real-time computer graphics, they should work well with interactive panorama viewers such as FreePV, and it has been requested before.
I would keep the definition of the faces that make the projection generic enough (perhaps specified in a script) to allow shapes other than cubes to be defined (I see there is interest in foldable panoramas and other shapes, including ceating a Pilosphere). This will require work on making enblend and enfuse blend across distinct faces. I don't know any algorithms that can do this well, but I would be willing to research it and experiment.
Marko Kuder, Utility for Creating A Philosphere
Here is the application I intend to submit to Google, combined with the information I gave on the mailing list it should answer the questions in the template.
Hugin is a software tool for simple design of panoramic images. It's main function is to aid in combining multiple photographs, taken from a point in space in different directions, by finding common control points in neighbouring views and applying necessary transformations so that the images can be joined into a single, bigger picture. This can of course become an even tougher problem as photographs usually suffer from irregular lighting and Hugin is able to compensate for that also by color and light balancing.
The ultimate application of these functions is the creation of 360 degree panoramas. But such photographs have an innate problem - how to represent them in a natural and more attractive way than long and sometimes oddly curved pictures. The function that is yet missing from Hugin and would solve this problem is the ability to transform an image in such a way that it can be applied to a spherical body or, in practice, the automatic creation of models which can be printed and then folded into a polyhedron or sphere. The development and addition of this feature to Hugin would be my assignment on this project.
The original idea description can be found at: http://wiki.panotools.org/SoC_2008_ideas#Utility_for_creating_a_Philosphere
Printing panoramas as foldable cut-out models has been somewhat popularized by Philippe Hurbain, a French electronics engineer and a panoramic photography enthusiast (his homepage is accessible at http://www.philohome.com). The rhombicuboctahedron, which he often uses for displaying his panoramic photographs, was nicknamed »Philosphere«, after Hurbain's first name. Even though printing panorama models has little technical potential, it is an idea that could bring panoramic photography closer to the general public, as it displays the photographs in a more interesting way. For this reason and because image transformations are a problem domain which can be handled with my education without much practical work experience (which I lack), I decided to apply for this specific idea. This way I could get some much needed programming experience, which is often expected when applying for a job, familiarize myself with working on a bigger project, which I would have to get to know without being involved in it's previous development, and work on something that is interesting to me. Other projects which are part of Google Summer of Code mostly require special skills or their project ideas are a very integrated part of the software they are based on. Since I plan to attend classes and pass my exams in June, I had to be realistic about my abilities and choose a project that I could handle in the time I would have. I was also a bit late in finding out about GSoC, so I have chosen only one project, this one, and spent the free time I had recently getting to know Hugin, the mentoring organization's project (which I haven't used before) and making sure I was fit for the job.
While the basic idea of projecting an image on a sphere seems simple, it can get quite complicated if we want the projection to be as realistic as possible. The problem is that we have many types of cameras with different lenses and consequently multiple types of projections e.g. rectilinear, equirectangular, cylindrical and fisheye. Each of these should be handled differently and considering the fact that what we are working with are panoramas constructed out of multiple photographs, the simple problem can be expanded quite far. I do not intend to pretend that the original problem isn't trivial. If we had to map multiple non-stiched overlapping images to a single body, it would be difficult, but Hugin already contains the necessary code to join these images into one large panorama, so our task becomes easier. We can map an image to a sphere simply by converting it to so called »orange slices« (as shown on http://www.philohome.com/rhombicuboctahedron/rhombicuboctahedron.htm), which can be done by transforming each horizontal line of a slice into a shorter one, so it fits into the set form. There is a problem with this approach though. The final result depends greatly on the projection of the original image and the number of slices. The more slices we choose, the more equirectangular should the original photo be (as opposed to a fisheye projection), but such a model can be very hard to fold when printed. The alternative to orange slices which can be easier to assemble is the use of polyhedra like a rhombicubioctahedron or icosahedron, although model images of these could prove to be harder to construct, as multiple different transformations would have to be applied to separate parts of the original image. Some software which offers 3D panorama model construction already exists, like PTGui (http://www.ptgui.com) and IP-slicer script (http://www.bruno.postle.net/neatstuff/ip-slicer), but the former, which is a commercial application, is only able to construct a rhombicubioctahedron from a 360x180 degree panorama and the latter only converts an image into orange slices. Both lack additional user options, such as projection correction and incomplete panorama support, which I would like to include in my project. In my recent discussion with Hugin's group, Bruno Postle - one of it's members - kindly pointed out some solutions to similar problems which I could use as basis for my project. One is his image patterner (http://www.bruno.postle.net/neatstuff/image-patterner), which allows mapping images to arbitrary shapes (but by his words has some problems with performance and is written in perl, so it can't be used directly), and the second is the ability to specify arbitrary 3D meshes as foldable models by seam definition in the well known 3D modelling application Blender (http://www.blender.org/documentation/htmlI/x5336.html). Both present a possibility of parametrization of goal shapes and as such greater flexibility for further expansion of user options. While this would be great, I'm not sure at the moment if it could be developed in the time at hand, but it absolutely is an idea that I would consider, if not otherwise as further development after GSoC.
PROJECT TIME PLAN
At first I intend to get to know Hugin and it's source code as much as I need to visualize my planned code in it (including planning the GUI modifications). After that I should make some research and study the mathematical basis for my problem, so I could define the required code. At the start of the first coding phase, in June, I will have some exams to pass, so it could be expected that I will be less productive for a period of 2-3 weeks in that month, but I intend to get a head start on the project by beginning coding before the official start of GsoC coding phase one and I plan to already have something to show the mentoring organization at that time.
Coding phase one:
After my exams I will most probably be able to commit my full time to the project. Till the 14th of July, when mid-term evaluations begin, I intend to complete the inclusion of at least one of the planned transformations in Hugin with full functionality, so I could show some practical results.
Coding phase two:
After mid-term evaluations I would add as many additional functions and user options as possible, before doing the final bug fixes and polishing up the finished code and GUI to complete the project.
I was born and raised in Trbovlje, a small town in Slovenia and at the age of 22 I'm in the 4th year of the Computer Software university program at the Faculty of Computer and Information Science in Ljubljana, the capital of our country. My main interest is in computer graphics, although my dream would be to work in a small company designing software and electronic devices of various types, because I like inventing new things. From my education I have a lot of experience in Java programming and some in Python, C and C++, which would be my programming language of choice, for computer graphics in combination with OpenGL. Recently I have been working with our school's AI lab by doing research in the field of search algorithms and heuristics. I'm not much into photography, which may be strange considering the project I chose for my application, but I think my problem domain is separated enough from Hugin's main functionality that my lack of expertise in the area shouldn't be a problem. Google Summer of Code appears as an opportunity to prove to myself that I can work on real software projects and I hope I will be given the chance to show the same thing to Hugin's project team.
Michael Ploujnikov, Automatic Test Suite
This project is to develop an automated test suite to perform functional (black box) and structural (white box) testing of the Hugin internals as well as the user interface. The goal is to increase developers' confidence in the current code and future changes to the code as well as to improve the user experience of the application.
- a suitable framework will either be chosen from among existing frameworks or a new one will be developed based on analysis of the project needs and discussion with the core developers
- the test suite will be integrated with the built system
- a number of useful test cases will be created
- procedures for creating unit tests, running tests and analyzing test results will be documented
My name is Michael Ploujnikov and I am currently finishing up the last year of a Computer Science program at the York University of Canada.
I have written a lot of C programs for school assignments, just learning the language, personal projects and free/opensource software projects (FOSS). Most of my C++ programming experience is in working with FOSS projects.
I shoot hand-held panoramas some of which can be viewed in my public photo gallery. Making even "regular" panoramas with Hugin is so much fun that I have not had a chance to try anything fancy.
I have not been involved with Hugin in the past. However, I have been involved in programming roles with other free/opensource software projects. My official resume has some more details about my involvement.
I want to work on this project for a number of reasons. I strongly believe in test-driven development and wish to make it a possible (to some extent) for the current and future developers of a really fun application - Hugin! I am currently taking a university course on software testing and would like to apply the learned concepts while they are fresh in my mind. I want to learn how Hugin works and writing test for the code base is arguably one of the best ways to do so. I want to help the experienced Hugin/panorama developers who volunteer their free time to work on this application by doing a rather tedious and boring task of setting up an automatic test suite for them. As a result, I hope the developers will have more time to spend on less "trivial" tasks. Finally, I want to get a feel for some unit testing frameworks (other than JUnit) that could be useful for me in the future career and FOSS projects.
I'm committed to work at least 40 hours per week for the duration of the GSoC on this project except for the first week in April when I will be busy with final exams, and another one week (to be determined) when I will be gone on vacation.
In Chronological order I will
- Finish school exams
- make sure I can run Hugin compiled from source
- Research existing testing frameworks specifically for the C/C++ languages and wx/GTK GUI toolkits
- Research testing frameworks that work well with cmake
- Make a list of test suite features I would like to have
- Modify the test suite feature list by discussing it with core Hugin developers to address their needs
- Decide (with core Hugin developers) whether an existing framework (could be more than one) is suitable for Hugin
- setup the existing framework(s) with stub tests or start creating a new framework (depending on how the previous decision goes)
- write some simple tests
- add concise usage documentation for the test framework on the wiki
- integrate the existing tests (I only found stuff in hugin/trunk/src/hugin1/tests)
- think about the testing techniques learned in school and use an appropriate one to come up with a "good" set of tests cases
- update the documentation because at this point the testing framework will probably have been modified
- write some tests for currently open bugs
- search the google group for proposed test cases and try to integrate them into the suite
- keep adding more tests until time runs out!
Onur Kucuktunc, Automatic Feature Matching for Panoramic Images
Aim of this Summer of Code project is to develop an efficient method for automatically matching local image features. Automatic image stitching, the process of combining images to form panoramic picture, can be performed in two steps: first step is to extract feature points (scale, rotation, illumination invariant), which is handled by "Feature Detection" project. Second step, also the goal of this project, is to find a set of corresponding features between images by using a matching algorithm.
Cover trees (a nearest neighbor matching algorithm) and RANSAC outlier pruning (an iterative method for fitting models in a data set with many outliers) will be implemented in order to find the correspondence between images. EXIF heuristics can also be used for improving the speed of feature matching. The time a picture was taken, and its filename give some information about the order of images in a panorama.
In a panorama photo stitcher software, an automatic feature matching algorithm needs to find the correspondence between feature points and images. The problem is to develop a robust and efficient matching algorithm.
Using nearest neighbor matching and RANSAC outlier with EXIF heuristics seem to increase robustness and efficiency of the feature matching method. Although the complexity of matching n images with a straight-forward approach is O(n^2), we can reduce it to O(nlogn) by using a nearest neighbor algorithm . Cover tree  is a remarkably fast nearest neighbor algorithm, which have documentation and implementation in . RANSAC (Random Sample Consensus Algorithm) is an iterative method for outlier pruning. Fischler and Bolles propose and discuss RANSAC model fitting with image analysis in . There is also an overview and pseudo-code for RANSAC in wikipedia .
EXIF heuristics can also improve the efficiency and effectiveness of the method. If available, the time a picture was taken, its filename (like DSC_0001.jpg), camera orientation, etc. are useful informations which can both decrease the running time and increase its correctness. Geolocation information in EXIF format also give clues whether or not images belong to the same panorama. "Exchangeable image file format:wikipedia" document  gives details about the format.
We need to be sure that applied algorithms really improves robustness and speed of the method. Therefore, some comparisons should be made on a dataset, which we manually give matching feature points. Correctly retrieved matching points (hit), retrieved but not correct ones (false positive), and the matching points we could not retrieve (miss) could be plotted to a ROC curve. By this way, we can improve the performance by selecting the right methods and adjusting parameters to the value which give highest hit rate. For example, we can compare k-d trees, which Brown and Lowe used in , with cover trees. Running time of the algorithm is also important for our purpose. This can be calculated both by analyzing the complexity of the algorithms, and recording running times in each test.
I plan to invest at least 30h or more per week to this project. I will be on vacation for 4-5 days, and another week I will probably be busy with moving my house. But I will balance it by working hard in order not to fall behind schedule. My planned actions will be as follows:
- Before May 26
- Getting to know the developers, mentors, reading documentation,
- Getting used to hugin/panotools, your coding strategy, bug reporting and fixing,
- Finding and/or generating representative datasets for testing and evaluation purposes,
- Reading scientific material, understanding all the concepts of matching and pruning procedures,
- Beginning documentation,
- Deciding inputs and outputs of the project,
- Making contact with the developer of Feature Detection part,
- Making a survey to learn the habits of photographers for effective EXIF heuristics.
- May 26 to July 7-14 (midterm evaluations)
- Combining algorithms by using and adapting existing cover tree and RANSAC algorithms,
- Adding heuristic to implementation, test if it really improves efficiency and effectiveness,
- Preparing the libraries, standalone applications
- Getting ready for mid-term evaluations
- July 15 to August 11-18 or September 1 (final evaluations)
- Testing with a large number of images, offering code for public review,
- Improving heuristics and other parts with respect to the feedbacks and test results,
- Completing documentations, preparing libraries, applications, etc.
- Getting ready for final evaluations
- A library for automatic feature matching which uses proposed algorithms,
- Documentation created by using the doxygen tool,
- Comparisons of the algorithms, test cases for running time analysis,
- A standalone application (works in cross-platform) that uses the library,
- Fully-commented source codes both for the library and Hugin application that uses the library.
I'm Onur Kucuktunc, a MSc student in computer engineering at Bilkent University, Turkey. I'm currently working on our Multimedia Database System (http://cs.bilkent.edu.tr/~bilmdg/), which includes lots of parts related to computer vision. I have taken Basics of Signals and Systems, Image Analysis, Computer Vision, and Pattern Recognition courses. I'm also a photographer.
I have chosen this topic and Hugin/panotools project because last semester for Computer Vision course, I worked on a project titled as "3D Face Reconstruction from 2D Images for Effective Face Recognition". We proposed a 3D face reconstruction method by automatically matching SIFT features, since most of the current implementations still require manual selection of feature points among different images. You can find additional information in my website .
I'm mostly experienced in Java and Matlab programming; but I have also written low-level, operating system-related codes in C, and graphic codes (using OpenGL) in C++. As an OSX user, I also believe I will be able to report and fix platform-dependent bugs in hugin. I have i18n and translation experience in some open-source projects, but it will be my first time to actually contribute as a developer.
- J.Beis, D. Lowe. "Shape indexing using approximate nearest-neighbor search in high-dimensional spaces."
- A.Beygelzimer, S.Kakade, J.Langford. "Cover Trees for Nearest Neighbor." ICML 2006.
- M.A.Fischler, R.C.Bolles. "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography." Comm. of the ACM 24
- RANSAC, http://en.wikipedia.org/wiki/RANSAC
- M.Brown, D.G.Lowe. "Recognizing Panoramas."
- Personal website, http://cs.bilkent.edu.tr/~onurk/