Build a MacOSX Universal Hugin bundle with Xcode
The "normal" way of compiling Hugin is via Cmake. With the current versions of MacPorts, Fink and Cmake it is very difficult to make a universal bundle due to the way endianness is dealt with in MacPorts and Fink. This HowTo explains how to make a universal bundle with Xcode.
The creation of a bundle is actually a two step process:
- Build all libraries and binaries which Hugin depends on. This is done outside Xcode. From here on we will call these libraries and binaries "External Programs". To compile and build all "External Programs" as universal, we will follow a different process compared to the normal "straight-forward" process of building libraries with MacPorts or Finks as described in Hugin Compiling OSX. We do not need nor use MacPorts and/or Fink. Some may even prefer to put them "out of the way" in order to make sure we will not link with the libraries they provide. However, they provide convenient ways to install a few of the tools that we require in the later process and they should not be moved out of the way if they are where cmake is installed on your system. Some of the external package build scripts rely on cmake.
- Build Hugin and it's "internal" tools in Xcode and create the bundle. As the title suggests: This is done in Xcode.
Note: This Howto does not explain how to build a Hugin the "cmake way". Follow the Howto Hugin Compiling OSX.
You should have Mac OS X 10.4 or above. Older systems are not recommended and how to build on those systems will not be included in this document.
Download and install XCode
Download and install the latest version of XCode Tools for your Mac OS X: Xcode 2.5.x for Mac OS X 10.4 (Tiger), Xcode 3.1.x for 10.5 (Leopard) and XCode 3.2.x for 10.6 (Snow Leopard). We do not support 10.3.9 any longer. You need to register as a developer to be able to download XCode, but it is free and you will not be bugged by all kind of mails (unless you choose to receive them).
Leopard comes with SVN installed. The version is way too old though. This means that you need to install it yourself. The simplest way to get SVN is to use MacPorts or Fink. If you are on Tiger, you need to install it anyway. The simplest way to get SVN is to use MacPorts or Fink.
If you fancy a nice GUI you can download the Open-Source SVNX. You still need svn installed as it is only a graphical shell and I won't explain SVNX here (I only used it once, I still prefer the terminal).
You may also find SCPlugin handy for some quick operations.
Preparations for building the "External Programs"
Building the necessary "External Programs" (the libraries and binaries Hugin depends on) is completely scripted. This part describes not how to use "./configure" or "make; make install". It will explain (advise) how and where to create the necessary directory structure, configure the base environment script and, more or less, tell you in which order to run the package build scripts.
Installing Necessary Tools
Before start, we need to install couple of tools used in the process:
- GNU gettext
You may use either binary distributions like MacPorts and Fink. You can also download the official version and manually compile and install ("./configure; make; sudo make install"). Note though that in that case you need to take yourself of all dependencies.
$ sudo port install subversion gettext mercurial
If you use Fink:
$ sudo apt-get install svn gettext mercurial-py25
Note: mercurial is in the Fink "unstable" Source distribution. Set Fink to look for distributions in the unstable list. As of this writing there are three mercurial listings: mercurial-py24, mercurial-py25 and mercurial-py26. Py25 is presumably for python 2.5. The command python -V reports what python version is currently installed.
Start a fresh Terminal session before proceeding to the next steps. Otherwise the hg command may not be recognized.
Get Hugin from the Mercurial repository
For these instructions you need a folder named "development" residing at /Users/<your_username>. Create this folder in lowercase with the Finder if you do not already have it. This folder starts the path for what is called the development tree herein.
Tip: In subsequent instructions you will be entering folder path names and file names at the Terminal prompt. Instead of manually typing these in you can have Terminal provide the text for you by dragging the folder or filename item from an open Finder window into the Terminal window. When you do this Terminal deposits the name of the folder or filename you are dragging to the location you drop it in Terminal. For example in the following instructions you are to enter "cd development". Type in "cd " and then drag the development folder into the Terminal window to finish entering the text. You will find this technique is extremely convenient for the more involved path and file names.
Cd into your development tree and download hugin from the Mercurial hg repository like:
$ cd ~ $ cd development $ hg clone http://hugin.hg.sourceforge.net:8000/hgroot/hugin/hugin hugin
Inside /Users/<your_username>/development, you will now have a directory hugin. The full path to your "External Programs" development tree will be /Users/<your_username>/development/hugin/mac/ExternalPrograms.
On subsequent retrievals from the Mercurial repository you simply cd into the hugin development directory on your mac and you issue the following two commands:
hg pull hg update
Location of the "External Programs" development tree
This part describes where we want to place the development tree for our "External Programs" that hugin depends on. Although the "External Programs" directory structure is placed inside the Hugin tree by default, this does not necessarily need to be in the same place as the Hugin source itself. The easiest way to place those files in custom places is to put symbolic link in the default place.
The first question is: where do you want to have your development tree? As you (might) know, the "normal" location is /usr/local, and MacPorts uses /opt/local by default and Fink, /sw. We do not want to use these locations!
Apart from the fact that it is a bad idea to mix up development trees, another drawback is that these directories are not in "user space", therefore always requiring a root authorization, e.g. "sudo make install" as a last step. When keeping the development tree in user space (e.g. /Users/<your_username>/development/ or /Users/Shared/development/), you don't need to "sudo". Note that the latter option also creates a development directory but keeps it away from your "normal" user data.
So, from this moment "we" have decided to build our development tree in user space.
Note: As mentioned before: If you position your development tree outside user space, you need to run everything as root user. The scripts are not tailored towards that "sudo" kind of use and need modification to work that way.
Inside hugin tree
The "External Programs" development tree is placed inside the hugin tree when you download Hugin. After you downloaded Hugin from Mercurial, you will find inside the hugin directory the following directory structure:
- /...other directories inside hugin
- /<more files inside mac>
- /...other directories inside hugin
Say you have downloaded hugin in /Users/<your_username>/development (Remember that "we" decided to keep it in user space?), than your "External Programs" build tree will be inside /Users/<your_username>/development/hugin/hugin/mac/ExternalPrograms/.
Note: You will also find a mac directory inside the platforms directory. This hugin/platforms/mac/ directory is not meant for compiling cross-platform tools, but for hosting platform specific tools like Erik Krause's droplet scripts, which you will find in platforms/windows.
Outside Hugin tree
Based on what I explained above you could also decide to place your build tree for the "External Programs" outside the hugin tree. An option might be /Users/<your_username>/development/ExternalPrograms/.
Pro's and Cons of "outside" Hugin tree
- You have your "External Programs" build tree separate from the hugin source. You can delete and recreate the Hugin directory anyway and anytime you want without touching your carefully built "External Programs".
- If you plan to build more universal software using this approach, you can share this directory (or just as well build another one).
- Many of the default paths assume the "inside" approach; you will have to map some of the directories with symbolic link.
Creation of the "External Programs" development tree
If you leave the "External Programs" development tree inside the Hugin SVN tree, you don't have to do anything and for simplicity this HowTo focuses on that way of working.
If you want to create it outside the hugin tree, I advise you to first create a development directory inside your home directory and then create the ExternalPrograms directory inside that development directory. Currently some part of hugin's Xcode project assumes the "binary repository" (explained below) is located at mac/ExternalPrograms/repository, inside the same directory as the source code you have downloaded with Mercurial's hg.
The recommended for "outside" approach is to:
- make your "External Programs" directory
- place symbolic link to the "binary repository" directory in your "External Programs" directory at /<path_to_hugin_source>/mac/ExternalPrograms/repository
- optionally place symbolic link to /<path_to_hugin_source>/mac/ExternalPrograms/scripts in your "External Programs" directory
$ myPathToHuginSource="/Users/<your_username>/development/hugin/hugin-hg" $ myExternalProgramsDir="/Users/<your_username>/development/hugin/ExternalPrograms" $ mkdir -p "$myExternalProgramsDir" $ mkdir -p "$myExternalProgramsDir/Repository-dynamic" $ ln -s "$myExternalProgramsDir/Repository-dynamic" "$myPathToHuginSource/mac/ExternalPrograms/repository" $ ln -s "$myPathToHuginSource/mac/ExternalPrograms/scripts" "$myExternalProgramsDir/scripts"
Building the "External Programs"
Prepare the build environment
Our build environment uses a lot of preset environment variables. The template of those variables can be found inside hugin/mac/ExternalPrograms/scripts, named "SetEnv-universal.txt" and "SetEnv-leopard.txt". SetEnv-universal.txt is for conventional 2-way universal binary, and SetEnv-leopard.txt is for 4-way universal binary with 64bit computing support. We take SetEnv-universal.txt in this document. Read #32bit versus 64bit for more info.
First, you should copy either of those files and name it "SetEnv.txt". In the top of this new file you will find the following two lines:
# has to be the absolute path from / myREPOSITORYDIR="/PATH2HUGIN/mac/ExternalPrograms/repository";
The path in the myREPOSITORYDIR variable needs to exactly match the path you use. This example show the (current) default one from the SVN. So, if you are Spiderman and you build inside your HOME directory you need to specify:
Check it, and check it again !!
You do not need to modify anything below this line, unless you cannot resist to customise something for yourself.
The list of programs and their versions that you should compile are given in the hugin/mac/ExternalPrograms/readme.txt. Some libraries are recommended not to be compiled as dynamically linked library, but as statically linked library instead. The scripts for static build are found in hugin/mac/ExternalPrograms/scripts/static/.
Note: There is a good chance those build scripts not on the list are outdated and do not work.
Build the "External Programs"
Building the necessary libraries and binaries which Hugin depends on, the so called "External Programs" is now relatively easy.
- You cd into your ExternalPrograms subdirectory, like
- As mentioned in #Prepare the build environment we need to set our build environment before we can start compiling our libraries and binaries. This only needs to be done once at the start of the build process, so now would be a good time to do it:
$ source scripts/SetEnv.txt
Note: The above command does not work in the tcsh shell. Prior to entering the above source command switch to the bash shell first with the following:
You may enter this following command to verify that SetEnv properly set myREPOSITORYDIR.
$ echo $myREPOSITORYDIR
- Download the necessary source packages (Google for them, copy them from your MacPorts and/or Fink base if available). The readme.txt in the ExternalPrograms directory lists the ones you need.
However, it's also a good idea to look at the corresponding script to verify the version number, as the scripts tend to be updated more often then the documentation. For example: The readme.txt might list Boost (1.39) when in fact the boost.sh script might have a line BOOST_VER="1_40" that causes the script to look for boost related file names with 1_40 instead of 1_39. Therefore you should be downloading boost version 1.40. As a general rule, check the scripts if you see if they indicate a more recent version to use than what is listed in the readme.txt file. The opposite is now true. The readme.txt currently (6/26/10) lists boost version 1.41.
- untar/unbzip2 the source packages. It's best to do this in the ExternalPrograms directory so that you will have all kind of subdirectories containing the source, like jpeg-6b, tiff-3.8.2, enblend, wxMac-2.8.7 and so on. Perhaps the easiest way to unpack the sources is to double click on the source packages in a Finder window. They can also be unpacked within the Terminal window:
As an example for libpng:
$ bunzip2 libpng-1.2.24.tar.bz2 $ tar -xvf libpng-1.2.24.tar
- Build and install the source package. The basic recipe is this:
$ cd <source code directory> $ sh ../scripts[/static]/<script_name>.sh $ cd ..
Here's what that looks like for libpng:
$ cd libpng-1.2.24 $ sh ../scripts/libpng.sh $ cd ..
- A recent addition to the project is a script named build-all.sh. This script has all the commands needed to build all of the tools. This script is highly recommended. Edit the script prior to using it to make sure it points to the correct external source folder names that you are using and to make sure it points to the scripts folder. Spiderman would edit the cd line in his version of build-all.sh to be: cd /Users/Spiderman/development/hugin/mac/ExternalPrograms/scripts
- The build-all.sh script writes a "<<<< building xxxxxxx >>>>>" line prior to starting each package build script. If you use Terminal, when this script is running or after the script has finished you can search (Edit Find) for the text "< building". This takes you to the beginning of each build process where you can see any errors that have occurred at the end of the prior build script or see any errors at the start of the next build. These two locations are the most likely spots where you can see that something that has gone wrong. You can keep a second copy of build-all.sh, renamed to build-user.sh, where you comment out all but the one package you may need to rebuild. So if a package does not build for some error like the version number is wrong, and you spot it soon enough, you can start a second Terminal session and run the specific build-user.sh for that package while the other build-all is still running on the other packages.
- After building all the tools, you need to download the Perl module Image-ExifTool-7.96 (source version), and extract the archive in the ExternalTools directory. Nothing else needs to be done for this program.
Tip: Maintain a folder outside of your development tree to stash the compressed external source packages you downloaded in case you need to start over with one of these. When necessary, rename the compressed file according to the package name and version.
Note: cmake is required for at least two of the packages, autopano and pano. Your system has to be able to find cmake. If it is at /opt/local/bin or /sw/bin then do not move those folders prior to building. Verify cmake's availability by entering "cmake -version" in the Terminal. You will be up the creek without a paddle if you do not get a cmake version response.
Order of building the "External Programs"
Some libraries and programs are dependent on other libraries. This means that these libraries need to be built first. As a rule of thumb, build your libraries in the following order (shell script names):
boost.sh, libexpat.sh, libiconv,sh, gettext.sh, libjpeg.sh, libpng.sh, libtiff.sh, wxmac28.sh, ilmbase.sh, openexr16.sh, pano13.sh, static/libexiv2.sh, lcms.sh, static/libxmi.sh, static/libglew.sh
And for the executables:
gnumake.sh, enblend31.sh, autopano-sift-C.sh, panomatic.sh
Note: You need to examine the scripts before executing them as some script use major and minor library numbers. These numbers are set from the script and need to be changed if your library version changes. For example libpng.sh contains a version number that may to be corrected.
Note 2: There is a build_all.sh script. This script will build all libaries and binaries and it will do that in the correct order. Please do check the versions of the source as they will relate to the folder names used in the script.
Apart from the wxmac (wxwindows) source tree, you can remove every library source tree if you want to.
The wxmac source tree is necessary for the Xcode project. Xcode needs the “localization” source files.
When you are finished building you can also reinstate the Macports or Fink directories you had disabled (see #Get MacPorts and/or Fink out of the way).
Xcode compiling Hugin
First, you should edit the configuration file. Duplicate BuildConfig.xcconfig.orig and name the new copy "BuildConfig.xcconfig".
Inside, you will find variables that needs be matched with your External Programs configuration. Please edit at least the following variables accordingly:
REPOSITORY_ABSOLUTE_PATH WX_LOCALE_DIR EXIFTOOL_DIR
If you have chosen to compile 4-way universal binary with 64bit support, uncomment RELEASE_ARCHS_64 line.
XCode basic walk-through
Note: This walk-through is base on XCode 2.4.x/2.5.x. The 3.x version has changed but in general this walk-through is still valid.
This HowTo will not discuss how to use Xcode. It will only explain some very basic steps to get you going. The rest is up to you (Xcode - the Final Frontier...............To boldly go where no man has gone before.) Sometimes small changes need to be made to the Xcode project due to added tools (matchpoint recently) or added or removed source files. These kind of actions will not be explained either in this HowTo.
When you double-click the hugin.xcodeproj, Xcode will start and show you the following screen:
In the Top section you see the Menu/Toolbar.
On the left side you see the navigation panel.
On the right side you see the File section.
|In the left Navigation panel you see little triangles in front of the icons and their descriptions. These triangles can be used to open or close the sub-sections. Double-clicking the icons has another function and will bring you to the properties of that subsection. If you click the little triangle in front of Hugin, you see the directory structure of the files the Hugin project uses. Please note that this is not a real representation of the hugin directory but a user-created representation. Note however that the "files" in here are actually links to your real files. If you double-click them, the Xcode editor will open them for editing and save them back to the file system.|
|Below the Hugin icon/description, you'll find the Targets section. Here you need to define what needs to be compiled, linked, copied and so on to create a binary or library, or a bundle containing these binaries and libraries. In case of a complex build project like Hugin, you first need to compile underlying tools and libraries, than build hugin and link hugin against these underlying tools and libraries, and finally create the bundle including "some copy work" to get the "External Programs" like the PT* tools, enblend, enfuse, align_image_stack and the like inside the bundle.|
Other options in the Navigation panel are not relevant or interesting, although you might see the error panel quite frequently in your early attempts.
Checking Project Info
Double-click the blue icon before Hugin in the top-left corner of the Navigation pane. If you do this, the following screen will open.
This "General" tab defines the location where your Hugin.app, and the intermediate files, will be built. By default a build directory will be created in the same directory where your Hugin.xcodeproj resides. Changing this value is not recommended.
You can use Subversion with Xcode. It is a very handy feature to use in order to keep the files up-to-date and, especially with Xcode 3's improved Subversion integration, to commit changes back to repository. Follow the configuration of "SCM System".
The second tab is the "Build" tab (see below).
This is the list of master configuration of the project. The values that you have set in BuildConfig.xcconfig (above) are reflected, and referenced by other settings. All relative paths are relative to the location of the hugin.xcodeproj directory structure. You should not need to change anything, but do check. Please note some of the settings are different among "Configuration", e.g. Release, Debug, Development, etc. but paths should not be different among them.
The other tabs are not relevant, but feel free to expand your knowledge.
Compile and build our Hugin.app
Set the "Active Target" to "configure" and "Active Build Configuration" to "Release". Now click the "Build" icon. This will only take a few seconds or less. Please run this target for 2 to 3 times as it sometimes updates Xcode project setting itself and the updated variables are only used in the following execution.
In the status bar you will see the message concerning this step. It should say "Build succeeded" on the left and "Succeeded" on the right.
If this is the status message, you can really start building your Hugin.app.
This build target updates the pre-release version tags, and is not propagated (does not automatically run) from other targets. You should run this target after every SVN update. Also, before compiling anything that you give away to other people, make sure that you "Clean all targets" and then build this "configure" target first.
Tips: If you have multiple Mac OS X machines, you may be able to distribute the compilation over multiple machines on the same network. Install Xcode Tools on all machines and check out the Preferences of Xcode, under "Distributed Builds".
If everything compiles and builds correctly, you will finally get a status message like:
Note that the build may have succeeded even though you see some error messages (in this screen dump). Double-click the "error" icon to display the errors.
If you did stick to all the "default settings", you will find your Hugin.app inside ../mac/build/Release among lots and lots of other files. All these other (intermediate build) files are not relevant. If built correctly, the Hugin.app is a completely portable application and everything is inside Hugin.app.
Now you need to test run your bundle. The first test is to see whether your application runs at all. Quick test can be done by "Run" (Cmd-R) within Xcode. If Hugin launches, it's a party time!!! So, go party and when you're finished come back for the next test.
"Run"ing in Xcode is useful for debugging with Xcode, but unfortunately using external executable within Hugin confuses the GDB and Xcode, so you cannot stitch or run autopano programs this way. If Hugin.app launches finely from Xcode, get to Finder and double-click the Huing.app icon and try to make a panorama. If that works fine, go for another glass of wine.
Now that you managed to get a working Hugin.app via Xcode we need to check if it is really a portable application. If you were completely successful in building your Hugin.app, than all binaries, tools and libraries should be "inside" the bundle and should know "how to find each other". The best way to test this is to copy the Hugin.app to another Mac and run Hugin.app there.
As this is not always possible, You could follow the following dirty technique:
Warning: This is not exactly a safe operation. Do at your own risk.
- Copy your Hugin.app to another location on your mac and rename
- your mac directory inside your hugin source directory to mac.org,
- your /opt/local to /opt/local.org (in case you have Macports),
- your /sw directory to /sw.org (in case you have Fink),
- your usr/local directory to /usr/local.org
(You can off course rename your directories to anything you like).
Renaming your directories will prevent Hugin from trying to link to the libraries inside these directories. If Hugin does it will crash and show an error message that Hugin tried to link to <path to>/ExternalPrograms/repository/lib/<some library> instead of the bundle library (or even worse for example to /opt/local/lib/<some library>, which means that you did not correctly #Get MacPorts and/or Fink out of the way).
The Complete-bundle.sh script mentioned in #Edit Complete-bundle.sh will alter the library paths inside Hugin and inside the libraries to "internal" paths relative to the executable location, using the install_name_tool, to make sure that they can find each other inside the bundle. If this did not work correctly (one of the non-fatal errors for Xcode), your Hugin.app will not run on another system as it will still try to use the libraries inside your build environment. This build environment is not available on another "Xcode and Hugin free" Mac.
To see errors directly you can start Hugin.app from the command line, e.g. from a terminal:
$ Hugin.app/Contents/MacOS/Hugin &
Alternatively you can double-click Hugin.app from Finder while following the (error) messages on /Applicatons/Utilities/Console.app.
Next to this you should check the logs (in case of crashes that is). You will find these in /Users/<user name>/Library/Logs/CrashReporter/. If "things" go wrong you can find there logs like Hugin.crash.log. These logs are not recreated but new error reports are just added to the log, making them bigger and bigger (But off course you won't run into errors). The same crash logs may be displayed by the CrashReporter immediately after Hugin crashes. You can configure the CrashReporter's behaviour with /Developer/Applications/Utilities/CrashReporterPrefs.app.
If everything worked fine you really made a portable application. Congratulations!
Have a big celebration and have a happy hacking Hugin time!
32bit versus 64bit
Tiger (Xcode 2.4) enables you to build universal binaries and libraries for PPC and i386. Leopard and Snow Leopard (XCode 3.x) enable you to build universal binaries for PPC and i386, but also for PPC64 and x86_64. If you want to do this, you should refer to SetEnv-leopard.txt. At this moment this is "experimental" as:
- some "Linux derived" libraries and binaries may not work properly.
- they are not well optimized for Core 2 processors.
- most users do not benefit from 64bit because it is required only when making a huge panorama (>2GB).
- 64bit part is only for (Snow) Leopard users on 64bit hardware (G5, Xeon, or Core 2). Those platforms can run 32bit anyway.
- almost doubles the binary size. The 2-part universal version alone weighs more than 100MB.
Use at your own risk.
Command line building with Xcode
Xcode has also a command line version named xcodebuild. If you prefer the command line than this tool is nice. You miss the nice integrated editor off course, so you need vi or pico (or some other editor) to change source code. (I use it for for remote ssh builds using vi as code editor).
Say you want to use (or experiment) with the command line builder, you need to cd into the mac directory and issue the following commands:
$ cd <path_to>/hugin/mac $ xcodebuild -project Hugin.xcodeproj -alltargets -configuration Release
Note: Even if you run the build from the command line, the complete environment will be opened. When finished, it will close again.
If you want more info just issue a xcodebuild --help for short help or a man xcodebuild for more extensive help. And you can read the docu/helpfiles from inside Xcode.
If you want to make automated nightly builds of Hugin you can easily script that with the command line version (svn refresh, command line build, command line creation of the dmg, ftp to website). You could even issue the svn, dmg creation and ftp commands from the Xcode project which means that you only have to script the xcodebuild.
There are some command line tools in Hugin package and related tool chain that users would want to use independently of Hugin.app. The best way to make those command line tools is to build them statically linked of all non-standard libraries.
You need to (that's a lie, but let me simplify) compile another binary depository with this time compiling the second list in ExternalPrograms/readme.txt. If you have taken the 'outside' approach, you can just make another repository and proceed the next step. If not, the right location to have your repository will be given in the next step. That's all for tools that are built in External Programs manner including enblend and autopano-sift-c.
For Xcode, the trick is to make another folder that parallels mac/ in the Hugin source:
- Make a new folder next to mac/ and name it "mac-static".
- Copy BuildConfig.xcconfig, Version.xcconfig, and Hugin.xcodeproj files to the new folder.
- Make a new folder in mac-static/ named "ExternalPrograms".
- Either make the static repository here, or place a symbolic link named "repository" to the outside one.
- Edit BuildConfig.xcconfig and update the REPOSITORY_ABSOLUTE_PATH variable.
Now you are ready for Xcode. Just get the Hugin.xcodeproj open, run the 'configure' build target as usual, select the 'static tools' target or any specific 'static' tools, and build. You are done. If you check the compiled executable file with 'otool -L ', you should see only the libraries in standard OS X installation (that is, usually, ones in '/usr/lib').
- What is endianness.
- MacPorts set endianness read-only for the platform it's installed on.
- MacOSX online Darwin man pages from the HMUG user group.
- Cross-Development Programming Guide and Universal Binary Programming Guidelines, Second Edition– more about building fully compatible universal binaries on OS X