Compile source code - and solve problems
Building software from source - that's a bit old-school, isn't it? Who wants to wrestle with the command line, hunting down dependencies and coaxing the GCC compiler into running properly? Well, it does sound like a strange thing to do in this world of binary packages and online repositories.
We have thousands of packages available via the internet, all neatly compiled for our distros, thereby usually nullifying the need to get down and dirty with a Makefile. Or so it seems... Read on to find out why you may want to compile a program from its source code, and deal with the problems that can crop up.
As great as they are, binary packages have a lot of limitations that can only be overcome by compiling a program from its source code. Here are some of the benefits of building by hand:
- Timeliness Get software as soon as it's available. When CoolApp 2.1 is released, your distro will probably have only version 2.0, and (unless it's a rolling-upgrade distro like Gentoo) you'll have to wait until the next major release of your distro to get the new app. But by building from source, we can get the newest version of anything when it's released.
- Features In the interests of stability, distro vendors often disable experimental features when making a package, leaving you with a rather plain (albeit reliable) piece of software. When we go down the source-code route, however, we can turn on extra goodies - it's our choice!
- Optimisations When you compile from source, you get the chance to boost the program for your machine. Binary packages are built for a wide range of x86 processor types. Using certain compilation options, you can build apps specifically for your exact model of CPU, helping to wring extra performance from your PC.
To prove that it's not quite as difficult as it sounds, let's see how this works. The skills you learn here will put you in good stead for your Linux (and Unix-using) career, as the vast majority of programs can be compiled in this manner.
We're going to take a stock, unmodified Ubuntu 8.04 installation and install the Audacity sound editor, enabling some extra features along the way. Audacity is a good choice because it usually throws up some strange errors during the compilation process, so you'll learn how to deal with every eventuality.
The stock Ubuntu Audacity package: good, but we can add a couple of effects by compiling from source.
First off, we need to get hold of the Audacity source code. You can find this on the project's website at http://audacity.sf.net. Save the tar.gz file of the latest version to your desktop (eg audacity-src-1.2.6.tar.gz) on to your desktop, then open a command line window (Applications > Accessories > Terminal) and enter the following:
cd Desktop tar xfvz audacity-src-1.2.6.tar.gz cd audacity-src-1.2.6
The first command switches us into the Desktop folder, and the second command extracts the compressed .tar.gz archive. (Note that for archives with a .tar.bz2 suffix, you should use tar xfvj instead.) Finally, we switch into the newly created source code directory.
Enter ls to see the list of files in the expanded archive. Along with various scripts (highlighted in green) and sub-directories (blue), you'll see a README.txt file. It's always worth having a quick peek at any file called README.txt or INSTALL.txt, as it may have useful information on building the program.
Enter less README.txt to view the file, and hit the Q key to quit. In this case, we don't need to read through it all - we can get straight to work on the compilation process.
The configure script will prepare the Audacity source code for compilation. Enter this command:
This executes the configure script (the ./ is required, as we're running it in the current directory), telling it to spew out all available compilation options. If you scroll up through the terminal, you can see that there's a huge range of options available to us. But don't fret - we can happily accept the defaults.
Here's the output we see when running ./configure --help - you'll see all the pre-building configuration options that are available.
We're going to make a couple of small changes though: firstly we're going to enable the SoundTouch library, which provides extra pitch and tempo manipulation tools that aren't included in the default Ubuntu package.
Another tweak we can make is to set the location of the installed program when it has been built. To do this, we use the --prefix option with the configure script. For instance, to install Audacity in the /usr/local directory (away from other software and easier to maintain), we use --prefix=/usr/local.
As mentioned earlier, another great benefit of compiling software from source is the choice of optimisations. By setting the CFLAGS (for C code) and CXXFLAGS (for C++) environment variables before we run the configure script, we can select the exact GCC compiler optimisations we want to use. This is a complicated subject, but we can get an extra speed boost very easily. Consider these commands:
export CFLAGS="-O3 -march=core2" export CXXFLAGS="-O3 -march=core2"
Here we're setting up the aforementioned environment variables with optimisations suitable for the Intel Core 2 CPU architecture. The -O3 bit at the start has a letter O, not a zero, and requests the third level of optimisation. (Most distributions compile their packages with -O2 for slightly smaller executables and easier debugging.)
If you have a machine with a AMD64 chip, use -march=athlon64, and if you're not sure, just use -march=i686 for wide compatibility. You can consult the GCC manual page (man gcc) for a complete list of supported CPU architectures, but it's heavy reading!
So, we have our optimisations set up, and we're ready to run the configure script. Here we go:
./configure --prefix=/usr/local --with-soundtouch=system
This tells the script that we want to install into the /usr/local directory, and enable the SoundTouch option that was listed when we ran ./configure --help before. (The 'system' part for the soundtouch option indicates that we want to use the version of SoundTouch available in the distro, as the one supplied with Audacity is broken.)
Why compile at all?
If you haven't even glimpsed at a chunk of source code before, you may be perplexed by some of the terminology in use here, so let's clarify the terms and processes.
Compiling is the process of converting human-readable source code - a software recipe that could be written in any high-level programming language - into the binary instructions used by a specific computer's processor.
For the most popular programming languages, C and C++, the GNU Compiler Collection (GCC) does the hard work of converting the source code into a binary executable file. It reads the source files, translating the C or C++ code listing into machine code instructions for your CPU.
Different CPU families have different types of instructions; for instance, if you compile an app's source code to run on an Intel processor, the resulting program won't run on a PowerPC. You need to compile it for each type of processor on which it will be used.
Note that for interpreted languages, such as Python, Perl and Ruby, each line of the source code is translated on the fly. With those languages, there is no compilation process - the language interpreter steps through the program line by line.
Oops - something's not quite right. After you've run the ./configure command, you'll get this error back: 'C compiler cannot create executables'. We need to install a development toolchain to build binaries from source, so enter: sudo apt-get install build-essential
On Ubuntu, this installs a meta-package that pulls in all the tools we need to compile bare-bones command-line programs. If you're running another distro, look for GCC, G++ and Binutils in your package manager, and go ahead and install them. Now run the previous configure script again. (A handy time-saver for those new to the command line: hit the up cursor key and you'll be able to retrieve previously entered commands.)
Now the script will proceed more healthily, but it will stop and inform us when it can't find a program on the system called wx-config. Audacity uses a graphical toolkit called WXWidgets to render its interface, and we don't have it installed, hence the error message that you'll see. So we need to grab WXWidgets from the Synaptic package manager.
Before you run off with the mouse, though, it's important to note that many Linux programs, especially libraries and toolkits on which other programs depend, are supplied in two versions.
One version is the run-time library that a program can link to; the other is a bundle of development libraries and header files used for compiling software. So, open Synaptic (under the System > Administration menu) and search for WXWidgets.
Scroll down the list of results and you'll see libwxgtk2.6-0. This is the run-time library on which WXWidgets-using programs depend - it provides a GTK layer on top of the WXWidgets framework. Check that for installation, and also check libwxgtk2.6-dev, which will provide the necessary development bits and bobs for our source code build, and then apply the changes. (In other distros, these packages will have similar names, so search for WXWidgets or wxgtk to find them. Development packages will almost always end with a -dev or -devel extension.)
We can install an app's dependencies with a couple of mouse clicks in Synaptic.
Now re-run the configure script. You'll see that Audacity requires an older version of WXWidgets - namely, 2.4.x! This is pretty much the standard process for building from source code, which is why we've illustrated it for you here; it often takes a bit of trial and error to work out exactly what library dependencies and versions are required - and it can sometimes be frustrating. So, head back into Synaptic, remove WXWidgets 2.6, and install version 2.4 (don't forget the -dev package).
The magic of checkinstall
If you've got several PCs, or you're an administrator in charge of a large number of corporate machines, you probably won't have time to follow the whole compilation procedure for every single computer. Thankfully then, there's an RSI-eliminating tool that enables you to create binary packages of your source-built programs.
Checkinstall, available from http://tinyurl.com/2uodas generates Deb, RPM and .tgz packages for Slackware.
After you've installed Checkinstall, run the ./configure script for the software you want to build as described in the main guide. Then run make to compile the software, but don't run the usual make install command afterwards. Instead, as root, run checkinstall.
You'll then be asked a variety of questions, such as the type of package you want to build, and whether you want to include any documentation or not. When Checkinstall has finished its work, you'll have a binary package that you can then install on as many other machines as you like (providing they're running the same distro and have the same libraries installed).
Build the code
Run the configure script once more and you'll see that everything goes smoothly this time. For other programs, you may have more dependencies to sort out before you can build - as mentioned, it's all a trial-and-error process.
For instance, if a program's configure script complains that it can't find the PNG development libraries, look for a package named something like libpng-dev. If you're building a Qt-based app, you'll need libqt-dev. Often the README file will provide a full list.
The command we need to kick off the GCC compiler is (drum roll please)...
Simple as that. The 'make' utility consults a newly created file called Makefile, which details the order in which the program's source code files are to be compiled.
Depending on the size of your program - and speed of your machine - the compilation process can take anywhere between 10 seconds (for a small command line app) to several hours (for a leviathan like OpenOffice.org).
During the compilation phase, you'll see that GCC is being executed using the CFLAGS and CXXFLAGS settings we defined before. You may also see some warning messages about non-strict code, but those are nothing to worry about.
You will, however, get a bit of a shock when the compilation process halts abruptly with an error. You'll see that SoundTouchEffect.o couldn't be compiled - does that mean SoundTouch is missing?
Well, yes, and the configure script should have alerted us to that. But not all is perfect in this world, so we have to fix it ourselves. You may recall, back at the start, that we passed the --with-soundtouch=system option to the configure script, instead of =local, because the version supplied with Audacity is broken. (You would've found out that =local is broken, so we've saved you a bit of time!)
So, configure let us down a bit there. But no big deal: open Synaptic, search for libsoundtouch and install it (along with its -dev sibling). We don't need to re-run the configure step again, as that succeeded, so enter make again and watch the output.
Now you'll see that it can't find FLAC. D'oh! So fire up Synaptic and install the libflac-dev library package. Enter 'make' once more, and you'll see that the compilation progresses a bit further, but then halts with a rather terrifying sequence of errors, including this:
undefined reference to 'soundtouch::SoundTouch::SoundTouch()'
Well, it looks like the compiler still can't find SoundTouch, even though we've installed it and it didn't complain during the configure phase. So during the last part of the compilation, when all of the program's files are being linked together into the executable, GCC doesn't know where SoundTouch is.
We need to alter the Makefile which controls the compilation process. Enter:
Scroll down to line 31, where you'll see 'LIBS' - that is, the libraries with which Audacity needs to be linked. Add -libSoundTouch to the end of the line (it's case-sensitive), save the file and exit the editor. Run make again. You'll see that the main Audacity binary compiles correctly, but the process hits another wall: it can't find the msgfmt utility to handle documentation.
This time we need to rely on Google: search for 'msgfmt' and 'Ubuntu' (both terms in the same search, but not in quotes as it's not a phrase) and you'll see that other users have had similar problems when compiling programs. You'll discover that it's in the gettext package - but, tediously, the Ubuntu team has both gettext and a gettext-base packages.
The latter was installed as part of the build-essential bundle, whereas the former includes our essential msgfmt tool. Maybe one day distro developers will stop fragmenting programs into a squillion ambiguously named packages, but at least the solution was a quick Google away here.
So, install gettext to get msgformat, and run make again. In the words of Borat, great success! You'll land back at the command line prompt with no errors. All we need to do now is install the newly built Audacity into the filesystem, so enter:
sudo make install
This action needs to be performed as the root (admin) user, hence sudo. On non-Ubuntu distros, you may need to switch to root with su and then enter make install. And now you can run the program:
If you want to uninstall the program, do sudo make uninstall. You can also remove the source code directory whenever you want - the final program is now in /usr/local/.
Font of knowledge
You'll notice that this version of Audacity isn't looking so pretty on the font front. That's because the version of WXWidgets included with Ubuntu we used built against GTK 1, which doesn't have anti-aliased fonts. Recompiling WXWidgets to use GTK 2 is another exercise that you can try when you've finished this guide!
Anyway, load up a sound sample and click on the Effect menu. You'll see two new options that weren't available in the stock Ubuntu version: Change Pitch and Change Tempo.
Thanks to our geektastic compiling powers, we've got one up on Ubuntu. Eat it, Shuttleworth! Given that it's quite fiddly to add SoundTouch support and get those two extra features, it's not surprising that the Ubuntu team has steered clear.
Et voila: our recompiled Audacity. It may be using Gtk 1, but we now have two funky new effects to play with.
Most of the programs you compile will go completely smoothly or just require a few obvious dependencies, but we've looked at lots of issues that can crop up, so now you're trained and ready for any eventuality. Now go forth and compile!
First published in Linux Format magazine