Detox your Linux box!
Let's get one thing straight - we're not saying that Linux isn't stable. There are Linux servers that have been running for years without a single reboot. But things are slightly different for desktop users. The problem is that we like to install things. Lots of things. In fact, you only need look at the average Linux package manager selection to see that one of the main reasons people use Linux is because there's a massive library of things to install.
And if someone hasn't developed the tools you want, then there are many users who are prepared to try and write their own. The net effect on the average Linux installation is that things will eventually start to break. It might not be in the first six months, or even the first year, but there will be a point when things start to fail. The detritus from two years of wanton package installation and compiling things from source will start to clog the smooth running of your system.
Lacklustre performance is usually the first sign that your system may be suffering. If your desktop starts to take considerably longer to appear, or the boot process is getting caught by a new addition to the start sequence, your system may be inhibited by the number of applications or servers being loaded at boot time. But it's system stability that's the surest sign.
When applications start to crash it usually means there's a software conflict. The problem is that very few of us stick to the official package provided by our distribution's vendor, and there's a good reason for this. The vendor seldom provides packages for new software, new versions or feature updates.
The people responsible for most distributions usually prefer to hold back on feature-adding upgrades, and use these as an incentive to make their users upgrade to the latest and greatest release of their distribution - particularly if they're charging money for it. The result is that many of us resort to third-party packages compiled by helpful members of the community. Unfortunately, this often means that third-party packages are flawed, either by linking against other unstable packages, or in the quality control of the application itself.
Things are worse if you need to compile new software yourself, because with most distributions this is going to break the dependency database used by your package manager. It's the final installation stage when compiling your own software that causes the problems - typing make install as your system administrator.
This gives carte blanche to the install script, which now has the power to copy and overwrite any file on your system. New libraries will replace old ones, and the delicate dependency balance that's handled by your distribution's package manager is broken. Dependency problems cause applications to stop functioning, but they shouldn't cause your machine to crash.
If this is happening, it usually means there's a problem with your kernel installation. It could be because you've installed your own kernel drivers or applied your own patches to the kernel. ATI and Nvidia's kernel drivers are infamous for creating problems like these, and sometimes there's no easy way to diagnose the problem. Upgrading the kernel might help, but this might also have a knock-on effect for the stability of the rest of your system.
In cases like this, it's usually counter productive to try and remedy each issue. A simple reinstall of the operating systems will solve most problems. But the average Linux user views this as throwing the baby out with the bathwater - you not only lose the instability but also all your hard-fought configuration, customised themes and applications settings. There is an answer - back up the stuff you need and let the rest go.
But what exactly is the stuff you need to keep? Where does it live and what's the most painless way to reinstall your Linux system without losing any of your valuable data? These are the questions we're going to answer over the next few pages.
Even though reinstalling Linux may seem an extreme measure, few of us stick with the same distribution for long, so it could be just the excuse you need to try out a new distribution, or change the way your system is configured. A new installation is the perfect excuse to get things right.
One thing you might want to consider is switching to LVM, the Logical Volume Manager that turns all your hard drives into a single logical device. If you find yourself running out of storage space (and who doesn't?), integrating a new drive into a stable installation isn't easy. Using LVM, on the other hand, means you can dynamically allocate space spread across each of your drives much more efficiently, and this is something you can only do before you install the operating system.
Pump up the volumes
Logical Volume Management is a good example of something you might want to consider with a fresh installation. But there are numerous other reasons, and whether it's because your system needs a serious detox or that you want to try something new, Linux offers more control over your current data than virtually any other operating system.
When compared with OS X or Windows, for example, Linux makes it relatively easy to migrate your current configuration options and data to a new installation. If you're lucky, you might even be able to get away with not touching your home partition, simply overwriting the root partition with the new operating system instead. All modern computers are blighted by performance and stability issues without the occasional purge, we should just count ourselves lucky that we don't need to start completely from scratch with Linux.
Step by step: Create a home partition
When you reinstall Linux, your distribution will want to reformat your hard drive. This is a good idea because it ensures that nothing survives, but it also means there's no haven for your own data. Creating either a separate data or home partition means that you can then safely reinstall your distribution without removing your data. This will be the most convenient solution for those of you with a thirst for new distros.
Launch GParted: Most distributions will let you repartition your hard drive at install time using a tool such as GParted. Just select the disc you want to partition and any delete any existing partitions. This will completely erase your data, so you must be either using a new drive or have already backed up the data you want to keep.
Create partitions: You'll need three partitions: for the root filesystem, for home or data partition and for Linux swap space. The first two should use the 'ext3' filesystem, while the swap partition needs to be 'linux swap'. The size of the first two depends on the space you have, while the swap partition is twice the size of installed memory.
Apply changes: After pressing Apply your changes will be written to the disk. You'll then be able to select these partitions when you re-install Linux. Remember to choose the corresponding mount point for each separate distribution.
Preparation and backup
If we were just going to do a fresh installation of Linux, everything would be relatively easy. We've all installed and reinstalled Linux many times, but the main consideration with a reinstallation is passing your valuable data on to the new installation while discarding rubbish that's holding your system back.
In principle, it should be simple: copy the contents of your home directory to a DVD, CD or USB memory stick, wipe your current installation, install afresh and copy your old home directory back across. But it's seldom that straightforward. First, you need to prune the data from your home directory, and second, you might need to keep some data that isn't in your home directory to start with.
A tool like Filelight can help you see exactly where your data is being stored.
It all comes down to the software you use, and how each piece of software differentiates between your vital data and its own configuration files. Your response to this might be "Well, most applications let me save my own documents to a custom directory", but what about your browser bookmarks or your painstakingly edited desktop backgrounds, your font configuration, your instant messaging buddy list, your email accounts and your email?
Most Linux desktops keep this information far from prying eyes. This helps with stability (by keeping it from the clutches of user error), but it can be a nightmare migrating this information to a new installation.
The main issue is that a fresh install is rarely as simple as installing the same version of Linux. Most users upgrade their distribution to the latest version in the same process, with the result that the new installation uses newer versions of all your favourite applications - it might use a newer version of Gnome or KDE, for example, and new versions like to change the configuration file.
If the programmer of your favourite application has been lazy, then your old settings become meaningless. If the programmer has been careless, an upgrade can cause all kinds of unpredictable problems. With the KDE instant messenger, for instance, things might appear to work at first, but you quickly find that you can't connect to the MSN network.
Even after deleting your login details and recreating the account, MSN still won't work. The configuration file has become corrupt as it now includes values from the previous and current versions of the messaging client at the same time. The only option is to delete the configuration file and start again. This is why, most of the time, a simple distribution upgrade fails to work and why a hand-picked selection of configuration files is better than arbitrarily copying every file you can find.
There are several different categories of application we need to account for. The first, and the toughest, is the desktop. It's tough because it's typically a sprawling set of applications developed by a gaggle of developers each with their own ideas on how to store settings.
They may rally under the same desktop, but they seldom agree on where to place a configuration file and how to use it. When faced with this, there's a lot to be said for blowing your configuration and starting from scratch with a fresh installation, but this shouldn't be necessary. With a little careful pruning, you can pass virtually everything that's useful on to a fresh installation. But you need to know what to save.
Each Linux desktop takes a different approach to configuration, with KDE being the most constant, but whichever you use, you need to prepare yourself for the hidden directories. These are the mainstays of configuration files, and can be a real pain to deal with. Most of the time, "out of sight, out of mind" works well for configuration files, because there's rarely a need to edit these manually. But this is obviously not the case when backing up your data, and you need to know exactly where they live or you're likely to miss a few in the process.
Hidden files and directories start with a dot, '.' and are mostly tucked away from view. In your file browser, you need to explicitly enable an option to be able to see these files and directories. In Gnome's Nautilus, you can press 'Ctrl+h' to show hidden files, while Konqueror users need to select Show Hidden Files from the View menu. The reason why this is important is because you need to see what you're doing as you prune the various directories and configuration files within the hidden hierarchy.
If you use KDE, take a look in .kde. The only directory we're interested in at this level is share. KDE mimics the root filesystem for your personal settings, which makes this version of share the equivalent of /usr/share, or whichever directory your distribution may use to hide the KDE files. As you change settings for KDE applications, they save their configuration information into the hidden share KDE directory.
Gnome is quite different, because it uses the gconf database to store most settings. These can be found as XML files inside the .gconf directory but you still need to be careful. The Evolution email client, for instance, uses gconf for your account settings and the .evolution directory to store your email. If you're not sure when to find information you know is somewhere on your system, check online with other users before pressing the terminal Reformat button.
What to save
What to keep and what to throw away is the vital question for a reinstallation. You want to keep everything that's useful, but not at the expense of reintroducing any of the same problems you may be suffering from. There's a lot to be said for a new start, but there's a lot of data you can safely keep without worrying about the consequences. Here's a list of some of the most important bits.
KDE is the worst offender for lumping useful information right alongside the temporary stuff. Ideally, you should be able to migrate your settings to a new installation by simply copying the hidden .kde directory from your old home to your new one, but in reality, this will include a lot of data you don't need.
The two directories you need to focus on are ~/.kde/share/apps and ~/.kde/share/config. For each application you use, there's likely to be a configuration file in the 'config' directory, and any data associated with the application in the apps directory.
For example, Kopete keeps your contact list in the apps/kopete directory and its account information in the config/kopeterc file. You'll need to copy both if you want to save your settings. Similarly, Konqeuror keeps your bookmarks in apps/konqueror/bookmarks.xml and your layout configuration in config/konquerorrc.
Other applications worth saving include KMail for your email settings, KHotKeys for customised keyboard layout, KWallet for your passwords, and Amarok for your music collection. You used to be able to rely on your POP email being saved into .mail or Mail in your home directory, but now it depends on the email client. With KMail, you need to save the share/apps/kmail/mail directory.
Gnome's internal configuration is more consistent than KDE's, thanks to the use of the gconf configuration database. You may have come across gconf when using the gconf-editor to update various Gnome settings. It's this database that stores the configuration options for the majority of Gnome applications. Email accounts for the Evolution email client can be found in the apps>evolution>mail folder. This is reflected in how data is stored in your home folder.
Each configuration file is an XML formatted document found under the same folder in the hidden ~/.gconf directory. For Evolution email accounts, this means you need to backup and restore the apps/evolution/mail folder, although it's easier to just store the entire Evolution folder. However, your actual email is stored in the ~/.evolution/mail directory, so you need to be careful.
Other Gnome applications you may want to save through gconf include Ekiga, Rhythmbox, Tomboy and Nautilus. Unfortunately, not every Gnome application is gconf compliant - make sure you check for hidden directories to make sure you catch them all.
Applications that are neither Gnome or KDE are likely to use their own local configuration directory - usually a hidden version of the application's name. Firefox will use a hidden .mozilla directory, and under this you'll find .firefox.
It's not worth holding on to your plug-ins, as these may cause problems when you reinstall, but your bookmarks should be saved. You can do this from either Firefox's Organise Bookmarks window, or by copying the bookmarks.html file hidden in the .default profile directory.
Older mail clients use either a hidden ~/.mail directory, or a simple 'mail' directory, and you can just copy this over to your backup medium. There are also system-wide settings to consider, but if you're using various database and custom configuration files, presumably you know where they are and how to back them up.
Step by step: Backup
There are as many ways to back up your data as there are to cook an egg, but choosing the easiest means that there's even less chance of a mistake. We use a tool called Simple Backup Suite, or sbackup, but the principle is identical to many other backup/restore tools. Create an archive of your files that keeps their file permissions and put this in a safe place to restore from when your new installation is complete.
Install the backup tool: Install the Simple Backup Suite using your distribution's package manager. It's a great tool that provides a graphical frontend for backing up your files, as well as selecting the ones you want to restore. Once installed, it will need to be run with system administrator privileges.
Tune your backup: This is the most important step. Make sure you include all the directories that contain your data. This might be the /home directory, but it could include a data directory. See the What to save boxout (p 58) for information on what to back up, but if in doubt, err on the side of caution.
Use external storage: Make sure you copy the backup data to a remote device, whether that's an external hard drive or USB stick, a network drive, CD or DVD. Do this by copying the backup to the device, or going to /var/backup and copying the files from there when the process has completed.
Restoring your data
Restoring data should be as easy as making sure the files you backed up are copied to the same place. But it's not. There are a few things you need to look out for. The most important things when recreating your previous settings are file permissions.
After you've restored a backup to a temporary location on your hard drive, take a look at the file permissions and user ownership using a file manager or by typing 'ls -l' on the command line. Even if you've created exactly the same user names, and placed your home directories in exactly the same locations, there's still a good chance that your distribution may have monkeyed about with file permissions, and things won't work.
If you've not come across file permissions before, they're easy enough to understand. Each file and directory has permissions for the owner, an associated group, and the whole system. Permissions can be either 'read access', 'write access', 'execute access', or any combination of the three.
This is a system-wide configuration, and it's used for everything from your email directory to your soundcard for audio playback. In Ubuntu, for example, if you look at the permissions for the soundcard device, you get the following:
ls -l /dev/dsp crw-rw---- 1 root audio 14, 3 2007-06-19 09:41 /dev/dsp
This means that Root is the owner of the '/dev/dsp' device, which is also a member of the 'audio' group. Both have read and write access (rw), and with a device, this means they can send data (play audio) as well as read data (record audio). The last dashes in the permissions show that there is no system-wide access available: you need to be either root or a member of the audio group to access the audio hardware.
Permissions on your own files work in exactly the same way. After you've installed your new system, have a look at the files you backed up from your old installation. You might see something like this:
drwxr-xr-x 6 1002 1002 4096 2007-04-11 13:51 sakamoto
Where are the owner and the group names? The problem is that to your Linux system, usernames and group names are really just a front for an identifier number. It's similar to the way the internet works, where hostnames are really just a front for an IP address and your computer needs to retrieve an IP address before it can do anything useful.
The two instances of the number 1002 represent the username and the group for the owner of the 'sakamoto' directory, and the reason why it's a number rather than a name is because the 1002 username and group don't exist on the system. You might even find that instead of 1002, you get an entirely different username and group. This is because another user has been given the user ID of 1002 and been swapped into the listing.
The answer to all these problems is to ensure you're using the same user and group ID as your previous installation. If you're sticking with the same distribution and you create each user in the same order, they will almost certainly have the same assignment. But if you're switching distribution, you'll encounter differences.
User group assignments and identification numbers are much easier to manage through a GUI.
Some distributions start the user ID assignment at 500, for instance, while others start at 1,000, as with our example. Most include a user manager that will let you manually define the user and group ID you want to use, as well as change those that have been set already. But you can also use the command line, and the command you want is 'adduser'.
Fortunately, you won't need to worry about any system-wide configuration files, as these are typically owned by the 'root' account, and this is one area that's the same across all Linux distributions. Root always takes the user and group identification of 0, which means that files you back up with root ownership will be restored as root ownership.
Step by step: Restore your data
The good thing about restoring your Linux installation is that you should already know where everything goes. The Gnome and KDE directories are going to be in the same place, and unless you've jumped to a vastly different Linux distribution, all other files should be placed in their original position.
We'll use the Simple Backup Suite to restore our previously backed up data onto a freshly installed Linux installation. The easiest approach is to restore the directories you want to a temporary folder before pruning the backup for files you no longer want to keep.
Select your backup: The first step is to launch the Simple Backup Restore application. You need to tell it exactly where your backups are stored. If you've used an external drive, then plug it in and click on the Use Custom checkbox before clicking on the folder icon to point the requester at the location of your device, This will usually be found under the /mnt or /media directories.
Restore to disk: After clicking on Apply, the list of available backups will be populated. You need to choose your full backup from the dropdown list. The files and folder that are included in your backup will then appear in the bottom panel. Select the directories you want to restore but click on the Restore As button in order to choose an alternative destination.
Prune new data: You don't want to overwrite your current data before you've pruned your backup files, so restore the backup to somewhere temporary. Then you can move the files and directories back to your working installation one at a time. Depending on the type of files you restore, you might need to restart your machine to update any system configuration changes.
Customise your approach
You might also want to consider using a temporary user to restore your setting. You can then use this user to test whether the applications you use are still working with the restored data. Does your instant messenger still connect? Has your email database survived intact?
If you're having problems, you can try restoring your data again by omitting a few of the files and directories that aren't necessary. Even if you're left with the bare minimum of usable data, it's still better than nothing, which is what you get with a fresh install.
Another advantage is that you learn a good deal more about your system than you otherwise might, helping you to upgrade in the future and make better use of a separate home partition, which is a lot more than can be said for doing the same thing with an alternative operating system.
But for many of us, reinstalling our Linux system time and time again has become a fact of life. Over many years, you've probably lost count of the number of times you've inserted a fresh Linux boot disc into your machine. We're all addicted to installing the latest distribution releases as well as destroying perfectly working installations with a glut of downloaded software. It's something we've all got used to. Which is why you will no doubt have your own ideas on how to best navigate the rough seas of a reinstallation.
Even here at TuxRadar we all have a different approach to upgrading our favourite operating system. We each use different tools to keep our data intact - ranging from the blind system upgrade, to separate 'home' partitions and large USB sticks.
But you may choose to simply write-off your personal data and configuration files, or keep them permanently stored on a remote drive somewhere on your network. You might even simply press the upgrade button whenever you get a new installation. Whatever you do, we'd like to hear you own personal tips for surviving an upgrade.
You're likely to have your own finely tuned approach to the problem, and we'd like to build a database of ideas for newbies and experts alike. Is there anything that you feel we've missed? Or is there something you think that can really help? If you take the time to post a comment below, you'll help lots of other people!
Five quick detox tips
- Customise your distro Revisor, bundled with the latest Fedora release, is perfect for creating your own customised distribution and putting it on to a USB stick. A customised distribution will enable you to fine tune your own requirements as well as helping to keep your data to a minimum.
- Rescue is at hand If you run into problems restoring from a backup or reinstalling the operating system, have something such as SystemRescueCd close to hand for troubleshooting. This can help with LVM and booting as well as getting at your backup archives and even old data.
- Ghost a working install Use Partimage to make a ghost copy of a freshly installed and configured Linux installation. This can save you a great deal of time if you just need to quickly reinstall your system and it's perfect as a backup medium too.
- Remote storage Store your transferable data on an alternate drive. You can then wipe your Linux installation knowing that your important data is still safe. If it's network-attached storage, there's the added advantage of universal access.
- Use symlinks Symbolically link directories you know you want to keep using ln -s source destination. If the destination is on a remote drive, you only need to restore the symbolic links on a fresh installation to get access to your old data. Links will be saved as part of a backup, which is handy.
First published in Linux Format magazine