Wednesday, December 06, 2006

Mac OS X Backup (Free Alternative!)

Jon’s First Rule of Computer Use: There are two kinds of computer users: people who have lost data and people who will.

The best part about the Rule is that you get to decide what kind of user you want to be. Personally, I have been the former far too often, so I’m a wee bit paranoid about data loss. Especially at work. I tend to set up elaborate schemes and systems to back up and archive my data. It’s all about the paranoia my friends. Like most of humanity, fright motivates me more often than will or anything else. Sad, but true.

Let’s say you are an enterprising sort of person/geek/dork/normal computer user and you have taken Jon’s First Rule of Computing to heart, nestled it under your pillow at night all snug as a bug, you’ve chosen the Right Way and now you want to make your data safe.

Let’s clear up a couple of terms first:

Archiving == moving the data from a hard drive to some other permanent media (i.e., CD-ROM, DVD, tape, stone tablets, etc.)

Back-up == copying the data you love from the hard drive you use on a daily basis to a different hard drive. This is so if your daily HD goes south for the winter, you can get your precious data back, hopefully by doing nothing more than closing your eyes and clicking your heels together seven or eight times.

Now with any good backup strategy, the key is that it’s easy to use, simple and you don’t have to think about it too much. (Get the feeling I’m trying to emphasize something here?) If it’s not easy, then you won’t do it. Also, it helps enormously if it’s automatic and “just happens.” Now you can go and spend your hard fought shekels on programs like Retrospect, SuperDuper or Chronosync, but then you wouldn’t have ramen noodle money. And plus with a little gumption and pluck you can use the BUILT IN tools that come on every single Mac that rolls out of the shiny Apple factories. It’s all UNIX stuff, but don’t let that scare you, good people. It’s easy UNIX. Plus, you are going to do the UNIX heavy lifting up front.

So go out to Fry’s or CompUSA or wherever and get yourself a Big Ol’ Hard Drive. FireWire or hi-speed USB are both fine. If you are simply backing up your computer’s data, you may skip to step 4. If you are backing up data that lives on other computers start from step one.

OK, let’s get to the good stuff. It’s a bit complicated, especially the beginning stuff, but remember... it’s Free! Plus, you get to play around in the Terminal and that’s ALWAYS, ALWAYS fun!

Assumptions:

One: you either have static IPs, local DNS or DHCP with static IPs. Or DHCP addys that probably never change, which if you are on a home network, is probably the case. Any of these are easy to “fix.”

Two: You have “admin” on all the machines you need to back up. If you wanted to make a bootable backup volume, you must have the “root” user enabled and there are a few extra and complicated steps, but let’s skip that for now. I’m assuming that we’ll just be backing up “User” folders like your Mail, Photos, etc. The good stuff that you *really* care about. You can always re-install the System and apps, the data is the key. :-]

Three: Enable “Remote Login” in the Sharing Preference Panel of the machines you want to back up. You have to have admin powers in order to do this and it essentially lets you log into the remote machine via SSH in the Terminal. Theoretically, this is a security risk, but it’s probably minor and hopefully your firewall doesn’t allow SSH into your network. If you have your firewall set up to forward and let SSH packets in (port 22 by default), beef up your admin passwords. If you are backing up an OS X Server, SSH is on by default (as it should be).

For this example I’m going to use the following users and machine names. Substitute the real thing as you see fit. I am also assuming that you are doing all this *from* the G5, since in this example it’s the most hunky, most likely to have an extra drive hanging off it, etc. :-]

MacBook — admin user: roamer (AKA the short username) IP addy: 192.168.1.10
iMac — admin user: bob IP addy: 192.168.1.11
G5 — admin user: fred IP addy: 192.168.1.12
The name of the Big Ol’ Hard drive where we’ll be putting the data on the G5 is called “Squid”. Also, there is a folder on “Squid” called “backup-o-rama” where I want the data to land. And inside “backup-o-rama” I’ve made individual folders for each Mac, i.e., “G5”, “MacBook” and “iMac”. See screenie:

Step 1: Create a public encryption key and then copy that public key over to the machines to be backed up. This is the mucky part, but in the end it saves you because the script you write won’t depend on logins and passwords. Or volumes being mounted on Desktops. As long as the one machine can “see” the other one, it’ll work. SSH and neat-o encryption will take care of all that for you. Some people might say this sort of thing (i.e., that allowing “password-less” logins is dangerous, but those people are WAY paranoid.) Everything will be fine. :-] In the Terminal type:

ssh-keygen -t dsa

then hit return three times to answer default for everything. You are generating an encryption key which lives on your machine and you will eventually copy the “public” encryption key to the other machines. The default location is ~/.ssh/id_dsa The “~” (tilde) means *your* Home directory. DO NOT give a passphrase. Just hit “return” twice when it asks for a passphrase. The -t dsa flag tells the ssh-keygen program that you want to generate a “dsa” or “type 2” key. You want type 2. More secure. Better bits. :-] You can also add a ”-b 2048” flag after the ”-t dsa” flag. This make the encryption key über long. The default of 1024 bits is probably sufficient for most everyone.

Here’s a screen shot:

Step 2: OK, now let’s do some more Terminal fun! Let start by setting up the imac.

type the following:

ssh and hit “return”

if you’ve never SSH’ed into the machine, you’ll get a big ol’ warning. This is normal and you must type “yes” to continue. Then give the bob’s admin password for the imac.

If all is well, you’ll have a prompt that says “[imac:~] bob%” depending on the machine name and the username.

This next step is really a bit obtuse, but bear with me. From the imac terminal session, go ahead and SSH *back* to the G5:

ssh

then you’ll probably have to do the same “warning” and typing “yes” thing before you can enter your password. After you have logged into the G5, go ahead and log right back out by typing ”exit” This will get you back to the “[imac:~] bob%” prompt. The reason we do this is so the “~/.ssh” directory is created automatically by the SSH process on the imac. Otherwise, you’d probably have to create the directory manually and then ”sudo chmod -R 700” the directory so the permissions would be correct. The SSH process handles this much more better, IMHO. :-]

OK, still with me?

Step 3: Fire up another Terminal session (on the G5, of course).

type this:

scp ~/.ssh/id_dsa.pub :

type in the admin password for the imac. Here we are using scp (secure copy) to copy the file “id_dsa.pub” over to the imac. Don’t forget the trailing “:” (colon) at the end.

Go back to the “imac” Terminal window session. Type this:

cat ~/id_dsa.pub >> ~/.ssh/authorized_keys

This command takes the contents of “id_dsa.pub” and pours it into a file you are making here called “authorized_keys”. SSH on the imac will check against this file and if the keys match (you send your public key from the g5 when using SSH), it trusts you implicitly and will grant you access without typing a password. Test it by logging out from the imac and then logging back in with ”ssh ” from the g5; you should immediately get to a prompt. No password required. If this didn’t work, start over, probably at the “scp” step. Essentially we have now built a trusted relationship between the g5 and the imac. Do the “scp” step to each of the other Macs from the g5 and you should then be able to login *from* the g5 to each of them without a password. You just need to have that “id_dsa.pub” data in an “authorized_keys” file on each of the remote machines.

Now in that same “imac” terminal session, go ahead and delete the “id_dsa.pub” file by typing:

rm ~/id_dsa.pub

You can also simply delete it via the Finder next time you are in front of the imac. It’s in the root level of the bob user’s Home folder.

Step 4: Now we are ready to do the “rsync” stuff. The real heavy lifting of all of this crap. The mucky part is done now. rsync can be tricky, but our set up here will be quite simple.

The basic rsync command goes something like this:

rsync [OPTIONS] (stuff you want copied) (place you want the stuff copied to)

Here’s a nice little command that will copy the contents of the bob’s Music folder over to the back up hard drive

rsync -a ~/Music /Volumes/Squid/backup-o-rama/G5/ --progress --stats

Let’s break down what’s happening up there real quick. rsync invokes the command. -a option tells the rsync command to march through the source directory (which comes next in the command) and copy the directory structure as well as the files to the destination. Basically, it mean that the nice organization of your files/folders is replicated on the backup. Which is nice. ~/Music is the source. You could also use /Users/fred/Music Remember, The “~” is a shortcut in UNIX-land for the current users Home directory. Next in the command we have /Volumes/Squid/backup-o-rama/G5/ which tells the rsync command where to put the stuff while it is copying. --progress and --stats tells the command to give a read out of how things are going as the copy moves along. They are not necessary, but personally I like to watch the bits fly by, but that’s just me.

Now let’s modify the command a teeny bit:

rsync -a ~/ /Volumes/Squid/backup-o-rama/G5/ --progress --stats

This does *everything* in your home folder, “Music”, “Documents”, even all the crap on your very messy desktop. (I’ve seen your Desktop, my friend, you NEVER clean it! What’s up with that?). Depending on how much data you have, this could take a while. But as you watch the files fly by, note that your Music folder was NOT copied. That’s because rsync is pretty clever. It looked at the Source (your home folder) and at the Destination folder, compared the two and saw that everything that was in the “Movies” folder in the Source was ALREADY in the “Movies” folder on the destination.

So now we have made a backup of the G5’s home folder. But let’s modify that command just a little bit more.

rsync -a ~/ /Volumes/Squid/backup-o-rama/G5/ --progress --stats --delete

This command will do the same comparison and anything on the destination end that isn’t in the source will be deleted. You can see where this might be dangerous, so whenever I started adding the --delete option I always add --dry-run to the end of the command:

rsync -a ~/ /Volumes/Squid/backup-o-rama/G5/ --progress --stats --delete --dry-run

This tells rsync to act like it’s doing what you told it to do, but NOT REALLY DO IT. It’s a “dry run”, get it? Massively useful for the paranoid or in my case, the scaredy cat-ish.

So what the practical upshot of adding the delete command? It keeps the destination disk from filling up, basically. Let’s say you have a file in your Documents folder called, “schmoopy.txt” You run the rsync command and it dutifully copies the file over. Now let’s say you toss “schmoopy.txt” into the Trash and then run the command. Well, without the --delete option, “schmoopy.txt” will live on over at the destination. Maybe this is what you want, but remember, we are making a backup, not an archive. Eventually, you’ll fill up your external hard drive with all the clutter of your life and that’s probably not what you want. So add the --delete option as you see fit.

If you are only backing up one machine locally and don’t need any of the network stuff, you may now skip to Step 6.

Step 5: Now that we have the G5 backed up, let’s move on to getting files off the networked Macs, the MacBook and iMac. Here is the command that will backup the bob Home folder on the iMac:

rsync -a -e :~/ /Volumes/Squid/backup-o-rama/iMac/ --progress --stats

What we have added is the -e and : stuff. The -e tells rsync the we are going to issue a shell command (the default is to use Secure Shell, i.e., SSH). The : tells SSH where to to get the data. In this case we are logging into the iMac (at addy 192.168.1.11) and logging in as user “bob” and then the Source directory is defined ~/, i.e., bob’s Home directory. Then we are telling rsync to plop that data from bob’s machine into the Squid. Depending on the speed of the network and how much data, it’ll either take a long time or a really long time to vacuum the data off the iMac.

Now one massively useful thing here is that you can push and pull the data from just about anywhere on the Intarweb where you have SSH privileges and some storage space. You can push data from your machine to a remote machine simply by switching the order of source and destination. For example:

rsync -a -e stevejobs@his_personal_machine.apple.com:~/Documents/secret_stuff “/Volumes/Jon’s Big external HD/stuff I’m stealing from Steve/”

Now since I don’t know the IP or DNS name of Steve personal machine, nor do I know his password, the command will fail. But you get the idea, I hope. And if I don’t have the password-less stuff up, I’ll have to type steve’s password when I run the command. Also, note that the back part of the command “/Volumes/Jon’s Big external HD/stuff I’m stealing from Steve/” is contained within quote marks. That’s because UNIX doesn’t appreciate spaces the way you and I do. It interprets a space as a new command or a new option to the current command. So if we have spaces or “funky” characters in our drive paths, we have to use quote marks (single or double) to set them off from the rest of the command.

Also, for long-ish commands, I find it quite useful to use a text editor, like BBEdit or TextMate to build my commands, instead of the Terminal. Easier to edit the command and then copy and paste into the Terminal. So now I have these three commands built:

rsync -a ~/ /Volumes/Squid/backup-o-rama/G5/ --progress --stats --delete
rsync -a -e :~/ /Volumes/Squid/backup-o-rama/iMac/ --progress --stats --delete
rsync -a -e :~/ /Volumes/Squid/backup-o-rama/MacBook/ --progress --stats --delete

I’ve tested each of the commands individually, and I know they work and my data will get copied properly, so let’s make this into a script that will run without my needing to type all that mess up every time I want to back the machines up.

Step 6: Fire up “Script Editor” which lives in /Applications/AppleScript and type (copy/paste as well) the following:

do shell script¬ “rsync -a ~/ /Volumes/Squid/backup-o-rama/G5/ --progress --stats --delete 2>>~/rsync_error_G5.txt || echo -n” do shell script¬ “rsync -a -e :~/ /Volumes/Squid/backup-o-rama/iMac/ --progress --stats --delete 2>>~/rsync_error_iMac.txt || echo -n”

do shell script¬ “rsync -a -e :~/ /Volumes/Squid/backup-o-rama/MacBook/ --progress --stats --delete 2>>~/rsync_error_MacBook.txt || echo -n”
You get the “¬” character by hitting Option + Return. It tells AppleScript to go to the next line, but not start a new command. It’s just so the spaghetti code we are making looks prettier and more legible. Notice that we’ve added quotes around the rsync command. If you have used any quotes in your command, because of funky characters or spaces, change them to single quotes. You can nest single and double quotes to your heart’s content, though I get confused as they get “deeper”. Also, note that I’ve added 2>>~/rsync_error_MacBook.txt || echo -n to the command. This tells rsync that if it encounters any errors, (the “2") to write those errors to a file in the G5’s Home directory. The “echo” command simply tells the shell script to hit “return” on any messages the rsync command might generate. If you didn’t make any mistakes, the script will compile and then you can test run it. If all is well, you can delete the --progress and --stats options. They aren’t necessary at all since this will run unattended.

Compile and Save the script somewhere on your machine (the G5 in this example, since we are running everything from the G5).

Step 7: Open iCal.

You can use an existing calendar, but let’s go ahead and make a new Calendar.

Make a new event:

Make an alarm and attached a script to it:

Make the alarm run your script. Set it to go off every day.

That’s it!

One caveat. If one of your Macs goes to sleep or can’t be reached on the network, the script will fail. I don’t know of a good way (besides Apple Remote Desktop or some other VNC program), to wake a remote Mac from sleep. If you are running OS X Server, it’ll never go to sleep, so this won’t be a problem. If you have a laptop or desktop that goes to sleep, you might want to change when the script runs or how soon after inactivity the Mac sleeps. Maybe just knock out the display, but let the Mac continue to go. If anyone has a decent way of waking up a remote Mac from sleep via AppleScript or the Terminal, please let me know. I run scripts like this throughout the day at work and no one even knows it’s happening. rsync doesn’t add a lot of overhead to the machine while it’s running. Only while it’s counting and comparing files does it register a blip on processor usage.

Also, all of the above might be obsolete once Leopard hits the streets. It’s suppsed to have some rather ginchy auto-magic backup stuff built in. Leopard probably won’t do the remote stuff, though.

Jon scribbled this mess on 12/06/06 at 01:27 PM, best we can tell it fits in the category of Regular Post. This many folks had something to say about that, The permanent home of this entry is here: Link

Wednesday, December 06, 2006

Mac OS X Backup (Free Alternative!)

Twitter

Favorite Entries

Search

Categories

Recent

Archives