Wednesday, February 06, 2008

Hi, I’m OS X Server, Leopard Edition! Let’s Waste Some Money Together!

Normal (non-nerdly) updates will resume tomorrow. I just kind of had to get this off my chest.

Let’s start with these two screen grabs:

This is a graph of my company’s weekly bandwidth usage. The blue lines represent the data we downloaded from the outside world. The green lines are the data we sent out to the IntarWeb Planet.

And another one:

This is zoomed in view of the “hourly” usage.

What do we see here kidlets? We see TREMENDOUS spikes occurring at regular intervals and then stopping.

What do we think is causing all this data transfer? What pray tell, are we downloading off the Internet? Have we suddenly developed a taste for P2P file sharing? Are we downloading every movie there is from the iTunes Store? Is someone at my work a voracious pr0n hound?

Nope.

That is one of my Xserves, screwing me with my pants on.

Yes, I am quite annoyed.

The problem lies in Leopard Server’s Software Update service. (10.5.1, even) This is a neat little thing where the Server downloads all software updates from Apple and then distributes them locally to all the computers in the studio. It’s lovely for a couple reasons. One, it saves on bandwidth costs, since machines can just go to MY server instead of Apple’s servers to get 10.5.1, updates to iLife 08, etc. Yes, a 150 MB OS X update isn’t that big a deal, but multiply that out by 20 machines and it’s a whole lot of usage. Two, it’s way faster to get updates locally than to have to truck all the way to Apple’s servers.

So I use this little function quite a bit. It’s very handy.

Except that it’s horribly, horribly broken in 10.5.1.

First off, upgrading from 10.4.11 ruined the Software Update setup. When I tried to turn that on, I’d get an error. Unfortunately by the time I started digging around I already knew that I’d be re-installing the whole she-bang (because Open Directory/Kerberos hates me, but that’s a whole ‘nother story I won’t go into now). So I didn’t make a note of the error. But as I recall, it has something to do with a .bundle not being updated properly. So the fancy new Leopard stuff is trying to use a creaky old Tiger .bundle Theoretically, you could extract the new Software Update .bundle off the Leopard Server Install DVD and plop it into the proper location, after making sure all the permissions were correct.

Yeah, a big freaking chore and one you might not get right. If you have upgraded to 10.5 and things aren’t working right, plan on re-installing. Or praying that the 10.5.2 update fixes all this crap.

Second thing with Software Update, which relates to the graphs above:

After the clean re-install, I got it to work fine and I went about setting up all ten of our new Mac Pros. (And upgrading three or four other machines to Leopard). All is right with the world, except that I was sleep deprived.

But then!

Sunday morning at about 1 AM, the Mac I was setting up could no longer reach my update server and the Software Updaters on the other Macs were spewing “can’t find index.sucatalog at server_name” and no updates were going out.

“Hrrrrm,” I said sleepily, “That’s odd. I haven’t changed anything on the server. Why isn’t it working?”

I stopped and started the Software Update service and there was No Joy in Jonville.

Fine.

Reboot the server.

Nope. Still not working. Same error, they can’t find the file they need on the server. Let’s look at log files! Whee! In the Software Update Error Log we find:

[Mon Feb 4 01:55:34 2008] [error] [client 192.168.0.51] File does not exist: /usr/share/swupd/html/index.sucatalog

OK, let’s SSH into the box and see what’s there. Where is the index.sucatalog file? Maybe I can just manually find the little bugger and put it where it’s supposed to go. Or I can just change the config of the Software Update process to the right location.

Turns out that index.sucatalog is merely a symbolic link and the real catalog file is here:

/usr/share/swupd/html/content/catalogs/index.sucatalog

Fine.

Except.

When I dig down in the “content” directory, guess what I do not find? If you guessed “a directory named ‘content’,” you win a cookie. There’s a directory called “downloads” and in there is where all 18GB of Apple Software updates can be found, but no “content” directory. Which is how the Software Update service does its mojo. It keeps track of everything on Apple’s servers and itself and syncs between the two.

See? No “content” directory.

It’s royally hosed.

But the thing that is flummoxing to me is that not only is the service hosed from the outside and my Macs can’t get their updates (even though I did NOTHING), but it’s fried internally as well. Because it just keeps downloading the individual software update packages OVER and OVER and OVER again. It would start in the middle of the night, go for 7 hours and then just stop. And then later in the day it would start up again, completely saturating our 7MBit connection. All told, it download about 70 GBs worth of crap.

Our “normal” bandwidth usage in a MONTH is around 55 GBs. We only pay for 75 and everything over that is $MONEY.

So this clever little custom apache process called “Software Update” that is supposed to save me money on bandwidth? Will end up costing us a grundle.

I hope the irony is not lost on you.

I called Apple Enterprise Support (We have Premium support for servers). This is the gist of the conversation:

Me: YOU ARE KILLING ME.

Apple Enterprise Support Droid: Yes sir, we know about this issue.

Me: For the love Pete on a popsicle stick, when are you fixing it? Can it please be VERY SOON? I’m feeling very fragile over here. I’m not well.

Apple Enterprise Support Droid: We are very sorry about it. A fix is coming very soon.

Me: OK, let me know, please. I’ll be the geek in the corner, weeping quietly into his caffeinated beverage.

Here’s the hack/fix/workaround:

1. Make sure the Software Update service is off in Server Admin.

2. ssh into the server and remove ALL the directories BELOW /usr/share/swupd/html/ (Yes, any updates you have already downloaded will be nuked.) BE CAREFUL while slinging around the rm command, OK?!

3. Restart the server.

4. Start up the Software Update service.

5. The service should auto-magically re-make the index.sucatalog file and all the appropriate linkages. Log back into the box and see if it did.

6. Go away for a while and go outside. Talk to human beings for a bit. When you come back, see if the updates are making their way down to your server again. Note for the 800th time that though the Software Update service is pretty ginchy and useful, it would be EVEN nicer if there were some way to gauge how long the downloads will take. Progress Bars. LET ME SHOW THEM TO YOU.

7. Profit!

Tune in tomorrow, because I’ll be back to “regular programing” with a story about how my oldest son is either a genius or criminal mastermind in embryo! Whee!

Jon scribbled this mess on 02/06/08 at 04:01 PM, best we can tell it fits in the category of Regular Post Rants. This many folks had something to say about that, The permanent home of this entry is here: Link

Twitter

    Favorite Entries

    If you are new around here, the following entries have been reasonably well received. You might want to peruse these.

     

    Holy Crap! Look at all this STUFF down here. It's awesome!

     

    Really, I'm glad you made it down here. Almost no one ever comes down here. I'm like in a freaking dungeon down here. I get lonely. But not you. YOU made it all the way to the end of the page. For this I think I've a little crush on you. I don't know, is "love" to strong a word to use in this situation? Well, if it's not "love," then it's very strong "like." I'm totally in like with you for coming down here. You are awesome. Please love me back! I know, I know, I shouldn't be all needy, it's not attractive at all, but you don't know how it is to be stuck down here. Who scrolls all the way to the end of a page anymore these days? Anyway, thanks for shedding some light down here in the depths. I appreciate it. Shoot me an email and I'll send you a dollar, OK?


    ©2005-2010 Jon B. Deal All Rights Reserved. I'm not kidding around here, I know people who know other people who would be willing to beat you up or similarly infringe on your rights, should you happen to infringe on my rights.