Rolled Up Sleeves

Even though I bill myself on the “about” page of this blog as an archaeologist, I haven’t actually posted much here about archaeology. So here goes.

Well, actually, it’s not so much about archaeology, per se, as it is about my old stomping grounds, the Museum Applied Science Center for Archaeology (MASCA) at the University of Pennsylvania Museum of Archaeology and Anthropology. I worked, in varying capacities, in and for MASCA from 1993 to 2005. It’s there, running the computer lab, that I got serious about using computers in archaeology, and it was through MASCA that I was sent all over the Middle East as a surveyor. So it should be no surprise that I’ve remained interested in my old department. It, however, was a surprise to learn of its imminent dissolution.

Now, as anyone who’s worked at a museum can tell you, there is a tendency for museum departments to stagnate, and persist out of sheer inertia, even after they are no longer useful to the museum as a whole. Sometimes this is due to the unwillingness to deal with a persnickety hanger-on (in hopes that they’ll just grow old, die, and give up their office space), and sometimes it’s due to a weak managerial structure that lacks the power to reorganize when necessary. Though I would argue against it, I’m willing to concede that one could make a case for MASCA’s obsolescence. What I won’t concede, however, is the necessity of retaining good scientists at a research institution such as the University Museum. So I, along with the rest of the archaeological community, was shocked to learn that the Museum’s new director, Richard Hodges, had decided not only to dissolve the department but to also lay off its staff. (See the Daily Pennsylvanian articles about it here and here.)

Mind you, some of these scholars had worked at the Museum for decades. They’re productive and active researchers who bring the Museum lots of good publicity and help study the materials from dozens excavations, Penn sponsored and otherwise. Though that wing of the Museum is a dump, the work carried out there is of the highest quality. So the only plausible reason for dumping these researchers would be a really tight financial bind. That is, in fact, the rationale that Hodges gave. Unfortunately, however, the University President, Amy Gutmann, stated that it was a restructuring exercise unrelated to the recent financial crisis. So, clearly, these two are not on message. There was also very little transparency in the process, so if it was financially motivated, there was no public exploration of where money could be saved prior to the announcement.

In the past month, many prominent archaeologists have written open letters decrying the decision and urging other archaeologists to write letters to Hodges and Gutmann in hopes that they reconsider. So here is the letter that I sent them today. It’s not an especially well written or stirring piece of prose, but given my pedigree, I had to do my part.

Dear Drs. Hodges and Gutmann:

As a former MASCAteer and a Penn graduate, I have watched the recent imbroglio over MASCA’s fate with great personal and professional interest. Though even I could make a case for the dissolution of MASCA and the absorption of its researchers into other departments, your treatment of those researchers—their hasty firings—is, in a word, appalling.

Your failure to openly discuss your options prior to November’s announcement should trouble any outside observer. But, having announced your decision, the confusion of your public statements—each of you declaring different reasons for the decision—reeks of a poorly managed process. Since I was certainly not privy to any part of the process, I cannot imagine what your actual motives might have been. But, in light of the public outcry, I urge you to reconsider that decision.

MASCA’s involvement in major Penn-sponsored excavations, as well as in outside projects, has trained and employed Penn’s researchers and graduate students for years. Its ongoing contributions to the field have ensured that MASCA, through its ups and downs, remains an important nexus in the study and interpreta- tion of archaeological remains. Its dissolution would cheapen the Museum’s reputation and make Penn a decidedly less attractive school for incoming students.

But, of course, MASCA is more than just a legacy department, a bauble or reminder of past glory. It is, in fact, home to researchers who have built their professional careers there and, through their work, built MASCA’s and the Museum’s reputation. Pat McGovern, Naomi Miller, and Kathleen Ryan, in particular, are productive scholars, much esteemed by their colleagues. Their work continues to advance their respective fields and bring publicity—professional and popular—to the University Museum. Losing them will do the Museum irreparable damage, as it will greatly undermine its reputation as a serious research institution.

If the department truly cannot be spared—a proposition which has not been effectively articulated—then you must at least, for the sake of the Museum, spare these researchers’ jobs.

Sincerely,

Paul C. Zimmerman
PhD, Anthropology, Penn 2008
MA, Anthropology, Penn 1998
Research Associate, Near East Section, UPM 2008–2011
Research Associate, MASCA, UPM 1997–2005
Research Assistant, MASCA, UPM 1993–1997

Well, as both of you may have noticed, this blog has gone pretty silent lately. As a chronicle of my tech trials, travails, and triumphs, this makes some sense: things are pretty quiet at work, and the problems have been mundane and not particularly blog-worthy. We’re still mighty busy, but there haven’t been any big projects, or problems.

The other reason, though, for my silence—and, no, this won’t be a “sorry for not blogging lately” post, because I’m not sorry—is my latest obsession.

Back in ’94 and ’95, on a dig in Yemen, I fell in love with the old Land Cruisers that we’d drive around. Not the Jeep looking like FJ40s, but the early SUV style FJ60s and FJ62s. Roomy, capacious, functional, bulletproof, and absolutely brilliant off-road, they quickly shot onto my list of all time favorite cars alongside 1958 and 1959 Chevys, and 1970 1/2 Firebird Formulas. So, in return for finally finishing my dissertation, my wife figured out that she’d let me get one if we could find it cheap. Which she did…on craigslist…and only 40 minutes from my house. A beige 1985 FJ60, with under 150k miles; stock, with a clean interior and rusty undercarriage, for under $1000.

That was the end of September. Though I fired it up before buying it, a broken fuel line—rusted out, natch—kept me from driving it. Instead, I borrowed my brother-in-law’s SUV, rented a trailer, and hauled the thing home. (It was the most harrowing drive of my life.) The following week, I was away in L.A. for a festschrift honoring my undergraduate advisor, so I couldn’t start working on it until October. Since then, I’ve been getting an hour or two each weekend to dink around with the new rig, trying to rehabilitate it.

Slowly, slowly, I chipped and hacked away at the rust, dropped the gas tank, replaced the rusted out fuel lines, and got the whole thing bolted back together again. Two weeks ago, I fired the engine up again, and last weekend I took the Mrs. for a joy ride all the way to the end of the block. (It’s not registered, and doesn’t have plates, so I didn’t want to risk running into the police.) As excited as I am about the toy and the process of working on it, there hasn’t really been much to blog about it. It’s actually pretty mundane mechanical work. But, boy does it feel good to skin my knuckles and get grease under my fingernails again. I haven’t done that in years.

As yet unnamed, my new obsession is being referred to as “The Truck” or “My Truck”—so that may be its true name. I’ll let you know when it (she? he? it’s also currently un-gendered) tells me what to call it.

One little computer-related comment on the whole process: it turns out that Land Cruiser obsessives are a particularly chatty bunch, and apparently are online during all waking hours that they’re not working on or playing with their trucks. On Sunday I found a loose wire hanging in my engine compartment, so I took a picture of it and posted it to the forum on ih8mud.com (a site for devotees of these vehicles). Within minutes, literally, I had a positive identification and an explanation of why it was hanging there with no apparent purpose. Man, this internet thing is useful. If word gets out, it could really take off!

I’ve been off with my kids visiting my Mom in Minnesota for the last two weeks. Nice little vacation, with lots of hiking, swimming, and beer swilling. (Though the latter was mostly me, not the kids.)

Anyhow, now I’m back at work and getting back into the swing of the server upgrades and imaging issues before school resumes. Our servers needed a lot of TLC because we had an unintentional power outage when I was away—an electrician accidentally flipped the master switch on our server room’s UPS—and they didn’t all come back online properly. So I spent way too much time yesterday and today fighting with some balky issues, the worst of which was my inability to log into the administrative account on our old Xserve.

That particular account is an Active Directory mobile account, and the server is our Open Directory master in the “Golden Triangle” on our network, so I figured that the problem was network-related. The problem itself was a repeatedly crashing systemuiserver and runaway processes upon logging into the administrative account. Other accounts, including root, however, weren’t affected. Deleting the account and re-creating it didn’t solve the problem, though, and I couldn’t replicate the problem with any other AD accounts, either. Thinking that it was a cache problem, I’d deleted the /Users/administrator as well as the contents of /Library/Caches and /System/Library/Caches, but the problem persisted no matter how many times I deleted the administrator account or how I rebuilt it.

Finally, at the end of the day today, I found a little hint in the system logs that suggested that the problem was a corrupted font cache. I thought I’d cleared all of those in my prior troubleshooting, but it turns out that Leopard also stores cache files in subdirectories of /var/folders. So, finding and deleting the directories there that were owned by administrator solved the problem. I’m still not sure whether or not the problem was actually a corrupted font cache, as opposed to some other cache stored in /var/folders, but I can now log back into the Xserve with the administrator account, and for that I’m happy.

And on a side note, the Trader Joe’s “Charles Shaw Blend” Sauvignon Blanc is surprisingly good for a sub-$5 bottle of wine.

Update 11/30/08
I’m not sure whether I’ll get back to regular blogging any time soon, but I figured I’d post a quick update to the above note about font caches. As it turns out, there’s an actual official way of clearing Leopard’s font caches. “atsutil databases -removeUser” clears them for the currently logged in user; “sudo atsutil databases -remove” removes them for all users and the system; and “sudo atsutil server -shutdown” restarts the ATSServer. I wish I’d known about it sooner.

Really. It says so right on my “About” page. I know that I’ve been blogging pretty much exclusively about my tech hassles at work, but I also do other things too. Not much, to be sure, but still…

Anyhow, last summer I built a bedroom in our basement for my father-in-law. (It’s actually his house, so it’s only fair.) I got the whole thing done except for the trim by early Fall. Probably would have finished the whole job, but my brother-in-law, who lent me his chop saw for the job, needed it back. Without that, I only had a cheap plastic mitre box, and that made awful corners. So after trimming out the bedroom door with hand tools, I gave up. But a couple weeks ago, I went to the Sears and bought a new chop saw and an electric brad gun. What a difference the right tools make! In about 10 hours total (spread out over two weeks), I’ve installed all the baseboards, trimmed the windows, and put a shelf in the closet. It’s not great work, but it ain’t half bad. Now I just need to paint it, install the closet doors, and move the furniture back in. I’ll post some pictures when it’s all done, and it’ll feel great to check another big project off of the to-do list.

One other little tidbit (aside from the value of using the right tools for the job): I’ve been using MDF trim, as well as planks for the shelving and boxing out the windows. I’d never used the stuff before, but had heard all sorts of good things about it. That and the fact that it was way cheaper than even the crappy quality, full of knots made me give it a go. Now I’ll never go back. Fast to cut, smooth, easy to handle, lighter than particle board, more substantial feeling than cheap softwood, and relatively inexpensive. So if you’ve got a carpentry project in the works, definitely consider MDF instead of wood. I know I will on my next project.

Unfortunately, though, this one wasn’t really such an enjoyable brew.

I’ve been holding onto a bottle of Dogfish Head 120 Minute IPA since December or January, to be opened when I finally sent of my dissertation to my committee. So Thursday was the big day; three copies were mailed to my readers, and I finally cracked open the vaunted brew. People that know me know that I like beer. A lot. And people that know the beers that I like know that I like heavy, hoppy IPAs. And people that know the breweries that I like know that I like Dogfish Head. Not always my favorites, but ones that I respect for their craftsmanship. I’d held onto this bottle, waiting for the occasion to drink it, on a friend’s recommendation. This beer, though, just sat wrong. Too sweet; too thick; and it smelled on alcohol. (Which, I suppose, I should have expected since 20% of it is.) Gave me a hangover, too.

Oh well, at least the dissertation is off.

A while back I declared 10.5.2 Enterprise Ready. This was based on my experience with binding to Active Directory and the behavior of WGM under 10.5.0 and 10.5.1. Well, last week 10.5.3 was released with a long list of bug fixes, among which were some substantial fixes for binding to Active Directory. “Yippee!” I thought, “this will make AD authentication even better on our Macs.” Of course, I forgot rule #1 of fixing things: if it ain’t broke, don’t fix it. So, as I should have predicted, 10.5.3 broke our ability to authenticate to our Active Directory domain. On our brand new Xserve, no less. Grrrr!

Anyhow, long story short and all that, yesterday I took the Active Directory plugin (version 1.6.1) off of a 10.5.2 client, and replaced the version 1.6.2 plugin that came with 10.5.3. And it worked! Everything is fine now, on OS X Server and every Leopard client Mac that I’ve tested.

I’ve reported the problem and the workaround to Apple, but if you find that 10.5.3 has broken your ability to authenticate to AD, then by all means give this a try. Not that I’m happy with Frankenstein systems, since upgrades tend to break these hacks, but a working system is better than a non-working system. (And that’s rule #2 of fixing things.)

Well, we never did figure out what exactly cleared up our wireless problems, but since we’ve been running w/o major problems for over a month now, we’ve decided to close our tickets with Apple and Aruba. So probably tomorrow will be hell. My other major project, the dissertation, is also rapidly coming to a close. So hopefully, I’ll have some more interesting things to do and say in the upcoming weeks. Thanks to Tom and Genki and everyone else who commented or emailed me off list. I’m not sure that we solved anything, but it’s nice to hear from other techies and compare notes.

I can’t say what the magic sauce was, but for the last few weeks our wireless has been remarkably solid. Maybe it was a combination of the “AirPort Extreme Update 2008-001 for Tiger” driver and an OS upgrade that we made to the Aruba controller. Maybe it was something environmental. Maybe it was something that I haven’t yet thought of. Or maybe we’ve just been lucky, and the flakiness will crop back up as soon as I declare it fixed. I really don’t know. But for now, we’re cautiously optimistic.

Since my last post over two weeks ago—they’re coming in slowly thanks to my dissertation, which is taking up all my spare time and energy—we’ve only had one real technical development. Two weeks ago, one of the Aruba engineers helped us upgrade our controller software and create new rules so we could put some of the AP-61s on B/G and some on A with the same SSID. This was to be a stopgap measure until we could get our hands on some AP-65s. The upgrade went well, and seems to have cleared up some problems we were having with some of our access points randomly power cycling. (We honestly don’t know what was causing that, but they stopped doing it after the upgrade—so I assume that it helped.) The down side to this setup is that we found out that Macs will roam from A to B/G, preferring B/G over A, when both bands have the same SSID. Testing since these changes were made has been very difficult, but it seems that we’re running better. A clearly works well, but B/G doesn’t seem to be giving us the same sorts of problems as it previously was I’m still testing, though, to see if these impressions are real or just a fluke brought on by a series of good days. (See the previous post for what happens when we test on a bad day.) When that testing is done, we’ll deploy some loaner AP-65s and AP-70s in place of our AP-61s in our worst-affected locations to see if we find any improvement.

If the problem is environmental, and there’s just too much interference on B/G in our location for consistently good performance, Apple changing their roaming behavior to prefer A over B/G would definitely help, particularly with the new access points. Also, however, Aruba tells me that they have a beta version of their controller software that does “client steering” to move clients from B/G to A when possible; we’ll probably upgrade to this version soon to take advantage of this feature, but I’d like to get some solid testing data before further complicating our setup.

The other interesting development was the long phone call that we had with Apple’s engineering team yesterday. They were very responsive, made some suggestions, and are committed to fixing this for us and the other clients in similar situations. The call also had an Aruba rep online, so they’ve agreed to combine their tickets and bring any software shared between us under a single NDA, which will really help the troubleshooting process. I’m not sure we’re running solidly right now, but if we’re not I feel confident (for the first time since last summer) that the solution is imminent.

Apple released AirPort Extreme Update 2008-001 for Tiger the other day. In our ongoing troubleshooting with Aruba and Apple, we’d been seeded a couple of pre-releases of this patch. (We were under an NDA, which is why I couldn’t mention them here.) So seeing this new patch cheered me up thinking that they’d finally gotten to the root of the problem.

Unfortunately, it didn’t. On Friday, we brought four Summer 2007 MacBooks w/ 10.4.11 and this new patch into one of our worst rooms for testing. Within 10 minutes, two of the four machines had lost their FirstClass connections. Whatever the cause of the disconnects, we can be sure that utilization wasn’t one of them; our school’s on Spring Break, and there were only a handful of people in the building, no more than a dozen of whom were on wireless (in many different locations).

So, to recap the latest events: Aruba’s made some configuration changes on our controller that have helped somewhat—most significantly, blocking UDP 5353 (mDNS/Bonjour) on the wireless—and Apple’s found the problems with the old drivers and declared them fixed with the new driver. But we’re still experiencing problems. When Aruba’s engineers were on-site, they found lots of interference around 2.4 GHz, so we’ll be looking into testing 802.11a more thoroughly and probably switching as much or our equipment off of 802.11g as is feasible.

Of course, this would be less of an issue if FirstClass weren’t such a shit stain. And it would have been less of an issue if we’d bought AP-65 access points instead of AP-61s. (When we made the purchase, the cards in all of Apple’s machines were b/g only, and because of Apple’s secrecy, we had no idea that they’d be coming out with 802.11a-capable machines only a few months later.) But these are our legacy, and they’re what we have to support.