I recently got back from my one-per-year business trip, to attend the 2014 LISA Conference, and now that I’ve presented to my group, it’s time to make a more public write-up. I’ll start with some of the more personal or social things I did, and then dig into the sessions I attended and what if anything I learned.

This year I’ve been flying JetBlue a lot, now that they have reasonable schedules to the West Coast (and by “reasonable” I really mean “the return flight isn’t a red-eye”); their regular seat pitch in coach is wider than premium seating from either UA or DL, and when you’re checking a bag, the old-line network carriers’ bag fees really add up. JetBlue has non-stops from Logan to both San Diego (where my parents live) and Seattle, not to mention LA (where I usually end up spending at least a few days every other April or thereabouts) and the Bay Area. There is, however, one drawback: the B6 winter schedule has no transcontinental flights returning to Boston on Saturday. So I could either return on a Friday red-eye, or stay an extra day on my own dime, or fly another airline (Alaska is the biggest carrier at SEA and also had non-stops to Boston, code-shared with AA).

I chose to stay the extra day, and used the Saturday after the conference to do touristy things — I went to Pike Place Market for lunch, and then took the #40 bus out to Fremont to see the Theo chocolate factory. I was lucky in that, although the factory tour was fully booked for Saturday, there was a group of eight no-shows, so they let me join the tour five minutes in, after paying my $10. The tour was pretty meh. There really isn’t a whole lot to see in a chocolate factory, particularly on a Saturday when there’s nothing in production. The tour did take us into the kitchen, where the non-chocolate confectionery is made (caramel centers, ganache for truffles, and fruit and nut flavors that are mixed into the bars); it’s also where their enrobing machine is located — but again, with no production happening, you can’t really see anything. (And even when the factory is operating, you still can’t see much, since much of the machinery is closed up for food safety — there’s no giant Willy Wonka-style fountain of chocolate.) I did learn a few things: their organic sugar, which comes from Brazil, is distributed by a company from Arlington, Mass.; the bars with mix-ins were historically blended by hand, but now (with a recent factory expansion) have moved to a more automated production line; and they make a large number of SKUs that are not nationally distributed — some of them are sold only in the factory store. (It doesn’t hurt that Fremont is a trendy Seattle neighborhood, with plenty of residential and office development nearby and very frequent bus service, so the factory store is rather more accessible than the typical zone industrielle where food factories would be located.) I ended up spending just shy of $200 at the factory store, buying nearly everything (except coffee) that wasn’t available at my local Whole Foods. (I bought a few things that Whole Foods doesn’t carry but, as it turned out, I could have gotten from my local REI instead!)

At Pike Place Market I had a salmon club sandwich for lunch, and also bought several bags of chocolate-covered cherries from Chukar Cherries. A nut vendor sold hazelnuts (their own) and smoked almonds (from a California maker). I didn’t waste time watching the fishmongers throw fish around, but I also probably only saw a quarter of what was on offer there. (There were a number of things I might have bought except that I had no safe way of getting them home in one piece — and I don’t care to find out what the airline, even JetBlue, would do to a jar of jam or honey in my checked baggage!)

I have an unfortunate habit of forgetting one really important thing whenever I travel. I very nearly forgot my cell phone, which would have been a disaster — thankfully, the Logan Express bus ticket agent has an EMV credit-card terminal, and as I was paying with my corporate travel card, I didn’t have the PIN memorized and had to look it up. Groping for my phone, I realized that I had left it in the car, and I dashed off to grab it at the last minute. (As it turned out, I was in so much of a hurry that I forgot to re-lock my car after grabbing the phone — good thing nobody tried to break into it while it was parked in the lot.) Having successfully remembered my phone, what it actually turned out I had forgotten was any clothing warm enough for the frigid weather which gripped Seattle — and the rest of the nation — during the six days I was there. Normally at this time of year, daytime highs in Seattle are around 50° — but this year, this week, it struggled to get much above 40°, and to compound the insult, it was windy as hell. I brought only lightweight jackets and my usual short-sleeve knit shirts, plus a couple of shirts from previous LISA conferences. The LISA’11 shirt was the only long-sleeved shirt I had, and it was grossly inadequate to the task. (The LISA’14 conference shirt, which was short-sleeved, continued the trend of Usenix buying very light-weight fabrics that are suitable only as underwear or in very warm weather. The LISA’99 shirt was made of heavier fabric, but was, like most of the others, a T-shirt.) The conference hotel, as is the way of these things, was frigid, although not quite as cold as outside.

It was at LISA’13 in Washington that I made a fairly important lifestyle decision, after talking with LISA and Usenix regular David Parter (of UW-Madison). Parter had lost a good deal of weight, and he credited a mobile app that he was using. At the time, I was having to take medication twice a day for glucose tolerance and high blood pressure, and I was constantly paying to repair small tears around the seams in my shirts. I had been exercising on-and-off on a stationary bike with no appreciable results, and it sounded like the app that Parter was using might help me implement enough calorie restriction to actually make a difference.

It turned out that it did, and that as I lost some weight, it also became easier to exercise, and I was able to incrementally lengthen my workout from barely 25 minutes total to over an hour over the course of the following six months. (The Winter Olympics in February greatly aided this, but it’s really baseball that makes the biggest difference — with games nearly every night, I’m not at the mercy of 22- and 48-minute TV shows for the visual distraction that keeps me on the bike late at night when I’d rather be relaxing. This fall I subscribed to NHL Center Ice so that I could watch hockey during the baseball off-season.) I knew that I wanted (and in practical terms needed, in the presence of all the free food a conference entails) to keep up my exercise regimen as best I could, and since I was staying in a big-city business hotel, I knew both that there would be a fitness center I could use, and that I needed to buy something to wear while doing so — hotels not being so keen on guests walking through the public areas in their underwear! Luckily, when I stopped by my local REI, they had a bunch of cycle clothing on clearance, and I bought a couple of shirts and a couple pairs of shorts (one pair of double shorts that would be OK to wear walking around — if you don’t mind the big chamois butt-pad — and one pair of “lycra bandit”-type shorts). I made sure to try them on before leaving, and made a shocking discovery: I can actually wear a size medium — at least if it’s made of Spandex. A year ago, I would never have believed I could ever again wear anything smaller than an XL. But I’ve gone down seven inches in waistline, and for some styles and manufacturers, that puts me in a medium, at least for bottoms. I went back to REI the next day to buy more cycle wear (including a pair of “M” tights), which felt ridiculously good despite the expense.

(I made a similar discovery in the market for new underwear a few months ago: a 33 waist might be a small, medium, or large depending on which maker’s size chart you go by. I’ve been gradually replacing the old XL underwear, but I’m never going to have the build of an underwear model, so I don’t think I’ll ever be wearing size “small” smalls no matter how much more weight I manage to lose.)

I arrived in Seattle around midday on Tuesday, which was an unusual bit of lucky scheduling for me — far more often I arrive at the conference hotel well after the badge pickup desk has closed, requiring me to stand in line early the next morning. When I checked in, the hotel clerk asked me if I wanted to waive housekeeping for the duration of my stay, in exchange for a $5 voucher each day, which certainly takes the “feel good and let us not wash the towels every day” a step further. (I declined, but apparently he didn’t key it correctly, so the next morning a voucher was shoved under my door and sure enough, no housekeeping. I didn’t care enough to complain, though, until the toilet paper ran out!) I watched the local TV news, where they were talking about roads being blocked and houses being damaged by falling trees, all a result of the uncharacteristically high winds sweeping the region. I set up my airchecking station near the window, and was greatly puzzled by the extremely poor reception — there must have been some sort of conductive coating on the windows, because my ninth-floor hotel room, facing north, should have been able to get far better signals (and usable HD) from the West Tiger Peak and Cougar Mountain transmitter sites; as it was, the only stations I had a good signal from were those with in-town transmitter sites (a couple of translators and a few stations on Capitol Hill and Queen Anne Hill). Then I went to find the fitness center, and found my resolve to exercise every night wavering a bit, despite the consequences: I was in the “Union” tower of the hotel, and the fitness center was at the very top of the “Pike” tower — I would have to go all the way down to the conference floor, or worse still, the lobby, in my sweaty workout clothes. And when I got to the fitness center, it was absolutely frigid. (This turned out to be a good thing, but I was just shivering there, in my street clothes at resting metabolism.)

After all that, I got in touch with my former co-worker (and recent-ex-Amazonian) Noah, and we went to a seafood restaurant a few blocks from the hotel, which coincidentally was also just a few blocks from the office building where Noah used to work. We got caught up, having not seen each other for a few years, and he gave me a bit of an insider’s view of the Amazon sausage factory. (This week he started on his new job working for a software company elsewhere in Seattle — the sort of place that didn’t really exist 15 years ago when I first visited the area. Back in 1999, there was Microsoft, and Boeing, and their suppliers, Starbucks, and UW, and that was pretty much it. The whole region is much more economically diverse now than it was then, although it’s still fairly concentrated, industry-wise.) Noah attended the conference, and we met again a few more times — on Wednesday, after seeing how awful the free lunch looked, we met up with another former co-worker, Eric, and had lunch at an Asian restaurant a few blocks from the hotel. (Eric lives on Vashon Island, and I had originally planned to go out there on Saturday to take a tour of the many radio towers that call Vashon home, but in view of the cold weather, long travel time, and limited daylight, I decided to put this off until some other time.)

I met up with a number of the “usual suspects” after the conference began. I often spend time at LISA with Steve VanDevender, who works at the University of Oregon in Eugene; Steve also teaches a summer undergraduate course at UofO in system administration. There’s also a guy whose name I can never remember who works at the University of Alberta in Edmonton — one of the few people for whom the weather in Seattle was actually warmer than what they had left. There were as always a lot of Googlers, but I only really know one, Chris Davis, who joined Google with their acquisition of ITA Software, which was an MIT-spinoff company in the travel industry. (ITA built airline fare search and reservation systems, along the way discovering that air travel planning is NP-hard, before being acquired by Google.) There are other regular attendees who I know by name — David Blank-Edelman, Tom Limoncelli, Lee Damon — who I will say “hello” to, but don’t really socialize with. (These people tend to be the serial organizers and presenters; I’m really only acquainted with them by dint of being on the edges of conversations and occasionally putting in a word.) I don’t think David Parter was there (if he was, I didn’t see him), but there were plenty of other regulars who I recognized but have never known by name.

Now for the conference itself: there were a number of organizational improvements in the conference this year, some of which will hopefully be repeated next year in Washington. There were two new tracks introduced: mini-tutorials, which are 90-minute introductions to material that a regular tutorial takes six hours to cover, and vendor talks, which give vendors (who are also conference sponsors) an opportunity to present their products in a lecture format, rather than the one-on-one selling of the vendor show floor. I have no idea how well either of these worked out as I didn’t go to any of them. There were a few mini-tutorials I was interested in, but they were always scheduled against talks I wanted to see. (This is a problem in general with LISA, for me at least: all too often, the 90-minute blocks of the technical program are either completely uninteresting or have multiple talks I want to see.) At least the Google “vendor BoF” (read: recruiting event with free beer) was scheduled well in advance, for the very last slot on Thursday, after the off-site conference reception, so that anyone who wanted to organize a successful BoF could choose a better night. These “vendor BoFs” suck the life out of the regular BoF track — the Google one especially — and the conference would really be better off without them, but the fees the companies pay help to offset the cost of running the conference, and keeping these events “inside” helps keep them somewhat under control.

Another problem I have to deal with is the decreasing relevance of the LISA program to anything that I do. While it can be interesting to learn what the “industry leaders” are doing, all the cheerleading in the program (“DevOps! Agile! Rah rah rah!”) can leave the impression that if you don’t work for either a gigascale Web shop or an equally enormous *aaS provider, you don’t really belong in this industry. (Until recently, we had exactly one “dev” — and she’s my officemate. Now we have two developers. In order to “deploy thousands of times a day”, you have to have thousands of things to deploy in a day. As I explained to someone, we don’t make software: our product is called a “Ph.D.”, and no amount of “agile” is going to reduce the average delivery time of eight years per diploma.) So when the LISA program is announced in late summer, I always have to take a long hard look to be certain I can justify spending $3000 of our limited travel budget to go there.

Since I went to dinner with Noah fairly late on Tuesday evening, I didn’t get to do some of the BoFs that I was interested in. I walked into the very tail end of the “Why FreeBSD Is For Me” BoF, which was organized by Dru Lavigne of the FreeBSD Foundation, then went to a documentation BoF organized by a guy from Red Hat. I didn’t get much out of either one, and was pretty tired after the end of that, so I went back to my room and crashed (without exercise). On Wednesday, the vendor expo provided free food (not very good and in far too great a quantity), so I was able to attend the early BoF sessions; I went to the monitoring BoF and learned about what people were doing for alerting and trending. Most people are still using separate systems for this, which makes perfect sense to me, but one BoF attendee talked about how he was dumping all of his metrics collection into ElasticSearch and doing alerting by running queries against the ES database. (This was a nice antidote to all the metrics talks in the main conference program, most of which were given by people in the metrics-as-a-service business. See below.)

After the monitoring BoF, I went to the PuppetLabs vendor BoF, and learned a bit about how PuppetLabs was planning on breaking our Puppet infrastructure in the future, as well as some prophylactic measures we could take to prevent it. The PuppetLabs release engineer was incredulous that anyone would actually install the latest version of Puppet automatically, and repeatedly said that it was not supported to run newer Puppet agents against older servers. (In our experience, newer Puppet servers are much more likely to break stuff that we depend upon than newer agents fail to work with old servers — we have that situation now where some client machines are running 3.7 — because they run FreeBSD and that’s the version that’s available — but we have to keep the server back at 3.6. After the BoF, I spent some more time talking with the PuppetLabs guy about this, and he tried to convince me that everything would be wonderful if they built packages with their own, renamed, versions of Ruby and OpenSSL. I said “No thank you”, and he persisted in thinking that this was wonderful. I said “I’d prefer to have timely security updates”, and he didn’t seem to hear me. Dear vendors: THIS IS A STUPID IDEA. Package builders insist on using a single copy of library and interpreter dependencies for a good reason, and however much you think you’re smarter than they are, we the users know better.)

I didn’t see anything interesting in the 9 PM BoF slot, and decided to brave the cold in the fitness center to at least try to get some exercise. My stationary bike at home is an old Vision Fitness unit, with a battery-powered LCD display and a fixed-drag braking system; the hotel fitness center had Life Fitness bikes with built-in (analog) TV tuners and a weird and discomfiting variable-resistance system. I was unable to find my comfort cadence regardless of how I set the resistance, but at least it was easy enough to get the seat height right. My bike has a completely uncalibrated calorie display, and I was really shocked to find out how far off the hotel bike’s results were from my unit at home — a workout that my home bike thought was good for 1000 calories was only 700 on the hotel one. I have reason to believe that the hotel unit’s readings were more accurate, however, and until I have the money to replace my old bike I’ll be basing my workouts on those numbers. I worked out again on Thursday, after the conference reception (again, far too much “free” food, and my phone died while inside the museum so it was impossible even to keep track of what I was eating), and on Friday before stopping in at the “dead dog” party (which is really for the conference organizers but somehow I’ve been allowed to attend since I was an invited speaker in San Diego a decade ago).

Now as for the main technical program: I thought the actual quality of the sessions I attended was pretty variable. Some were excellent, some were mediocre but interesting enough to stay, a few were downright boring. The opening keynote was given by Ken Patchett of Facebook, who talked about data center design (I did that back in 2004, and unless I change employers it’s highly unlikely that I will ever again be involved in such a project); it was moderately interesting, particularly the bits about getting more efficiency by keeping power distribution at 480Y/277 rather than stepping down to 208Y/120 as is normal. (Not sure how well that would work for our Canadian friends, where 600Y/346 is used instead of 480 — I can imagine a lot of employers would not care to have 600-volt power distribution just due to the increased training and insurance expense. But Facebook is pushing this as a standard through something called “Open Compute”, so maybe they’d use 480 in Canada anyway. In Europe they would presumably stick with the normal 400Y/240 that’s already used for three-phase power distribution.) Facebook also does a lot with outside (unconditioned) air, which reduces costs substantially, but only for those DCs that have cool enough air for enough of the year, which not all of them can, given the other infrastructure constraints that affect DC siting.

For the first session on Monday, I attended Dieter Plaetinck’s talk on “Metrics 2.0“, the first of several talks on that theme in this LISA; his presentation was about standardizing the representation of metrics metadata, so that more of the work in presenting metrics to administrators can be automated — a worthwhile goal, at least, given the huge quantity of data being generated and collected today. (We currently use Munin, which bypasses this issue by permitting only fixed presentation defined by the data source. But Munin has lots of limitations and we’d like to replace it with something more flexible.) The second talk in the session was about resource accounting for HPC facilities, and even though our operation is very HPC-like in a lot of ways, this talk was totally irrelevant to me and I left about ten minutes in to join the hallway track.

After lunch (with my former co-workers Eric and Noah at Wild Ginger), I attended the paper session, but found only two papers to be at all interesting. “An Administrator’s Guide to Internet Password Research” was a survey paper summarizing results about what sort of password policies were most effective given rational sorts of threat models. The presenter made the claim that there were only two sorts of attacks worth considering: online attacks, which can be foiled by a fairly simple password quality checks in combination with account lockout controls, and offline attacks, security against which requires much stronger passwords — always assuming the underlying authentication mechanism is actually implemented competently. (Incompetently implemented authentication mechanisms are much easier to break by directly attacking the mechanism rather than by attacking the password, so there is no need to consider them — they are all easier than online guessing.) The presenter agrees with me that the commonly referenced alphabetic entropy model for password quality is at best misleading if not completely erroneous. The second interesting paper presentation, “Realtime High-Speed Network Traffic Monitoring Using ntopng”, was apparently not interesting enough for me to remember what was presented, but you can read the paper, which received the LISA’14 best paper award. I read mail or snoozed through the other paper presentations.

For the second session Tuesday afternoon, I attended talks in the “metrics” track. David Josephsen of Librato, one of those metrics-as-a-service companies, talked about “Feature Flagging at Scale”. He described how they tested and incrementally deployed an architectural change to how Librato collects metrics from Amazon Web Services, saving substantial expense for themselves, without taking substantial risks of business interruption or losing client data. They have an internal chatbot that, among other things, developers use to initiate deployments and reconfigure feature flags. The second talk of that session was Theo Schlossnagle of Circonus — another metrics-as-a-service company — explaining how almost everybody had failed to study statistics and was doing metrics wrong, and giving a few examples of how one might get the statistics right without hiring your own team of statisticians. Of particular note, the non-normality of most of the metrics we collect (e.g., request latency) means that the most common statistical techniques (like mu-sigma) give meaningless results, and the non-stationarity of the distributions that we do so means that most machine-learning techniques don’t work for anomaly detection. (<wavy lines/>The last time LISA was in Seattle, in 1999, I saw the guy from WebTV — remember WebTV? — talking about using Holt-Winters models to do anomaly detection on the metrics they were collecting with Cricket.<wavy lines/>) There was a poster session after this, but I didn’t go. The “free” food in the vendor expo passed for my dinner on Wednesday, but I don’t recall what it was (only that it was better than the lunch offering had been).

Thursday morning’s plenary was a “Rah rah rah! Agile! DevOps! Go team!” talk by Gene Kim, which was totally uninteresting to me, but since I was already awake, I went to it anyway. Kim seems to conflate “high performing IT organizations” with those that employ hundreds if not thousands of developers. I’m sure somebody got something out of this talk, but I found it a complete waste of time. After the morning break, I attended Brian Atkisson’s talk about “Open Source Identity Management in the Enterprise”, which was a bit less relevant than it might have been despite being easily the most informative talk of the whole conference. Unfortunately, Atkisson works for Red Hat, which means that “Open Source Identity Management” means, with a few exceptions, “Red Hat products for which there happen to be related open-source projects so we can claim to be an open-source company”. Most of these products were acquired by Red Hat from Sun, which had acquired them in the dissolution of Netscape a decade ago, and it shows. I played around a bit with “389 Directory Server”, which is indeed open-source (there’s even an Ubuntu package), after I got home, but the documentation for it seems to be locked up behind a Red Hat paywall, so I didn’t bother to pursue it any further. (We would probably do well to put a more serious effort into it, particularly because we could stand to have a better Certificate Authority implementation than my hand-hacked Ruby CA that I’ve been running since 2002, but I really don’t have time to figure it out this week, at any rate.) The one particularly relevant non-Red Hat product that he talked about was (MIT) Kerberos, which we’re obviously already running. There was some more interesting stuff about integration of one-time password technology (required for Sarbanes-Oxley and PCI compliance) with their systems. Interestingly, Red Hat’s engineers hate OTP, and love the SSO experience they get with Kerberos — wish our users were like that. (Our users hate Kerberos and love the only-one-login-ever-until-I-reboot experience they get with ssh-agent forwarding. Doesn’t work with AFS, though, so they can’t have it, no matter how much they would prefer it.)

After lunch on Thursday (taken in the vendor expo, but far better food than Wednesday’s that I skipped), I attended Hal Stern’s talk “You Have Too Much Data”, which I didn’t get much out of, and then in the same session David Josephsen’s “Data Storage at Librato” talk, which at least told me that Librato was one of those companies that had the statistics wrong in the way Theo Schlossnagle was talking about Wednesday afternoon (specifically, using mu-sigma on non-Gaussian data). This was one of the sessions that I could have completely skipped, except that I didn’t have anything else to do that would have been more interesting. For the late afternoon session, I attended Mikey Dickerson’s talk which wasn’t really about “the healthcare.gov meltdown” — more a pitch for the newly established “US Digital Service”, given remotely from a White House teleconference room with no audience interaction. Dickerson was a Googler, who took leave to join the team that “fixed” healthcare.gov, and was then persuaded to lead the USDS, which is an Executive Branch office that helps other federal agencies figure out how to deliver digital services to citizens and other clients. He worked on the Obama campaigns in ’08 and ’12 and will presumably be looking for work in 2017. I probably should have gone instead to the talks “Managing Large Scale Cloud Infrastructure at Rackspace” (by Jesse Keating at Rackspace) and “Building a ‘Multi-Landlord’ Public Cloud” (by Peter Desnoyers at Northeastern) — I’ll have to make do with the videos, once they’re released.

Thursday evening’s social event was the conference reception, held at the Paul AllenEMP Museum. I saw some interesting exhibits, particularly ones showcasing guitars and science-fiction memorabilia in the museum’s collection. I only saw about half of the museum; it would have been nice to have a bit more time there without the conference crowd.

Friday’s plenary is always the conference closer, so there are two sessions on Friday morning. In the early session, I went to “Software is Eating the Network: SDN and Devops” by John Willis, which was not particularly relevant to me (takeaway: “Docker and OpenVSwitch are going to rule the world! Thousands of containers on ever server! You will be assimilated!!!!1!!”), which I say through mainly to see the Lightning Talks, which were scheduled for the second half of the session. I also wanted to attend Ben Rockwood’s “I Am SysAdmin (And So Can You!)” and Tom Limoncelli’s mini-tutorial “Live Upgrades on Running Systems”, both of which were scheduled against the Lightning Talks, and Lee won. At the second morning session, I went back to the metrics track, but Caskey Dickson’s “Gauges, Counters, and Ratios, Oh My!” was not especially interesting — aside from introducing a bit of additional vocabulary, it didn’t offer anything that I hadn’t already heard from the Wednesday metrics talks. I definitely should have gone to one of the other sessions (either Luca Deri’s mini-tutorial on ntopng or Abe Singer’s talk on password policies).

I stuck with the metrics track again for the Friday afternoon session. Eldon Koyle’s talk was mostly live demo and not very interesting (Koyle didn’t write the software, and it doesn’t do IPv6 or indeed pretty much anything that isn’t a single IPv4 /16); Toufic Boubez (another one of those metrics-as-a-service guys!) talked about anomaly detection, and most of what he said was the same as what Theo Schlossnagle said on Wedneaday, aside from adding a few references to useful techniques and textbooks that might help people actually get metrics right. (And thinking about it now, I’m wondering how much I’ve conflated Boubez’s and Schlossnagle’s talks.)

The closing plenary had been originally scheduled to be a NASA person talking about planetary exploration, but that speaker had to cancel at the last minute and was replaced by a Nordstrom VP. Her talk was utterly uninteresting. I spent the time cleaning up my email; if this speaker had been in the advance program rather than an emergency replacement, I would have arranged to fly home on Friday. Instead, I sat through a boring speech and then went to dinner with Steve VanDevender, a Dutch Googler from their Zurich office, and a German fellow whose employment I don’t recall.

So what did I learn? Well, metrics are hard, and particularly uninformative if you get the statistics wrong. There are some LDAP tools from Red Hat that I should really investigate. DevOps cheerleaders have not gotten any less irritating. PuppetLabs is going to keep on breaking shit, Munin and Nagios still suck, and Google treats ops people better than Amazon does. There are still a whole lot of efficiency improvements possible in data center design (but only if you get to build them from scratch at cloud-provider scale), and SDN might be relevant to my job some day. Discount hotels have better radio reception in guest rooms. Metrics-as-a-service is ungodly expensive. mod_auth_mellon might be interesting if we find ourselves needed to implement SAML. “Federated logout” is a thing, and the way SAML specifies it is unimaginably stupid.

Oh, and despite exercising four nights out of five, I still gained five pounds. I blame the free food.