What I saw (and did) at (and around) LISA’14

I recently got back from my one-per-year business trip, to attend the 2014 LISA Conference, and now that I’ve presented to my group, it’s time to make a more public write-up. I’ll start with some of the more personal or social things I did, and then dig into the sessions I attended and what if anything I learned.

This year I’ve been flying JetBlue a lot, now that they have reasonable schedules to the West Coast (and by “reasonable” I really mean “the return flight isn’t a red-eye”); their regular seat pitch in coach is wider than premium seating from either UA or DL, and when you’re checking a bag, the old-line network carriers’ bag fees really add up. JetBlue has non-stops from Logan to both San Diego (where my parents live) and Seattle, not to mention LA (where I usually end up spending at least a few days every other April or thereabouts) and the Bay Area. There is, however, one drawback: the B6 winter schedule has no transcontinental flights returning to Boston on Saturday. So I could either return on a Friday red-eye, or stay an extra day on my own dime, or fly another airline (Alaska is the biggest carrier at SEA and also had non-stops to Boston, code-shared with AA).

I chose to stay the extra day, and used the Saturday after the conference to do touristy things — I went to Pike Place Market for lunch, and then took the #40 bus out to Fremont to see the Theo chocolate factory. I was lucky in that, although the factory tour was fully booked for Saturday, there was a group of eight no-shows, so they let me join the tour five minutes in, after paying my $10. The tour was pretty meh. There really isn’t a whole lot to see in a chocolate factory, particularly on a Saturday when there’s nothing in production. The tour did take us into the kitchen, where the non-chocolate confectionery is made (caramel centers, ganache for truffles, and fruit and nut flavors that are mixed into the bars); it’s also where their enrobing machine is located — but again, with no production happening, you can’t really see anything. (And even when the factory is operating, you still can’t see much, since much of the machinery is closed up for food safety — there’s no giant Willy Wonka-style fountain of chocolate.) I did learn a few things: their organic sugar, which comes from Brazil, is distributed by a company from Arlington, Mass.; the bars with mix-ins were historically blended by hand, but now (with a recent factory expansion) have moved to a more automated production line; and they make a large number of SKUs that are not nationally distributed — some of them are sold only in the factory store. (It doesn’t hurt that Fremont is a trendy Seattle neighborhood, with plenty of residential and office development nearby and very frequent bus service, so the factory store is rather more accessible than the typical zone industrielle where food factories would be located.) I ended up spending just shy of $200 at the factory store, buying nearly everything (except coffee) that wasn’t available at my local Whole Foods. (I bought a few things that Whole Foods doesn’t carry but, as it turned out, I could have gotten from my local REI instead!)

At Pike Place Market I had a salmon club sandwich for lunch, and also bought several bags of chocolate-covered cherries from Chukar Cherries. A nut vendor sold hazelnuts (their own) and smoked almonds (from a California maker). I didn’t waste time watching the fishmongers throw fish around, but I also probably only saw a quarter of what was on offer there. (There were a number of things I might have bought except that I had no safe way of getting them home in one piece — and I don’t care to find out what the airline, even JetBlue, would do to a jar of jam or honey in my checked baggage!)

I have an unfortunate habit of forgetting one really important thing whenever I travel. I very nearly forgot my cell phone, which would have been a disaster — thankfully, the Logan Express bus ticket agent has an EMV credit-card terminal, and as I was paying with my corporate travel card, I didn’t have the PIN memorized and had to look it up. Groping for my phone, I realized that I had left it in the car, and I dashed off to grab it at the last minute. (As it turned out, I was in so much of a hurry that I forgot to re-lock my car after grabbing the phone — good thing nobody tried to break into it while it was parked in the lot.) Having successfully remembered my phone, what it actually turned out I had forgotten was any clothing warm enough for the frigid weather which gripped Seattle — and the rest of the nation — during the six days I was there. Normally at this time of year, daytime highs in Seattle are around 50° — but this year, this week, it struggled to get much above 40°, and to compound the insult, it was windy as hell. I brought only lightweight jackets and my usual short-sleeve knit shirts, plus a couple of shirts from previous LISA conferences. The LISA’11 shirt was the only long-sleeved shirt I had, and it was grossly inadequate to the task. (The LISA’14 conference shirt, which was short-sleeved, continued the trend of Usenix buying very light-weight fabrics that are suitable only as underwear or in very warm weather. The LISA’99 shirt was made of heavier fabric, but was, like most of the others, a T-shirt.) The conference hotel, as is the way of these things, was frigid, although not quite as cold as outside.

It was at LISA’13 in Washington that I made a fairly important lifestyle decision, after talking with LISA and Usenix regular David Parter (of UW-Madison). Parter had lost a good deal of weight, and he credited a mobile app that he was using. At the time, I was having to take medication twice a day for glucose tolerance and high blood pressure, and I was constantly paying to repair small tears around the seams in my shirts. I had been exercising on-and-off on a stationary bike with no appreciable results, and it sounded like the app that Parter was using might help me implement enough calorie restriction to actually make a difference.

It turned out that it did, and that as I lost some weight, it also became easier to exercise, and I was able to incrementally lengthen my workout from barely 25 minutes total to over an hour over the course of the following six months. (The Winter Olympics in February greatly aided this, but it’s really baseball that makes the biggest difference — with games nearly every night, I’m not at the mercy of 22- and 48-minute TV shows for the visual distraction that keeps me on the bike late at night when I’d rather be relaxing. This fall I subscribed to NHL Center Ice so that I could watch hockey during the baseball off-season.) I knew that I wanted (and in practical terms needed, in the presence of all the free food a conference entails) to keep up my exercise regimen as best I could, and since I was staying in a big-city business hotel, I knew both that there would be a fitness center I could use, and that I needed to buy something to wear while doing so — hotels not being so keen on guests walking through the public areas in their underwear! Luckily, when I stopped by my local REI, they had a bunch of cycle clothing on clearance, and I bought a couple of shirts and a couple pairs of shorts (one pair of double shorts that would be OK to wear walking around — if you don’t mind the big chamois butt-pad — and one pair of “lycra bandit”-type shorts). I made sure to try them on before leaving, and made a shocking discovery: I can actually wear a size medium — at least if it’s made of Spandex. A year ago, I would never have believed I could ever again wear anything smaller than an XL. But I’ve gone down seven inches in waistline, and for some styles and manufacturers, that puts me in a medium, at least for bottoms. I went back to REI the next day to buy more cycle wear (including a pair of “M” tights), which felt ridiculously good despite the expense.

(I made a similar discovery in the market for new underwear a few months ago: a 33 waist might be a small, medium, or large depending on which maker’s size chart you go by. I’ve been gradually replacing the old XL underwear, but I’m never going to have the build of an underwear model, so I don’t think I’ll ever be wearing size “small” smalls no matter how much more weight I manage to lose.)

I arrived in Seattle around midday on Tuesday, which was an unusual bit of lucky scheduling for me — far more often I arrive at the conference hotel well after the badge pickup desk has closed, requiring me to stand in line early the next morning. When I checked in, the hotel clerk asked me if I wanted to waive housekeeping for the duration of my stay, in exchange for a $5 voucher each day, which certainly takes the “feel good and let us not wash the towels every day” a step further. (I declined, but apparently he didn’t key it correctly, so the next morning a voucher was shoved under my door and sure enough, no housekeeping. I didn’t care enough to complain, though, until the toilet paper ran out!) I watched the local TV news, where they were talking about roads being blocked and houses being damaged by falling trees, all a result of the uncharacteristically high winds sweeping the region. I set up my airchecking station near the window, and was greatly puzzled by the extremely poor reception — there must have been some sort of conductive coating on the windows, because my ninth-floor hotel room, facing north, should have been able to get far better signals (and usable HD) from the West Tiger Peak and Cougar Mountain transmitter sites; as it was, the only stations I had a good signal from were those with in-town transmitter sites (a couple of translators and a few stations on Capitol Hill and Queen Anne Hill). Then I went to find the fitness center, and found my resolve to exercise every night wavering a bit, despite the consequences: I was in the “Union” tower of the hotel, and the fitness center was at the very top of the “Pike” tower — I would have to go all the way down to the conference floor, or worse still, the lobby, in my sweaty workout clothes. And when I got to the fitness center, it was absolutely frigid. (This turned out to be a good thing, but I was just shivering there, in my street clothes at resting metabolism.)

After all that, I got in touch with my former co-worker (and recent-ex-Amazonian) Noah, and we went to a seafood restaurant a few blocks from the hotel, which coincidentally was also just a few blocks from the office building where Noah used to work. We got caught up, having not seen each other for a few years, and he gave me a bit of an insider’s view of the Amazon sausage factory. (This week he started on his new job working for a software company elsewhere in Seattle — the sort of place that didn’t really exist 15 years ago when I first visited the area. Back in 1999, there was Microsoft, and Boeing, and their suppliers, Starbucks, and UW, and that was pretty much it. The whole region is much more economically diverse now than it was then, although it’s still fairly concentrated, industry-wise.) Noah attended the conference, and we met again a few more times — on Wednesday, after seeing how awful the free lunch looked, we met up with another former co-worker, Eric, and had lunch at an Asian restaurant a few blocks from the hotel. (Eric lives on Vashon Island, and I had originally planned to go out there on Saturday to take a tour of the many radio towers that call Vashon home, but in view of the cold weather, long travel time, and limited daylight, I decided to put this off until some other time.)

I met up with a number of the “usual suspects” after the conference began. I often spend time at LISA with Steve VanDevender, who works at the University of Oregon in Eugene; Steve also teaches a summer undergraduate course at UofO in system administration. There’s also a guy whose name I can never remember who works at the University of Alberta in Edmonton — one of the few people for whom the weather in Seattle was actually warmer than what they had left. There were as always a lot of Googlers, but I only really know one, Chris Davis, who joined Google with their acquisition of ITA Software, which was an MIT-spinoff company in the travel industry. (ITA built airline fare search and reservation systems, along the way discovering that air travel planning is NP-hard, before being acquired by Google.) There are other regular attendees who I know by name — David Blank-Edelman, Tom Limoncelli, Lee Damon — who I will say “hello” to, but don’t really socialize with. (These people tend to be the serial organizers and presenters; I’m really only acquainted with them by dint of being on the edges of conversations and occasionally putting in a word.) I don’t think David Parter was there (if he was, I didn’t see him), but there were plenty of other regulars who I recognized but have never known by name.

Now for the conference itself: there were a number of organizational improvements in the conference this year, some of which will hopefully be repeated next year in Washington. There were two new tracks introduced: mini-tutorials, which are 90-minute introductions to material that a regular tutorial takes six hours to cover, and vendor talks, which give vendors (who are also conference sponsors) an opportunity to present their products in a lecture format, rather than the one-on-one selling of the vendor show floor. I have no idea how well either of these worked out as I didn’t go to any of them. There were a few mini-tutorials I was interested in, but they were always scheduled against talks I wanted to see. (This is a problem in general with LISA, for me at least: all too often, the 90-minute blocks of the technical program are either completely uninteresting or have multiple talks I want to see.) At least the Google “vendor BoF” (read: recruiting event with free beer) was scheduled well in advance, for the very last slot on Thursday, after the off-site conference reception, so that anyone who wanted to organize a successful BoF could choose a better night. These “vendor BoFs” suck the life out of the regular BoF track — the Google one especially — and the conference would really be better off without them, but the fees the companies pay help to offset the cost of running the conference, and keeping these events “inside” helps keep them somewhat under control.

Another problem I have to deal with is the decreasing relevance of the LISA program to anything that I do. While it can be interesting to learn what the “industry leaders” are doing, all the cheerleading in the program (“DevOps! Agile! Rah rah rah!”) can leave the impression that if you don’t work for either a gigascale Web shop or an equally enormous *aaS provider, you don’t really belong in this industry. (Until recently, we had exactly one “dev” — and she’s my officemate. Now we have two developers. In order to “deploy thousands of times a day”, you have to have thousands of things to deploy in a day. As I explained to someone, we don’t make software: our product is called a “Ph.D.”, and no amount of “agile” is going to reduce the average delivery time of eight years per diploma.) So when the LISA program is announced in late summer, I always have to take a long hard look to be certain I can justify spending $3000 of our limited travel budget to go there.

Since I went to dinner with Noah fairly late on Tuesday evening, I didn’t get to do some of the BoFs that I was interested in. I walked into the very tail end of the “Why FreeBSD Is For Me” BoF, which was organized by Dru Lavigne of the FreeBSD Foundation, then went to a documentation BoF organized by a guy from Red Hat. I didn’t get much out of either one, and was pretty tired after the end of that, so I went back to my room and crashed (without exercise). On Wednesday, the vendor expo provided free food (not very good and in far too great a quantity), so I was able to attend the early BoF sessions; I went to the monitoring BoF and learned about what people were doing for alerting and trending. Most people are still using separate systems for this, which makes perfect sense to me, but one BoF attendee talked about how he was dumping all of his metrics collection into ElasticSearch and doing alerting by running queries against the ES database. (This was a nice antidote to all the metrics talks in the main conference program, most of which were given by people in the metrics-as-a-service business. See below.)

After the monitoring BoF, I went to the PuppetLabs vendor BoF, and learned a bit about how PuppetLabs was planning on breaking our Puppet infrastructure in the future, as well as some prophylactic measures we could take to prevent it. The PuppetLabs release engineer was incredulous that anyone would actually install the latest version of Puppet automatically, and repeatedly said that it was not supported to run newer Puppet agents against older servers. (In our experience, newer Puppet servers are much more likely to break stuff that we depend upon than newer agents fail to work with old servers — we have that situation now where some client machines are running 3.7 — because they run FreeBSD and that’s the version that’s available — but we have to keep the server back at 3.6. After the BoF, I spent some more time talking with the PuppetLabs guy about this, and he tried to convince me that everything would be wonderful if they built packages with their own, renamed, versions of Ruby and OpenSSL. I said “No thank you”, and he persisted in thinking that this was wonderful. I said “I’d prefer to have timely security updates”, and he didn’t seem to hear me. Dear vendors: THIS IS A STUPID IDEA. Package builders insist on using a single copy of library and interpreter dependencies for a good reason, and however much you think you’re smarter than they are, we the users know better.)

I didn’t see anything interesting in the 9 PM BoF slot, and decided to brave the cold in the fitness center to at least try to get some exercise. My stationary bike at home is an old Vision Fitness unit, with a battery-powered LCD display and a fixed-drag braking system; the hotel fitness center had Life Fitness bikes with built-in (analog) TV tuners and a weird and discomfiting variable-resistance system. I was unable to find my comfort cadence regardless of how I set the resistance, but at least it was easy enough to get the seat height right. My bike has a completely uncalibrated calorie display, and I was really shocked to find out how far off the hotel bike’s results were from my unit at home — a workout that my home bike thought was good for 1000 calories was only 700 on the hotel one. I have reason to believe that the hotel unit’s readings were more accurate, however, and until I have the money to replace my old bike I’ll be basing my workouts on those numbers. I worked out again on Thursday, after the conference reception (again, far too much “free” food, and my phone died while inside the museum so it was impossible even to keep track of what I was eating), and on Friday before stopping in at the “dead dog” party (which is really for the conference organizers but somehow I’ve been allowed to attend since I was an invited speaker in San Diego a decade ago).

Now as for the main technical program: I thought the actual quality of the sessions I attended was pretty variable. Some were excellent, some were mediocre but interesting enough to stay, a few were downright boring. The opening keynote was given by Ken Patchett of Facebook, who talked about data center design (I did that back in 2004, and unless I change employers it’s highly unlikely that I will ever again be involved in such a project); it was moderately interesting, particularly the bits about getting more efficiency by keeping power distribution at 480Y/277 rather than stepping down to 208Y/120 as is normal. (Not sure how well that would work for our Canadian friends, where 600Y/346 is used instead of 480 — I can imagine a lot of employers would not care to have 600-volt power distribution just due to the increased training and insurance expense. But Facebook is pushing this as a standard through something called “Open Compute”, so maybe they’d use 480 in Canada anyway. In Europe they would presumably stick with the normal 400Y/240 that’s already used for three-phase power distribution.) Facebook also does a lot with outside (unconditioned) air, which reduces costs substantially, but only for those DCs that have cool enough air for enough of the year, which not all of them can, given the other infrastructure constraints that affect DC siting.

For the first session on Monday, I attended Dieter Plaetinck’s talk on “Metrics 2.0“, the first of several talks on that theme in this LISA; his presentation was about standardizing the representation of metrics metadata, so that more of the work in presenting metrics to administrators can be automated — a worthwhile goal, at least, given the huge quantity of data being generated and collected today. (We currently use Munin, which bypasses this issue by permitting only fixed presentation defined by the data source. But Munin has lots of limitations and we’d like to replace it with something more flexible.) The second talk in the session was about resource accounting for HPC facilities, and even though our operation is very HPC-like in a lot of ways, this talk was totally irrelevant to me and I left about ten minutes in to join the hallway track.

After lunch (with my former co-workers Eric and Noah at Wild Ginger), I attended the paper session, but found only two papers to be at all interesting. “An Administrator’s Guide to Internet Password Research” was a survey paper summarizing results about what sort of password policies were most effective given rational sorts of threat models. The presenter made the claim that there were only two sorts of attacks worth considering: online attacks, which can be foiled by a fairly simple password quality checks in combination with account lockout controls, and offline attacks, security against which requires much stronger passwords — always assuming the underlying authentication mechanism is actually implemented competently. (Incompetently implemented authentication mechanisms are much easier to break by directly attacking the mechanism rather than by attacking the password, so there is no need to consider them — they are all easier than online guessing.) The presenter agrees with me that the commonly referenced alphabetic entropy model for password quality is at best misleading if not completely erroneous. The second interesting paper presentation, “Realtime High-Speed Network Traffic Monitoring Using ntopng”, was apparently not interesting enough for me to remember what was presented, but you can read the paper, which received the LISA’14 best paper award. I read mail or snoozed through the other paper presentations.

For the second session Tuesday afternoon, I attended talks in the “metrics” track. David Josephsen of Librato, one of those metrics-as-a-service companies, talked about “Feature Flagging at Scale”. He described how they tested and incrementally deployed an architectural change to how Librato collects metrics from Amazon Web Services, saving substantial expense for themselves, without taking substantial risks of business interruption or losing client data. They have an internal chatbot that, among other things, developers use to initiate deployments and reconfigure feature flags. The second talk of that session was Theo Schlossnagle of Circonus — another metrics-as-a-service company — explaining how almost everybody had failed to study statistics and was doing metrics wrong, and giving a few examples of how one might get the statistics right without hiring your own team of statisticians. Of particular note, the non-normality of most of the metrics we collect (e.g., request latency) means that the most common statistical techniques (like mu-sigma) give meaningless results, and the non-stationarity of the distributions that we do so means that most machine-learning techniques don’t work for anomaly detection. (<wavy lines/>The last time LISA was in Seattle, in 1999, I saw the guy from WebTV — remember WebTV? — talking about using Holt-Winters models to do anomaly detection on the metrics they were collecting with Cricket.<wavy lines/>) There was a poster session after this, but I didn’t go. The “free” food in the vendor expo passed for my dinner on Wednesday, but I don’t recall what it was (only that it was better than the lunch offering had been).

Thursday morning’s plenary was a “Rah rah rah! Agile! DevOps! Go team!” talk by Gene Kim, which was totally uninteresting to me, but since I was already awake, I went to it anyway. Kim seems to conflate “high performing IT organizations” with those that employ hundreds if not thousands of developers. I’m sure somebody got something out of this talk, but I found it a complete waste of time. After the morning break, I attended Brian Atkisson’s talk about “Open Source Identity Management in the Enterprise”, which was a bit less relevant than it might have been despite being easily the most informative talk of the whole conference. Unfortunately, Atkisson works for Red Hat, which means that “Open Source Identity Management” means, with a few exceptions, “Red Hat products for which there happen to be related open-source projects so we can claim to be an open-source company”. Most of these products were acquired by Red Hat from Sun, which had acquired them in the dissolution of Netscape a decade ago, and it shows. I played around a bit with “389 Directory Server”, which is indeed open-source (there’s even an Ubuntu package), after I got home, but the documentation for it seems to be locked up behind a Red Hat paywall, so I didn’t bother to pursue it any further. (We would probably do well to put a more serious effort into it, particularly because we could stand to have a better Certificate Authority implementation than my hand-hacked Ruby CA that I’ve been running since 2002, but I really don’t have time to figure it out this week, at any rate.) The one particularly relevant non-Red Hat product that he talked about was (MIT) Kerberos, which we’re obviously already running. There was some more interesting stuff about integration of one-time password technology (required for Sarbanes-Oxley and PCI compliance) with their systems. Interestingly, Red Hat’s engineers hate OTP, and love the SSO experience they get with Kerberos — wish our users were like that. (Our users hate Kerberos and love the only-one-login-ever-until-I-reboot experience they get with ssh-agent forwarding. Doesn’t work with AFS, though, so they can’t have it, no matter how much they would prefer it.)

After lunch on Thursday (taken in the vendor expo, but far better food than Wednesday’s that I skipped), I attended Hal Stern’s talk “You Have Too Much Data”, which I didn’t get much out of, and then in the same session David Josephsen’s “Data Storage at Librato” talk, which at least told me that Librato was one of those companies that had the statistics wrong in the way Theo Schlossnagle was talking about Wednesday afternoon (specifically, using mu-sigma on non-Gaussian data). This was one of the sessions that I could have completely skipped, except that I didn’t have anything else to do that would have been more interesting. For the late afternoon session, I attended Mikey Dickerson’s talk which wasn’t really about “the healthcare.gov meltdown” — more a pitch for the newly established “US Digital Service”, given remotely from a White House teleconference room with no audience interaction. Dickerson was a Googler, who took leave to join the team that “fixed” healthcare.gov, and was then persuaded to lead the USDS, which is an Executive Branch office that helps other federal agencies figure out how to deliver digital services to citizens and other clients. He worked on the Obama campaigns in ’08 and ’12 and will presumably be looking for work in 2017. I probably should have gone instead to the talks “Managing Large Scale Cloud Infrastructure at Rackspace” (by Jesse Keating at Rackspace) and “Building a ‘Multi-Landlord’ Public Cloud” (by Peter Desnoyers at Northeastern) — I’ll have to make do with the videos, once they’re released.

Thursday evening’s social event was the conference reception, held at the Paul AllenEMP Museum. I saw some interesting exhibits, particularly ones showcasing guitars and science-fiction memorabilia in the museum’s collection. I only saw about half of the museum; it would have been nice to have a bit more time there without the conference crowd.

Friday’s plenary is always the conference closer, so there are two sessions on Friday morning. In the early session, I went to “Software is Eating the Network: SDN and Devops” by John Willis, which was not particularly relevant to me (takeaway: “Docker and OpenVSwitch are going to rule the world! Thousands of containers on ever server! You will be assimilated!!!!1!!”), which I say through mainly to see the Lightning Talks, which were scheduled for the second half of the session. I also wanted to attend Ben Rockwood’s “I Am SysAdmin (And So Can You!)” and Tom Limoncelli’s mini-tutorial “Live Upgrades on Running Systems”, both of which were scheduled against the Lightning Talks, and Lee won. At the second morning session, I went back to the metrics track, but Caskey Dickson’s “Gauges, Counters, and Ratios, Oh My!” was not especially interesting — aside from introducing a bit of additional vocabulary, it didn’t offer anything that I hadn’t already heard from the Wednesday metrics talks. I definitely should have gone to one of the other sessions (either Luca Deri’s mini-tutorial on ntopng or Abe Singer’s talk on password policies).

I stuck with the metrics track again for the Friday afternoon session. Eldon Koyle’s talk was mostly live demo and not very interesting (Koyle didn’t write the software, and it doesn’t do IPv6 or indeed pretty much anything that isn’t a single IPv4 /16); Toufic Boubez (another one of those metrics-as-a-service guys!) talked about anomaly detection, and most of what he said was the same as what Theo Schlossnagle said on Wedneaday, aside from adding a few references to useful techniques and textbooks that might help people actually get metrics right. (And thinking about it now, I’m wondering how much I’ve conflated Boubez’s and Schlossnagle’s talks.)

The closing plenary had been originally scheduled to be a NASA person talking about planetary exploration, but that speaker had to cancel at the last minute and was replaced by a Nordstrom VP. Her talk was utterly uninteresting. I spent the time cleaning up my email; if this speaker had been in the advance program rather than an emergency replacement, I would have arranged to fly home on Friday. Instead, I sat through a boring speech and then went to dinner with Steve VanDevender, a Dutch Googler from their Zurich office, and a German fellow whose employment I don’t recall.

So what did I learn? Well, metrics are hard, and particularly uninformative if you get the statistics wrong. There are some LDAP tools from Red Hat that I should really investigate. DevOps cheerleaders have not gotten any less irritating. PuppetLabs is going to keep on breaking shit, Munin and Nagios still suck, and Google treats ops people better than Amazon does. There are still a whole lot of efficiency improvements possible in data center design (but only if you get to build them from scratch at cloud-provider scale), and SDN might be relevant to my job some day. Discount hotels have better radio reception in guest rooms. Metrics-as-a-service is ungodly expensive. mod_auth_mellon might be interesting if we find ourselves needed to implement SAML. “Federated logout” is a thing, and the way SAML specifies it is unimaginably stupid.

Oh, and despite exercising four nights out of five, I still gained five pounds. I blame the free food.

Posted in Computing | Tagged , , , , , , , | Leave a comment

Other people’s recipes: Four & Twenty Blackbirds’ black walnut pie

Almost a week late, here’s a quick rundown on the delicious black walnut pie I made six days ago. The recipe comes from Emily and Melissa Elsen, The Four & Twenty Blackbirds Pie Book, like most of the pies I’ve baked recently, and it’s the last of their “Fall” seasonal pies on my list. It starts, of course, with toasted black walnuts. Black walnuts are the native American walnut, but because they are more convoluted than English (aka Persian) walnuts, they are more difficult to shell, which makes them more expensive; they are only seen as smallish broken pieces, never “whole”, unless you happen to have a black walnut tree in your yard. (And you probably don’t want to: walnuts whether black or English are drupes, and the ripe fruit — which must be stripped off to get to the nut — contains a very powerful dark brown pigment that stains anything it touches.)

Black walnuts have a delicate, fruity fragrance which makes them far more interesting in flavor, however, to make up for the increased scarcity. I bought a pound of them by mail order from Hammons Products, one of the principal processors and distributors of black walnuts, located in Missouri near the most of the orchards that grow black walnuts as a cash crop. (In addition to retail and wholesale black walnuts, Hammons also sells black-walnut confections.)

The Elsens recommend their “cornmeal” pie crust rather than the “all-butter” crust used for most of the other recipes I’ve tried so far. After partial blind baking it looks like this:
Patrially prebaked cornmeal crust

I honestly could not tell the difference between this shell and the regular (all wheat flour) recipe. The recipe continues with an “apple cider caramel”, which looked to me nothing like any caramel I’ve ever made. (Not for the first time with this cookbook, either — one of these days I’ll have to do a side-by-side comparison of their recipe and a normal caramel to see which one actually turns out better. My experience with the Elsens’ “caramel” recipes is that the foaming of the butter, which they add at the beginning, makes it very difficult to actually tell when the sugar is at the right stage, and I don’t think the temperatures they suggest are anywhere near correct for proper caramel. A normal caramel procedure would be to brown the sugar, then quench with liquid, then stir in butter off the heat — see, for example, the Cook’s Illustrated chocolate caramel walnut tart, which I’ll be making again this year with leftover black walnuts.) In any event, the sugar mixture is cooled and used to make an egg custard, with the addition of some cornmeal as a thickener, and, of course, the toasted black walnuts. The whole process is very similar to a pecan pie, or to the black bottom oatmeal pie I did a few weeks back, but with a flavorful cooked-sugar syrup rather than corn syrup.

I found that my pie was way underdone after the recommended 30 minutes of cooking time, and ended up giving it another ten minutes. When I finally took it out, it looked like this:
Finished pie, hot from the oven

The Elsens do caution, however, “Be careful not to overbake, or the custard can separate”, and I suspect that’s what happened here:
Slice of black walnut pie
If you look closely, you can see that the filling has in fact separated into a yellow, eggy custard layer on the bottom and a sweet, syrupy layer between that and the nutty bits on top. Nobody at the office seemed to notice, but I did, which tells me that my pie-doneness-determining skills still need some work. Here’s what the rest of the pie looks like:
Pie minus one slice

Nutrition

I figured this pie based on eight slices, which makes it look a bit worse than other pies I’ve done recently, but is perhaps more realistic. (I always have trouble trying to get ten slices out of a pie — maybe I should buy one of those professional pie slicing devices they use in bakeries!)

Nutrition Facts
Serving size: 1/8 pie
Servings per container: 8
Amount per serving
Calories 471 Calories from fat 261
% Daily Value
Total Fat 29g 45%
 Saturated Fat 14g 69%
 Monounsaturated Fat 0g
Trans Fat 0g
Cholesterol 122mg 41%
Sodium 178mg 7%
Potassium 55mg 2%
Total Carbohydrate 47g 16%
 Dietary fiber 2g 8%
 Sugars 28g
Proteins 9g 18%
Vitamin A 16%
Vitamin C 1%
Calcium 2%
Iron 7%

Here is the data for the cornmeal pie crust alone:

Nutrition Facts
Serving size: 1/8 pie
Servings per container: 8
Amount per serving
Calories 178 Calories from fat 100
% Daily Value
Total Fat 18g 17%
 Saturated Fat 7g 35%
Trans Fat 0g
Cholesterol 30mg 10%
Sodium 74mg 3%
Potassium 11mg 0%
Total Carbohydrate 16g 5%
 Dietary fiber 1g 3%
 Sugars 0g
Proteins 3g 5%
Vitamin A 8%
Vitamin C 0%
Calcium 0%
Iron 2%
Posted in Food | Tagged , , ,

Limited posting schedule for the rest of this month

I’ll be making two trips to the west coast later this month, and that will put a crimp on blog posts for a while (especially the food posts). There will be one more post from last weekend’s cooking, which should come tomorrow or Thursday depending on when I get off my butt and write it. I have a couple of other posts percolating: a big idea post about the two most important lenses through which we see the world (and how we don’t do a very good job at teaching them), and a much smaller post reflecting on a year of daily commuting in a hybrid — not sure if either of these will get done before the month is out.

CSAIL Holiday Baking Fest will be the Monday after I get back from California, so there will be some more recipe posts for the treats I end up baking for that (currently planning on doing three: two chocolate tarts and a Polish tea bread).

Aside | Posted on by

Other people’s recipes: Cook’s Country’s Five-Alarm Chili

This gallery contains 10 photos.

It’s been a busy cooking weekend for me: in addition to yesterday’s truffles, I also did a black-walnut pie from Four & Twenty Blackbirds’ cookbook (post to come later this week), and I finished the weekend off with a huge … Continue reading

Gallery | Tagged , , , , ,

Other people’s recipes: Alice Medrich’s Extra-Bittersweet Ganache Truffles

This gallery contains 6 photos.

This week’s first recipe was born out of a dilemma: last week’s recipe called for a quarter cup of heavy cream. I don’t have anything useful to do with heavy cream, but it’s sold in pint bottles, so I had … Continue reading

Gallery | Tagged , , , ,

Other people’s recipes: Four & Twenty Blackbirds’ Black Bottom Oatmeal Pie

This gallery contains 7 photos.

I was nearly ready to write this pie off as a complete failure, for reasons I’ll detail below, but I put it in the refrigerator overnight and brought it into work, and my co-workers loved it — they practically gobbled … Continue reading

Gallery | Tagged , , , , | 2 Comments

Recipe quick take: Kenji Lopez-Alt’s Easy pan-fried fish

I stumbled across this great post on SeriousEats by culinary director Kenji Lopez-Alt. There’s nothing especially new here, but it’s nice to have this step-by-step tutorial for doing this. I didn’t worry about the exact ingredients; the procedure is what’s important. I did it with cod (readily available here in New England, although I wonder if the stocks have really recovered enough for it to be as plentiful as it appears). I added a couple teaspoons of minced fresh tarragon to the egg mixture — this allowed me to add some herbal flavor while keeping the quick-to-burn ingredients still relatively protected. It all came out exactly as Kenji suggested: perfectly cooked fish, golden-brown coating, and no mess or lingering odor to clean up. I used a ten-inch non-non-stick stainless-steel skillet, but for my pound of cod a slightly larger size would have been a bit better. No matter: it still came out great.

Proportions I used:

  • 1 lb cod fillet
  • ½ cup all-purpose flour
  • 1 tsp kosher salt
  • fresh ground black pepper (not measured)
  • 2 medium eggs (how did I end up with these?)
  • about 2 tsp minced fresh tarragon
  • 1½ tbl canola oil
  • 1 cup panko

There was plenty of all of the breading stages left over to do at least another half-pound of fish, if not a whole pound.

Posted in Food | Tagged , , , ,

Pumpkin Pie Fest part III: the results

This gallery contains 9 photos.

This was the scene as a hundred or more hungry graduate students descended on our group’s lunch area to take part in CSAIL Pumpkin Pie Fest: We gave out more than 200 portions of pumpkin pie, and according to some … Continue reading

Gallery | Tagged , ,

Pumpkin Pie Fest part II: the pies

Eight recipes, seven cookbooks, six piesOn Sunday, I baked five of the six pies. I ran into two issues that prevented me from baking pie #6 on Sunday: I ran out of both pumpkin and eggs. (This was perhaps because my shopping list was based on making four rather than six pies.) So today — Monday — I delivered the first five pies to the refrigerator at work (throwing away some clearly spoiled vegetables in the process of making room), then went grocery shopping so I could finish pie #6, Pecan Pumpkin Pie, about which more below.

As I mentioned in part I, the pies varied quite a bit in their construction. Some called for blind-baking the pie crust, either fully (as with Joanne Chang’s recipe) or partially (the Elsens’ and Richard Sax’s recipes). The butter content of these pie doughs made the whole process a bit of a challenge, and those crusts that were not blind-baked definitely did not perform as well as those that were (see in particular the Moosewood recipe below). Four of the recipes I initially identified as having potential got left out: Alice Waters’ recipe was very plain, so I figured that someone else would probably be bringing something very similar; Richard Sax’s pumpkin chiffon didn’t interest me very much, but I gave a photocopy to Dorothy Curtis; Alton Brown’s recipe was one that I had done before, and with three cheesecakes contributed by others, I did not think there was a need for another crumb crust; and I didn’t like the idea of using yams, called for in the Cook’s Illustrated recipe from 2008. (Jason Miller made that recipe, however, so tasters will get a chance to try it anyway.)

Starting out, I tried to overlap some of the cooking steps, so that one pie could be in the oven alongside the crust being blind-baked for the next pie, but after the first two pies I found that this was too much to juggle, particularly as the recipes called for different oven temperatures (and in some cases, changing oven temperatures). The first pie to be completed was Honey Pumpkin Pie by David Page and Barbara Shinn, from their cookbook Recipes from Home:
David Page and Barbara Shinn's Honey Pumpkin Pie
Although this pie is called “Honey Pumpkin Pie”, only a third of the sweetener is honey; I used McLure’s orange blossom honey. (The “Golden Pumpkin Pie”, below, from King Arthur Flour Whole Grain Baking is sweetened entirely with honey.) As I mentioned in part I, I did not use their pie crust, but instead used half of a double-crust recipe from Richard Sax’s Classic Home Desserts which Sax attributes to Nick Malgieri.

The second pie to be finished was Joanne Chang’s Super-Pumpkiny Pumpkin Pie, from her cookbook Flour:
Joanne Chang's Super-Pumpkiny Pumpkin Pie
Chang’s recipe is the only one I made that calls for the traditional combination of evaporated milk and sweetened condensed milk (I used Carnation for the former and Eagle Brand for the latter) in addition to heavy cream. Additional sweetness comes from brown sugar, which is cooked with the pumpkin for a fairly long time to concentrate its flavor. Chang’s crust is enriched with egg, and unlike all the others is fully blind-baked. You can see the difference in this photo, which shows the first two pies side by side:

Two pies side-by-side

Honey Pumpkin Pie (l) and Super-Pumpkiny Pumpkin Pie (r)

The third pie to be completed was Brown Butter Pumpkin Pie, from Emily and Melissa Elsen’s Four & Twenty Blackbirds Pie Book:
Emily & Melissa Elsen's Brown Butter Pumpkin Pie
As you can see from the crack in the center (which grew even more in cooling), I somewhat overcooked the custard. The brown butter in the title forms part of a butterscotch, which is the only sweetener for this pie, which has the most unusual ingredients list of the six pies, including both lemon and carrot juice.

Pie #4 was King Arthur’s Golden Pumpkin Pie. As noted above, the custard is sweetened entirely with honey (nine ounces of non-varietal honey from a local apiary); the crust is made with ground rolled oats and traditional (red) whole wheat flour, and it’s sweetened with brown sugar (I used India Tree dark muscovado):
King Arthur Flour's Golden Pumpkin Pie
Beyond the honey, the other unusual ingredients in this custard are dark rum and a tablespoon of melted butter. The recipe called for half-and-half, but since I had plenty of milk and cream, I made my own rather than buying an additional dairy product I wouldn’t be able to use for anything else.

Sunday’s fifth and final pie was “Best-Ever” Pumpkin Pie, from Richard Sax’s Classic Home Desserts. He had an unusual blind-baking procedure which I don’t think worked very well — it didn’t involve using weights, so even after docking, steam from the butter lifted the bottom crust off the surface of the pie plate, and the edges of the crust slumped down in the extra-deep 9½” pie plate specified for this recipe:
Richard Sax's Best-Ever Pumpkin Pie
While making this recipe, I ran out of my homemade pumpkin purée and had to substitute some canned pumpkin (actually Dickinson squash) to make up the difference. This recipe calls for rum as a fairly significant flavoring — three tablespoons — but only two eggs (all the other custards used three or four eggs), and brown sugar is the primary sweetener.

Finally, the straggler completed on Monday was Pecan Pumpkin Pie from the Moosewood Restaurant Book of Desserts. This pie is an interesting concept: a full-blown pecan pie on the bottom, with a pumpkin custard on the top (“so your holiday guests don’t have to choose”, say the authors). Unfortunately, it is marred by a number of execution difficulties — it calls for a ten-inch pie plate, which is not a normal size, and does not call for blind-baking the crust, leading to slumping, as seen here:

Moosewood's Pecan Pumpkin Pie (pecan layer)

Before the pecan layer went into the oven, the crust was sitting on the rim of the dish. Now it’s barely above the surface of the pecan filling.


The fluted-edge ceramic plate was the only one I could find that actually claimed to be ten inches, but I have a suspicion that it actually isn’t, and that this recipe was written for a wider, shallower pie plate. That seems to be borne out by the instructions, which claim that the pecan filling should be set after only 20 minutes of baking — I found that it took 30 minutes. But the pumpkin custard was done (indeed overdone) well before the 40 minutes called for:
Moosewood's Pecan Pumpkin Pie
This will clearly be a challenge to portion and serve, with the crust buried well below the top of the filling. If I decide I want to make this pie again (keeping in mind that I haven’t tasted this or any of the others yet), some changes will clearly be required — reducing the quantities to fit in a normal pie plate, blind-baking the crust to keep it from slumping, or making smaller (individual- or two-serving) pies using the same procedure. (Perhaps it would work better in an extra-tall-sided tart pan?)

Sorry folks, I’m not compiling nutrition information for all of these pie recipes! Any that I make primarily for myself, I’ll post again with more pictures, and give the nutrition results then.

Posted in Food | Tagged , , , , , , , ,

Pumpkin Pie Fest part I: the pastry

When you’re baking a whole bunch of something, it pays to arrange things assembly-line style, so that you’re not constantly moving ingredients from cabinet to work area and back, and so that the tools, measures, and bowls that are needed will be close at hand. Since I’m making six different pumpkin pies this weekend, that means preparing the pumpkin purée all at once (did that on Thursday), cubing and freezing the butter (Friday), then making all of the pastry (what this post is about), and finally making the custard fillings and baking each pie (Sunday). Once all the pies have fully cooled, I can pre-slice them, wrap them up tightly, and on Monday I’ll drive in to the office and put them all in the refrigerator.

Most of the pastry recipes are fairly similar, as you might expect. Except in the two recipes that specify otherwise, I used an equal combination by weight of pastry flour and all-purpose flour, as suggested by one of the King Arthur baking books; for recipes that did not give the mass of flour I assumed five ounces per cup. All of the cutting-in of butter was done in the food processor, regardless of the procedure specified in the recipe, except for Joanne Chang’s recipe which calls for a stand mixer; the liquid ingredients were mixed in by hand, except for Moosewood’s and Joanne Chang’s recipes, which were done in the food processor and the stand mixer, respectively. Some of the recipes did not call for resting the dough, but on general principle (and also to make the assembly-line principle work for a single baker) I decided that all of them would be rested for about twenty-four hours.

Notes on the individual crusts:

  • The Moosewood Collective, “Best All-Purpose Pie Crust”, Moosewood Restaurant Book of Desserts: I used the full amount of (optional) sugar specified in the recipe. For the pie recipe, the crust is not blind-baked, but it is double-baked — first with the pecan filling and then again with the pumpkin custard.
  • Richard Sax, “Nick Malgieri’s Flaky Butter Pie Dough”, Classic Home Desserts: This unusual pie dough calls for cake flour and baking powder in addition to the usual flour, butter, salt, and water. I probably didn’t use quite enough water, but the long rest should help the flour to fully hydrate anyway. This recipe makes a double crust, so I’ll use half for the Richard Sax pie (“Best-Ever Pumpkin Pie”) and half for the David Page and Barbara Shinn Pie (“Honey Pumpkin Pie”, Recipes from Home). Sax’s pie recipe includes blind baking, but the Page/Shinn recipe uses an uncooked crust.
  • Emily and Melissa Elsen, “All-Butter Crust”, The Four & Twenty Blackbirds Pie Book: I’ve made this one before; the distinctive ingredient is cider vinegar, which helps to prevent gluten development. I made a double recipe, and will freeze half to use with another one of their recipes later in the fall. Their pie recipe calls for “partial” blind baking, for which they have a standard procedure shared by all similar pies.
  • “Golden Pumpkin Pie”, King Arthur Flour Whole Grain Baking: as you might expect, the pastry for this pie is rather different from all of the preceding. It contains two whole grains — ground oats and red whole wheat — and is sweetened with brown sugar rather than granulated or not at all. In addition, the crust includes a large amount of cinnamon (¾ tsp) and uses milk for the liquid rather than ice water as all the previous recipes do.
  • Joanne Chang, “Pâte Brisée II”, Flour: Chang’s recipe is the most unusual, as it is enriched with egg and milk, and as mentioned above, is made in a stand mixer. Her pie recipe calls for fully blind-baking the shell, unlike the other recipes which are only partially blind-baked.

More to follow tomorrow!

UPDATE (2014-10-12): I ran out of eggs for pie #6, so the post with all of the finished pies will have to wait until Monday evening after I’ve had a chance to go to the store. (I also ran out of homemade pumpkin puree, so I had to supplement pie #5 with canned pumpkin — actually canned Dickinson squash — and pie #6 will be entirely made with canned.)

Posted in Food | Tagged , , , , , , ,