Quote of the day: Atul Gawande on priorities

Everywhere I see the mistake of ignoring that people have priorities in their lives besides merely surviving another day. Even in severe illness or frailty, people desire connections to others and to purposes of their own choosing. I think we’ve been wrong— I think we’ve been rather limited about what we think our job is in building systems of care for human existence. We think our job is to ensure health and survival, but really it is larger than that. It is to enable wellbeing, and wellbeing is ultimately about sustaining the reasons one wishes to be alive.

Atul Gawande, “The Idea of Wellbeing“, BBC Reith Lecture, 2014-12-16, at 22:36

Quote | Posted on by | Tagged , , | Leave a comment

CSAIL Holiday Baking Fest

This gallery contains 11 photos.

Last Monday, we held the first ever (so far as I can tell) CSAIL Holiday Baking Fest. It was not quite as well attended as I would have liked, but all the food did get eaten. (I ate rather too … Continue reading

Gallery | Tagged , , , , , | Leave a comment

Something useful I learned at Whole Foods

I was looking for chocolate meeting particular specifications for some recipes I’ll be writing about soon. I went to the cheese specialty department at my local Whole Foods, where they sell broken blocks of chocolate, but the labeling is too general to be useful in determining what products they are actually selling. (The labeling on Valrhona products is usually pretty specific and accurate, but not so for Callebaut.) Once I found a clerk who understood what I was asking for, I was able to get them to tell me what the various blocks of chocolate actually were. I don’t know how much this varies from one unit to another (department managers at Whole Foods have a fair amount of flexibility in what they carry), but for the benefit of those who don’t want to mail-order their Callebaut, here’s what I found out:

  1. The chocolate labeled “bittersweet” is L-60-40-NV (60% cacao).
  2. The chocolate labeled “semisweet” is 811NV (54.5%), or perhaps D811NV (52.6%).

I only asked whether they had any 811 or 815, which is why I don’t know which of the 811s it is; I would have preferred the 815NV, but I took what I could get reasonably easily in the small quantity (less than 1 kg) that I needed. I don’t normally make recipes that specifically call for low-test chocolate; my normal chocolate for baking would be TCHO or Valrhona, but in this case I wanted to prepare the recipes as written, and both called for 55% rather than the 66% or 70% I would usually reach for.

Of course it would be much simpler if they labeled the Callebaut packages with the actual product inside, like they do with Valrhona (and used to do with El Rey back when they carried that).

Aside | Posted on by | Tagged ,

Random thoughts in Los Angeles

I’m in Southern California for the Thanksgiving holiday, and I just wanted to collect a bunch of tweets that I made to preserve them for posterity, or something. At least those of you who don’t follow me on Twitter will get to see, anyway.

(Pretty much everything that exists today other than the basic bus network was built in the wake of the Northridge earthquake — although I believe the Metro Red Line was in planning and/or construction already when that happened. MetroLink commuter rail was inaugurated to relieve pressure after parts of the 10 freeway collapsed in the quake.)

(I seem to be the bad-luck fairy for hockey; I don’t think the home team has ever won an NHL game that I attended.)

(About an hour in both cases.)

(The Comfort on Lankershim was about $25 more expensive a night than this place, and it’s not all that wonderful either. Terrible radio reception, for one thing.)

Aside | Posted on by | Tagged , , , , , , ,

Quote of the Day: Fred Vultee on the Fox Agenda

It’s been a while since I’ve posted a Quote of the Day (Month?). Here’s Wayne State journalism prof Fred Vultee riffing on a partisan jab from the Baltimore Sun‘s TV writer:

Fox might or might not think its viewers are stupid, but it certainly doesn’t seem to mind when they’re happily misinformed about the world at large. Rational viewers aren’t good for Fox, because they might turn into rational voters: able to make autonomous, evidence-based decisions that reflect their own interests about stuff like terrorism, immigration, the climate, and the threats posed by infectious diseases, apology tours and Muslim baby names. That, in turn, makes it harder to enforce a sense of national panic in which you hand your civil liberties over to friends of Roger Ailes for the duration. (If that’s starting to sound like a seminar in Copenhagen School securitization theory, there’s a reason for that.)

Fox isn’t the first media outlet to dominate its market through paranoia, jingoism and tabloid fascination with celebrity culture, or the first to take a histrionic dive when the ref was watching. You could call it a living tribute to the McCormick-Patterson axis, putting it rather squarely in the mainstream of Industrial Age journalism. To borrow from another Baltimore media critic, Fox has figured out what The People want and gives it to them good and hard. Fox viewers don’t have to be stupid as long as they continue to get the drama of inconvenient truth mixed up with the thrill of convenient fiction. Benghazigate, Grubergate and those friendly games of chance at the county fair were made for such an audience.

– “Another Day on Planet Fox“, Heads Up: The Blog, 2014-11-22

Quote | Posted on by | Tagged , , ,

What I saw (and did) at (and around) LISA’14

I recently got back from my one-per-year business trip, to attend the 2014 LISA Conference, and now that I’ve presented to my group, it’s time to make a more public write-up. I’ll start with some of the more personal or social things I did, and then dig into the sessions I attended and what if anything I learned.

This year I’ve been flying JetBlue a lot, now that they have reasonable schedules to the West Coast (and by “reasonable” I really mean “the return flight isn’t a red-eye”); their regular seat pitch in coach is wider than premium seating from either UA or DL, and when you’re checking a bag, the old-line network carriers’ bag fees really add up. JetBlue has non-stops from Logan to both San Diego (where my parents live) and Seattle, not to mention LA (where I usually end up spending at least a few days every other April or thereabouts) and the Bay Area. There is, however, one drawback: the B6 winter schedule has no transcontinental flights returning to Boston on Saturday. So I could either return on a Friday red-eye, or stay an extra day on my own dime, or fly another airline (Alaska is the biggest carrier at SEA and also had non-stops to Boston, code-shared with AA).

I chose to stay the extra day, and used the Saturday after the conference to do touristy things — I went to Pike Place Market for lunch, and then took the #40 bus out to Fremont to see the Theo chocolate factory. I was lucky in that, although the factory tour was fully booked for Saturday, there was a group of eight no-shows, so they let me join the tour five minutes in, after paying my $10. The tour was pretty meh. There really isn’t a whole lot to see in a chocolate factory, particularly on a Saturday when there’s nothing in production. The tour did take us into the kitchen, where the non-chocolate confectionery is made (caramel centers, ganache for truffles, and fruit and nut flavors that are mixed into the bars); it’s also where their enrobing machine is located — but again, with no production happening, you can’t really see anything. (And even when the factory is operating, you still can’t see much, since much of the machinery is closed up for food safety — there’s no giant Willy Wonka-style fountain of chocolate.) I did learn a few things: their organic sugar, which comes from Brazil, is distributed by a company from Arlington, Mass.; the bars with mix-ins were historically blended by hand, but now (with a recent factory expansion) have moved to a more automated production line; and they make a large number of SKUs that are not nationally distributed — some of them are sold only in the factory store. (It doesn’t hurt that Fremont is a trendy Seattle neighborhood, with plenty of residential and office development nearby and very frequent bus service, so the factory store is rather more accessible than the typical zone industrielle where food factories would be located.) I ended up spending just shy of $200 at the factory store, buying nearly everything (except coffee) that wasn’t available at my local Whole Foods. (I bought a few things that Whole Foods doesn’t carry but, as it turned out, I could have gotten from my local REI instead!)

At Pike Place Market I had a salmon club sandwich for lunch, and also bought several bags of chocolate-covered cherries from Chukar Cherries. A nut vendor sold hazelnuts (their own) and smoked almonds (from a California maker). I didn’t waste time watching the fishmongers throw fish around, but I also probably only saw a quarter of what was on offer there. (There were a number of things I might have bought except that I had no safe way of getting them home in one piece — and I don’t care to find out what the airline, even JetBlue, would do to a jar of jam or honey in my checked baggage!)

I have an unfortunate habit of forgetting one really important thing whenever I travel. I very nearly forgot my cell phone, which would have been a disaster — thankfully, the Logan Express bus ticket agent has an EMV credit-card terminal, and as I was paying with my corporate travel card, I didn’t have the PIN memorized and had to look it up. Groping for my phone, I realized that I had left it in the car, and I dashed off to grab it at the last minute. (As it turned out, I was in so much of a hurry that I forgot to re-lock my car after grabbing the phone — good thing nobody tried to break into it while it was parked in the lot.) Having successfully remembered my phone, what it actually turned out I had forgotten was any clothing warm enough for the frigid weather which gripped Seattle — and the rest of the nation — during the six days I was there. Normally at this time of year, daytime highs in Seattle are around 50° — but this year, this week, it struggled to get much above 40°, and to compound the insult, it was windy as hell. I brought only lightweight jackets and my usual short-sleeve knit shirts, plus a couple of shirts from previous LISA conferences. The LISA’11 shirt was the only long-sleeved shirt I had, and it was grossly inadequate to the task. (The LISA’14 conference shirt, which was short-sleeved, continued the trend of Usenix buying very light-weight fabrics that are suitable only as underwear or in very warm weather. The LISA’99 shirt was made of heavier fabric, but was, like most of the others, a T-shirt.) The conference hotel, as is the way of these things, was frigid, although not quite as cold as outside.

It was at LISA’13 in Washington that I made a fairly important lifestyle decision, after talking with LISA and Usenix regular David Parter (of UW-Madison). Parter had lost a good deal of weight, and he credited a mobile app that he was using. At the time, I was having to take medication twice a day for glucose tolerance and high blood pressure, and I was constantly paying to repair small tears around the seams in my shirts. I had been exercising on-and-off on a stationary bike with no appreciable results, and it sounded like the app that Parter was using might help me implement enough calorie restriction to actually make a difference.

It turned out that it did, and that as I lost some weight, it also became easier to exercise, and I was able to incrementally lengthen my workout from barely 25 minutes total to over an hour over the course of the following six months. (The Winter Olympics in February greatly aided this, but it’s really baseball that makes the biggest difference — with games nearly every night, I’m not at the mercy of 22- and 48-minute TV shows for the visual distraction that keeps me on the bike late at night when I’d rather be relaxing. This fall I subscribed to NHL Center Ice so that I could watch hockey during the baseball off-season.) I knew that I wanted (and in practical terms needed, in the presence of all the free food a conference entails) to keep up my exercise regimen as best I could, and since I was staying in a big-city business hotel, I knew both that there would be a fitness center I could use, and that I needed to buy something to wear while doing so — hotels not being so keen on guests walking through the public areas in their underwear! Luckily, when I stopped by my local REI, they had a bunch of cycle clothing on clearance, and I bought a couple of shirts and a couple pairs of shorts (one pair of double shorts that would be OK to wear walking around — if you don’t mind the big chamois butt-pad — and one pair of “lycra bandit”-type shorts). I made sure to try them on before leaving, and made a shocking discovery: I can actually wear a size medium — at least if it’s made of Spandex. A year ago, I would never have believed I could ever again wear anything smaller than an XL. But I’ve gone down seven inches in waistline, and for some styles and manufacturers, that puts me in a medium, at least for bottoms. I went back to REI the next day to buy more cycle wear (including a pair of “M” tights), which felt ridiculously good despite the expense.

(I made a similar discovery in the market for new underwear a few months ago: a 33 waist might be a small, medium, or large depending on which maker’s size chart you go by. I’ve been gradually replacing the old XL underwear, but I’m never going to have the build of an underwear model, so I don’t think I’ll ever be wearing size “small” smalls no matter how much more weight I manage to lose.)

I arrived in Seattle around midday on Tuesday, which was an unusual bit of lucky scheduling for me — far more often I arrive at the conference hotel well after the badge pickup desk has closed, requiring me to stand in line early the next morning. When I checked in, the hotel clerk asked me if I wanted to waive housekeeping for the duration of my stay, in exchange for a $5 voucher each day, which certainly takes the “feel good and let us not wash the towels every day” a step further. (I declined, but apparently he didn’t key it correctly, so the next morning a voucher was shoved under my door and sure enough, no housekeeping. I didn’t care enough to complain, though, until the toilet paper ran out!) I watched the local TV news, where they were talking about roads being blocked and houses being damaged by falling trees, all a result of the uncharacteristically high winds sweeping the region. I set up my airchecking station near the window, and was greatly puzzled by the extremely poor reception — there must have been some sort of conductive coating on the windows, because my ninth-floor hotel room, facing north, should have been able to get far better signals (and usable HD) from the West Tiger Peak and Cougar Mountain transmitter sites; as it was, the only stations I had a good signal from were those with in-town transmitter sites (a couple of translators and a few stations on Capitol Hill and Queen Anne Hill). Then I went to find the fitness center, and found my resolve to exercise every night wavering a bit, despite the consequences: I was in the “Union” tower of the hotel, and the fitness center was at the very top of the “Pike” tower — I would have to go all the way down to the conference floor, or worse still, the lobby, in my sweaty workout clothes. And when I got to the fitness center, it was absolutely frigid. (This turned out to be a good thing, but I was just shivering there, in my street clothes at resting metabolism.)

After all that, I got in touch with my former co-worker (and recent-ex-Amazonian) Noah, and we went to a seafood restaurant a few blocks from the hotel, which coincidentally was also just a few blocks from the office building where Noah used to work. We got caught up, having not seen each other for a few years, and he gave me a bit of an insider’s view of the Amazon sausage factory. (This week he started on his new job working for a software company elsewhere in Seattle — the sort of place that didn’t really exist 15 years ago when I first visited the area. Back in 1999, there was Microsoft, and Boeing, and their suppliers, Starbucks, and UW, and that was pretty much it. The whole region is much more economically diverse now than it was then, although it’s still fairly concentrated, industry-wise.) Noah attended the conference, and we met again a few more times — on Wednesday, after seeing how awful the free lunch looked, we met up with another former co-worker, Eric, and had lunch at an Asian restaurant a few blocks from the hotel. (Eric lives on Vashon Island, and I had originally planned to go out there on Saturday to take a tour of the many radio towers that call Vashon home, but in view of the cold weather, long travel time, and limited daylight, I decided to put this off until some other time.)

I met up with a number of the “usual suspects” after the conference began. I often spend time at LISA with Steve VanDevender, who works at the University of Oregon in Eugene; Steve also teaches a summer undergraduate course at UofO in system administration. There’s also a guy whose name I can never remember who works at the University of Alberta in Edmonton — one of the few people for whom the weather in Seattle was actually warmer than what they had left. There were as always a lot of Googlers, but I only really know one, Chris Davis, who joined Google with their acquisition of ITA Software, which was an MIT-spinoff company in the travel industry. (ITA built airline fare search and reservation systems, along the way discovering that air travel planning is NP-hard, before being acquired by Google.) There are other regular attendees who I know by name — David Blank-Edelman, Tom Limoncelli, Lee Damon — who I will say “hello” to, but don’t really socialize with. (These people tend to be the serial organizers and presenters; I’m really only acquainted with them by dint of being on the edges of conversations and occasionally putting in a word.) I don’t think David Parter was there (if he was, I didn’t see him), but there were plenty of other regulars who I recognized but have never known by name.

Now for the conference itself: there were a number of organizational improvements in the conference this year, some of which will hopefully be repeated next year in Washington. There were two new tracks introduced: mini-tutorials, which are 90-minute introductions to material that a regular tutorial takes six hours to cover, and vendor talks, which give vendors (who are also conference sponsors) an opportunity to present their products in a lecture format, rather than the one-on-one selling of the vendor show floor. I have no idea how well either of these worked out as I didn’t go to any of them. There were a few mini-tutorials I was interested in, but they were always scheduled against talks I wanted to see. (This is a problem in general with LISA, for me at least: all too often, the 90-minute blocks of the technical program are either completely uninteresting or have multiple talks I want to see.) At least the Google “vendor BoF” (read: recruiting event with free beer) was scheduled well in advance, for the very last slot on Thursday, after the off-site conference reception, so that anyone who wanted to organize a successful BoF could choose a better night. These “vendor BoFs” suck the life out of the regular BoF track — the Google one especially — and the conference would really be better off without them, but the fees the companies pay help to offset the cost of running the conference, and keeping these events “inside” helps keep them somewhat under control.

Another problem I have to deal with is the decreasing relevance of the LISA program to anything that I do. While it can be interesting to learn what the “industry leaders” are doing, all the cheerleading in the program (“DevOps! Agile! Rah rah rah!”) can leave the impression that if you don’t work for either a gigascale Web shop or an equally enormous *aaS provider, you don’t really belong in this industry. (Until recently, we had exactly one “dev” — and she’s my officemate. Now we have two developers. In order to “deploy thousands of times a day”, you have to have thousands of things to deploy in a day. As I explained to someone, we don’t make software: our product is called a “Ph.D.”, and no amount of “agile” is going to reduce the average delivery time of eight years per diploma.) So when the LISA program is announced in late summer, I always have to take a long hard look to be certain I can justify spending $3000 of our limited travel budget to go there.

Since I went to dinner with Noah fairly late on Tuesday evening, I didn’t get to do some of the BoFs that I was interested in. I walked into the very tail end of the “Why FreeBSD Is For Me” BoF, which was organized by Dru Lavigne of the FreeBSD Foundation, then went to a documentation BoF organized by a guy from Red Hat. I didn’t get much out of either one, and was pretty tired after the end of that, so I went back to my room and crashed (without exercise). On Wednesday, the vendor expo provided free food (not very good and in far too great a quantity), so I was able to attend the early BoF sessions; I went to the monitoring BoF and learned about what people were doing for alerting and trending. Most people are still using separate systems for this, which makes perfect sense to me, but one BoF attendee talked about how he was dumping all of his metrics collection into ElasticSearch and doing alerting by running queries against the ES database. (This was a nice antidote to all the metrics talks in the main conference program, most of which were given by people in the metrics-as-a-service business. See below.)

After the monitoring BoF, I went to the PuppetLabs vendor BoF, and learned a bit about how PuppetLabs was planning on breaking our Puppet infrastructure in the future, as well as some prophylactic measures we could take to prevent it. The PuppetLabs release engineer was incredulous that anyone would actually install the latest version of Puppet automatically, and repeatedly said that it was not supported to run newer Puppet agents against older servers. (In our experience, newer Puppet servers are much more likely to break stuff that we depend upon than newer agents fail to work with old servers — we have that situation now where some client machines are running 3.7 — because they run FreeBSD and that’s the version that’s available — but we have to keep the server back at 3.6. After the BoF, I spent some more time talking with the PuppetLabs guy about this, and he tried to convince me that everything would be wonderful if they built packages with their own, renamed, versions of Ruby and OpenSSL. I said “No thank you”, and he persisted in thinking that this was wonderful. I said “I’d prefer to have timely security updates”, and he didn’t seem to hear me. Dear vendors: THIS IS A STUPID IDEA. Package builders insist on using a single copy of library and interpreter dependencies for a good reason, and however much you think you’re smarter than they are, we the users know better.)

I didn’t see anything interesting in the 9 PM BoF slot, and decided to brave the cold in the fitness center to at least try to get some exercise. My stationary bike at home is an old Vision Fitness unit, with a battery-powered LCD display and a fixed-drag braking system; the hotel fitness center had Life Fitness bikes with built-in (analog) TV tuners and a weird and discomfiting variable-resistance system. I was unable to find my comfort cadence regardless of how I set the resistance, but at least it was easy enough to get the seat height right. My bike has a completely uncalibrated calorie display, and I was really shocked to find out how far off the hotel bike’s results were from my unit at home — a workout that my home bike thought was good for 1000 calories was only 700 on the hotel one. I have reason to believe that the hotel unit’s readings were more accurate, however, and until I have the money to replace my old bike I’ll be basing my workouts on those numbers. I worked out again on Thursday, after the conference reception (again, far too much “free” food, and my phone died while inside the museum so it was impossible even to keep track of what I was eating), and on Friday before stopping in at the “dead dog” party (which is really for the conference organizers but somehow I’ve been allowed to attend since I was an invited speaker in San Diego a decade ago).

Now as for the main technical program: I thought the actual quality of the sessions I attended was pretty variable. Some were excellent, some were mediocre but interesting enough to stay, a few were downright boring. The opening keynote was given by Ken Patchett of Facebook, who talked about data center design (I did that back in 2004, and unless I change employers it’s highly unlikely that I will ever again be involved in such a project); it was moderately interesting, particularly the bits about getting more efficiency by keeping power distribution at 480Y/277 rather than stepping down to 208Y/120 as is normal. (Not sure how well that would work for our Canadian friends, where 600Y/346 is used instead of 480 — I can imagine a lot of employers would not care to have 600-volt power distribution just due to the increased training and insurance expense. But Facebook is pushing this as a standard through something called “Open Compute”, so maybe they’d use 480 in Canada anyway. In Europe they would presumably stick with the normal 400Y/240 that’s already used for three-phase power distribution.) Facebook also does a lot with outside (unconditioned) air, which reduces costs substantially, but only for those DCs that have cool enough air for enough of the year, which not all of them can, given the other infrastructure constraints that affect DC siting.

For the first session on Monday, I attended Dieter Plaetinck’s talk on “Metrics 2.0“, the first of several talks on that theme in this LISA; his presentation was about standardizing the representation of metrics metadata, so that more of the work in presenting metrics to administrators can be automated — a worthwhile goal, at least, given the huge quantity of data being generated and collected today. (We currently use Munin, which bypasses this issue by permitting only fixed presentation defined by the data source. But Munin has lots of limitations and we’d like to replace it with something more flexible.) The second talk in the session was about resource accounting for HPC facilities, and even though our operation is very HPC-like in a lot of ways, this talk was totally irrelevant to me and I left about ten minutes in to join the hallway track.

After lunch (with my former co-workers Eric and Noah at Wild Ginger), I attended the paper session, but found only two papers to be at all interesting. “An Administrator’s Guide to Internet Password Research” was a survey paper summarizing results about what sort of password policies were most effective given rational sorts of threat models. The presenter made the claim that there were only two sorts of attacks worth considering: online attacks, which can be foiled by a fairly simple password quality checks in combination with account lockout controls, and offline attacks, security against which requires much stronger passwords — always assuming the underlying authentication mechanism is actually implemented competently. (Incompetently implemented authentication mechanisms are much easier to break by directly attacking the mechanism rather than by attacking the password, so there is no need to consider them — they are all easier than online guessing.) The presenter agrees with me that the commonly referenced alphabetic entropy model for password quality is at best misleading if not completely erroneous. The second interesting paper presentation, “Realtime High-Speed Network Traffic Monitoring Using ntopng”, was apparently not interesting enough for me to remember what was presented, but you can read the paper, which received the LISA’14 best paper award. I read mail or snoozed through the other paper presentations.

For the second session Tuesday afternoon, I attended talks in the “metrics” track. David Josephsen of Librato, one of those metrics-as-a-service companies, talked about “Feature Flagging at Scale”. He described how they tested and incrementally deployed an architectural change to how Librato collects metrics from Amazon Web Services, saving substantial expense for themselves, without taking substantial risks of business interruption or losing client data. They have an internal chatbot that, among other things, developers use to initiate deployments and reconfigure feature flags. The second talk of that session was Theo Schlossnagle of Circonus — another metrics-as-a-service company — explaining how almost everybody had failed to study statistics and was doing metrics wrong, and giving a few examples of how one might get the statistics right without hiring your own team of statisticians. Of particular note, the non-normality of most of the metrics we collect (e.g., request latency) means that the most common statistical techniques (like mu-sigma) give meaningless results, and the non-stationarity of the distributions that we do so means that most machine-learning techniques don’t work for anomaly detection. (<wavy lines/>The last time LISA was in Seattle, in 1999, I saw the guy from WebTV — remember WebTV? — talking about using Holt-Winters models to do anomaly detection on the metrics they were collecting with Cricket.<wavy lines/>) There was a poster session after this, but I didn’t go. The “free” food in the vendor expo passed for my dinner on Wednesday, but I don’t recall what it was (only that it was better than the lunch offering had been).

Thursday morning’s plenary was a “Rah rah rah! Agile! DevOps! Go team!” talk by Gene Kim, which was totally uninteresting to me, but since I was already awake, I went to it anyway. Kim seems to conflate “high performing IT organizations” with those that employ hundreds if not thousands of developers. I’m sure somebody got something out of this talk, but I found it a complete waste of time. After the morning break, I attended Brian Atkisson’s talk about “Open Source Identity Management in the Enterprise”, which was a bit less relevant than it might have been despite being easily the most informative talk of the whole conference. Unfortunately, Atkisson works for Red Hat, which means that “Open Source Identity Management” means, with a few exceptions, “Red Hat products for which there happen to be related open-source projects so we can claim to be an open-source company”. Most of these products were acquired by Red Hat from Sun, which had acquired them in the dissolution of Netscape a decade ago, and it shows. I played around a bit with “389 Directory Server”, which is indeed open-source (there’s even an Ubuntu package), after I got home, but the documentation for it seems to be locked up behind a Red Hat paywall, so I didn’t bother to pursue it any further. (We would probably do well to put a more serious effort into it, particularly because we could stand to have a better Certificate Authority implementation than my hand-hacked Ruby CA that I’ve been running since 2002, but I really don’t have time to figure it out this week, at any rate.) The one particularly relevant non-Red Hat product that he talked about was (MIT) Kerberos, which we’re obviously already running. There was some more interesting stuff about integration of one-time password technology (required for Sarbanes-Oxley and PCI compliance) with their systems. Interestingly, Red Hat’s engineers hate OTP, and love the SSO experience they get with Kerberos — wish our users were like that. (Our users hate Kerberos and love the only-one-login-ever-until-I-reboot experience they get with ssh-agent forwarding. Doesn’t work with AFS, though, so they can’t have it, no matter how much they would prefer it.)

After lunch on Thursday (taken in the vendor expo, but far better food than Wednesday’s that I skipped), I attended Hal Stern’s talk “You Have Too Much Data”, which I didn’t get much out of, and then in the same session David Josephsen’s “Data Storage at Librato” talk, which at least told me that Librato was one of those companies that had the statistics wrong in the way Theo Schlossnagle was talking about Wednesday afternoon (specifically, using mu-sigma on non-Gaussian data). This was one of the sessions that I could have completely skipped, except that I didn’t have anything else to do that would have been more interesting. For the late afternoon session, I attended Mikey Dickerson’s talk which wasn’t really about “the healthcare.gov meltdown” — more a pitch for the newly established “US Digital Service”, given remotely from a White House teleconference room with no audience interaction. Dickerson was a Googler, who took leave to join the team that “fixed” healthcare.gov, and was then persuaded to lead the USDS, which is an Executive Branch office that helps other federal agencies figure out how to deliver digital services to citizens and other clients. He worked on the Obama campaigns in ’08 and ’12 and will presumably be looking for work in 2017. I probably should have gone instead to the talks “Managing Large Scale Cloud Infrastructure at Rackspace” (by Jesse Keating at Rackspace) and “Building a ‘Multi-Landlord’ Public Cloud” (by Peter Desnoyers at Northeastern) — I’ll have to make do with the videos, once they’re released.

Thursday evening’s social event was the conference reception, held at the Paul AllenEMP Museum. I saw some interesting exhibits, particularly ones showcasing guitars and science-fiction memorabilia in the museum’s collection. I only saw about half of the museum; it would have been nice to have a bit more time there without the conference crowd.

Friday’s plenary is always the conference closer, so there are two sessions on Friday morning. In the early session, I went to “Software is Eating the Network: SDN and Devops” by John Willis, which was not particularly relevant to me (takeaway: “Docker and OpenVSwitch are going to rule the world! Thousands of containers on ever server! You will be assimilated!!!!1!!”), which I say through mainly to see the Lightning Talks, which were scheduled for the second half of the session. I also wanted to attend Ben Rockwood’s “I Am SysAdmin (And So Can You!)” and Tom Limoncelli’s mini-tutorial “Live Upgrades on Running Systems”, both of which were scheduled against the Lightning Talks, and Lee won. At the second morning session, I went back to the metrics track, but Caskey Dickson’s “Gauges, Counters, and Ratios, Oh My!” was not especially interesting — aside from introducing a bit of additional vocabulary, it didn’t offer anything that I hadn’t already heard from the Wednesday metrics talks. I definitely should have gone to one of the other sessions (either Luca Deri’s mini-tutorial on ntopng or Abe Singer’s talk on password policies).

I stuck with the metrics track again for the Friday afternoon session. Eldon Koyle’s talk was mostly live demo and not very interesting (Koyle didn’t write the software, and it doesn’t do IPv6 or indeed pretty much anything that isn’t a single IPv4 /16); Toufic Boubez (another one of those metrics-as-a-service guys!) talked about anomaly detection, and most of what he said was the same as what Theo Schlossnagle said on Wedneaday, aside from adding a few references to useful techniques and textbooks that might help people actually get metrics right. (And thinking about it now, I’m wondering how much I’ve conflated Boubez’s and Schlossnagle’s talks.)

The closing plenary had been originally scheduled to be a NASA person talking about planetary exploration, but that speaker had to cancel at the last minute and was replaced by a Nordstrom VP. Her talk was utterly uninteresting. I spent the time cleaning up my email; if this speaker had been in the advance program rather than an emergency replacement, I would have arranged to fly home on Friday. Instead, I sat through a boring speech and then went to dinner with Steve VanDevender, a Dutch Googler from their Zurich office, and a German fellow whose employment I don’t recall.

So what did I learn? Well, metrics are hard, and particularly uninformative if you get the statistics wrong. There are some LDAP tools from Red Hat that I should really investigate. DevOps cheerleaders have not gotten any less irritating. PuppetLabs is going to keep on breaking shit, Munin and Nagios still suck, and Google treats ops people better than Amazon does. There are still a whole lot of efficiency improvements possible in data center design (but only if you get to build them from scratch at cloud-provider scale), and SDN might be relevant to my job some day. Discount hotels have better radio reception in guest rooms. Metrics-as-a-service is ungodly expensive. mod_auth_mellon might be interesting if we find ourselves needed to implement SAML. “Federated logout” is a thing, and the way SAML specifies it is unimaginably stupid.

Oh, and despite exercising four nights out of five, I still gained five pounds. I blame the free food.

Posted in Computing | Tagged , , , , , , ,

Other people’s recipes: Four & Twenty Blackbirds’ black walnut pie

Almost a week late, here’s a quick rundown on the delicious black walnut pie I made six days ago. The recipe comes from Emily and Melissa Elsen, The Four & Twenty Blackbirds Pie Book, like most of the pies I’ve baked recently, and it’s the last of their “Fall” seasonal pies on my list. It starts, of course, with toasted black walnuts. Black walnuts are the native American walnut, but because they are more convoluted than English (aka Persian) walnuts, they are more difficult to shell, which makes them more expensive; they are only seen as smallish broken pieces, never “whole”, unless you happen to have a black walnut tree in your yard. (And you probably don’t want to: walnuts whether black or English are drupes, and the ripe fruit — which must be stripped off to get to the nut — contains a very powerful dark brown pigment that stains anything it touches.)

Black walnuts have a delicate, fruity fragrance which makes them far more interesting in flavor, however, to make up for the increased scarcity. I bought a pound of them by mail order from Hammons Products, one of the principal processors and distributors of black walnuts, located in Missouri near the most of the orchards that grow black walnuts as a cash crop. (In addition to retail and wholesale black walnuts, Hammons also sells black-walnut confections.)

The Elsens recommend their “cornmeal” pie crust rather than the “all-butter” crust used for most of the other recipes I’ve tried so far. After partial blind baking it looks like this:
Patrially prebaked cornmeal crust

I honestly could not tell the difference between this shell and the regular (all wheat flour) recipe. The recipe continues with an “apple cider caramel”, which looked to me nothing like any caramel I’ve ever made. (Not for the first time with this cookbook, either — one of these days I’ll have to do a side-by-side comparison of their recipe and a normal caramel to see which one actually turns out better. My experience with the Elsens’ “caramel” recipes is that the foaming of the butter, which they add at the beginning, makes it very difficult to actually tell when the sugar is at the right stage, and I don’t think the temperatures they suggest are anywhere near correct for proper caramel. A normal caramel procedure would be to brown the sugar, then quench with liquid, then stir in butter off the heat — see, for example, the Cook’s Illustrated chocolate caramel walnut tart, which I’ll be making again this year with leftover black walnuts.) In any event, the sugar mixture is cooled and used to make an egg custard, with the addition of some cornmeal as a thickener, and, of course, the toasted black walnuts. The whole process is very similar to a pecan pie, or to the black bottom oatmeal pie I did a few weeks back, but with a flavorful cooked-sugar syrup rather than corn syrup.

I found that my pie was way underdone after the recommended 30 minutes of cooking time, and ended up giving it another ten minutes. When I finally took it out, it looked like this:
Finished pie, hot from the oven

The Elsens do caution, however, “Be careful not to overbake, or the custard can separate”, and I suspect that’s what happened here:
Slice of black walnut pie
If you look closely, you can see that the filling has in fact separated into a yellow, eggy custard layer on the bottom and a sweet, syrupy layer between that and the nutty bits on top. Nobody at the office seemed to notice, but I did, which tells me that my pie-doneness-determining skills still need some work. Here’s what the rest of the pie looks like:
Pie minus one slice


I figured this pie based on eight slices, which makes it look a bit worse than other pies I’ve done recently, but is perhaps more realistic. (I always have trouble trying to get ten slices out of a pie — maybe I should buy one of those professional pie slicing devices they use in bakeries!)

Nutrition Facts
Serving size: 1/8 pie
Servings per container: 8
Amount per serving
Calories 471 Calories from fat 261
% Daily Value
Total Fat 29g 45%
 Saturated Fat 14g 69%
 Monounsaturated Fat 0g
Trans Fat 0g
Cholesterol 122mg 41%
Sodium 178mg 7%
Potassium 55mg 2%
Total Carbohydrate 47g 16%
 Dietary fiber 2g 8%
 Sugars 28g
Proteins 9g 18%
Vitamin A 16%
Vitamin C 1%
Calcium 2%
Iron 7%

Here is the data for the cornmeal pie crust alone:

Nutrition Facts
Serving size: 1/8 pie
Servings per container: 8
Amount per serving
Calories 178 Calories from fat 100
% Daily Value
Total Fat 18g 17%
 Saturated Fat 7g 35%
Trans Fat 0g
Cholesterol 30mg 10%
Sodium 74mg 3%
Potassium 11mg 0%
Total Carbohydrate 16g 5%
 Dietary fiber 1g 3%
 Sugars 0g
Proteins 3g 5%
Vitamin A 8%
Vitamin C 0%
Calcium 0%
Iron 2%
Posted in Food | Tagged , , ,

Limited posting schedule for the rest of this month

I’ll be making two trips to the west coast later this month, and that will put a crimp on blog posts for a while (especially the food posts). There will be one more post from last weekend’s cooking, which should come tomorrow or Thursday depending on when I get off my butt and write it. I have a couple of other posts percolating: a big idea post about the two most important lenses through which we see the world (and how we don’t do a very good job at teaching them), and a much smaller post reflecting on a year of daily commuting in a hybrid — not sure if either of these will get done before the month is out.

CSAIL Holiday Baking Fest will be the Monday after I get back from California, so there will be some more recipe posts for the treats I end up baking for that (currently planning on doing three: two chocolate tarts and a Polish tea bread).

Aside | Posted on by

Other people’s recipes: Cook’s Country’s Five-Alarm Chili

This gallery contains 10 photos.

It’s been a busy cooking weekend for me: in addition to yesterday’s truffles, I also did a black-walnut pie from Four & Twenty Blackbirds’ cookbook (post to come later this week), and I finished the weekend off with a huge … Continue reading

Gallery | Tagged , , , , ,

Other people’s recipes: Alice Medrich’s Extra-Bittersweet Ganache Truffles

This gallery contains 6 photos.

This week’s first recipe was born out of a dilemma: last week’s recipe called for a quarter cup of heavy cream. I don’t have anything useful to do with heavy cream, but it’s sold in pint bottles, so I had … Continue reading

Gallery | Tagged , , , ,