Overdue recipe report: Luisa Weiss’s Pfeffernüsse

I decided that my holiday baking this year would be three recipes from Luisa Weiss’s Classic German Baking (Ten Speed Press, 2016) — after all, many of our holiday traditions, and a good portion of American baking more generally, come from German immigrants, which accounts in part for the wide differences between US and UK Christmas food traditions. (And, of course, my family name also comes from German immigrants!) Among the most classic German-American traditions at Christmas are pfeffernüsse, small spice cookies (often confused in appearance, but not flavor, with Russian/Italian/Mexican tea cakes/wedding cookies) flavored with ground pepper and gingerbread spice (the latter is called lebkuchengewürz in German and is a common prepared mixture of ground spices used in many recipes). I remember my father making cookies like these when I was little, probably using the recipe from the 1976 edition of Joy of Cooking. Rombauer’s Joy recipe is quite different from the recipe in Weiss’s book (pp. 228–30), and I suspect it reflects a more “Americanized” version. Among the most notable differences, Rombauer uses molasses, black pepper, baking powder, brandy, and dry confectioner’s sugar where Weiss uses honey, white pepper, ammonium carbonate (“baker’s ammonia”), rum, and a lemon-sugar glaze, respectively. The most significant difference is that the Joy recipe contains butter whereas the only fat in Weiss’s recipe comes from the egg; Joy also uses candied citron and chopped almonds, flavoring ingredients absent from the Weiss’s pfeffernüsse. (By the way, in this post I will use English capitalization since I’m writing in English, but I will follow the German morphology for number, so it’s one pfeffernuss and two pfeffernüsse.) Weiss says that her recipe comes from Offenbach — like many traditional European recipes, there is a lot of regional variation — and it’s entirely possible that Rombauer’s recipe is equally authentic but from a different part of Germany.

Before starting to make the pfeffernüsse, it was necessary to gather the requisite ingredients for the lebkuchengewürz — since, unlike in Germany, it’s not something sold in grocery stores. The recipe is quite simple, and makes far more than is required for one or even two batches of pfeffernüsse: 30 g of ground cinnamon (I used true cinnamon and not cassia), 1½ tbl of ground cloves, 1 tsp of ground allspice, 1 tsp ground cardamom, 1 tsp ground ginger, 1 tsp ground mace (I’ve never seen a recipe use that much mace), and ¾ tsp ground aniseed. That last ingredient was a bit of a challenge to find — it’s one of those spices that isn’t stocked in my local Whole Foods, but luckily the conventional supermarket nearby does have it (it still took a bit of searching the shelves) so I didn’t have to mail-order. All these spices are simply mixed well and sealed up in a container.

Mise en place
Now for the pfeffernüsse proper, starting with the usual mise en place. Starting from bottom left, you can see the lebkuchengewürz, of which this recipe uses only two teaspoons. Next to that is ⅛ tsp of freshly ground white pepper, and then a teaspoon of ammonium carbonate (next to the bottle). The dry ingredients — 310 g flour, 1 tsp cocoa powder, and ¼ tsp salt, have already been mixed together. In the saucepan at upper right is 160 g of honey and 75 g of sugar, and hiding behind the pan is a single egg. There should have been 1 tablespoon of slightly warmed rum in that bowl in the center, but I mistakenly used my half-tablespoon measure instead. Oops.

Honey-sugar syrup
The recipe begins with heating the honey-sugar mixture on the stovetop until it’s hot enough to fully dissolve the sugar. While the mixture is still hot, the spices are stirred into it (this helps to “bloom” the spices and bring out their volatile flavors). Meanwhile, the ammonium carbonate is mixed with the rum, causing it to fizz a bit, and hopefully dissolve. (I found that even when I used the correct amount of rum, it wasn’t quite enough to fully dissolve the ammonium carbonate.) It will fizz quite a bit, losing some of the leavening power in the process (since there’s nothing to capture the gas at this point). Ammonium carbonate — (NH4)2CO3 — was once a popular leavening agent, since it decomposes entirely into gas (ammonia, carbon dioxide, and water) leaving nothing behind, unlike baking powder, which adds sodium or other metal ions like aluminum depending on the specific chemistry. The downside of ammonium carbonate is that half of the leavening power comes from gaseous ammonia, which is not a scent you want to have in your food, so it’s only usable in applications where you can be sure that all the ammonia will be driven off in the baking process.

After adding ammonium carbonate to hot syrup
When the rum-leavening mixture is added to the still-warm honey-sugar-spice mixture, a great deal more gas is evolved, giving the baker a strong whiff of ammonia. After stirring to fully incorporate, the resulting mixture is stirred or kneaded into the dry ingredients along with the egg to form a sticky dough.

Dough mostly combined
At this point, the dough is formed into small balls and deposited on parchment-lined baking sheets. Weiss calls for one-inch (25 mm) dough balls, for a yield of 48 cookies; when I used a #100 (⅝ oz) disher, I only got 31 — I was so surprised by this that I actually did the entire recipe a second time, and had the exact same yield. Because there’s no fat in this recipe, the cookies do not spread in baking, and they can be spaced quite close — I had no trouble putting 20 dough balls on a standard cookie sheet, and if you manage to get the 48 that Weiss calls for, there’s no reason you couldn’t fit them all on two baking sheets.

Fully baked dough balls
The pfeffernüsse are baked in a 375°F (190°C) oven for ten minutes — since there’s very little moisture in them to start with, it doesn’t take long, and you definitely don’t want them to completely dry out. While the first batch is baking, a simple lemon-sugar glaze is made from 100 g confectioner’s sugar, 1 tbl fresh lemon juice, and 1 tbl water.

Cookies after brushing with lemon glaze
Using a pastry brush, the glaze is applied to the the cookies while they are still hot. Just to keep the kitchen clean it helps to put the cooling rack over a sheet pan to catch the excess glaze.

Glazed cookies cooling and draining
You should probably do a better job than I did of evenly covering the surface of the cookies. Once cool, the pfeffernüsse should be sealed in an airtight for a minimum of several days to ripen — as the moisture from the glaze is absorbed by the cookies proper, the dry, crisp crumb becomes soft and chewy.

As I mentioned above, I ended up making two batches of these cookies, which left me with enough to half-fill a Christmas cookie tin that I already had. I had enough (about 60) to eat quite a few of them myself, give a lot away at the family Christmas party, give more away at Christmas dinner, and still bring a dozen into the office after Christmas where almost nobody was around to keep me from eating the rest.


The nutrition details below reflect my yield for this recipe and not Weiss’s. If you can actually get four dozen pfeffernüsse, you’ll have to cut these numbers by about a third.

Nutrition Facts
Serving size: 3 cookies
Servings per recipe: 10
Amount per serving
Calories 240 Calories from fat 4
% Daily Value
Total Fat <1​g 1%
 Saturated Fat <1​g 1%
Trans Fat 0​g
Cholesterol 19​mg 6%
Sodium 69​mg 3%
Total Carbohydrate 55​g 18%
 Dietary fiber 1​g 5%
 Sugars 30​g
Proteins 3​g 6%
Vitamin A 1%
Vitamin C 1%
Calcium 0%
Iron 3%
Posted in Food | Tagged , , ,

Several long-overdue recipe quick takes

There comes a point when you simply have to declare “blog bankruptcy” and clear out your writing backlog. That doesn’t allow you to discharge your writing debt in its entirety, but it does mean that you (well, I) can feel less bad about giving some posts really short shrift. So here are a few really quick hits on recipes I’ve made since my trip to Worldcon 75 in Helsinki last August that I haven’t gotten around to saying anything about yet.

First, the brownie recipe (entitled simply “brownies”) from Little Flower Baking by Christine Moore with Cecilia Leung (Prospect Park Books, 2016). Awful, awful, awful: overbaked, cakey, and tasteless. I find in impossible to believe that they sell brownies this bad in the eponymous cafe. The proportions don’t look terribly far from the average brownie, so this recipe would probably work out fine with the right cooking time and temperature.

Second, “Truffle Chocolate Cream Pie” from Joanne Chang’s Baking with Less Sugar (Chronicle Books, 2015). Unlike the last recipe, this one was a real winner, with the two distinct-textured rich chocolate fillings and only the sugar originally in the chocolate itself. I had trouble getting the chocolate mousse topping to whip up as much as I thought it should have (I have the same trouble with whipped ganache filling for cakes, and probably for the same reason — maybe it’s my expectation that’s wrong). Despite the reduced sugar, it’s by no means a low-cal dessert: with low-fat milk and Callebaut 60-40-NV for the chocolate it still clocks in at 517 kcal for 1/12 of a pie. Recommended. I have a few photos of this one:

The recipe uses Chang’s standard pâte brisée, with egg yolk and milk but no sugar.

This shows the pie with the bottom “truffle” filling (basically a very thick chocolate custard, with egg yolks, butter, and cream).

The pie is topped with a whipped chocolate mousse and then chocolate shavings are sprinkled on top of that.

This shows the cross-section of the pie, with the two fillings barely distinguishable (more by apparent density than by color). I have yet to master the proper technique for truly flaky all-butter pie crust.

I had a bunch of cream left over after the last recipe, and you know what that means: truffles! I turned once again to Alice Medrich’s Seriously Bitter Sweet (Artisan, 2013), but this time I used a different recipe, “Cold Creamy Truffles” (pp. 102ff). Unlike the “Ganache Truffles” I’ve made before, which involve making a ganache and allowing it to recrystallize at room temperature overnight, the centers of these truffles are formed from a refrigerator-cooled ganache and set in the freezer before coating with melted dark chocolate. This technique avoids the need for tempering the coating chocolate, at the cost of having to refrigerate the truffles until shortly before serving time. The texture of the cold, creamy centers contrasts nicely with the crisp coating shell. (True story: I didn’t intend to make this recipe; I was actually thinking I was going to make a different one, and didn’t notice until I had all of my mise done for this one, which I actually scaled up by 50% to use up all my excess cream.) No photos for this one. I used Valrhona Caraque for the centers and Valrhona Caraïbe for the coating, and ended up making them about 50% larger than the recipe called for. At this size, they came in at 111 kcal each, which is still pretty reasonable if you are comparing with a &approx;350-kcal cookie or brownie.

Fourth: I decided to go full-on German this Christmas, thanks to Luisa Weiss’s acclaimed Classic German Baking (Ten Speed Press, 2016). There wasn’t enough time to do Lebkuchen (leavened with potash, the dough takes months to ripen!) but plenty of time for Pfeffernüsse, Christbrot, and Mohntorte. I intend to do separate write-ups of all three of these recipes later on this month.

Finally, my birthday cake this year will be Mycroft’s Delight Revisited. This past weekend, I made the Gianduja-flavored Swiss meringue buttercream that represents my principal contribution to this recipe, but this time I used Valrhona Noisette Noir instead of the Noisette Lait I used last time, and I can report that it is much better in terms of color and flavor in this recipe. So if you want to try this yourself, get the dark. (There’s still plenty of butter in this frosting so you’ll get your butterfat one way or another.)

Posted in Food | Tagged , , , , , , , ,

World Cup Men’s Bobsled, November 9 at Lake Placid

This gallery contains 33 photos.

As I mentioned in an earlier post, there were two runs of men’s bobsled at the Lake Placid World Cup. The first was on Thursday, November 9, in the early evening, and was quite chilly. The second was the following … Continue reading

Gallery | Tagged , ,

World Cup Women’s Bobsled, November 9 at Lake Placid

This gallery contains 112 photos.

The BMW IBSF World Cup tour may have moved on from Lake Placid to Park City — and indeed from Park City to Whistler, B.C., by the time you see this — but I still have photos from Lake Placid … Continue reading

Gallery | Tagged , ,

World Cup Women’s Skeleton: medals ceremony

This gallery contains 18 photos.

Bet you didn’t think I could get this many blog posts about a single race, did you? I wasn’t expecting this one either, but you can thank some schoolchildren. On the official, published schedule, there is supposed to be a … Continue reading

Gallery | Tagged , , | 1 Comment

World Cup Women’s Skeleton run 2

This gallery contains 77 photos.

As seen in the previous post, I started out run 2 of the BMW IBSF World Cup women’s skeleton at the start of the track. After the second competitor left the start area, I walked down the track to get … Continue reading

Gallery | Tagged , ,

Slideshow: skeleton starts

This gallery contains 60 photos.

The regular gallery format isn’t great for showing sports action photos, but I wanted to show what a skeleton start looks like. These photos were taken during the second heat of the BMW IBSF World Cup women’s skeleton race at … Continue reading

Gallery | Tagged , ,

World Cup Women’s Skeleton run 1

This gallery contains 46 photos.

These photos were taken during the first heat of the women’s skeleton race at the BMW IBSF World Cup at Lake Placid on November 9, 2017. All except the first were taken from the spectator side of the finish area; … Continue reading

Gallery | Tagged , ,

Trip report: BMW IBSF World Cup at Lake Placid

Here’s a different kind of trip report. On Thursday, November 9, I attended the BMW IBSF Bob + Skeleton World Cup races at the Olympic Sports Center in Lake Placid, New York, site of the 1932 and 1980 Winter Olympics. The conditions were quite chilly on Thursday, but very good for racing on an artificial ice track — according to the TV commentary I watched after the event, it was an amazing turn-around from the previous weekend when late-season rain washed out the track. I bought tickets for both Thursday and Friday (an excellent deal at $28 total) but when Friday’s weather turned out windy with highs in the low teens Fahrenheit (around −8°C), I bailed on the second day of competition and drove back home. I did see the events I was most interested in seeing, the women’s skeleton and bobsled, as well as the first race of men’s two-man bob. (Because of the track conditions, day 2’s four-man bobsled race was swapped with next week’s scheduled two-man race in Park City, Utah, so there were two two-man races in Lake Placid and two four-man races in Park City. The first race on Friday morning, which I would have gone to see had the weather been a bit nicer, was men’s skeleton.)

There are only four sanctioned bobsled tracks in North America, at each of the last four Winter Olympic venues: Lake Placid and Park City in the U.S. and Calgary and Whistler in Canada. Lake Placid is the site of the U.S. Olympic Training Center, and also the only track within reasonable driving distance of my home outside Boston — but the Lake Placid Combined Track, which is the facility currently in use, was built after the 1980 Winter Olympics. (The old track, known as the “1932–1980 track”, is no longer used for competition.) It’s called the “Combined Track” because, unlike the previous track, it’s approved for all three of the Olympic sliding sports: bobsled, skeleton, and luge. (Skeleton was not on the Olympic program in 1980 when the old track was in use.) The Combined Track is kept open for as long as weather conditions allow, which gives many teams and the general public an opportunity to use it after the competitive season is over, meaning that many non-U.S. athletes have experience training at Lake Placid. (And I’m sorely tempted to sign up for the “Skeleton Experience”, which for a modest fee gives ordinary people the opportunity to slide a skeleton sled down the track from Start 4, which is about halfway up the track, when the track is not otherwise in use for training or competitions.)

The sliding sports are biggest in the German-speaking countries, Latvia, and Russia, and nearly all of the on-track advertising erected for the international broadcasts is from German and Russian advertisers, notably the title sponsor BMW, whose logo is seen on all bobsleds, helmets, and skeletoners’ race bibs. Teams also have a variety of national sponsors, and some individual athletes have their own sponsors, especially those from small countries. (The track announcer said that one British bobsleigh driver was told by her federation that she had to raise £30,000 to compete this year — which she raised from individual donors on GoFundMe!) The differences in national support also mean that some teams travel with few or no support personnel; when I post the photos of the bobsled events, you’ll see how some of the poorer teams have nobody at the race finish area to help the athletes pull their sleds off the track.

As I mentioned, there are three sliding sports: bobsled (“bobsleigh” in the Commonwealth countries), skeleton, and luge. Bobsled and skeleton are governed by the International Bobsleigh and Skeleton Federation (IBSF); luge has a different governing body, the Fédération Internationale de Luge (FIL), and thus a separate competition schedule; I won’t have much more to say about it, and don’t plan to go back to Lake Placid for the luge competition in December. The IBSF is one of a number of sports federations that in recent years has changed its name from French to English; while still based on Lausanne, Switzerland, it used to be called the Fédération Internationale de Bobsleigh et Tobogganing (FIBT), and the change is recent enough that there is still ephemera in circulation with the old name. The IBSF manages four international skeleton circuits, of which the World Cup is the top level, followed by the Inter-Continental Cup, and at the lowest level, the Europa Cup and the North American Cup. Bobsled has three circuits; as a team sport, it requires more competitors from each national program, so there is no intermediate level corresponding to the ICC.

Because 2018 is an Olympic year, the IBSF schedule is compressed, with all of the World Cup events completed before the PyeongChang Olympics begin in mid-February. Like most other top-level international winter sports circuits, the IBSF World Cup awards points to each competitor (skeleton athlete or bobsled driver) on the basis of their standing in each race, with an overall season points winner awarded a “crystal globe”. In a pre-Olympic season, the World Cup points are also used to determine which national teams get to compete at the Olympics, and how many competitors they can enter in each race; many national teams will also use the points total to determine which of their athletes will compete in the Olympics. (In regular years, the World Cup standings also determine the number of entries each national team gets in the following year’s World Cup season, and the position in the starting-order drawing for each individual athlete or driver.) This means that a number of national teams that normally can’t afford to bring their athletes to the North American stops on the World Cup are doing so this year in the hope of earning a place in the Olympics, so there were more competitors than usual in the first heat of each race. (Both bob and skeleton are normally two-heat races, with start order in heat 2 the reverse of the heat 1 times and the winner determined by total time; only the top 20 finishers in heat 1 make the cut for heat 2 and have the chance to earn World Cup points.)

Lake Placid is a really small town — technically, a “village” in New York law — and 1980 was the last time the Winter Olympics were ever (or will ever be) held in so small a community. While Lake Placid still has all of the sports facilities required to host an Olympics, it is far behind modern requirements for communications and transportation infrastructure, not to mention accommodations for athletes and media, and Adirondack Park is subject to strict limits on development. Lake Placid is a fine place for a competition at this level, however, even if it means that the athletes are staying in the same budget hotels as the spectators. (The Austrian team was in the Quality Inn where I stayed, and not only did I run into them at breakfast, but I saw their technicians working on their sleds in a room in the basement next to the hotel’s laundry. I was told by the hotel manager that the Austrian team comes every year for World Cup, but they also had two smaller teams staying this year.)

I’ll have more to say about the individual races when I post my photos of them. In the mean time, I’ll conclude this part of my report with some pictures of the Lake Placid Combined Track itself. According to the TV announcers, the track saw significant work over the summer of 2017, including improvements to the refrigeration system and construction of a new shelter at the finish area; it also appears that the facility roadways have been improved by adding a new paved road connecting the management offices, finish, and start areas; spectators desiring to climb to the start area do so via the old road, which they share with track maintenance workers and a few athletes warming up. (In general, I was quite surprised by how close athletes and spectators get at this event; you’ll see more of this in later posts.)

Facility offices
Track substructure
Refrigerant pipes
Refrigerant tank

In the photo below, looking down from the end of the run-out towards the finish area, you can see the new shelter (white metal structure). Beyond that is where the track makes a big loop (also painted white). Off in the distance is the visitor parking lot.
New shelter over finish area

Bobsleds and skeleton sleds come into the finish area with an enormous amount of momentum and require a very long run-out after the finish line to come to a stop. In the next two photos, you can see how far uphill the sleds run before stopping, after crossing the finish line at more than 100 km/h. In the second photo, I’m looking very far down towards the bobsled finish line — the skeleton finish is even lower down the hill, if I’m interpreting the TV pictures correctly. Sleds that run into trouble don’t generally don’t make it all the way up the run-out to the finish area; the building on the left beyond the parked cars is just above the bobsled finish.
Looking up the run-out
Looking back down toward the finish line

Now up by the start house, which is divided into “VIP” and “plebe” sides — the VIPs mix with the athletes and race officials on the right-hand side of the track, facing downward, and the rest of us are on the left side. (However, with the reduced crowds on frigid Friday, I saw from the TV broadcast that the Russian women were standing on the “plebe” side to cheer on their male counterparts who were racing that day, so I’m making it sound more classist than it really is.)
Start area

There are temporary video monitors set up at the start and finish areas so that competitors, coaches, race officials, and spectators can watch (but not hear) the TV broadcast. There’s a separate on-track commentary from a local announcer which is broadcast from speakers mounted all around the track; the local announcer also provides schedule information for competitors, spectators, and maintenance crews.
Athletes' side of start area

The darker ice at the far end isn’t actually darker, it’s just an effect of different light conditions heading into the first curve on the track.
Looking down the track toward curve 1
Old road past the start houses

A few pictures of the infrastructure brought in to support the broadcasts. IBSF handles all of the broadcast production for the World Cup events, with English-language commentary and “international” feeds made available to broadcasters; the English feed is streamed live on the IBSF’s YouTube account, where recorded broadcasts are also found, but final runs are geoblocked for US viewers to ensure that we see NBC’s advertising (with some of the competitors edited out for time).
Water for track touch-up
Very chilly tower camera position
Tower camera operator
Camera position at the end of the run-out
Remote camera encoder
Camera mounted over track
Microphone for track sound pickup

I’ll follow up as soon as I’m able with photos of the competitors from each of the events I attended.

Posted in sports, travel | Tagged , , ,

Trip report: Usenix Large Installation System Administration Conference 2017

Hi folks, it’s time for some work- and computing-related stuff. I just recently got back from the 2017 edition of the Usenix Association‘s annual system administration conference, LISA’17, which was held in San Francisco. I’ve gone to most LISA conferences since 1998 (when it was in Boston), but this is the first time that it’s actually been held in The City that I can recall, even among the ones I didn’t attend. It was very expensive, with membership discounts for registration having been withdrawn and the conference hotel getting a whopping $239 a night (at the discounted conference rate), not to mention the cost of airfare; my total cost (paid by my employer, thankfully) will be around $3,000 — something I could not possibly afford if I had to pay my own way — and that’s without taking any tutorials. That total only includes a couple of meals, because one major problem with this conference is the excess of “free” food, most of which is quite unhealthy, piled on buffets at every break, meal-time, and vendor BoF (Birds of a Feather) event. I gained five pounds in one week, and because of the time shift I found it nearly impossible to use the hotel gym the entire week. (I did spend nearly all of Tuesday walking around San Francisco, which helped a bit, except that I also went to some of the city’s famous bakeries and chocolatiers — so the conference isn’t entirely to blame for my weight gain.) Most of this excess food is paid for by the vendors, but I’d really have preferred if they could have found something else for the vendors to sponsor, and limited “refreshment” breaks to nothing more than coffee and soda — especially in a city like San Francisco where getting out of the hotel at mealtimes would have been much more rewarding than many of the other venue cities.

I’ve seen a number of people, including invited speaker Tanya Reilly and tutorial instructor Tom Limoncelli, post their comments about the program, so I’m going to do the same. In general, my impression is a bit more negative than theirs, and I was left wondering if we actually attended the conference. However, I’d also note that this conference has become more and more dominated by corporate IT and especially Web startups, whose organizations, problems, and space of feasible solutions (generally starting with “throw money and/or developers at it”) are nothing at all like mine. I’m seriously considering not attending the next LISA in Nashville, given the lack of take-home value this time around, whereas at the last few LISAs I’ve had difficulty deciding between two or three great sessions in nearly every time slot. I said as much in the official after-the-conference survey, but I’m honestly not sure how much the program committee cares at this point, or whether they even see R&E shops like us to be within the target audience of the conference. (Hey, I’m not saying that scalability isn’t cool — but nothing that I do will ever scale higher than n=4.) The weakness of the program was a surprise to me, given that I volunteered for the Content Recruitment Team and actually had a chance to double-blind review many of the submissions; after that process I was quite excited about the program and I didn’t bother to review what actually made it before registering.

So anyway, let’s go through the program session by session. The opening plenary was, unusually, divided into two 45-minute slots, and I unfortunately had to race to the bathroom shortly into the first slot so I didn’t really get to see either speaker. Wednesday’s second session was the only one of the entire conference where I seriously had trouble deciding which track to attend; I ended up going to the Mini-Tutorial “Automating System Data Analysis Using R”, taught by Robert Ballance, who covered the same ground in more detail in a half-day tutorial on Monday that I didn’t attend. I felt that the compressed 90-minute format was not a good match for this material; I’ve done some elementary data analysis in R already, and it took well into the second half before he really got into the things that would actually be useful for me at work. This is the sort of thing where I suspect an interactive “lab” or “workshop” format would be much better, with a “bring your own data” element that could actually be more helpful than the synthetic datasets used to present this tutorial. I’ll have to remember to check out the materials for the full-length tutorial to see if there are techniques or packages in there that he didn’t have time for in the mini-tutorial. (Apparently I missed a great talk by attending this tutorial: the commentary on Matt Provost’s “Never Events” talk makes me think I should have gone — waiting for the video to be posted.)

The first conference lunch was held inside the vendor expo, and is effectively paid for by the vendors as an inducement to get the conference attendees to stop by their booths. I noted the absence of a number of vendors this time around: publishers like No Starch and O’Reilly, service providers like PagerDuty and DataDog, and major hardware and software vendors like Dell and Splunk were not to be found. Of the 40 total exhibitors, a quarter were there solely for recruiting purposes, and nearly as many were non-profit organizations there either to raise awareness or (in the case of Princeton University) to recruit sysadmins to participate in a study.

After lunch on Wednesday I stuck with the “Talks II” track for the rest of the day. I thought Silvia Boutros’s talk “Working with DBAs in a DevOps World” was interesting enough despite having no take-home value for me (I’m actually the closest thing we have to a DBA, at least insofar as I wrote a number of internal applications that use a database and have strong opinions about how it should be done). The next talk was “Queueing Theory Practice: Performance Modeling for the Working Engineer” was also interesting without being especially useful, although the most counterintuitive theoretical result that Eben Freeman introduced was one that I already knew (tail latency goes to hell when utilization goes over 80% in a memoryless single-service model with random arrivals). There was some other good stuff in the talk, about balancing coordination overheads against parallelism. The third talk in the session, Stella Cotton on “Distributed Tracing: From Theory to Practice”, had no plausible applicability to anything I do and I tuned out fairly quickly.

The second PM session on Wednesday was divided into two talks. (This confused me a bit: why were some sessions three half-hour talks and some two 45-minute talks? I found in general that the half-hour talks were too compressed: the speakers spent too much time on the motivation and not nearly enough time on the actual results or engineering they were supposed to be describing, and didn’t leave any time for the Q&A that might have brought out more interesting applications.) I sat through Daniel Barker’s “Becoming a Plumber: Building Deployment Pipelines”, but found it uninteresting and remember little of it. Then Tanya Reilly came up and gave one of the three best talks of the whole conference, “Have You Tried Turning It Off and Turning It On Again?” — which was about engineering services to survive a disaster like a power outage that takes down a whole data center. She pointed out that in a “microservices” would, our “technology stack” can easily degenerate into a “technology pile” unless careful attention is paid to avoiding circular dependencies — especially non-obvious multi-node cycles in the dependency graph. This talk really spoke to me because a big part of my responsibilities at work is specifically maintaining those services that have to be up and working before anyone else’s stuff can run — network, time, authentication, directory, database, and other services that the rest of the infrastructure needs to have in order to start up or in order to be managed by other members of my team. As one of the few people who has been around for every facility power outage going back to 1997, it falls on me in particular to worry about this dependency graph, and what happens when we (for example) virtualize services that might be required to boot the virtualization environment. Her talk also reminded me of a Graydon Saunders blog post from last year that it turns out was actually more detailed in my memory than its actual text. (tl;dr: Given modern global supply chains, how many people does it actually take for the global economy to function? Saunders guesses at least a billion.)

After the last session of the day it was time for dinner — in the expo hall, because, well, “free”, and also there really isn’t sufficient time in the schedule to get together with some people and find a restaurant before the beginning of the BoF sessions, especially in a city like San Francisco where the good restaurants are not all in one place and tend to be busy enough that reservations are advisable. The BoF track was pretty uniformly disappointing this year, with nearly all of the potentially interesting BoFs scheduled for the same two time slots, Wednesday at 7 PM and Thursday at 8. In addition to the OpenZFS BoF, which I attended, there were two different monitoring BoFs — we got kicked out of the room just as the first one was getting past the usual introductions, so I went to the second one on Thursday evening as well. I honestly think having fewer and smaller rooms would have been an improvement, forcing the BoF organizers to spread their slots out more. Of course, there is also the usual problem of the “vendor BoFs” — with more free food and alcohol — sucking the life out of the actual shared-interest BoFs.

Thursday’s plenary was moved to the evening slot, rather than the usual first-thing-in-the-morning schedule. So I started out the morning in a talk by Nina Schiff of Facebook about “Disaggregating the Network”, which illustrated why things that work at Facebook are not really practical for the rest of us (see Corey Quinn’s talk below). It’s a nice idea, to commoditize top-of-rack switching in the data center and replace proprietary switch operating systems with the same Linux stack and configuration management used on the servers in those racks, but it’s not a practical exercise for those of us who don’t build a whole new data center to roll out a new service. After that talk I moved into the other session to learn about “Charliecloud: Unprivileged Containers for User-Defined Software Stacks in HPC”, because our environment, though not an HPC cluster at the scale of Los Alamos, shares a lot of the same use model and many of the same software release management problems as HPC centers have — except that in our case, nearly all of that work is done by graduate students who should be doing something else. Containers in general are supposed to abstract away a lot of those problems by allowing multiple independent (and immutable) software stacks to execute on the same machine. There’s still a pretty long way to go before our environment — with lots of GPUs requiring matching kernel drivers, not to mention storage on AFS — can really take advantage of this.

In the second morning session, I went to Trever Vaughan’s talk “Operational Compliance: From Requirements to Reality”, and I have it marked in my program as one that I thought was good, but even after reviewing the slides posted online I’m not sure why. This was followed by two people from NEC Labs presenting a rehash of a 2016 paper (from some other venue) about a tool they developed (and which is not available outside NEC) to automatically cluster log messages and use the results to generate log-parsing patterns that will maximize the amount of useful data extracted under conditions of limited processing power per log message. The tool actually generates multiple sets of parsing patterns which vary in their coverage of the input messages; the user must then choose their desired CPU vs. completeness trade-off. The third half-hour talk in this session was Dan O’Boyle of Stack Overflow explaining why you should give all your cryptographic keys to Google or Amazon rather than allowing your operations staff to touch them. (I suppose that way you can claim you didn’t know that the government was reading all your communications because the National Security Letter wasn’t addressed to you.)

Thursday’s lunch was again “free” in the vendor expo. After lunch, I went to Chris McEniry’s mini-tutorial on “The Ins-and-Outs of Networking in the Big Three Clouds”, which was an overview of how client networks work in AWS, Google Cloud, and Azure, with an emphasis on translations between the names used and capabilities implemented by each provider. (Most important lesson: don’t expect broadcast or multicast protocols for things like service discovery or mastership elections to work in cloud providers!) At the 3:30 PM break, I bailed completely, and did not attend the afternoon plenary (a panel on “Attracting and Retaining a Diverse Workforce”), although those who did thought it was both good and too short. Instead, I took the F-Market streetcar all the way to the “wharf” end, in a tacky, touristy area near the terminus of the Powell-Hyde cable-car line, Ghirardelli Square, and Fisherman’s Wharf. After seeing the long lines waiting to ride the cable car, I chose to talk back to the hotel instead — but following the steep hills of the Powell-Mason and California St. cable cars rather than the flat Embarcadero route of the streetcars. I returned in time for the conference reception (more “free” food that I shouldn’t have eaten), which was in the main atrium lobby area of the hotel this year. (In the distant past, they would rent a museum or some other interesting venue, but in recent years it’s been confined to a hotel ballroom, so I can’t really claim to be disappointed.) The reception was followed by more BoF sessions, of which I attended the second monitoring BoF, which went over its scheduled time by a bit, and the “DevOps Poetry Slam”.

I should say a bit more about monitoring/metrics BoFs: there has been one at every single LISA I’ve ever attended, and it’s quite clear that there is still a great deal of unhappiness with the solutions different organizations have adopted — whether it’s over resource demands, cost of third-party software and outsourced monitoring services, or the difficulty of building dashboards that actually collect all the business-relevant metrics, it’s clear that there’s still a lot of work to be done. We’re not especially happy with our setup either, but we have exactly zero budget in either money or personnel for the sort of solutions that might have a chance of making us happy. Some day, one of us will take it on, and then when that person leaves it will fall into disrepair.

Friday began with a talk by Martin Van Horenbeeck, “An Internet of Governments: How Policymakers Became Interested in ‘Cyber'”, which was exactly what it says on the tin. Following that was a talk by Evan Gilman and Doug Barth about work they had done when both were at PagerDuty. The title of their talk was “Clarifying Zero Trust: The Model, the Philosophy, the Ethos”, but I felt like that oversold the content somewhat. It was interesting to see the specific problem that they were trying to solve, how it related to their business requirement to operate across multiple public cloud providers, and their choice to use IPsec and packet filters to enforce security policies rather than the VPN offerings of each cloud provider (which would harm availability by creating single points of failure in each availability zone). I have it noted on my program that I didn’t much care for the presentation, but I’m not sure why. (The Zero Trust model is a very attractive one for us — indeed, many of the fundamental ideas where developed at MIT in the 1980s — but fails to meet many of our users’ needs or threat models.)

For Friday’s second morning session, I started out with Peter Lega’s talk “DevOps in Regulatory Spaces: It’s only 25% What You Thought It Was”, which was basically about how you convince compliance people in a regulated industry that modern software development methodologies really can address the risks that regulators are most concerned about — in Merck’s case, by integrating the compliance documentation and procedures with the development process, reducing the adversarial relationship between developers (“move fast and break things”) and regulators (“first do no harm”). I then switched sessions to see Corey Quinn’s talk, “‘Don’t You Know Who I Am?!’ The Danger of Celebrity in Tech”, which was excellent and exactly on-point to many of the concerns I’ve had with LISA programs in the past. Quinn made the point that most of the world is not Google, Facebook, Netflix, or Twitter, and that (a) it’s probably not a good idea for organizations that aren’t anything like those companies probably should think carefully before adopting the technology or the methodologies that those companies use. Quinn gave the example of a bank IT director watching a talk about Netflix’s “Simian Army” and wanting to take that approach back home — the methodology that’s appropriate for a company that doesn’t actually do anything important (sorry, Netflixers) may not be something you want people doing when people’s lives (or money) are at stake. Quinn also lit into the other side of the problem, people who work for those companies and use their resumes to shut down discussion of techniques that actually would be appropriate for someone who isn’t working at one of the top five Web origins. I strongly urge anyone to watch this talk when the video is released by Usenix in the coming weeks. (Yes, I got up and asked a question.)

After yet another “free” lunch, the final regular session of talks started with Ben Hartshorne of Honeycomb.io talking about various techniques for sampling application trace data, with a particular emphasis on using business requirements to determine appropriate sampling rates for different events (e.g., sampling important clients at a higher rate, but sampling successful transactions at a lower rate than errors). I didn’t find this talk especially interesting, but I wanted to get a good seat for David Blank-Edelman’s talk, “Where’s the Kaboom? There Was Supposed to Be an Earth-Shattering Kaboom!” I didn’t read the abstract of his talk ahead of time, so I honestly had no idea what it was about — it turned out to be about how lessons we can take from the demolition industry. David delivered the talk with his typical flair, but I did think it was less interesting (and entertaining) than several other talks on similar themes he has done at past LISAs, some of which rated a plenary slot.

After another refreshment break (finally, the “free” food was bags of mass-produced snack foods, which I had no trouble resisting), the conference closed with a plenary address by Jon Kuroda of UC Berkeley. Jon avers that this exact talk was rejected in 2013, but he decided to resubmit it this time around, and the PC was so excited by it that they asked him to speak for 90 minutes rather than just 45. He went into the history of several modern engineering fields, including space flight, nuclear energy, and (the specific focus of this talk) commercial aviation: along with computing, all date to the early post-WW2 period, but unlike computing, all have developed strong protocols for reasoning about and ultimately ensuring safety. (Medical technology and pharmaceuticals, too, date from this period or just slightly later, and have very strong safety cultures now.) Jon went through a number of well-known commercial aircraft accidents, and identified how the operator’s corporate safety culture made these incidents either better or worse than they might have been, exploring what lessons we should bring into our industry as computing systems are more and more involved in making decisions that can cause serious individual harm.

I returned home on Saturday morning, but it’s taken me until now to actually write this. Later today, I’ll be off to Lake Placid to watch some athletic young people in skinsuits hurtle down a mountain face-first while balanced atop a tea tray — which is how I had enough time this evening/morning to finally write this summary.

Posted in Computing, travel | Tagged ,