Thinking about pumpkin pie

Posted on 2014-09-25 by

I just got back from a short (two-day) trip to Toronto, which is my last travel before November. In Canada, they celebrate Thanksgiving in October, on our Columbus Day, which is only two and a half weeks away — so the well-prepared home cook will already be thinking about menus. We, on the other hand, still have a couple of months, but I at least will be traveling for a good chunk of November, so it behooves me to start thinking about Thanksgiving myself — and since I won’t be taking care of the main meal (thanks to my parents’ relocation to the other end of the country), I’ll have to get my holiday cooking fix in beforehand.

Which brings me to the question of pumpkin pie. As I noted in my recipe pointers pages, there are a lot of pumpkin pie recipes — even once you discount the standard back-of-the-can recipes that vary mostly in how over-spiced they are, and whether they use sweetened condensed milk or not. I went through my cookbooks to identify interesting pumpkin pie recipes, and I’m going to use my Columbus Day holiday to bake some of them, and bring them in to work as a combination treat/tasting adventure. (Because obviously I’m not going to eat four or more pumpkin pies on my own!) If it goes well, I may do more on a subsequent weekend. Here are the contenders:

  • Alton Brown, “Pumpkin Pie”, Good Eats 3: The Later Years, p. 370 (from “American Classics IX”, Good Eats episode 239) — one I’ve done before, this uses a gingersnap crust and is otherwise spiced only with nutmeg
  • Joanne Chang, “Super-Pumpkiny Pumpkin Pie”, Flour, p. 214 — standard spices; vanilla, heavy cream, brown sugar, and evaporated milk; pumpkin puree is reduced significantly as a flavor boost
  • Emily Elsen and Melissa Elsen, “Brown Butter Pumpkin Pie”, The Four & Twenty Blackbirds Pie Book, p. 168 — uses cream and milk; heavily sweetened with molasses and brown sugar; lightly spiced; vanilla, lemon juice, and carrot juice
  • Moosewood Collective, “Pecan Pumpkin Pie”, Moosewood Restaurant Book of Desserts, p. 57 — a layered pie, with a(n optionally maple-)pecan filling on the bottom, and a standard pumpkin custard on top; uses an unusually large 10-inch pie plate
  • David Page and Barbara Shinn, “Honey Pumpkin Pie”, Recipes from Home, p. 394 — heavy cream; honey, brown and white sugar; standard spices except no cloves; cornstarch
  • Richard Sax, “Best-Ever Pumpkin Pie”, Classic Home Desserts, p. 539 (recipe also appears in Brooke Dojny, The New England Cookbook) — black pepper in addition to the usual spices; milk and heavy cream; adds bourbon or rum; brown and white sugar, with an option for maple syrup
  • Richard Sax, “Sour Cream Pumpkin Chiffon Pie”, Classic Home Desserts, p. 541 — a layered chiffon (refrigerator) pie, with gelatin; vanilla; standard spices; pecans
  • Alice Waters, “Pumpkin Pie”, The Art of Simple food, p. 368 — standard spices; uses cream for the custard rather than evaporated or condensed milk; lightly sweetened with brown sugar
  • anon., “Golden Pumpkin Pie”, King Arthur Flour Whole Grain Baking, p. 471 — a special oat and whole-wheat pâte brisée; sweetened only with honey; dairy is half-and-half; standard spices plus optional dark rum

Surprisingly, I didn’t find anything especially interesting in the Test Kitchen files: they have one recipe from 2008 that uses candied yams in addition to the pumpkin, a recipe from very early on that’s very heavy and heavily spiced as well, and a Cook’s Country recipe that is similar in concept to the Moosewood recipe above, except that the pecan layer is on top rather than on the bottom. There’s also a no-bake recipe that uses a precooked custard filling (rather than chiffon) and a graham-cracker crust.

Stay tuned for the choices and the results.

Posted in Food | Tagged , , | 3 Comments

Not Mastering the Art of French Cooking

Posted on 2014-09-21 by

I recently bought a copy of Julia Child et al.‘s Mastering the Art of French Cooking (50th anniversary edition boxed set), the cookbook that made Julia Child a household name (at least among the aspirational middle class). I’ve been scanning it for potential recipe pointers, saying to myself, “Nope — that one’s got mushrooms. No arthropods, no veal, no duck, no cognac.” It feels slightly sacrilegious, but I’ll get over it. There’s still more than enough to try out, and I’m only halfway through volume 1. Obviously there’s no way to get around the wine, though, but at least it’s used in large enough quantities that it’s not a problem to get rid of the unused portion.

No recipe posts today as I’m traveling this week. I did try making a quadruple batch of pie dough from the Four & Twenty Blackbirds Pie Book (enough for two double-crust pies), which did not go well. The Elsens said that this was the minimum quantity they would consider making in a food processor. My food processor is not a small one, but the flour alone filled the work bowl to the rim. Their recipe calls for about three times the liquid as is required, as well. I think in the future when a recipe calls for a standard all-butter pâte brisée, I will ignore their recipes and go for one of Kenji Lopez-Alt’s (either the vodka one he did for Cook’s Illustrated, although that one calls for shortening which I don’t use, or the non-alcohol version he developed at SeriousEats. There should be plenty of opportunities coming up this fall — I’m planning on making half a dozen or so pies. Reminder: I’m still taking requests!

Posted in Administrivia, Food | Tagged ,

The Recipe Pointers Just Keep On Coming

This batch finishes off all of the cookbooks that I currently have on hand — at least those that I thought worth extracting the interesting bits from. There are still plenty of other cookbooks that I use more as reference works (including four editions of Joy, but I already have so many recipes to choose from, I’m not sure if I’ll ever need to get back to them. I’m still taking requests for future recipe writeups — please leave comments on that page if there’s anything you’d like to see. The latest set:

Aside | Posted on 2014-09-17 by | Tagged

Chocolate tasting results for (extra dark) championship week 1

Posted on 2014-09-16 by

We held the first of two championship tastings Monday, concentrating on the 70% end of our “extra dark” range. As I wrote on our wiki:

In order to keep the championship round at two weeks, we ended up with eight chocolates per tasting, and some of the tasters noted a bit of fatigue by the time they reached the eighth chocolate — all of which were fairly similar, but the panel was still fairly clear on the differences, and the result was quite certain. (However, it may be interesting to compare panelists comments with previous tastings where the same products were included!) With four first-place votes, the winner was Equal Exchange Very Dark Chocolate (71%). In second place, with three first-place votes, was Theo Pure 70% Dark Chocolate, which I had expected to win. Rounding out the field, Jelina Noir (72%) came in third, followed by Whole Foods Costa Rican Dark Chocolate 71% in fourth, Vivani Dark Chocolate (71%) and Nói Síríus Traditional Icelandic Chocolate 70% tied for fifth, Divine Intensely Rich Dark Chocolate (70%) in seventh, and Dick Taylor 72% Madagascar (Sambirano) in last place.

More to come in a couple weeks when we do the 85% end of the spectrum.

Quote | Posted on 2014-09-16 by | Tagged , ,

Now taking recipe writeup requests…

Hey, folks! I know there are a bunch of you following me who seem to be interested in my recipe posts (because you click the “like” button). I’m currently working out a schedule for things I want to make for the fall and winter, and there are some empty spaces. Go have a look at my Recipe Pointers pages and let me know in comments here if there’s anything you’d be interested in seeing. (Not responsible for lost, stolen, or damaged game pieces. The decisions of the judges are final. Void where prohibited.)

I’ll be sticking primarily to things that are (a) practical for a single guy to make, (b) under 500 calories per serving, and (c) don’t require exotic ingredients that I’ll be stuck with for a long time. Some exceptions may be made for special occasions, and at least two more cookbooks’ worth of Recipe Pointers are in the queue. I have seven weekends open for the rest of the year, and will probably fit five recipes in, possibly more depending on the sweet/savory balance, cost, and overall level of effort required. The first open weekend is October 4–5, so you’ve got some time to get your requests in!

Aside | Posted on 2014-09-15 by | Tagged , | 2 Comments

My recipe, for once: whole wheat sandwich bread

Posted on 2014-09-14 by

This is a slightly revised and updated version of my Wheat sandwich bread, Mk. 4 recipe, developed a number of years ago based on a number of sources (see the link) — now with pictures. As I’m presenting it here, it’s a half-whole-wheat bread, but I’ve had good results using all (white) whole wheat as described in the original recipe; if you want a white bread recipe instead, there are better choices (including Diane Duane’s Tessinerbrot from last January and Joanne Chang’s brioche from July). The procedure here owes a lot to Alton Brown; the specialty ingredients are all from King Arthur Flour.

Note that this recipe uses white whole wheat flour. This is a specific type of whole-wheat flour which is made from hard winter wheat. I’m not sure how easily available this is in other countries, and I’ve never tried it with standard (soft, red) whole wheat. I invite comments from anyone who decides to try this with other flours and report back.

Sponge

This bread starts with a sponge:

Sponge ingredients
7½ oz 210 g white whole wheat flour
1 oz 30 g butter
13 oz 370 mL water, 115°F (45°C)
2 tsp 10 g SAF instant yeast
2 tsp 10 g diastatic malt powder
3 tbl 30 g vital wheat gluten

Procedure:

  1. Add the butter to the warm water and let it melt.
  2. Sift and combine all other ingredients in a large bowl.
  3. Add the water and melted butter to the bowl and stir until smooth.
  4. Cover the bowl with a sheet of plastic film and let stand at room temperature for half an hour.
  5. Place the covered bowl in the refrigerator for at least 12 hours.

Despite the color, white whole wheat still includes all of the bran of the wheat kernels. When milled into flour, wheat bran turns into sharp fragments which can cut the gluten strands as they develop during the kneading process; this recipe takes two measures to avoid this: first, adding additional gluten (and using high-protein flour in the first place) ensures that enough gluten is formed, and second, allowing the sponge to ferment for a long period softens the bran fragments so they are less able to interrupt the gluten structure. The added gluten is probably not necessary in this version of the recipe, because of the bread flour used below, but is important to a good texture and rise as the ratio of whole-wheat to white flour increases. If you don’t have any, substitute an equal quantity of high-protein bread flour — the result won’t be quite as good but still serviceable.

Note that this bread includes no sugar other than what is found in the malt powder. Diastatic malt powder contains an enzyme which helps to break down some of the starch in the flour. (Non-diastatic malt lacks this enzyme, and should be saved for homemade bagels.)

Loaf

Ingredients for one 9×5 loaf
​ sponge recipe, above
7½ oz 210 g high-protein bread flour
1 oz 30 g Baker’s Special nonfat dry milk powder
1½ tsp 5 g salt
​ egg wash
​ baking spray

Procedure:

  1. Bring sponge to room temperature; this may take a couple of hours.
  2. Preheat your oven to its lowest temperature and turn it off. Alternatively, put a kettle of water on to boil.
  3. Sift together the dry ingredients.
  4. Using a stand mixer with a dough hook, mix together the sponge and the dry ingredients until the dough just comes together. It will look somewhat shaggy. Wait for ten minutes before proceeding.
  5. Still in the stand mixer, knead the dough for ten minutes or so. The dough should be somewhat moist, but if it is too sticky to handle, add a tablespoon of flour and knead for another minute. When the dough is ready, a small ball of dough can be flattened and stretched into a thin membrane (“windowpaning”) without breaking.
  6. Turn the dough out onto a lightly floured surface and form a tight ball by tucking loose ends underneath and rolling the ball around on the work surface between your hands.
  7. Lubricate a large bowl with baking spray and place the dough ball in the bowl. Cover with a sheet of plastic film and place in the warm oven or another warm place. (If you did not preheat the oven, pour some boiling water into a pan and put it in the oven with the bowl of dough.) Let the dough rise for about an hour.
  8. Turn the risen dough out onto a floured work surface.
  9. Using your knuckles, flatten the dough ball into a rectangle as shown in the photo below, taking care to pop any large bubbles of gas.
  10. Fold the flattened dough over in thirds, like a trifold wallet or a business letter. Using a pastry brush to remove any excess flour from the surface before each fold.
  11. Repeat the last two steps twice over, turning the dough 90 degrees each time.
  12. Form the dough into a log and pinch the seams shut.
  13. Spray a standard 9″x5″ loaf pan with baking spray and place the dough inside. Cover with plastic film and return to the oven. If the oven is no longer warm, add more boiling water.
  14. Let the dough proof until it crowns the top of the loaf pan by about an inch (25 mm) (see photo below); this may take anywhere from 45 to 90 minutes depending on the how warm it is.
  15. Remove the loaf and any remaining container of water from the oven and put a rack in the lower middle position. Preheat to 375°F (190°C).
  16. Using a pastry brush, apply egg wash to the top of the loaf. If you prefer a crisper crust, use melted butter instead of egg wash. You can also apply seeds, rolled oats, or salt to the crust at this time.
  17. Using a sharp slicer or serrated bread knife, cut a slit in the center of the loaf, about 1/8″ (3 mm) deep.
  18. Bake the loaf for about 35 minutes, or until it reaches an internal temperature of 190°F (90°C) on an instant-read thermometer.
  19. Remove the bread from the oven and let cool in the pan for 15 minutes, then turn it out onto a cooling rack and allow to cool for another hour before slicing.

I used Sir Lancelot bread flour, which is higher in protein even than King Arthur’s regular bread flour, and almost certainly obviates the need for the extra gluten in the sponge. The yield obviously depends on the thickness of the slice: from a 9″ pan you should be able to get 18 ½” slices if you’re a better hand with the bread knife than I am; I got 16 usable slices. (Would that I could just bring my loaf into the supermarket bakery and use their slicer!) If you like thicker slices (and you probably do), 12 is a reasonable target. The nutrition data presented below is based on 17 slices.

Pictures

The sponge looks like this, prior to fermentation:
Sponge, prior to fermentation

After fermenting for 16 hours, it looks like some biological activity has taken place (and smells slightly of alcohol):
Sponge, 16 hours later

After mixing the dough, it still looks quite shaggy. We pause at this point for ten minutes to allow the additional flour to hydrate:
Dough, just mixed

The dough has been kneaded, and in this case I had to add a couple extra tablespoons of flour. It’s easy to form into a ball, and into the proofing bowl it goes:
Dough, after kneading

After proofing, the dough is nice and warm and poofy:
Dough, after first proof

We begin the wallet fold procedure by flattening the dough into an oblong (or what would be an oblong if I were a bit better baker):
Flattened dough ball

After making the first of nine folds. Note that the back side of the bread has picked up a lot of flour from the work surface; it’s important to brush off any excess flour, or it will turn into a chewy seam inside the loaf.
First fold

The dough is folded and turned and folded and turned and eventually you get this log shape and pinch the seams shut. Into a lubricated loaf pan it goes for second rise (bench proofing):
Dough ready for bench proof

The dough is done proofing when it crowns the pan by about an inch. If I had done a better job, this would be more even, rather than having a short end and a tall end — that’s indicative of the dough having started out uneven.
Fully proofed dough, showing extent of rise

Then an egg wash is applied and a release cut is made down the center of the loaf, and it’s ready for baking:
Dough ready for baking

At least in my oven, the baking time is a pretty consistent 35 minutes. When it’s done, the top crust is this lovely golden brown, but the sides and bottom are still pale, just like regular sandwich bread.
Fully baked loaf

Because I’m a single guy on a diet and only eat bread in sandwiches, I need to slice it as soon as it’s cool so that I can freeze most of the loaf and thaw only what I need — since this bread does not contain any preservatives (or even all that much salt) it is an ideal culture medium for mold, and nobody wants to eat moldy bread. Unfortunately, I’m a lousy hand with the slicing knife, even given the assistance of a ruler or slicing guide, but I managed to get 17 slices out of this — one of which I immediately toasted up, buttered, and ate, because FRESH BREAD:
Messy job of slicing

Nutrition

The figures below do not include the egg wash, which makes a negligible contribution to nutrition.

Nutrition Facts
Serving size: 1 slice
Servings per container: 16-18
Amount per serving
Calories 114 Calories from fat 18
% Daily Value
Total Fat 2g 3%
 Saturated Fat 1g 5%
 Monounsaturated Fat 0g ​
 Trans Fat 0g ​
Cholesterol 5mg 2%
Sodium 164mg 7%
Potassium 0mg ​
Total Carbohydrate 18g 6%
 Dietary fiber 2g 8%
 Sugars 1g ​
Proteins 5g 10%
Vitamin A 1%
Vitamin C 0%
Calcium 3%
Iron 5%
Posted in Food | Tagged , , ,

Chocolate tasting results for week 7

Posted on 2014-09-11 by

I was expecting week 7, chock full of artisanal chocolates from boutique American makers, to be the culmination of our two months of preliminary heats. Boy was I wrong:

This tasting was a bit of a disaster for the artisanal chocolate makers — in fact, our two reference chocolates, Green & Black’s 85% and Equal Exchange Very Dark Chocolate tied for first place, each receiving three first-place votes and two second-place votes. Excluding those two, the top finisher was Dick Taylor 72% Madagascar, despite the sourness that all tasters noted, with one first-place vote, followed by the old bar of Rogue Silvestre 75% with two second-place votes. Rogue Balao 75% also received one first-place vote.

I’m not quite sure what happened — was it my tasting panel, or the protocol, or even the chocolates themselves? I mean, if you’re going to pay 24 cents a gram, you’d hope that at least someone would think it was the best they’d ever had! One theory of mine, though, is that it’s just the format of these bars:

I suspect that the size of the pieces may have significantly affected the reception of these artisanal chocolates. Unlike the two supermarket bars included as references, the five on-theme products are cast in very thin bars, which makes them less conducive to the tasting protocol we have been using. Tasters received approximately 50% larger pieces of the Green & Black’s 85% than they did of the Dick Taylor bars (at 56 grams each, the smallest in our entire series of tastings), and the Equal Exchange bar came in even larger pieces. I suspect that all of these products would have done better as somewhat thicker bars with portions between 4 and 6 grams.

(See the full details on our wiki.)

The championship round will run for two weeks, although the schedule is not determined yet. I tried to organize the chocolate lineup for each week to minimize the apples-to-oranges effects, so the first week will be:

  • Theo Pure 70% Dark Chocolate (runner-up, week 1)
  • Vivani Dark Chocolate (71%, winner, week 2)
  • Nói Síríus Traditional Icelandic Chocolate 70% (runner-up, week 3)
  • Whole Foods Costa Rican Dark Chocolate 71% (winner, week 4)
  • Divine Intensely Rich Dark Chocolate (70%, Ghana, winner [tie], week 5)
  • Jelina Noir (72%, mixed origin, winner, week 6)
  • Equal Exchange Very Dark Chocolate (71%, winner [tie], week 7)
  • Dick Taylor 72% Madagascar (Sambirano) (top on-theme product, week 7)

Check back here next week for the results!

Quote | Posted on 2014-09-11 by | Tagged , ,

Other people’s recipes: Joanne Chang’s honey-cinnamon ice cream & Cook’s Illustrated’s rice and lentils

Posted on 2014-09-08 by

It’s a two-for-one today: first the honey-cinnamon ice cream from Joanne Chang’s first cookbook, Flour (p. 260), and then from this month’s Cook’s Illustrated, it’s “Rice and Lentils with Crispy Onions” (pp. 8–9).

Let’s start with the ice cream, which I made first but tried second (since it wasn’t finished freezing last night). Chang describes how a honey-vanilla ice cream turned into honey-cinnamon ice cream when a supplier failed to deliver her vanilla order on time to Rialto, the restaurant which she had her first job as a pastry chef. Since all the desserts were accompanied by ice cream, she had to substitute, and decided to use cinnamon. Other than that one substitution, it’s a fairly standard French-style ice cream, made with half cream and half milk plus lots of egg yolks to form a custard base. The primary sweetener is honey — I used McLure’s orange blossom honey — with a small amount of sugar added, and the cinnamon gives it the sort of understated spiciness for which Chang is known. The custard base is wonderfully smooth after mellowing for 18 hours, and remains silky smooth even after a day’s hardening in the back of the freezer. Behold:
A serving of ice cream in a dessert cup

I’m actually interested in turning this back into a honey-vanilla ice cream, and possibly experimenting with different types of honey to see what sort of flavors are expressed. With a full ¾ cup of honey in the custard, the honey flavor definitely carries through to at least some extent. But I have lots of other things I want to try before I get back to this one! (Ingredient notes: I used High Lawn Farm low-fat milk and Sky Top Farms unhomogenized heavy cream for the dairy, and Country Hen extra-large organic eggs for the egg yolks; the nutrition data below does not take this into account.)

That brings us to the Test Kitchen’s “Rice and Lentils with Crispy Onions” with a tangy yogurt sauce — also known as mujaddara or megadarra — a complete-meal vegetarian dish from the Middle East. The Achilles’ Heel of this dish is definitely the crispy onions: although I thought I followed the directions fairly closely, I still ended up with blackened, nearly burnt onions which absorbed nearly half (¾ cup!) of the frying oil. If I made this dish again — and I might well do — I would probably substitute caramelized onions, which add far less fat to the dish and are less finicky. (Or I could just buy crispy onions from the store, but that seems like cheating, particularly to get the equivalent of two pounds of fresh onions!) Nonetheless, I found myself irresistibly drawn to snacking on the onions, to the point where there wasn’t enough left to sprinkle on top as directed, so I just stirred all the onions in.

I made the onions a day ahead, and by the time I actually needed them, they had lost a great deal of their crispness, so I put them in the oven on a parchment sheet at 250°F to drive off some of the remaining water — would that I could do that with the fat! — with reasonably success. Of course, this just made them even more snackable.
Crispy onions, slightly burnt, on a sheet of parchment for reheating

As prepared, the recipe made a little over three pounds — which made Cook’s Illustrated‘s “serves 4 to 6” seem a little silly: as a side dish, it probably serves 8 to 10, and as a vegetarian main dish, 6 to 8. To serve 4, you’d be talking 12 ounces of mujaddara per 900-calorie serving. I’ve used 6 servings for the nutrition computation below (which, note well, also includes an entire cup of canola oil!), but for dinner tonight I had a six-ounce portion (¾ serving) alongside four ounces of store-bought chicken salad. (Portion weights do not include yogurt sauce, measured separately, but the calorie counts do.) This is Sunday’s dinner plate:
A serving of the finished rice-and-lentil dish, with yogurt sauce, on a dinner plate
If I had been less tired, I would have steamed some spinach to add some more green to the plate.

Nutrition

For the ice cream:

Nutrition Facts
Serving size: ½ cup
Servings per container: about 10
Amount per serving
Calories 308 Calories from fat 190
% Daily Value
Total Fat 20g 31%
 Saturated Fat 15g 73%
 Monounsaturated Fat 2g ​
 Polyunsaturated Fat 1g ​
 Trans Fat 0g ​
Cholesterol 235mg 78%
Sodium 93mg 4%
Potassium 28mg 1%
Total Carbohydrate 24g 8%
 Dietary fiber 0g ​
 Sugars 24g ​
Proteins 4g 8%
Vitamin A 23%
Vitamin C 0%
Calcium 41%
Iron 3%

For the mujaddara:

Nutrition Facts
Serving size: 8 ounces (excluding yogurt sauce)
Servings per container: about 6
Amount per serving
Calories 589 Calories from fat 351
% Daily Value
Total Fat 39g 61%
 Saturated Fat 4g 19%
 Monounsaturated Fat 22g ​
 Polyunsaturated Fat 11g ​
 Trans Fat 0g ​
Cholesterol 6mg 2%
Sodium 1015mg 42%
Potassium 719mg 21%
Total Carbohydrate 47g 16%
 Dietary fiber 15g 59%
 Sugars 11g ​
Proteins 15g 29%
Vitamin A 4%
Vitamin C 33%
Calcium 13%
Iron 24%
Posted in Food | Tagged , , , ,

The network nightmare that ate my week

Posted on 2014-09-05 by

A few hours ago, I sent the following “dear colleagues” email (lightly edited to remove some private details) to all my users at work:

This has been a very trying week. For those of you whose work was disrupted this week by unplanned network outages, my deepest apologies. I am writing to you to explain what we know so far about the cause of these problems, what we have done to resolve them, and
what actions still remain to be taken.

Over the course of the summer, we have been suffering from a variety of difficult-to-identify network problems. Our network switches have been observed using far more CPU than has historically been the case, we have had a variety of packet storms that appear to have been caused by forwarding loops despite the fact that we run a protocol designed to prevent such loops from taking place, and we have had a variety of unexplained switch crashes.

First issue

Starting very early Wednesday morning, the switch that serves the server room 32-399 began to crash unexpectedly, and in a way that (contrary to design) requires a human to physically power-cycle it in order to restore service. This switch crash affected a variety of research groups’ systems, as well as central AFS servers and the newest set of OpenStack hypervisors. We initially thought that this was a consequence of the power failure we experienced on Tuesday evening, and the vendor (Juniper Networks) suggested a reinstallation procedure for the member nodes in this switch to recover from possible corruption of the flash media from which the switch member nodes boot. Unfortunately, this did not resolve the problem, although it did put off the recurrence for a few hours.

On Thursday morning, an engineer from Juniper came to Stata to help us determine the cause of that crash and come up with a resolution. He was assisted by five other Juniper engineers from their Advanced Technical Assistance Center via teleconference. It took the rest of the day Thursday to come up with an action plan to resolve the issue (although still without any identifiable root cause), because the failing switch nodes were unresponsive on their console as well as the network, and none of the more obvious fixes we tried had any effect. (The switch logs, on reboot, stopped just before the crash.) Eventually we decided to upgrade this one switch to the latest version of Juniper’s firmware for this platform (EX4200), and simultaneously to make several configuration changes to reduce the use of code paths which are less well tested. This, in combination with what I’m about to explain next, appears to have resolved the issues with this particular switch. I will be monitoring this switch over the weekend to make sure it remains stable. Juniper has dispatched replacement hardware for all three member nodes of this switch, in case it proves necessary, but at this point we believe that the problem was caused by software and not a hardware failure (and thus, the association with the power outage was a red herring).

Second issue

Over the summer we have been experiencing a variety of issues with our core switches. We continued to experience these issues even after upgrading the firmware as recommended by Juniper, which we did (on an emergency, unscheduled basis) two Mondays ago. The most serious issue is that there appear to be bridge loops occasionally being created which saturate the CPU on the core switches, thereby preventing them from running necessary protocol processing. I believe (but don’t have official confirmation yet) that this can result in a priority inversion relative to flooding broadcast and multicast traffic through the network, such that the process that implements the Spanning-Tree Protocol (which constructs a loop-free topology by deactivating redundant links) is unable to run, which causes all of the access switches to think that their uplink is no longer connected to a spanning-tree bridge, which causes Sorceror’s Apprentice packet amplification as multicasts are then forwarded down both core links simultaneously — which only adds to the overload on the core switches. Sometimes the core switches would recover from this condition without human intervention, but several times they did not, and I was forced to physically power-cycle one of them to break the loop.

We are trying to develop some configuration changes that will make this less likely in the future, by changing some of the assumptions in the Spanning-Tree Protocol to limit the possibility of bridge loops forming in the first place. This work has yet to be done, so in the mean time, we have made some changes to our network to reduce the CPU load on the core switches and make it less likely that the spanning-tree process can get starved in the first place.

The principal change that I have made in this regard is to disable IPv6 on most CSAIL networks. I have come to the conclusion that so much in IPv6 design and implementation has been botched by protocol designers and vendors (both ours and others) that it is simply unsafe to run IPv6 on a production network except in very limited geographical circumstances and with very tight central administration of hosts.

Technical details

The fundamental design problem with IPv6 is related to how it functions over shared layer-2 networks like Ethernet. In IPv4, there is a separate protocol (ARP) which is used by hosts to find the MAC address of other stations on the network. To do this, a host that wants to send a packet sends a layer-2 broadcast frame asking “who has IP address 128.30.xxx.yyy?” The intended recipient, if it’s on the network, sends a reply that says “Hey, I’m 128.30.xxx.yyy, and my MAC address (in case you missed it) is 01:23:45:67:89:ab!” If the intended recipient isn’t on the network, the sender keeps on broadcasting periodically until it either gets a response or gives up.

In IPv6, this is not a separate protocol; it’s called “neighbor discovery”, and uses ICMPv6 packets. Because IPv6 does not have broadcasts, the soliciation packets must be sent to a multicast address. But rather than use the standard “all hosts” multicast address, the Neighbor Discovery Protocol instead specifies that every host must join another multicast group, one of 4 billion distinct “solicited node” multicast groups chosen as a function of the system’s IPv6 address, and Neighbor Discovery packets are sent to this group rather than being broadcast. (This means that the vast majority of all IPv6 multicast groups in use anywhere in the universe have a single member.)

In theory, that should be no worse than sending an ARP broadcast, but in practice it is much worse, because IPv6 systems must also implement the Multicast Listener Discovery protocol, by which all stations on the network report, when requested, all of the multicast groups they are members of — and they send these reports to the group, which means flooding those reports throughout the network, because the network switches have no way of knowing Ethernet multicast addresses are desired on which ports. Furthermore, MLD packets are required by the protocol to be transmitted with a “router alert” option, which causes routers to bounce these packets from hardware forwarding into software, meaning that while the flooding of ARP broadcasts can be fast-pathed and are usually implemented in silicon, MLD listener report multicasts must be slow-pathed — and since IPv6 is still not implemented on very large layer-2 networks like a campus network or our building, MLD processing is generally poorly tested and not optimized by network vendors. Our core switches send MLD all-groups queries every 150 seconds, and direct end hosts to splay their responses over a 10-second interval, both as recommended in the MLD RFC.

In theory, we could implement “MLD snooping” across our network to reduce the overhead of flooding MLD listener report packets all over. However, this is very new and raw code, at least in Juniper’s operating system, and not well exercised anywhere that I’m aware of. Even if we did this (and we tried), that would require at least two entries in the multicast zone of every switch’s TCAM (hardware forwarding table) for every host on every IPv6-enabled network in CSAIL, just to handle the Neighbor Discovery multicast groups — and in our switches, the entire TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL, but for one major issue: “privacy” addresses.

IPv6 “privacy” addresses are an incredible botch added to IPv6 a few years ago to simulate the “good old days” of dial-up Internet where machines changed their IP (v4) addresses all the time. In normal IPv6 usage, every host generates an IPv6 address by listening for broadcasts from routers telling them what network prefix (the first 64 bits of the IPv6 address) to use, and appending to that a fixed function of their network interface card’s MAC address. In “privacy” addresses, the host simply generates a 48-bit pseudorandom number, pretends that it’s a MAC address, and applies the same function. The host will also maintain a “traditional” IPv6 address, used only for incoming connections — the random address is used for outgoing packets. What’s worse, the random address is changed regularly, typically daily, but the old random addresses are kept around for a fairly long time, on the order of a week, in case some other host out there on the network wants to “call back” after a new address has been generated. Thus, a typical machine — say, an Ubuntu 14.04 workstation with the default configuration — will end up claiming eight IPv6 addresses at any one time. That means nine IPv6 multicast groups, which means that those 3,000 TCAM entries can be exhausted by as few as 333 Ubuntu workstations.

This is generally not an issue for portable machines like laptops, because they forget all their old random addresses whenever their IPv6 prefix changes (which happens whenever they are disconnected from one IPv6-capable network and connected to another) but it is a very serious issue for workstations and, of course, servers, that are connected full-time to a single network. (The random addresses are also very problematic for me as a network administrator, because they mean that I am unable to trace back problem machines to their owners if they have been removed even briefly from the network in the interim.)

I used Ubuntu as an example, but it is hardly the worst offender. We have seen Windows machines with more than 300 IPv6 addresses — which, recall, means that every 150 seconds they will be transmitting 30 multicast packets per second which have to be flooded through the network. That problem was caused by a broken Intel NIC driver — Windows attempts to offload IPv6 processing to the NIC while the workstation is in standby mode, to support wake-on-LAN over IPv6 — but we had to get an updated driver from Intel to fix the problem (Windows Update was still distributing the broken driver). We’ve seen other machines that merely flood the network with copies of a single MLD listener report — sometimes hundreds of packets in less than a second. I first learned about this back in July when we started to observe CPU overload on the switches that serve our area; it turned out to be one of the public Windows machines outside the TIG offices, but once we knew what to look for, we saw many other machines doing the same thing. (Pretty much any new Dell workstation running Windows has this bug, unless the Intel Ethernet driver has been updated.)

For this reason, we will not be making IPv6 more broadly available again until we have a reliable way of ensuring that “privacy” addresses are disabled on all clients. (They are already disabled in CSAIL Ubuntu, but there are of course many other clients that we can’t control centrally.) We will probably move towards not supporting IPv6 “stateless” autoconfiguration at all, and rely on DHCPv6 to assign stable, traceable addresses to all IPv6 clients, but we’re a ways away from being able to implement that at this time.

I finally got home last night around 1 AM, after being awake for nearly 40 hours. If I never see another dawn in the office again, it will be too soon.

UPDATE (2014-09-06): As Stéphane Bortzmeyer was the first to point out, RFC 7217 addresses all of my issues with “privacy” addresses. Let implementation come soon!

CORRECTION (2014-09-06): There are actually only 16,777,216 “solicited node” multicast groups, not 4 billion (as I posted here). I originally said 65,536 in my email, but realized that was wrong, and miscorrected it when I posted here.

Posted in Computing | Tagged , , , , | 64 Comments

From the pages of Stupid Software Design Quarterly…

Posted on 2014-09-01 by

About a month ago, printing stopped working on my machine. I had a clunky workaround, so I didn’t worry too much about it. I had just upgraded the cups package, which manages printing, on the print server, and it seemed to think the printer wasn’t “available”.

Let’s back up a bit. My print server (which is also my desktop) runs FreeBSD. The standard solution for printing on FreeBSD these days is to use cups (originally a commercial open-source product from an independent software house, now owned by Apple). I bought a Xerox Phaser 6140 really cheap (under $150 including shipping), but adding an Ethernet port would have doubled the price (not to mention wasting an IP address for something that shouldn’t be public anyway), so I had to use the USB interface. Fine, I said, and got a long USB cable to plug it in; it showed up as /dev/ulpt0, and I configured cups to print to that device, and all was well.

(Well, actually, not all was well — I had to configure some hacks in /etc/devfs.conf to allow cupsd(8) to access the printer device, and I also ended up creating another hack for devd(8) to automatically pause and restart printing when the printer was powered on and off, since laser printers generate a lot of heat and are energy hogs.)

When I upgraded the cups package and printing stopped working, I discovered quickly that I had no difficulty printing directly to /dev/ulpt0 — I just had to remember the usual bit about sending a control-D to tell the printer’s PostScript job manager that the file I had just transmitted was done. So clearly whatever had broken printing wasn’t in the operating system, but some new misbehavior on the part of cupsd. After digging around (and finding lots of totally unhelpful Linux forum sites that talked about “blacklisting modules” and other crap that FreeBSD users don’t have to put up with), I was able to find out the name of the program that cupsd outsources USB printing to: /usr/local/libexec/cups/backend/usb, which allowed me to find it in the process table, which allowed me to ktrace(1) it, which told me why I was tearing my hair out:

  5950 usb      CALL  open(0x80082f3f1,0,0)
  5950 usb      NAMI  "/dev/usbctl"
  5950 usb      RET   open 7
  5950 usb      CALL  ioctl(0x7,USB_READ_DIR,0x7fffffff9950)
  5950 usb      RET   ioctl 0
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen0.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen1.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen2.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen3.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen4.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen5.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen6.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen6.2"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen3.2"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  ioctl(0x7,USB_READ_DIR,0x7fffffff9950)
  5950 usb      RET   ioctl 0
  5950 usb      CALL  close(0x7)
  5950 usb      RET   close 0
  5950 usb      CALL  write(0x2,0x7fffffff9510,0x20)
  5950 usb      GIO   fd 2 wrote 32 bytes
       "DEBUG: libusb_get_device_list=0
       "
  5950 usb      RET   write 32/0x20
  5950 usb      CALL  close(0x5)
  5950 usb      RET   close 0
  5950 usb      CALL  close(0x6)
  5950 usb      RET   close 0
  5950 usb      CALL  write(0x2,0x7fffffff7a00,0x2f)
  5950 usb      GIO   fd 2 wrote 47 bytes
       "INFO: Waiting for printer to become available.
       "
  5950 usb      RET   write 47/0x2f

What’s it trying to do there? Why did it try to open a whole bunch of device special files it had no business looking at, and then print the message, “Waiting for printer to become available”?

Then it dawned on me what mindbogglingly stupid thing it must be doing: it’s enumerating all of the USB devices, and then trying to open them by USB address rather than using the proper driver! Which of course means that it will be going through the wrong device special file, and the permissions I’ve configured to be automatically applied via /etc/devfs.conf won’t have any effect. Obviously, I’m not going to allow cupsd(8) to interact with every USB device on my system, so I guess I’ll just have to hope that the USB address of my printer never changes — and that I can remember this whole mess when I replace my print server!

Posted in FreeBSD | Tagged , , ,