Chocolate tasting results for (extra dark) championship week 1

We held the first of two championship tastings Monday, concentrating on the 70% end of our “extra dark” range. As I wrote on our wiki:

In order to keep the championship round at two weeks, we ended up with eight chocolates per tasting, and some of the tasters noted a bit of fatigue by the time they reached the eighth chocolate — all of which were fairly similar, but the panel was still fairly clear on the differences, and the result was quite certain. (However, it may be interesting to compare panelists comments with previous tastings where the same products were included!) With four first-place votes, the winner was Equal Exchange Very Dark Chocolate (71%). In second place, with three first-place votes, was Theo Pure 70% Dark Chocolate, which I had expected to win. Rounding out the field, Jelina Noir (72%) came in third, followed by Whole Foods Costa Rican Dark Chocolate 71% in fourth, Vivani Dark Chocolate (71%) and Nói Síríus Traditional Icelandic Chocolate 70% tied for fifth, Divine Intensely Rich Dark Chocolate (70%) in seventh, and Dick Taylor 72% Madagascar (Sambirano) in last place.

More to come in a couple weeks when we do the 85% end of the spectrum.

Quote | Posted on by | Tagged , , | Leave a comment

Now taking recipe writeup requests…

Hey, folks! I know there are a bunch of you following me who seem to be interested in my recipe posts (because you click the “like” button). I’m currently working out a schedule for things I want to make for the fall and winter, and there are some empty spaces. Go have a look at my Recipe Pointers pages and let me know in comments here if there’s anything you’d be interested in seeing. (Not responsible for lost, stolen, or damaged game pieces. The decisions of the judges are final. Void where prohibited.)

I’ll be sticking primarily to things that are (a) practical for a single guy to make, (b) under 500 calories per serving, and (c) don’t require exotic ingredients that I’ll be stuck with for a long time. Some exceptions may be made for special occasions, and at least two more cookbooks’ worth of Recipe Pointers are in the queue. I have seven weekends open for the rest of the year, and will probably fit five recipes in, possibly more depending on the sweet/savory balance, cost, and overall level of effort required. The first open weekend is October 4–5, so you’ve got some time to get your requests in!

Aside | Posted on by | Tagged , | 1 Comment

My recipe, for once: whole wheat sandwich bread

This is a slightly revised and updated version of my Wheat sandwich bread, Mk. 4 recipe, developed a number of years ago based on a number of sources (see the link) — now with pictures. As I’m presenting it here, it’s a half-whole-wheat bread, but I’ve had good results using all (white) whole wheat as described in the original recipe; if you want a white bread recipe instead, there are better choices (including Diane Duane’s Tessinerbrot from last January and Joanne Chang’s brioche from July). The procedure here owes a lot to Alton Brown; the specialty ingredients are all from King Arthur Flour.

Note that this recipe uses white whole wheat flour. This is a specific type of whole-wheat flour which is made from hard winter wheat. I’m not sure how easily available this is in other countries, and I’ve never tried it with standard (soft, red) whole wheat. I invite comments from anyone who decides to try this with other flours and report back.


This bread starts with a sponge:

Sponge ingredients
7½ oz 210 g white whole wheat flour
1 oz 30 g butter
13 oz 370 mL water, 115°F (45°C)
2 tsp 10 g SAF instant yeast
2 tsp 10 g diastatic malt powder
3 tbl 30 g vital wheat gluten


  1. Add the butter to the warm water and let it melt.
  2. Sift and combine all other ingredients in a large bowl.
  3. Add the water and melted butter to the bowl and stir until smooth.
  4. Cover the bowl with a sheet of plastic film and let stand at room temperature for half an hour.
  5. Place the covered bowl in the refrigerator for at least 12 hours.

Despite the color, white whole wheat still includes all of the bran of the wheat kernels. When milled into flour, wheat bran turns into sharp fragments which can cut the gluten strands as they develop during the kneading process; this recipe takes two measures to avoid this: first, adding additional gluten (and using high-protein flour in the first place) ensures that enough gluten is formed, and second, allowing the sponge to ferment for a long period softens the bran fragments so they are less able to interrupt the gluten structure. The added gluten is probably not necessary in this version of the recipe, because of the bread flour used below, but is important to a good texture and rise as the ratio of whole-wheat to white flour increases. If you don’t have any, substitute an equal quantity of high-protein bread flour — the result won’t be quite as good but still serviceable.

Note that this bread includes no sugar other than what is found in the malt powder. Diastatic malt powder contains an enzyme which helps to break down some of the starch in the flour. (Non-diastatic malt lacks this enzyme, and should be saved for homemade bagels.)


Ingredients for one 9×5 loaf
sponge recipe, above
7½ oz 210 g high-protein bread flour
1 oz 30 g Baker’s Special nonfat dry milk powder
1½ tsp 5 g salt
egg wash
baking spray


  1. Bring sponge to room temperature; this may take a couple of hours.
  2. Preheat your oven to its lowest temperature and turn it off. Alternatively, put a kettle of water on to boil.
  3. Sift together the dry ingredients.
  4. Using a stand mixer with a dough hook, mix together the sponge and the dry ingredients until the dough just comes together. It will look somewhat shaggy. Wait for ten minutes before proceeding.
  5. Still in the stand mixer, knead the dough for ten minutes or so. The dough should be somewhat moist, but if it is too sticky to handle, add a tablespoon of flour and knead for another minute. When the dough is ready, a small ball of dough can be flattened and stretched into a thin membrane (“windowpaning”) without breaking.
  6. Turn the dough out onto a lightly floured surface and form a tight ball by tucking loose ends underneath and rolling the ball around on the work surface between your hands.
  7. Lubricate a large bowl with baking spray and place the dough ball in the bowl. Cover with a sheet of plastic film and place in the warm oven or another warm place. (If you did not preheat the oven, pour some boiling water into a pan and put it in the oven with the bowl of dough.) Let the dough rise for about an hour.
  8. Turn the risen dough out onto a floured work surface.
  9. Using your knuckles, flatten the dough ball into a rectangle as shown in the photo below, taking care to pop any large bubbles of gas.
  10. Fold the flattened dough over in thirds, like a trifold wallet or a business letter. Using a pastry brush to remove any excess flour from the surface before each fold.
  11. Repeat the last two steps twice over, turning the dough 90 degrees each time.
  12. Form the dough into a log and pinch the seams shut.
  13. Spray a standard 9″x5″ loaf pan with baking spray and place the dough inside. Cover with plastic film and return to the oven. If the oven is no longer warm, add more boiling water.
  14. Let the dough proof until it crowns the top of the loaf pan by about an inch (25 mm) (see photo below); this may take anywhere from 45 to 90 minutes depending on the how warm it is.
  15. Remove the loaf and any remaining container of water from the oven and put a rack in the lower middle position. Preheat to 375°F (190°C).
  16. Using a pastry brush, apply egg wash to the top of the loaf. If you prefer a crisper crust, use melted butter instead of egg wash. You can also apply seeds, rolled oats, or salt to the crust at this time.
  17. Using a sharp slicer or serrated bread knife, cut a slit in the center of the loaf, about 1/8″ (3 mm) deep.
  18. Bake the loaf for about 35 minutes, or until it reaches an internal temperature of 190°F (90°C) on an instant-read thermometer.
  19. Remove the bread from the oven and let cool in the pan for 15 minutes, then turn it out onto a cooling rack and allow to cool for another hour before slicing.

I used Sir Lancelot bread flour, which is higher in protein even than King Arthur’s regular bread flour, and almost certainly obviates the need for the extra gluten in the sponge. The yield obviously depends on the thickness of the slice: from a 9″ pan you should be able to get 18 ½” slices if you’re a better hand with the bread knife than I am; I got 16 usable slices. (Would that I could just bring my loaf into the supermarket bakery and use their slicer!) If you like thicker slices (and you probably do), 12 is a reasonable target. The nutrition data presented below is based on 17 slices.


The sponge looks like this, prior to fermentation:
Sponge, prior to fermentation

After fermenting for 16 hours, it looks like some biological activity has taken place (and smells slightly of alcohol):
Sponge, 16 hours later

After mixing the dough, it still looks quite shaggy. We pause at this point for ten minutes to allow the additional flour to hydrate:
Dough, just mixed

The dough has been kneaded, and in this case I had to add a couple extra tablespoons of flour. It’s easy to form into a ball, and into the proofing bowl it goes:
Dough, after kneading

After proofing, the dough is nice and warm and poofy:
Dough, after first proof

We begin the wallet fold procedure by flattening the dough into an oblong (or what would be an oblong if I were a bit better baker):
Flattened dough ball

After making the first of nine folds. Note that the back side of the bread has picked up a lot of flour from the work surface; it’s important to brush off any excess flour, or it will turn into a chewy seam inside the loaf.
First fold

The dough is folded and turned and folded and turned and eventually you get this log shape and pinch the seams shut. Into a lubricated loaf pan it goes for second rise (bench proofing):
Dough ready for bench proof

The dough is done proofing when it crowns the pan by about an inch. If I had done a better job, this would be more even, rather than having a short end and a tall end — that’s indicative of the dough having started out uneven.
Fully proofed dough, showing extent of rise

Then an egg wash is applied and a release cut is made down the center of the loaf, and it’s ready for baking:
Dough ready for baking

At least in my oven, the baking time is a pretty consistent 35 minutes. When it’s done, the top crust is this lovely golden brown, but the sides and bottom are still pale, just like regular sandwich bread.
Fully baked loaf

Because I’m a single guy on a diet and only eat bread in sandwiches, I need to slice it as soon as it’s cool so that I can freeze most of the loaf and thaw only what I need — since this bread does not contain any preservatives (or even all that much salt) it is an ideal culture medium for mold, and nobody wants to eat moldy bread. Unfortunately, I’m a lousy hand with the slicing knife, even given the assistance of a ruler or slicing guide, but I managed to get 17 slices out of this — one of which I immediately toasted up, buttered, and ate, because FRESH BREAD:
Messy job of slicing


The figures below do not include the egg wash, which makes a negligible contribution to nutrition.

Nutrition Facts
Serving size: 1 slice
Servings per container: 16-18
Amount per serving
Calories 114 Calories from fat 18
% Daily Value
Total Fat 2g 3%
 Saturated Fat 1g 5%
 Monounsaturated Fat 0g
Trans Fat 0g
Cholesterol 5mg 2%
Sodium 164mg 7%
Potassium 0mg
Total Carbohydrate 18g 6%
 Dietary fiber 2g 8%
 Sugars 1g
Proteins 5g 10%
Vitamin A 1%
Vitamin C 0%
Calcium 3%
Iron 5%
Posted in Food | Tagged , , , | Leave a comment

Chocolate tasting results for week 7

I was expecting week 7, chock full of artisanal chocolates from boutique American makers, to be the culmination of our two months of preliminary heats. Boy was I wrong:

This tasting was a bit of a disaster for the artisanal chocolate makers — in fact, our two reference chocolates, Green & Black’s 85% and Equal Exchange Very Dark Chocolate tied for first place, each receiving three first-place votes and two second-place votes. Excluding those two, the top finisher was Dick Taylor 72% Madagascar, despite the sourness that all tasters noted, with one first-place vote, followed by the old bar of Rogue Silvestre 75% with two second-place votes. Rogue Balao 75% also received one first-place vote.

I’m not quite sure what happened — was it my tasting panel, or the protocol, or even the chocolates themselves? I mean, if you’re going to pay 24 cents a gram, you’d hope that at least someone would think it was the best they’d ever had! One theory of mine, though, is that it’s just the format of these bars:

I suspect that the size of the pieces may have significantly affected the reception of these artisanal chocolates. Unlike the two supermarket bars included as references, the five on-theme products are cast in very thin bars, which makes them less conducive to the tasting protocol we have been using. Tasters received approximately 50% larger pieces of the Green & Black’s 85% than they did of the Dick Taylor bars (at 56 grams each, the smallest in our entire series of tastings), and the Equal Exchange bar came in even larger pieces. I suspect that all of these products would have done better as somewhat thicker bars with portions between 4 and 6 grams.

(See the full details on our wiki.)

The championship round will run for two weeks, although the schedule is not determined yet. I tried to organize the chocolate lineup for each week to minimize the apples-to-oranges effects, so the first week will be:

  • Theo Pure 70% Dark Chocolate (runner-up, week 1)
  • Vivani Dark Chocolate (71%, winner, week 2)
  • Nói Síríus Traditional Icelandic Chocolate 70% (runner-up, week 3)
  • Whole Foods Costa Rican Dark Chocolate 71% (winner, week 4)
  • Divine Intensely Rich Dark Chocolate (70%, Ghana, winner [tie], week 5)
  • Jelina Noir (72%, mixed origin, winner, week 6)
  • Equal Exchange Very Dark Chocolate (71%, winner [tie], week 7)
  • Dick Taylor 72% Madagascar (Sambirano) (top on-theme product, week 7)

Check back here next week for the results!

Quote | Posted on by | Tagged , , | Leave a comment

Other people’s recipes: Joanne Chang’s honey-cinnamon ice cream & Cook’s Illustrated’s rice and lentils

It’s a two-for-one today: first the honey-cinnamon ice cream from Joanne Chang’s first cookbook, Flour (p. 260), and then from this month’s Cook’s Illustrated, it’s “Rice and Lentils with Crispy Onions” (pp. 8–9).

Let’s start with the ice cream, which I made first but tried second (since it wasn’t finished freezing last night). Chang describes how a honey-vanilla ice cream turned into honey-cinnamon ice cream when a supplier failed to deliver her vanilla order on time to Rialto, the restaurant which she had her first job as a pastry chef. Since all the desserts were accompanied by ice cream, she had to substitute, and decided to use cinnamon. Other than that one substitution, it’s a fairly standard French-style ice cream, made with half cream and half milk plus lots of egg yolks to form a custard base. The primary sweetener is honey — I used McLure’s orange blossom honey — with a small amount of sugar added, and the cinnamon gives it the sort of understated spiciness for which Chang is known. The custard base is wonderfully smooth after mellowing for 18 hours, and remains silky smooth even after a day’s hardening in the back of the freezer. Behold:
A serving of ice cream in a dessert cup

I’m actually interested in turning this back into a honey-vanilla ice cream, and possibly experimenting with different types of honey to see what sort of flavors are expressed. With a full ¾ cup of honey in the custard, the honey flavor definitely carries through to at least some extent. But I have lots of other things I want to try before I get back to this one! (Ingredient notes: I used High Lawn Farm low-fat milk and Sky Top Farms unhomogenized heavy cream for the dairy, and Country Hen extra-large organic eggs for the egg yolks; the nutrition data below does not take this into account.)

That brings us to the Test Kitchen’s “Rice and Lentils with Crispy Onions” with a tangy yogurt sauce — also known as mujaddara or megadarra — a complete-meal vegetarian dish from the Middle East. The Achilles’ Heel of this dish is definitely the crispy onions: although I thought I followed the directions fairly closely, I still ended up with blackened, nearly burnt onions which absorbed nearly half (¾ cup!) of the frying oil. If I made this dish again — and I might well do — I would probably substitute caramelized onions, which add far less fat to the dish and are less finicky. (Or I could just buy crispy onions from the store, but that seems like cheating, particularly to get the equivalent of two pounds of fresh onions!) Nonetheless, I found myself irresistibly drawn to snacking on the onions, to the point where there wasn’t enough left to sprinkle on top as directed, so I just stirred all the onions in.

I made the onions a day ahead, and by the time I actually needed them, they had lost a great deal of their crispness, so I put them in the oven on a parchment sheet at 250°F to drive off some of the remaining water — would that I could do that with the fat! — with reasonably success. Of course, this just made them even more snackable.
Crispy onions, slightly burnt, on a sheet of parchment for reheating

As prepared, the recipe made a little over three pounds — which made Cook’s Illustrated‘s “serves 4 to 6″ seem a little silly: as a side dish, it probably serves 8 to 10, and as a vegetarian main dish, 6 to 8. To serve 4, you’d be talking 12 ounces of mujaddara per 900-calorie serving. I’ve used 6 servings for the nutrition computation below (which, note well, also includes an entire cup of canola oil!), but for dinner tonight I had a six-ounce portion (¾ serving) alongside four ounces of store-bought chicken salad. (Portion weights do not include yogurt sauce, measured separately, but the calorie counts do.) This is Sunday’s dinner plate:
A serving of the finished rice-and-lentil dish, with yogurt sauce, on a dinner plate
If I had been less tired, I would have steamed some spinach to add some more green to the plate.


For the ice cream:

Nutrition Facts
Serving size: ½ cup
Servings per container: about 10
Amount per serving
Calories 308 Calories from fat 190
% Daily Value
Total Fat 20g 31%
 Saturated Fat 15g 73%
 Monounsaturated Fat 2g
 Polyunsaturated Fat 1g
Trans Fat 0g
Cholesterol 235mg 78%
Sodium 93mg 4%
Potassium 28mg 1%
Total Carbohydrate 24g 8%
 Dietary fiber 0g
 Sugars 24g
Proteins 4g 8%
Vitamin A 23%
Vitamin C 0%
Calcium 41%
Iron 3%

For the mujaddara:

Nutrition Facts
Serving size: 8 ounces (excluding yogurt sauce)
Servings per container: about 6
Amount per serving
Calories 589 Calories from fat 351
% Daily Value
Total Fat 39g 61%
 Saturated Fat 4g 19%
 Monounsaturated Fat 22g
 Polyunsaturated Fat 11g
Trans Fat 0g
Cholesterol 6mg 2%
Sodium 1015mg 42%
Potassium 719mg 21%
Total Carbohydrate 47g 16%
 Dietary fiber 15g 59%
 Sugars 11g
Proteins 15g 29%
Vitamin A 4%
Vitamin C 33%
Calcium 13%
Iron 24%
Posted in Food | Tagged , , , , | Leave a comment

The network nightmare that ate my week

A few hours ago, I sent the following “dear colleagues” email (lightly edited to remove some private details) to all my users at work:

This has been a very trying week. For those of you whose work was disrupted this week by unplanned network outages, my deepest apologies. I am writing to you to explain what we know so far about the cause of these problems, what we have done to resolve them, and
what actions still remain to be taken.

Over the course of the summer, we have been suffering from a variety of difficult-to-identify network problems. Our network switches have been observed using far more CPU than has historically been the case, we have had a variety of packet storms that appear to have been caused by forwarding loops despite the fact that we run a protocol designed to prevent such loops from taking place, and we have had a variety of unexplained switch crashes.

First issue

Starting very early Wednesday morning, the switch that serves the server room 32-399 began to crash unexpectedly, and in a way that (contrary to design) requires a human to physically power-cycle it in order to restore service. This switch crash affected a variety of research groups’ systems, as well as central AFS servers and the newest set of OpenStack hypervisors. We initially thought that this was a consequence of the power failure we experienced on Tuesday evening, and the vendor (Juniper Networks) suggested a reinstallation procedure for the member nodes in this switch to recover from possible corruption of the flash media from which the switch member nodes boot. Unfortunately, this did not resolve the problem, although it did put off the recurrence for a few hours.

On Thursday morning, an engineer from Juniper came to Stata to help us determine the cause of that crash and come up with a resolution. He was assisted by five other Juniper engineers from their Advanced Technical Assistance Center via teleconference. It took the rest of the day Thursday to come up with an action plan to resolve the issue (although still without any identifiable root cause), because the failing switch nodes were unresponsive on their console as well as the network, and none of the more obvious fixes we tried had any effect. (The switch logs, on reboot, stopped just before the crash.) Eventually we decided to upgrade this one switch to the latest version of Juniper’s firmware for this platform (EX4200), and simultaneously to make several configuration changes to reduce the use of code paths which are less well tested. This, in combination with what I’m about to explain next, appears to have resolved the issues with this particular switch. I will be monitoring this switch over the weekend to make sure it remains stable. Juniper has dispatched replacement hardware for all three member nodes of this switch, in case it proves necessary, but at this point we believe that the problem was caused by software and not a hardware failure (and thus, the association with the power outage was a red herring).

Second issue

Over the summer we have been experiencing a variety of issues with our core switches. We continued to experience these issues even after upgrading the firmware as recommended by Juniper, which we did (on an emergency, unscheduled basis) two Mondays ago. The most serious issue is that there appear to be bridge loops occasionally being created which saturate the CPU on the core switches, thereby preventing them from running necessary protocol processing. I believe (but don’t have official confirmation yet) that this can result in a priority inversion relative to flooding broadcast and multicast traffic through the network, such that the process that implements the Spanning-Tree Protocol (which constructs a loop-free topology by deactivating redundant links) is unable to run, which causes all of the access switches to think that their uplink is no longer connected to a spanning-tree bridge, which causes Sorceror’s Apprentice packet amplification as multicasts are then forwarded down both core links simultaneously — which only adds to the overload on the core switches. Sometimes the core switches would recover from this condition without human intervention, but several times they did not, and I was forced to physically power-cycle one of them to break the loop.

We are trying to develop some configuration changes that will make this less likely in the future, by changing some of the assumptions in the Spanning-Tree Protocol to limit the possibility of bridge loops forming in the first place. This work has yet to be done, so in the mean time, we have made some changes to our network to reduce the CPU load on the core switches and make it less likely that the spanning-tree process can get starved in the first place.

The principal change that I have made in this regard is to disable IPv6 on most CSAIL networks. I have come to the conclusion that so much in IPv6 design and implementation has been botched by protocol designers and vendors (both ours and others) that it is simply unsafe to run IPv6 on a production network except in very limited geographical circumstances and with very tight central administration of hosts.

Technical details

The fundamental design problem with IPv6 is related to how it functions over shared layer-2 networks like Ethernet. In IPv4, there is a separate protocol (ARP) which is used by hosts to find the MAC address of other stations on the network. To do this, a host that wants to send a packet sends a layer-2 broadcast frame asking “who has IP address” The intended recipient, if it’s on the network, sends a reply that says “Hey, I’m, and my MAC address (in case you missed it) is 01:23:45:67:89:ab!” If the intended recipient isn’t on the network, the sender keeps on broadcasting periodically until it either gets a response or gives up.

In IPv6, this is not a separate protocol; it’s called “neighbor discovery”, and uses ICMPv6 packets. Because IPv6 does not have broadcasts, the soliciation packets must be sent to a multicast address. But rather than use the standard “all hosts” multicast address, the Neighbor Discovery Protocol instead specifies that every host must join another multicast group, one of 4 billion distinct “solicited node” multicast groups chosen as a function of the system’s IPv6 address, and Neighbor Discovery packets are sent to this group rather than being broadcast. (This means that the vast majority of all IPv6 multicast groups in use anywhere in the universe have a single member.)

In theory, that should be no worse than sending an ARP broadcast, but in practice it is much worse, because IPv6 systems must also implement the Multicast Listener Discovery protocol, by which all stations on the network report, when requested, all of the multicast groups they are members of — and they send these reports to the group, which means flooding those reports throughout the network, because the network switches have no way of knowing Ethernet multicast addresses are desired on which ports. Furthermore, MLD packets are required by the protocol to be transmitted with a “router alert” option, which causes routers to bounce these packets from hardware forwarding into software, meaning that while the flooding of ARP broadcasts can be fast-pathed and are usually implemented in silicon, MLD listener report multicasts must be slow-pathed — and since IPv6 is still not implemented on very large layer-2 networks like a campus network or our building, MLD processing is generally poorly tested and not optimized by network vendors. Our core switches send MLD all-groups queries every 150 seconds, and direct end hosts to splay their responses over a 10-second interval, both as recommended in the MLD RFC.

In theory, we could implement “MLD snooping” across our network to reduce the overhead of flooding MLD listener report packets all over. However, this is very new and raw code, at least in Juniper’s operating system, and not well exercised anywhere that I’m aware of. Even if we did this (and we tried), that would require at least two entries in the multicast zone of every switch’s TCAM (hardware forwarding table) for every host on every IPv6-enabled network in CSAIL, just to handle the Neighbor Discovery multicast groups — and in our switches, the entire TCAM space dedicated to IPv6 multicast is only 3,000 entries. That would be just barely enough to support all of CSAIL, but for one major issue: “privacy” addresses.

IPv6 “privacy” addresses are an incredible botch added to IPv6 a few years ago to simulate the “good old days” of dial-up Internet where machines changed their IP (v4) addresses all the time. In normal IPv6 usage, every host generates an IPv6 address by listening for broadcasts from routers telling them what network prefix (the first 64 bits of the IPv6 address) to use, and appending to that a fixed function of their network interface card’s MAC address. In “privacy” addresses, the host simply generates a 48-bit pseudorandom number, pretends that it’s a MAC address, and applies the same function. The host will also maintain a “traditional” IPv6 address, used only for incoming connections — the random address is used for outgoing packets. What’s worse, the random address is changed regularly, typically daily, but the old random addresses are kept around for a fairly long time, on the order of a week, in case some other host out there on the network wants to “call back” after a new address has been generated. Thus, a typical machine — say, an Ubuntu 14.04 workstation with the default configuration — will end up claiming eight IPv6 addresses at any one time. That means nine IPv6 multicast groups, which means that those 3,000 TCAM entries can be exhausted by as few as 333 Ubuntu workstations.

This is generally not an issue for portable machines like laptops, because they forget all their old random addresses whenever their IPv6 prefix changes (which happens whenever they are disconnected from one IPv6-capable network and connected to another) but it is a very serious issue for workstations and, of course, servers, that are connected full-time to a single network. (The random addresses are also very problematic for me as a network administrator, because they mean that I am unable to trace back problem machines to their owners if they have been removed even briefly from the network in the interim.)

I used Ubuntu as an example, but it is hardly the worst offender. We have seen Windows machines with more than 300 IPv6 addresses — which, recall, means that every 150 seconds they will be transmitting 30 multicast packets per second which have to be flooded through the network. That problem was caused by a broken Intel NIC driver — Windows attempts to offload IPv6 processing to the NIC while the workstation is in standby mode, to support wake-on-LAN over IPv6 — but we had to get an updated driver from Intel to fix the problem (Windows Update was still distributing the broken driver). We’ve seen other machines that merely flood the network with copies of a single MLD listener report — sometimes hundreds of packets in less than a second. I first learned about this back in July when we started to observe CPU overload on the switches that serve our area; it turned out to be one of the public Windows machines outside the TIG offices, but once we knew what to look for, we saw many other machines doing the same thing. (Pretty much any new Dell workstation running Windows has this bug, unless the Intel Ethernet driver has been updated.)

For this reason, we will not be making IPv6 more broadly available again until we have a reliable way of ensuring that “privacy” addresses are disabled on all clients. (They are already disabled in CSAIL Ubuntu, but there are of course many other clients that we can’t control centrally.) We will probably move towards not supporting IPv6 “stateless” autoconfiguration at all, and rely on DHCPv6 to assign stable, traceable addresses to all IPv6 clients, but we’re a ways away from being able to implement that at this time.

I finally got home last night around 1 AM, after being awake for nearly 40 hours. If I never see another dawn in the office again, it will be too soon.

UPDATE (2014-09-06): As Stéphane Bortzmeyer was the first to point out, RFC 7217 addresses all of my issues with “privacy” addresses. Let implementation come soon!

CORRECTION (2014-09-06): There are actually only 16,777,216 “solicited node” multicast groups, not 4 billion (as I posted here). I originally said 65,536 in my email, but realized that was wrong, and miscorrected it when I posted here.

Posted in Computing | Tagged , , , , | 64 Comments

From the pages of Stupid Software Design Quarterly…

About a month ago, printing stopped working on my machine. I had a clunky workaround, so I didn’t worry too much about it. I had just upgraded the cups package, which manages printing, on the print server, and it seemed to think the printer wasn’t “available”.

Let’s back up a bit. My print server (which is also my desktop) runs FreeBSD. The standard solution for printing on FreeBSD these days is to use cups (originally a commercial open-source product from an independent software house, now owned by Apple). I bought a Xerox Phaser 6140 really cheap (under $150 including shipping), but adding an Ethernet port would have doubled the price (not to mention wasting an IP address for something that shouldn’t be public anyway), so I had to use the USB interface. Fine, I said, and got a long USB cable to plug it in; it showed up as /dev/ulpt0, and I configured cups to print to that device, and all was well.

(Well, actually, not all was well — I had to configure some hacks in /etc/devfs.conf to allow cupsd(8) to access the printer device, and I also ended up creating another hack for devd(8) to automatically pause and restart printing when the printer was powered on and off, since laser printers generate a lot of heat and are energy hogs.)

When I upgraded the cups package and printing stopped working, I discovered quickly that I had no difficulty printing directly to /dev/ulpt0 — I just had to remember the usual bit about sending a control-D to tell the printer’s PostScript job manager that the file I had just transmitted was done. So clearly whatever had broken printing wasn’t in the operating system, but some new misbehavior on the part of cupsd. After digging around (and finding lots of totally unhelpful Linux forum sites that talked about “blacklisting modules” and other crap that FreeBSD users don’t have to put up with), I was able to find out the name of the program that cupsd outsources USB printing to: /usr/local/libexec/cups/backend/usb, which allowed me to find it in the process table, which allowed me to ktrace(1) it, which told me why I was tearing my hair out:

  5950 usb      CALL  open(0x80082f3f1,0,0)
  5950 usb      NAMI  "/dev/usbctl"
  5950 usb      RET   open 7
  5950 usb      CALL  ioctl(0x7,USB_READ_DIR,0x7fffffff9950)
  5950 usb      RET   ioctl 0
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen0.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen1.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen2.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen3.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen4.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen5.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen6.1"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen6.2"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  open(0x7fffffff9a90,0x2,0)
  5950 usb      NAMI  "/dev/ugen3.2"
  5950 usb      RET   open -1 errno 13 Permission denied
  5950 usb      CALL  ioctl(0x7,USB_READ_DIR,0x7fffffff9950)
  5950 usb      RET   ioctl 0
  5950 usb      CALL  close(0x7)
  5950 usb      RET   close 0
  5950 usb      CALL  write(0x2,0x7fffffff9510,0x20)
  5950 usb      GIO   fd 2 wrote 32 bytes
       "DEBUG: libusb_get_device_list=0
  5950 usb      RET   write 32/0x20
  5950 usb      CALL  close(0x5)
  5950 usb      RET   close 0
  5950 usb      CALL  close(0x6)
  5950 usb      RET   close 0
  5950 usb      CALL  write(0x2,0x7fffffff7a00,0x2f)
  5950 usb      GIO   fd 2 wrote 47 bytes
       "INFO: Waiting for printer to become available.
  5950 usb      RET   write 47/0x2f

What’s it trying to do there? Why did it try to open a whole bunch of device special files it had no business looking at, and then print the message, “Waiting for printer to become available”?

Then it dawned on me what mindbogglingly stupid thing it must be doing: it’s enumerating all of the USB devices, and then trying to open them by USB address rather than using the proper driver! Which of course means that it will be going through the wrong device special file, and the permissions I’ve configured to be automatically applied via /etc/devfs.conf won’t have any effect. Obviously, I’m not going to allow cupsd(8) to interact with every USB device on my system, so I guess I’ll just have to hope that the USB address of my printer never changes — and that I can remember this whole mess when I replace my print server!

Posted in FreeBSD | Tagged , , ,

Other people’s recipes: Cook’s Illustrated’s Cream-cheese brownies

Last weekend I made the notably unsuccessful “cherries and cream pie” (see writeup), and of course that meant I had leftover cream cheese (that recipe only took a quarter of a cup) and sour cream, two ingredients which I rarely have any use for in everyday cooking. So in order to use them up before they spoiled, I went looking for recipes that would use both. Cheesecakes were out of the question, since they typically require much more than I had on hand, and tend to be written to require an exact number of packages of the Leading National Brand. Any number of recipes call for sour cream alone, which didn’t help dispose of the cream cheese, but I settled on the cream-cheese brownie recipe from last month’s Cook’s Illustrated. (For subscribers, that’s this recipe from 2014, not this recipe from 1999.)

This recipe was a bit fussier than the traditional brownie recipe, which is hardly surprising considering that what we’re making is effectively two desserts — a brownie and a cheesecake — which are baked together in one pan, but are supposed to retain at least some of their distinctness. That means dirtying a lot of bowls (I ended up using four — one for the chocolate and butter, one for the dry team, one for the cream-cheese mixture, and one for the wet ingredients). I think it turned out pretty good:
An 8x8 pan of cream-cheese brownies on a cooling rack, cut into nine individual brownies, with one removed

I didn’t do as good a job of swirling the cheesecake into the brownie batter as shown in the printed recipe. Also, the recipe calls for 2″x2″ brownies, which is ridiculous; an 8×8 pan makes nine brownies, no more. Here’s the center piece:
Cream-cheese brownie on a plate

Of course, that does mean that as I made them, they’re a bit calorie-heavy. I see no reason you couldn’t make this recipe in a 9×9 pan instead, and then it wouldn’t be ridiculous to get 16 (somewhat thinner) brownies. But doubling the recipe and baking in a 9×13 pan (with appropriate adjustment to the cooking time) would seem quite practical as well, if you’re not in the position that I was of trying to get rid of leftovers! (I think commercial bakers tend to make brownies in half-sheet or even full-sheet pans, which are just too shallow for proper brownie construction, in my humble opinion.)


Multiply everything by 9/16 (approximately 0.56) for the 2×2 brownies specified in the recipe (yield 16). Either way, a special treat, not an every-day dessert. I’ll be freezing some of these, or giving them away to colleagues.

Nutrition Facts
Serving size: 1 brownie
Servings per container: 9
Amount per serving
Calories 423 Calories from fat 235
% Daily Value
Total Fat 26g 40%
 Saturated Fat 16g 81%
 Monounsaturated Fat 1g
Trans Fat 0g
Cholesterol 97mg 32%
Sodium 216mg 9%
Potassium 42mg 1%
Total Carbohydrate 42g 14%
 Dietary fiber 0g
 Sugars 30g
Proteins 5g 10%
Vitamin A 15%
Vitamin C 0%
Calcium 7%
Iron 11%
Posted in Food | Tagged , ,

Week 6 chocolate tasting results

I noticed that I forgot to post the results of our sixth weekly chocolate tasting. From our wiki:

Once again, one of our “reference” chocolates took top honors: the Jelina Noir 72% that we had previously tasted back in week 1 (however, of this week’s participants, only Sue Felshin had attended the tasting in week 1). The Jelina was tied for first place with Idilio Origins Finca Torres, both having two first-place votes each, but gets the nod on the basis of its two second-place votes. Valrhona Araguani and Idilio Origins Carenero Urrutia Superior tied for third place, each receiving one first-place and one second-place vote; Idilio Origins Selección Cata Ocumare came in fifth with one second-place vote. Now I need to go buy another bar of the Jelina for our championship round.

Because this week is the first week of classes, we’re delaying week 7, the last of the preliminary round, until next week, once people’s schedules settle down a bit.

Quote | Posted on by | Tagged , ,

Other people’s recipes (sort of): Cherries & cream pie

This gallery contains 10 photos.

As I discussed last weekend, the cherry pie I made used less than half of the cherries I had bought. So of course I had to make something else with cherries (I didn’t want to make the same pie twice … Continue reading

Gallery | Tagged , , | 1 Comment