I’m thinking about organizing a chocolate tasting at work, and hoping other people out there will have done something similar. This is just intended for fun, but I still want to run a proper taste-test protocol, and actually determine the real collective preferences, if there are any. Of course all the raw data would be made public.

A few issues I’m thinking about:

Protocol

The protocol will obviously depend on how many people are interested in participating and on whether I accept participant suggestions (or ask them to bring something — would require some care to make sure people didn’t all bring the same bar of Lindt they got at the mall store).

Obviously, the usual precautions (random order of tasting, identifying marks removed to the extent possible, portions controlled) should be taken. That means that I probably don’t get to include my own preferences, if I’m running the tasting. I’m not sure what the ideal number of samples per round would be, but I think it’s probably no more than six. I’m not sure whether there is an ideal number of tasters per round, or indeed whether the same tasters should do all rounds. Presumably I would need to hold back some of each sample for use in later rounds.

Some sort of palate-cleanser will need to be provided — seltzer is good enough for me, but not for many other people. Something with some fat (milk, maybe) or a non-polar solvent like rum may be good. I might just ask tasters to bring their own.

Evaluation

I’m thinking the tasters should provide both individual evaluations for each chocolate tasted, and an overall ranking of all of the chocolates in each round. The overall ranking in each round can be determined from the individual rankings, with ties broken on the basis of individual scores. How many moved on to the next round would depend on how many samples there were, and the number of rounds. (So with 24 chocolates and three rounds, that might be six quarter-finals of four chocolates each in the first round, with the top two moving on to two semi-finals of six each, and then the top three from each semi moving on to the finals.)

For the individual evaluations, I’m thinking a 1-5 scale (5 being the best) on aroma, texture, sweetness, bitterness, and flavor, plus freeform comments.

Categories

Obviously it is important to compare only, um, comparable chocolates against each other. The ones I’m most interested in doing would be high-end extra-dark eating chocolate, but given the variety of chocolates out there on the market, I can easily see doing any of these categories (but not all at once):

• Unadulterated dark chocolate, at least 60% cacao (optionally, vanilla, lecithin) — might need to split this into 60–70% and over-70% as some people would likely find truly high-test chocolate unpalatable and give it poor ratings
• Semisweet chocolate (anything brown with less than 60% cacao but no dairy ingredients)
• Milk chocolate (for those who insist)
• Chocolate with stuff in it:
  • Fruit
  • Mint
  • Nuts (whole, in pieces, or ground)
  • Spices (cassia, ginger, chile — any flavoring other than vanilla)
  • Anything else — bacon, caramel, potato chips, wafers, etc.

Some of these categories would be more difficult to organize than others, and obviously anything with animal flesh in it would need to be set aside for the vegetarians. I think I could do all of the “anything else” category from the Vosges catalogue alone, given how many different varieties they do.

If I do this, I’d be happy to include remote submissions of chocolate provided they meet the qualifications for whichever category I end up doing.

Analysis

Obviously I would make all of the data available so other people could analyze it. I’m not a statistician myself and would probably write some simple scripts to help with data reduction, with all of the evaluations stored in a database. (Probably keyed by hand, though, unless someone can point out an existing Free Software application that would allow the users to enter their own evaluations on a mobile device.)

Thoughts?

UPDATE: Moved further discussion to the “Tasters” web in CSAIL FosWiki.