Judging the Judges

Judging WinesThe perils of rating wines are many and, well, perilous. Can it be that the educated palate isn’t educated at all? Is this an oenological dirty little secret?

A four-year study in the United States, made public last week, showed that only 10 percent of wine judges were consistently able to give the same wine the same rating twice, to wines sampled multiple times in blind tastings.

To use the adage, “A little knowledge is a dangerous thing”, it is one thing to be well-versed in wine and another to have a trained palate with level of skill and experience to consistently and accurately analyze wine. The conundrum exposed by the study brings to light the flaws of wine shows and the credulous trust that consumers place in ‘experts’, whereas they would perhaps be better off having more confidence in their own palates.

Every wine writer or industry professional must grapple with scoring at some point, discomfortingly critical as it is to one’s reputation and possibly the only way to establish credibility. Personally, I have a love-hate relationship with scoring. As tempting as it is to gloat in the power of assigning numbers, I continue to be ambivalent. In Asia a growing caste of wine drinkers are obsessed with scores, along with wine merchants and marketers who seek to sell wine through the blatant exploitation of ratings. Even more worrying is that the average Asian wine drinker is oblivious to the issues, new to western-style wine and reared on the 100-point scoring system.

This epidemic stems from the United States where such individuals are otherwise known as ‘score whores’ — wine snobs – according to a Master of Wine friend who recently visited the US and was amazed at the American devotion to wine scores, largely because of publications like the Wine Spectator, and the omnipotent Robert Parker, who after all invented the 100-point scoring system.

I have utmost respect and admiration for Parker. Clearly, he has an extraordinary palate matched by an exhaustive knowledge of wine. Actually, I get more out of his tasting notes than the scores. However, that doesn’t mean I agree with him at all times. I am most certainly not impressed by formulaic wines that are blatantly fashioned to please the Parker palate and there is evidence that consumers in the US are also tiring of his preferences, particularly syrupy, overblown Barossa Valley and McLaren Vale shiraz. I remain skeptical of his methodology of determining that a wine should only be given 89 points, just a mere one point short of that magical 90 points.

But a few years ago when I was emceeing an options tasting — the entertaining if not masochistic sport of identifying masked wines guided by multi-choice questions, one of the participants announced, in an irritating manner, that she and her husband only drank wines rated 95 points and above. Blind tastings are merciless to wine snobs and our self-proclaimed connoisseur humiliated herself with an unequivocal preference for a non-rated, humble Portuguese red (HK$240 per bottle) over a celebrated 1998 Henschke Hill of Grace Shiraz, rated 97 points by Parker and valued at more than HK$3,000 at the time. Another recent wine experience highlighting human behavioral inadequacy and its relation to wine was when I attended a luncheon hosted by the celebrated Château Haut-Brion to taste several wines. The woman sitting next to me kept badgering me on which wine I liked the most, and didn’t understand my answer: “All of them!” She missed the point that the 2004 Château Laville Haut-Brion Blanc was as enjoyable as the 2005 Château Haut-Brion Blanc, although completely different in style or vintage expression. Ditto for the 1998 Château La Mission Haut-Brion or the 1995 Château Haut-Brion. I’d be damned if I was going to be intimidated by this captious group.

What is it with humans that we always have to be contentiously comparative when there is more than one wine served or become super critical when there are extraordinarily expensive bottles involved? It would be naive to suggest that we could do without personal opinions or ratings altogether. But wine drinkers need to move away from the herd mentality and simply become more reliant on their own tastes.

Scores aside, the predictable question wine writers are always asked is, “What is your favorite wine?” To which I reply, “The wine I haven’t tried yet.” I gain the most pleasure in discovering new taste sensations. My palate for food and wine is greatly influenced by my mood and by different cuisines, although inevitably, there are certain flavors, varieties and regions I enjoy more than others.

The very notion of drinking one type or style of wine continuously escapes me completely. And yet, seemingly the exact opposite is happening to wine consumers’ palates; they are being dictated by scores or powerful critics and corralled by herd mentality and slaves of fashion, naively ensnared in vinous mediocrity. The situation is not helped by the plethora of pedestrian emails in circulation from wine merchants, the contents almost entirely reproductions of scores and tasting notes — little wonder that consumers are becoming lazier and coerced by scores.

The embellishing of every point above 90 is rampant. Anything above 95 is money in the bank for vintners, although a perfect 100 is more likely to cause a massive headache for the distributor and is a milestone that haunts producers for the rest of their days.

It is not that I am questioning the competence of those who choose to score or rate wines. On the contrary, there are many writers, critics, winemakers, wine and hospitality industry professionals with encyclopedic knowledge and seriously sharp palates who can dissect and assess a wine with the precision and thoroughness of a forensic scientist performing an autopsy. But it is crucial to keep in mind that individual scores are highly subjective and that you need to synchronize the personality and particular tastes or track record of the critic with your own likes or dislikes.

Wine JudgesWhere scores are a result of a tasting panel, we encounter one of the most serious flaws associated with numerical ratings. The permutations in calculating the mean score are highly manipulative, and like any statistic, the range or difference between the largest and smallest value alters the outcome significantly. Panel tasting outcomes can also be greatly obscured by invited guest judges, particularly international ones, as it invariably takes time to acclimatize palates and their scores can be wildly divergent from the locals.

There is also a tendency to assess wine at the tasting bench and in exhaustive batches. The method lacks soul, and there is the unavoidable element of comparisons that are often unjust. There is also the question of the taster’s methodology or philosophy and whether he or she is assessing the wine in context of the region’s known characteristics or an accepted style, or have an all-encompassing qualitative view.

This issue is controversial, with some critics having to defend their assessments. James Halliday, arguably Australia’s most respected wine critic, scored a pinot noir rosé 94 points out of 100 points, in my opinion, rightly so. But how could a simple rosé be scored only three points away from Penfold’s Grange, Australia’s icon, which scored 97 points, the highest level in the 2008 Australian Wine Companion?

Technical issues aside, the main flaw in clinically rating wines is that there is no accounting for the flavor and enhancing influences of food, mood or your state of mind at the time of drinking — all of which profoundly affect your opinion. I can think of endless personal experiences that demonstrate this. It is one thing to travel the hills of the Languedoc, where Roman generals chose to retire among the joie de vivre cultures of Catalonia and France, to luxuriate in jovial hospitality on an impeccably radiant Provincial afternoon, and quite another to sample wines on a miserable wet afternoon in a warehouse. In the Languedoc, I could be compelled to score a wine a perfect 100 points.

I recall a bittersweet experience of running a small business when I discovered that my administration manager, who had impressed the entire wine community with his unrivalled knowledge of old and rare wines, was enjoying his research at my expense. After he was turfed out, we unearthed a magnum of 1949 Comte de Vogue, `Le Musigny’ Vielles Vignes Grand Cru red Burgundy, which would be served in thimbles if at a formal wine dinner. In an act of decadence, I decided to share the wine with three devoted drinking comrades. We were in nirvana as we toasted both the fired executive’s exodus and expertise, immersing ourselves in our gluttony and the captivating sweet perfume and evocative secondary aromas of antique woods, wild mushrooms, truffles and earthy minerals. It remains my benchmark for an aged pinot noir to this day. As for a score, I would rate it 200 points out 100 points.

Price/quality rapport is perilously subjective and clearly linked to disposable income, perhaps the overriding factor in categorizing wine, period. Invariably there are discernable reference points to make a judgment whether a wine is over-delivering, or under-delivering for that matter, at its price point. Moreover, a tangible appreciation and anecdotal corroboration among informed wine drinkers that often set wines apart and naturally invokes the laws of supply and demand.

I suspect that many Asians, regardless of their wealth, appreciate good value or, even better, a bargain. The greatest value and bargains in today’s highly competitive world of wine is the mid-ground, where passionate, dedicated owner-operators are trying harder than ever to gain recognition and an edge. When it comes to magazines publishing ratings or using regional or comparative tastings as features, I personally believe the star-rating system is the most just, even ethical method. It might not be as glamorous or attention-grabbing as the 100-point scale but it is not as controversial, placing a wine in a category or range of points that allows for the shortfalls of panel tastings yet, at the same time giving a clear enough indication of quality.

This method is used by the highly respected magazine Decanter and by the world doyen of wine writers, Hugh Johnson. Johnson, I believe, is even more correct, and perhaps diplomatic, in focusing his star rating on the vineyard itself rather than individual wines. As you become more familiar with wine and begin travelling the wine regions of the world, you come to realize it is the producer that is paramount, and ratings or even vintages have less relevance. Assessing any wine is only a snapshot of a ‘living thing’ that undergoes a delicate, continuous transformation of chemistry and integration of organisms.


By Curtis Marsh | Buying Wine | Related to: | 2 comments

You might also like:

Wine into Water – Cambodia
A variety of Thai food
When Thai meets wine
Biodynamic Wines & Wine Feng Shui 酒之风水
Start Your Own Wine Bar at Home!
Chianti Vineyard, Italy
Tuscan Reveries

2 Comments to Judging the Judges | Comments Feed

  • Switalski says

    This is such a great resource that you are providing and you give it away for free. I love seeing web sites that understand the value of providing a quality resource for free. It?s the old what goes around comes around routine.

    November 10, 2010
  • Thanks for shouting about Camden

    December 4, 2011
  • The comments are closed.