The HOT LIST 2007: Truth, not advertising: Bomb & Gouge: Golf Digest

Print
E-Mail
Feeds

Bomb and Gouge Equipment Blog: golfdigest.com

The HOT LIST 2007: Truth, not advertising

Wednesday, January 3, 2007 Comments

GOUGE: It is a little uncomfortable reading the advance reviews of our annual Hot List project, things like the following seen recently on the otherwise worthwhile equipment-talk website golfwrx.com. Notes one leader of the misinformed:
"It seems like whichever companies give Golf Digest the most $$ for advertising always ends up winning the most "Editors Choice" awards." Or this: "How does a companies "marketing influence" (insert any other pointless category that is not performance related here) affect how well you or me hit their product? It doesn't... So why is it a category?" Study more here.
So how bad does it feel to be viewed as a sell-out, pardsy? You can't win an argument like this. The perception that our annual Hot List project is subject to the whims and pressures of the business side is without question the most difficult complaint for us to understand. The belief that advertisers, or worse those within the company concerned exclusively with the business side, somehow influence our choices, implies a dereliction of duty that is unsettling, not just because it's wrong, but because it in some small way suggests that we've lost the trust readers have in what we do. I'm troubled, friend. How do you get that trust back, aside from screaming from the highest mountain, as you know I'm wont to do, NO ONE, AND I MEAN NO ONE, ASIDE FROM THE FOUR JUDGES ON THE HOT LIST PANEL, HAS A VOTE OF ANY KIND ON THE HOT LIST. NO. ONE.

BOMB: You’re right, you can’t win an argument like this. But that doesn’t mean that it sticks in my craw any less. Want to push my buttons? Impugning my integrity will do it every time. That said, I understand where it comes from. It’s hard not to think that way if you don’t see the process in action. But someone please go tell my wife, 9-year-old and 12-year-old why I spent some 60 days on the road away from them last year if all I had to do was award the biggest advertisers Hot List honors? Or why I’m up at 7 a.m. on my computer on a Saturday or Sunday? Or why I sit in a room with you and our other two judges knocking heads for weeks trying to decipher the slightest differences in products across all five of our criteria? I’m not looking for sympathy, just saying this is all part of what we do to try and be as diligent as possible. I have no problem with those that question our process or debate our selections. That’s all fair game. But implying we’re advertiser-driven in our selections just gets to me in a way few things do because we go above and beyond in so many ways to avoid even the appearance of that. For those that don’t know, we do not accept free trips from manufacturers. The equipment editors pay for the equipment that is in their bags. When we dine with representatives from equipment companies, we pick up the check. And that’s not something many other golf publications can say. Most important of all, no one—not our owners, not our CEO, not our publisher, not our chairman and editor, has ever told us what club to put in what spot on the Hot List. As you said, it is the decision of the four judges. Period. And anyone who thinks otherwise simply doesn’t know that they’re talking about.

GOUGE: If they’re still listening, let’s review another complaint about our Hot List criteria. There is much hemming and hawing that Buzz Factor ought not to play any sort of role in our decision, that sales have nothing to do with how far a driver goes or how many three-footers a particular putter might make, that the idea that “a product must be in demand” is some kind of euphemism for “spends the most money in advertising with Golf Digest.” With the Hot List we are attempting to highlight the most significant products in the game. It is, after all, the Hot List, not the What-driver-goes-farther-on-5/8-inch-off-center-hits-at-an-87-miles-per-hour-swing-with-a-delofting-by-one-degree-at-impact-angle-of-attack List. In addition, proven success in the marketplace is another indication of a product’s or company’s ability to produce quality. So Buzz Factor is a measure of success. If you can generate interest, it goes well beyond advertising, and it indicates a level of performance that goes beyond logo lust. More importantly, however, Buzz Factor has been reduced to 15 percent of a product’s total score, half the weight of both Performance/Playability and Technology/Innovation. The highest marks in both of those criteria are what gets you on the Hot List. Our process, and like you mention our entire way of operating is aimed at striving for a consistent and overriding sense of fairness and objectivity. Quite simply, we attempt to be thorough. Finally, what would be the financial upside for satisfying one advertiser over another? The only master we attempt to serve in the Hot List process is the reader, and given that the Hot List has been rated by readers the magazine's most interesting and useful article for three years in a row, that’s about all you can ask for. But we welcome the criticism, especially you web junkies. It’s how we improve.

Categories:

Keywords:

Today's Rant

Previous Post: The Hot List . . . almost

Next Post: Hot List questions and answers

Bomb & Gouge Main

Rating

Rate it:

1 star 2 stars 3 stars 4 stars 5 stars

Comments

Archived Comments (9) Click to expand

I am writing to inform Bomb and Gouge that I was the poster on golfwrx.com which stated:

"It seems like whichever companies give Golf Digest the most $$ for advertising always ends up winning the most "Editors Choice" awards."

You can call me misinformed but you failed to tell the readers of this blog the rest of my post which went on to say:

"The only reason I enjoy this issue is to use it as a reference when looking at new clubs I am thinking about purchasing...I always take them to the course first before buying."

I believe one of your articles in your 2005 Hot List edition informed me to do that exact thing. Here is the quote from that article.

"We must emphasize the Golf Digest Hot List is not an end in and of itself. The buying process must also include a rigorous examination on your own that involved research at company websites, consultation with your club pro or trusted club fitter, significant play-testing, and at the final stage a thorough fitting session"

And then from the most recent Hot List:

"How you go about your search is an individual pursuit, but whatever your process, you must extensively evaluate products on the range and on the course before you make the commitment to put them in the bag."

The reason a lot us golfers in the "masses" think the "Editors Choice is swayed by advertising dollars is that sometimes the picks do not reflect what the "masses" think are the best clubs.

Another reason the "Editors Choice" award is sometimes not taken very seriously may be because of inconsistent rankings of particular clubs. Take for example your Hot List 2005 rated the Callaway Steelhead X-18 in 8th Place...and then in 2006 the exact same club was rated "Editors Choice." To me that says something other than the "Judges swayed the vote.

I am not writing this comment to stir up anymore criticism of the "Hot List", because it is my favorite edition of Golf Digest. I just wanted to inform you as to why some subscribers think that the "Editor's Choice" award is swayed by advertising dollars.

I commend you on all your hard work making the "Hot List" but just know that there are hundreds if not thousands of us golfers out there that would love to have your jobs testing and tinkering with all the latest and greatest clubs.

GOUGE: "To me that says something other than the "Judges swayed the vote."

Thanks for your thoughts, but when you say the above you are quite simply dead wrong. We can talk about specific cases all you want, but you have to understand our process before you criticize it. Performance/playability is one criteria of five. If a product serves notices in its first year that tens of thousands of golfers use it and like it, that is an indication of success. If that product is still a flagship product the next year, it is reevaluated across all five our our criteria. You can disagree with our process, certainly. But this is why you may think you see "inconsistency" when there is none. I am happy that you have also made the point that the Hot List is not an end in and of itself. YOUR METHOD OF USING THE HOT LIST SHOULD BE THE STANDARD FOR ALL OUR READERS. The worst thing we think anyone could do is to take our Hot List and just buy something without some kind of fitting consultation and extensive trial.

Posted by Scott January 3, 2007 5:55 PM

You're indignation might carry some credibility if your process and grading system that you so vociferously defend wasn't so pitifully transparent. Including "buzz" as a significant factor is asinine. And how exactly do clubs receive different ratings on "performance" than technology. A club gets the highest mark for technology but something less for performance? What the heck was the technology so good for? And was the performance rated for golfers with high swing speeds, low swing speeds, seniors, professionals, hackers...? It is, quite simply, so poorly constructed and conceived, that there is no room for any other conclusion that it's veiled advertising.

GOUGE: Another explanation for the poor misguided Hot List naysayers. Nowhere in our definition of Buzz Factor does the word advertising appear. Conclusion: Advertising is not something we consider. As to varying scores between Performance and Technology, equating the two is a cop out. It suggests that we can't think beyond if a club works and why it works. All of these clubs work, and they work exceptionally well. The Hot List criteria are designed to measure significance. The Technology grade reflects qualities that do not encompass simple performance (Fitting options for one, ideas that attempt to stretch the category's design paradigm, for another). Also, and most importantly, a crucial element in our assessment of technology and innovation is how well is a company's technology explained to the judges in a scientifically thoughtful and meaningful way. These things are all clearly different than Performance. More importantly, these things are important to us in our assessment of products. If it's not important to you, then simply ignore it and sort your list by the criteria that are important to you. It is not unlike the way a car magazine might study stopping distance or engine size as being of some importance in assessing the latest cars. Those things are not important in my choice of car. So I ignore them. In short, I have functioning neurons. I choose to use them. I would hope others do the same.

Posted by RL January 5, 2007 11:24 AM

Any chance of you guys including anything on club shafts in future Hot Lists? Club heads are nice, but the shaft is FAR more important in the overall scheme of getting the right equipment for a particular player, isn't it?

BOMB GOUGE: The shaft is important, but last we checked no one's hit a 300-yard drive off the shaft yet. But your point is one we have considered for years. What we need are some good thoughts on methodology that would hold up without requiring a team of human clones to be on the range and in the lab for nine consecutive months without coming up for air or a FedEx Cup point update. We're open to suggestions, but it's not as easy as it sounds. Best suggestion wins a dozen Pro V1 balls. The new ones that you haven't seen yet. How's that for love from Bomb Gouge?

Posted by Bob Locke January 6, 2007 3:51 AM

The knee-jerk, "sell out" criticism is so, well, knee-jerk that I can't understand why anyone would honestly take such comments seriously. What do these people, Scott and RL, know about the Golf Digest, journalism, or the golf business? There are as many ads for Cialis in golf magazines and on TV today as there are Callaway ads, so can we be sure it isn't big pharma buying the Hot List?

I have wondered about some things in the process--like whether Buzz is legit or not. Fine, we're all entitled to our opinion. But what I like about the Golf Digest ratings are that there's a LOT more to it than 40 guys reading comments into a tape recorder. One of the recent Golf Magazine ratings printed a quote from a golfer, who was like a 17 handicap, that said something like "you could feel the ball stay on the face a little longer." You've got to be kidding me. I'll take my chances with Golf Digest's journalistic integrity any day.

I think if the critics can manage to stay objective and use some gray matter for just a few minutes and really look at this "Hot List", you can only conclude that it's nothing short of spectacular. Thank God we finally have some sort of road map to help us spend our money a bit more wisely. Rock on, Bomb Gouge.

BOMB GOUGE: Salient points. And the debate rages on.

Posted by LD January 8, 2007 4:05 PM

Obviously, you've both already posted a give-and-take with readers critical of your own critiques of products. I think the deeper issue is to be found not in whether or not one club is ranked higher than another because of advertising dollars, but whether one club is actually better than another club.

You allege that some putters make more three-foot-putts than other putters. Really? Seriously? Putters, themselves, make or miss three-foot-putts. Bad putters miss them, good putters make them.

Nobody starts dropping three-foot-putts because they spent $170 on the latest putter. They start dropping three-foot-putts because they hauled themselves out to a practice green and got to work.

Or perhaps we can put this another way. The guys that I play golf with aren't the ones spending thousands of dollars a year on clubs. We don't play golf at fancy courses that allow only the richest one percent of one percent to play. They use older clubs, dinged up and dirty. Some of them make putts. Some of them miss putts. But I've never seen the putter in their hand change and the quality of the putt itself improve.

You guys are always peddling this ridiculous notion that a struggling player can purchase a better game, and in one sense (not yours), that's true. A player can purchase a better game by purchasing practice time and range time. But a seventeen-handicapper who drops $4,000 on brand new equipment won't be an eight handicapper the next day, and he won't be one two years later.

There are players with good swings, who hit good shots with whatever club is in their hands. There are players with middling swings, who hit middling shots with whatever is in their hands. There are players with bad swings, who hit bad shots with whatever is in their hands.

It isn't the equipment; it's the practice. And that's why the Hot List is so absurd. Not because it might be biased - I'm sure you guys aren't taking Eli Callaway's cash and writing good reviews - but because you're flat wrong about what improves a game. That isn't any personal failing of yours. It's just the way things are.

GOUGE: Please point out where we implied that a particular putter makes three-foot putts. Certainly, one of our panelists did say something to the effect that if I had a putter just for three-footers this would be it, in referencing the Heavy Putter. We certainly believe one proven method of attacking the yips is with an extremely heavy putter. Research with the SAMS putting machine says as much, and SAMS has no affiliation with Heavy Putter. But what does it matter? I'm not sure you're in a mood to be proven wrong. Nowhere did we say that spending $4,000 on new clubs will turn a chop into a scratch in a day. But if you and your blue-collar buddies aren't consulting with a decent pro or clubfitter when you buy a new driver, then how can you pretend to think you're going to get better using something that doesn't match your swing and game. The trouble is making any buying decision isn't as obvious as you'd think. Partake of some knowledge about modern technology, and I don't see how it can hurt your game. But as to whether one club is better than another, let's reemphasize that the Hot List is no more a test than the Academy Awards is. We are trying to identify the products we think are most significant. We have chosen certain products as being the most exemplary in their category. Our position is that we've helped the reader interested in discovering the possibilities of new equipment. If you don't want that help, that's your choice.

Posted by Sam Wilkinson January 9, 2007 10:30 PM

So we can just dismiss the categories we aren't interested in and focus on the ones that seem pertinent to us? That would be fine, except for the fact that the less useful categories serve to keep some clubs off of the list entirely. If you want to be taken seriously, publish the results and data for all the clubs tested. Let's see which ones make the list because of "buzz" and which ones get left off in spite of high performance.

And let's explore further how we would use "functioning neurons" and concentrate only on the information we find pertinent:

I start by completely dismissing "Buzz", because it has nothing to do with how the club will perform; because a category that purports to measure "consumer acceptance" when half the drivers listed haven't been released seems suspect; and because it relies partly on the opinion of retailers, who have already placed large orders and have much to gain by creating interest for their largest customers.

Next, I dismiss Personal Preference, because it's vague, the sample size of judges is so small, and I'm not sure why I should care to begin with.

Next, I review Value. Now, I want to view this as a relevant category, but then realize that I have no idea what "Value" is relative to. I'm told it's relative to the "benefit" it provides, but is any of that "benefit" related to buzz or your personal preference? If this strictly relates to performance/price, then it may be of use. I doubt that's the case.

Which brings me to Performance and Technology. Now we're getting to something that matters. Surely, we can find some useful information here. I begin by looking for some details regarding how performance was rated. Was quantitative data gathered or did panelists simply give a rating; how was the data assimilated; was product testing performed by a large enough sample and on numerous different days; how were various performance factors (distance, accuracy, feel) considered and weighted. After reviewing the article, I found no such guidance. I only found a single sentence description that stated the performance measure is based on interviews with panelists who were asked if the product performs in a "comparitively unique" way. Huh? So now I realize I can't determine what is being measured in this category and have no way of knowing whether a high performance rating reflects characteristics I'm interested in or not. So I move on to technology.

Here I'm told that the club is evaluated based on whether the technology is innovative and how well the company explains it's technology. LOL. I can just see it now--Hello Edwin Watts, I'd like to demo the driver with the most well written literature please. If this was confined to technology innovation, it would be useful information. However, we'd still be left with the confusing notion that innovative technology doesn't necessaily result in highly rated performance. I realize there are non-technological factors that impact performance, but they aren't club specific factors.

Further evaluation reveals that not one single club from a niche or boutique manufacturer made the list. Not 1??!! It's hard to believe that a credible process wouldn't identify even 1 noteworthy club from someone other than the usual suspects.

I should not have implied that the Hot List is advertising in objective analysis' clothing. That does question your integrity and crosses the line. I'm willing to accept your statement that you are under no pressure from advertisers and fear absolutely no reprisal if you were to rate poorly the clubs of your biggest advertisers.

I believe the Hot List is simply poorly conceived. It could be so much more useful.

Inability to use the hot list is not a function not having or using "functioning neurons". It is those neurons that recognize that the Hot List doesn't contain useful information.

GOUGE: Wonderfully misguided but passionate soul: I'm so sorry you think the Hot List is not useful. Let's not talk about what you think isn't important, let's talk about what you think is. First of all: Boutique houses not being on the Hot List. What about Miura irons? Guerin Rife putters? Bobby Jones drivers, fairway woods and hybrids? Not boutique enough for you? OK, but let's remember that having the potential ability to actually find the product in your neighborhood shop does weigh somewhat in our consideration. It's not a mandate, but by the same token it doesn't hurt a product's chances.

But really your discussion is about Performance/Playability and Technology/Innovation. We utilize our group of players to get a sense of Performance/Playability. We utilize our scientists to better understand technology/innovation. We do not consider extensive robot testing for the simple reason that there are too many variables to consider. Do you test at one swing speed and one set of launch conditions? Of course not. Six? Three? And how long do you think that might take in a field of say 85 drivers? What if you reduced it to 30? I'll give you an example. The very fun and interesting website bombsquadgolf.com put together a driver evaluation that took so long to execute that most of the drivers they got results for were on their way out by the time the results came out. It's an admirable test, but it's not what the Hot List is trying to be. We're evaluating products, as we state over and over again, in terms of significance. We continue to look for ways to make our evaluation more thorough.

Posted by rl January 10, 2007 3:20 PM

You asked for recommendations about making the "Hot List" better. Here is my idea. Robotic testing by an independent lab. No buzz, technology, value rating allowed; just average distance and dispersion measurements. The best club will be very obvious to everyone who reads the results. I also think that you should find some high quality clones and see if they can beat the originals just for the fun of it.

GOUGE: What is it with you troglodytes and clones? Wouldn't matter to me if they went twice as far at half the price. They're just wrong.

Posted by Bill Gritton January 10, 2007 5:13 PM

On shaft comparisons (and I'm thinking this would probably need to be limited to graphite shafts in drivers to start):

You'd have to have a handful of clubfitters available, obviously. And I would think you would want players of varying skill levels AND swing speeds, since swing speed is the primary determinant in selecting the proper shaft. Also, as the shafts would be the "center of attention", you might want to severely limit the clubhead selections (maybe the marquee lines from the top brands, in multiple lofts). Naturally, a couple of launch monitors would be helpful.

After pulling those things together, it would be a process of hitting balls for the launch monitor and on the range (both for accuracy and distance), and then having the play-testers take their top handful of choices out to the course.

I would think if you did that you could come up with a reasonable listing of "best" shaft/head combinations (i.e. 10.5 degree 905R head with an Aldila Green NV shaft) for the various subgroups of handicaps and swing speeds. It wouldn't take nearly as long to pull together as the Hot List, and since the driver is the club that gets us all started (and gets us the most pumped when hit well) I would think that idenitifying the best head/shaft combination would be important to your readership.

Just my $0.02, and worth exactly what you paid for it.

Posted by Bob Locke January 10, 2007 9:32 PM

LD,

I never said that Golf Digest is a "sell out". I just pointed out my OPINION on what I believe are some of the problems in the Hot List evaluation process. as stated in my previous comment I think this is the best issue that GD puts out and I look forward to it every year. I just think the evaluation process and judging needs to be tweaked (again). ;)

As far as testing shafts for the '08 Hot List...my suggestion would be to use an "Iron Byron" type device to test shafts. That would eliminate a lot of variables that human testing creates.

I believe that this test should be done only for the Driver. You could categorize shafts by swing speeds (i.e. SS of 100-110, best shaft for distance is XYZ shaft.) It could even be broken down into "Best for Distance" "Best for Accuracy" and "Best Overall Shaft"

As far as what clubheadand golf ball to use I would leave that up to the Golf Digest, but why not use the previous years "Editors Choice" Driver Clubhead and "Editors Choice" Ball?

Those are my suggestions for shaft testing...Gouge feel free to send me an e-mail so I can give you my address for those Pro V1's ;)

BTW I am still waiting for my copy of the "Hot List" to arrive in the mail. Hey Bomb or Gouge do you think you can tell GD distribution to hurry up and get me my copy ;)

GOUGE: We'll get circulation on the case. A robot is consistent, but it does not stress shafts the way a human swing does. Very simply, the shaft does not matter to the robot, as a robot can make almost any shaft work. As for using one head as the control, who's to say certain shafts might not work better with certain heads. Would that be fair?

Posted by Scott January 10, 2007 11:58 PM