Know When You're a Loser
I recently attended the DEMO Conference in Scottsdale, Arizona. Rather than a bunch of bleeding edge consumer devices and web applications being unveiled this year, DEMO was loaded with enterprise apps. In particular, there were a number of companies dealing with email management (search, sorting, filtering, etc.). And of particular interest to the DEMO audience was Spam filtering.
A centerpiece of Monday's program (2.17.03) was a "Spam Challenge" between two spam-fighting startups, MailFrontier and Cloudmark. The two anti-spam startups agreed to allow representatives of NetworkWorld Mangazine to independently test the effectiveness of their respective products. The good folks at Network World determined that MailFrontier was decidedly more effective than Cloudmark at achieving a high filtering rate while maintaining a relatively low false positive rate. What was most interesting to me wasn’t the fact that Cloudmark got spanked – it was more the fact that Cloudmark didn't seem to know that they were going to get spanked. Moreover, because Cloudmark was directly involved in NetworkWorld's testing, it gave that reasearch Cloudmark's own imprimatur despite the negative outcome for the company.
The Spam Challenge reminded me a bit of an independent study that was jointly commissioned some time ago by two competing photo printing websites, Ofoto and Shutterfly. Ofoto and Shutterfly split the cost of an InfoTrends study on photo print quality among 5 leading online print sites. It turned out to be a great deal for Ofoto. InfoTrends determined that 75% of people surveyed found Ofoto's prints the highest in quality, compared to the mere 10% who chose Shutterfly (the second highest in the test). Worse yet for Shutterfly, they had validated the study by co-commissioning it with their biggest competitor.
Both these independent studies beg the question, didn't the losers know they were going to lose? And if they didn't, why not? The first thing a young prosecutor learns about cross examining a witness is to never ask a question to which she don't know the answer already. I would think the same should apply here. Never endorse an independent comparison of your product with a competitor's unless you are absolutely certain that you will come out on top.
