Wednesday, April 30, 2008 by Niels Hartvig

By default I don't trust companies that produces market analyzes, research, benchmarks, evaluations and likewise (let's just call them benchmarks) and I rarely trust companies who claim to be impartial.

There's exceptions and my rule of thumb for trusting an impartial benchmark is openness and transparency on the economic aspects. That's an easy - albeit not bulletproof - way to test the level of neutrality. A report financed by the American car industry is as likely to promote Japaneese Hybrids as I am to promote Silverlight for the umbraco UI ;-)

This is why I have a principle of never paying to get umbraco evaluated. Today that principle just got umbraco excluded from a Benchmark that usually been good to us. But Benchmarks should be financed by the users, not by the participants that gets evaluated, as it's the only way to ensure neutrality. If we were to finance the report, we would take the money from our (tiny) marketing budget - and I think that says it all. It would no longer be an evaluation, it would become an ad in disguise.

3 comment(s) for “Benchmarks, evaluations, principles and credibility”

  1. Neil Says:

    Who was it? Tell tell!

  2. Søren Sprogø Says:

    Good choice!

    And I bet the finished Benchmark Study won't mention anything about how it's funded :-)

  3. Peter Gregory Says:

    Good on you for not participating. Why should you pay to be evaluated? Makes no sense.

Leave comment:

Comments are disabled for older posts