The AmLaw Associate Satisfaction Survey: Data or Inkblot?

The American Lawyer's annual associates satisfaction survey is here (full version) and here (summary ranking), and I'll have some deeper analysis coming up.  But for now, my initial reaction as an armchair statistician and amateur student of market research methodology at the feet of my wife, the marketing executive, is that the viewing this ranking as "quantitative" research—statistically reliable, reproducible, verifiable, etc.—may be the wrong view.  Because of the enormous variation year to year (a striking 15 of the top 25 this year weren't in the top 25 last year), I'm inclined to view it more as "qualitative" research—meaning that it's a genuinely valuable exercise in exploring associates' attitudes, gaining insight into what matters for morale and associate development, and similar issues, but not a definitive exercise in ranking.  A focus group, in other words, not a Gallup poll.

That said, I know enough about the good folks at The American Lawyer to applaud their diligence and their ingenuity at undertaking this ambitious and important project.  I'd be curious to hear if any readers have a similar initial take to mine.