Fundamental to forming convincing, critical arguments is skeptically questioning information presented as fact.
Regardless if the issues are about government bailouts, the college football BCS, or if user interface design and interaction design are one and the same, a critical thinker identifies and reacts against twisted statistics or flawed logic.
However, it’s difficult to recognize fishy data published online by unfamiliar albeit seemingly reputable institutions. Even more challenging is understanding the value of their conclusions when these reports are cited by others, referenced in research, or included in legitimate press releases.
Self-Promotional Usability Marketing Reports
For example, the Australian business technology web site iTnews recently published “E-Commerce Sites Failing on Simple Usability“, which describes the results of testing 3,600 volunteers in Europe across 14 different web sites. The article summarizes that “the average success rate for each task among UK users was 52 percent at best, and that the figure fell to around a third for some sites”.
Unfortunately, we’re not privy to what those tasks were, save for a brief mention of when shipping costs are provided and how to return damaged goods.
But it’s important to realize that returning damaged goods is a niche, fringe use-case. And without knowing what is exactly a failure, we cannot determine if the user simply made a few wrong turns but ultimately found his way, or if he threw the computer out the nearest window in frustration.
David Hamill of Good Usability also expresses his skepticism with the assigned tasks, commenting that “somehow I doubt it was ‘Buy an iPod on Amazon’.
Most importantly, we can’t even be sure of the integrity of the research. Since a usability company partnered with a research firm, and was not commissioned by one particular company to provide in-depth analysis, we can skeptically presume this report is just to publicize each both names to drum up future business.
It’s easy to believe a scenario where an unscrupulous company could provide a web usability test designed to stump volunteers, then in turn go to that company whose site just “failed” their test and offer to provide recommendations to increase their site’s usability. Is this the case here? I don’t know, but unfortunately we can’t ignore the possibility.
Web Site Rankings
Many web site top-ten lists select “winners” based on criteria that’s either subjective or based on irrelevant data.
For example, the “Webometrics Ranking of World Universities” is a project by the Cybermetrics Lab, a research group belonging to the Consejo Superior de Investigaciones Científicas (CSIC), the largest public research body in Spain.
The study’s mission is to measure
the activity and visibility of the institutions and it is a good indicator of impact and prestige of universities. Rank summarizes the global performance of the University, provides information for candidate students and scholars, and reflects the commitment to the dissemination of scientific knowledge.
Though I’m clearly not in a position to question the quality of Spain’s largest public research consortium, we should read this particular report with a skeptic eye. Even this particular study’s mission statement is enough to alarm the casual reader. Can a web site adequately indicate the “prestige” of its university?
And while I understand this study originated in Spain and there could be some meaning lost in the translation to English, I’m left to assume the rank “summarizes the global performance of the [University's]” web site and not the University itself.
If not, then the folks at Yale (39), Duke (33), and NYU (28) were bested by the likes of my own alma mater, the University of Maryland (at 19 and one step ahead of the University of Chicago, 20).
Even worse is when we dig deeper into how these results were founded:
Four indicators were obtained from the quantitative results provided by the main search engines as follows:
Size (S). Number of pages recovered from four engines: Google, Yahoo, Live Search and Exalead.
Visibility (V). The total number of unique external links received (inlinks) by a site can be only confidently obtained from Yahoo Search, Live Search and Exalead.
Rich Files (R). After evaluation of their relevance to academic and publication activities and considering the volume of the different file formats, the following were selected: Adobe Acrobat (.pdf), Adobe PostScript (.ps), Microsoft Word (.doc) and Microsoft Powerpoint (.ppt). These data were extracted using Google, Yahoo Search, Live Search and Exalead.
Scholar (Sc). Google Scholar provides the number of papers and citations for each academic domain. These results from the Scholar database represent papers, reports and other academic items.
The problem, of course, is that we cannot evaluate the scholarly value of a document based only on Google Scholar or simply count how many PowerPoint presentations exist on a web server to infer said “global performance”.
Sheer quantity of pages is also irrelevant, since we do not know how many of these pages are scholarly research and how many pages are sports highlights from a 1993 fraternity-league flag football game.
Gauging ranked universities’ reactions to this report is also worth monitoring.
For instance, MIT, the list’s #1 university web site based on this ‘research’, published their top ranking in an internal staff newsletter in a brief blurb, whereas Stanford and Harvard apparently make no mention of the report at all, based on site searches (I’m such a sleuth. Sign me up for CSI-DC).
And what of my undergraduate institution, the home of the Terrapins? Well, they went the opposite route and published a press release, aptly titled UM Ranked Among Globe’s Most Cybersavvy Universities.
