What a reputation survey measures

The survey instrument

Every major global ranking includes a peer-assessment component. In the QS World University Rankings it accounts for forty percent of the total score — the single heaviest indicator. The Times Higher Education World University Rankings assigns a smaller share to academic reputation but still treats it as a distinct input, gathered by annual questionnaire sent to tens of thousands of scholars worldwide. The mechanics are straightforward: a respondent selects a discipline, names the institutions she considers excellent in research within that field, and submits. The aggregated nominations become a score.

What the instrument cannot easily distinguish is whether a respondent is recalling actual engagement with an institution's output — reading its papers, attending its conferences, tracking its faculty — or recalling its position in last year's published table. The distinction matters because the two sources of information produce very different things. Direct engagement is a genuine signal. Recalled rank is circular: the score reflects the ranking, and the ranking then reflects the score.

Autocorrelation and what it documents

Autocorrelation, in this context, means that an institution's reputation score in year n predicts its score in year n+1 more reliably than any change in its underlying research activity does. This has been measured. Scholars examining QS data over multi-year periods find that reputation scores are highly stable across consecutive editions ↗, moving far less than research-output or citation indicators in the same period. An institution that scored well in reputation last year almost certainly scores well this year, regardless of what its faculty published in the interval.

This stability is not, on its face, evidence of circularity — reputations built over decades can be legitimate and slow-moving. Harvard, MIT, and Oxford carry research traditions that no single year's output overturns, and a respondent recalling their standing is not necessarily wrong. The problem is that the survey cannot separate the two cases. A university that climbed the table on the back of a heavy citation-boosting strategy — or a temporary spike in indexed output — will accumulate reputation survey nominations that then persist long after the underlying driver has faded. The score hardens around an impression that may already be out of date.

The population of respondents compounds this. Survey invitations go out to large but not randomly selected samples of academics. Response rates vary substantially by region and discipline. Scholars at highly ranked institutions are, by some analyses, more likely to respond, and they are also more likely to name institutions within their own network of conferences and collaborators — which skews toward the same institutions already prominent in the table. The result is a reputation survey ↗ that reflects the sociology of academic networking as much as it reflects research quality.

What the score is actually measuring

Strip out the circularity and the network bias, and what remains is a form of brand recognition — something real, but not the same as educational or research excellence. A very large, very old institution in an anglophone country has structural advantages in both dimensions: more alumni dispersed through global academia, more publications in English-language journals indexed by the databases that citation indicators also draw on, and more decades of table positions for respondents to have absorbed.

This is why the reputation component of a weighted composite ranking behaves differently from the bibliometric components. Citation counts and faculty-to-student ratios can shift meaningfully in three to five years if an institution changes its hiring or publishing strategy. Reputation scores do not. They act as an anchor that prevents the composite from moving as much as the other indicators would push it — which can be read either as stability or as inertia, depending on which direction the underlying data has moved.

Ranking producers are aware of the critique. QS has adjusted its methodology periodically, and THE has altered how it weights its reputation data. Neither has abandoned the survey instrument, because the alternative — a composite built entirely on bibliometric proxies — carries its own distortions, systematically favouring STEM fields and high-volume publishing over humanities scholarship and pedagogical innovation. The reputation survey stays because nothing else currently measures what it is trying, imperfectly, to reach.

Citation counts and faculty-to-student ratios can shift meaningfully in three to five years if an institution changes its hiring or publishing strategy.