Which is a better measure of university quality(mostly research) - citations per faculty or reputation?

Question

I did my own simple experiment and looked at the correlation between the two metrics for the top 19 or so universities for per capita citations, using the THES data for 2011. 0.0081 was the coefficient; in other words, the two metrics were uncorrelated. For some universities, the reputation score was even missing because it was out of the 100 in terms of reputation but in the top 25 in terms of citations per faculty and so on - a situation which strikes me as strange because intuitively a university is high in research quality(which is supposedly measured by citations per faculty) will undoubtely have high reputations in that field, and in combination, in a variety of fields, which will translate to the reputation of that university.

Another observation was that while per capita citations were pretty mildly downward sloping as the rank went down, but that reputation was a winner-gets-all kind of thing; Harvard's reputation was three to four times greater than that of the tenth ranked university, say Yale or Princeton or Caltech's.

I'm leaving out other metrics, such as total citations, number of phds granted, phd student to undergraduate student ratio, student to faculty ratio, employer ratings, industry income per capita and so forth - I think these bias in favor of larger and research oriented universities, or are otherwise cursory or irrelevant to the task of ranking a university.

Answer 1

Hyun Jae,

 

Several observations:

 

Presumably, one is interested in quantifying University research quality to estimate its current state and project the likely near-term future change of that quality. 

 

Therefore, university reputation would be a lagging explanatory variable based on past achievements and perceptions that may no longer be warranted.  For example, City College New York (CCNY) had a very high academic reputation in the 1940s and 1950s that got severely damaged by changes in student and faculty recruitment policies initiated in the late 1960s.  But even an outdated reputation can have value for job applications, career networking, and social interactions. However, if one is interested in selecting a university to pursue a Phd or research, then it is more valuable to know where good research work is being done now and where top-flight faculty & equipment are currently.

 

Per capita measures are crude and can be very misleading.  A large capita in the denominator as in most State universities will water down any islands of excellence that may exist in individual departments.  Thus, a smaller, focused university (say M.I.T.) will show up better in its area of focus (technology & science) than a broad university, in which science & technology may only represent say 20% of its activities, no matter how excellent.

 

Published citations are also problematic given the pressure to publish and the trend toward multiple authorship as well as reduced standards of publications.  How to get through this clutter?  Perhaps, the number of citations above a baseline number that would vary by area of research or department.  Then, the more important or seminal publications will most likely be cited over and over again.  Of course, this may not work well for small or emerging areas of scholarship.

 

The goals should be relevance and timeliness. What are you searching for, is it there now, and will it remain vital later?

These get submerged and lost when aggregated to the university or even the department level.  

 

So it is probably most sensible to rely on current practitioners in each specialty, who are most familiar with the current research and by whom and where it is being done, to rank the most productive places for pursuing such work.  Thus, where are the 10 best overall math departments, or places to pursue topology topics?  Then, the top 3 can be considered the best, the next 4 as very good, and the bottom 3 as good for those that want to do state of the art research and be in the "inner circle".  For others, there are many other universities that offer quality programs, that are "lesser" in the sense that one's work is being done outside the "inner circle" and thus one is not as well or as immediately connected. Of course, this does not preclude in doing excellent work.  For example, Einstein made his breakthroughs in Physics while working in a government patent office.  But all he needed was mind, creativity, pencil & paper. What if you need access to expensive or rare equipment?  Good luck if you're not connected.