Turning the tables: A university league-table based on quality not quantity

Criteria used by league tables

The International Ranking Expert Group (IREG) audit university league tables and aim to strengthen public awareness and understanding of university rankings. A recent inventory by IREG found 17 international league tables⁶, although two are based solely on web traffic and one concerns environmental sustainability. Of the remaining 14 tables, 12 use publication numbers, and 12 use citations.

Although papers and citations are commonly used, every league table uses their own method to count them. Variations include:

The differences between league tables could reflect genuine differences of opinion in the best way to use the data. It could also be somewhat due to a desire by league tables to differentiate themselves and so produce novel results. It could also be because papers and citations are imperfect proxies of quality, and so there are multiple opinions on how best to refine them.

Criticisms of league tables

A seminal paper on institutional ranking (including hospitals and schools) in 1996 by Goldstein and Spiegelhalter stated that responsible rankings, “may provide relevant information to universities, students, funders and governments”⁷. However, they also cautioned about the need to consider data quality, uncertainty in the rankings, gaming by institutions, and unwarranted conclusions based on small changes in ranks. A report on the use of public league tables recommended that every table should have an appropriate and prominent “health warning” about their limitations⁸.

The criteria used by university league tables have been criticised for lacking construct validity⁹ and for experiencing implausibly large changes from year to year¹⁰, some of which were due to calculation errors and methodological changes¹¹.

A report on the use of citation statistics warned that “citation data provide only a limited and incomplete view of research quality”¹². An analysis of misprints in citations suggested that most researchers simply copy citations without reading the actual paper¹³, which undermines their face validity as a ranking criteria. Citations and paper numbers can be gamed^14,15, and gaming by researchers can greatly alter a university’s ranking¹¹. Concerns about the misuse of simplistic metrics in research led to the Leiden Manifesto in 2015, which set out ten principles for the proper use of metrics for evaluating researchers and institutions¹⁶. In 2017 the Leiden group created ten more principles for responsibly ranking universities¹⁷, which included transparency and acknowledging the uncertainty in rankings.

Good research practice

To our knowledge, no current international league table includes a measure of best publication practices, by which we mean established methods that increase the robustness, transparency and reproducibility of research. There is an international league table of potentially questionable research practice, which is the Retraction Watch table of individual researchers ranked by their number of retracted papers¹⁸.

Examples of good research practice are:

Unlike the traditional metrics, such as the number of publications, used by current league tables, these metrics are prerequisites to solving recognised problems in science. Recent evidence points to a growing reproducibility crisis in many fields of research, which is only possible to examine when sharing of data, code, materials and methods takes place.

Good research practices help reduce research waste, which can occur when researchers cut corners in order to progress in the “publish or perish” game. Avoidable research waste is an enormous problem and an estimated 85% of the current investment in health and medical research is wasted due to poor research practice, which is billions of dollars per year²³.

In this paper we examine one of these good research practices by examining when authors cited an EQUATOR reporting guideline²⁴. EQUATOR stands for: Enhancing the QUAlity and Transparency Of health Research, and they are a wide-ranging suite of more than 400 guidelines that cover every common research study design. There is evidence that using a reporting guideline improves the quality of the published paper^25,26. Our key assumption is that citing the guideline is an indicator of good research practice. An important difference from our approach compared with previous league tables, is that we reward the universities whose researchers give the citation, not the universities of researchers who receive the citation.

There are four EQUATOR centres around the world (UK, France, Canada and Australasia) with the aim of promoting the use of the guidelines worldwide. Many of the most commonly used EQUATOR guidelines have been translated into multiple languages.

There is a wide literature on rankings and university league tables including discussions of policy²⁷, design²⁸ and statistical critiques⁷, as well as systematic reviews²⁹ and books³⁰. We do not review this literature in detail, as our primary aim was to identify whether a league table could be constructed based on good research practice.

Papers included

We counted papers that cited one of the EQUATOR guidelines for clinical trials (CONSORT)³¹, systematic reviews (PRISMA)³², and observational studies (STROBE)³³. We chose these three guidelines because they cover three commonly used study designs. Each guideline was published simultaneously across multiple journals, which was done to increase their reach into multiple fields. We therefore counted citations to any of the original papers or updates to the guidelines (see Supplementary List 1)³⁴. If a paper cited multiple EQUATOR papers, then only one was counted.

To include only papers that adhered to the first item on the CONSORT and PRISMA guideline check-lists, which is to include the study design in the title, we only included papers that included the following in their title:

We did not include a restriction for STROBE papers because there are many observational study designs and any list we created might exclude valid papers.

To focus on original research, we included publication types of Articles or Reviews, and excluded Editorials, Commentaries and Corrections.

We aimed to sum citations per year and we examined the two most recent complete years of data by using papers published in 2016 or 2017.

We used Scopus to identify citations because it is a recognised database for citations that is used by three international league tables, and because of the ease of extracting the data using the rscopus package in R³⁵ (Version 0.6.3). We used the rentrez package in R (version 1.2.1) to extract meta-data on the papers from Pubmed³⁶. Papers were excluded if they did not have a digital object identifier (DOI), because this was the key linking variable for extracting the affiliation data. The data extraction from Scopus was performed on 19 December 2018.

Cleaning affiliations

We extracted all authors’ countries and affiliations. The affiliation data is free text and required extensive cleaning to extract a standardised set of universities. Affiliations were changed to:

We changed vague affiliations to missing, for example “Faculty of Health”.

We standardised affiliations to ensure that citations were consolidated into a single university rather than being split over two or more universities and hence creating a falsely low position in our league table.

A flow chart of the data collection and management is in Supplementary Figure 1³⁴.

Creating our league table

To create a score per university, we summed the total number of citing papers per university per year. To better divide the credit from a citation, we used an organisational-level fractional count of author affiliations per paper³⁷. So, for example, if a paper had two affiliations in the address list, one from Queensland University of Technology and one from Ottawa Hospital Research Institute, then each university would gain 0.5. A fractional count avoids the situation where universities gain a full point even when their staff member was only one of multiple authors.

We examined the amount of missing affiliation data by country to look for biases in the affiliation data that may disadvantage particular universities or geographic regions in our league table. We also included “Missing” as a separate university, in order to show the relative importance of missing data.

We accounted for uncertainty in our league table using a bootstrap procedure³⁸. We randomly resampled with replacement from all the citing papers and recalculated each university’s score and rank. We repeated this resampling 1,000 times. To summarise this uncertainty we created a bootstrap 95% confidence interval for the rank.

We examined changes over time by comparing the ranks of universities in the top 200 in 2016 and 2017. We used a Bland–Altman plot to examine how ranks changed between these two years³⁹. For comparison, we also used a Bland–Altman plot of the THE World University Rankings using their research criterion, which combines a reputation survey, data on research income and paper numbers⁴⁰.

We qualitatively self-assessed our league table against the ten principles for responsible ranking from the Leiden group⁴¹.

As a comparison to our good research practice table, we created a standard league table based on counting each university’s papers for the years 2016 and 2017. We counted articles only, not books, editorials or letters. To match our good practice table which is focused on health and medical research, we only included papers in the three subject areas of Dentistry, Health Professions and Nursing. These data were from Scopus.

Clustering universities into similar groups

We present our results as a table using the total score per university per year and give an integer rank to universities in each year. This implies a monotonic order, where each university performed better than the university below it. This is unlikely to be true, and to give a better impression of performance we used clustering to group universities into five clusters. We chose five as an a priori opinion of the number of meaningful clusters. We used a Bayesian clustering model defined as:

where S(i, t) is our score for university i in year t. The five cluster means ($\overline{x}$) are ordered from low to high. For each university we estimate their cluster, c(i, t) ∈ c(1, 2, 3, 4, 5), which comes from a categorical distribution with five probabilities π(1), . . . , π(5). These probabilities came from the sum of five uniform prior distributions which were formulated so that the minimum probability for each cluster was 1% (π ≥ 0.01). This was an attempt to avoid small clusters of just a few universities. We only applied the clustering algorithm to universities with a score of 2 or above, which removed the large number of universities with small samples sizes and low scores. We cross-tabulated the median clusters by year to show how many universities changed between 2016 and 2017.

The data extraction and analyses were made using R version 3.5.2⁴². The clustering model was fitted in WinBUGS (version 1.4.3)⁴³ and we visually checked the mixing of the Markov chain Monte Carlo estimates. The data and code that created the tables is available here: https://github.com/agbarnett/league.tables.

In summary, the aim of our table was to score universities using the EQUATOR guidelines, with higher scores indicative of better research practice. We also included measures of uncertainty via the bootstrap and attempted to cluster similar universities. We report our results using the STROBE guidelines³³.

Missing affiliations

The number and percent of missing affiliation data are shown by country in Table 1. If the country was missing then the affiliation was also likely to be missing. The most amount of missing data was in the USA. Overall the percent of missing affiliation data was small, at just 0.5% of all affiliations.

Highest ranking regions and countries

Before examining institutions, we first examine the scores by regions and countries, and the top ten regions and countries are shown in Table 2. The rank order of the top ten was the same for the regions and countries, except for the tenth ranked country, which was Denmark in 2016 and Spain in 2017. Every region and country in the top ten had a higher total score in 2017 than 2016, reflecting an increased use of the EQUATOR guidelines. The highest ranking regions and countries in the table are familiar producers of research.

Highest ranking universities

The top ten universities in each year are in Table 3. We have presented the scores in this paper to one decimal place, but would use rounded integers in public tables to discourage readers over-interpreting small differences. The University of Toronto had the highest score for papers citing the EQUATOR guidelines in both years. Although the proportion of missing affiliation data in the entire data set is small (just 0.5%), “Missing” was in the top ten in both years.

The University of Toronto was ranked highest for good research practice in both years, and there was little uncertainty in this top ranking as the bootstrap confidence intervals were rank 1 to 2 in 2016 and rank 1 to 1 in 2017. The University of Sydney was ranked second in both years.

The clustering model selected only a small number of universities to be in the highest category of ‘5’, despite our attempt to avoid small clusters by formulating a minimum prior probability of 1%. Summary statistics for the five clusters are in Supplementary Table 1³⁴.

There was relatively little movement in clusters between years for the best clusters of ‘3’ to ‘5’ (Table 4). There was more movement over time between the lowest two clusters of ‘1’ and ‘2’. Only two universities moved by two or more clusters, which was from ‘1’ to ‘3’.

The 95% bootstrap intervals were wider for universities outside the top ten. For example, for the university ranked 100 in 2017, the 95% interval was from rank 63 to 176. The width of the interval increased by an average of 13.6 for every 10 increase in rank (95% CI 13.0 to 14.1 using linear regression; see Supplementary Figure 2³⁴). This increase was due to the reduced sample size (number of papers) for lower ranked universities.

The universities in our top 10 had varied results using a standard ranking, with some being in the top 10 and others outside the top 100. Two Chinese universities ranked in the top ten in our good research practice ranking, but were outside the top 100 using the standard table. Erasmus University and The University of Ottawa also did much better on the good research practice ranking that the standard ranking. The Spearman’s rank correlation between the standard ranking and our good practice ranking was 0.59.

Complete tables for all universities with a score of two or above are available online: https://aushsi.shinyapps.io/equator (available until 2020). These interactive tables allow examination of the results by year, geographical region and selected countries. The top 50 universities per year are shown in Supplementary Tables 2 and 3³⁴.

Agreement in ranks between years

We show the agreement in university ranks between years using Bland–Altman plots in Figure 1. For both our league table and the THE table, there was less change in the highest ranking universities, and more movement between years at lower ranks. The Bland–Altman limits of agreement were –60 to 60 in our table and –46 to 43 for the THE table.

Figure 1. Bland–Altman plots of the agreement in university league table ranks between 2016 and 2017 for our good research practice league table and the Times Higher Education league table for research.

We only examine universities in the top 200 in both years, which is 161 in our table and 184 in the THE table. The dashed horizontal lines are the Bland–Altman limits of agreement.

Assessment against the ten Leiden principles for ranking universities

We assessed our Good Research Practice league table against the ten Leiden principles in Table 5.

Future ranking criteria

Lindner et al recently examined whether metrics and incentives could be developed to encourage scientists to use high-quality methods and publish “negative” studies⁴⁴. They concluded that, “If rigorous, innovative studies of significant issues and publication of valid, reproducible results are desired, the best way to achieve those objectives is to explicitly evaluate and reward scientists based on those criteria.”

Lane suggested that new metrics should capture “the essence of what it means to be a good scientist”⁴⁵ and future league tables could include:

However, league tables generally rely on large volumes of data to create scores, meaning these criteria would need to be automated. At present we could only likely automate whether matching data or protocol paper existed, and not whether the data was complete or whether the authors followed the protocol. Detailed data that cannot be automated can be collated on a smaller scale using audits^47,48.

We could expand our criteria to include more of the EQUATOR guidelines, such as the STARD guidelines for diagnostic accuracy studies⁴⁹. Including more EQUATOR guidelines would increase the sample size per university and so would likely reduce some of the variation between years shown in Figure 1.

We did not adjust for the size of the university to produce a relative measure of performance. Hence our table is biased towards larger universities that have more staff, an issue recognised by the Leiden manifesto on metrics¹⁷. An ideal standardisation would be to adjust for the number of papers that failed to cite an EQUATOR guideline when appropriate. This could be used to give an indication of performance regardless of size, and would also show the potential improvement for each university.

One surprising result from our tables was the high rank of “Missing”. This shows the importance of correctly completing affiliations, and universities could increase their rankings (in our table and others) by promoting a clear and consistent affiliation to their staff. We recommend that all league tables report the amount of missing data and show its ranking in their tables. We also recommend, as have others^7,17, that all league tables include a measure of ranking uncertainty.

Limitations

There are many limitations to constructing a university league table, and our tables should be treated as suggestive rather than definitive⁷.

It is impossible to numerically validate our table because there is no gold standard ranking against which we can compare our results. We qualitatively assessed our own performance against the ten Leiden principles, but others may be more critical.

A valid concern with our table is that it would be gamed, with researchers simply citing an EQUATOR guideline without engaging with it. This is very likely to happen, but we cannot estimate the scale of this problem. This is less likely in journals that appropriately implement reporting guidelines because there is an internal check. The harms from such gaming could be outweighed by the number of researchers and universities that genuinely engage with the EQUATOR guidelines. Benefits would likely include greater awareness of the guidelines, and prompting researchers who were already aware of them to use them more rigorously. Complete and transparent reporting has been indicated as an essential prerequisite in dealing with the reproducibility crisis⁵⁰. Some token engagement with a guideline could be spotted by the paper’s peer reviewers, although peer reviewers often have limited time and have an imperfect record of spotting mistakes in papers⁵¹. It may be possible to automate how the paper has adhered to the guidelines and produce a report that is shared with the authors, reviewers and editor(s), and there is an ongoing trial at the journal BMJ Open of such a tool⁵².

The free text affiliation data from Scopus were challenging to process as they were often incomplete and inconsistent. Some universities have multiple versions of their name, including acronyms and English-language versions. We made extensive searches and asked international colleagues to check where consolidations could be made. However, we are very likely to have missed some consolidations, and hence some universities may be too low in our tables because their data has been spread across multiple names.

We tried to examine a correlation in ranks between our tables and those of the Times Higher Education World University Rankings and CWTS Leiden Ranking. However, it was very difficult to correctly merge the data because of the large variation in affiliation names. Just one of many examples is we use “Mayo Clinic”, whereas the Times Higher Education uses “Mayo Medical School”, and this institute is not included in the CWTS Leiden Ranking.

Related study

We could only find one previous related study, which was an international ranking that aimed to measure research quality by using membership on academic editorial boards of professional journals⁵³. They extracted researchers’ names from the websites of 115 economics journals creating a sample of over 3,700 researchers, and created league tables of researchers and universities. Their conclusion was that their table could be used to find experts to evaluate research quality.

Underlying data

A random selection of 500 rows of the data has been made available (see below). The public sharing of data for the purpose of reproducibility with a specific party is permissible upon written request and explicit written approval and the dataset remains with the customer/research. Requests can be made to: integrationsupport@elsevier.com. Zenodo: agbarnett/league.tables: Ready for journal submission. https://doi.org/10.5281/zenodo.2594016³⁴.

Extended data

Zenodo: agbarnett/league.tables: Ready for journal submission. https://doi.org/10.5281/zenodo.2594016³⁴.

Where appropriate, extended data are held under the MIT License.