1. An unpromising starting point

In a 2018 article, Alberto Baccini and Giuseppe De Nicolao described the Italian academic system as “an unprecedented in vivo experiment in governing and controlling research and teaching via automatic bibliometric tools”. Italian universities and research institutions are subject to widespread bibliometric measurements. Their use in research evaluation exercises (Valutazione della qualità della ricerca or VQR) and in the national scientific qualification for professorships (Abilitazione scientifica nazionale or ASN), which we will discuss, are just two of many examples. But the Italian research evaluation machine is not only bibliometric: it is also administrative and centralized.

ANVUR, the Italian National Agency for the Evaluation of the University and Research, plays a central role. ANVUR is neither an autonomous agency nor a quango run at arm’s length by the government. It is instead a governmental agency: its board consists of seven professors directly nominated by the Minister of Education [now: Ministro dell’università e della ricerca or MUR]. Moreover, ANVUR acts principally by implementing activities directly defined by ministerial decrees, such as: research assessment exercises, quality assurance for teaching, evaluating the administrative tasks of universities, and assessing the qualifications of candidates for professorship. Among similar European institutions, such as AERES in France or ANECA in Spain, none concentrate so much power and so many functions in one place.¹

Against this background, the agency’s signature of the COARA agreement and the presence of one of its board members on the COARA steering committee seems surprising. It is true that, after having locked Italian research in a steel cage, in violation of Article 33 of the Italian Constitution, only the ministry (MUR) and the agency (ANVUR) can repair the damage they have caused. But are they willing to do so? And does its subordination to the government really give the agency the power to honor the commitments it has formally signed?

2. A disappointing first step

As reported in our August statement, ANVUR has ruled that Open Research Europe (ORE) is not a scientific journal because it does not publish in “clear, self-contained fascicles that are not open to further updating” and because it does not practice anonymous peer review. But the substance of this decision, controversial as it may appear, is not so important. Even if the ANVUR had declared ORE to be scientific, the core issue would have remained the same: in Italy, it is not the researchers who decide what is scientific and what is not, but a government-appointed agency, whose many tasks include identifying which journals in the humanities and social sciences are scientific and which are not, and which of the latter are “excellent” (A rating or “di classe A”) and which are not.
Since publications in journals declared scientific or excellent by the agency are necessary for the career of Italian researchers, the very existence of this bureaucratic system of lists contradicts the COARA principles signed by ANVUR.

1. If what is scientific and what is not is determined by lists drawn-up by a government-appointed body, the freedom of researchers to choose the topics, methods, theories, and applications of their investigations is trapped within the agendas and editorial policies of journals approved by the ANVUR, against the safeguard of the freedom of scientific inquiry which is the content of the second principle for overarching conditions.²

2. The first COARA principle for evaluation criteria and processes links research quality to openness as “early sharing of knowledge and data and open collaboration”.³ But openness will continue to be a lip service if what is scientific and what is not is decided by a government-appointed authority in a zero-sum game designed to pit researchers and institutions against each other.

Moreover, the ANVUR specifically violates at least three of the COARA commitments:

1. First commitment (p. 5): if evaluation should be about content rather than containers, why continue to compile lists of journals administratively labeled “scientific”?

2. Second commitment (p. 5): if the main method of assessing quality should be rigorous and transparent peer review, how can transparency be possible when the only type of review that ANVUR considers “scientific” is anonymous?

3. Third commitment (p. 6): if the “inappropriate use of journal- and publication-based metrics in research assessment” is to be abandoned, the ANVUR lists should also have been dropped, since they are used to evaluate researchers on the basis of the number of publications and thus on the basis of journal- and publication-based metrics.

3. An inconsistent conclusion

In October 2023, it became clear that ANVUR’s rejection of ORE was no accident. Bibliometrics, based either on the H-index or on citations or on the list of journals compiled by ANVUR, was and still is the cornerstone of both the national research evaluation exercise (VQR) and the national scientific qualification for professorships (ASN).

If we look at the last calls of ANVUR, we can see that nothing has changed significantly.

The call for the next National Research Assessment Exercise (VQR 2020-2024) implements the guidelines established by a previous decree of the Ministry of University and Research, even if it includes at least two patent violations of COARA commitments.
ANVUR will draw by lot the 75% of the members of the panels of expert evaluators, bureaucratically known as “gruppi di esperti della valutazione” or GEVs. In order to be included in the pool from which the lots will be drawn, candidates should meet quantitative thresholds based on the calculation of their citations and H-index in the STEM fields, and on the number of articles published in the above-mentioned lists of journals in the SSH fields (Art. 3.3), against the letter of the third commitment (p. 6). Furthermore, the first paragraph of article 7 of the VQR call allows the use of bibliometric indicators in the evaluation of research, although “such indices cannot replace an accurate evaluation of the merits of the research product, nor can they lead to the automatic assignment of the product” to an evaluation category (art. 7.2). Such a clause, however, can be compatible with any kind of purely bibliometric evaluation, provided that it is reformulated in ad hoc qualitative terms.

The drawing of lots is not a tribute to ancient democracy’s mistrust of the aristocratic risks of elections. The remaining 25% of the members of the GEVs will be appointed by ANVUR, again on a bibliometric basis (art. 3.4); ANVUR will also supplement the GEVs (art. 3.6)  if the procedure does not produce groups that meet the requirements of article 3.4, and will appoint their coordinators (art. 3.16). Finally (art. 3.22), the ANVUR board may replace the members of a GEV in the case of “critical issues that have arisen during the in itinere review of the progress of the evaluation process, after hearing the coordinator” – appointed by it – “(with special reference to failing to perform assigned work, delaying, or violating the Code of Ethics)”. Despite the drawing of lots, the strategic appointment of the coordinators, the possibility of dismissing evaluators almost at will and the very weakness of officials who are chosen by lot rather than elected place the entire VQR exercise under the hierarchical control of the ANVUR board.

As far as Open Access is concerned, the VQR call takes away with one hand what it seems to give with the other. Article 8 (first paragraph, point a) requires open access “in the case of publications dealing with research results financed at least 50% by public funds and, in general, for all publications for which the publisher allows it.” However, access can be postponed according to the extraordinarily long embargo of Law 112 of 2013, of one and a half years for STEM and of two years for SSH. And point (b) of the same subsection, which regulates access to reference metadata “for products related to research results that have received less than 50% of their funding from public funds, or with embargo periods longer than those indicated in point (a), or in all cases where dissemination is not authorized by the publisher” – suggests that ANVUR’s open access will take place on Tibb’s Eve, at the discretion of publishers. Oddly, the opinion of the original copyright holders, the authors, seems to be irrelevant.
It is worth recalling that Italy is also a longstanding violator of the 2012 Recommendation on access to and preservation of scientific information (C(2012) 4890 final) (C(2012) 4890 final), which was one of the cornerstones of the EU Open Access policy. To avoid the crystallization of rules that should and could be improved, the ANVUR calls could have more simply required that if a publication is already legally Open Access, it must be uploaded as such to the ministry platform.

And what if an Italian scholar wants to apply for national scientific qualification for professorship (ASN)? According to the latest ministerial decree, he or she must meet at least two quantitative thresholds (“indicatori di impatto della produzione scientifica”, Art. 2.4 c), which have nothing to do with Open Access, but are still determined by proprietary or administrative bibliometrics. In the STEM fields, the thresholds are the H-index, the number of articles and the number of citations, as obtained from two proprietary databases, Clarivate Analytics and Scopus. For SSH, the thresholds are the number of books, the number of articles published in “scientific” journals, and the number of articles published in “excellent” (“di classe A”) journals. As mentioned above, in Italy a journal can only be recognized as scientific or “excellent” if it is included in ANVUR’s lists.
These bibliometric requirements are not part of an evaluation that includes qualitative assessments as well, as in a responsible use of metrics: if scholars do not meet the required thresholds, their application must be rejected, even if they were Nobel laureates.
And again, the commissioners who will judge them are also drawn by lot from a pool of candidates who must meet some stricter bibliometric requirements, without regard to the open accessibility of their works.

4. Some final questions

Why did ANVUR sign the COARA agreement but failed to fulfill its commitments?
The ANVUR employees could justify themselves by saying that they are obliged to follow the orders of the Ministry according to the Italian laws and administrative regulations in force. Yet, if this is the case, why did they sign an agreement that they have no authority to apply? Why does an ANVUR official sit on the COARA steering board? In what capacity does the ANVUR participate in the meetings of the Italian National Chapter?
On the other hand, if the ANVUR has at least some power to break the bureaucratic chains that imprison Italian research, why has it not used that power? Did it sign the agreement from the beginning with the hidden intention of changing things only to keep them the same?
In any case, whether the agency does not respect its signature because it cannot or because it does not want to, two questions are hard to avoid. Is its participation in the steering board and in the Italian national chapter still compatible with the COARA agreement? And how can COARA ensure that its members honor their commitments if they seem unable or unwilling to do so?

Update on ORE (January 19, 2024)

This text is licensed under CC BY-SA 4.0 license

1. Alberto Baccini and Giuseppe De Nicolao, How Pseudoscientific Rankings Are Distorting Research, 2018
2. Agreement on Reforming Research Assessment, I , p. 3.
3. Agreement on Reforming Research Assessment, I , p. 3.

Accessi: 730