On statistics and virtue: ANVUR’s criteria for the evaluation of research quality (VQR 2020-2024) and the European reform of research evaluation (COARA)

Posted on , updated on by Maria Chiara Pievatolo

The European Union, recognizing that quantitative evaluation of research harms quality for the sake of quantity, urged evaluators, universities, research institutions, and scholarly societies to join together in a coalition (COARA) for the reform of evaluation itself. Even the Italian evaluation agency. ANVUR, joined the coalition and committed itself to reducing bibliometrics to a complement to qualitative evaluation (peer review), which cannot be done without reading the texts. However, ANVUR has hardly fulfilled its commitments: it continues to use bibliometrics in the five-year evaluation of research quality (Valutazione della qualità della ricerca or VQR) and in the national scientific qualification for professorships (Abilitazione scientifica nazionale or ASN). In the SSH field, a list of scientific journals drawn up by experts appointed by the Agency is and will continue to be maintained, while in the STEMMfields, bibliometric criteria calculated on proprietary databases are still used. Despite the COARA commitments, these criteria are mandatory and not complementary for the selection of candidates and potential commissioners for the ASN, as well as for the eligibility of expert evaluators in the VQR. And in cases where they are formally complementary, such as in the evaluation of works subject to the VQR, the rule can be easily circumvented under the cloak of anonymous review.

Why did ANVUR not honor its signature? One possible explanation is that it is not an autonomous entity, and COARA may have made a mistake in including it in its coalition instead of the Italian Ministry of University and Research. However, the literature produced by scholars who are practically and theoretically close to Italian state evaluation suggests at least one other hypothesis: peer review involves reading of texts, a process that is not scalable. Moreover, that personal idiosyncrasies influence both the selection of evaluators and the evaluations themselves. This may explain why a state and mass evaluation agency such as the Italian one is inclined to cling to bibliometrics As a mass evaluation agency, it needs bibliometrics as a weapon of mass evaluation to maintain its pervasive power. And as a state evaluation agency, it can more easily hide its authoritarian nature behind a veil of statistics.

1. Bibliometrics in the VQR 2020-2024

State evaluation of research, especially when it is as hierarchical and pervasive as in Italy, is an act of mistrust in the freedom of public use of reason. This approach undermines the legitimacy of both state universities and research institutions, which are portrayed as so flawed that they need to be evaluated by a source outside the scientific community. Furthermore, it undermines the legitimacy of the government which enforces it as an administrative measure, without being a scientific authority.

Because state evaluation relies on coercion rather than on science, it is not surprising that many state evaluation agencies employ bibliometric criteria in their assessments. These agencies rely on the quantity of publications, the journals in which they are published, and the number of citations, rather than the quality of scholarship, which is incomprehensible to administrators. Even the European Union has recently acknowledged that the quantitative approach to evaluation results in quantity rather than scientific quality: governments always get what they want, often to discover that those outcomes were not, in fact, what they wanted. For this reason, the EU sponsored a coalition, COARA, with the objective of transitioning the evaluation process to a more qualitative approach, primarily based on peer review. The main commitments of those who join are as follows:

  1. recognize the diversity of researchers’ contributions and careers;
  2. base research evaluation primarily on qualitative evaluations focused on peer review, supported by responsible use of quantitative indicators;
  3. abandon the inappropriate use in research evaluation of journal- and publication-based metrics such as JIF and H-index
  4. avoid the use of rankings of research organizations in research evaluation.

The European reform is about how we evaluate, but it doesn’t address who, for what purpose, and with what legitimacy. Consequently, state agencies such as ANVUR, which have imposed and utilized predominantly bibliometric forms of evaluation, have also joined COARA.

As already noted, ANVUR has effectively disregarded commitments 2 and 3 by imposing mandatory and non-complementary bibliometric requirements on both applicants and evaluators involved in the ASN, as well as on expert evaluators who can be appointed or drawn by lot in the five-year evaluation exercise of universities and research institutions, bureaucratically called VQR 2020-2024.

On the other hand, the VQR 2020-2024 expert evaluators’ criteria meet COARA’s commitments. Indeed,

in the VQR 2020-2024 Research Quality Assessment exercise, the GEV assesses the quality of each product using the peer review methodology.[…] This approach also takes into account the provisions of the Coalition for Advancing Research Assessment’s second recommendation, which states that the assessment should be primarily based on qualitative aspects, for which the role of peer review supported by responsible use of quantitative indicators is central.1<

However, we have to ask whether in the so-called bibliometric fields (STEMM) this respect does not run the risk of being merely pro forma. Citation indicators from expensive databases which are in the hands of commercial oligopolies such as Elsevier (Scopus) and Clarivate Analytics (WoS),2 may “inform” the process of peer review. In other words, they can be used to support peer review, but not to determine it.3 However, there is no impediment to the continued influence of bibliometric data in the secrecy of anonymous evaluation, without the need for in-depth reading and human justification of judgments, since their writing could be entrusted to text generation systems marketed as “artificial intelligence”. ASN commissioners also have a heavy workload, which makes it tempting for them to use such tools. But since their names are known, they have to take public responsibility for the texts that are generated. Why are VQR evaluators granted the privilege of avoiding it?

2. A fragile compromise

COARA Commitments 2 and 3 require that peer review be the primary method of evaluating research, which necessitates the reading of texts. However, this does not preclude the use of quantitative and journal- and publication-based criteria, provided that they are employed “responsibly” and “appropriately”. But if evaluating researchers requires reading and understanding texts, how can we use indicators that require neither reading nor understanding “responsibly” – it is not clear to whom – and “appropriately”? How can we possibly view popularity within an expensive journal system, controlled by a few commercial publishing oligopolists, be considered a complement to peer review? We can also ask ourselves a more radical question: can pursuing research quality really be complementary to trying to get published and cited by the journals in this proprietary ecosystem, despite the unreliability of the literature produced under the pressures of “publish or perish” and bibliometrics, which have the effect of making research “not about curiosity anymore,” but “just a career”?

A recent article,4 The forced battle between peer-review and scientometric research assessment: Why the CoARA initiative is unsound, helps address these doubts. Bibliometrics – it asserts – does not assess the quality of research, but rather its impact. “Like any goods producer, more is needed for a researcher than merely producing high-quality research products; instead, they must be disseminated effectively, akin to the necessity for selling good”: the good researcher, in other words, must know how to sell his or her (paper) products, and bibliometrics measures this ability. Indeed, “it is not about curiosity anymore, it’s just a career.” “Would a company“- the article asks rhetorically – ”ever evaluate the success of a product already launched in the market by convening expert panels instead of relying on quantitative sales analysis?”

It happens in business that mediocre products, effectively promoted, are nevertheless highly successful. But when researchers are forced to sell themselves – or rather, give themselves away – in an oligopolistic pseudo-market because it is administratively imposed and circumscribed, we must treat their research papers as outputs of mass production systems and not as unique pieces of craftsmanship. And when it comes to making mass evaluations of mass productions, the article argues, we must recognize that qualitative evaluation becomes unreliable, if not unfeasible, under these conditions of overload.

So, is the use of bibliometrics as a weapon of mass evaluation inevitable? The answer is yes, but only if we want mass evaluations to continue.

3. A political question

The primacy of peer review, which is among COARA’s commitments, can only be systematically applied under one condition: that mass evaluation be minimized, if not eliminated. And here the interests of ANVUR, whose raison d’être and extraordinary power depend largely on mass evaluation, do not necessarily coincide with those of science.

Not surprisingly, a significant proportion of the agency’s actions have not aligned with its commitments.5 Furthermore, the recent action plan for COARA implementation continues to rely on bibliometric, journal-based evaluation methods rather than content-based assessment approaches. The action plan assumes the continued existence of lists of journals whose scientificity and excellence are directly or indirectly defined by the ANVUR (pp. 6, 7, 10) and the possibility of emancipation from the proprietary databases of Elsevier and Clarivate Analytics is never discussed, not even in the most general terms.

While its action plan (p. 2) presents ANVUR as an independent agency, Roberto Caso (pp. 9 ff.) reminds us that it is, in fact, not legally independent. In 2008, Fiorella Kostoris wrote:

ANVUR’s independence is undermined by its lack of third-party status with respect to the government and by the excessive control exerted by various stakeholders. Notably, all members of its Board are selected directly or indirectly by the Minister for Universities and Research (MUR) and report, inform, propose to the Minister or his department.

In light of the above, it is possible that ANVUR is evading the substance of the commitments it has signed, not because it does not want to, but because it cannot. And that COARA may have been mistaken in including it instead of the body who really makes the decisions, namely the Italian Ministry of Universities and Research.

And yet, on the Italian side, there may be deep-rooted reasons why an agency that clings to bibliometrics and presents itself as independent is a member of COARA. The aforementioned The forced battle between peer review and scientometric research assessment: Why the CoARA initiative is unsound remarks that peer review is heavily influenced by personal bias, and therefore it is better to rely on objective bibliometric experts, possibly supported by SALAMI, who treat scientists as “limited resources” whose use must be “optimized”.

Let us resist the temptation to reply that the use of statistics, whether automatic or not, as weapons of mass evaluation, still aggregates subjective biases. Indeed, authors with a long commitment to state evaluationhave asked: how can research evaluation be reformed in line with COARA standards, to balance or even replace bibliometrics? And they were answered that researchers should be judged on their virtues, i.e. on their motivations and character traits. But in the homeland of Giovanni Gentile, the suggestion that a government-appointed agency be tasked with judging the dianoetic and ethical virtues of researchers may evoke memories – perhaps welcome for some – of the fascist ethical state of the first half of the 20th century.

What would happen if a non-independent, government-appointed agency judged the character of researchers and ranked the most virtuous institutions? It would become much more evident that the state evaluation process is quite authoritarian. Conversely, an esoteric veil of statistics, preferably based on closed and proprietary data, diverts the time of others to tedious administrative procedures and impotent arguments about indicators among subordinates. And this ultimately serves to hide that the Italian research evaluation is not a scientific evaluation, but a state evaluation.

  1. This quotation is taken from article 4 of the expert evaluators panel documents, which can be accessed here.↩︎

  2. Although it is possible to conceive and foster open alternatives.︎ ↩︎

  3. As stated in article 6 of the documents of the expert panels evaluating the so-called bibliometric (STEMM) fields, which can be found here. ↩︎

  4. Its author is a researcher who also works as an official in the service of ANVUR. His positions, just because they are in structural conflict between the Mertonian reasons of science and the administrative reasons of the agency, deserve the greatest attention, at least from the administrative point of view. His article complains that the invocation of a “responsible” use of bibliometrics degrades the professionalism and scientific competence of bibliometricians. And this would certainly be true if the exercise of bibliometrics were only scientific, and therefore open to free discussion and adoption by communities of scholars, and not also administrative, and therefore unquestionable and imposed on the basis of direct or indirect governmental appointment and not on the basis of spontaneous recognition of scientific authority. ↩︎

  5. In addition to the above-mentioned obligatory and non-complementary use of bibliometric criteria for granting participation as a candidate and as a commissioner in the ASN, as well as for the draw and appointment of reviewers in the VQR 2020-2024, we recall the attempts to deny or administratively downgrade the scientificity of open peer review and to minimize the open access requirement with regard to the works evaluated in the VQR 2020-2024, leaving it to the discretion of the publisher and to an extraordinarily long embargo (ASN and VQR after Italy’s accession to COARA, p. 7). ↩︎

This text is licensed under CC BY-SA 4.0 license

COARA action plan of the Associazione Italiana per la promozione della Scienza Aperta

Posted on , updated on by Maria Chiara Pievatolo

The only evaluation activity carried out by AISA is that related to the awarding of a prize for theses dedicated to Open Science.

The prize is awarded by a panel of three commissioners appointed by the President in consultation with the Board, whose names are made public. The panel selects the theses relevant to the topic and makes a collegial assessment based on a justified implementation of qualitative criteria clearly stated in the call for applications.

In the spirit of Open Science, in order to participate in the call, theses must be made available in a public institutional or disciplinary repository, i.e. managed by libraries, universities and research institutions, under an open license. We recommend, but do not require, the Creative Commons Attribution-Share-Alike license.

If the research of the theses depends on new data, and it is legally possible to do so, we also ask that the data be made available, preferably in FAIR format. The data format is a recommendation, not a requirement, so as not to discriminate against young scholars whose university has not provided support in this regard.

These principles, if not already in place, will be included in this year’s call for applications, which will be published in the summer of 2024.

ANVUR’s rule: state evaluation and open peer review in Italy

Posted on , updated on by Maria Chiara Pievatolo

This summer the Italian National Agency for the Evaluation of University and Research Systems (ANVUR) denied scientificity and excellence (“classe A”) to Open Research Europe (ORE) for general sociology. Recently, however, ANVUR has updated its rules for classifying journals, adding an article 18 entitled ‘Transitional regulation for open peer review’.

The ORE affair was the first step in ANVUR’s systematic violation of the principles of COARA, the European coalition for the reform of research evaluation towards more open science and a less irresponsible and extensive use of bibliometrics, of which ANVUR is a member. And it was also a diplomatic incident: by denying the scientific quality of ORE, ANVUR told Italian sociologists who wanted to pursue an academic career in Italy that publishing their texts on a website set up by the European Commission to submit articles written by recipients of European funding to open peer review was tantamount to throwing them in the trash.

However, ANVUR now states, albeit provisionally, that

1. As an alternative to the requirements of Art. 13 c. 4.1, for journals using open peer review procedures, the following requirements shall be provided, subject to compliance with the previously defined rules on ethics and management of conflicts of interest
a. the tracking of all versions of papers produced during the review process
b. the assignment of a specific status to papers that have passed at least two successful peer reviews and their subsequent indexing;
c. clearly indicating any additions, changes, or corrections made by authors to previous versions. In accordance with the provisions of c. 1, journals that adopt appropriate open review procedures are considered eligible for the classification procedure for inclusion in the lists of scientific journals and class A journals, with reference only to those articles that have been definitively accepted or have successfully passed peer review and have subsequently been indexed.

Could Open Research Europe finally be included in the Italian agency’s lists? Not yet: Article 2(2) states that “(a) provide for the publication of multiple units (fascicles, volumes, issues), with continuity and without a predetermined end date; (b) provide formally identifiable and citable units of publication (numbered and dated) that are self-contained and allow the identification of individual contributions within them (through progressive page numbering and/or DOI code assigned to each article)”.

Perhaps ORE could – but why should it? – bother to hire a box-ticker to periodically produce self-contained, numbered and dated fascicles to satisfy the Italian agency. But beyond the pragmatic interest, we must ask at least two questions:

    .

  1. Why is it so important to the agency that articles that are already identifiable in other ways, e.g. by DOI, be organized in fascicles as if they were going to be printed?
  2. Could an article that was rejected after open peer review, but remained public with the reviewers’ critiques and the author’s responses, still be considered by committees evaluating his scientific qualifications for a professorship (ASN) or deciding whether to hire him? Imagine, for example, that a young Galileo Galilei had a manuscript entitled “Sidereus Nuncius” rejected by an open peer review of the open-access Journal of Ptolemaic Studies, which was rated excellent (“di classe A”) by ANVUR. Galileo, however, is so sure of his theory that he decides to leave his “Sidereus Nuncius” with the reviewers’ negative opinions and his replies on the journal’s website. Could a Copernican ASN or a hiring committee recognize the scientific value of the “Sidereus Nuncius” even though ANVUR’s evaluation of journals is required to ignore it, according to the last paragraph of Article 18? Galileo perhaps could not meet the bibliometric thresholds required for application: but this circumstance avoids the problem administratively, but does not solve it scientifically.

As Alessandro Figà Talamanca points out in Il fattore d’impatto nella valutazione della ricerca e nello sviluppo dell’editoria scientifica, irregularity in publication was reason enough to exclude even well-established but artisanal journals from ISI’s commercial database (now in the hands of Clarivate Analytics) on which the Journal Impact Factor (JIF) is calculated. This exclusion had nothing to do with scientific quality, but only with the convenience and cost of calculating the JIF. And, according to the ANVUR regulations (Article 15(3) and (4)), the impact of a journal is still an aspect to be taken into account when awarding a “di classe A” rating.

A former president of ANVUR argued that a predicament like that of the young Galileo Galilei is rare and unlikely. Under state evaluation, scholars are precociously trained to behave as petty clerks of research and to submit orthodox texts to the Ptolemaic journals evaluated by ANVUR. In this way, such excellent journals will never have the chance to become scientifically stimulating venues for the discussion of Copernican provocations that could never see the light of day.

And yet the imaginary Galilean predicament suggests that ANVUR’s new regulation on journals has not a scientific purpose but a bibliometric one, and that even the transitional Article 18 works to make open peer review so bureaucratic that it can be reconciled with a mainly bibliometric and centralized evaluation of research, whose principle is that a government-appointed agency is entitled to determine what is scientific and what is not. ANVUR is a participant in COARA – and one of its members sits on its Steering Board – but, despite the third commitment of the agreement, it continues to evaluate researchers, at the service of the Italian Ministry of Universities and Research, with metrics based on journals and their rankings, taking them as decisive criteria and not merely complementary. In fact, a scenario in which the peers of the scientific community, outside ANVUR’s control, actually evaluated published texts without ANVUR’s bibliometrics would weaken the power of ANVUR and the Ministry of Universities and Research behind it. And this may be the reason why ANVUR would rather try to reduce open peer review back to bureaucracy than to reduce itself to science.

This text is licensed under CC BY-SA 4.0 license

ITRN 2023 Reproducibility Award

Posted on by Silvia Bello

Italian Reproducibility Network is awarding a prize for contributions to reproducibility, replicability or open science over the past 5 years.

Submission Deadline: January 20, 2024

Details and submission link <https://www.itrn.org/initiatives/reproducibility-awards>

The prize includes a certificate, 1-year ITRN membership and a reward of 200€. The awarding moment will be an in-person ceremony during the 3rd Annual ITRN Conference scheduled to take place in Bologna on February 23, 2024.

In addition, if you’d like to join the ITRN conference free of charge and showcase your work on open science, through an oral presentation, simply follow this link <https://www.itrn.org/itrn-scientific-events/itrn-meeting>

We look forward to your submissions!

AISA nominates Emanuele Conte for COARA Steering Board

Posted on by Maria Chiara Pievatolo

Professor Emanuele Conte, a member of our Board of Directors, has agreed to be our candidate for the the Steering Board of COARA. We nominated him not only because of his long and significant experience in Open Access scholarly publishing, but more importantly because, as a legal scholar and historian, he could help to steer research evaluation towards Open Science even where laws and regulations have made it a centralized governmental exercise, and thus impervious to collaborative and participatory reform initiatives.

Taking all the running one can do, to keep in the same place: ANVUR’s complicated relationship with the COARA agreement

Posted on , updated on by Maria Chiara Pievatolo

1. An unpromising starting point

In a 2018 article, Alberto Baccini and Giuseppe De Nicolao described the Italian academic system as “an unprecedented in vivo experiment in governing and controlling research and teaching via automatic bibliometric tools”. Italian universities and research institutions are subject to widespread bibliometric measurements. Their use in research evaluation exercises (Valutazione della qualità della ricerca or VQR) and in the national scientific qualification for professorships (Abilitazione scientifica nazionale or ASN), which we will discuss, are just two of many examples. But the Italian research evaluation machine is not only bibliometric: it is also administrative and centralized.

ANVUR, the Italian National Agency for the Evaluation of the University and Research, plays a central role. ANVUR is neither an autonomous agency nor a quango run at arm’s length by the government. It is instead a governmental agency: its board consists of seven professors directly nominated by the Minister of Education [now: Ministro dell’università e della ricerca or MUR]. Moreover, ANVUR acts principally by implementing activities directly defined by ministerial decrees, such as: research assessment exercises, quality assurance for teaching, evaluating the administrative tasks of universities, and assessing the qualifications of candidates for professorship. Among similar European institutions, such as AERES in France or ANECA in Spain, none concentrate so much power and so many functions in one place.1

Against this background, the agency’s signature of the COARA agreement and the presence of one of its board members on the COARA steering committee seems surprising. It is true that, after having locked Italian research in a steel cage, in violation of Article 33 of the Italian Constitution, only the ministry (MUR) and the agency (ANVUR) can repair the damage they have caused. But are they willing to do so? And does its subordination to the government really give the agency the power to honor the commitments it has formally signed?

2. A disappointing first step

As reported in our August statement, ANVUR has ruled that Open Research Europe (ORE) is not a scientific journal because it does not publish in “clear, self-contained fascicles that are not open to further updating” and because it does not practice anonymous peer review. But the substance of this decision, controversial as it may appear, is not so important. Even if the ANVUR had declared ORE to be scientific, the core issue would have remained the same: in Italy, it is not the researchers who decide what is scientific and what is not, but a government-appointed agency, whose many tasks include identifying which journals in the humanities and social sciences are scientific and which are not, and which of the latter are “excellent” (A rating or “di classe A”) and which are not.
Since publications in journals declared scientific or excellent by the agency are necessary for the career of Italian researchers, the very existence of this bureaucratic system of lists contradicts the COARA principles signed by ANVUR.

1. If what is scientific and what is not is determined by lists drawn-up by a government-appointed body, the freedom of researchers to choose the topics, methods, theories, and applications of their investigations is trapped within the agendas and editorial policies of journals approved by the ANVUR, against the safeguard of the freedom of scientific inquiry which is the content of the second principle for overarching conditions.2

2. The first COARA principle for evaluation criteria and processes links research quality to openness as “early sharing of knowledge and data and open collaboration”.3 But openness will continue to be a lip service if what is scientific and what is not is decided by a government-appointed authority in a zero-sum game designed to pit researchers and institutions against each other.

Moreover, the ANVUR specifically violates at least three of the COARA commitments:

1. First commitment (p. 5): if evaluation should be about content rather than containers, why continue to compile lists of journals administratively labeled “scientific”?

2. Second commitment (p. 5): if the main method of assessing quality should be rigorous and transparent peer review, how can transparency be possible when the only type of review that ANVUR considers “scientific” is anonymous?

3. Third commitment (p. 6): if the “inappropriate use of journal- and publication-based metrics in research assessment” is to be abandoned, the ANVUR lists should also have been dropped, since they are used to evaluate researchers on the basis of the number of publications and thus on the basis of journal- and publication-based metrics.

3. An inconsistent conclusion

In October 2023, it became clear that ANVUR’s rejection of ORE was no accident. Bibliometrics, based either on the H-index or on citations or on the list of journals compiled by ANVUR, was and still is the cornerstone of both the national research evaluation exercise (VQR) and the national scientific qualification for professorships (ASN).

If we look at the last calls of ANVUR, we can see that nothing has changed significantly.

The call for the next National Research Assessment Exercise (VQR 2020-2024) implements the guidelines established by a previous decree of the Ministry of University and Research, even if it includes at least two patent violations of COARA commitments.
ANVUR will draw by lot the 75% of the members of the panels of expert evaluators, bureaucratically known as “gruppi di esperti della valutazione” or GEVs. In order to be included in the pool from which the lots will be drawn, candidates should meet quantitative thresholds based on the calculation of their citations and H-index in the STEM fields, and on the number of articles published in the above-mentioned lists of journals in the SSH fields (Art. 3.3), against the letter of the third commitment (p. 6). Furthermore, the first paragraph of article 7 of the VQR call allows the use of bibliometric indicators in the evaluation of research, although “such indices cannot replace an accurate evaluation of the merits of the research product, nor can they lead to the automatic assignment of the product” to an evaluation category (art. 7.2). Such a clause, however, can be compatible with any kind of purely bibliometric evaluation, provided that it is reformulated in ad hoc qualitative terms.

The drawing of lots is not a tribute to ancient democracy’s mistrust of the aristocratic risks of elections. The remaining 25% of the members of the GEVs will be appointed by ANVUR, again on a bibliometric basis (art. 3.4); ANVUR will also supplement the GEVs (art. 3.6)  if the procedure does not produce groups that meet the requirements of article 3.4, and will appoint their coordinators (art. 3.16). Finally (art. 3.22), the ANVUR board may replace the members of a GEV in the case of “critical issues that have arisen during the in itinere review of the progress of the evaluation process, after hearing the coordinator” – appointed by it – “(with special reference to failing to perform assigned work, delaying, or violating the Code of Ethics)”. Despite the drawing of lots, the strategic appointment of the coordinators, the possibility of dismissing evaluators almost at will and the very weakness of officials who are chosen by lot rather than elected place the entire VQR exercise under the hierarchical control of the ANVUR board.

As far as Open Access is concerned, the VQR call takes away with one hand what it seems to give with the other. Article 8 (first paragraph, point a) requires open access “in the case of publications dealing with research results financed at least 50% by public funds and, in general, for all publications for which the publisher allows it.” However, access can be postponed according to the extraordinarily long embargo of Law 112 of 2013, of one and a half years for STEM and of two years for SSH. And point (b) of the same subsection, which regulates access to reference metadata “for products related to research results that have received less than 50% of their funding from public funds, or with embargo periods longer than those indicated in point (a), or in all cases where dissemination is not authorized by the publisher” – suggests that ANVUR’s open access will take place on Tibb’s Eve, at the discretion of publishers. Oddly, the opinion of the original copyright holders, the authors, seems to be irrelevant.
It is worth recalling that Italy is also a longstanding violator of the 2012 Recommendation on access to and preservation of scientific information (C(2012) 4890 final) (C(2012) 4890 final), which was one of the cornerstones of the EU Open Access policy. To avoid the crystallization of rules that should and could be improved, the ANVUR calls could have more simply required that if a publication is already legally Open Access, it must be uploaded as such to the ministry platform.

And what if an Italian scholar wants to apply for national scientific qualification for professorship (ASN)? According to the latest ministerial decree, he or she must meet at least two quantitative thresholds (“indicatori di impatto della produzione scientifica”, Art. 2.4 c), which have nothing to do with Open Access, but are still determined by proprietary or administrative bibliometrics. In the STEM fields, the thresholds are the H-index, the number of articles and the number of citations, as obtained from two proprietary databases, Clarivate Analytics and Scopus. For SSH, the thresholds are the number of books, the number of articles published in “scientific” journals, and the number of articles published in “excellent” (“di classe A”) journals. As mentioned above, in Italy a journal can only be recognized as scientific or “excellent” if it is included in ANVUR’s lists.
These bibliometric requirements are not part of an evaluation that includes qualitative assessments as well, as in a responsible use of metrics: if scholars do not meet the required thresholds, their application must be rejected, even if they were Nobel laureates.
And again, the commissioners who will judge them are also drawn by lot from a pool of candidates who must meet some stricter bibliometric requirements, without regard to the open accessibility of their works.

4. Some final questions

Why did ANVUR sign the COARA agreement but failed to fulfill its commitments?
The ANVUR employees could justify themselves by saying that they are obliged to follow the orders of the Ministry according to the Italian laws and administrative regulations in force. Yet, if this is the case, why did they sign an agreement that they have no authority to apply? Why does an ANVUR official sit on the COARA steering board? In what capacity does the ANVUR participate in the meetings of the Italian National Chapter?
On the other hand, if the ANVUR has at least some power to break the bureaucratic chains that imprison Italian research, why has it not used that power? Did it sign the agreement from the beginning with the hidden intention of changing things only to keep them the same?
In any case, whether the agency does not respect its signature because it cannot or because it does not want to, two questions are hard to avoid. Is its participation in the steering board and in the Italian national chapter still compatible with the COARA agreement? And how can COARA ensure that its members honor their commitments if they seem unable or unwilling to do so?

Update on ORE (January 19, 2024)

This text is licensed under CC BY-SA 4.0 license

“Open”, “Openness”

Posted on , updated on by Enrico Pasini

Open’ is the opposite of ‘closed’. Openness has different degrees, from ‘ajar’ to ‘wide open’, and it applies to very different things, from closets to archives, from consultation to access, from politics to science.

In Galileo’s view, nature is an open book in front of our eyes that can be studied by anyone who has the right intellectual tools.
In the age of Galileo, scientists went to great lengths to disseminate scientific results. But they also used riddles and anagrams to prove publicly, for future reference, that they were the sole owners of a discovery they did not want or could not yet publish. Even the practical applications of science could be a source of controversy: Galileo produced and sold an ingenious measuring instrument and sued an imitator to maintain exclusivity.

Since then, the nature of scientific research has changed:

  • It has become very complex, both technically and socially;
  • The system of scientific publishing has become an international economic sector in its own right.

Nature is still open, but the tools of science and the communication of results have become part of closed system.

The answer to this closure is openness as a program and a motto. It refers first of all to access to cultural works and scientific materials (publications, methods, data), then to science itself. It affects the publication models and dissemination practices of science and qualifies its social and democratic function.

In this usage, “open” means first and foremost “free”, not only from costs (“free”), but also from obstacles due to various reasons of secrecy (political, economic, competitive), administrative evaluation obligations or material difficulties. Open licenses promote the free circulation of knowledge. The principles of Open Access consist in the free consultation and reproduction of scientific publications and results (while respecting the moral rights of authors).

Moreover, “open” means not only “free” but also “accessible” and “transparent” (as in the case of open peer review). Openness, free circulation, accessibility and transparency are the hallmarks of open science.

Transl. by Maria Chiara Pievatolo

This text is licensed under CC BY-SA 4.0 license

Dicer’s oaths: ORE, COARA and the Italian National Agency for the Evaluation of the University and Research Systems

Posted on , updated on by Maria Chiara Pievatolo

As reported by Roars, the Italian National Agency for the Evaluation of University and Research Systems (ANVUR)  has refused to recognize Open Research Europe (ORE) as a scientific “journal” because it does not publish in single, self-contained fascicles and does not practice anonymous peer review.

Open Research Europe is an infrastructure provided by the European Commission to authors whose research is the result of European funding. ORE can be simultaneously an archive, a public discussion forum, and a series of journals from different disciplines, since it is free from the technological constraints of print, for which the publication of texts selected ex ante also signals their value (filter, then publish).

There are many kinds of open peer review. Among them, ORE has chosen the one in which specially selected experts are appointed, but the whole discussion is open to the public: the public debate helps to recognize their responsibilities and merits, to expose any conflicts of interest, and to transform an opaque, hierarchical process prone to failure even catastrophic into an open scientific conversation .

ORE is objectionable because its management has been entrusted not to public libraries and archives, but to F1000 Research, which is owned by a private commercial oligopolist that is part of the so-called surveillance publishing, which has very little to do with the protection of independent and public knowledge. But this is not what the ANVUR is concerned about: for those who want to use the articles deposited there for the purposes of their academic career in the sociological field, ORE is, according to the agency, neither scientific nor “excellent”, whatever that means, because it does not publish in “clear, self-contained fascicles that are not open to further updating” and because it does not practice anonymous review.

Let us resist the temptation to laugh at those who, more than thirty years after the invention of the World Wide Web, demand that online publishing should not make texts available as soon as they are ready, but should manage to mimic the technical and economic limitations of printing by publishing them in “fascicles”. Let us, for the moment, avoid criticizing their attempt to harness scientific discussion by means of processes inspired by ideas that are newer and more questionable than they probably imagine. And let us also forget the debate that is raging beyond the Italian border, on the parasitism and obsolescence of commercial scientific journals. Instead, let us focus on who evaluated ORE and why they were entitled to do so.

The current system of research evaluation in Italy is administrative and centralized, rather than scientific and decentralized, under the pervasive control of a National Agency for University and Research Evaluation, whose board of directors is appointed by the government. One of its many tasks is to determine which journals in the humanities and social sciences are scientific and which are not, and which of the latter are “excellent” (“di classe A”) and which are not. Despite Article 33 of the Italian Constitution, Caesar est supra grammaticos: it is not the researchers who decide what is scientific and what is not, but an authority appointed by the government.

The same bureaucratic and hierarchical principle inspires the regulation for the rating of journals in the so-called non-bibliometric fields, written by the ANVUR board: in addition to having the final say, the board controls the process from the beginning by appointing the members of the working groups responsible for the preliminary evaluation.

The ANVUR is one of the formal signatories of the European Agreement on Reforming Research Assessment, and one of its board members also sits on the steering committee of the Coalition for Advancing Research Assessment (COARA). But, with its decision, it not only tells Italian sociologists that publishing on the platform provided by the European Commission is like throwing their papers in the trash: it also violates some of the principles and commitments of the COARA itself.

First, the ANVUR violates COARA’s second principle for overarching conditions, which is to “safeguard the freedom of scientific research”.

Safeguard freedom of scientific research. By putting in place assessment frameworks that do not limit researchers in the questions they ask, in their research implementation, methods or theories. By limiting the assessment frameworks to only those necessary, as assessment must be useful for researchers, institutions and funders.1

If what is scientific and what is not is determined by lists written by a government-appointed agency, the freedom of researchers to choose the topics, methods, theories, and applications of their investigations is trapped within the agendas and editorial policies of journals approved by the ANVUR.

Second, among the principles for evaluation criteria and processes, the ANVUR violates at least the first one, which links research quality to openness: “openness corresponds to early knowledge and data sharing, as well as open collaboration including societal engagement where appropriate”.2 Again, how can we promote open collaboration when what is scientific and what is not is decided by a government-appointed authority in a zero-sum game designed to pit researchers and institutions against each other?

The ANVUR is also in specific violation of at least three of the commitments in the COARA Agreement:

  • the first commitment (p. 5), which requires recognition of all the “valuable contributions that researchers make to science and for the benefit of society, including diverse outputs beyond journal publications and irrespective of the language in which they are communicated”. If evaluation should be about content, wherever it is made available, rather than about its containers, why continue to compile lists of journals administratively labeled “scientific”?
  • the second commitment (p. 5) recognizes that “peer review is the most robust method known for assessing quality and has the advantage that it is in the hands of the research community”. Therefore, “it is important that peer review processes are designed to meet the fundamental principles of rigor and transparency” – a transparency that is not possible if the only type of review that the ANVUR considers “scientific” is anonymous.
  • the third commitment (p. 6) requires to “abandon inappropriate uses in research assessment of journal- and publication- based metrics”. But what would be the use of ANVUR’s lists of scientific journals if not the evaluation of researchers on the basis of the number of publications in the journals included in the lists, and thus on the basis of journal- and publication-based metrics?

Its decision on ORE was not essential in order to recognize that the administrative system of journal lists was in contradiction with the European commitments formally signed by the ANVUR. It was already clear that the Italian regulation on research evaluation conflicted with COARA’s principles. To be consistent, the ANVUR should have at least updated its regulation to make its lists more inclusive or, better yet, abolished the list system altogether. But the ANVUR did neither, as if it did not know the meaning of the agreement it had signed, or, knowing it well, it had the hidden intention of changing things only to keep them the same.

Update on ORE (January 19, 2024)

This text is licensed under CC BY-SA 4.0 license

Open Science and centralized and administrative research assessment: an odd couple?

Posted on , updated on by Maria Chiara Pievatolo

AISA responds to COARA‘s Call for Working Groups with its own proposal: “Open Science and centralized and administrative evaluation of research: an odd couple?”

The aim of the working group is to understand whether and how a centralized and administrative assessment of research, which evaluates its “products” in a zero-sum game, can be made compatible with Open Science.

The full text of the proposal is available here. COARA member organizations wishing to endorse and enhance it are most welcome.

You can reach us at aisa@aisascienzaperta.org .