Investigating Changes in Common Vocabulary Terms in eLife Assessments Across Versions, in a Publish, Review, Curate Model

Nicola Adamson,¹ Andy Collings¹

Objective

In October 2022, eLife announced a Publish, Review, Curate model¹ in which all peer-reviewed submissions are published as reviewed preprints, accompanied by an eLife Assessment² and public reviews. Authors decide which revisions to undertake and when to declare a version of record (VOR), without an accept/reject decision after peer review. We investigated the extent to which authors revise and improve their work.

Design

An eLife Assessment is prepared for each submission sent for review and subsequent revisions, which summarizes the strength of evidence and significance of findings using terms from a common vocabulary. We extracted terms used in eLife Assessments for all 2918 reviewed preprints and their subsequent versions to the end of 2024. To ensure publications had completed the entire process and we could evaluate the full extent of changes, we selected the 1504 where a VOR had been declared. From these, 213 were excluded because either the VOR or original reviewed preprint had included multiple terms to describe the evidence. Using a retrospective analysis, we studied the remaining 1291 reviewed preprints and their respective VOR, calculating the distributions of and changes to terms between first and final versions and the number of rounds of revision.

Results

Panel A in Figure 25-0989 shows the terms used to describe the strength of evidence in VORs and the first version of the reviewed preprints. The evidence term improved between versions in 39.4% of cases (n = 509) and remained the same in 57.7% of VORs (n = 745). Of VORs in which the evidence was described as incomplete in the first reviewed preprint, 76.5% (n = 199) improved to solid or better, and in papers originally described as solid, 49.3% improved to convincing or better (n = 217). In papers in which the evidence was originally described as inadequate (n = 23), this improved in 78.3% of instances (n = 18) and approximately one-half ended as solid or better (47.8%, n = 11). The majority of VORs (82.6%, n = 1066) were declared after 1 round of revision and 13.5% (n = 174) after 2 or more rounds. Of the 4.0% of VORs declared without revisions (n = 51), only 13.7% had an evidence term of inadequate (n = 0) or incomplete (n = 7). The significance of the findings (Figure 25-0989, B) was most frequently described using valuable (VOR: 31.1%, n = 401) or important (VOR: 49.0%, n = 633); overall, significance terms remained the same in 78.2% of cases (n = 1010).

Conclusions

Authors generally complete at least 1 round of revision, even though they have the option to proceed without, and multiple rounds of revision are uncommon. Where the evidence was originally described as incomplete, the majority of authors revised to improve this before the VOR. Terms used to describe the significance of the findings change less often than evidence terms, perhaps due to significance being more closely linked to the original research question under investigation.

References

1. Eisen MB, Akhmanova A, Behrens TE, et al. Scientific publishing: peer review without gatekeeping. eLife. October 20, 2022;11:e83889. doi:10.7554/eLife.83889

2. eLife’s New Model: What is an eLife assessment? eLife. October 20, 2022. Accessed January 30, 2025. https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment

¹eLife Sciences Publications Ltd, 95 Regent Street, Cambridge, CB2 1AW, UK, n.adamson@elifesciences.org.

Conflict of Interest Disclosures

We have no competing interests to declare. Both authors are employed by eLife Sciences Publications Ltd; we received no external funding for this work.

Acknowledgment

We thank Fiona Hutton, Fred Atherden, Emily Packer, and George Currie for their comments and suggestions on the submission, and Fred Atherden for extracting data for analysis.