Methods of Evaluating Scientific Papers

Introduction

The quality of applied research can be evaluated based on whether a new technology or a new successful product is created. The result of basic research is primarily publications – scientific papers in which authors present to the professional community what they discovered and by what procedure.

The evaluation of scientific papers can be divided by method into qualitative and qualitative approaches. Qualitative evaluation of scientific papers is carried out through the opinions of experts in the given field – within the peer review process (i.e., peer review) or through case studies. This also includes surveys of research workers' opinions (user surveys), or surveys conducted by senior executives or government offices. Quantitative evaluation is based on the use of scientometric indicators and bibliometric data – citation indexes, the Hirsch index, the Impact Factor, national information databases of science and research, and also data from one's own surveys.

Recognition of quality work comes when other scientists cite it in their publications – they mention a certain result with a reference to the original publication. As a rule, a scientific paper is cited more often if it is more important and more interesting. However, this is not always the case. Not only the relevance or impressiveness of the work is a significant factor, but also the focus of the topic and the field in which the author publishes. The more narrowly specified the topic is, the fewer citations the paper may receive. Similarly, the number of specialists and scientists working on a given topic affects citation rates. A more specific article intended for a particular group of specialists and scientists tends to have lower citation rates. It should be noted that the language in which the work is published, as well as the country in which the author works, and the journal (institution) where the work was published, may also influence citation frequency. The quality and importance of scientific papers can therefore be at least approximately evaluated or assessed through their citation rates. Citation data are often requested when evaluating candidates for academic teaching titles, in competitions for significant administrative research positions, and frequently in grant applications. One option is to count the actual citations of a specific author's works over a certain period. However, to assess the citation impact, it is necessary to wait several years, which is why the so-called impact factor of the journal ("the impact or influence factor of the journal") in which the work was published is often used instead.

Peer Review Process

A qualified assessment of a work guarantees its quality. The function of the peer review process is to eliminate low-quality and insignificant contributions, and to remove errors and inaccuracies from accepted texts, so that the reader receives reliable and trustworthy findings.

The peer review process addresses the following elements:

• novelty of research,
• importance to the subject matter,
• relevance (interest) for readers,
• originality, authenticity, originality – research conducted personally by the author,
• correctness of the methodological aspects of the research.

The peer review report should primarily be a clear statement by the reviewer regarding the justification of publishing the evaluated work. Such a position must always be stated in the conclusion of the report.

The main criterion for this statement should be the professional quality of the work. Another important criterion is the evaluation of the author's familiarity with the literature (both domestic and foreign). The comprehensiveness and relevance of the literature used is assessed. For textbooks and manuals, the reviewer should pay attention to the comprehensibility of the text from the reader's perspective. When evaluating textbooks, it is necessary to determine (and express) whether the submitted text has the character of a university textbook. A scientific monograph – the reviewer evaluates whether the author's work is based on their own research, which must be briefly characterized, whether the topic has been developed by the author to the necessary extent and in an original manner (i.e., to express the author's contribution to the development of the given discipline).¹ An academic book publication – does not require evaluation of the author's contribution to the development of the scientific discipline.

The reviewer should adhere to the following principles:

• agree to review a contribution only if they are aware of their expertise in the subject matter,
• respect the confidentiality of the review process and not disseminate information about the proceedings (except for necessary communication with the editorial office),
• not abuse review process information for their own benefit or to the detriment of other persons,
• declare in advance any potential conflict of interest or bias regarding the author or the topic of the contribution,
• prepare an objective and constructive review,
• avoid personal attacks and derogatory remarks about the author.

In evaluating a manuscript, the reviewer assesses the following:

• whether the objective of the material is clearly defined,
• originality,
• the structure, stylistic, and linguistic level of the contribution,
• the methods used,
• the correctness of citations,
• the clarity of the abstract and keywords,
• the quality of the graphic appendices.²

In the conclusion of their review, the reviewer indicates: a) the manuscript is recommended for publication without reservations, b) the manuscript is recommended for publication with reservations (the reviewer subsequently states whether the comments were addressed by the author and whether the manuscript is recommended for publication without reservations), or c) the manuscript is not recommended for publication.

Categories for Recording Publication Activity

In accordance with Decree No. 397/2020 of the Ministry of Education, Science, Research and Sport of the Slovak Republic on the Central Register of Publication Activity and the Central Register of Artistic Activity of 5 December 2020, the following categories for recording publication activity and categories for recording responses to publication activity outputs are established:

V – Scientific Output of Publication Activity V1 – scientific output of publication activity as a whole – monograph, critical source edition, critical annotated translation, cartographic work, catalog of artistic works, edited book, proceedings. V2 – scientific output of publication activity as part of an edited book or proceedings – chapter, contribution, abstract, conference abstract, conference poster, conference contribution. V3 – scientific output of publication activity from a journal – abstract, article, conference abstract, conference poster, conference article.

O – Professional Output of Publication Activity O1 – professional output of publication activity as a whole – book publication, review work, annotated interpretation, anthology, catalog of artistic works, cartographic work, dictionary, encyclopedia, proceedings. O2 – professional output of publication activity as part of a book publication or proceedings – chapter, contribution, abstract, conference abstract, conference poster, conference contribution, review, entry. O3 – professional output of publication activity from a journal – abstract, article, conference abstract, conference poster, conference article, review.

P – Educational Output of Publication Activity P1 – educational output of publication activity as a whole – university textbook, secondary school textbook, primary school textbook, course script, educational text, workbook, didactic manual. P2 – educational output of publication activity as part of a textbook or course script – chapter.

U – Artistic Output of Publication Activity U1 – artistic output of publication activity as a whole – book publication, anthology, catalog of artistic works, musical score (sheet music), dramatic work, architectural study, fiction, short story collection, poetry collection, proceedings. U2 – artistic output of publication activity as part of a book publication or proceedings – chapter, contribution. U3 – artistic output of publication activity from a journal – article.

D – Intellectual Property Rights Document D1 – intellectual property rights document – patent application, patent, utility model application, utility model, design, semiconductor product topography, product geographical indication, product geographical indication, breeding certificate.

I – Other Output of Publication Activity I1 – other output of publication activity as a whole – publications that cannot be classified into categories V, O, P, U, or D. I2 – other output of publication activity as part of a publication or proceedings – parts that cannot be classified into categories V, O, P, U, or D. I3 – other output of publication activity from a journal – articles that cannot be classified into categories V, O, P, U, or D.

Categories for Recording Responses to Publication Activity Output:

1. citation in a publication registered in citation indexes,
2. citation in a publication including citations in publications registered in databases other than citation indexes,
3. review and art criticism in a publication. ³

Peer Review in Journals

Large open-access journals focus their peer review process only on two criteria: originality (authenticity, originality – research conducted personally by the author) and correctness of the methodological aspects of the research (soundness - only peer review).

Currently, the most widely used type of peer review is evaluating a manuscript before publication. After an article is submitted to a journal, the editor first evaluates it. If the editor determines that the topic does not match the journal's focus, the article may be immediately rejected (so-called desk rejection). Approximately 20% of submitted manuscripts end at this stage (Ware 2008). Another 40% are typically rejected after review by reviewers (these are average figures; Greaves et al. 2006). Due to the large number of texts awaiting evaluation in editorial offices, even an immediate rejection may not be communicated to the author until several months later. However, these cases do not occur frequently and are generally the result of an editorial error. If the editor determines that the article could be suitable for the journal in terms of its thematic focus and processing, they contact reviewers. These may agree to evaluate the article or decline to do so. If the author mentions potential reviewers in their cover letter, the editor may select them but is not obligated to accept the proposal. Editors use such contacts when they have difficulty finding a sufficient number of suitable reviewers in their own database for a given field. The editor must always verify that the reviewers are not biased, do not come from the same institution as the author, and have not co-authored publications with the author in the past 3 to 5 years. The editor works with two reviewers; if both reviewers provide positive assessments, the article is typically published in the journal after approval by the editorial board and editorial corrections. If the reviewers' opinions differ, a third reviewer is invited whose opinion is decisive.

Critical thinking is the reviewer's main tool. It is the ability to independently evaluate the information presented in light of one's knowledge and expertise, form an opinion, and make a decision. The manuscript is evaluated with appropriate attention, the reviewer can separate the essential from the non-essential, identify the strengths and weaknesses of the methodology, and formulate constructive comments.

The reviewer's first step is to consider their own abilities and the limits of their expertise – they should decline the review if asked to evaluate an article on a topic they do not sufficiently master. They must also not conceal any potential conflict of interest with the manuscript, whether personal, financial, professional, political, or other in nature, which could raise doubts about the reviewer's objectivity and impartiality. The reviewer should also consider whether they have sufficient time to prepare their evaluation. In the case of declining a review, they can help the editorial office by recommending another suitable reviewer (COPE 2017). Journals differ in the form of peer review requirements, e.g., whether they require verification of sources, allow (or encourage) contact between the reviewer and the author, or have different policies regarding content sharing. Information obtained by the reviewer during the process is confidential, and they may not disclose it to their students or colleagues without the editorial office's permission. The reviewer should be impartial, detailed, constructive, without personal remarks, and should support their opinions with facts or references to literature. Editorial offices usually allow reviewers to submit comments for the editor as well as comments accessible to the author. An important aspect is the consistency of the evaluation – comments for the editor and for the author should not differ, and the final recommendation for accepting or rejecting the manuscript should not contradict the comments. In cases where the results are not conclusive, the reviewer may request additional results, assess the linguistic (terminological) level, evaluate the overview of the subject matter, the novelty, the adequacy of the methods, and their application. The reviewer remains in contact with the editorial office even after submitting their evaluation, commenting on new versions of the manuscript.

Accepting an article with reservations means that the manuscript is returned to the author for revision based on comments that may take the form of minor formal corrections (minor revisions). Comments are substantive (major revisions) if they concern serious deficiencies in the processing, methods, or interpretation. A significant number of substantive deficiencies in an article, combined with interesting results and the editor's continued interest, allows for rejection of the article with the possibility of resubmission, provided that the author fundamentally rewrites the article (reject with the possibility to resubmit). If the editor does not accept the work for a given journal, some publishers allow the manuscript to be forwarded to another journal in their portfolio; in this case, the review may be transferred and the review process does not have to start from scratch (portable peer review). If the second journal has different requirements and the author has revised the article (or left it in its original form), the reviewer adjusts their evaluation accordingly. A manuscript may be returned multiple times until the reviewers, editor, and author reach agreement. After acceptance of the article, editorial corrections, and typesetting (arranging the text and images into their final form), the author receives the article for a final check, and then the article is published. Compliance with the journal's formal requirements includes signing a licensing agreement, and sometimes transferring copyright to the publisher. When an article is rejected, the author has several options. They may appeal and request re-evaluation, but it is generally easier to submit the article to another journal. If the author accepts the reviewers' comments as valid, they should revise the article accordingly.

During the second half of the last century, the specialization of scientific fields and the increasing number of articles led to the introduction of the peer review process in all scientific journals. The peer review process was also applied in other academic areas, such as evaluating grant applications. To avoid conflicts within the scientific community, the review process began to be conducted with varying degrees of anonymity. Single-blind peer review – the author does not have information about the identity of the reviewers; Double-blind peer review – the author does not know the reviewers' names, and the reviewers do not know the author, to prevent the emergence of bias (conscious or unconscious). In highly specialized scientific communities, however, the nature of the work, the location, materials used, or other elements may reveal the author's identity. Many authors emphasize the necessity of double-blind peer review, but more and more scientists no longer have historical prejudices and do not require anonymity of the process, if given the choice. Conversely, the large volume of manuscripts presents a problem due to the difficulty of finding qualified and willing reviewers. Open peer review should ensure that the public values the work of reviewers, who will therefore engage with manuscripts more professionally. If a reviewer approves a low-quality manuscript or rejects a good manuscript without sufficient justification, it may damage their reputation. The British Medical Journal has been publishing signed reviewer reports along with the article since 2014, which has been appreciated by both authors and reviewers. The journal PeerJ provides reviewers with the option to sign their reviews and provides authors with the option to publish the peer review history (PeerJ 2014). Opening the peer review process often raises concerns about a decline in the number of reviewers, but this may not necessarily occur.⁴

Scientometric Indicators as a Criterion for Evaluating Scientific Work

1. Citation Indexes

• the subject of citation indexes is tracking the number of citations in terms of citation responses,
• citation response (cited reference) is one of the criteria for evaluating the quality of an individual's scientific work, or that of an institution, group of authors, etc.,
• citation responses also serve as a primary source of information for determining the importance of scientific journals according to impact factor and other journal evaluation measures,
• the author's citation index – the number of citations attributed to their published works over a certain period,
• an author has a higher citation index the more frequently their publications are cited by other authors,
• the results are not entirely accurate, as negative citations questioning the given publication and self-citations – the author citing their own articles – are also counted as citations; the results also depend on the scientific field and other aspects, vary according to the database from which the data are obtained, and change over time.

Among the most significant citation databases are Web of Science Core Collection, Scopus (citations from 1995), as well as freely available sources Google Scholar, Pubmed (for the field of biomedicine), NASA ADS (for the fields of astronomy, astrophysics, physics, and geophysics), etc.

2. Hirsch Index (h-index)

• an index named after Jorge Eduardo Hirsch (proposed in 2005),
• it is a quantitative evaluation of the quality of scientific articles published by a single researcher, or an institution, group of authors,
• the Hirsch index of an individual (or institution) can be determined from the citation indexes of individual scientific papers. It is the number h, indicating the number of articles that have a citation index equal to or higher than h. For example, if an author's eighth most-cited article has a citation index of 15 and the ninth article has a citation index of 7, then for that scientist, the Hirsch index is h=8.

The h-index value of a specific author (or institution) can be obtained from the Web of Science Core Collection database (the h-index includes self-citations; the h-index without self-citations must be determined manually), from the Scopus database, where the h-index includes self-citations, which can be removed – "Exclude from citation overview" (the result is an h-index without self-citations of all co-authors – "Self citations of all authors", or of the specific author – "Self citations of selected author"); from Google Scholar (which is freely available), for which the Firefox add-on – Scholar H-Index Calculator can be used. Also, in specific databases such as INSPIRE-HEP, which can automatically generate the h-index for researchers in particle physics. Examples of erroneous conclusions regarding the h-index can be summarized as follows: when searching, one must be careful about name matches or variations in the institution's name; the h-index changes over time – it is valid as of the date of determination; errors may occur in databases; comparing the performance of scientists in different fields; also, a scientist working in a field longer than a colleague may be assumed to have a higher h-index; it is possible that each database may calculate a different number for the same scientist due to different coverage; WoS and Scopus have different record depths and do not index the same sources; a possible different result in Google Scholar; it is necessary to specify the source or method of determination; the h-index may also differ within WoSCC (generated automatically, value including self-citations).

3. Impact Factor

• the impact or influence factor of a journal,
• it is contained in the Journal Citation Reports (JCR) database, which is one of the databases on the Web of Science platform – a product of Clarivate Analytics. JCR is a source of other journal evaluation measures,
• the impact factor is valid only for the journal as a whole (journal impact factor) and for a given year. It allows for the evaluation and comparison of scientific journals. It does not apply to individual articles from which it was calculated, or to their authors,
• the impact factor, due to its simplicity and accessibility, is one of the criteria for evaluating the publication activity of individuals and institutions,
• it does not replace the citation rate of an article (author), i.e., its individual specific citation responses; however, if an author publishes in a journal with a high impact factor value, then their work is better "scored" in the evaluation of scientific and publication activity.⁵

IF of a journal for a given year = number of citations (A) / number of publications from the previous 2 years (B) (A) the number of citations of all articles published in the journal in the previous 2 years that were cited by indexed journals during the given year (B) the total number of publications in the given journal from the previous 2 years * * does not include editorials, corrections (errata), notes, retractions, discussions

Conclusion

Currently, one can find authors who leave the evaluation of the quality of their scientific work to the content of the work itself and to the opinion of experts from the same scientific field, who can objectively assess the quality of the work, rather than relying on evaluation in the form of citation frequency and the impressiveness of the journal's impact factor. Similarly, there are authors who focus primarily on publishing in journals with a high impact factor, sometimes also for their own personal prestige.

It is true that scientometric indicators do not always fully reflect the quality of scientific work, its contribution, importance, or originality. These indicators can be distorted by many phenomena, such as negative citations, increasing personal ratings through self-citations, or forming certain communities of colleagues who preferentially cite the works of their colleagues over those of competitors in the scientific field. Several scientists also question the significance of the impact factor and point out that major world journals favor scientists from certain circles and countries. There have also been cases in the past where journals refused to publish scientists' work, yet these scientists later became world-renowned and respected. It is difficult to assess retrospectively whether this was due to the incompetence of reviewers and editors, which led to an inability to recognize the quality of the work, or due to conflicts of interest for personal or financial reasons.

In any case, the evaluation of scientific papers using scientometric indicators cannot be considered an objective method of evaluation, as it is carried out through assessments by people from the given scientific field, or by other people who cite the work. Finally, it is important to highlight the work of experienced reviewers who responsibly and with high professionalism evaluate scientific papers, and in doing so contribute to the progress of science and help popularize it.

Author: JUDr. Branislav Benkovič

References

COPE Council. Ethical Guidelines For Peer Reviewers. Committee on Publication Ethics. 2017. Available online: https://publicationethics.org/files/4111798b762b4349e6124a7f15dc7aac ČIŽNÁR, I. Fundamentals of Scientific Research Work. Bratislava, 2012, p. 143. ISBN 978-80-89352-68-5 DOBBERSTEINOVÁ, J., HUDECOVÁ, S., STOŽICKÁ, Z. Guide to the World of Scientific Publishing. Bratislava: Open Science Publishing, Centre for Scientific and Technical Information of the Slovak Republic, 2019, p. 294. ISBN 978-80-89965-17-5 GREAVES, S., SCOTT, J., CLARKE, M., MILLER, L., HANNAY, T., THOMAS, A., CAMPBELL, P. Nature's trial of open peer review. In: Nature, 444 (7122), 2006, pp. 971-972. Available online: http://www.nature.com/nature/peerreview/debate/nature05535.html HENDERSON, M. Problems with peer review. In: BMJ, 340 (1409), 2010. ISSN 1468-5833. Available online: https://doi.org/10.1136/bmj.c1409 HIRSCH, J.,E. An index to quantify an individual's scientific research output. 2005. Available online: https://www.pnas.org/content/102/46/16569.full HOŘEJŠÍ, V. Can We Measure the Quality of Scientific Work? In: Vesmír, No. 2, 2000. Available online: https://vesmir.cz/cz/casopis/serialy/scientometrie-aneb-muzeme-merit-kvalitu-vedecke-prace.html ILAVSKÁ, J. Evaluation of Science and Research at Comenius University in Bratislava. Bratislava: Comenius University. 2018. Available online: http://www.sakba.sk/dokumenty/2018/SAK-scientometriaUK-2018.pdf Criteria for Evaluating the Level of Research, Development, Artistic, and Other Creative Activities within the Comprehensive Accreditation of Higher Education Institution Activities. Bratislava: EUBA. 2013. Available online: https://www.euba.sk/www_write/files/916cf2faa309cda6158fec2e3dd3903c PEERJ STAFF. Who's Afraid of Open Peer Review? PeerJ Blog, 2014. Available at: https://peerj.com/blog/post/100580518238/whos-afraid-of-open-peerreview/ Detailed Evaluation Rules Applied in the Relevant Research Areas. Accreditation Commission, Advisory Body of the Government of the Slovak Republic. Available online: http://www.akredkom.sk/zapis/364fo/Pravidla_hodnotenia_vyskumu_AK_uplne%20znenie.pdf Peer Review Process and Rules for Reviewers. Institute of Ethnology and Social Anthropology of the Slovak Academy of Sciences. Available online: https://uesa.sav.sk/?q=sk/recenzne-konanie-pravidla-pre-recenzentov Editorial Activity Guidelines. Faculty of Special Technology. Trenčín: Trenčín University of Alexander Dubček in Trenčín, 2013. Available online: https://fst.tnuni.sk/fileadmin/smernice/Smernica_pre_EC_FST_2013.pdf Rector's Directive No. 5/2006 Editorial Activity Guidelines. Bratislava: Bratislava University of Law. 2006. Available online: https://www.paneurouni.com/wp-content/uploads/2017/03/smernica-c_5_2006.pdf Centre for Scientific Information, Institute of Physiology of the Czech Academy of Sciences. Available online: http://sun2.biomed.cas.cz/fgu/knihovna/veda_hodnoceni.html SULLIVAN, G., M. What to Do When Your Paper Is Rejected. In Journal of Graduate Medical Education, 2015, vol. 7, nr. 1. Available online: https://www.jgme.org/doi/10.4300/JGME-D-14-00686.1?mobileUi=0 WARE, M. Peer review: benefits, perceptions and alternatives. Publishing Research Consortium, 2008, vol. 4. Available online: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.214.9676&rep=rep1&type=pdf WARE, M., MABE, M. The STM Report - An overview of scientific and scholarly journal publishing. Netherlands: International Association of Scientific, Technical and Medical Publishers, 2018. Available online: http://www.stmassoc.org/2015_02_20_STM_Report_2015.pdf Decree No. 397/2020 of the Ministry of Education, Science, Research and Sport of the Slovak Republic on the Central Register of Publication Activity and the Central Register of Artistic Activity

¹ Editorial Activity Guidelines. Faculty of Special Technology. Trenčín: Trenčín University of Alexander Dubček in Trenčín, 2013. (p. 17) ² Peer Review Process and Rules for Reviewers. Institute of Ethnology and Social Anthropology of the Slovak Academy of Sciences ³ Decree No. 397/2020 of the Ministry of Education, Science, Research and Sport of the Slovak Republic on the Central Register of Publication Activity and the Central Register of Artistic Activity (pp. 8-11) ⁴ DOBBERSTEINOVÁ, J., HUDECOVÁ, S., STOŽICKÁ, Z., Guide to the World of Scientific Publishing. Bratislava: Open Science Publishing, Centre for Scientific and Technical Information of the Slovak Republic, 2019. ISBN 978-80-89965-17-5, (pp. 67-74) ⁵ Centre for Scientific Information, Institute of Physiology of the Czech Academy of Sciences