Assessment of the academic quality of scientific journals of Ukraine in physical culture
DOI:
https://doi.org/10.15561/health.2025.0202Keywords:
academic quality, Ukrainian scientific journals, physical culture, indexing status, internationalization, bibliographic quality, methodological rigor, statistical analysisAbstract
Background and Study Aim. In the context of Ukraine’s integration into the global academic space, a systematic assessment of the quality of national scientific journals is of particular importance. The aim of the study was to determine the level of academic quality of Ukrainian scientific journals on physical culture and to identify differences between groups of journals with different indexing statuses.
Materials and Methods. The analysis covered 29 Ukrainian journals with varying levels of indexing (WoS, Scopus, WoS/Scopus, DOAJ, Category B, non-indexed). A structured assessment model was applied based on eight criteria: Novelty, Methodology, Relevance, Analysis, Practical Value, Structure, Sources, and Internationalization. Article evaluation was conducted using an AI tool followed by expert verification. Statistical processing included descriptive statistics, the Shapiro–Wilk test, Spearman’s rank correlation coefficient, and the Mann-Whitney U test.
Results. Journals indexed in WoS/Scopus showed the highest mean scores across all criteria, particularly for Relevance (4.8), Novelty (4.7), and Structure (4.7). Category B journals had the lowest mean values, especially for Sources (3.13) and Internationalization (2.33). The Mann–Whitney U test revealed statistically significant differences (p ≤ 0.001) between indexed and non-indexed journals across all criteria. Spearman’s correlation analysis identified strong positive associations among most criteria, with Internationalization showing relatively weaker correlations. The agreement between the results of the proposed model and the official WoS/Scopus metrics was 86.1%, confirming its validity and compliance with internationally recognized thresholds for a high level of agreement.
Conclusions. The proposed assessment model enables an objective evaluation of the academic quality of Ukrainian journals, highlighting both strengths and areas for improvement. Aligning internationalization indicators, the quality of bibliographies, and methodological rigor with international standards may enhance the visibility and impact of Ukrainian journals in physical culture research.
References
Higher Attestation Commission of Ukraine. Resolution No. 1a/5 dated 22.05.1997. On the approval of lists of scientific journals in which the main results of dissertations may be published [Internet]. Kyiv; 1997 [cited 2025 Aug 5]. Available from: https://zakon.rada.gov.ua/rada/show/v05_7330-99
Iermakov SS. Specialized professional journals of Ukraine in the global and European educational space. Bulletin of the Higher Attestation Commission of Ukraine, 2006;10:3–6. (In Ukrainian).
Iermakov SS. Scientific journals of Ukraine: European dimension. Bulletin of the Higher Attestation Commission of Ukraine, 2008;2:13–15. (In Ukrainian).
Iermakov SS. The level of defended dissertations in citation indicators of publications by participants in the attestation process. Bulletin of the Higher Attestation Commission of Ukraine, 2010;1:10–14. (In Ukrainian).
Iermakov SS. Problems and prospects of publishing an article in a Ukrainian journal included in an international scientometric database. Visnyk of Chernihiv National Pedagogical University. Series: Pedagogical Sciences. Physical Education and Sport, 2013;112(2):104–112. (In Ukrainian).
Podrigalo L, Iermakov S. Duplicate references in the ’Introduction’ and ’Discussion’ sections of scientific articles on physical education and sports. Pedagogy of Health. 2023;2(2):65-73. https://doi.org/10.15561/health.2023.0203 DOI: https://doi.org/10.15561/health.2023.0203
Iermakov S, Korobeynikov G. Assessment of factors influencing the citation level of scientific publications in the field of sport and physical activity. Physical Culture, Recreation and Rehabilitation. 2025;4(1):35-49. https://doi.org/10.15561/physcult.2025.0104 DOI: https://doi.org/10.15561/physcult.2025.0104
Popov MV, Bohatel NV. Indexing of scientific publications in international databases. Science, Technologies, Innovations. 2018;3:16–27. (In Ukrainian).
Mryhlod OI, Mryhlod IM. Collective authorship in Ukrainian science: marginal effect or new phenomenon? Visnyk of the National Academy of Sciences of Ukraine. 2020;7:34–48. (In Ukrainian). https://doi.org/10.15407/visn2020.07.034 DOI: https://doi.org/10.15407/visn2020.07.034
Mingers J, Willmott H. Taylorizing business school research: On the ‘one best way’ performative effects of journal ranking lists. Human Relations, 2013;66(8): 1051–1073. https://doi.org/10.1177/0018726712467048 DOI: https://doi.org/10.1177/0018726712467048
Jaafar R, Pereira V, Saab SS, El-Kassar AN. Which journal ranking list? A case study in business and economics. EuroMed Journal of Business, 2021;16(4): 361–380. https://doi.org/10.1108/EMJB-05-2020-0039 DOI: https://doi.org/10.1108/EMJB-05-2020-0039
Björk BC. Scholarly journal publishing in transition – from restricted to open access. Electronic Markets. 2017;27(2):101–9. https://doi.org/10.1007/s12525-017-0249-2 DOI: https://doi.org/10.1007/s12525-017-0249-2
Chan L, Okune A, Hillyer R, Albornoz D, Reyes Posada A, editors. Contextualizing openness: situating open science. Ottawa: University of Ottawa Press; 2019.
UNESCO. UNESCO Recommendation on Open Science [Internet]. 2021 [cited 2025 Apr 15]. Available from: https://doi.org/10.54677/MNMH8546 DOI: https://doi.org/10.54677/MNMH8546
Moed HF. Citation Analysis in Research Evaluation. Dordrecht: Springer; 2005. https://doi.org/10.1007/1-4020-3714-2
Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997;314(7079):498–502. https://doi.org/10.1136/bmj.314.7079.497 DOI: https://doi.org/10.1136/bmj.314.7079.0f
Archambault É, Larivière V. History of the journal impact factor: Contingencies and consequences. Scientometrics. 2009;79(3):635–49. https://doi.org/10.1007/s11192-007-2036-x DOI: https://doi.org/10.1007/s11192-007-2036-x
Bornmann L, Marx W. The use of altmetrics for measuring societal impact: A review of current studies. EMBO Rep. 2014;15(12):1224–1227. https://doi.org/10.15252/embr.201439608 DOI: https://doi.org/10.15252/embr.201439608
Sugimoto CR, Larivière V. Measuring research: What everyone needs to know. Oxford: Oxford University Press; 2018. DOI: https://doi.org/10.1093/wentk/9780190640118.001.0001
Waltman L. A review of the literature on citation impact indicators. J Informetr. 2016;10(2):365–391. https://doi.org/10.1016/j.joi.2016.02.007 DOI: https://doi.org/10.1016/j.joi.2016.02.007
Frantsvåg JE, Strømme TE. Few open access journals are compliant with Plan S. Publications. 2020;8(2):11. https://doi.org/10.3390/publications8020011 DOI: https://doi.org/10.3390/publications7020026
DORA. Declaration on Research Assessment (DORA). San Francisco; 2012 [cited 2025 Aug 5]. Available from: https://sfdora.org/read
Ukrainian Register of Scientific Journals. IAS "Ukrainian Scientific Periodicals" [Internet]. Ukrainian Institute of Scientific and Technical Expertise and Information; 2025 [cited 2025 Aug 6]. (In Ukrainian). Available from: https://nfv.ukrintei.ua
Hames I, [ed.]. Peer Review and Manuscript Management in Scientific Journals: Guidelines for Good Practice.. 1st ed. Wiley; 2007. https://doi.org/10.1002/9780470750803 DOI: https://doi.org/10.1002/9780470750803
Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Methodology Review Group (ed.) Cochrane Database of Systematic Reviews, 2007;2010(1). https://doi.org/10.1002/14651858.MR000016.pub3 DOI: https://doi.org/10.1002/14651858.MR000016.pub3
Elsevier. Reviewer Guidelines [Internet]. 2023 [updated 2025 Jun; cited 2025 Jun 10]. Available from: https://www.elsevier.com/reviewers
Springer Nature. Reviewer Guidelines [Internet]. 2023 [updated 2025 Jun; cited 2025 Jun 10]. Available from: https://www.springer.com/gp/campaign/discover-journals/reviewer-guidelines?srsltid=AfmBOooOdCgQ8MkNR53OkLo9FFh32oyWTt6oBEhd4-OQ4SD1E9Ia8ri-
COPE. Ethical Guidelines for Peer Reviewers [Internet]. 2023 [updated 2025 Jun; cited 2025 Jun 10]. Available from: https://publicationethics.org/guidance/guideline/ethical-guidelines-peer-reviewers
Scopus. Learning & instruction registered report guidelines [Internet]. 2024 [updated 2025 Jun; cited 2025 Jun 10]. Available from: https://legacyfileshare.elsevier.com/promis_misc/JLI%20Registered%20Report%20Guidelines%20FINAL.pdf
Bornmann L, Mutz R, Daniel HD. A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants. Rogers S (ed.) PLoS ONE, 2010;5(12): e14331. https://doi.org/10.1371/journal.pone.0014331 DOI: https://doi.org/10.1371/journal.pone.0014331
Piro FN, Aksnes DW, Rørstad K. A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing. Journal of the American Society for Information Science and Technology, 2013;64(2): 307–320. https://doi.org/10.1002/asi.22746 DOI: https://doi.org/10.1002/asi.22746
Xavier Garcia Lopez. Scientific Article Analysis Tool (SAAT) [Internet]. 2025 [updated 2025 May 31; cited 2025 Jul 12]. Available from: https://www.gptshunter.com/gpt-store/NTYzMzEwNTYxOTI5MzMwYzEx
Carabantes D, González-Geraldo JL, Jover G. ChatGPT could be the reviewer of your next scientific paper. Evidence on the limits of AI-assisted academic reviews. El Profesional de la información, 2023; e320516. https://doi.org/10.3145/epi.2023.sep.16 DOI: https://doi.org/10.3145/epi.2023.sep.16
Tai AMY, Meyer M, Varidel M, Prodan A, Vogel M, Iorfino F, et al. Exploring the potential and limitations of ChatGPT for academic peer-reviewed writing: Addressing linguistic injustice and ethical concerns. J Acad Lang Learn. 2023;17(1):T16–T30.
Saad A, Jenko N, Ariyaratne S, Birch N, Iyengar KP, Davies AM, et al. Exploring the potential of ChatGPT in the peer review process: An observational study. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 2024;18(2): 102946. https://doi.org/10.1016/j.dsx.2024.102946 DOI: https://doi.org/10.1016/j.dsx.2024.102946
Reed MS, Ferré M, Martin-Ortega J, Blanche R, Lawford-Rolfe R, Dallimer M, et al. Evaluating impact from research: A methodological framework. Research Policy, 2021;50(4): 104147. https://doi.org/10.1016/j.respol.2020.104147 DOI: https://doi.org/10.1016/j.respol.2020.104147
Wang Z, Zhang H, Chen H, Feng Y, Ding J. Content-based quality evaluation of scientific papers using coarse feature and knowledge entity network. Journal of King Saud University - Computer and Information Sciences, 2024;36(6): 102119. https://doi.org/10.1016/j.jksuci.2024.102119 DOI: https://doi.org/10.1016/j.jksuci.2024.102119
Horbach SPJM, Halffman W. The ability of different peer review procedures to flag problematic publications. Scientometrics, 2019;118(1): 339–373. https://doi.org/10.1007/s11192-018-2969-2 DOI: https://doi.org/10.1007/s11192-018-2969-2
Pranckutė R. Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World. Publications, 2021;9(1): 12. https://doi.org/10.3390/publications9010012 DOI: https://doi.org/10.3390/publications9010012
Mongeon P, Paul-Hus A. The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics, 2016;106(1): 213–228. https://doi.org/10.1007/s11192-015-1765-5 DOI: https://doi.org/10.1007/s11192-015-1765-5
Singh VK, Singh P, Karmakar M, Leta J, Mayr P. The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis. Scientometrics, 2021;126(6): 5113–5142. https://doi.org/10.1007/s11192-021-03948-5 DOI: https://doi.org/10.1007/s11192-021-03948-5
MONU. List of scientific professional publications of Ukraine [Internet]. 2025 [updated 2025 May 31; cited 2025 Jul 12]. Available from: https://mon.gov.ua/nauka/nauka-2/atestatsiya-kadriv-vishchoi-kvalifikatsii/naukovi-fakhovi-vidannya
Mason J. Qualitative researching. 2nd ed. London: SAGE Publications; 2002.
White MD, Marsh EE. Content Analysis: A Flexible Methodology. Library Trends, 2006;55(1): 22–45. https://doi.org/10.1353/lib.2006.0053 DOI: https://doi.org/10.1353/lib.2006.0053
O'Leary Z. The essential guide to doing your research project. 3rd ed. London: SAGE Publications; 2017.
Luukkonen T, Tijssen RJW, Persson O, Sivertsen G. The measurement of international scientific collaboration. Scientometrics, 1993;28(1): 15–36. https://doi.org/10.1007/BF02016282 DOI: https://doi.org/10.1007/BF02016282
OECD. OECD Science, Technology and Innovation Outlook 2016. OECD; 2016. https://doi.org/10.1787/sti_in_outlook-2016-en DOI: https://doi.org/10.1787/sti_in_outlook-2016-en
Kwiek M. International Research Collaboration and International Research Orientation: Comparative Findings About European Academics. Journal of Studies in International Education, 2018;22(2): 136–160. https://doi.org/10.1177/1028315317747084 DOI: https://doi.org/10.1177/1028315317747084
SSTL. Open Ukrainian Scientific Content Initiative [Internet]. State Scientific and Technical Library of Ukraine, 2025 [updated 2025 May 31; cited 2025 Jul 12]. Available from: https://ouci.dntb.gov.ua/en/about/
Crossref. Metadata Search [Internet]. Crossref, 2025 [updated 2025 May 31; cited 2025 Jul 12]. Available from: https://search.crossref.org
Superchi C, González JA, Solà I, Cobo E, Hren D, Boutron I. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Medical Research Methodology, 2019;19(1): 48. https://doi.org/10.1186/s12874-019-0688-x DOI: https://doi.org/10.1186/s12874-019-0688-x
Kousha K, Thelwall M. Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper. Journal of the Association for Information Science and Technology, 2024;75(3): 215–244. https://doi.org/10.1002/asi.24810 DOI: https://doi.org/10.1002/asi.24810
Bornmann L. Measuring impact in research evaluations: a thorough discussion of methods for, effects of and problems with impact measurements. Higher Education, 2017;73(5): 775–787. https://doi.org/10.1007/s10734-016-9995-x DOI: https://doi.org/10.1007/s10734-016-9995-x
Agustian HY. Methodological Rigor in Laboratory Education Research. Laboratories, 2024;1(1): 74–86. https://doi.org/10.3390/laboratories1010006 DOI: https://doi.org/10.3390/laboratories1010006
Kosmützky A. The Precision and Rigor of International Comparative Studies in Higher Education. In: Huisman J, Tight M (eds.) Theory and Method in Higher Education Research, Emerald Group Publishing Limited; 2016. p. 199–221. https://doi.org/10.1108/S2056-375220160000002010 DOI: https://doi.org/10.1108/S2056-375220160000002010
Synowiec C, Fletcher E, Heinkel L, Salisbury T. Getting Rigor Right: A Framework for Methodological Choice in Adaptive Monitoring and Evaluation. Global Health: Science and Practice, 2023;11(Supplement 2): e2200243. https://doi.org/10.9745/GHSP-D-22-00243 DOI: https://doi.org/10.9745/GHSP-D-22-00243
Traag VA, Malgarini M, Sarlo S. Metrics and peer review agreement at the institutional level. Quantitative Science Studies, 2025;6: 328–350. https://doi.org/10.1162/qss_a_00352 DOI: https://doi.org/10.1162/qss_a_00352
Tunger D, Schaer P. On the Alignment of Post-Publication Reviews & Bibliometric and Altmetric Impact -- A Case Study on Expert Statements from the Science Media Center Germany. 2025. https://doi.org/10.48550/ARXIV.2503.22594
McHugh ML. Interrater reliability: the kappa statistic. Biochemia Medica, 2012;22(3): 276–282. DOI: https://doi.org/10.11613/BM.2012.031
Matthew Graham, Anthony Milanowski, Jackson Miller Westat. Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. Center for Educator Compensation Reform; 2012.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics, 1977;33(1): 159–174. DOI: https://doi.org/10.2307/2529310
Waltman L, van Eck NJ, van Leeuwen TN, Visser MS, van Raan AFJ. On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff. Scientometrics, 2011;88(3): 1017–1022. https://doi.org/10.1007/s11192-011-0425-7 DOI: https://doi.org/10.1007/s11192-011-0425-7
Traag VA, Waltman L. Systematic analysis of agreement between metrics and peer review in the UK REF. Palgrave Communications, 2019;5(1): 29. https://doi.org/10.1057/s41599-019-0233-x DOI: https://doi.org/10.1057/s41599-019-0233-x
Santini A. The Importance of Referencing. J Crit Care Med (Targu Mures). 2018;4(1):3–4. DOI: https://doi.org/10.2478/jccm-2018-0002
Kumar A. Quality of references reveal the merit of the research. Journal of Indian Society of Periodontology, 2022;26(6): 519–520. https://doi.org/10.4103/jisp.jisp_410_22 DOI: https://doi.org/10.4103/jisp.jisp_410_22
Mammola S, Fontaneto D, Martínez A, Chichorro F. Impact of the reference list features on the number of citations. Scientometrics, 2021;126(1): 785–799. https://doi.org/10.1007/s11192-020-03759-0 DOI: https://doi.org/10.1007/s11192-020-03759-0
Tykhonkova I. References are the important indicator of articles` quality. How to escape an extra work. In: Ukrainian science in the global information space. 2015. P. 100–106. https://doi.org/10.13140/RG.2.1.4176.5842
Vlokh R. Bibliometric and statistical analyses of Ukrainian scientific journals. Physics journals. Ukrainian Journal of Physical Optics, 2010;11(4): S11. https://doi.org/10.3116/16091833/11/4/S11/2010 DOI: https://doi.org/10.3116/16091833/11/4/S11/2010
Demin D. National Analytical Scientific Base of Ukraine as a resource of quality scientific publications for Web of Science and Scopus: conceptual solution and information and mathematical support. ScienceRise, 2024;(2): 73–95. https://doi.org/10.21303/2313-8416.2024.003559 DOI: https://doi.org/10.21303/2313-8416.2024.003559
Nazarovets S. Analysis of publications by authors of Ukrainian institutes in Scopus‐delisted titles. Learned Publishing, 2022;35(4): 499–515. https://doi.org/10.1002/leap.1464 DOI: https://doi.org/10.1002/leap.1464
Nazarovets S, Mryglod O. Ukrainian arts and humanities research in Scopus: a bibliometric analysis. Library Hi Tech, 2025;43(1): 156–179. https://doi.org/10.1108/LHT-05-2023-0180 DOI: https://doi.org/10.1108/LHT-05-2023-0180
Mryglod O, Nazarovets S, Kozmenko S. Universal and specific features of Ukrainian economic research: publication analysis based on Crossref data. Scientometrics, 2021;126(9): 8187–8203. https://doi.org/10.1007/s11192-021-04079-7 DOI: https://doi.org/10.1007/s11192-021-04079-7
Downloads
Published
How to Cite
Issue
License
Copyright (c) 2025 Sergii Iermakov, Georgiy Korobeynikov, David Curby
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright Holder - Author(s). more
Abstract views: 451 / PDF downloads: 232
