Support Programs to Increase the
Number of Scien6fic Publica6ons Using Bibliometric Measures: The Turkish Case
Yaşar Tonta
Hace%epe University
Department of Informa5on Management 06800 Beytepe, Ankara, Turkey
yunus.hace%epe.edu.tr/~tonta/tonta.html yasartonta@gmail.com
@yasartonta
ISSI 2015, June 29-‐July 3, 2015, Boğaziçi University, İstanbul, Turkey
Plan
• Use of bibliometric measures in research
evalua5on (JIF, h-‐index, Ar5cle Influence Score)
• TUBITAK’s Support Program of Interna5onal Scholarly Publica5ons
• Method
• Findings
• Discussion and conclusions
Turkey in Brief
• 185 universi5es
• Over 5M students
• 151K faculty
• 7K academic book 5tles published p.a.
• Close to 1,700 academic journals published
• About 400K papers in total in WoS-‐indexed journals
Source: h%ps://ista5s5k.yok.gov.tr/; h%p://tuik.gov.tr/PreTablo.do?alt_id=1086; Kozak, 2014; h%p://dergipark.ulakbim.gov.tr; h%p://webofscience.com
Year # of papers
1990 1181
1991 1404
1992 1748
1993 2287
1994 2933
1995 3423
1996 4408
1997 5183
1998 6058
1999 6708
2000 6984
2001 8428
2002 10819
2003 13206
2004 16358
2005 17653
2006 20114
2007 24175
2008 25128
2009 28525
2010 29480
2011 30425
2012 32841
2013 35717
2014 34231
0 5000 10000 15000 20000 25000 30000 35000 40000
1990 1995 2000 2005 2010 2015
Number of papers with Turkish affilia5ons
Source: h%p://webofscience.com, 26 April 2015
Research evalua5on
• Peer review
• Academic tenure and performance
• Research and publica5on support
– E.g.,Research Excellence Framework (REF)
– Publica5on support (e.g., TUBITAK, universi5es)
Impact-‐factor obsession
Source: Hicks et al., 2015, p. 431
Journal Impact Factor (JIF)
• Cita5on indexes
• Web of Science (WoS)
• Journal Cita5on Reports (JCR)
• Journal Impact Factor
– Developed to help librarians in collec5on development – Average number of cita5ons to papers published in a
journal (# of cita5ons / # of citable items)
Drawbacks of cita5on-‐based metrics
• Skewed cita5on distribu5ons
– 43% of 11,500 journals listed in JCR 2012 have JIF’s between 0 and 1
• Different publica5on and cita5on prac5ces in different disciplines
– Top Economics paper received 60 cita5ons whereas top Science paper received more than 1,000 cita5ons between 2008-‐2012 (hence JIFs vary)
• Changing publishers policies
– Web of Science indexed c. 8,000 journals in 2006 and 11,500 in 2012
• “Gaming” and manipula5on
• “Bean coun5ng”
• Do not to use “journal-‐based metrics . . . as a surrogate measure of the quality . . .” (San Francisco, 2012)
• But . . . the Higher Educa5on Council and TUBITAK are using it
Sources: Kaynak: Al ve Soydal, 2014; Casadevall ve Fang, 2014; Hicks, 2015; Seglen, 1997; Sgroi ve Oswald, 2013, s. F256; Tonta, 2014a,
2014b, 2015; LSE, 2011, s. 26; SCImago, 2015, h%p://www.scimagojr.com
TUBITAK’s Support Program of Int’l Scholarly Papers
• 1993-‐2012: based on JIFs taken from JCR
– Journals classified as A, B, C and D based on their JIFs
• 2013: based on JIF5 and cited half-‐life (which has more to do with the obsolescence of the paper than its quality)
• 2014: based on Ar5cle Influence Score (AIS), which measures the average impact of a paper (similar to Google’s PageRank algorithm) and is highly correlated with JIF
Kaynak: Arendt, 2010; Franceschet, 2010; Tonta, 2014a, 2014b, 2015
TUBITAK’s Support Program of Int’l Scholarly Papers
2011 2012 2013 2014
8.01 8.15
10.06
11.5
11,342 11,708
10,653
11,530
11,721 12,053
11,199
12,449
Teşvik Tutarı (Milyon TL) Yayın Sayısı Araşyrmacı Sayısı
Source: Satoğlu, 2015 h%p://193.140.98.158/wp-‐content/uploads/2015/03/ULAKBIM_EYES-‐Acilis.pptx
Amount of support (MTL) # of papers # of researchers
Purpose
• To examine the impact of TUBITAK’s most recent algorithmic changes (2013-‐2014) and compare them with that of pre-‐2012
• To understand the mo5ves behind changes and their effects on journal scores
determining the amount of TUBITAK’s
monetary support
Method
• Stra5fied sample
– from TUBITAK’s list of 2012 journals (N=11,562) – Strata: journals grouped under A (36%), B (21%), C
(41%), and D (2%)
– One third were Social Science journals
• Sample size: 2% (n=232)
Popula5on parameters & sample sta5s5cs
Findings
Amount of support (in Turkish Lira)
Sca%er of journals by amount of TUBITAK support (2012-‐2014)
Pearson’s r’s
• 2012-‐2013 r = 0.29
• 2012-‐2014 r = 0.23
• 2013-‐2014 r = 0.77
Correla5ons
• Group A journals of 2012 received less support in 2013 and 2014
• Out of 84 A journals in 2012, only 15 (18%)
maintained their top posi5ons in 2013 and 2014 (12 Science and 3 Social Science journals)
• No discernible pa%erns between the amount of support in 2012 and the succeeding years
– 2012-‐2013 r = 0.29 (p = .000) – 2012-‐2014 r = 0.23 (p = .000)
• Moderate correla5on between 2013 and 2014
– 2013-‐2014 r = 0.77 (p = .000)
Comparison of 2013 and 2014 journals
!
r = 0.77
Rela5onship between journal score and the amount of support in 2013
!
Rela5onship between journal score and the amount of support in 2014
!
TUBITAK’s support threshold is low
• Almost all WoS indexed journals supported
• One third barely meet the minimum criteria
• Support to more than 3,000 journals can be discon5nued
• 80%-‐90% of journals received less than 2,500TL
• Only 5% of journals received more than 4,000TL
• Social Science journals are worse
Amount of TUBITAK support for Science vs. Social Science journals (2014)
!
Discussion
• Journals that performed poorly in 2012 did so in succeeding years, too
• 2013 and 2014 algorithms do not differ much
– 2013: avg=701.00TL, median꞊564.00TL – 2014: avg=770.00TL, median꞊577.00TL
• Because both JIFs and AISs are based on the number of cita5ons and highly correlated
• TUBITAK nullified its earlier decision of not suppor5ng Group C Science journals, although a few Group C
Science journals performed differently in 2013 and 2014
• TUBITAK’s support does not seem to encourage
authors to publish in more pres5gious journals
Conclusions
• JIF: “fatal a%rac5on”, “poor man’s cita5on analysis”
• bibliometric performance measures alone are not the sole criteria for research assessment and . . . they “should be
applied only as a collec6ve group (and not individually)” (IEEE, 2013, original emphasis)
• JIF should not supplant peer review but support it
• Different publica5on/cita5on prac5ces by different fields should be taken into account
• Bibliometric measures should not be used to measure the quality of papers, researchers and ins6tu6ons and compare them with each other
Source: Van Raan, 2005; IEEE, 2013; Hicks et al., 2015; Marx & Bornmann, 2013; h%p://am.ascb.org/dora/
Epilogue
“Not everything that counts can be counted, and not everything that can be counted counts.”
-‐-‐ William Bruce Cameron, 1963
“When a measure becomes a target, it ceases to be a good measure.”
-‐-‐ Charles Goodhart, 1975
Kaynak: h%p://quoteinves5gator.com/2010/05/26/everything-‐counts-‐einstein/; h%p://en.wikipedia.org/wiki/Goodhart%27s_law
Sources
• Al, U. & Soydal. İ. (2014). Akademinin ayf dizinleri ile savaşı. Hace>epe Üniversitesi Edebiyat Fakültesi Dergisi, 31(1): 23-‐42.
h%p://yunus.hace%epe.edu.tr/~umutal/publica5ons/war.pdf
• Arendt, J. (2010). Are ar5cle influence scores comparable across scien5fic fields? Issues in Science and Technology Librarianship, No. 60.
h%p://www.istl.org/10-‐winter/refereed2.html
• Casadevall, A. & Fang, F.C. (2014). Causes for the persistence of impact factor mania. mBio, 5(2). tarihinde h%p://mbio.asm.org/content/5/2/e00064-‐14.full.pdf
• Centre for Science and Technology Studies. (2007). Scoping study on the use of bibliometric analysis to measure the quality of research in UK higher educa5on ins5tu5ons. Report to HEFCE. Leiden: Centre for Science and Technology Studies, Leiden University.
• Franceschet, M. (2010). Journal influence factors. Journal of Informetrics, 4(3), 239-‐248. Retrieved, January 26, 2015, from h%ps://users.dimi.uniud.it/~massimo.franceschet/publica5ons/joi10b.pdf
• Hicks, D., Wouters, P., Waltman, L., de Rijcke, S. & Rafols, I. (2015, 23 Nisan). The Leiden Manifesto for research metrics. Nature, 520, 429-‐431.
h%p://www.nature.com/polopoly_fs/1.17351!/menu/main/topColumns/topLe}Column/pdf/520429a.pdf
• IEEE. (2013, September 9). Appropriate use of bibliometric indicators for the assessment of journals, research proposals, and individuals. Retrieved, April 7, 2015, from h%p://www.ieee.org/publica5ons_standards/publica5ons/rights/ieee_bibliometric_statement_sept_2013.pdf
• Kozak, N. (2014). Türkiye akademik dergiler rehberi-‐2014. Ankara: Detay Yayıncılık.
• LSE Public Policy Group. (Nisan 2011). Maximizing the impacts of your research: A handbook for social scien5sts.
h%p://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf
• Marx, W. & Bornmann, L. (2013). Journal Impact Factor: “the poor man’s cita5on analysis” and alterna5ve approaches. European Science EdiPng, 39(2), 62-‐63.
h%p://www.ease.org.uk/sites/default/files/aug13pageslowres.pdf.
• Satoğlu, M.M. (2015, 4 Mart). Bilimsel yayınlarda kaliteye ulaşmak: Editör ve yazar eği5m seminerleri.
h%p://193.140.98.158/wp-‐content/uploads/2015/03/ULAKBIM_EYES-‐Acilis.pptx
• SCImago. (2015). SJR — SCImago Journal & Country Rank. h%p://www.scimagojr.com.
• Seglen, P.O. (1997, Şubat 15). Why the impact factor of journals should not be used for evalua5ng research. Bri5sh Medical Journal, 314(7079):498-‐502.
h%p://www.dcscience.net/seglen97.pdf
• Sgroi, D. & Oswald, A.J. (2013). How should peer-‐review panels behave? The Economic Journal, 123(570), F255–F278.
• Tonta, Y. (2014a). Akademik Performans, Öğre5m Üyeliğine Yükseltme ve Yayın Destekleme Ölçütleriyle İlgili Bir Değerlendirme (yayımlanmamış makale).
h%p://yunus.hace%epe.edu.tr/~tonta/yayinlar/tonta-‐yukseltme-‐kriterleri-‐hakkinda-‐degerlendirme-‐11-‐Temmuz-‐2014.pdf.
• Tonta, Y. (2014b). Use and Misuse of Bibliometric Measures for Assessment of Academic Performance, Tenure and Publica5on Support. Metrics 2014: Workshop on Informetric and Scientometric Research (SIG/MET). 77th Annual MeePng of the AssociaPon for InformaPon Science and Technology, October 31-‐November 5, 2014, Sea>le, WA. Bildirinin tam metni: h%p://bit.ly/1ur3cGN; Slaytlar: h%p://slidesha.re/10VuVTf
• Tonta, Y. (2015). Support Programs to Increase the Number of Scien5fic Publica5ons Using Bibliometric Measures: The Turkish Case (bildiri). ISSI 2015, June 29-‐July 4, 2015, İstanbul.
• Van Raan, A.F.J. (2005). Fatal a%rac5on: conceptual and methodological problems in the ranking of universi5es by bibliometric methods. Scientometrics, 62(1), 133–43.
