1068
■Τ8
;äf îüieU TY If ШЙЛ'гЯ Ш ЙШ.?Т®
Вѣиі.
Û f E F İ ESSAYS 3Υ Ш . е т Й Ш Ѵ Ш И»шітші
г г і ® ш $Ш»ІЙТТЕЭ TS TÎS FASILTY
W
ЙШАШТІЕ! ІЛЕ ^ЕГГШ
і
; ;
іÏ S
ί'όίΤίΤΐΤΕ
Üİ-
icsasisss
asî
îîîîâi
і і ш ш B'^L^ıiîT ϋ ί ϊ ΐ ϊ ι ί Β Γ ί if. ^
ш ш р м м ш ! ы іш т т ш ш т
■r-fî.Ş·
Ущр
В£'§Й—в в ? fä'.'ii· ί£?- 4?‘? jTHE RELIABILITY OF HOLISTIC AND ANALYTIC EVALUATIONS OF EFL ESSAYS BY TURKISH UNIVERSITY
PREPARATORY STUDENTS
A THESIS
SUBMITTED TO THE FACULTY OF LETTERS AND HUMANITIES AND THE IN ST IT U T E OF ECONOMICS AND SOCIAL SCIENCES
OF BILKENT UNIVERSITY
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS
IN THE TEACHING OF ENGLISH AS A FOREIGN LANGUAGE
ia.c.r.fjc'cn i
BY
§EHNAZ §AHINKARAKA§ AUGUST 1993
pe
l o é ?
to my h u s b a n d w ith my lo v e .
ABSTRACT
T itle : T h e r e lia b ility of h o lis tic a n d a n a ly tic e v a lu a tio n s of th e EFL e s s a y s b y T u r k is h U n iv e r s ity p r e p a r a t o r y s t u d e n t s
A u th o r: Ş e h n a z Ş a h in k a r a k a ş
T h e s is C h a ir p e r s o n : D r. Dan J . T a n n a c ito , B ilk e n t U n iv e r s ity , MA TEFL P ro g ra m
T h e s is C om m ittee M em bers: Ms. P a tr ic ia B r e n n e r , D r. L in d a L a u b e , B ilk e n t U n iv e r s ity , MA TEFL P ro g ra m
T h is s t u d y a tte m p te d to in v e s t ig a te a re lia b le m eth o d of s c o r in g
e s s a y s . Two h y p o th e s e s w ere t e s t e d . O b s e rv a tio n s w ere m ade p e r ta in in g to th e s c o r in g s y s te m u s e d a t th e p r e p a r a t o r y sch o o l of Ç u k u ro v a U n iv e r s ity . A to ta l of 150 EFL p r e p a r a t o r y s t u d e n t s p a r tic ip a te d in th e s t u d y . T h e se s t u d e n t s w ro te tw o e s s a y s : one f o r th e f i r s t h y p o th e s is a n d one f o r th e s e c o n d . T h e f i r s t e s s a y s w ere r a t e d an eily tically b y th e t e a c h e r s a t
Ç u k u ro v a U n iv e r s ity . T h e s e c o n d e s s a y s w ere r a t e d h o lis tic a lly a n d a n a ly t ic a lly b y f o u r r a t e r s who h a v e e x p e r ie n c e a t EFL te a c h in g s itu a tio n fo r a t le a s t fiv e y e a r s . C o r r e la tio n s w ere m ade to fin d th e r e la tio n s h ip s b e tw e e n th e s c o r e s g iv e n b y th e r a t e r s f o r th e s c o r in g m e th o d s.
T h e f i r s t h y p o th e s is w as t h a t th e s c o r in g s y s te m u s e d a t Ç u k u ro v a U n iv e r s ity d id n o t h a v e a h ig h le v e l of r e lia b ility . T h e c o r re la tio n a l a n a ly s is of d a ta r e je c te d th i s h y p o th e s is ( r = .9 7 ) . H o w ev e r, d e s c r ip tiv e a n a ly s is sh o w ed t h a t th e c o r r e la tio n of th e s c o r e s alone w ould n o t be s u f f ic ie n t to claim t h a t th i s s y ste m w as re lia b le . In f a c t , o b s e r v a tio n s in d ic a te th e r a t e r s who s c o r e d e s s a y s f o r th e s e c o n d tim e saw th e f i r s t s c o r e s , t h u s c r e a tin g a s e lf - f u lf illin g b ia s .
T h e s e c o n d h y p o th e s is w as t h a t h o lis tic a lly s c o r e d e s s a y s h a v e s ig n if ic a n tly g r e a t e r r e lia b ility th a n a n a ly tic a lly s c o r e d o n e s in th is e d u c a
tio n a l c o n te x t . T h e a n a ly s is of d a ta w as tw o fo ld : i n t e r r a t e r re lia b ility a n d i n t r a r a t e r r e lia b ility . T h e c o r re la tio n f o r i n t e r r a t e r re lia b ility
in d ic a te d t h a t b o th s c o r in g s y s te m s h ad h ig h r e lia b ilitie s . T he i n t e r r a t e r re lia b ility of h o lis tic s c o r in g m ethod w as .8 5 , a n d of a n a ly tic s c o rin g m ethod w as .8 4 . T h e d iff e r e n c e is n e g lig ib le .
in v e s t ig a te d th e r e lia b ility of e ac h c a te g o r y in d iv id u a lly a s well a s th e to ta l. T h e a n a ly s is of c a te g o r ie s r e v e a le d t h a t th e re lia b ility of th e c a te g o r ie s w as n o t a s h ig h a s th e to ta l s c o r e s f o r aneilytic r a t in g . T h e i n t e r r a t e r r e lia b ility w as .75 f o r c o n te n t , .69 f o r o r g a n iz a tio n , .80 fo r v o c a b u la r y , .82 f o r la n g u a g e u s e , a n d .71 f o r m e c h a n ic s.
T h e c o r r e la tio n s f o r i n t r a r a t e r r e lia b ility sh o w ed t h a t th e r e w as n o t a s ig n if ic a n t d if f e r e n c e b e tw ee n th e tw o s c o r in g m e th o d s (p < .0 1 f o r b o th s c o r i n g ) . T h e i n t r a r a t e r re lia b ility of h o lis tic s c o r in g r a n g e d from .70 to .85 a n d of a n a ly tic s c o r in g from .65 to .8 6 .
H o w ev e r, th e c a te g o r ie s s c o r e d on th e a n a ly tic r u b r i c h a d low i n t r a r a t e r re lia b ilitie s . T h e i n t r a r a t e r re lia b ility r a n g e d from .34 to .83 fo r c o n te n t , from .2 3 to .81 f o r o r g a n iz a tio n , from .46 to .80 f o r v o c a b u la ry , from .6 3 to .77 f o r la n g u a g e u s e , a n d from .55 to .80 f o r m e ch a n ic s.
We may c o n c lu d e t h a t h o lis tic s c o r in g is m ore re lia b le th a n a n a ly tic s c o r in g . A lth o u g h th e to ta l s c o r e s of a n a ly tic s c o r in g m ig h t h a v e h ig h r e lia b ility , th e c a te g o r ie s of th i s s c o r in g m ethod m ig h t h a v e v e r y low re lia b ility w h ich may rciise a q u e s tio n a b o u t th e r e lia b ility of a n a ly tic s c o r in g .
BILKENT UNIVERSITY
IN ST IT U T E OF ECONOMICS AND SOCIAL SCIENCES MA THESIS EXAMINATION RESULT FORM
A u g u s t 31, 1993
T h e e x am in in g com m ittee a p p o in te d b y th e I n s t i t u t e of Econom ics a n d S ocial S c ie n c e s f o r th e
th e s i s e x am in atio n of th e MA TEFL s t u d e n t Ş e h n a z Ş a h in k a r a k a ş h a s r e a d th e th e s i s of th e s t u d e n t . T h e com m ittee h a s d e c id e d t h a t th e t h e s i s of th e s t u d e n t is s a t i s f a c t o r y . T h e s is T itle T h e s is A d v is o r
Com m ittee M em bers
R eliab ility of h o lis tic a n d a n a ly tic e v a lu a tio n of th e EFL e s s a y s of T u r k is h u n iv e r s ity p r e p a r a to r y s t u d e n t s
D r. Dan J . T a n n a c ito
B n k e n t U n iv e r s ity , MA TEFL P ro g ra m Ms. P a tric ia B r e n n e r
B ilk e n t U n iv e r s ity , MA TEFL P ro g ra m D r. L in d a L au b e
VI
We c e r tif y t h a t we h a v e re a d th i s th e s i s a n d t h a t in o u r com bined o p in io n i t is fu lly a d e q u a te , in sco p e a n d in q u a lity , a s a th e s i s f o r th e d e g r e e of M a ste r of A r ts .
l i c A m u m r
P a tr ic ia B r e n n e r (C om m ittee M em ber) L in d a L au b e (C om m ittee M em ber) A p p ro v e d f o r th eI n s t i t u t e of Econom ics a n d Social S c ie n c e s
Ali K araosm anoglu D ire c to r
v u ACKNOWLEDGEMENTS
I am m ost g r a t e f u l to my a d v is o r . D r. T a n n a c ito , f o r h is e n d le s s p a tie n t a n d h e lp . He t a u g h t me n o t o n ly how to do r e s e a r c h b u t also how to e n jo y d o in g r e s e a r c h . I will a lw ay s re m e m b er him a n d h is re c o m m en d a tio n s in my f u t u r e s t u d i e s .
I w ould lik e to th a n k my th e s i s com m ittee m e m b e rs, L in d a a n d P a tric ia , a n d my t u t o r , R u th f o r t h e i r s u p p o r t .
I am v e r y g r a t e f u l to my c la s s m a te s A lev , Ş a d iy e , AU a n d A d n a n , who h e lp e d me a lo t f o r my s tu d y a n d all my o t h e r c la s s m a te s f o r e n c o u r a g in g me w h e n e v e r I n e e d e d .
I w ould lik e to th a n k th e a d m in is tr a to r s , th e t e s t i n g co m m ittee, my c o lle a g u e s a n d th e s t u d e n t s a t Ç u k u ro v a U n iv e r s ity f o r t h e i r h elp a n d
u n d e r s t a n d i n g . D eep th a n k s to H ü ly a , N ü k h e t a n d M eral who n e v e r le f t me a lo n e.
More th a n th a n k s a re d u e to T ü r k a y f o r h e r g r e a t s u p p o r t a n d h e lp in my s ta t is tic a l a n a ly s is . How c o u ld I do s t a t i s t i c s w ith o u t h e r ?
A nd my h u s b a n d , my m o th e r, my m o th e r - in - la w , my c h ild r e n a n d F atm a. My th a n k s a re a lso to y o u f o r liv in g w ith me all th r o u g h my s t u d y .
Vlll TABLE OF CONTENTS LIST OF TABLES...x CHAPTER 1 INTRODUCTION ... 1 B ackground o f th e P r o b l e m ...1 P u rp o se o f th e S t u d y ... 3 S ig n i f ic a n c e o f th e S t u d y ... 4
D e lim ita tio n s and L i m ita tio n s o f th e S tu d y ... 4
C o n ce p tu a l D e f i n i t i o n s ... 5
CHAPTER 2 LITERATURE REVIEW... 7
T e s tin g W r i t i n g ... 7 Types o f W ritin g T e s t s ...7 S c o rin g W ritin g M e th o d s ... 9 H o l i s t i c S c o rin g ... 10 A n a ly tic S c o rin g ... 14 R e l i a b i l i t y ... 15 I n t e r r a t e r R e l i a b i l i t y ... 16 I n t r a r a t e r R e l i a b i l i t y ... 17 C o n c l u s i o n ... 18 CHAPTER 3 METHODOLOGY ... 19 I n t r o d u c t i o n ... 19
R ese arch D esign ... 19
S o u rce s o f D a t a ... 20
M e a s u re m e n ts ...21
I n s t r u m e n t s ... 21
P r o c e d u r e ... 22
O b se rv a tio n o f th e S c o rin g Method a t Çukurova U n i v e r s i t y ... 22
D ata C o l l e c t i o n ...23
R a tin g P r o c e d u r e ... 24
S t a t i s t i c a l T ech n iq u es f o r th e D a t a ... 25
CHAPTER 4 RESULTS AND DISCUSSION...26
I n t r o d u c t i o n ...26 D e s c r ip tio n o f th e W ritin g T e s t P ro c e s s a t Ç U ...26 The T e s t ... 26 P r e - r a t i n g ...27 R a tin g S e s s io n ... 27 S p o t - c h e c k i n g ... 29 R e l i a b i l i t y o f ÇU S c o rin g S y s t e m ...29 I n t e r r a t e r R e l i a b i l i t y ...30 I n t e r r a t e r R e l i a b i l i t y o f H o l i s t i c S c o rin g Method . . .30
I n t e r r a t e r R e l i a b i l i t y o f A n a ly tic S c o rin g Method . . .31
I n t e r p r e t a t i o n o f A n a l y s i s ...32 I n t e r r a t e r R e l i a b i l i t y o f th e I n d i v i d u a l C a te g o rie s o f A n a ly tic S c o r i n g ... 32 I n t e r r a t e r R e l i a b i l i t y o f C o n te n t C ateg o ry . .32 I n t e r r a t e r R e l i a b i l i t y o f O rg a n iz a tio n C a t e g o r y ... 33 I n t e r r a t e r R e l i a b i l i t y o f V o cab u lary C a te g o s ry ... 35 I n t e r r a t e r R e l i a b i l i t y o f Language Use C a t e g o r y ... 36 I n t e r r a t e r R e l i a b i l i t y o f M echanics C a t e g o r y ... 37
IX
I n t e r p r e t a t i o n o f D ata T o g e th e r w ith th e A n a ly tic
C a t e g o r i e s ...38 I n t r a r a t e r R e l i a b i l i t y ...39 I n t r a r a t e r R e l i a b i l i t y o f H o l i s t i c S c o rin g ... 39 I n t r a r a t e r R e l i a b i l i t y o f A n a ly tic S c o rin g ... I n t r a r a t e r R e l i a b i l i t y o f C o n te n t C ateg o ry . I n t r a r a t e r R e l i a b i l i t y o f O rg a n iz a tio n C ateg o ry ... I n t r a r a t e r R e l i a b i l i t y o f V o cab u lary C ateg o ry ... I n t r a r a t e r R e l i a b i l i t y o f Language Use C ateg o ry ... I n t r a r a t e r R e l i a b i l i t y o f M echanics C ateg o ry ... I n t e r p r e t a t i o n o f A n a ly s is f o r I n t r a r a t e r R e l i a b i l i t y 40 .41 .42 .42 .43 .44 .44 CHAPTER 5 CONCLUSION... 46 S u m m ary ...46 P e d a g o g ic a l I m p l i c a t i o n s ...47 S u g g e s tio n s f o r F u r th e r R e s e a r c h ...48 BIBLIOGRAPHY... 50 APPENDICES... 53 Appendix A A ppendix B A ppendix C A ppendix C C onsent F o r m ... 53
R u b ric Used a t QU f o r W ritin g S e c t i o n ... 55
T e s t o f W ritte n E n g lis h (T W E )... 56
LIST OF TABLES
TABLE PAGE
1 F i r s t & Second H o l i s t i c C o r r e l a t i o n s o f th e Second E ssay s ... 30 2 F i r s t & Second A n a ly tic C o r r e l a t i o n s o f th e Second E ssay s ... 31 3 F i r s t & Second C o n te n t C ateg o ry C o r r e l a t i o n s o f th e
Second E ssa y s ... 33 4 F i r s t & Second O rg a n iz a tio n C ateg o ry C o r r e l a t i o n s o f th e
Second E ssay s ... 34 5 F i r s t & Second V o cab u lary C ateg o ry C o r r e l a ti o n s o f th e
Second E ssay s ... 35 6 F i r s t & Second Language Use C ateg o ry C o r r e l a t i o n s o f th e
Second E ssa y s ... 36 7 F i r s t & Second M echanics C ateg o ry C o r r e l a t i o n s o f th e
Second E ssay s ... 37 8 I n t e r r a t e r R e l i a b i l i t i e s o f H o l i s t i c , A n a ly tic , and A n a ly tic C a te g o r ie s ... 38 C o r r e l a t i o n C o e f f i c i e n t s o f H o l i s t i c S c o rin g f o r I n t r a r a t e r R e l i a b i l i t y ... .40 10 C o r r e l a t i o n C o e f f i c i e n t s o f A n a ly tic S c o rin g f o r I n t r a r a t e r R e l i a b i l i t y ... .40 11 C o n te n t C ateg o ry C o e f f i c i e n t s f o r I n t r a r a t e r R e l i a b i l i t y ...41 12 O rg a n iz a tio n C a te g o ry C o e f f i c i e n t s f o r I n t r a r a t e r R e l i a b i l i t y . . . 42 13 V o cab u lary C ateg o ry C o e f f i c i e n t s f o r I n t r a r a t e r R e l i a b i l i t y . . . . 43 14 Language Use C ateg o ry C o e f f i c i e n t s f o r I n t r a r a t e r R e l i a b i l i t y . . . 43 15 M echanics C ateg o ry C o e f f i c i e n t s f o r I n t r a r a t e r R e l i a b i l i t y ...44 16 I n t r a r a t e r R e l i a b i l i t i e s o f H o l i s t i c , A n a ly tic , and
CHAPTER 1 INTRODUCTION B a c k g ro u n d of th e Problem
EFL te a c h e r s in c r e a s in g ly a re fa c e d w ith a n e e d to e v a lu a te w ritin g b o th f o r c la ssro o m a n d in s titu tio n a l p u r p o s e s . E v a lu a tin g w ritin g is an im p o r ta n t m e a s u re of a l e a r n e r 's com m unication a b ility . T h e r e s u l t s c an be u s e d f o r t e s t i n g s t u d e n t s ' le v e l of p r o f ic ie n c y , f o r p la c e m e n t p u r p o s e s , a n d so f o r t h . T h e n e e d to e v a lu a te w ritin g h a s le d to th e d e v elo p m en t of s e v e r a l m e th o d s of a s s e s s m e n t in TEFL.
Two of th e s e m e th o d s a r e : a n a ly tic a n d h o lis tic m e th o d s of a s s e s s m e n t. A n a ly tic a s s e s s m e n t u s e s a d e ta ile d r u b r i c in w hich a l i s t of f e a tu r e s a n d c h a r a c t e r i s t i c s of w ritin g to be e v a lu a te d a r e m e n tio n e d in fu ll d e ta il. T h e a d v a n ta g e of t h i s a s s e s s m e n t is t h a t i t allow s th e te a c h e r s to o b s e r v e th e d e v e lo p m e n t of th e s t u d e n t s in d i f f e r e n t w ritin g c h a r a c t e r i s t i c s . F o r e x am p le, a n a ly tic m ethod c an be u s e d to fin d if a s t u d e n t is b e t t e r a t o rg a n iz a tio n th a n la n g u a g e u s e , o r b e t t e r a t v o c a b u la r y , o r a t c o h e r e n c e , a n d so f o r t h . C a re y (1988) d e s c r ib e s th e a d v a n ta g e s of a n a ly tic a s s e s s m en t:
T h is p r o c e d u r e h e lp s a te a c h e r fo c u s on r e le v a n t a s p e c ts of s t u d e n t s ' r e s p o n s e s a n d p r o v id e s a s y s te m a tic way to a s s ig n a p a r tia l c r e d i t . J u s t a s im p o r ta n t, i t allow s s t u d e n t s to see w h e re th e y lo s t p o in ts . U sin g th is m e th o d , t e a c h e r s c an sum m arize th e g r o u p 's p e rfo rm a n c e on main c o m p o n e n ts, a n a ly z e e r r o r s , a n d u se th e e r r o r a n a ly s is to e v a lu a te a n d r e v is e i n s t r u c t i o n , (p . 191)
C a re y m e n tio n s t h a t i t is e a s y f o r th e t e a c h e r s to u n d e r s ta n d w h at p ro b lem s s t u d e n t s face in w ritin g b y u s in g a n a ly tic m ethod b e c a u s e te a c h e r s e v a lu a te w ritin g w ith in d if f e r e n t c a te g o r ie s , n o t a s a w hole u n it, a n d so te a c h e r s will h a v e th e c h a n c e to re e x am in e th e p ro b lem p a r t s of w ritin g .
H olistic a s s e s s m e n t u s e s a le s s d e ta ile d r u b r i c , u s in g o n ly a g e n e r a l im p re s sio n s c a le . T h is a s s e s s m e n t d o es n o t allow t e a c h e r s to e v a lu a te
w ritin g a b ility w ith in d i f f e r e n t c a te g o r ie s . R a t h e r , i t h e lp s t e a c h e r s e v a lu a te w ritin g q u a lity a s a w hole u n it. T he im p re s sio n of th e te a c h e r w hen s / h e r e a d s an e s s a y once is v e r y im p o r ta n t f o r th e e v a lu a tio n . Mann (1988) s t a t e s th e d if f e r e n c e b e tw e e n th e a n a ly tic a s s e s s m e n t a n d h o listic a s s e s s m e n t:
H olistic s c o r in g d if f e r s from a n a ly tic s c o r in g in a d ra m a tic w ay . I n s te a d of a s s e s s in g s e le c te d co m p o sitio n f e a t u r e s , i t r e s p o n d s to
s t u d e n t w ritin g a s a u n it. D ev elo p ed u n d e r th e a u s p ic e s of E d u c a tio n a l T e s tin g S e r v ic e , h o lis tic s c o r in g is q u ic k a n d p r a c tic a l a s w ell a s
c o s t e f f e c tiv e . M ost im p o r ta n t, h o w e v e r, is th e f a c t t h a t th is s c o r in g o p tio n p e rm its th e r a t i n g of to ta l e f f e c tiv e n e s s of th e w ritin g sam p le, n o t j u s t of c e r ta in f e a t u r e s , ( p . 6)
Mann view s h o lis tic s c o r in g a s a b e t t e r m ethod of a s s e s s m e n t th a n a n a ly tic s c o r in g b e c a u s e th e r a t e r will h a v e a c h a n c e to e v a lu a te w ritin g a s a whole a n d b e c a u s e h o lis tic is m ore p r a c tic a l.
T h e s e tw o m e th o d s of a s s e s s m e n t h a v e a d v a n ta g e s a n d d is a d v a n ta g e s . Some r e s e a r c h e r s t h in k t h a t h o listic is a m ore re lia b le m ethod to e v a lu a te q u a lity of w r itin g . On th e o t h e r h a n d , some r e s e a r c h e r s th in k t h a t a n a ly t ic is a m ore re lia b le m eth o d . J a c o b s , Z in k g ra f , W orm uth, H a rtfie l, a n d H u g h e y (1981) e x p la in s r e lia b ility a s th e e x t e n t to w hich a t e s t y ie ld s c o n s is te n t r e s u l t s , i . e . , a r e th e t e s t s c o r e s p r e c is e , s ta b l e , a n d d e p e n d ab le? T h e r e lia b ility c an be t e s t e d in s e v e r a l w a y s; tw o of them — i n t e r r a t e r a n d i n t r a r a t e r reH abiU ty — will be ta k e n in to c o n s id e ra tio n in th is s t u d y . I n t e r r a t e r r e lia b ility is a w ay to t e s t how c o n s is te n t two o r m ore r a t e r s a r e in r a t i n g th e sam e w ritin g sam p le. I n t r a r a t e r r e lia b il ity is a n o th e r w ay to t e s t how c o n s is te n t one r a t e r is in s c o r in g th e same w ritin g sam ple tw ice w ith a s p e c ific tim e in te r v a l b e tw e e n th e two r a t i n g .
To d e te rm in e th e re lia b ility of a n y a s s e s s m e n t m ethod to be u s e d is of g r e a t im p o rta n c e . T h e re lia b ility of a m e th o d , w h e th e r a n a ly tic o r h o lis
tic , c an be im p ro v e d th r o u g h a c a r e f u l tr a in in g of r a t e r s . If r a t e r s a re u n tr a in e d th e y s c o r e e s s a y s in c o n s is te n tly .
P u rp o s e of th e S tu d y
T h e f i r s t g o al of th is r e s e a r c h w as to i n v e s t ig a te th e re lia b ility of th e m eth o d of a s s e s s m e n t u s e d a t Ç u k u ro v a U n iv e r s ity P r e p a r a to r y School. In th is in s t i t u t i o n a p p ro x im a te ly te n th o u s a n d e s s a y s a r e r a te d a y e a r . T h e q u e s tio n s t h u s a r is e : W hat is th e c o n s is te n c y , t h a t i s , th e a g re e m e n t of th e g r a d e s g iv e n to e ac h e s s a y b y d i f f e r e n t r a t e r s , of th e c u r r e n t a n a ly tic a s s e s s m e n t s y ste m u s e d in th i s in s titu tio n ? I f t h e r e is an
in c o n s is te n c y , w h a t i s / a r e th e s o u r c e ( s ) of th i s in c o n s is te n c y ? Is i t th e r u b r i c its e lf ? I s i t th e la c k of tr a in in g ? I s i t th e b a c k g r o u n d of th e r a t e r s ? I s i t th e c o n d itio n s f o r e v a lu a tio n ? I s i t th e p r o c e s s of a s s e s s m ent? How lo n g d o e s th e r a t i n g p r o c e d u r e la s t?
M o re o v er, in o r d e r to d e te rm in e th e b e s t m eth o d of a s s e s s m e n t, w h e th e r a n a ly tic o r h o lis tic , a s e c o n d q u e s tio n w as i n v e s t ig a te d : Do h o listic a lly
s c o re d e s s a y s o r a n a ly tic a lly s c o r e d e s s a y s h a v e s ig n if ic a n tly g r e a t e r re lia b ility ? To a n s w e r th is q u e s tio n th i s s tu d y u s e d tw o d i f f e r e n t r u
b r ic s : th e T e s t of W ritten E n g lish (TWE) (B o y d , 1990) a s r e p r e s e n t a t iv e of h o listic s c o r in g (se e A p p e n d ix C) a n d th e ESL C om position P ro file ( J a c o b s , Z in k g ra f, W orm uth, H a rtfie l & H u g h e y , 1981) a s r e p r e s e n t a t i v e of a n a ly tic s c o r in g (se e A p p e n d ix D ).
T h is s tu d y t e s t e d two h y p o th e s e s . T he f i r s t h y p o th e s is w as t h a t th e a n a ly tic s c o r in g m ethod u s e d a t Ç u k u ro v a U n iv e r s ity P r e p a r a to r y School d id n o t h av e h ig h r e lia b ility . T h e se c o n d h y p o th e s is w as t h a t h o U stically
s c o re d e s s a y s h a d s ig n if ic a n tly g r e a t e r r e lia b ility th a n a n a ly tic a lly s c o re d e s s a y s .
S ig n ific a n c e of th e S tu d y
T h e p r e s e n t s tu d y is s ig n if ic a n t f o r th e in s ti tu t i o n s t h a t a r e ad m in is te r i n g EFL w ritin g e x am s. T h is s tu d y e x p lic a te s th e r e lia b ility of two m e th o d s of w ritin g a s s e s s m e n t: h o lis tic a n d a n a ly tic m e th o d s. I t also e x p la in s w hich of th e s e two m e th o d s is le s s tim e -c o n su m in g a n d s u ita b le to a p p ly .
T h e s tu d y is also s ig n if ic a n t f o r TEFL r e s e a r c h e r s b e c a u s e am ong th e s tu d ie s t h a t h a v e b e e n d o n e c o m p a rin g re lia b ility of h o listic a n d a n a ly tic s c o r in g , d i f f e r e n t r e s u l t s h a v e b e en fo u n d . Some r e s e a r c h claim s t h a t h o lis tic s c o r in g h a s g r e a t e r re lia b ility th a n a n a ly tic w hile o t h e r r e s e a r c h claim s a n a ly tic s c o r in g is m ore re lia b le th a n h o lis tic . T h e r e f o r e , th e s e tw o s c o r in g m e th o d s will be r e e v a lu a te d to see how th e y a p p ly to T u r k is h U n iv e rs ity p r e p a r a t o r y d e p a r tm e n ts .
In su m , th i s r e s e a r c h is s ig n if ic a n t f o r i n s titu tio n s a n d a d m in is tra t o r s who w a n t to fin d o u t th e m ost re lia b le way of a s s e s s in g t h e i r s t u d e n t s ' w r itin g , a n d f o r th e field r e s e a r c h e r s who w a n t to exam ine d if f e r e n t w ay s of a s s e s s m e n t.
D elim itatio n s a n d L im itatio n s of th e S tu d y
T h e d elim ita tio n of th i s s tu d y w as t h a t th e d a ta w as c o lle c te d f o r o nly u n d e r g r a d u a t e a d v a n c e d le v e l s t u d e n t s . In th e p r e p a r a t o r y d e p a r tm e n t of th e u n i v e r s i t y , w h e re th e r e s e a r c h e r c o lle c te d d a ta , th e r e a r e a b o u t one th o u s a n d s t u d e n t s a t f o u r le v e ls : b e g in n e r , lo w e r - in te r m e d ia te , u p p e r - in te rm e d ia te a n d a d v a n c e d . T h e se le v e ls a r e d iv id e d in to tw o: g r a d u a te a n d u n d e r g r a d u a t e s t u d e n t s . T he d a ta c o n s is te d of th e e s s a y s w ritte n by o n ly u n d e r g r a d u a t e a d v a n c e d le v e l s t u d e n t s . I t is p o s sib le to g e n e ra liz e th e r e s u l t s of th o s e s tu d e n t s to all le v e ls in th is s itu a tio n s in c e all
le v e l s t u d e n t s a re ta k in g th e same ty p e of w ritin g exam , t h a t is , d ir e c t w ritin g . T h e d iff e r e n c e is t h a t th e b e g in n e r a n d in te rm e d ia te le v e l s t u d e n t s a r e a s k e d to w rite s h o r t p a r a g r a p h s o r g u id e d c o m p o sitio n s.
D e sp ite th e d if f e r e n c e , th e w ritin g s e c tio n in exam s is e v a lu a te d w ith th e sam e ty p e of s c o r in g m ethod (a k in d of a n a ly tic a s s e s s m e n t) . T h e r e f o r e , it is p o s s ib le to a p p ly th e m ost re lia b le s c o r in g s y s te m w hich w as i n v e s t ig a t e d in th i s s t u d y to all le v e ls in th is in s ti t u t i o n .
T h e main lim itatio n of th i s s tu d y w as th e n u m b e r of w ritin g sam p les r a t e d . A lth o u g h th e r e s e a r c h e r c o lle c te d 150 e s s a y s , th is n u m b e r h ad to be r e d u c e d to 50 b e c a u s e of th e tim e lim ita tio n . S in ce th is r e s e a r c h w as to be d o n e in a s h o r t tim e , i t w ould be v e r y h a r d to e x p e c t th e r a t e r s to s c o r e a ll 150 e s s a y s w ith o u t f a tig u e . T h e d a ta w ere lim ited to fif ty e s s a y s in o r d e r to av o id f a tig u e .
T h e o t h e r lim itatio n of th i s s tu d y w as t h a t th e p a r t i c i p a n t s co u ld n o t w rite a s e c o n d e q u iv a le n t e s s a y to co m p are h o listic a n d a n a ly tic s c o r in g m e th o d s. I t w ould be too b u rd e n so m e on th e a d m in is tra tio n to le t th e r e s e a r c h e r c o lle c t m ore d a ta sin c e s t u d e n t s w ro te th e e s s a y s d u r in g t h e i r c la s s tim e. F u r th e r m o r e , i t w ould be a b u r d e n on th e r a t e r s as well to r a t e f if ty m ore e s s a y s in a lim ited tim e.
C o n c e p tu a l D e fin itio n s
T h e re a r e s e v e r a l d e fin itio n s f o r h o lis tic a n d a n a ly tic s c o r in g sy ste m s in th e fie ld of com position e v a lu a tio n . Among t h e s e , th e r e s e a r c h e r u s e d a d e fin itio n b y K atz (1988) sin c e sh e b e s t d e s c r ib e s how th e g u id e lin e s of TWE s c o r in g m eth o d (u s e d in th is s tu d y ) c a n be u s e d to re s p o n d s t u d e n t w ritin g a s a u n it. She e x p la in s "ev e n th o u g h e a c h le v e l of th e g u id e lin e s d e s c r ib e s s p e c if ic , r e q u ir e d a b ilitie s , e v a lu a tio n r e q u i r e s r a t e r s to ta k e all of th e s e p a r t s a n d b le n d them in to a whole s c o re " (p . 199).
T h e d e fin itio n f o r a n a ly tic s c o r in g is t h a t of H u g h e y , W orm uth, H a rtf- ie l, a n d J a c o b s (1983) sin c e t h e i r d e fin itio n b e s t e x p la in s how th e ESL P ro file ( u s e d in th is s tu d y ) c an be u s e d to r e s p o n d s t u d e n t w ritin g a s s e p a r a te f e a t u r e s . T h e y d e s c r ib e a n a ly tic s c o r in g a s " p ro v id in g a sid e v iew , a n o u tlin e , of an ESL w r i t e r 's s u c c e s s a t co m p o sin g o r s y n th e s iz in g
th e main e le m e n ts of w ritin g in to a c o n n e c te d , c o h e r e n t, e ffe c tiv e p iece of w r itte n d is c o u r s e " (p . 139).
On th e i n t e r r a t e r a n d i n t r a r a t e r r e lia b ility , th e r e s e a r c h e r u s e d th e d e fin itio n s b y C a rlso n a n d B rid g e m a n (19 8 6 ). T h e y d e fin e i n t e r r a t e r r e li a b ility a s a way to t e s t how c o n s is te n t tw o o r m ore r a t e r s a r e in r a tin g th e same w ritin g sam ple a n d i n t r a r a t e r re lia b ility a s a w ay to t e s t how c o n s is te n t one r a t e r is in s c o r in g th e sam e w ritin g sam ple tw ice w ith a s p e c ific tim e in t e r v a l b e tw ee n th e tw o r a t i n g .
CHAPTER 2 LITERATURE REVIEW T e s tin g W ritin g
T e s tin g is an im p o r ta n t is s u e b o th in f o r e ig n la n g u a g e a n d in se c o n d la n g u a g e e d u c a tio n b e c a u s e t e s t s a r e u s e d f o r a v a r ie ty of p u r p o s e s . T he w o rd " te s t" is h ig h ly v a lu e d in th e s o c ie ty a c c o r d in g to L lo y d -J o n e s
(19 8 7 ). He p o in ts o u t t h a t th e so cial g o al of u n iv e r s a l e d u c a tio n h a s made th e n u m b e rs of s t u d e n t s so la r g e t h a t m an ag em en t r e q u i r e s p ro o f of accom p lis h m e n t.
H o w ev e r, t e s t i n g is d if f ic u lt. I t is e s p e c ia lly d if f ic u lt f o r w ritin g te a c h e r s b e c a u s e th e r e a r e v a rio u s m odes a n d s ty le s w h ich c a u s e co m p lex ity in ju d g e m e n t. T h e main r e a s o n f o r th i s c o m p le x ity is t h a t e v e r y o n e le a r n s w ritin g th r o u g h o u t t h e i r lif e , m ostly o u t- o f - s c h o o l, a n d from d i f f e r e n t p e o p le , a n d in d if f e r e n t s ty le . In a d d itio n , L lo y d -J o n e s m e n tio n s a n o th e r d iff ic u lty w ith th e t e s t i n g of w ritin g : " I t r e p r e s e n t s a n e f f o r t to re c o r d q u a n tita tiv e ly th e q u a lity of th e w ritin g o r w ritin g s k ills of a g ro u p of p eo p le so t h a t a d m in is tr a to r s c an make p o lic ie s a b o u t e d u c a tio n a l p r o g r e s s " (p . 155).
T e a c h e r s c a n re d u c e th e p ro b le m s to a minimum b y le a r n in g e n o u g h a b o u t th e a r t s of t e s t i n g . F o r e x am p le, w ritin g te a c h e r s c a n le a r n e n o u g h a b o u t th e ty p e s of t e s t in g w ritin g to a p p ly th e a p p r o p r ia te ty p e to t h e i r s itu a tio n . T h e y c a n a lso le a r n e n o u g h a b o u t th e r e lia b ility of th e s c o rin g m ethod th e y a r e u s in g .
A c c o rd in g ly , i t is im p o r ta n t f o r t e a c h e r s of w ritin g to know a b o u t th e w ays to t e s t w r itin g , th e s c o r in g m e th o d s a n d th e r e lia b ility of th e s e
s c o r in g m e th o d s .
T y p e s of W ritin g T e s ts
T e s tin g w ritin g h a s two k in d s of m e a s u re m e n t: I n d i r e c t a n d d ir e c t t e s t s of w r itin g . An in d ir e c t t e s t a s k s s t u d e n t s to r e s p o n d to q u e s tio n s a b o u t co m p o sitio n ( J a c o b s , Z in k g ra f, W orm uth, H a rtfie ld a n d H u g h e y , 1981) o fte n
in a m u ltip le ch o ice fo rm a t (C a rls o n & B rid g e m a n , 1986). A lth o u g h in d ir e c t t e s t s of w ritin g a r e commonly r e f e r r e d to a s o b je c tiv e , H am p -L y o n s (1990) th in k s th i s is a m isn o m er, s in c e hum an ju d g e m e n t is s till c e n t r a l while
c r e a tin g th e a r r a y of q u e s tio n s a n d p o s s ib le a n s w e r s . H am p -L y o n s d e s c r ib e s an i n d ir e c t m e a su re of w ritin g b y s a y in g t h a t :
I t d o e s n o t r e q u i r e th e t e s t t a k e r to w rite c o n tin u o u s p r o s e , a lth o u g h s h e o r he may w rite some w o rd s , a n d th e r e is no room f o r p e rs o n a l i n t e r p r e t a t i o n b y th e t e s t t a k e r s in c e p o s s ib le a n s w e r s a r e p ro v id e d a n d th e " c o r r e c t" one a lr e a d y d e c id e d u p o n . (p . 6)
I n d i r e c t m e a s u re s of w ritin g becam e le s s p o p u la r in th e 1970s a f t e r em p h a s is on la n g u a g e a s com m unication w h ich c a lls f o r d i r e c t t e s t of w ritin g in c r e a s e d . A d i r e c t t e s t of w ritin g h a s a t le a s t th e follow ing fiv e c h a r a c t e r i s t i c s , a c c o rd in g to H am p -L y o n s:
1. E ach in d iv id u a l a c tu a lly , p h y s ic a lly w r ite s a t l e a s t one p iece of c o n tin u o u s t e x t .
2. While th e w r i t e r is p r o v id e d w ith a s e t of i n s t r u c t i o n s a n d m a te ri a l, s / h e is g iv e n a c o n s id e ra b le room w ith in w hich to c r e a te a re s p o n s e to th e p ro m p t.
3. E ach w r itte n t e x t is r e a d b y a t l e a s t o n e , u s u a lly m o re, hum an r e a d e r - j u d g e s who h a s b e e n th r o u g h some p r e p a r a t io n o r tr a in in g f o r e v a lu a tio n p r o c e s s .
4. Each ju d g e m e n t m ade b y r e a d e r s a r e tie d to some common s ta n d a r d mea s u r e m e n t, s u c h a s a d e s c r ip tio n of e x p e c te d p e rfo rm a n c e a t c e r ta in
le v e ls o r one o r s e v e r a l r a t i n g s c a le s .
5. R e a d e r s ' r e s p o n s e s to th e w ritin g a r e e x p r e s s e d a s a n u m b e r o r n u m b e rs of some k in d , n o t w r itte n o r v e r b a l com m ents.
With th e new a p p ro a c h to la n g u a g e a s co m m u n icatio n , a c c o rd in g to C a rlso n & B rid g em an (1 9 8 6 ), d ir e c t m e a s u re s of w ritin g s t a r t e d s e r v in g a s th e p r e f e r r e d m eans f o r a s s e s s in g w ritin g p e rfo rm a n c e b e c a u s e th e y more n e a r ly
a p p ro x im a te r e a l d is c o u r s e . F u r t h e r , th e y r e p o r t t h a t w ritin g sam p les p e rm it th e e v a lu a tio n of a s p e c ts of w r itin g , s u c h a s o r g a n iz a tio n , c o h e r e n c e , a n d th e e la b o r a tio n of id e a s w h ich a r e n o t m e a s u re d w ith in d ir e c t m e a s u re s . A c c o rd in g to J a c o b s a t a l. (1 9 8 1 ), th e b e n e f its of a d ir e c t t e s t of w ritin g a r e t h a t i t ;
1. e m p h a s iz e s to le a r n e r s th e im p o rta n c e of la n g u a g e f o r com m unication; 2. p ro m o te s a c lo s e r m atch b e tw e e n w h a t is t a u g h t a n d w h a t is te s t e d ; 3. is m ore v a lid ; 4. is e a s i e r to p r e p a r e ; 5. p r o d u c e s m ore m e a n in g fu l a n d i n t e r p r e t a b l e r e s u l t s ; 6. c a n in d ic a te le v e l of p ro f ic ie n c y a n d s t r e n g t h s a n d w e a k n e s s e s in th e w r itin g s k ill; 7. c a n b e h ig h ly re lia b le if p r o p e r ly a d m in is te r e d a n d e v a lu a te d ; 8. u tiliz e s th e im p o r ta n t in t u i t i v e , a lb e it s u b je c tiv e , r e s o u r c e s of o t h e r p a r t i c i p a n t s in th e com m unication p r o c e s s - t h e r e a d e r s of w ritte n d is c o u r s e .
S c o rin g W ritin g M ethods
T h e im p o rta n c e of d i r e c t t e s t s of w ritin g fo r c e d th e s e a r c h f o r r e li ab le a n d v a lid s c o r in g m e th o d s. C a rlso n & B rid g e m a n m en tio n th e n e e d to c h a n g e th e c u r r e n t s c o r in g m e th o d s u s e d w ith in d i r e c t t e s t s w hen th e y s a y :
With th e d e v e lo p m e n t of c o m p eten ce in b a sic com m unication s k ills ( w rit i n g , s p e a k in g lis te n in g a n d r e a d in g ) a s a p rim a ry g o al f o r e d u c a tio n a n d w ith th e r e c o g n itio n t h a t m any s t u d e n t s p a s s th r o u g h o u r e d u c a tio n al s y s te m w ith in a d e q u a te E n g lis h - la n g u a g e c o m p e te n c e , e d u c a to r s a re r e a p p r a i s i n g t h e i r m eth o d s a n d r e d e f in in g t h e i r o b je c tiv e s , (p . 126) T h e s e a r c h f o r d i r e c t w ritin g a s s e s s m e n t b r o u g h t two main s c o rin g m e th o d s: h o lis tic a n d a n e d y tic . T he fo rm e r e v a lu a te s w ritin g a s a w hole w h e re a s th e l a t t e r e v a lu a te s d if f e r e n t f e a tu r e s of w r itin g , s u c h a s c o h e r e n c e , v o c a b u l a r y , a n d la n g u a g e u s e .
H olistic S c o rin g
H o listic s c o r in g r e s p o n d s to s t u d e n t w ritin g a s a w hole d is c o u r s e . T h e r a t e r s a r e tr a i n e d on a s e t of i n s t r u c t i o n s , c a lle d a r u b r i c , to g u id e
t h e i r r a t i n g . T h e TOEFL T e s t of W ritten E n g lish (TWE) u s e s a h o listic s c o r in g m eth o d w ith a r u b r i c s c o r e d from 1 to 6 (se e A p p e n d ix C ).
H o listic s c o r in g h a s b e e n th e m ost p o p u la r a s s e s s m e n t to o l in w ritin g . As H uot (1990) m e n tio n s "m any s c h o la r s see i t a s th e m ajor m eans of d ir e c t w ritin g e v a lu a tio n . O th e rs c o n te n d t h a t hoU stic s c o r in g h a s p ro v e n to be th e b e s t eco n o m ical, fle x ib le a n d a p p lic a b le of th e d i r e c t w ritin g i n s t r u m en ts" ( p . 2 0 1 ).
G re g o ry (1991) l i s t s six r e a s o n s f o r th e p o p u la r ity of h o listic s c o r in g :
1. Low c o s t, e sp e c ia lly if c o m p ared w ith m u ltip le ch o ice ty p e of
s c o r in g . T h e b ig g e s t e x p e n s e will be to r a t e r s b u t s in c e m ost p r o je c ts a re b r ie f i t will n o t be v e r y e x p e n s iv e ;
2. T h e e ffic ie n c y of t e s t a d m in is tra tio n : T e s ts can be a d m in is te re d in a 45-50 m in u te c la s s p e rio d ;
3. H igh r e lia b ility ( o v e r .8 5 );
4. T h e a p p e a l of a h o listic a p p ro a c h is to see th in g s a s u n i t s , as c o m p le te , a n d a s w h o les;
5. H o listic r e a d in g is t h o u g h t to be f a c e - to - f a c e e n c o u n te r b e c a u s e th e w r i t e r 's m ind em bodied in w ritte n e x p r e s s io n a n d r e a d e r 's mind a tte m p t in g to se e w h a t is b e in g com m u n icated ;
6. S c o re d e s c r ip ti o n s do n o t v a r y from y e a r to y e a r . A p a p e r e a r n in g a 6 in 1990 s h o u ld e a r n 6 in 1993.
B e sid e s th e s e a d v a n ta g e s " h o lis tic s c o r in g m ethod h a s th e a d v a n ta g e of
b e in g v e r y ra p id " (H u g h e s , 1989, p . 8 6 ). H u g h e s s t a t e s t h a t an e x p e r ie n c e d r a t e r c an s c o r e a o n e -p a g e p iece of e s s a y in j u s t a co u p le of m in u te s o r
e v e n le s s . T h e re a re some o th e r r e s e a r c h e r s who a g re e w ith him (C a rls o n &
B rid g e m a n , 1986; C a r ls o n , B rid g e m a n , Camp & W a a n d e rs, 1985; C o o p er, 1977; G re g o ry , 1991; a n d M ann, 1988). T h e y all a g r e e t h a t th e h o listic r a tin g p r o c e d u r e r a r e l y r e q u i r e s more th a n two m in u te s p e r p a p e r . On th e o th e r h a n d , s in c e in th e h o listic s c o r in g m ethod " th e s c o re m u st r e p r e s e n t w h a t a s o p h is tic a te d r e a d e r i n t e r p r e t s a s a to ta l e ffe c t" (L lo y d - J o n e s , 1987,
p . l 6 4 ) , i t is b e lie v e d to be im p r e s s io n is tic a n d h e n c e b y some r e s e a r c h e r s , u n r e lia b le .
H uot (1 9 9 0 ), f o r e x am p le, re e x a m in in g th e r e lia b ility a n d especiaU y v aH d ity of h o lis tic s c o r in g , p o in ts o u t t h a t h o lis tic s c o r in g is a t a
d i s t i n c t d is a d v a n ta g e b e c a u s e i t is an in d iv id u a lly s c o r e d t e s t . A c co rd in g to him , th e s c o r e s m u st be g e n e r a b z e d to show h o lis tic s c o r in g r e s u l ts r e f le c tin g w ritin g q u a lity . H uot s t a t e s t h a t :
th e m ore re lia b le a t e s t , th e m ore we c an g e n e ra liz e a b o u t i t s o u t
com es. . . . In o th e r w o rd s , we m u st be a b le to generaM ze s c o r e s if we w ish to claim t h a t h o listic s c o r in g r e s u l t s r e f le c t w ritin g q u a lity a n d a b ility . So, th e a b ility to g e n e ra liz e a b o u t s c o r e s re c e iv e d from h o lis tic r a tin g p r o c e d u r e s is lim ited d u e to i t s low r e lia b ility , (p . 203)
In a s t u d y c o m p a rin g s c o r in g te c h n i q u e s , P e r k in s (1983) m e n tio n s t h a t p u b lis h e d r e s e a r c h on h o listic s c o r in g in te rm s of re lia b ility a n d c o n c u r r e n t v a lid ity h a s y ie ld e d c o n tr a d ic to r y f in d in g s . He r e p o r t s th e r e s u l t s of a s tu d y m ade b y D ie d e ric h (1974, as d is c u s s e d in P e r k in s , 1983) w hich show t h a t " o u t of th e 300 e s s a y s g r a d e d , 101 re c e iv e d e v e r y g r a d e from 1 to 9, 94% re c e iv e d e i t h e r s e v e n , e ig h t o r n in e d i f f e r e n t g r a d e s ; a n d no e s s a y re c e iv e d le s s th a n fiv e d if f e r e n t g r a d e s from 53 r e a d e r s " (p . 6 5 3 ).
F o r tu n a te ly , i t is p o s sib le to re a c h re a s o n a b ly h ig h r e b a b ility w ith h o listic s c o r in g w hen th e follow ing p o in ts a re c o n s id e r e d . T h o ro u g h tr a in in g of th e r e a d e r s is n e c e s s a r y . C a rlso n & B rid g e m a n (1986) b e lie v e t h a t th e p ro b lem t h a t comes o u t w ith th e d is p a r ity of th e s t u d e n t s ' e x p e c t
ed s k ills c a n be r e s o lv e d b y tr a in in g a n d re a c h in g a c o n s e n s u s a b o u t how to e v a lu a te s u c h e s s a y s . C a rlso n & B rid g e m a n r e p o r t h ig h re lia b ility (.8 0 to
.8 5 ) f o r hoU stic s c o r in g a f t e r t r a i n i n g , in a s tu d y w ith n a tiv e s t u d e n t , >
p o p u la tio n .
A d d itio n a lly , h o lis tic s c o r in g c a n be h ig h ly re lia b le w h en r a t e r s from sim ilar b a c k g r o u n d s a r e c a r e f u lly t r a i n e d . C o o p er (1977) beH ev es t h a t in a d d itio n to t r a i n i n g , r a t e r s b a c k g r o u n d is also im p o r ta n t. He s t a t e s t h a t i t is p o s s ib le to im p ro v e re lia b ility from a r a n g e of .30 to .75 b e fo re tr a in in g to a r a n g e of .73 to .98 a f t e r t r a i n i n g . T h e follow ing s ta te m e n t b e s t d e s c r ib e s w h a t he b e lie v e s a b o u t th e re lia b ility of h o lis tic s c o r in g :
When r a t e r s a r e from sim ila r b a c k g r o u n d s a n d w hen th e y a r e tr a in e d w ith a h o lis tic s c o r in g g u id e —e i t h e r one th e y b o rro w o r d e v is e f o r th e m
s e lv e s of th e s p o t— th e y c a n a c h ie v e n e a r ly p e r f e c t a g re e m e n t in c h o o sin g th e b e t t e r of a p a ir of e s s a y s ; a n d th e y a c h ie v e s c o r in g re lia b ilitie s in th e h ig h e ig h tie s a n d low n in e tie s on t h e i r sum m ed s c o r e s from m u ltip le p ie c e s of a s t u d e n t 's w ritin g , ( p . 19)
In a s tu d y t h a t e x am in es TOEFL T e s t of W ritte n E n g lish (TW E), B oyd (1990) p o in ts o u t th e im p o rta n c e of tr a in in g f o r r e lia b ility of h o lis tic s c o r in g .
She s t a t e s in h e r fin a l a n a ly s is t h a t " it is n o t th e S c o rin g G uide t h a t g u a r a n te e s re lia b le s c o r in g of TWE p a p e r s , b u t th e n a t u r e of th e tr a in in g t h a t th e r e a d e r s re c e iv e " (p . 101).
M itchell a n d A n d e rso n (1986) r e p o r t th e h ig h re lia b iU ty of h o listic s c o r in g th e y u s e d in a s tu d y a b o u t th e r e lia b ility of h o lis tic s c o r in g .
Each e s s a y w as r e a d b y two r a t e r s a n d if th e p a p e r s r e c e iv e d more th a n one s c o re d is a g r e e m e n t, a t h i r d r e a d in g w as n e e d e d . M itchell a n d A n d e rso n p o in t o u t t h a t "a t h i r d re a d in g w as r e q u ir e d on 5.3% of th e p a p e r s " (p . 77 2 ). T he reU abU ity w as . 94.
H am b u rg (1984) also m en tio n s th e e f f e c t of tr a in in g on th e r e lia b ility of h o listic s c o r in g in a s tu d y t h a t d is c u s s e s th e r e la tio n s h ip b e tw ee n
s u b je c tiv e e v a lu a tio n a n d o b je c tiv e m e a s u re s of ESL w ritin g p ro fic ie n c y . He c o n c lu d e s t h a t " h o lis tic e v a lu a tio n of ESL c o m p o sitio n s, w ith tr a in in g to fam iliarize r e a d e r s w ith th e ty p e s of f e a t u r e s p r e s e n t in ESL com posi tio n s , c an be c o n s id e r e d to be a d e q u a te ly re lia b le a n d v a lid " (p . 103).
G re g o ry (1991) in h is s tu d y e x am in in g th e w e a k n e s s e s of h o listic a s s e s s m e n t a c c e p ts th e h ig h re lia b ility (.8 0 o r a b o v e ) of h o listic s c o r in g . To m ention h is b e lie f a b o u t th e r e lia b ility , he q u o te s H ogan a n d M ish ler (1981) who s a y t h a t "m ost r e s e a r c h e r s a g r e e t h a t th is le v e l of reU abU ity (o v e r .8 0 ) is p o s s ib le , d e s p ite a w id e s p r e a d n o tio n to th e c o n t r a r y am ong la y p e r s o n s " (p . 7 ) .
N e v e r th e le s s , we c an fin d some s tu d ie s t h a t show low r e lia b ility f o r th e h o lis tic m ethod w ith tr a in e d r a t e r s . V a u g h an (1 9 9 0 ), f o r ex am p le,
c o n d u c te d a s tu d y in w hich s h e e v a lu a te d th e " p ro c e s s " b y w hich r a t e r s make t h e i r d e c is io n s d u r in g h o lis tic a s s e s s m e n t. In h e r r e s e a r c h th e p a s s in g
e s s a y s r e c e iv e d 4, 5, o r 6 a n d fa ilin g e s s a y s r e c e iv e d 3 , 2, o r 1. T he r a t e r s w ere tr a in e d a n d a s k e d to g r a d e six e s s a y s com m enting v e rb a lly in a th in k - a lo u d p r o c e d u r e a s th e y r e a d . T h e s e e s s a y s w ere r a te d b e fo re h a n d a t th e u n iv e r s ity b y t e a c h e r s of t h a t u n i v e r s i t y . V a u g h an r e p o r t s t h a t " th e o rig in a l r a t e r s ( te a c h e r s of th e u n iv e r s ity ) p a s s e d o n ly tw o of th e six
e s s a y s (33%). On th e o th e r h a n d , th e r a t e r s in th i s s tu d y a w a rd e d e s s a y s a p a s s in g g r a d e 57 p e r c e n t of th e tim e" (p . 11 5 ). V a u g h a n 's s tu d y show s low re lia b ility of h o lis tic s c o r in g m eth o d . T h e e s s a y s t h a t w ere r a te d by th e o rig in a l te a c h e r s of th e u n iv e r s ity re c e iv e d 33% s u c c e s s w h e re a s th e r a t e r s in V a u g h a n 's s tu d y p a s s e d 57% of th e sam e e s s a y s .
H olistic s c o r in g h a s some o th e r d is a d v a n ta g e s b e s id e s th e q u e s tio n of low re lia b ility of th e s c o rin g m eth o d . Mann (1988) s ta t e s t h a t th e u se of h o listic s c o r in g is in a p p r o p r ia te in te a c h in g s itu a tio n s w h e re d ia g n o stic d a ta is r e q u ir e d sin c e i t g iv e s only a g e n e r a l r a t i n g .
Y e t, G re g o ry (1991) s e e s th is a s an a d v a n ta g e . He p o in ts o u t t h a t
" la n g u a g e is n o t le a r n e d from s u b s e t to s u b s e t , s t r u c t u r e to s t r u c t u r e . I n s t e a d , i t is u n d e r norm al c o n d itio n s 'l e a r n e d a n d u s e d a ll of a p ie c e ' so to s p e a k in a h o lis tic fa sh io n " (p . 1 0 ).
T h e r e f o r e , i t is p o s sib le to fin d v a r io u s r e s u l t s f o r th e r e lia b ility
a n d th e a d v a n ta g e s of h o listic s c o r in g b u t th e a d v a n ta g e s seem to o u tw e ig h th e d is a d v a n ta g e s .
A n aly tic S c o rin g
T h e a n a ly tic s c o r in g m ethod is a n o th e r m eth o d of d i r e c t w ritin g a s s e s s m en t w hich p r o v id e s s e p a r a te s c o r e s . P e r k in s (1983) d e f in e s th i s p r o c e d u r e a s i t in v o lv e s th e s e p a r a tio n of th e v a r io u s f e a t u r e s of a co m p o sitio n in to co m p o n en ts f o r s c o r in g p u r p o s e s . T h is d e fin itio n c a n be a p p lie d to th e ESL C om position P ro file (s e e A p p e n d ix D) a s an a n a ly tic s c o r in g m e th o d .
A lth o u g h th e a u t h o r s c o n s id e r th e ESL P ro file a s a h o lis tic s c o r in g m ethod (J a c o b s e t a d ., p . 2 9 ), i t fu lfills o u r d e fin itio n of an a n a ly tic m ethod by s e p a r a tin g s c o r in g f o r c o n te n t , o r g a n iz a tio n , v o c a b u la r y , la n g u a g e u s e , a n d m e c h a n ic s.
Like h o lis tic s c o r in g , a n a ly tic s c o r in g h a s some a d v a n ta g e s a n d d is a d v a n ta g e s . I t is b e lie v e d to p e rm it re lia b le s c o r e s am ong r a t e r s b y some r e s e a r c h e r s ( C a r e y , 1988; C o n n o r, 1990; H u g h e s , 1989; J a c o b s e t a l, 1981; P e r k in s , 1983). C o n n o r, f o r e x am p le, r e p o r t s th e r e lia b ility of a n a ly tic s c o r in g b e tw e e n .81 to .9 1 . A n o th e r a d v a n ta g e of a n a ly tic s c o r in g is t h a t a s C a re y s t a t e s : " it p ro v id e s a s y s te m a tic w ay to a s s ig n p a r tia l c r e d i t . J u s t a s im p o r ta n t, i t allow s s t u d e n t s to se e w h e re th e y lo s t p o in ts " (p . 191).
H o w ev e r, C a rlso n a n d B rid g em an see th i s a d v a n ta g e of a n a ly tic s c o r in g a s an illu sio n s ta t in g t h a t " th e r e a d e r 's g e n e r a l im p re s sio n is lik e ly to in flu e n c e r a t i n g s on e a c h of th e s e p a r a te a s p e c ts b e in g e v a lu a te d " (p .
E d w a rd W hite (1985, c ite d in M ann, 1988) s t a t e s th e p ro b le m w ith
14
15
a n a ly tic s c a le s in th e follow ing w ay:
T h e re is no e v id e n c e t h a t w ritin g q u a lity is th e r e s u l t of th e accu m u la tio n of a s e r ie s of s u b s k ill s . To th e c o n t r a r y , th e la c k of a g r e e m en t of s u b sk iU s in th e p ro f e s s io n s u g g e s t s t h a t w ritin g re m a in s more th a n th e sum of i t s p a r t s a n d t h a t th e a n a ly tic th e o r y t h a t s e e k s to d e fin e a n d a d d u p th e s u b s k ills is fu n d a m e n ta lly fla w e d . A n aly tic s c o r in g is u n eco n o m ical, u n r e lia b le , p e d a g o g ic a lly u n c e r ta in o r d e s t r u c t i v e a n d th e o r e tic a lly b a n k r u p t , ( p . 6)
White th in k s t h a t th e a n a ly tic s c o r in g m eth o d h a s lo s t it s im p o rta n c e b e c a u s e i t e v a lu a te s w ritin g a s s u b s k ills a n d b e c a u s e i t is n e ith e r p r a c t i cal f o r t e s t i n g p ro f ic ie n c y n o r re lia b le .
T h e main d is a d v a n ta g e of anadytic s c o r in g is t h a t i t is tim e -c o n su m in g . S te p h e n Wiseman (c ite d in C a rlso n a n d B rid g e m a n , 1986) fo u n d t h a t " fo u r g e n e r a l im p re s sio n m a rk in g s (h o listic m a rk in g ) w ere e q u iv a le n t in time a n d e f f o r t to one a n a ly tic m a rk in g " (p . 145). T h e r e f o r e , r a t in g la rg e g ro u p of e s s a y s w ith a n a ly tic s c o r in g c o u ld c a u s e f a tig u e in r a t e r s a n d c o n s e q u e n tly m ig h t a f f e c t i t s r e lia b ility .
In s tu d ie s c o m p a rin g h o lis tic a n d a n a ly tic s c o r in g m e th o d s , b o th h a v e show n some re lia b le r e s u l t s . C a n a le , F r e n e tte a n d B é la n g e r (19 8 8 ), in a s tu d y e v a lu a tin g s t u d e n t w ritin g in f i r s t a n d s e c o n d la n g u a g e , fin d r e li a b ility f o r h o lis tic s c o r in g q u ite h ig h ( r a n g in g from .83 to .92) a n d
g e n e r a lly h ig h f o r a n a ly tic ( r a n g in g from .59 to 9 0 ). H o w ev er, th e r a n g e of a n a ly tic s c o r e s w as w id e r th a n of h o lis tic s o r e s . C a r ls o n , B rid g e m a n , Camp a n d W aan d ers (1985) also r e p o r t h ig h r e lia b ilitie s f o r b o th s c o rin g m eth o d s: from .80 - .8 5 f o r h o listic a n d from .80 - .8 4 f o r a n a ly tic .
R eliab ility
V alid ity is th e m ajor c o n c e rn in la r g e - s c a le te s t in g p ro g r a m s . S ta n s f - ie ld a n d R oss (1988) d e fin e s v a lid ity a s " th e in te r f e r e n c e s made a b o u t a t e s t s c o re : i . e . , th e d e g re e to w hich i t is u s e f u l a s a m e a su re of a
16
p a r t i c u l a r t r a i t f o r a p a r t i c u l a r p u r p o s e a n d f o r a p a r t i c u l a r exam inee" (P . 4 ).
R eliab ility is a n o t h e r c o n c e rn in s u c h t e s t in g p ro g r a m s . "R elia b ility r e f e r s to th e c a p a c ity of th e a s s e s s m e n t p r o c e d u r e s to r a n k - o r d e r th e same sam p les of w ritin g p e rfo rm a n c e c o n s is te n tly in th e same w a y ". (H e n n in g , 1991) H e n n in g m eans a re lia b le t e s t y ie ld s th e same s c o r e s o r r a n k - o r d e r in g f o r th e sam e g r o u p of s t u d e n t s u n d e r d i f f e r e n t c o n d itio n s .
S ta n s fie ld a n d R o ss also s ta t e t h a t in e s s a y t e s t i n g , re lia b ility is of g r e a t e r th a n n o rm al im p o rta n c e b e c a u s e e s s a y t e s t s e x h ib it a good d e a l of face v a lid ity . T h e face v a lid ity of e s s a y t e s t s is t h a t th e y r e q u ir e th e ex am in ee to p e rfo rm in s te a d of d e m o n s tra tin g k n o w led g e a b o u t how to p e r form . H o w ev e r, i t is n o t p o s sib le to s a y th e same fo r th e re lia b ility of e s s a y t e s t s s in c e th e y a r e o p en to s o u r c e s of e r r o r t h a t a r e n o t p r e s e n t in m u ltip le -c h o ic e t e s t s . T h e r e f o r e , in d i r e c t m e a s u re s , s u c h a s e s s a y t e s t s , re lia b ility becom es a n e x te n s iv e c o n c e r n .
T he a c c e p ta b le le v e l of r e lia b ility d if f e r s a c c o rd in g to d if f e r e n t u s e s of th e t e s t . C o o p er (1977) s t a t e s t h a t "a r e lia b ility c o e ffic ie n t of .80 is c o n s id e re d h ig h e n o u g h f o r p ro g ra m e v a lu a tio n , w hile a re lia b ility c o e ffic ie n t of . 90 f o r in d iv id u a l g ro w th m e a su re m e n t in te a c h in g a n d r e s e a r c h " ( p . 1 8 ). S im ilarly , J a c o b s , e t a l. r e p o r t s t h a t " re lia b ility c o e f fic ie n ts of .85 a r e u s u a lly c o n s id e re d a d e q u a te f o r t e s t s u s e d fo r p la ce m e n t p u r p o s e s , b u t h ig h e r re lia b ilitie s — in th e n in e tie s — a re d e s ir a b le a n d r e q u e s t e d f o r t e s t s w hich will be th e b a s is f o r d e c is io n s a b o u t in d iv id u a ls " (p . 6 9 ). T h e re lia b ility of .80 w as a c c e p te d h ig h e n o u g h in th i s s t u d y .
I n t e r r a t e r R eliab ility
R e lia b ility c a n be m e a su re d in a n u m b e r of w ay s; i n t e r r a t e r reU abiU - t y , i n t r a r a t e r r e lia b ility , t e s t - r e t e s t re lia b ility , s p lit- h a lf re lia b ili
re lia b ility m e a s u re m e n ts; i n t e r r a t e r a n d i n t r a r a t e r r e lia b ility .
P e r h a p s th e m ost common m ethod f o r a s s e s s in g th e r e lia b ility of w rit in g -s a m p le t e s t s is to d e te rm in e th e i n t e r r a t e r r e lia b ility . ( G r e e n b e r g ,
1986; H atch & L a z a r a to n , 1991; H e n n in g , 1991; L a u e r & A s h e r , 1988) I n t e r r a t e r r e lia b ility e s tim a te s th e e x t e n t to w hich two o r m ore r a t e r s a g re e on th e s c o re t h a t s h o u ld be a s s ig n e d to a n e s s a y . T h a t i s , tw o r a t e r s wiU. g iv e th e sam e s c o r e s f o r th e same e s s a y s .
I n t e r r a t e r r e lia b ility in v o lv e s d e te r m in in g th e in te r c o r r e la tio n of two o r m ore r a t e r s f o r th e sam e w ritin g sam p le, a n d th e n a d ju s tin g th e o b ta in e d c o e ffic ie n ts b y u s e of th e S p e a rm a n -B ro w n P ro p h e c y fo rm u la , to r e f le c t th e a v e r a g e r a t i n g of th e r a t e r s .
T he n u m b e r of th e r a t e r s is im p o r ta n t in d e te r m in in g th e i n t e r r a t e r re lia b ility b e c a u s e s t u d e n t s ' fin a l s c o r e s a r e th e co m b in atio n o r a v e r a g e of th e r a t i n g s . H e n ce , th e m ore r a t e r s , th e h ig h e r th e r e lia b ility . R a te r tr a in in g is a lso im p o r ta n t f o r h ig h i n t e r r a t e r re lia b ility a s w ell. If th e r a t e r s a r e tr a in e d c a r e f u lly , i t is p o s sib le to a c h ie v e h ig h i n t e r r a t e r r e l i a b i l i t y .
I n t r a r a t e r R eliab ility
I n t r a r a t e r re lia b ility in d ic a te s how c o n s is te n t a s in g le r a t e r is in
s c o r in g th e sam e s e t of e s s a y s tw ice w ith a s p e c ifie d tim e i n te r v a l b etw een th e f i r s t a n d s e c o n d s c o r in g . T h a t i s , a n y p a r t i c u la r r a t e r w ould g iv e th e sam e s c o r e on b o th r a t i n g s .
T he in te r c o r r e la tio n of th e two s c o r in g of one r a t e r f o r th e same w ritin g sam p les is d e te rm in e d a n d th e n th e c o e ffic ie n t is a d ju s te d to th e S p e a rm a n -B ro w n fo rm u la to r e f le c t th e i n t r a r a t e r r e lia b ility .
As i t is in i n t e r r a t e r r e lia b ility , r a t e r tr a in in g is im p o rta n t to in c r e a s e i n t r a r a t e r reU ab U ity . If th e r a t e r s a re tr a in e d w ell in each s c o rin g s e s s io n , i t is p o s sib le to a c h ie v e h ig h i n t r a r a t e r re lia b ility .
C o n clu sio n
T h is re v ie w of l i t e r a t u r e te lls how v a rio u s r e s e a r c h e r s h a v e view ed th e two main s c o r in g m e th o d s of t e s t in g w r itin g , h o lis tic a n d a n a ly tic , w hich th is s tu d y h a s e x a m in e d . T he two m e th o d s h a v e b o th a d v a n ta g e s a n d d is a d v a n ta g e s . E ach of th e two m eth o d s h a s b e en fo u n d re lia b le b y some r e s e a r c h e r s , u n re lia b le by o th e r s .
T h is s tu d y ex am in ed th e re lia b ility of th e s e tw o s c o r in g m eth o d s in EFL c o n te x t in T u r k is h U n iv e r s ity P r e p a r a to r y s c h o o ls , fo c u s in g o n ly th e
i n t e r r a t e r a n d i n t r a r a t e r re lia b ility .
CHAPTER 3 METHODOLOGY I n tr o d u c tio n
T h is s tu d y aim s to d e te rm in e w hich is th e m ore re lia b le w ay of s c o rin g EFL w ritin g sam p les in th e c o n te x t of T u r k is h u n i v e r s i ti e s . T he s c o r in g m ethod u s e d in th e p r e p a r a t o r y d e p a r tm e n t of Ç u k u ro v a U n iv e r s ity (ÇU) — a v a r ia n t of a n a ly tic s c o r in g — w as t e s t e d u s in g a sim ple c o r r e la tio n a l
s ta t is tic a l te c h n iq u e in o r d e r to exam ine th e r e lia b ility of th e m ethod a t th is u n i v e r s i t y . In a d d itio n , two d i s t i n c t s c o r in g m e th o d s , a n a ly tic
(C om position P ro file S cale) a n d h o listic (TWE) s c o r in g m e th o d s , w ere c o r r e la te d to t e s t w h e th e r a n a ly tic o r h o lis tic s c o r in g is m ore re lia b le in th i s s e t t i n g .
R e s e a rc h D esig n
I u s e d a fo c u s e d d e s c r ip tiv e com bined w ith a c o r r e la tio n a l d e s ig n in o r d e r to t e s t th e re lia b ility of th e s c o r in g m eth o d t h a t is u s e d in Ç u k u ro v a u n iv e r s ity a n d to d e te rm in e w h e th e r a n a ly tic o r h o lis tic s c o r in g m ethod is more re lia b le . T h e s tu d y is called a fo c u s e d d e s c r ip ti v e b e c a u s e in s u c h s tu d ie s th e r e s e a r c h e r o b s e r v e s th e a c tiv itie s a n d ta k e n o te s (e . g . , o b s e r v a tio n a l s t u d i e s ) , a n d n a rro w s th e sc o p e of th e s tu d y to a p a r t ic u l a r s e t of v a r ia b le s (L a r s e n -F r e e m a n a n d L o n g , 1991). In th i s s tu d y I n a rro w e d th e s tu d y to th e s y ste m f o r s c o r in g w ritin g sam p les a t th e a d v a n c e d le v e l. I u s e d n o te s o b ta in e d d u r in g o b s e r v a tio n to d e s c r ib e how th e s c o r in g s y ste m to o k place a t Ç u k u ro v a U n iv e r s ity . T h e n u m b e r of r a t e r s , th e n u m b e r of th e p a p e r s e ac h r a t e r r a t e s , th e s c o r in g m e th o d , th e r a t e r s ' t r a i n in g , a s well a s th e p r o c e d u r e s f o r s p o t- c h e c k in g w ere d e s c r ib e d . T h is q u a lita tiv e d a ta w as u s e d to te ll w h e th e r th e s c o r in g p r o c e d u r e s tr e n g t h e n e d o r w e ak e n ed th e re lia b ility of th e s c o r in g s y ste m in th is i n s titu tio n .
T h is s tu d y is te rm e d c o rre la tio n a l a s w ell b e c a u s e c o rre la tio n a l s tu d ie s t r y to e s ta b lis h a re la tio n s h ip b e tw e e n s c a le d o r s c o re d d a ta on one v a ria b le w ith th o s e on a n o th e r (H a tc h & L a z a r a to n , 1991). In th is
s t u d y , I c o r r e la te d th e s c o r e s g iv e n b y th e r a t e r s f o r th e two s c o rin g m e th o d s. T h is q u a n tita tiv e d a ta ex am in ed th e r e la tio n s h ip b e tw ee n th e s e s c o r e s to fin d th e r e lia b ility of th e s c o r in g m e th o d s.
T he follow ing s e c tio n s d e a l w ith who th e s u b je c ts a r e , how th e d a ta w ere c o r r e l a t e d , a n d w h a t s te p s w ere follow ed.
S o u rc e s of d a ta
T he p o p u la tio n of th i s s tu d y is a b o u t 1000 p r e p a r a t o r y EFL s tu d e n t s a t fo u r p ro f ic ie n c y le v e ls : b e g in n e r , lo w e r - in te r m e d ia te , u p p e r - in te r m e d ia te a n d a d v a n c e d le v e l of s t u d e n t s . A s t u d e n t 's p ro f ic ie n c y le v e l is d e t e r m ined b y a s ta n d a r d iz e d p la c e m e n t t e s t , called T h e P r o fic ie n c y E xam ination p r e p a r e d b y C am b rid g e U n iv e r s ity .
At QU p r e p a r a t o r y sc h o o l, s t u d e n t s s tu d y E n g lish i n te n s iv e ly , i. . e . , fiv e d a y s a w eek . S tu d e n ts s tu d y th e f o u r s k ills 25 h o u r s p e r w eek . At th e b e g in n e r , lo w e r-in te rm e d ia te a n d u p p e r - in te r m e d ia te le v e ls , 3 h o u r s o u t of 25 a r e d e v o te d to th e w ritin g s k ills a n d a t a d v a n c e d le v e l 4 h o u rs
T he sam ple f o r th i s s tu d y is 150 s t u d e n t s who a r e a t th e a d v a n c e d le v e l. S tu d e n ts a t th i s le v e l w ere s e le c te d s in c e th e y w e re ab le to w rite e s s a y s r a t h e r th a n s h o r t p a r a g r a p h s . T h e a g e a n d sex of th e s t u d e n t s w ere n o t c o n s id e r e d a s m o d e ra to r v a r ia b le s b e c a u s e th e s e v a ria b le w ere n o t
th o u g h t to a f f e c t th e r e lia b ility of th e s c o r in g m e th o d s. T h is sam ple (150 e s s a y s ) w as u s e d to t e s t th e r e lia b ility of s c o r in g m ethod u s e d in th is u n iv e r s ity .
F ifty o u t of 150 s t u d e n t s w ere s e le c te d b y a sim ple ra n d o m sam pling p r o c e d u r e . I p r e p a r e d 150 p ie c e s of p a p e r s . I m a rk e d 50 of th e s e p a p e r s a n d p u t th em to g e t h e r . On th e o th e r h a n d , I n u m b e re d th e e s s a y s . I p ic k e d one of th e p ie c e s fo r e ac h n u m b e re d e s s a y . If th e p iece w as m a rk e d , th e e s s a y w as s e le c te d . T h is sam ple (50 e s s a y s ) w as u s e d to t e s t w h e th e r a n a ly tic o r h o lis tic s c o r in g m ethod is m ore re lia b le .
T h is s tu d y c o n s is ts of in te r v a l d a ta — th e m a rk s g iv e n b y r a t e r s fo r
