Best Test Design

Citation preview

TEST DESIGN

BEST TEST DESIGN Benjam in D. W right M ark H. Stone University o f Chicago

M E S A PRESS C h ic a g o 1979

M E S A PRE S S 5 8 3 5 K im b a r k A v e . C h ica g o, I L 60637

C o p y r ig h t © 1 9 7 9 b y B e n ja m in D . W r ig h t a n d M a r k H . S to n e A l l rig h ts res er ved P r in te d in th e U n ite d S ta tes o f A m e r ic a L ib r a r y o f C on g res s C a ta lo g C a r d N u m b e r 7 9 - 8 8 4 8 9 C o v e r D es ig n : A n d r e w W . W r ig h t G r a p h ic s a n d T y p e s e ttin g : B e tty S to n e c ip h e r

ACKNOW LEDGEMENTS

T h is b o o k o w e s m o r e t o G e o r g R a s ch th a n w o r d s ca n c o n v e y . T h e m a n y m o n th s o f p a in s ta k in g a n d in s p ir e d tu to r in g w ith w h ic h h e in tr o d u c e d m e to th e o p p o r tu n itie s a n d n eces s ities o f m e a s u r e m e n t d u r in g 1 9 6 0 a r e th e fo u n d a tio n o n w h ic h M a r k S to n e a n d I h a ve b u ilt. I a m a ls o in d e b te d t o J a n e L o e v in g e r fo r h e r c o m m o n s en s e d u r in g th e 1 9 6 5 M id w e s t P s y c h o lo g ic a l A s s o c ia tio n

S y m p o s iu m

o n “ S a m p le - fr e e P r o b a b ility M o d e ls in

P s y c h o lo g ic a l M e a s u r e m e n t,” to D a r r e ll B o c k f o r e n c o u r a g in g m e to ta lk a b o u t “ S a m p lefr e e T e s t C o n s tr u c tio n a n d T e s t- fr e e A b il it y M e a s u r e m e n t” a t th e M a r ch , 1 9 6 7 m e e tin g o f th e P s y c h o m e tr ic S o c ie ty a n d to B e n ja m in B lo o m f o r in s is tin g o n “ S a m p le - fr e e T e s t C a lib r a tio n

and

P ers on

M e a s u r e m e n t”

at

th e

1967

E TS

In v ita tio n a l C o n fe r e n c e

on

T e s tin g P r o b le m s . B e n ja m in D . W r ig h t

D a v id F a r r in itia te d a n A E R A T r a in in g S es s ion o n “ S a m p le - fr e e T e s t C a lib r a tio n a n d P e r s o n M e a s u r e m e n t in E d u c a tio n a l R e s a r c h ” in 1 9 6 9 . S in ce th e n , th a n k s to th e s u p p o r t of

R ic h a r d

J a eg er

and

W illia m

R u s s ell,

th e r e h a ve b ee n s ever a l A E R A

“ p r es es s io n s .”

T h e s e fo r u m s f o r d e b a te o n th e th e o r y a n d p r a c tic e o f R a s ch m e a s u r e m e n t h a ve fo s te r e d m a n y in s tr u c tive c o lla b o r a tio n s . F o r th es e les s on s in h o w to m a k e R a s ch m e a s u r e m e n t u s e fu l w e a r e e s p e c ia lly in d e b te d t o L o u is B a s h a w , R ic h a r d W o o d c o c k , R o b e r t R e n tz a n d F r e d F o r s te r .

T h e lo n g h a r d w o r k o f m a th e m a tic a l a n a lys is , c o m p u te r p r o g r a m m in g a n d e m p ir ic a l v e r ific a tio n w h ic h m a k es R a s ch m e a s u r e m e n t n o t o n ly p r a c tic a l b u t e ve n ea s y to u s e w a s d o n e w ith fr ie n d s in th e M E S A p r o g r a m a t T h e U n iv e r s ity o f C h ic a g o . T h is g r e a t a d ven tu r e b eg a n in 1 9 6 4 w ith B r u ce C h o p p in a n d N a r g is P a n ch a p a k es a n a n d c o n tin u e s v i t h D a vid A n d r ic h , G r a h a m D o u g la s , R o n a ld M e a d , R o b e r t D r a b a , S u s a n B e ll a n d G e o ffr e y M a s ters . W e d e d ic a te o u r b o o k t o th e m . B e n ja m in D . W r ig h t M a r k H . S to n e

The U niv e rs ity o f C hicago M a rch 30, 1979

v

FO RW ARD

0 .1

H O W T O U S E T H IS B O O K

T h is is a h a n d b o o k f o r le a r n in g h o w to d o R a s ch m e a s u r e m e n t. W e g ive s o m e th e o ­ r e tic a l e x p la n a tio n s , b u t w e e m p h a s ize p r a c tic e . C h a p ter s 2 , 4 , 5 a n d 7 u s e a s m a ll p r o b ­ le m

to illu s tr a te th e a p p lic a tio n o f R a s ch m e a s u r e m e n t in c o m p le te d e ta il. T o th o s e o f

you

w h o lea r n b e s t b y d o in g w e r e c o m m e n d g o in g d ir e c tly to C h a p ter s 2 a n d 4 a n d

w o r k in g

s ec tio n s

of

C h a p te r s 5 a n d 7 th a t c o n tin u e th e a n a lys is o f th is p r o b le m a n d th e n g o b a ck to C h a p te r

th r o u g h

1.

B e h in d

th e

m e th o d o lo g ic a l

th e

p r a c tic a l p r o b le m

p r a c tic a l

p roced u res

is s u es th a t m o tiv a te

C h a p te r s 1 , 5 a n d 6 . T o

d e v e lo p e d

of

R a s ch

a n d g o ve r n

th o s e o f y o u

th e r e .

N e x t s tu d y

th e

m e a s u r e m e n t a r e th e ir rea s on s .

Th e

R a s ch m e a s u r e m e n t a r e d e v e lo p e d

in

w h o lik e to b e g in w ith th e o r y , w e r e c o m m e n d

r e a d in g th e s e c h a p ter s b e fo r e w o r k in g th e p r o b le m in C h a p ter s 2 a n d 4 . F in a lly , i f R a s ch m e a s u r e m e n t is e n tir e ly n e w

to y o u , y o u m ig h t w a n t to b eg in w ith S e c tio n 0 .3 o f th is

F o r w a r d w h ic h is a n in tr o d u c tio n to th e to p ic g iven a t th e O c to b e r 2 8 , 1 9 6 7 E T S In v i­ ta tio n a l C o n fe r e n c e o n T e s tin g P r o b le m s (W r ig h t, 1 9 6 8 ).

S e c tio n

0 .2

r e vie w s

th e

m o tiv a tio n

and

h is to r y

o f th e id ea s th a t c u lm in a te d in

R a s c h ’s P ro b a b ilis tic M od e ls fo r S o m e In te llig e n ce and A tta in m e n t Tes ts ( 1 9 6 0 ) . T h e r e fe r e n c e s

c ite d th e r e a n d e ls e w h e r e in th is b o o k a r e fo c u s e d o n w o r k th a t ( 1 ) b ea rs

d ir e c tly o n th e d is cu s s ion , ( 2 ) is in E n g lis h a n d ( 3 ) is e ith e r r e a d ily a va ila b le in a u n i­ v e r s ity lib r a r y o r ca n b e s u p p lie d b y us.

0 .2

M O T IV A T IO N A N D H IS T O R Y

F if t y ye a r s a g o T h o r n d ik e c o m p la in e d th a t c o n te m p o r a r y in te llig e n c e tes ts fa ile d to s p e c ify “ h o w fa r it is p r o p e r to a d d , s u b tr a ct, m u ltip ly , d iv id e , a n d c o m p u te r a tio s w ith th e m ea s u res o b ta in e d . ” ( T h o r n d ik e , 1 9 2 6 , 1 ). A g o o d m e a s u r e m e n t o f a b ility w o u ld b e o n e “ o n w h ic h z e r o w ill r e p r e s e n t ju s t n o t a n y o f th e a b ility in q u e s tio n , a n d 1, 2 , 3 , 4 , a n d s o o n w ill r e p r e s e n t a m o u n ts in cr ea s in g b y a c o n s ta n t d iffe r e n c e . ” ( T h o r n d ik e , 1 9 2 6 , 4 ) . T h o r n d ik e h a d th e c o u r a g e to c o m p la in

b eca u s e h e b e lie ve d h e h a d w o r k e d o u t a

s o lu tio n t o th e p r o b le m f o r h is o w n in te llig e n c e tes t. S o d id T h u r s to n e ( 1 9 2 5 ).

T h u r s to n e ’s m e th o d w a s to tr a n s fo r m

th e p r o p o r tio n in a n a g e g r o u p p a s s in g a n y

ite m in to a u n it n o r m a l d e v ia te a n d to u s e th es e va lu es as th e b a s is fo r s ca lin g . C o m m o n s ca le va lu es f o r d iffe r e n t a g e g r o u p s w e r e o b ta in e d b y a s s u m in g a lin e a r r e la tio n s h ip b e ­ tw e e n th e d iffe r e n t s ca le va lu es o f ite m s s h a red b y tw o o r m o r e te s t fo r m s u s in g th e d iffe r e n t g r o u p m ea n s a n d s ta n d a r d d e v ia tio n s as th e p a r a m eter s fo r a tr a n s fo r m a tio n o n to a c o m m o n s ca le. T h u r s to n e r e d id a p ie c e o f T h o r n d ik e ’s w o r k to s h o w th a t h is m e th o d w a s b e tte r (T h u r s to n e , 1 9 2 7 ). H is “ a b s o lu te s ca le ” ( 1 9 2 5 , 1 9 2 7 ) y ie ld s a m o r e o r les s in te r va l s ca le. B u t o n e w h ic h is q u ite d e p e n d e n t o n th e a b ility d is tr ib u tio n o f th e s a m p le

u s ed .

In

a d d itio n

to

ite m

h o m o g e n ie ty ,

th e T h u r s to n e m e th o d

r eq u ir es

th e

a s s u m p tio n th a t a b ility is n o r m a lly d is tr ib u te d w ith in a g e g r o u p s a n d th a t th er e e x is t

FORWARD

v iii

r e leva n t fix e d p o p u la tio n p a ra m eter s fo r th es e d is tr ib u tio n s . S h ou ld th e s p e c ific a tio n o f p o p u la tio n b e in a p p r o p r ia te s o w ill th e e s tim a ted s ca le va lu es . S h ou ld th e s a m p lin g o f in te n d e d p o p u la tio n s b e in a d eq u a te in a n y w a y s o w ill th e e s tim a ted s ca le va lu es . T h e y c a n n o t b e in va r ia n t to s a m p lin g. S a m p les d iffe r in g in th e ir a b ility d is tr ib u tio n s w ill p r o ­ d u ce s ca le va lu es d iffe r e n t in m a g n itu d e a n d d is p er s io n . T h u r s to n e u s ed th e 1 9 2 5 ver s io n o f h is m e th o d fo r th e r es t o f h is life , b u t th e m a jo r ­ ity o f te s t ca lib r a tor s h a ve r e lie d o n th e s im p ler tech n iq u es o f p e r c e n tile ra n k s a n d s ta n d ­ a rd s cores . T h e in a d eq u a cies o f th es e m e th o d s w e r e c la r ifie d b y L o e v in g e r ’s 1 9 4 7 a n a lys is o f th e c o n s tr u c tio n a n d e va lu a tio n o f tes ts o f a b ility (L o e v in g e r , 1 9 4 7 ). L o e v in g e r s h o w ed

th a t tes t h o m o g e n ie ty

a n d s ca le m o n o to n ic ity

w e r e es s en tia l

c r ite r ia fo r a d eq u a te m ea s u r em en t. In a d d itio n , “ A n a cc ep ta b le m e th o d o f s ca lin g m u s t r es u lt in a d e r ive d s ca le w h ic h is in d e p e n d e n t o f th e o r ig in a l s ca le a n d o f th e o r ig in a l g r o u p te s te d .” (L o e v in g e r , 1 9 4 7 , 4 6 ). S u m m in g u p th e te s t c a lib r a tio n s itu a tio n in 1 9 4 7 , L o e v in g e r s a ys , “ N o s ys tem o f s ca lin g has b een p r o ve d a d eq u a te b y th e c r ite r ia p r o p o s e d h er e,

th o u g h

th es e c r ite r ia c o r r es p o n d

to

th e cla im s m a d e b y

T h u r s to n e ’s s ys tem .”

(L o e v in g e r , 1 9 4 7 , 4 3 ). A s fo r r elia b ilitie s b a s ed o n co r r ela tio n s , “ U n til a n a d q u a te s ys tem o f s ca lin g is fo u n d , th e c o r r e la tio n b e tw e e n tes ts o f a b ilities , even b e tw e e n tw o tes ts o f th e s a m e a b ility , w ill b e a ccid en ta l to an u n k n o w n d e g r e e .” (L o e v in g e r , 1 9 4 7 , 4 6 ).

In 1 9 5 0 G u llik s en c o n c lu d e d h is The ory o f M e n ta l Tests w ith th e o b s e r va tio n th a t

R e la tiv e ly little e x p e r im e n ta l o r th e o r e tic a l w o r k has b een d o n e o n th e e ffe c t o f g r o u p ch a n ges o n ite m p a ra m eter s . I f w e a s s u m e th a t a g iven ite m r eq u ir es a cer ta in a b ility , th e p r o p o r tio n o f a g r o u p a n s w er in g th a t ite m c o r r e c tly w ill in crea s e a n d d ecrea s e as th e a b ility le ve l o f th e g r o u p c h a n g e s .. . . A s y e t th ere* h as b een n o s ys tem a tic th e o r e tic a l tr e a tm e n t o f m ea s u res o f ite m d iffic u lty d ir e c te d

p a r ticu la r ly

to w a r d

d e te r m in in g th e n a tu re

o f th e ir va r ia tio n w ith

r e s p e ct to ch a n ges in g r o u p a b ility . N e ith e r has th e e x p e r im e n ta l w o r k o n ite m a n a lys is

b een

d ir e c te d

p a ra m eter s w ith

to w a r d

d e te r m in in g

s ys tem a tic ch a n ges in

th e

r e la tive in va r ia n ce

o f ite m

th e a b ility le ve l o f th e g r o u p tes ted

(G u llik s e n , 1 9 5 0 , 3 9 2 - 3 9 3 ).

A t th e 1 9 5 3 E T S In vita tio n a l C o n fe r e n c e o n T e s tin g P r o b lem s , T u c k e r s u g ges ted th a t, “ A n id ea l tes t m a y b e c o n c e ive d as o n e f o r w h ic h th e in fo r m a tio n tr a n s m itte d b y ea ch o f th e p os s ib le s ca led s cores rep r es en ts a lo c a tio n o n s o m e u n ita r y c o n tin u u m s o th a t u n i­ fo r m

d iffe r e n c e s b e tw e e n s ca led s cores c o r r e s p o n d to u n ifo r m d iffe r e n c e s b e tw e e n tes t

p e r fo r m a n c e s fo r a ll s cor e levels ” (T u c k e r , 1 9 5 3 , 2 7 ). H e a ls o p r o p o s e d th e com p a r is o n o f g r ou p s d iffe r in g in a b ility as a s tr o n g m e th o d fo r eva lu a tin g te s t h o m o g e n e ity (T u c k e r , 1 9 5 3 , 2 5 ). B u t th e o th e r p a r ticip a n ts in th e c o n fe r e n c e b e little d h is p r o p o s a ls as im p r a c­ tic a l a n d id ea lis tic .

In 1 9 6 0 A n g o ff w r o te in h is e n c y c lo p e d ia a r tic le o n m ea s u r em en t a n d s ca lin g th a t

M o s t o f th e te s t s ca les n o w in u se d er ive th e ir s ys tem s o f u n its fr o m d a ta ta k en fr o m

a ctu a l tes t a d m in is tr a tio n s , a n d th u s a re d e p e n d e n t o n th e p e r fo r m a n c e

o f th e g ro u p s tes ted . W h en s o c o n s tr u cted , th e s ca le h a s m e a n in g o n ly s o lo n g as th e g r o u p is w e ll d e fin e d a n d has m ea n in g , a n d b ea rs a res em b la n ce in s om e fa s h io n to th e g ro u p s o r in d ivid u a ls w h o la te r ta k e th e tes t fo r th e p a r ticu la r p u rp os es o f s ele ctio n , g u id a n ce, o r g r o u p eva lu a tio n . H o w e ve r , i f it is fo u n d

FOR W AR D

ix

th a t th e s a m p lin g f o r th e d e v e lo p m e n t o f a te s t s ca le h a s n o t b een a d e q u a te , o r th a t th e g r o u p o n w h ic h th e te s t h a s b ee n s ca led h a s o u tliv e d its u s efu ln es s , p o s s ib ly b eca u s e o f ch a n g es in th e d e fin e d p o p u la tio n o r b eca u s e o f ch a n ges in e d u c a tio n a l em p h a s es , th e n th e s ca le its e lf c o m e s in to q u e s tio n . T h is is a s er io u s m a tte r . A te s t w h ic h is to h a ve c o n tin u e d u s efu ln es s m u s t h a ve a s ca le w h ic h

d oes

n o t ch a n g e w ith

fa m ilia r ity w ith

th e s ys te m

th e tim es , w h ic h w ill p e r m it a c q u a in ta n ce a n d

o f u n its , a n d w h ic h

o f d a ta f o r h is to r ic a l c o m p a r is o n s

w ill p e r m it a n a c c u m u la tio n

( A n g o f f, 1 9 6 0 , 8 1 5 ).

A n d y e t th e fa u lte d m e th o d s r e fe r r e d to a n d c r itic iz e d b y L o e v in g e r , G u llik s en a n d A n g o f f a r e s till w id e ly u s ed in te s t c o n s tr u c tio n a n d m e a s u r em en t. T h is is in s p ite o f th e fa c t th a t c o n s id e r a b le e v id e n c e h a s a c c u m u la te d in th e p a s t tw e n ty - fiv e yea r s th a t m u ch b e tte r m e th o d s a r e p o s s ib le a n d p r a c tic a l.

T h e s e b e tte r m e th o d s h a ve th e ir r o o ts in th e 1 9 th c e n tu r y p s y c h o p h y s ic a l m o d e ls o f W e b e r a n d F e c h n e r . T h e y a re b a s ed o n s im p le m o d e ls f o r w h a t it s eem s r ea s o n a b le to s u p p o s e h a p p en s w h e n a p e r s o n r e s p o n d s to a te s t ite m . T w o s ta tis tica l d is tr ib u tio n s h a ve b e e n u s ed to m o d e l th e p r o b a b ilis tic a s p e c t o f th is e ve n t. T h e n o r m a l d is tr ib u tio n a p p ea rs as a b a s is f o r m e n ta l m e a s u r e m e n t in T h u r s to n e ’s L a w o f C o m p a r a tiv e J u d g e m e n t in th e 1 9 2 0 ’s. T h e

u se o f

th e

n orm a l

o g iv e as a n ite m

r es p o n s e m o d e l s eem s to h a ve b een

in itia te d b y L a w le y a n d F in n e y in th e 1 9 4 0 ’s. L o r d m a d e th e n o r m a l o g iv e th e c o r n e r ­ s to n e o f h is a p p r o a c h to ite m a n a lys is u n til a b o u t 1 9 6 7 , w h e n u n d e r B im b a u m ’s in flu ­ e n c e , h e s w itc h e d t o a lo g is tic r e s p o n s e m o d e l ( L o r d , 1 9 6 8 ).

T h e lo g is tic d is tr ib u tio n w a s u s ed b y b io m e tr ic ia n s to s tu d y g r o w th a n d m o r ta lity r a tes in th e 1 9 2 0 ’s a n d B e r k s o n h a s c h a m p io n e d its p r a c tic a l a d va n ta g es o v e r th e n o r m a l d is tr ib u tio n th r o u g h

e ve r

s in ce. T h e s e b io m e tr ic a p p lic a tio n s w e r e fin a lly p ic k e d u p , p r o b a b ly

th e w o r k o f B r a d le y a n d T e r r y in th e 1 9 5 0 ’s, a n d fo r m u la te d in to a lo g is tic

r e s p o n s e m o d e l f o r ite m a n a lys is b y B im b a u m ( 1 9 6 8 ) a n d B a k e r ( 1 9 6 1 ) . B a k e r d e v e lo p e d c o m p u te r p r o g r a m s f o r a p p ly in g lo g it a n d p r o b it ite m a n a lys is a n d s tu d ie d th e ir p e r fo r ­ m a n c e w ith e m p ir ic a l a n d s im u la te d d a ta (B a k e r , 1 9 5 9 , 1 9 6 3 ) .

In a ll o f th e s e a p p r o a c h e s to ite m

a n a lys is , h o w e v e r , a t lea s t tw o p a r a m eter s a re

s o u g h t f o r ea ch ite m . A tte m p ts a r e m a d e to e s tim a te n o t o n ly a n ite m d iffic u lty , th e r e s p o n s e o g iv e ’s h o r iz o n ta l in te r c e p t a t p r o b a b ility o n e - h a lf, b u t a ls o a n ite m d is cr im in a ­ tio n , th e o g iv e ’s s lo p e a t th is in te r c e p t. U n fo r tu n a te ly th is s e e m in g ly r ea s o n a b le e la b o r a ­ tio n

o f th e p r o b le m

in tr o d u c e s a n in s u r m o u n ta b le d iffic u lty in to a p p ly in g th es e id ea s

in p r a c tic e . T h e r e h a s b e e n a r u n n in g d e b a te f o r a t lea s t fifte e n yea r s as to w h e th e r o r n o t th e r e is a n y u s e fu l w a y b y w h ic h s o m e k in d o f e s tim a tes o f ite m p a r a m eter s lik e ite m d is c r im in a tio n a n d ite m “ g u es s in g ” ca n b e o b ta in e d .

T h e in e v ita b le r e s o lu tio n o f th is d e b a te h a s b ee n im p lic it e v e r s in ce F is h e r ’s in ve n ­ tio n o f s u ffic ie n t e s tim a tio n in th e 1 9 2 0 ’s a n d N e y m a n n a n d S c o tt’s w o r k o n th e c o n ­ s is te n c y o f c o n d itio n a l e s tim a to r s in th e 1 9 4 0 ’s. R a s ch ( 1 9 6 8 ) , A n d e r s e n (1 9 7 3 , 1 9 7 7 ) a n d B a m d o r ff- N ie ls e n ( 1 9 7 8 ) ea ch p r o v e d e c is iv e ly th a t o n ly ite m d iffic u lty ca n a c tu a lly be

e s tim a te d

c o n s is te n tly

and

s u ffic ie n tly

fr o m

th e r ig h t/ w r o n g ite m

r es p on s e d a ta

a va ila b le f o r ite m a n a lys is . T h e s e p r o o fs m a k e it c le a r th a t th e d ic h o to m o u s res p on s e d a ta a va ila b le f o r ite m

a n a lys is ca n o n ly s u p p o r t th e e s tim a tio n o f ite m

d iffic u lty a n d th a t

a tte m p ts to e s tim a te a n y o th e r in d ivid u a l ite m p a r a m eter s a r e n eces s a r ily d o o m e d .

T h e m a th e m a tic s o f th es e p r o o fs n e e d n o t b e m a s ter e d to b e c o m e c o n vin c e d o f th e ir

X

FORWARD

p r a ctica l im p lica tio n s . A n y o n e w h o a c tu a lly ex a m in es th e in n er w o r k in g s o f th e va riou s c o m p u te r p r o g r a m s a d ver tis ed to es tim a te ite m d is cr im in a tion s a n d tr ies to a p p ly th e m to a ctu a l d a ta , w ill fin d th a t th e r es u ltin g es tim a tes a re h ig h ly s a m p le d ep en d e n t. I f a ttem p ts a re m a d e in th es e c o m p u te r p r o g r a m s to ite r a te to a n a p p a r e n t c o n ve r g e n c e , th is “ c o n ­ ve r g e n c e ” ca n o n ly b e “ r e a c h e d ” b y in te r fe r in g a r b itr a r ily w ith th e in e vita b le te n d e n c y o f a t lea s t o n e o f th e ite m d is c r im in a tio n es tim a tes to d ive r g e to in fin ity . In m o s t p r o ­ gra m s

th is

in s u r m o u n ta b le p r o b le m

is s id es tep p ed e ith e r b y n o t ite r a tin g a t a ll o r b y

p r e ve n tin g a n y p a r ticu la r d is c r im in a tio n e s tim a te fr o m e x c e e d in g s o m e e n tir e ly a r b itr a r y c e ilin g s u ch as 2 .0 . A s fa r as w e ca n te ll, it w a s th e D a n is h m a th e m a tic ia n G e o r g R a s ch w h o fir s t u n d er ­ s to o d th e p o s s ib ilitie s f o r tr u ly o b je c tiv e m e a s u r e m e n t w h ic h r es id e in th e s im p le lo g is tic re s p o n s e m o d e l. A p p a r e n tly it w a s a ls o R a s ch w h o fir s t a p p lie d th e lo g is tic fu n c tio n to th e a ctu a l a n a lys is o f m e n ta l te s t d a ta fo r th e p r a c tic a l p u r p o s e o f c o n s tr u c tin g tes ts . R a s ch b eg a n h is w o r k o n p s y c h o lo g ic a l m e a s u r e m e n t in 1 9 4 5 w h e n h e s ta n d a r d ize d a g r o u p in te llig e n c e te s t f o r th e D a n is h D e p a r tm e n t o f D e fe n s e . I t w a s in c a r r y in g o u t th a t ite m a n a lys is th a t h e fir s t “ b ec a m e a w a r e o f th e p r o b le m o f d e fin in g th e d iffic u lty o f a n ite m

in d e p e n d e n tly o f th e p o p u la tio n a n d th e a b ility o f a n in d ivid u a l in d e p e n d e n tly o f

w h ic h ite m s h e h a s a c tu a lly s o lv e d . ” (R a s c h , 1 9 6 0 , v iii) . B y 1 9 5 2 h e h a d la id d o w n th e b a s ic fo u n d a tio n s f o r a n e w p s y c h o m e tr ic s a n d w o r k e d o u t tw o p r o b a b ility m o d e ls fo r th e a n a lys is o f o r a l r e a d in g tes ts . In

1 9 5 3 h e r e a n a ly ze d th e in te llig e n c e te s t d a ta a n d

d e v e lo p e d th e es s en tia ls o f a lo g is tic p r o b a b ility m o d e l f o r ite m a n a lys is .

R a s ch fir s t p u b lis h e d h is c o n c e r n a b o u t th e p r o b le m o f s a m p le d e p e n d e n t e s tim a tes in h is 1 9 5 3 a r tic le o n s im u lta n eo u s fa c to r a n a lys is in s ever a l p o p u la tio n s B u t h is w o r k o n ite m

a n a lys is w a s u n k n o w n

(R a s c h , 1 9 5 3 ).

in th is c o u n tr y u n til th e s p r in g o f 1 9 6 0

w h e n h e v is ite d C h ic a g o f o r th r e e m o n th s , g a ve a p a p e r a t th e B e r k e le y S y m p o s iu m o n M a th e m a tic a l

S ta tis tic s

(R a s c h ,

1 9 6 1 ),

and

p u b lis h e d P ro b a b ilis tic M o d e ls

fo r S o m e

In te llig e n ce and A tta in m e n t Tes ts (R a s c h , 1 9 6 0 ). In

h e r 1 9 6 5 r e v ie w

o f p e r s o n a n d p o p u la tio n as p s y c h o m e tr ic c o n c e p ts L o e v in g e r

w r o te ,

R a s ch ( 1 9 6 0 ) h a s d e v is e d a tr u ly n e w a p p r o a c h t o p s y c h o m e tr ic p r o b le m s . . . H e m a k e s u s e o f n o n e o f th e cla s s ica l p s y c h o m e tr ic s , b u t r a th e r a p p lie s a lg e b r a a n e w t o a p r o b a b ilis tic m o d e l. T h e p r o b a b ility th a t a p e r s o n w ill a n s w e r a n ite m c o r r e c tly is a s s u m ed t o b e th e p r o d u c t o f a n a b ility p a r a m e te r p e r ta in in g o n ly

to

th e

B eyon d

p ers on

and

a d iffic u l t y p a r a m e te r p e r ta in in g o n ly t o th e ite m .

s p e c ify in g o n e p e r s o n

as th e s ta n d a r d o f a b ility o r o n e ite m

as th e

s ta n d a r d o f d iffic u l ty , th e a b ility a s s ig n ed t o a n in d iv id u a l is in d e p e n d e n t o f th a t o f o th e r m e m b e r s o f th e g r o u p a n d o f th e p a r tic u la r ite m s w ith w h ic h h e is te s te d ; s im ila r ly f o r th e ite m d iffic u l t y . . . In d e e d , th e s e t w o p r o p e r tie s w e r e o n c e s u g g e s te d as c r ite r ia f o r a b s o lu te s c a lin g ( L o e v in g e r , 1 9 4 7 ) ; a t th a t tim e p rop os ed

s ch e m es

fo r

a b s o lu te

s c a lin g h a d

n o t b een

s h ow n

to

s a tis fy

th e

c r ite r ia , n o r d o e s G u ttm a n s c a lin g d o s o . T h u s , R a s c h m u s t b e c r e d ite d w ith a n o u ts ta n d in g c o n tr ib u tio n

to

o n e o f th e t w o

c e n tr a l p s y c h o m e tr ic p r o b le m s ,

th e a c h ie v e m e n t o f n o n a r b itr a r y m ea s u r es . R a s c h is c o n c e r n e d w ith a d iffe r e n t and

m ore

r ig o r o u s

G le s e r .

W h en

p ers on s

and

h is

k in d

m od el

o f th e

o f g e n e r a liz a tio n fits ,

p a r tic u la r

th e

r e s u lts

th a n a re

ite m s w ith in

C ro n b a ch ,

R a ja r a tn a m ,

in d e p e n d e n t o f

and

th e s a m p le o f

s o m e b r o a d lim its . W ith in th e s e

lim its , g e n e r a lity is , o n e m ig h t s a y, c o m p le te ( L o e v in g e r , 1 9 6 5 , 1 5 1 ) .

FOR W AR D

0 .3

xi

A N IN T R O D U C T IO N T O T H E M E A S U R E M E N T P R O B L E M

M y to p ic is a p r o b le m in m e a s u r e m e n t. I t is a n o ld p r o b le m in e d u c a tio n a l tes tin g . A l fr e d

B in e t w o r r ie d a b o u t it 6 0 ye a r s a g o . L o u is T h u r s to n e w o r r ie d a b o u t it 4 0 yea r s

a g o . T h e p r o b le m

is s till u n s o lve d . T o

c o n s id e r it c a r e fu lly , I th in k y o u

s o m e it m a y s eem a s m a ll p o in t. B u t w h e n y o u

w ill fin d th a t th is s m a ll p o in t is a m a tte r o f life a n d

d e a th to th e s cie n c e o f m e n ta l m e a s u r e m e n t. T h e tr u th is th a t th e s o - c a lle d m ea s u r em en ts w e n o w m a k e in e d u c a tio n a l te s tin g a r e n o d a m n g o o d !

E v e r s in ce I w a s o ld e n o u g h t o a r g u e w ith m y p a ls o v e r w h o h a d th e b e s t IQ ( I s a y “ b e s t” b eca u s e s o m e th o u g h t 1 0 0 w a s p e r fe c t a n d 6 0 w a s p a s s in g ), I h a ve b e e n p u zzle d b y m e n ta l m e a s u r e m e n t. W e w e r e m ix e d u p a b o u t th e s ca le. IQ u n its w e r e u n lik e a n y o f th o s e m ea s u r es o f h e ig h t, w e ig h t, a n d w e a lth w ith w h ic h w e w e r e le a r n in g to b u ild a s cie n c e o f life . E v e n th a t n o b le a c h ie v e m e n t, 1 0 0 p e r c e n t, w a s a m b ig u o u s . O n e h u n d r ed m ig h t s ig n ify th e w e lc o m e n e w s th a t w e w e r e s m a rt. O r it m ig h t m e a n th e te s t w a s ea s y. S o m e tim e s w e p r a y e d f o r e a s ier tes ts t o m a k e u s s m a r ter .

L a te r I le a r n e d o n e w a y a te s t s c o r e c o u ld m o r e o r les s b e u s ed . I f I w e r e w illin g to a c c e p t as a w h o le th e s e t o f ite m s m a k in g u p a s ta n d a r d ize d te s t, I c o u ld g e t a r e la tive m ea s u r e o f a b ility . I f m y p e r fo r m a n c e p u t m e a t th e e ig h tie th p e r c e n tile a m o n g c o lle g e m e n , I w o u ld k n o w w h e r e I s to o d . O r w o u ld I? T h e s a m e s c o r e w o u ld a ls o p u t m e a t th e e ig h ty - fifth

p e r c e n tile

s c h o o l s en io r s , a n d

a m on g

a b o ve

th e

c o lle g e

w o m e n , a t th e n in e tie th

n in e ty - n in th

p e r c e n tile a m o n g h ig h

p e r c e n tile a m o n g h ig h s c h o o l ju n io r s . M y

a b ility d e p e n d e d n o t o n ly o n w h ic h ite m s I to o k b u t o n w h o I w a s a n d th e c o m p a n y I k e p t!

T h e tr u th is th a t a s c ie n tific s tu d y o f ch a n g es in a b il it y —o f m e n ta l d e v e lo p m e n t— is fa r b e y o n d o u r fe e b le c a p a c itie s to m a k e m ea s u r em en ts . H o w

ca n w e p o s s ib ly o b ta in

q u a n tita tiv e a n s w er s to q u e s tio n s lik e : H o w m u ch d o e s r e a d in g c o m p r e h e n s io n in cr ea s e in th e fir s t th r e e y e a r s o f s c h o o l? W h a t p r o p o r tio n o f a b ility is n a tive a n d w h a t le a r n e d ? W h a t p r o p o r tio n o f m a tu r e a b ility is a c h ie v e d b y ea c h y e a r o f c h ild h o o d ?

I h o p e I a m r e m in d in g y o u o f s o m e p r o b le m s w h ic h a ffl ic t p r e s e n t p r a c tic e in m e n ta l m e a s u r e m e n t. T h e s ca les o n w h ic h a b ility is m ea s u r ed a r e u n c o m fo r ta b ly s lip p e r y. T h e y h a ve n o r e g u la r u n it. T h e ir m e a n in g a n d e s tim a te d q u a lity d e p e n d u p o n th e s p e c ific s et o f ite m s a c tu a lly s ta n d a r d ize d a n d th e p a r tic u la r a b ility d is tr ib u tio n o f th e c h ild r e n w h o h a p p e n e d t o a p p e a r in th e s ta n d a r d izin g s a m p le.

I f a ll o f a s p e c ifie d s et o f ite m s h a ve b e e n tr ie d b y a c h ild y o u w is h t o m ea s u r e, th e n you

ca n o b ta in h is p e r c e n tile p o s itio n a m o n g w h a te v e r g r o u p s o f c h ild r e n w e r e u s ed to

s ta n d a r d ize th e te s t. B u t h o w d o y o u in te r p r e t th is m ea s u r e b e y o n d th e c o n fin e s o f th a t s et o f ite m s a n d th o s e g r o u p s o f c h ild r e n ? C h a n g e th e c h ild r e n a n d y o u h a ve a n e w ya r d s tic k . C h a n g e th e ite m s a n d y o u h a ve a n e w y a r d s tic k a g a in . E a c h c o lle c tio n o f ite m s m ea s u res a n a b ility o f its o w n . E a c h m ea s u r e d e p e n d s f o r its m e a n in g o n its o w n fa m ily o f tes t ta k er s . H o w c a n w e m a k e o b je c tiv e m e n ta l m e a s u r em en ts a n d b u ild a s cien ce o f m e n ta l d e v e lo p m e n t w h e n w e w o r k w ith r u b b e r ya r d s tic k s ?

T h e g r o w th o f s cie n ce d e p e n d s o n th e d e v e lo p m e n t o f o b je c tiv e m e th o d s fo r tra n s ­ fo r m in g o b s e r va tio n in to m e a s u r e m e n t. T h e p h y s ic a l s cien ces a r e a g o o d e x a m p le . T h e ir b a s is is th e d e v e lo p m e n t o f m e th o d s f o r m ea s u r in g w h ic h a r e s p e c ific to th e m ea s u r em en t in te n d e d a n d in d e p e n d e n t o f v a r ia tio n in th e o th e r c h a r a cter is tics o f th e o b je c ts m ea s u red

x ii

FORWARD

o r th e m ea s u r in g in s tru m en ts u s ed . W h en w e w a n t a p h ys ica l m ea s u rem en t, w e s eld om w orry

a b o u t th e in d ivid u a l id e n tity

o f th e m ea s u r in g in s tru m en t. W e n eve r con cer n

ou rs elves w ith w h a t o b je c ts o th e r th a n th e o n e w e w a n t to m ea s u re m ig h t s o m etim e b e, o r o n c e h a ve b een , m ea s u red w ith th e s a m e in s tru m en t, It is s u ffic ie n t to k n o w th a t th e in s tr u m en t is a m e m b e r in g o o d s ta n d in g o f th e cla ss o f in s tru m en ts a p p r o p r ia te fo r th e jo b .

W h en a m a n s a ys h e is a t th e n in etieth p er c e n tile in m a th a b ility , w e n eed to k n o w in w h a t g r o u p a n d o n w h a t tes t b e fo r e w e ca n m a k e a n y sen s e o f h is s ta tem en t. B u t w h en h e s a ys h e is fiv e fe e t e le ve n in ch es ta ll, d o w e a sk to s ee h is ya r d s tic k ? W e k n o w ya r d ­ s tick s d iffe r in c o lo r , te m p er a tu r e , c o m p o s itio n s , w e ig h t— e ve n s ize. Y e t w e a s su m e th e y s ha re a s ca le o f len g th in a m a n n er s u ffic ie n tly in d e p e n d e n t o f th es e s eco n d a r y ch a ra c­ ter is tic s to g ive a m ea s u r em en t o f fiv e fe e t e leven in ch es o b je c tiv e m ea n in g . W e e x p e c t th a t a n o th e r m a n o f th e s a m e h e ig h t w ill m ea s u re a b o u t th e s a m e fiv e fe e t e le ve n e ve n o n a d iffe r e n t ya r d s tick . I m a y b e a t a d iffe r e n t a b ility p e r c e n tile in e ve r y g r o u p I c o m p a r e m y s e lf w ith . B u t I a m th e s a m e 1 7 5 p ou n d s in a ll o f th em . L e t u s ca ll m ea s u r em en t th a t p os s es s es th is p r o p e r ty “ o b je c tiv e ” . T w o c o n d itio n s a re n eces s a ry to

a ch ieve it. F ir s t, th e c a lib r a tio n

o f m ea s u rin g in s tru m en ts m u s t b e

in d e p e n d e n t o f th o s e o b je c ts th a t h a p p en to b e u s ed fo r c a lib r a tio n . S ec o n d , th e m ea s u re­ m e n t o f o b je c ts m u s t b e in d e p e n d e n t o f th e in s tr u m en t th a t h a p p en s to b e u s ed fo r m ea s u rin g. In p r a c tic e, th es e c o n d itio n s ca n o n ly b e a p p r o x im a te d . B u t th e ir a p p r o x i­ m a tio n is w h a t m a k es m ea s u r em en t o b je c tive . O b je c t- fr e e in s tr u m en t c a lib r a tio n a n d in s tr u m en t- fr ee o b je c t m ea s u r em en t a re th e c o n d itio n s

w h ich

m a k e it p os s ib le to

g en er a lize m ea s u r em en t b e y o n d

th e p a r ticu la r

in s tr u m en t u s ed , to c o m p a r e o b je c ts m ea s u red o n s im ila r b u t n o t id e n tic a l in s tru m en ts , a n d to c o m b in e o r p a r titio n in s tru m en ts to s u it n e w m ea s u r em en t r eq u ir em en ts .

T h e g u id in g s ta r to w a r d w h ic h m o d els fo r m e n ta l m ea s u r em en t s h ou ld a im is th is k in d o f o b je c tiv ity . O th e r w is e h o w ca n w e e ve r a ch ieve a q u a n tita tive gra sp o f m en ta l a b ilitie s o r e ve r c o n s tr u ct a s cien ce o f m en ta l d e ve lo p m e n t? T h e c a lib r a tio n o f tes t- item d iffic u lty m u s t b e in d e p e n d e n t o f th e p a r ticu la r p er s on s u s ed fo r th e ca lib r a tion . T h e m ea s u r em en t o f p er s o n a b ility m u s t b e in d e p e n d e n t o f th e p a r ticu la r te s t ite m s u s ed fo r m ea s u rin g.

W h en w e c o m p a r e o n e ite m w ith a n o th e r in o r d e r to ca lib r a te a tes t, it s h ou ld n o t m a tte r w h o s e res p on s es to th es e ite m s w e u s e fo r th e com p a r is o n . O u r m e th o d fo r tes t c a lib r a tio n s h ou ld g ive u s th e s a m e res u lts rega r d les s o f w h o m w e tr y th e tes t o n . T h is is th e o n ly w a y w e w ill e ve r b e a b le to co n s tr u ct tes ts w h ic h h a ve u n ifo r m m ea n in g r eg a r d ­ less o f w h o m w e c h o o s e to m ea s u re w ith th em . W h en w e e x p o s e p er s on s to a s e le c tio n o f tes t ite m s in o r d e r to m ea s u re th e ir a b ility , it s h ou ld n o t m a tte r w h ich s ele c tio n o f item s w e u se o r w h ic h item s th e y c o m p le te . W e s h ou ld b e a b le to c o m p a r e p ers on s , to a r rive a t s ta tis tica lly e q u iva le n t m ea s u r em en ts o f a b ility , w h a te ve r s e le c tio n o f item s h a p p en s to h a ve b ee n u s ed — e ven w h en th e y h a ve b een m ea s u red w ith e n tir e ly d iffe r e n t tes ts . E x h o r ta tio n s a b o u t o b je c tiv ity a n d sa rca s m a t th e e x p en s e o f p r es en t p r a ctices a re ea s y. B u t ca n a n yth in g b e d o n e a b o u t th e p r o b le m ? Is th e r e a b e tte r w a y? In th e o ld w a y o f d o in g th in gs , w e ca lib r a te a tes t ite m b y o b s e r vin g h o w m a n y p ers on s in a s ta n d a rd

FOR WAR D

x iii

s a m p le s u c ce ed o n th a t ite m . T h e tr a d itio n a l ite m “ d iffic u l ty ” is th e p r o p o r tio n o f c o r ­ r e c t r es p o n s es in s o m e s ta n d a r d izin g s a m p le. Ite m q u a lity is ju d g e d fr o m th e c o r r e la tio n b e tw e e n th es e ite m

r es p on s es a n d te s t s co res . P e r s o n a b ility is a p e r c e n tile s ta n d in g in

th e s a m e “ s ta n d a r d ” s a m p le. O b v io u s ly th is a p p r o a c h lea n s v e r y h e a vily o n a s s u m p tion s c o n c e r n in g th e a p p r o p r ia te n e s s o f th e s ta n d a r d izin g s a m p le o f p er s on s .

A

q u ite d iffe r e n t a p p r o a c h is p o s s ib le , o n e in w h ic h n o a s s u m p tio n s n e e d b e m a d e

a b o u t th e a b ility d is tr ib u tio n o f th e p er s o n s u s ed . T h is n e w a p p r o a c h a s s u m es in s tea d a v e r y s im p le m o d e l f o r w h a t h a p p en s w h e n a n y p e r s o n e n c o u n te r s a n y ite m . T h e m o d e l s a ys s im p ly th a t th e o u tc o m e o f th e e n c o u n te r s h a ll b e ta k e n to b e e n tir e ly g o v e r n e d b y th e d iffe r e n c e b e tw e e n th e a b ility o f th e p e r s o n a n d th e d iffic u lty o f th e ite m . N o th in g m o r e . T h e m o r e a b le th e p e r s o n , th e b e tte r th e ir ch a n ces f o r s u cces s w ith a n y ite m . T h e ea s ier th e ite m , th e m o r e lik e ly a n y p e r s o n is to s o lve it. I t is a s s im p le as th a t.

B u t th is s im p le m o d e l h a s s u r p r is in g co n s eq u e n ce s . W h en m e a s u r e m e n t is g o v e r n e d b y th is m o d e l, it is p o s s ib le to ta k e in to a c c o u n t w h a te ve r a b ilitie s th e p er s o n s in th e ca li­ b r a tio n s a m p le h a p p e n to d e m o n s tr a te a n d to fr e e th e e s tim a tio n o f ite m d iffic u lty fr o m th e p a r ticu la r s o f th es e a b ilitie s . T h e s co r es p er s o n s o b ta in o n th e te s t ca n b e u s ed to r e m o v e th e in flu e n c e o f th e ir a b ilitie s fr o m th e e s tim a tio n o f ite m d iffic u lty . T h e r es u lt is a s a m p le - fr e e ite m c a lib r a tio n .

T h e s a m e th in g ca n h a p p e n w h e n w e m ea s u r e p er s on s . T h e s co r es ite m s r e c e iv e in w h a te v e r s a m p le h a p p en s to p r o v id e th e ir c a lib r a tio n s ca n b e u s ed to r e m o v e th e in flu ­ e n c e o f ite m

d iffic u lty fr o m

th e e s tim a tio n o f p e r s o n a b ility . T h e r es u lt is a te s t- fr e e

p e r s o n m e a s u r e m e n t.1

'A d a p t e d

fr o m

P ro ce e d in g s o f the 1967 In v ita tio n a l C o n fe re n ce o n Te s tin g P rob le m s . C o p y r ig h t ©

1 9 6 8 b y E d u c a tio n a l T e s tin g S e r vic e . A l l r ig h ts r e s e r ve d . R e p r in te d b y p e r m is s io n .

CONTENTS

A C KN O W LE D G E M E N TS

v

FO RW ARD

1

2

3

4

5

v ii

TH E M E A S U R E M E N T M O D E L

1

1 .1

H o w T e s ts a r e U s e d to M ea s u r e

1

1 .2

H o w S cores a re U s ed

4

1 .3

W h a t H a p p e n s W h en a P e r s o n T a k e s a n Ite m

1 .4

T h e R a s ch M o d e l

15

9

1 .5

U s in g th e R a s ch M o d e l f o r C a lib r a tin g a n d M ea s u r in g

17

1 .6

A S im p le U s e fu l E s tim a tio n P r o c e d u r e

21

1 .7

H o w T r a d itio n a l T e s t S ta tis tic s A p p e a r in R a s ch M e a s u r e m e n t

24

IT E M C A L I B R A T I O N B Y H A N D

28

2 .1

In tr o d u c tio n

28

2 .2

Th e K n o x C u b e Tes t

28

2 .3

T h e D a ta f o r Ite m A n a ly s is

29

2 .4

C a lib r a tin g Ite m s a n d M ea s u r in g P er s o n s

30

2 .5

D is cu s s io n

44

IT E M C A L I B R A T I O N B Y C O M P U T E R

46

3 .1

In tr o d u c tio n

46

3 .2

B I C A L O u tp u t f o r a P R O X A n a ly s is o f th e K n o x C u b e T e s t D a ta

46

3 .3

C o m p a r in g P R O X b y H a n d w ith P R O X b y C o m p u te r

55

3 .4

A n a ly z in g K C T w ith th e U C O N P r o c e d u r e

56

3 .5

C o m p a r in g U C O N to P R O X w ith th e K C T D a ta

60

3 .6

A C o m p u tin g A lg o r ith m f o r P R O X

61

3 .7

T h e U n c o n d itio n a l P r o c e d u r e U C O N

62

T H E A N A L Y S IS O F F IT

66

4 .1

In tr o d u c tio n

66

4 .2

T h e K C T R e s p o n s e M a tr ix

66

4 .3

T h e A n a ly s is o f F it b y H a n d

69

4 .4

M is fittin g P e r s o n R e c o r d s

76

4 .5

M is fittin g Ite m R e c o r d s

77

4 .6

B r ie f S u m m a r y o f th e A n a ly s is o f F it

79

4 .7

C o m p u te r A n a lys is o f F it

80

C O N S T R U C T IN G A V A R IA B L E

83

5 .1

G e n e r a lizin g th e D e fin itio n o f a V a r ia b le

83

5 .2

D e fin in g th e K C T V a r ia b le

83

5 .3

In te n s ify in g a n d E x te n d in g th e K C T V a r ia b le

87

xv

xvi

6

7

8

CONTENTS

5 .4

C o n tr o l L in es fo r Id e n tity P lo ts

94

5 .5

C o n n e c tin g T w o Tes ts

96

5 .6

B u ild in g Ite m B a n k s

5 .7

B a n k in g th e K C T B D a ta

106

5 .8

C o m m o n P er s on E q u a tin g w ith th e K C T B

109

5 .9

C o m m o n Ite m E q u a tin g w ith th e K C T B

112

5 .1 0

C r ite r io n R e fe r e n c in g th e K C T V a r ia b le

118

5 .1 1

Ite m C a lib r a tio n Q u a lity C o n tr o l

1 21

5 .1 2

N o r m R e fe r e n c in g th e K C T V a r ia b le

126

98

D E S IG N IN G T E S T S

129

6 .1

In tr o d u c tio n

129

6 .2

T h e M ea s u r em en t T a r g e t

129

6 .3

T h e M ea s u rin g T e s t

131

6 .4

T h e S h a p e o f a B es t T e s t

133

6 .5

T h e P r ecis io n o f a B es t T e s t

134

6 .6

T h e E r r o r C o e ffic ie n t

135

6 .7

T h e D es ig n o f a B es t T e s t

137

6 .8

T h e C o m p le te R u les fo r B es t T e s t D es ig n

139

M A K IN G M E A S U R E S

141

7 .1

U s in g a V a r ia b le to M a k e M ea s u res

141

7 .2

C o n ve r tin g S cor es to M ea s u res b y U C O N , P R O X a n d U F O R M

142

7 .3

M ea s u res fr o m B es t Tes ts b y U F O R M

145

7 .4

In d ivid u a lize d T e s tin g

151

7 .5

S ta tu s T a ilo r in g

7 .6

P e r fo r m a n c e T a ilo r in g

,

7 .7

S e lf- T a ilo r in g

* 161

153 156

7 .8

P er s on F it a n d Q u a lity C o n tr o l

165

7 .9

D ia g n o s in g M is fit

170

7 .1 0

C o r r e c tin g a M ea s u re

181

C H O O S IN G A S C A L E

191

8 .1

In tr o d u c tio n

191

8 .2

F o r m u la s fo r M a k in g N e w S ca les

192

8 .3

T h e L e a s t M ea s u ra b le D iffe r e n c e

192

8 .4

D e fin in g th e S p a cin g F a c to r

195

8 .5

N o r m a tiv e S ca lin g U n its : N IT S

198

8 .6

S u b s ta n tive S ca lin g U n its : S IT S

199

8 .7

R es p o n s e P r o b a b ility S ca lin g U n its : C H IP S

201

8 .8

R e p o r tin g F o r m s

205

A P P E N D IC E S

211

T a b le A

2 1 2 & 213

T a b le B

214 & 215

T a b le C

216

RE FE RE N C E S

217

IN D E X

220

1

1 .1

THE M EA SU R EM EN T MODEL

HOW TESTS ARE USED TO M EASURE

T h is b o o k is a b o u t h o w to m a k e a n d u s e m e n ta l tes ts . In o r d e r to d o th is s u cces s ­ fu lly w e m u s t h a ve a m e th o d f o r tu r n in g o b s e r va tio n s o f te s t p e r fo r m a n c e in to m ea s u res o f m e n ta l a b ility . T h e id e a o f a m ea s u r e r e q u ir es a n id e a o f a va r ia b le o n w h ic h th e m ea s u r e is lo c a te d . I f th e va r ia b le is v is u a lize d as a lin e , th e n th e m ea s u r e ca n b e p ic tu r e d as a p o in t o n th a t lin e . T h is r e la tio n s h ip b e tw e e n a m ea s u r e a n d its va r ia b le is p ic tu r e d in F ig u r e 1 .1 .1 .

1

F I G U R E 1 .1 .1

I

A M E A S U R E O N A V A R IA B L E

T h e M e a s u re

—

.................. |

T h e V a r ia b le

W h en w e te s t a p e r s o n , o u r p u r p o s e is to e s tim a te th e ir lo c a tio n o n th e lin e im p lie d b y th e tes t. B e fo r e w e ca n d o th is w e m u s t c o n s tr u c t a te s t th a t d e fin e s a lin e. W e m u s t a ls o h a ve a w a y to tu r n th e p e r s o n ’s te s t p e r fo r m a n c e in to a lo c a tio n o n th a t lin e . T h is b o o k s h o w s h o w t o u s e te s t ite m s to d e fin e lin es a n d h o w to u s e r es p on s es to th es e ite m s to p o s itio n p er s o n s o n th e s e lin es .

In o r d e r f o r a te s t to d e fin e a va r ia b le o f m e n ta l a b ility , th e ite m s o u t o f w h ic h th e te s t is m a d e m u s t s h a re a lin e o f in q u ir y . T h is c o m m o n lin e a n d its d ir e c tio n to w a r d s in cr ea s in g a b ility ca n b e p ic tu r e d as a n a r r o w w ith h ig h a b ility to th e r ig h t a n d lo w a b ility to th e le ft. T h e m e a n in g o f th is a r r o w is g ive n b y th e te s t ite m s w h ic h d e fin e it. I f w e u se th e s y m b o ls 6 1 , 6 2 . . . 6 t . . . , to r e p r e s e n t th e d iffic u lty le ve ls o f item s , th e n ea ch 5 t m a r k s th e lo c a tio n o f a n ite m o n th e lin e . T h e 5 ’s a r e th e c a lib r a tio n s o f th e ite m s a lo n g th e va r ia b le a n d th es e c a lib r a te d ite m s a r e th e o p e r a tio n a l d e fin itio n o f w h a t th e va r ia b le m ea s u r es . H a r d ite m s w h ic h c h a lle n g e th e m o s t a b le p er s on s d e fin e th e h ig h , o r r ig h t, en d o f th e lin e . E a s y ite m s w h ic h e ve n th e lea s t a b le p er s o n s ca n u s u a lly d o s u cces s fu lly d e ­ fin e th e lo w , o r le ft, e n d o f th e lin e. F ig u r e 1 .1 .2 s h o w s a va r ia b le d e fin e d b y fo u r ite m s s p r ea d a cro s s its le n g th .

1

2

BEST T E ST D E S IG N

F I G U R E 1 .1 .2

D E F IN IN G A V A R IA B L E

P erson M e a su re

E asiest

H a rd e s t

It e m

It e m It e m C a lib ra tio n s

A va r ia b le b eg in s as a g en er a l id e a o f w h a t w e w a n t to m ea s u re. T h is g en er a l id ea is g iven s u b s ta n ce b y w r itin g te s t item s a im ed a t e lic itin g s ign s o f th e in te n d e d va r ia b le in th e b e h a vio r o f th e p ers on s . T h e s e te s t ite m s b e c o m e th e o p e r a tio n a l d e fin itio n o f th e va r ia b le. T h e in tu itio n o f th e te s t b u ild er a n d th e ca r e fu l' c o n s tr u ctio n o f p r o m is in g tes t item s , h o w e ve r , a re n o t en ou g h . W e m u s t a ls o g a th er e vid e n c e th a t a va r ia b le is in fa c t r e a lize d b y th e te s t item s . W e m u s t g ive th e item s to s u ita b le p ers on s a n d a n a lyze th e r es u ltin g r es p on s e p a tter n s to s ee i f th e ite m s f it to g e th e r in s u ch a w a y th a t res p on s es to th e m d e fin e a va r ia b le .

In o r d e r to lo c a te a p er s o n o n th is va r ia b le w e m u s t te s t th em w ith s o m e o f th e item s w h ic h d e fin e th e va r ia b le a n d th en d e te r m in e w h e th e r th e ir res p on s es a d d u p to a p o s itio n o n th e lin e. I f w e u se th e s y m b o l 0 to r ep r es en t th e a b ility le ve l o f th e p er s on , th en j3 m a rk s th e ir lo c a tio n o n th e lin e.

T h e p er s on m ea s u re 0 s h o w n in F ig u r e 1 .1 .2 lo ca te s th is p er s o n a b o ve th e th r ee ea s ies t item s a n d b e lo w th e h a r d es t o n e. W er e th is p er s on to ta k e a te s t m a d e u p o f th es e fo u r item s , th e ir m o s t p r o b a b le te s t s cor e w o u ld b e th r ee a n d w e w o u ld e x p e c t th em to g e t th e th r ee ea s ies t item s c o r r e c t a n d th e fo u r th , h a r d es t ite m , in c o r r e c t. T h is ob s er ­ v a tio n is m o r e im p o r ta n t th a n it m ig h t s eem b eca u s e it is th e b a s is o f a ll o u r m e th o d s f o r e s tim a tin g p er s on m ea s u res fr o m te s t s cores . W h en w e w a n t to k n o w w h e r e a p er s on is lo c a te d o n a va r ia b le, w e o b ta in th e ir res p on s es to s om e o f th e ite m s w h ic h d e fin e th e va r ia b le . T h e o n ly r ea s on a b le p la ce to es tim a te th e ir lo c a tio n fr o m th es e d a ta is in th e r e g io n w h e r e th e ir res p on s es s h ift fr o m m o s tly c o r r e c t o n ea s ier ite m s to m o s tly in c o r r e c t o n h a r d er on es . B e fo r e

we

ca n

es tim a te a p e r s o n ’s m ea s u re fr o m

th e ir s co r e, h o w e ve r , w e m u s t

e x a m in e th e ir p a tter n o f res p on s es . W e m u s t s ee i f th e ir p a tter n is con s is ten t w ith h o w w e e x p e c t th e ir item s to e lic it res p on s es . W h en th e item s w ith w h ic h a p er s on is tes ted h a ve b een ca lib r a ted a lo n g a va r ia b le fr o m ea s y to h a rd , th en w e e x p e c t th e p e r s o n ’s res p on s e p a tter n to b e m o r e o r less c o n s is ten t w ith th e d iffic u lty o r d e r o f th es e item s a lo n g th e

THE M EASUR EM ENT M ODEL

3

va r ia b le . W e e x p e c t th e p e r s o n to s u c ce ed o n ite m s th a t o u g h t to b e ea s y fo r th e m a n d to fa il o n ite m s th a t o u g h t t o b e h a r d fo r th e m .

F ig u r e 1 .1 .3 s h o w s t w o r e s p o n s e p a tter n s to th e s a m e te n ite m tes t. T h e te n item s a r e lo c a te d a lo n g th e va r ia b le a t th e ir le ve ls o f d iffic u lty . E a ch p a tte r n o f r es p on s es is r e c o r d e d a b o v e th e lin e o f th e va r ia b le . T h e l ’s r e p r e s e n t c o r r e c t a n s w ers . T h e 0 ’s r e p r e ­ s en t in c o r r e c t a n s w er s . B o th p a tte r n s p r o d u c e a s c o r e o f s ix .

In P a tte r n A th e s ix ea s ies t ite m s a r e c o r r e c t a n d th e fo u r h a r d es t o n es in c o r r e c t. I t s eem s in c o n c e iv a b le to lo c a te th is p e r s o n a n y w h e r e e x c e p t in th e r e g io n a b o ve 5 6 , th e m o s t d iffic u lt ite m

th e y g e t c o r r e c t, b u t b e lo w 5 7 , th e lea s t o f th e e ve n m o r e d iffic u lt

ite m s th e y g e t in c o r r e c t.

4

BEST T EST D E SIGN

P a tter n B , h o w e ve r , is v e r y d iffic u lt to r e c o n c ile w ith th e im p lica tio n s o f a s core o f s ix . T h is p er s on gets th e s ix h a rd es t item s c o r r e c t a n d th e fo u r ea s ies t on es in c o r r e c t! I f w e tr y to lo c a te th is p er s on a b o ve 5 1 0, th e h a r d es t item s th e y g e t c o r r e c t, w e h a ve to e x p la in h o w th e y g o t th e fo u r ea s ies t item s in co r r ec t. C o u ld a n yo n e b e th a t ca reles s ? If, o n th e o th e r h a n d , w e tr y to lo c a te th e m b e lo w 5 ^ , th e ea s ies t ite m th e y g e t in co r r ect, th en h o w d o w e e x p la in th e ir g e ttin g th e s ix h a r d es t item s in c o r r e c t? E ve r y o th e r lo c a tio n a lo n g th e va r ia b le, s u ch as b e tw e e n 6 6 a n d 6 7 fo r a s cor e o f s ix , is e q u a lly u n s a tis fa cto r y as a “ m ea s u r e” fo r th e p er s on w h o p r o d u c e d P a tter n B . T h is p a tter n o f res p on s es is n o t c o n s is ten t w ith a n y lo c a tio n

o n th e va r ia b le d e fin e d b y th es e item s . W e a re fo r c e d to

c o n c lu d e th a t s o m eth in g is w r o n g . E ith e r th e item s u s ed a r e m is ca lib r a ted o r th is p er s on d id n o t ta k e th e m in th e w a y w e in te n d e d . In a n y ca s e, n o r ea s on a b le m ea s u re ca n b e d e r ive d fr o m P a tter n B .

T h e P a tter n B e x a m p le is a n im p o r ta n t o n e b eca u s e it s h ow s u s th a t e ven w h en w e h a ve c o n s tr u cted item s th a t ca n d e fin e a va lid va r ia b le w e s till h a ve a ls o to va lid a te e ve r y p e r s o n ’s res p on s e p a tter n b e fo r e p r o c e e d in g to u se th e ir s cor e as a b a s is fo r es tim a tin g th e ir m ea s u re. W h en ite m ca lib r a tion s h a ve b een va lid a te d b y e n o u g h s u ita b le p ers on s , th en m o s t o f th e res p on s e p a tter n s w e e n c o u n te r a m o n g s u ita b le p ers on s w ill a p p r o x i­ m a te P a tter n A . H o w e ve r , th e p o s s ib ility o f occu r r en ces ver g in g o n P a tter n B fo r c e s us to e x a m in e a n d va lid a te r o u tin e ly th e res p on s e p a tter n o f e ve r y p er s on tes ted b e fo r e w e ca n p res u m e to es tim a te a m ea s u re fr o m th e ir te s t s core.

F o u r s tep s m u s t b e ta k en to u se a te s t to m ea s u re a p er s on . F ir s t, w e m u s t w o r k o u t a c lea r id ea o f th e va r ia b le w e in te n d to m a k e m ea s u res o n . S eco n d , w e m u s t c o n ­ s tr u ct item s w h ic h a re b elie va b le r e a liza tio n s o f th is id e a a n d w h ic h ca n e lic it s ign s o f it in th e b e h a vio r o f th e p ers on s w e w a n t to m ea s u re. T h ir d , w e m u s t d em o n s tr a te th a t th es e ite m s w h en ta k en b y s u ita b le p ers on s ca n lea d to res u lts th a t a re con s is ten t w ith o u r in te n tio n s . F in a lly , b e fo r e w e ca n u se a n y p e r s o n ’s s co r e as a b a s is fo r th e ir m ea s u re, w e m u s t d e te r m in e w h e th e r o r n o t th e ir p a r ticu la r p a tter n o f res p on s es is, in fa c t, c o n ­ s is ten t w ith o u r e x p e cta tio n s .

1 .2

HOW SCORES A R E USED

A te s t s cor e is in te n d e d to lo c a te a p er s on o n th e va r ia b le d e fin e d b y th e te s t item s ta k en . N e a r ly e ve r y o n e w h o u ses te s t s cores s u p p os es th a t th e p e r s o n ’s lo c a tio n o n th e va r ia b le is s a tis fa c to r ily d e te r m in e d e ith e r b y th e s core its e lf, o r b y s om e lin ea r fu n c tio n o f th e s cor e s u ch as a p e r c e n t c o r r e c t o r a n or m - b a s ed s ca le va lu e. I t is ta k en fo r g r a n ted th a t th e s cor e, o r its s ca le e q u iva le n t, tells u s s o m eth in g a b o u t th e p er s on te s te d th a t goes b e y o n d th e m o m e n t o r m a teria ls o f th e tes tin g . I t is a ls o ta k en fo r g r a n ted th a t s cores a re s u ita b le fo r u s e in th e a r ith m e tic n eces s a ry to s tu d y g r o w th a n d c o m p a r e grou p s . B u t d o s cores a ctu a lly h a ve th e p r o p e r tie s n eces s a ry to m a k e it r ea s on a b le to u s e th em in th es e w a ys ? In o r d e r f o r a p a r ticu la r s cor e to h a ve m ea n in g it m u s t c o m e fr o m a res p on s e p a tter n w h ic h is con s is ten t w ith item s th a t d e fin e a va r ia b le . B u t e ven th e d em o n s tr a tio n o f ite m v a lid ity a n d res p on s e v a lid ity d o e s n o t g u a ra n tee th a t th e s cor e w ill b e u s efu l. In o r d e r to g en er a lize a b o u t th e p er s on b e y o n d th e ir s cor e, in o r d e r to d is c o ve r w h a t th e ir s core im p lies , w e m u s t a ls o ta k e in to a c c o u n t a n d a d ju s t fo r th e p a rticu la rs o f th e tes t item s u s ed. H o w , th en , d o es a p e r s o n ’s te s t s cor e d ep en d o n th e ch a ra cteris tics o f th e item s in th e tes t th e y ta k e?

THE M EASUR EM ENT M ODEL

5

f

F I G U R E 1 .2 .1

I

HOW SCORES D EPEND O N TH E LE V E L A N D SPREAD O F T E S T IT E M D IF F IC U L T IE S

P e rs on

P

V e ry E a sy Test

6 ,

6 8 E x p e c te d S c o re 8

P e rs o n ,3

V e ry H a rd

1 .................. 1 1 ■ ■ ■

Test

S.

"1

E x p e c te d

°8

S c o re 0

P e rso n N a rro w H a rd

I ..................1

Test

'

Si E x p e c te d S c o re

1

Test C e n te r

N a rro w E a sy Test

W id e E a sy Test

Test C e n te r

'

Sg

-.

6

BEST T EST DESIGN

F ig u r e 1 .2 .1 s h ow s w h a t ca n h a p p en w h en o n e p er s on a t a p a r ticu la r a b ility le ve l ta k es fiv e d iffe r e n t tes ts a ll o f w h ich m ea s u re o n th e s a m e va r ia b le b u t w h ic h d iffe r in th e le ve l a n d s p rea d o f th e ir ite m d iffic u ltie s fr o m ea s y to h a r d a n d n a r r o w to w id e . T h e d iffic u ltie s o f th e e ig h t item s in ea ch tes t a re m a r k ed o n th e lin e o f th e va r ia b le. In o r d er to s ee ea ch tes t s ep a r a tely w e h a ve r ed r a w n th e lin e o f th e va r ia b le fiv e tim es , o n ce fo r ea ch tes t.

T h e a b ility o f th e p er s on o n th e m ea s u re is a ls o m a r k ed o n ea ch lin e s o th a t w e ca n s ee h o w th is p er s on s ta n d s w ith r es p ect to ea ch tes t. W h ile ea ch te s t has a d iffe r e n t p o s i­ tio n o n th e va r ia b le d ep en d in g o n th e d iffic u ltie s o f its item s , th is p e r s o n ’s p o s itio n , o f cou r s e, is th e s a m e o n ea ch lin e. F ig u r e 1 .2 .1 a ls o s h ow s th e s cores w e w o u ld e x p e c t th is p er s on m o s t o fte n to g e t o n th es e fiv e tes ts . T h e fir s t, V e r y E a s y T e s t, has item s s o ea s y fo r th is p er s on th a t w e e x p e c t a tes t s co r e o f eig h t. T h e s eco n d , V e r y H a r d T e s t, has s u ch h a r d item s th a t w e e x p e c t a s core o f ze r o . T h e th ir d , N a r r o w H a r d T e s t, has s even o f its item s a b o ve th e p e r s o n ’s a b ility a n d o n e b e lo w . In th is s itu a tio n th e s cor e w e w o u ld e x p e c t m o s t o fte n to s ee w o u ld b e a o n e. T h e fo u r th , N a r r o w E a s y T e s t, h as s even o f its item s b e lo w th e p er s o n ’s a b ility a n d s o w e e x p e c t a s cor e o f s even . F in a lly th e fifth , W id e E a s y T e s t, h a s fiv e item s w h ich s h ou ld b e ea s y fo r th em . E ven th o u g h th is te s t is c en te r e d a t th e s a m e p o s itio n o n th e va r ia b le as th e N a r r o w E a s y T e s t ju s t a b o ve it in F ig u r e 1 .2 .1 a n d s o has th e s a m e a vera ge d iffic u lty le ve l, n everth eles s , b eca u s e o f its g r ea ter w id th in ite m d iffic u lty , w e e x p e c t o n ly a s cor e o f five .

For

o n e p ers on

w e h a ve fiv e

e x p e c te d s cores :

ze r o ,

on e, fiv e , s even a n d e ig h t!

A lth o u g h w e k n o w th e p e r s o n ’s a b ility d o es n o t ch a n g e, th e fiv e d iffe r e n t s cores , as th e y s ta n d , s u gges t fiv e d iffe r e n t a b ilities . T e s t s cores o b vio u s ly d e p e n d as m u ch o n th e ite m ch a ra cteris tics o f th e tes t as o n th e a b ility o f th e p er s on ta k in g th e tes t.

I f th e m ea n in g o f a te s t s cor e d ep en d s o n th e ch a ra cteris tics o f th e te s t item s , h o w ­ ever , th en

b e fo r e

w e ca n

d e te r m in e a p e r s o n ’s a b ility fr o m

th e ir tes t s cor e w e m u s t

“ a d ju s t” th e ir s cor e fo r th e e ffe c ts o f th e p a r ticu la r tes t item s fr o m w h ic h th a t p a r ticu ­ la r s core com es . T h is a d ju s tm en t m u s t b e a b le to tu rn tes t- b ou n d s cores in to m ea s u res o f p er s on a b ility w h ic h a re te s t- fr ee.

U n fo r tu n a te ly , w ith te s t s cores lik e ze r o , in w h ic h th er e is n o in s ta n ce o f su ccess, a n d th e e ig h t o f o u r e x a m p le , in w h ic h th er e is n o in s ta n ce o f fa ilu r e, th er e is n o s a tis ­ fa c to r y w a y to s ettle o n a fin ite m ea s u re fo r th e p er s on . A ll w e ca n d o in th o s e s itu a tion s is to o b s er ve th a t th e p er s on w h o s co r ed a ll in c o r r e c t o r a ll c o r r e c t is s u b s ta n tia lly b e lo w o r a b o ve th e o p e r a tin g le ve l o f th e tes t th e y h a ve ta k en . I f w e w is h to es tim a te a fin ite m ea s u re fo r s u ch a p er s on , th en w e w ill h a ve to fin d a te s t fo r th em w h ic h is m o r e a p p r o ­ p r ia te to th e ir le ve l o f a b ility .

W e m ig h t b e te m p te d to in te r p r e t p e r fe c t s cores as “ c o m p le te m a s te r y.” B u t u n less th e tes t in q u es tio n a c tu a lly co n ta in ed th e m o s t d iffic u lt item s th a t c o u ld e ve r b e w r itte n fo r th is va r ia b le th er e w o u ld a lw a ys b e th e p o s s ib ility o f o th e r item s w h ich w e r e even m o r e d iffic u lt. T h e s e m o r e d iffic u lt item s m ig h t p r o d u c e in c o r r e c t a n s w ers , even w ith o u r p e r fe c tly s co r in g p er s on , r e ve a lin g th a t m a s ter y w a s n o t c o m p le te a fte r a ll. W h en a te s t is e x tr e m e ly ea s y, o f cou rs e, e ve r y o n e r e co g n izes th a t even a p e r fe c t s core is q u ite c on s is ten t w ith in te r m e d ia te a b ility.

THE M EASUR EM ENT M ODEL

Th e

d ep en d en ce

7

o f te s t s co r es o n ite m

d iffic u lty is a p r o b le m

w ith

w h ic h m o s t

te s t u sers a r e fa m ilia r . A lm o s t e v e r y o n e r e a lizes th a t f i f t y p e r c e n t c o r r e c t o n a n ea s y te s t d o e s n o t m e a n as m u ch as f i f t y p e r c e n t c o r r e c t o n a h a r d tes t. S o m e te s t u s ers e ve n r e a lize th a t s e v e n ty - fiv e p e r c e n t c o r r e c t o n a n a r r o w te s t d o e s n o t im p ly as m u ch a b ility as s e v e n ty - fiv e p e r c e n t c o r r e c t o n a w id e tes t. B u t th e r e is a n o th e r p r o b le m in th e u s e o f te s t s co r es w h ic h is o fte n o v e r lo o k e d .

I t is c o m m o n p r a c tic e t o c o m p u te d iffe r e n c e s in te s t s cor es to m ea s u r e g r o w th , to c o m b in e te s t s co r es b y a d d itio n a n d s u b tr a c tio n in o r d e r to c o m p a r e g r o u p s a n d to a d d a n d s u b tr a c t s q u a res a n d c r o s s - p r o d u c ts o f te s t s co r es in o r d e r to d o r eg r es s io n a n a lys is . B u t w h e n th e s e s im p le a r ith m e tic o p e r a tio n s a r e a p p lie d to te s t s cor es th e res u lts a re a lw a y s s lig h tly d is to r te d a n d ca n b e s u b s ta n tia lly m is lea d in g . A lth o u g h te s t s co r es u s u a lly e s tim a te th e o r d e r o f p e r s o n s ’ a b ilitie s r a th e r w e ll, th e y n e v e r e s tim a te th e s p a cin g s a t­ is fa c to r ily .

T e s t s co r es a r e n o t lin e a r in th e m ea s u res th e y im p ly a n d f o r w h ic h

th e y

a r e u s ed .

In th e s ta tis tica l u s e o f te s t s cor es , f l o o r a n d c e ilin g e ffe c ts a r e o c c a s io n a lly r e c o g ­ n ize d . B u t th e y a r e a lm o s t n e v e r a d ju s te d fo r . T h e s e b o u n d a r y e ffe c ts ca u s e a n y fix e d d iffe r e n c e s o f s c o r e p o in ts to v a r y in m e a n in g o v e r th e s co r e ra n g e o f th e tes t. T h e d is ­ ta n c e o n th e va r ia b le a p a r tic u la r d iffe r e n c e in s c o r e p o in ts im p lie s is n o t th e s a m e fr o m o n e e n d o f th e te s t to th e o th e r . A d iffe r e n c e o f fiv e s co r e p o in ts , fo r e x a m p le , im p lies a la r g e r ch a n g e in a b ility a t th e en d s o f a te s t th a n in th e m id d le .

F ig u r e 1 .2 .2 illu s tr a tes th is p r o b le m

w ith

te s t s co res . W e s h o w tw o p er s on s w ith

m ea s u res , 0 A a n d p B , w h o a r e a fix e d d is ta n ce a p a r t o n th e s a m e va r ia b le. B o th p er s o n s a r e a d m in is te r e d fiv e d iffe r e n t tes ts a ll m ea s u r in g o n th is va r ia b le . T h e p e r s o n s ’ lo c a tio n s a n d h e n c e th e ir m ea s u r a b le d iffe r e n c e o n th e va r ia b le r e m a in th e s a m e fr o m te s t to tes t, b u t th e ir m o s t p r o b a b le s co r es v a r y w id e ly . T h is is b eca u s e th e fiv e tes ts d iffe r in th e ir ite m

d iffic u lty

le ve l,

s p r ea d

and

s p a cin g . L e t ’s s ee h o w

th e r e s u ltin g e x p e c te d s cores

r e fle c t th e fix e d d iffe r e n c e b e tw e e n th es e t w o p er s on s .

T e s t I is c o m p o s e d o f e ig h t ite m s a ll o f w h ic h fa ll w e ll b e tw e e n P er s o n A a n d P er s on B . W e e x p e c t P e r s o n A to g e t n o n e o f th es e ite m s c o r r e c t fo r a s co r e o f z e r o w h ile w e e x p e c t P e r s o n B to g e t a ll e ig h t ite m s c o r r e c t f o r a s c o r e o f e ig h t. O n th is te s t th e ir a b ili­ tie s w ill u s u a lly a p p e a r to b e e ig h t s c o r e p o in ts a p a r t. T h a t is as fa r a p a r t in a b ility as it is p o s s ib le to b e o n th is tes t.

T e s t I I is c o m p o s e d o f e ig h t ite m s a ll o f w h ic h a r e w e ll b e lo w b o th p er s on s . W e e x ­ p e c t b o th p er s o n s to g e t s co r es o f e ig h t b eca u s e th is te s t is t o o ea s y fo r b o th o f th e m . N ow

th e ir e x p e c te d s c o r e d iffe r e n c e in te s t s cor es is z e r o a n d th e ir a b ilitie s w ill u s u a lly

a p p e a r t o b e th e s a m e!

T e s t I I I is c o m p o s e d o f e ig h t v e r y h a r d item s . N o w w e e x p e c t b o th p er s on s to g e t s co r es o f z e r o b eca u s e th is te s t is t o o h a r d fo r th e m . O n c e a g a in th e ir e x p e c te d s co r e d if­ fe r e n c e is z e r o a n d th e ir a b ilitie s w ill u s u a lly a p p ea r to b e th e s a m e.

T e s t I w a s s u cces s fu l in s ep a r a tin g P er s on s A a n d B . T e s ts I I a n d I I I fa ile d b eca u s e th e y w e r e t o o fa r o f f ta r g e t. P er h a p s it is o n ly n eces s a r y to c e n te r a te s t p r o p e r ly in o r d e r to o b s e r ve th e d iffe r e n c e b e tw e e n tw o p ers on s .

8

BEST T EST DE SIGN

__________ |

F I G U R E 1 .2 . 2

|_ _________

T H E N O N L IN E A R IT Y O F S C O R E S

0A

Pb

Test I

T e s t II

i I 5 1

................................. i . . . . . .

Pa

Pb

E x p e c te d

E x p e c te d

S c o re

S c o re

8

8

| 58

E x p e c te d S c o re d iffe r e n c e : 8 -8

= 0

Pa T est III ■

1 «1

T e s t IV

Test V

E x p e c te d

E x p e c te d

S c o re

S c o re

0

0

........................................ ........................... ...... 1 ^10

E x p e c te d S c o re d iffe r e n c e : 0 -0

= 0

THE M EASUR EM ENT M ODEL

• Test I V

9

is c e n te r e d b e tw e e n P er s o n A a n d P er s o n B b u t its ite m s a re s o s p r ea d o u t

th a t th e r e is a w id e g a p in its m id d le in to w h ic h P er s o n A a n d P er s o n B b o th fa ll. T h e r e s u lt is th a t b o th p er s o n s ca n b e e x p e c te d to a c h ie ve s co r es o f fo u r b eca u s e fo u r ite m s a r e t o o e a s y a n d fo u r ite m s a r e t o o h a r d f o r b o th o f th e m . E ve n f o r th is te s t w h ic h is m o r e o r les s c e n te r e d o n th e ir p o s itio n s , th e ir e x p e c te d s c o r e d iffe r e n c e is z e r o a n d th e ir a b ilitie s w ill s till u s u a lly a p p e a r to b e th e s a m e.

T e s t V , a t la s t, is b o th

w id e a n d fa ir ly w e ll c e n te r e d o n P er s on s A a n d B . I t c o n ­

ta in s tw o ite m s w h ic h fa ll b e tw e e n th e ir p o s itio n s a n d th e r e fo r e s ep a r a te th e m . W e e x p e c t P e r s o n A to g e t th e fo u r ea s ies t ite m s c o r r e c t f o r a m o s t p r o b a b le s co r e o f fo u r . A s fo r P e r s o n B , h o w e v e r , w e e x p e c t th e m n o t o n ly to g e t th e s a m e fo u r ite m s c o r r e c t b u t a ls o th e n e x t t w o h a r d e r o n e s b eca u s e th es e tw o ite m s a r e a ls o b e lo w P er s o n B ’s a b ility le ve l. Th u s on T es t V

th e e x p e c te d d iffe r e n c e in s co r es b e tw e e n P e r s o n ’s A a n d B b e c o m e s

tw o . O n th is te s t th e ir a b ilitie s w ill u s u a lly a p p e a r to b e s o m e w h a t, b u t n o t e x tr e m e ly , d iffe r e n t.

W h a t ca n w e in fe r a b o u t th e d iffe r e n c e s in a b ility b e tw e e n P er s o n s A

a n d B fr o m

s co r es lik e th e s e ? P er s o n s A a n d B w ill te n d to a p p e a r e q u a lly a b le o n T e s ts II, I I I , a n d I V , s o m e w h a t d iffe r e n t on . T e s t V b e tw e e n

a n d as d iffe r e n t as p o s s ib le o n T e s t I. I f d iffe r e n c e s

th e te s t s co r es o f th e s a m e t w o p er s o n s ca n b e m a d e t o v a r y s o w id e ly m e r e ly

b y c h a n g in g th e d iffic u ltie s o f th e ite m s in th e te s t, th e n h o w ca n w e u s e d iffe r e n c e s in te s t s co r es to s tu d y a b ility d iffe r e n c e s o n a va r ia b le ?

T h e a n s w e r is, w e c a n ’t. N o t as th e y s ta n d . In o r d e r to u s e te s t s cor es , w h ic h a re n o t lin e a r in th e va r ia b le th e y im p ly , to a n a ly ze d iffe r e n c e s w e m u s t fin d a w a y to tr a n s ­ fo r m th e te s t s co r es in to m ea s u r es w h ic h a p p r o x im a te lin e a r ity .

T e s t s co r es a lw a y s c o n ta in a p o te n tia lly m is le a d in g d is to r tio n . I f w e in te n d to u se te s t res u lts to s tu d y g r o w th m a k in g m ea s u res fr o m

a n d to c o m p a r e g r o u p s , th e n w e m u s t u s e a m e th o d fo r

te s t s co r es w h ic h m a r k s lo c a tio n s a lo n g th e va r ia b le in a n e q u a l

in te r v a l o r lin e a r w a y .

In th is s e c tio n w e h a ve illu s tr a te d tw o s er io u s p r o b le m s w ith te s t s co res . T h e fir s t illu s tr a tio n s h o w s h o w te s t s co r es a r e te s t- b o u n d a n d h o w w e h a ve to a d ju s t th e m fo r th e c h a r a c te r is tic s o f th e ir te s t ite m s b e fo r e w e ca n u s e th e s co r es as a b a s is f o r m ea s u r em en t. T h e s e c o n d illu s tr a tio n s h o w s h o w te s t s co r es d o n o t m a r k lo c a tio n s o n th e ir va r ia b le in a lin e a r w a y a n d h o w w e n e e d to tr a n s fo r m te s t s co r es in to m ea s u res th a t a r e lin ea r b e fo r e w e ca n u s e th e m to s tu d y g r o w th o r to c o m p a r e g r ou p s .

1 .3

W H A T H A P P E N S W H E N A P E R S O N T A K E S A N IT E M

T h e d is cu s s ion s in S e c tio n s 1 .1 a n d 1 .2 es ta b lis h o u r n e e d fo r 1 ) v a lid ite m s w h ic h ca n b e d e m o n s tr a te d to d e fin e a va r ia b le , 2 ) v a lid r es p o n s e p a tter n s w h ic h ca n b e u s ed to lo c a te p er s o n s o n th is va r ia b le , 3 ) te s t- fr e e m ea s u res th a t ca n b e u s ed to c h a r a c te r ize p e r ­ s on s in a g en er a l w a y a n d 4 ) lin e a r m ea s u res th a t ca n b e u s ed to s tu d y g r o w th a n d c o m ­ p a r e g r o u p s . N o w w e m u s t b u ild a m e th o d th a t c o m e s to g r ip s w ith th es e r eq u ir em en ts .

T h e r es p o n s es o f in d ivid u a l p er s o n s to in d ivid u a l ite m s a r e th e r a w d a ta w ith w h ic h w e b e g in . T h e m e th o d w e d e v e lo p m u s t ta k e th es e d a ta a n d m a k e fr o m th e m ite m ca li­ b r a tio n s a n d p er s o n m ea s u res w ith th e p r o p e r tie s w e r e q u ir e . F ig u r e 1 .3 .1 s h o w s a v e r y

10

BEST T EST DE SIGN

s im p le d a ta m a tr ix c o n ta in in g th e res p on s es o f e ig h t p er s on s to a fiv e ite m tes t. T h e fiv e item s a re n a m ed a t th e to p o f th e m a tr ix . T h e e ig h t p ers on s a re n a m ed a t th e le ft. T h e res p on s e o f ea ch p er s on to ea ch ite m is in d ic a ted b y “ 1 ” fo r a c o r r e c t res p on s e a n d “ 0 ” fo r an in c o r r e c t res p on s e. N o tic e th a t th e res p on s es in F ig u r e 1 .3 .1 h a ve b een s u m m ed a cros s th e ite m s a n d e n ter ed o n th e r ig h t s id e o f th e m a tr ix as p er s on s cores

and d ow n

th e p ers on s a n d e n ter ed a t th e b o tto m o f th e m a tr ix as ite m s cores .

J

F I G U R E 1 .3 .1

I

A D A T A M A T R IX O F O B S E R V E D R E S PO N S ES

It e m Person

N am e

P erson 1

2

3

4

5

a

1

0

0

0

0

b

0

1

0

0

0

1

c

1

1

0

0

0

2

d

1

0

1

0

0

2

0

0

3

1

0

3

N am e

e

1

1

1

f

1

1

0

S c o re 1

9

1

1

1

1

0

4

h

1

1

1

0

1

4

7

6

4

2

1

20

It e m S c o re

F ig u r e 1 .3 .1 s h ow s w h a t th e b a s ic d a ta lo o k lik e. B u t b e fo r e w e ca n p u t th es e d a ta to w o r k w e m u s t a n s w er a fu n d a m e n ta l q u es tio n . W h er e d o w e th in k th es e d a ta c o m e fr o m ? W h a t a re th es e ite m a n d p er s on s cores s u p p os ed to te ll u s a b o u t item s a n d p er ­ s on s? H o w d o w e th in k th es e p a tter n s o f l ’s a n d 0 ’s a re p r o d u ce d ? In o r d e r to fig u r e o u t h o w to u se th es e d a ta w e m u s t s et u p a r ea s on a b le m o d e l fo r w h a t w e s u p p os e h a p p en s w h en a p er s on a tte m p ts to a n s w er a n ite m .

W e w o u ld lik e a p er s on v ’s a b ility /?„, th a t is th e ir lo c a tio n o n th e va r ia b le, to g o ver n h o w fa r a lo n g th e va r ia b le w e ca n e x p e c t th em to p r o d u c e c o r r e c t res p on s es to item s . In d e e d th a t is th e o n ly s itu a tio n in w h ic h w e ca n u s e ite m

d iffic u ltie s a n d a p er s o n ’s

res p on s es to th e m as th e b a s is fo r m ea s u rin g th e p er s on . O f cou rs e w e ca n th in k o f o th e r fa c to r s w h ic h m ig h t a ffe c t a p e r s o n ’s res p on s es . I f item s a re m u ltip le- c h o ic e , s om e gu es s in g is b ou n d to o c c u r a n d p er s on s d iffe r in h o w m u ch

gu es s in g th e y a re w illin g to en g a ge in . T h e p o s s ib ilities o f d is tu r b in g in flu en ces

w h ic h

in te r fe r e

w ith

th e clea r ex p r es s ion a n d h en ce th e u n a m b ig u ou s o b s er va tio n o f

a b ility Eire en d les s . B u t, i f it is r e a lly th e p er s o n ’s a b ility th a t w e h o p e to m ea s u re, th en

11

T HE M EASUR EM ENT M ODEL

it w o u ld

b e u n r e a s o n a b le n o t to d o o u r b e s t to a r r a n g e th in g s s o th a t it is th e p e r s o n ’s

a b ility w h ic h

d o m in a te s th e ir te s t b e h a vio r . In d e e d , is n ’t th a t w h a t g o o d te s t a d m in is ­

tr a tio n p r a c tic e s a re fo r , n a m e ly , to c o n tr o l a n d m in im ize th e in tr u s io n o f in te r fe r in g in flu e n c e s .

We

w o u ld

d e te r m in e h o w

a ls o lik e

ite m

t’s d iffic u lty 5 t, th a t is its lo c a tio n o n th e va r ia b le , to

fa r a lo n g th e va r ia b le w e ca n e x p e c t c o r r e c t r es p on s es to th a t ite m to

o c c u r . A s w ith p er s o n s , w e ca n th in k u p ite m c h a r a cter is tics , s u ch as d is c r im in a tio n a n d v u ln e r a b ility to g u es s in g , w h ic h m ig h t m o d ify p e r s o n s ’ r es p on s es t o th e m . S o m e p s y c h o ­ m e tr ic ia n s a tte m p t to e s tim a te th es e a d d itio n a l ite m

ch a r a cter is tics e ve n th o u g h th e r e

a r e g o o d r ea s o n s to e x p e c t th a t a ll s u ch a tte m p ts m u s t, in p r in c ip le , fa il. B u t, a g a in , it h a r d ly s eem s r e a s o n a b le n o t to d o o u r b e s t to a r r a n g e th in g s s o th a t it is a n ite m ’s d if­ fic u lty w h ic h d o m in a te s h o w

p er s o n s o f va r io u s a b ilitie s r e s p o n d to th a t ite m . In a n y

ca s e, th e fa c t is th a t w h e n e v e r w e u s e u n w e ig h te d s cor es as o u r te s t res u lts w e a r e a s s u m in g th a t, f o r a ll p r a c tic a l p u r p o s es , it is ite m d iffic u ltie s , a n d p e r s o n a b ilitie s , th a t d o m in a te p e r s o n r es p on s es .

_ _ _ _ _ _ _ _ _ _ _ _ _ _ j

F IG U R E 1 .3 .2

|_ __ __ __ __ __ __ __ __ __ __ _

T H E E S S E N T I A L C O N D I T I O N S C A U S IN G A R E S P O N S E

P e rso n

O b s e rv e d

A b ility

R esponse

W h e n t h e res p o ns e is " c o r r e c t " , x = 1 W h e n t h e re s p o n s e is " in c o r r e c t " , x = 0

12

BEST T EST D ESIGN

T h es e con s id er a tio n s lea d u s to s et u p a res p on s e m o d e l th a t is th e s im p les t r ep r e­ s en ta tio n p os s ib le. F ig u r e 1 .3 .2 d ia g ra m s p er s on v w ith a b ility

a ctin g o n ite m i w ith

d iffic u lty 8 1 to p r o d u ce th e res p on s e x „ t. Th es e a re th e es s en tia l elem en ts w e w ill ta k e in to a c c o u n t w h en w e tr y to e x p la in th e d a ta in F ig u r e 1 .3 .1 . F ig u r e 1 .3 .2 p r o p o s es th a t th e r es p on s e x v l w h ic h occu r s w h en p er s on v ta k es ite m i ca n b e th o u g h t o f as g o ver n ed b y th e p e r s o n ’s a b ility 0V a n d th e ite m ’s d iffic u lty 5 ( a n d n o th in g els e. O u r n e x t s tep is to d e c id e h o w w e w a n t p er s on a b ility 0V a n d ite m d iffic u lty 5 1 to in te r a c t in o r d e r to p r o d u c e x „ t. W h a t is a r ea s on a b le a n d u s efu l w a y to s et u p a m a th e­ m a tic a l r e la tio n b e tw e e n 0V a n d 5 1? S in ce w e r e q u ir e th a t 0V a n d 6 1 r ep r es en t lo ca tio n s a lo n g o n e c o m m o n va r ia b le w h ich th e y s h a re, it is th e ir d iffe r e n c e (0 „ - 5 t) w h ich is th e m o s t c o n ve n ie n t a n d n a tu ra l fo r m u la tio n o f th e ir r e la tio n . Id e n tify in g th e d iffe r e n c e (0 „ -

5 t), h o w e ve r , d o e s n o t fin is h o u r w o r k b eca u s e w e

m u s t a ls o d e c id e h o w w e w a n t th is d iffe r e n c e to g o ver n th e va lu e o f th e res p on s e x v l. E ven w h e n a p er s on is m o r e a b le th a n an ite m

is d iffic u lt, s o th a t th e ir 0V is g r ea ter

th a n th e ite m ’s 5 ( , it w ill o cc a s io n a lly h a p p en th a t th is p er s on n ever th eles s fa ils to g ive a c o r r e c t a n s w er to th a t r e la tive ly ea s y ite m s o th a t th e r es u ltin g va lu e o f x v i is “ 0 ” . It w ill a ls o h a p p en o cc a s io n a lly th a t a p er s on o f m o d e r a te a b ility n everth eles s s u cceed s o n a v e r y d iffic u lt ite m . O b vio u s ly it is g o in g to b e a w k w a r d to fo r c e a d eter m in is tic r e la tio n ­ s h ip o n to th e w a y th is p r o b le m

-

5 t) g over n s th e va lu e o f res p on s e x v i. A b e tte r w a y to d ea l w ith

is to a c k n o w le d g e th a t th e w a y th e d iffe r e n c e (0V -

6 t) in flu en c es th e r e­

s p on s e x „ t ca n o n ly b e p r o b a b ilis tic a n d to s et u p o u r res p on s e m o d e l a c c o r d in g ly. F ig u r e 1 .3 .3 s h ow s h o w it w o u ld b e m o s t r ea s on a b le to h a ve th e d iffe r e n c e (0 „ - 8t) a ffe c t th e p r o b a b ility o f a c o r r e c t res p on s e. W h en

is la r g er th a n 5 ( , s o th a t th e a b ility

le ve l o f p er s on v is g r ea ter th a n th e d iffic u lty le ve l o f ite m i a n d th e ir d iffe r e n c e (/J„ - 5 t) is g r e a ter th a n ze r o , th en w e w a n t th e p r o b a b ility o f a c o r r e c t a n s w er to b e g r ea ter th a n o n e h a lf. W h en , o n th e o th e r h a n d , th e a b ility le ve l o f p er s on v is less th a n th e d iffic u lty le ve l o f ite m i, s o th a t th e ir d iffe r e n c e (0 „ — 6 1) is les s th a n ze r o , th e n w e w a n t th e p r o b a ­ b ility o f a c o r r e c t a n s w er to b e les s th a n o n e h a lf. F in a lly , w h en th e levels o f p er s on a b ility a n d ite m d iffic u lty a re th e s a m e s o th a t th e ir d iffe r e n c e (0 „ -

5 1) is ze r o , th en

th e o n ly p r o b a b ility th a t s eem s r ea s on a b le to assign to a c o r r e c t ( o r to a n in c o r r e c t) a n s w er is e x a c tly o n e h a lf. T h e cu r ve in F ig u r e 1 .3 .4 s u m m a rizes th e im p lic a tio n s o f F ig u r e 1 .3 .3 fo r a ll r ea s on ­ a b le r ela tion s h ip s

b e tw e e n

p r o b a b ilitie s

o f c o r r e c t res p on s es a n d d iffe r e n c e s b e tw e e n

p er s on a b ility a n d ite m d iffic u lty . T h is cu r ve s p ecifies th e c o n d itio n s o u r res p on s e m o d e l m u s t fu lfill. T h e d iffe r e n c e s (fiv -

5 t) c o u ld a ris e in tw o w a ys . T h e y c o u ld a rise fr o m a

v a r ie ty o f p er s on a b ilitie s r e a c tin g to a s in gle ite m o r th e y c o u ld a ris e fr o m a v a r ie ty o f ite m d iffic u ltie s te s tin g th e a b ility o f o n e p er s on . W h en th e cu r ve is d r a w n w ith a b ility

0 as its va r ia b le s o th a t it d es cr ib es a n ite m , it is c a lle d a n ite m ch a r a cter is tic cu r ve ( IC C ) b eca u s e it s h ow s th e w a y th e ite m e lic its res p on s es fr o m p er s on s o f e ve r y a b ility . W h en th e cu r ve is d r a w n w ith d iffic u lty 5 as its va r ia b le s o th a t it d es cr ib es h o w a p er s on r es p on d s to a v a r ie ty o f item s , w e ca n ca ll it a p er s on ch a r a cter is tic cu r ve (P C C ).

THE M EASUR EM ENT M ODEL

13

[

F IG U R E 1 .3 .3

|

H O W D IF F E R E N C E S B E T W E E N P E R S O N A B IL IT Y A N D IT E M D IF F IC U L T Y O U G H T T O A F F E C T T H E P R O B A B IL IT Y O F A C O R R E C T R E S P O N S E

K 1.

W h en

0 „> 5 t

f

( 0 „ - 5 t) > O

5,

2.

W hen

P„< 6t

I (Pit ~

8 t) < 0

a n d P |x j,t = l J

5 t

< ’/2

Pv

\

3 . W hen P it =

t 5t (/3„- 6 t) = 0 a n d P { x w = 1 |= %

T h e c u r ve in F ig u r e 1 .3 .4 is a p ic tu r e o f th e r es p o n s e m o d e l w e r e q u ir e in o r d e r to s o lve th e p r o b le m o f h o w th e p a r a m eter s , Pv a n d 5 1 w h ic h w e w a n t to e s tim a te , d e p e n d o n th e d a ta

w e ca n o b s e r ve . T o m ea s u r e a p er s o n , w e m u s t e s tim a te pv a n d to c a li­

b r a te a n ite m w e m u s t e s tim a te 6 t. In o r d e r to e s tim a te e ith e r o f th es e p a r a m eter s fr o m th e o b s e r ve d r es p on s es o f p er s o n s to ite m s w e m u s t c o n s tr u c t a m a th e m a tic a l fo r m u la ­ tio n w h ic h is tr u e to th e r e la tio n s h ip d r a w n in F ig u r e 1 .3 .4 a n d w h ic h r ela tes pv , 5 t a n d

x v l in a u s e fu l w a y . T h is fo r m u la tio n m u s t a ls o b e a b le to s h o w u s h o w to u s e d a ta o f th e k in d g ive n in

F ig u r e 1 .3 .1 to m a k e e s tim a tes o f p er s o n a b ility w h ic h a r e te s t- fr e e

e s tim a tes o f ite m d iffic u lty w h ic h a r e s a m p le- fr ee.

and

THE

I

| ___

CURVE

1.3.4

RESPONSE

F IG U R E

14 BEST T EST D E SIGN

THE M EASUR EM ENT M ODEL

1 .4

15

TH E RASCH M O DEL

In

o r d e r to c o n s tr u c t a w o r k a b le m a th e m a tic a l fo r m f o r th e c u r ve in F ig u r e 1 .3 .4

w e b e g in b y c o m b in in g th e p a r a m eter s , th r o u g h th e ir d iffe r e n c e ( P v -

f o r p er s o n a b ility a n d 6 t f o r ite m d iffic u lty

6 ( ). W e w a n t th is d iffe r e n c e to g o v e r n th e p r o b a b ility o f

w h a t is s u p p o s e d to h a p p e n w h e n p er s o n v u s es th e ir a b ility p v a g a in s t th e d iffic u lty 5 1 o f ite m i. B u t th e d iffe r e n c e (0 „ -

6 t) ca n v a r y fr o m m in u s in fin ity to p lu s in fin ity w h ile

th e p r o b a b ility o f a s u cces s fu l r e s p o n s e m u s t r e m a in b e tw e e n z e r o a n d o n e . T o d ea l w ith th is w e a p p ly th e d iffe r e n c e ( 0 „ - 6 t) as a n e x p o n e n t o f th e n a tu r a l c o n s ta n t e = 2 .7 1 8 2 8 . . . a n d w r ite th e r e s u lt as

e(Pv~ 8 t) = exp T h is e x p o n e n tia l e x p r e s s io n va r ie s b e tw e e n

z e r o a n d p lu s in fin ity a n d w e ca n b r in g it

in to th e in te r v a l b e tw e e n z e r o a n d o n e b y fo r m in g th e r a tio

e x p ( P „ - 5 t)/[1 + e x p ( 0 „- 5 t> ] .

T h is fo r m u la tio n h a s a s h a p e w h ic h fo llo w s th e o g iv e in F ig u r e 1 .3 .4 q u ite w e ll. I t ca n b e u s ed t o s p e c ify th e p r o b a b ility o f a s u cces s fu l r es p o n s e as

P l x ^ = I|j3 „ , 5 J = e x p ( Pv - 6 t ) / [ 1 + e x p (Pv - 5 t ) ]

[ 1 .4 .1 1

w h ic h is th e R a s ch m o d e l.

A n y m a th e m a tic a l fo r m w h ic h d es cr ib e s a n o g iv e o f th e s h a p e in F ig u r e 1 .3 .4 c o u ld p r o v id e a s o lu tio n b e tw e e n

0

and

to th e lin e a r ity p r o b le m b y tr a n s fo r m in g s co r es w h ic h a r e r e s tr ic te d

100

p e r c e n t in to

“ m ea s u r es ”

w h ic h

ru n fr o m

m in u s in fin ity to p lu s

in fin ity .

A n y m a th e m a tic a l fo r m

w h ic h

r e la tes th e p r o b a b ility o f x „ t to th e d iffe r e n c e b e ­

tw e e n pv a n d 5 1 a n d w h ic h h a s e s tim a b le p a r a m eter s c o u ld a llo w u s to s tu d y ite m a n d r e s p o n s e v a lid ity . A l l w e h a ve to d o is to s p e c ify a w o r k a b le m o d e l

f o r h o w (j3„ -

g o ve r n s th e p r o b a b ility o f x „ t, u s e th is m o d e l to e s tim a te p v a n d 5 1fr o m

5 t)

s o m e d a ta a n d

th e n e x a m in e th e w a y th e s e d a ta f i t w ith p r e d ic tio n s c a lc u la te d fr o m th e m o d e l.

A n y o g iv e a n d a n y fo r m u la tio n , h o w e v e r , w ill n o t d o . In fa c t, o n ly th e fo r m u la tio n o f E q u a tio n 1 .4 .1 , th e R a s ch m o d e l, a llo w s u s t o e s tim a te p v a n d 5 t in d e p e n d e n tly o f o n e a n o th e r in s u ch a w a y th a t th e e s tim a tes pv a r e fr e e d fr o m th e e ffe c ts o f th e 6 ( a n d th e e s tim a te s d t a r e fr e e d fr o m th e e ffe c ts o f th e P y ’s.

T h e lo g is tic fu n c tio n

in E q u a tio n

1 .4 .1 p r o vid e s a s im p le , u s e fu l r es p o n s e m o d e l

th a t m a k es b o th lin e a r ity o f s ca le a n d g e n e r a lity o f m ea s u r e p o s s ib le . A lth o u g h b io m e tr i­ cia n s h a ve u s ed th e lo g is tic fu n c tio n s in ce 1 9 2 0 , it w a s th e D a n is h m a th e m a tic ia n G e o r g R a s ch ( 1 9 6 0 ) w h o fir s t a p p r e c ia te d its p s y c h o m e tr ic s ig n ific a n c e . R a s ch ca lls th e s p ecia l c h a r a c te r is tic p o s s ib le

o f th e

“ s p e c ific

m a th e m a tic a l

s im p le lo g is tic

fu n c tio n

w h ic h m a k es g e n e r a lity in m ea s u r e m e n t

o b je c tiv ity . ” H e a n d o th e r s h a ve s h o w n th a t th e r e is n o a lter n a tive

fo r m u la tio n

fo r

th e o g iv e in F ig u r e 1 .3 .4 th a t a llo w s e s tim a tio n o f th e

p e r s o n m ea s u res P v a n d th e ite m

c a lib r a tio n s 6 t in d e p e n d e n tly o f o n e a n o th e r (R a s c h ,

1 9 6 1 , 1 9 6 7 ; A n d e r s e n , 1 9 7 3 , 1 9 7 7 ; B a m d o r ff- N ie ls e n , 1 9 7 8 ). W h en th e e s tim a to r s fo r P v a n d 5 1 a r e d e r iv e d b y m a x im izin g a c o n d itio n a l lik lih o o d th e y a r e u n b ia s ed , con s is ten t,

BEST T EST D ESIGN

16

e ffic ie n t, a n d s u ffic ie n t (A n d e r s e n , 1 9 7 0 , 1 9 7 1 , 1 9 7 2 a , 1 9 7 3 , 1 9 7 7 ; H a b er m a n , 1 9 7 7 ). S im p le

a p p r o x im a tio n s

fo r

th es e

c o n d itio n a l

m a x im u m

lik e lih o o d

es tim a tor s

w h ich

a re a ccu ra te en o u g h fo r a lm o s t a ll p r a ctica l p u rp os es a re d es cr ib ed in W r ig h t a n d P a n ch a p a k es a n

(1 9 6 9 ),

W r ig h t a n d

D ou g la s

(1 9 7 5 a ,

1975b ,

1977a ,

1 9 7 7 b ) a n d W r ig h t

a n d M ea d (1 9 7 6 ). T h es e p r o ced u r es h a ve b een u s efu l in a w id e v a r ie ty o f a p p lica tio n s (C o n n o lly ,

N a ch tm a n

and

P r itc h e tt,

1971;

W ood cock ,

1974;

W illm o tt a n d

F o w le s ,

1 9 7 4 ; R e n tz a n d B a s h a w , 1 9 7 5 , 1 9 7 7 ; A n d r ic h , 1 9 7 5 ; M ea d , 1 9 7 5 ; W r ig h t a n d M ea d , 1 9 7 7 ; C orn is h a n d W in es , 1 9 7 7 ; D ra b a , 1 9 7 8 ; E llio tt, M u r r a y a n d P ea r s on , 1 9 7 7 .

_ _ __ _ __ _ __ _ _ __ _ __ _ _ __ _ _ |

T A B L E 1 .4 .1

| _______________________

P E R S O N A B I L I T Y A N D IT E M D I F F I C U L T Y IN L O G IT S A N D T H E R A S C H P R O B A B IL IT Y O F A R I G H T A N S W E R

P erso n

It e m

A b ility

D if f i c u l t y

Pv

*»

R ig h t A n s w e r D iff e r e n c e

In f o r m a t io n

Odds

P r o b a b ility

exp (/3j, - 5 t)

nvi

in a R espo n se

5

0

5

.9 9

.0 1

4

0

4

5 4 .6

.9 8

.0 2

3

0

3

2 0 .1

.9 5

.0 5

2

0

2

7 .3 9

.8 8

.1 1

1

0

1

2 .7 2

.7 3

.2 0

0

0

0

1 .0 0

.5 0

.2 5

.2 0

148.

0

1

-

1

0 .3 6 8

.2 7

0

2

-

2

0 .1 3 5

.1 2

.1 1

0

3

-

3

0 .0 5 0

.0 5

.0 5

0

4

-

4

0 .0 1 8

.0 2

.0 2

0

5

-

5

0 .0 0 7

.0 1

.0 1

*

v VL = e x p $ v - 5 t ) / [ 1 + e x p (0 „ - 5 t )]

K i= nv i^ ~ nw)

W e ca n s ee in E q u a tio n 1 .4 .1 th a t w h en p er s on v is s m a r ter th a n ite m i is d iffic u lt, th en j3„ is m o r e th a n 6 1, th e ir d iffe r e n c e is p o s itive a n d th e p r o b a b ility o f s u ccess o n ite m i is g r ea ter th a n o n e h a lf. T h e m o r e th e p er s o n ’s a b ility su rpa sses th e ite m ’s d iffic u lty , th e g r ea ter th is p o s itive d iffe r e n c e a n d th e n ea r er th e p r o b a b ility o f s u ccess c o m e s to on e. B u t w h en th e ite m is to o h a rd fo r th e p er s on , th en

is less th a n 6 t, th e ir d iffe r e n c e

is n eg a tive a n d th e p er s o n ’s p r o b a b ility o f s u ccess is less th a n o n e h a lf. T h e m o r e th e ite m o ve r w h e lm s th e p er s o n , th e g r ea ter th is n eg a tive d iffe r e n c e b ec o m es a n d th e n ea r er th e p r o b a b ility o f s u ccess c o m es to ze r o .

THE M EASUR EM ENT M ODEL

17

T h e m a th e m a tic a l u n its fo r pv a n d 6 t d e fin e d b y th is m o d e l a r e c a lle d “ lo g its . ” A p e r s o n ’s a b ility in lo g its is th e ir n a tu r a l lo g o d d s f o r s u c ce ed in g o n ite m s o f th e k in d c h o s en to d e fin e th e “ z e r o ” p o in t o n th e s ca le. A n d a n ite m ’s d iffic u lty in lo g its is its n a tu r a l l o g o d d s f o r e lic itin g fa ilu r e fr o m p er s o n s w ith “ z e r o ” a b ility .

T a b le 1 .4 .1 g ives e x a m p le s o f va r io u s p er s o n a b ilitie s a n d ite m d iffic u ltie s in lo g its , th e ir d iffe r e n c e s (/3„ -

5 1) a n d th e s u cces s p r o b a b ilitie s w h ic h res u lt. T h e fir s t s ix r o w s

illu s tr a te va r io u s p e r s o n a b ilitie s a n d th e ir s u cces s p r o b a b ilitie s w h e n p r o v o k e d b y ite m s o f z e r o d iffic u lty . T h e la s t s ix r o w s g ive e x a m p le s o f va r io u s ite m d iffic u ltie s a n d th e p r o b a b ilitie s o f s u cces s o n th e m b y p er s o n s w ith z e r o a b ility .

T h e o r ig in a n d s ca le o f th e lo g its u s ed in T a b le 1 .4 .1 a r e a r b itr a r y . W e ca n a d d a n y c o n s ta n t to a ll a b ilitie s a n d a ll d iffic u ltie s w ith o u t c h a n g in g th e d iffe r e n c e (0 „ - 6 ( ). T h is m ea n s th a t w e ca n p la c e th e z e r o p o in t o n th e s ca le s o th a t n e g a tive d iffic u ltie s a n d a b ili­ tie s d o n o t o c c u r . W e ca n a ls o in tr o d u c e a n y s ca lin g fa c to r w e fin d c o n v e n ie n t in c lu d in g o n e la r g e e n o u g h to e lim in a te a n y n e e d f o r d e c im a l fr a c tio n s . C h a p te r 8 in ves tig a tes th es e p o s s ib ilitie s in d e ta il.

T h e la s t c o lu m n o f T a b le 1 .4 .1 g ives th e r e la tive in fo r m a tio n I „t = -nVL( 1 a b le in a r e s p o n s e o b s e r ve d a t ea ch (j3„ -

-nv l )a v a il

5 t). W h en ite m d iffic u lty 5 t is w ith in a lo g it o f

p e r s o n a b ility j3„, th e in fo r m a tio n a b o u t e ith e r 6 L o r

in o n e o b s e r va tio n is g r e a te r th a n

.2 0 . B u t w h e n ite m d iffic u lty is m o r e th a n tw o lo g its o f f ta r g e t, th e in fo r m a tio n is les s a n d .1 1 a n d f o r

\PV -

6 t| >

3 less th a n

s a m p lin g a n d b e s t te s t d es ig n

.0 5 . T h e im p lic a tio n s f o r e ffic ie n t c a lib r a tio n

a re th a t res p on s es in th e

\fiv -

5 1|
2 a n d m o r e th a n fo u r tim e s as m u ch as th o s e o u ts id e o f |j3„ — 5 1| > 3.

1 .5

U S IN G T H E R A S C H M O D E L F O R C A L IB R A T IN G A N D M E A S U R IN G

W e h a ve e s ta b lis h e d th e n e e d f o r a n e x p lic it a p p r o a c h to m e a s u r e m e n t a n d s h o w n h ow

m e a s u r e m e n t p r o b le m s ca n b e a d d r es s ed w ith

a m o d e l fo r w h a t h a p p en s w h e n a

p e r s o n ta k e s a n ite m . N o w w e a r e r e a d y to w o r k th r o u g h th e m a th e m a tic s o f th is m o d e l in o r d e r t o fin d o u t h o w w e ca n u s e th e m o d e l to c a lib r a te ite m s a n d m ea s u r e p ers on s . T h e m o d e l s p e c ifie s th e p r o b a b ility o f p er s o n v w ith a b ility j3„ g ivin g r es p o n s e x „ t to ite m

i w ith d iffic u lty 5 1 as

P {x w

| j3 „ ,5 t } = e x p [ x „ t( / 3 „ - 8 t ) ] / [ 1

+ e x p ( 0 „ - 5 t) ]

[1 .5 .1 ]

T h e r e s p o n s e x vl ta k es o n ly t w o va lu es ,

x „( = 0 w h e n th e r es p o n s e is in c o r r e c t a n d x „ t = 1 w h e n th e r es p o n s e is c o r r e c t.

W h en w e in s e r t ea ch o f th es e va lu es o f Xj,t in to E q u a tio n 1 .5 .1 w e fin d th a t it b rea k s d o w n in to th e c o m p le m e n ta r y e x p r es s io n s

P { x l,l = 1

| 0 „ , 5 t } = e x p ( ( J „ - 6 t) / [ 1 + e x p ( 0 „ - 5 t)]

[1 .5 .2 ]

fo r a c o r r e c t r e s p o n s e a n d

P {x „t = 0 1 5 ( } =

1 / [ 1 + e x p ( fl„ - 5 t) ]

[1 .5 .3 ]

18

BEST T EST DESIGN

fo r an in c o r r e c t res p on s e. Th es e tw o ex p res s ion s a d d u p to o n e b eca u s e to g e th e r th e y c o v e r e ve r yth in g th a t ca n h a p p en to x „ t. W h en a s et o f L item s is a d m in is ter ed to a s a m p le o f N p ers on s , th e r es u lt is a n N b y L c o lle c tio n o f res p on s es ( ( x „ t) ) fo r w h ic h v = 1, N a n d i = 1, L a n d th e d o u b le p a ren th es es ( ( ) ) r em in d u s th a t a w h o le N b y L ta b le o f x „ ( ’s is im p lie d . T h e s e d a ta ca n b e d is p la yed in a m a tr ix lik e th e o n e s h o w n in F ig u r e 1 .5 .1 . T h e m a rg in a l s u m s o f th e r o w s a n d colu m s o f th is d a ta m a tr ix a re th e p er s on s cores L

v = 1 ,N

a n d th e ite m s cores

t=

J

1 .L

F I G U R E 1 .5 .1

D A T A M A T R IX O F O B S E R V E D R E S PO N S ES

Ite m s

P erso n S cores L

Persons

N

It e m S cores

N

2

xvl = s,

T HE M EASUR EM ENT M ODEL

19

W h a t h a p p en s w h e n w e a n a ly ze th es e d a ta as th o u g h th e y w e r e g o v e r n e d b y th e m o d e l o f E q u a tio n 1 .5 .1 ? A c c o r d in g to th a t m o d e l th e o n ly s ys te m a tic in flu e n c e s o n th e p er s o n a b ilitie s (P v ) a n d th e L ite m d iffic u ltie s ( 6 ( ) . A s

p r o d u c tio n o f th e x ^ ’s a r e th e N

a r es u lt, a p a r t fr o m th es e p a r a m eter s , th e x ^ ’s a r e m o d e le d t o b e q u ite in d e p e n d e n t o f o n e a n o th e r . T h is m ea n s th a t th e p r o b a b ility o f th e w h o le d a ta m a tr ix ( ( x „ t) ) , g ive n th e m o d e l a n d its p a r a m eter s ( 0 „) a n d ( 5 1), ca n b e e x p r e s s e d as th e p r o d u c t o f th e p r o b a b ili­ tie s o f ea ch s ep a r a te x v l g ive n b y E q u a tio n

1 .5 .1 c o n tin u e d o v e r a ll v =

1, N

a n d a ll

i = 1, L .

T h is c o n tin u e d p r o d u c t is i

± ;|e x p [x „ ( ( / 3 „ - 5 ,)]

P { < ( * „) > l ( 0 „u s , l ) . n

W h en

we

m ove

th e

)

|

c o n tin u e d

.

p r o d u c t o p e r a to r s

[1.5.4]

N

L

n

and n

v

i

in th e n u m e r a to r o f

E q u a tio n 1 .5 .4 in to th e e x p o n e n tia l e x p r e s s io n

exp [ x w

6 t> ] ,

th e y b e c o m e th e s u m m a tio n o p e r a to r s N

L

2

and 2

N

L

n

n

v

i

N

L

s o th a t f" N

e x p [xv l(Pv ~ 6 t)] = exp

2

L

v

L

"1

2 x „ t ( 0 „ - 5 t) .

i

J

T h e n , s in ce

and N

L

2

2 Xm St = 2 st 5 t,

V

I

L

I

E q u a tio n 1 .5 .4 b e c o m e s exp

Oh*

n n [1 + ( e x p ( ^ -

v i

E q u a tio n

[1 .5 .5 ]

)J

1 .5 .5 is im p o r ta n t b eca u s e it s h o w s th a t in o r d e r to e s tim a te th e p a r a ­

m e te r s (/3j,) a n d ( 5 t), w e n e e d o n ly th e m a r g in a l s u m s o f th e d a ta m a tr ix , ( r „) a n d ( ^ ) . T h is is b eca u s e th a t is th e o n ly w a y th e d a ta ( ( x „ t) ) a p p ea r in E q u a tio n 1 .5 .5 . T h u s th e p e r s o n s co r es (r , ,) a n d ite m s co r es (s t) c o n ta in a ll th e m o d e lle d in fo r m a tio n a b o u t p er s on m ea s u r es a n d ite m c a lib r a tio n s . F in a lly , th e n u m e r a to r o f E q u a tio n 1 .5 .5 ca n b e fa c to r e d in to tw o p a rts s o th a t th e m o d e l p r o b a b ility o f th e d a ta m a tr ix b e c o m e s

20

BEST T EST DESIGN

N

[e x p (

L

2 r„ 0„f)

[e x p ( -

2 st 5t)]

P { ( ( x w ) ) |( /3 „ ) ,( 6 t ) } = ------ ------------------------------ '---------IIII

i

[1 .5 .6 ]

[1 + e x p ( 0 „ - 8 , ) ]

E q u a tio n 1 .5 .6 is im p o r ta n t b eca u s e it s h ow s th a t th e p er s on a n d ite m p a ra m eter s ca n b e e s tim a ted in d e p e n d e n tly o f o n e a n o th er . T h e s ep a r a tion o f

( 2 r„ v

pj

and L

( 2 st 5t) i

in E q u a tio n 1 .5 .6 m a k es it p os s ib le to c o n d itio n e ith e r s e t o f p a r a m eter s o u t o f E q u a tio n 1 .5 .6

w h en e s tim a tin g th e o th e r s et. T h is m ea n s , in th e la n gu a g e o f s ta tis tics , th a t th e

s cores ( r „) a n d (s t) a re s u ffic ie n t fo r e s tim a tin g th e p er s on m ea s u res a n d th e ite m c a li­ b ra tion s .

B eca u s e o f th is w e ca n u se th e p er s on s cores ( r „) to r e m o ve th e p er s on p a ra m eter s

( P „ ) fr o m E q u a tio n 1 .5 .6 w h en c a lib r a tin g item s . T h is fr ee s th e ite m ca lib r a tion s fr o m th e m o d e lle d ch a r a cter is tics o f th e p ers on s a n d in th is w a y p r o d u ce s s a m p le- fr ee ite m ca li­ b r a tion s . A s fo r m ea s u rin g p ers on s , w e c o u ld u se th e ite m s cores (s ( ) to r e m o ve th e ite m p a r a m eter s ( 8 t) fr o m

E q u a tio n 1 .5 .6 . W h en w e c o m e to p er s on m ea s u r em en t, h o w e ve r ,

w e w ill fin d it m o r e c o n ve n ie n t to w o r k d ir e c tly fr o m th e e s tim a te d ite m ca lib r a tion s

( d t ). T h e r e a re s ever a l w a ys th a t E q u a tio n 1 .5 .6 ca n b e u s ed to es tim a te va lu es fo r pv a n d 5 t. T h e id ea l w a y is to u se th e s u ffic ie n t s ta tis tics fo r p er s on s (r ,,) to c o n d itio n p er s on p a ra m eter s ( 0 „) o u t o f th e e q u a tio n . T h is lea ves a c o n d itio n a l lik lih o o d in vo lv in g o n ly th e ite m p a r a m eter s ( 6 t) a n d th e y ca n b e e s tim a ted fr o m th is c o n d itio n a l lik lih o o d (F is c h e r a n d S ch eib lech n er , 1 9 7 0 ; A n d er s e n , 1 9 7 2 a , 1 9 7 2 b ; W r ig h t a n d D ou g la s , 1 9 7 5 b , 1 9 7 7 b ; A lle r u p a n d S o r b er , 1 9 7 7 ; G u s ta fs s on , 1 9 7 7 ).

B u t th is id ea l m e th o d is im p r a c tic a l a n d u n n eces s a ry. C o m p u tin g tim es a re ex ces s ive. R o u n d - o ff er r or s lim it a p p lic a tio n to tes ts o f f ift y ite m s a t m o s t. A n d , in a n y ca s e, res u lts a re n u m e r ic a lly e q u iva le n t to th o s e o f q u ic k e r a n d m o r e r o b u s t m e th o d s . A c o n ve n ie n t a n d p r a ctica l a lter n a tive is to u se E q u a tio n 1 .5 .6 as it s ta n d s. T o lea r n m o r e a b o u t th is u n c o n d itio n a l

e s tim a tio n

of

ite m

p a ra m eter s

s ee W r ig h t a n d

P a n ch a p a k es a n

(1 9 6 9 ),

W r ig h t a n d D ou g la s (1 9 7 5 b , 1 9 7 7 a , 1 9 7 7 b ) a n d C h a p ter 3 , S e c tio n 3 .4 . o f th is b o o k .

E ven th is u n c o n d itio n a l m e th o d , h o w e ve r , is o fte n u n n eces s a rily d e ta ile d a n d c o s tly fo r p r a ctica l w o r k . I f th e p er s on s w e u se to ca lib r a te item s a re n o t t o o u n s ym m e tr ica lly d is tr ib u te d in a b ility a n d n o t to o fa r o f f ta r g e t s o th a t th e im p a c t o f th e ir a b ility d is tr ib u ­ tio n ca n b e m o r e o r less s u m m a r ized b y its m ea n a n d va r ia n ce, th en w e ca n u se a ve r y s im p le a n d w o r k a b le m e th o d fo r e s tim a tin g ite m d iffic u ltie s . T h is m e th o d , ca lle d P R O X , was

fir s t s u gges ted b y L e s lie C oh en in 1 9 7 3 (s ee W r ig h t a n d D ou g la s , 1 9 7 7 a ; W r ig h t,

1 9 7 7 ).

21

THE M EASUR EM ENT M ODEL

1 .6

A S IM P L E U S E F U L E S T IM A T IO N P R O C E D U R E

T h r e e m e th o d s o f p a r a m e te r e s tim a tio n w ill b e u s ed in th is b o o k . T h e g en er a l u n ­ c o n d itio n a l m e th o d c a lle d U C O N r e q u ir e s a c o m p u te r a n d a c o m p u te r p r o g r a m s u ch as B IC A L

(W r ig h t a n d

M e a d , 1 9 7 6 ). U C O N

is d is cu s s ed a n d illu s tr a te d in C h a p te r 3 . A

s e c o n d m e th o d c a lle d U F O R M , w h ic h ca n b e d o n e b y h a n d w ith th e h e lp o f th e s im p le ta b le s g ive n in A p p e n d ix

C , is d is cu s s ed a n d a p p lie d in C h a p te r 7. T h e th ir d m e th o d ,

P R O X , is c o m p le te ly m a n a g e a b le b y h a n d . In a d d itio n th e s im p lic ity o f P R O X h elp s u s to s ee h o w

th e R a s ch m o d e l w o r k s to s o lve m e a s u r e m e n t p r o b le m s . T h e d e r iva tio n s o f

th e U F O R M a n d P R O X e q u a tio n s a r e g ive n in W r ig h t a n d D o u g la s (1 9 7 5 a , 1 9 7 5 b ).

P R O X a s s u m es th a t p e r s o n a b ilitie s ( 0 „ ) a r e m o r e o r les s n o r m a lly d is tr ib u te d w ith m ea n M

a n d s ta n d a r d d e v ia tio n

a a n d th a t ite m d iffic u ltie s ( 5 t) a r e a ls o m o r e o r les s

n o r m a lly d is tr ib u te d w ith a ve r a g e d iffic u lty H a n d d iffic u lty s ta n d a r d d e v ia tio n co.

If

~ N

(M , a )

and

5t ~

N ( H , co2 ),

th e n f o r a n y p e r s o n v w ith p e r s o n s c o r e r „ o n a te s t o f L ite m s it fo llo w s th a t

b j, = H + X f in [ r j , / ( L -

r„ )]

[ 1 .6 .1 ]

a n d f o r a n y ite m i w ith ite m s co r es st in a s a m p le o f N p er s o n s it fo llo w s th a t

d t = M + Y 8 n [ ( N - st ) / s t ]

[ 1 .6 .2 ]

T h e c o e ffic ie n ts X a n d Y a r e e x p a n s io n fa c to r s w h ic h r e s p o n d in th e c l ,e o f X to th e d iffic u l ty d is p e r s io n o f ite m s a n d in th e ca s e o f Y to th e a b ility d is p er s io n o f p er s on s . In p a r tic u la r

X = ( 1 + w 2 / 2 . 8 9 ) 1/2

[1 .6 .3 ]

and

Y =

(1

+ o 2/ 2.89)Vx

T h e va lu e 2 .8 9 =

[1 .6 .4 ]

1 .7 2 c o m e s fr o m th e s ca lin g fa c to r 1 .7 w h ic h b rin g s th e lo g is tic

o g iv e in to a p p r o x im a te c o in c id e n c e w ith

th e n o r m a l o g iv e . T h is is b eca u s e th e lo g is tic

o g iv e f o r va lu es o f 1 .7 z is n e v e r m o r e th a n o n e p e r c e n t d iffe r e n t fr o m th e n o r m a l o g iv e f o r va lu e s o f z.

T h e e s tim a tes b „ a n d d t h a ve s ta n d a r d er r o r s

S E ( b „ ) = X [ L / r „ ( L - r „ ) ] Vx

[1 -6 .5 ]

S E ( d t ) = Y [ N / s t ( N - st ) ] 1/a

[ 1 . 6 .6 ]

22

BEST T EST D ESIGN

T h is e s tim a tio n m e th o d ca n b e a p p lied d ir e c tly to o b s er ved ite m s cores (s t) b y ca lcu ­ la tin g th e s a m p le s cor e lo g it o f ite m t as

x t = 8n [ ( N - st ) / s ( ]

[ 1 .6 .7 1

a n d th e ite m s cor e lo g it o f p er s on v as

y „ = Cn [ r „ / ( L - r „ ) ]

[1 .6 .8 ]

T h e ex p a n s io n fa c to r s X a n d Y a re th en e s tim a ted b y th e ex p r es s ion s

X = [{1 + U / 2 . 8 9 ) / ( 1 - U V / 8 . 3 5 ) ] 14

,

[1 .6 .9 ]

fo r th e p er s on lo g it ex p a n s io n fa c to r a n d

Y = [( 1 + V / 2 . 8 9 ) / ( 1 - U V / 8 . 3 5 ) ] * ,

[1 .6 .1 0 ]

fo r th e ite m lo g it ex p a n s io n fa c to r .

In th es e ex p r es s io n s 2 .8 9 = 1 .7 2 a n d 8 .3 5 = 2 .8 9 2 = 1 .7 4 a n d

U = ( 2 x ,2 -

I

1

L x .2 ) / ( L - 1 ),

[1 .6 .1 1 ]

th e ite m lo g it va r ia n ce a n d N

V = ( 2 y „ 2 -

N y .2 ) / ( N - 1 ),

[1 .6 .1 2 ]

0 th e p er s on lo g it va ria n ce. T o c o m p le te th is e s tim a tio n , w e s et th e te s t c e n te r a t ze r o s o th a t H = 0 . Th en

d t = M + Y x t = Y ( x t - x .)

[1 .6 .1 3 ]

fo r ea ch ite m d iffic u lty , a n d

b„ = H + X y „ = X y „

[1 .6 .1 4 ]

fo r ea ch p er s on a b ility .

S ta n d a r d er r or s a re

S E ( d t ) = Y [ N / s t {N - st ) ] 14 “

2 . 5 / N 54

[1 .6 .1 5 ]

and S E ( b v ) = X t L / r ^ L - rv ) ] ' A =

2 . 5 / L 54

[1 .6 .1 6 ]

F in a lly th e e s tim a ted p er s on s a m p le m ea n a n d s ta n d a rd d e via tio n b e c o m e

M

«

- Yx .

1 .7 { Y 2 -

[1 .6 .1 7 ]

1 )^

[1 .6 .1 8 ]

THE M EASUR EM ENT M ODEL

23

O n c e w e h a ve e s tim a te d b „ a n d d t w e ca n u s e th e m tw e e n

to o b ta in th e d iffe r e n c e b e ­

w h a t th e m o d e l p r e d ic ts a n d th e d a ta w e h a ve a c tu a lly o b s e r ve d . T h e s e res id u a ls

fr o m th e m o d e l a r e c a lc u la te d b y e s tim a tin g th e m o d e l e x p e c ta tio n s a t ea ch x „ t fr o m b y a n d d t a n d s u b tr a c tin g th is e x p e c ta tio n

fr o m

th e x „ t w h ic h w a s o b s e r ve d . T h e m o d e l

e x p e c ta tio n f o r x v i is

17VI

^ ( XW } =

w ith m o d e l va r ia n c e

V { * v i } = v vi ^ ~ 71vJ w h ere

v vl = e x p (Pv - 5 t ) / [ 1 + e x p ( 0 y - 6 t ) ]

.

A s ta n d a r d ize d r es id u a l w o u ld b e

(X w - i r w ) /t ir w ( 1 -

w ^ ) ] 54

.

[ 1 . 6 .1 9 1

I f th e d a ta f i t th e m o d e l th is s ta n d a r d ize d r es id u a l o u g h t t o b e d is tr ib u te d m o r e o r less n o r m a lly w ith m ea n z e r o a n d va r ia n c e o n e .

I f w e e s tim a te v Vl fr o m p v l w h e r e

pv i = e x p ( b v - d ( ) / [ 1 + e x p (b y - d t ) ]

[1 .6 .2 0 ]

th e n w e ca n u s e th e e r r o r d is tr ib u tio n s

Zyt ~

N (0,1)

7

z lH

and

2

Xi

as g u id e lin e s f o r e va lu a tin g th e e x te n t t o w h ic h a n y p a r tic u la r s e t o f d a ta ca n b e m a n a g ed b y o u r m e a s u r e m e n t m o d e l.'

W e ca n c a lc u la te th e s u m o f th e ir s q u a r ed res id u a ls zv l2 f o r ea ch p er s o n . A c c o r d in g to th e m o d e l th is s u m o f s q u a r ed n o r m a l d e via te s s h o u ld a p p r o x im a te b u tio n w ith a b o u t ( L -

a ch i- s q u a re d is tr i­

1 ) d eg r e e s o f fr e e d o m . T h is g ives u s a c h i- s q u a r e s ta tis tic

[ 1 .6.2 11

w ith d e g r e e s o f fr e e d o m

fy = (L -

1 )(N -

1 )/N

[1 .6 .2 2 ]

a n d a m e a n s q u a r e s ta tis tic

vy = C l / fy ~ F y °o

[1.6.23]

24

BEST T EST D ESIGN

w h ic h a p p r o x im a te s a n F - d is tr ib u tio n w h en th e p e r s o n ’s res p on s es f i t th e m o d e l.

T h e s u m o f s qu a red res id u a ls fo r ea ch ite m ca n b e u s ed in th e s a m e w a y to eva lu a te ite m fit. F o r item s

f

Z w = C t 2 ~ X f2i

[1 .6 .2 4 ]

ft = (N — 1 ) ( L — 1 ) / L

[1 .6 .2 5 ]

v i = c i2 / f i ~ F f t < °°.

[1 .6 .2 6 ]

w ith

and

F in a lly , s in ce x „ ( ca n o n ly e q u a l o n e o r ze r o , w e ca n u se th e d e fin itio n o f p „t g iven in E q u a tio n 1 .6 .2 0 to ca lcu la te z„, Vl a n d z „t2 2 d ir e c tly as

zw = (2 x w -

1) exp [(2 x w -

1 )(d t - b „ )/2 ]

[1 .6 .2 7 ]

and

z^ t = e x p [ ( 2 x „ , - 1 ) ( d t - b ^ , ) ]

[1 .6 .2 8 ]

T h is r e la tio n ca n a ls o b e w o r k e d b a ck w a r d s . I f w e ca lcu la te th e

a lr ea d y h a ve a z „t2 a n d w is h to

p r o b a b ility o f th e o b s er ved r es p on s e x „ t to w h ic h it r e fe r s in o r d e r to d e c id e

w h e th e r o r n o t th a t r es p on s e is to o im p r o b a b le to b e lie ve , th en w e ca n u se

P { x w I b „ ,d t } = 1 /( 1 + z „ t 2 ).

In

c o n tr a s t w ith

th e p ul o f E q u a tio n

[1 .6 .2 9 ]

1 .6 .2 0 w h ic h is th e e s tim a ted p r o b a b ility o f a

c o r r e c t a n s w er, th e p r o b a b ility o f E q u a tio n 1 .6 .2 9 a p p lies to x „ t w h a te ve r va lu e it ta k es , w h e th e r x „ t = 1 fo r a c o r r e c t a n s w er o r x „ t = 0 fo r a n in c o r r e c t on e.

1 .7

H O W T R A D I T IO N A L T E S T S T A T IS T IC S A P P E A R IN R A S C H M E A S U R E M E N T

S ec tio n s 1 .1 , 1 .2 a n d 1 .3 d is cu s s th e p u r p o s e o f tes ts , th e u se o f te s t s cores a n d th e p r o b le m s o f g e n e r a lity a n d lin e a r ity in m a k in g m ea s u res . S ec tio n s 1 .4 , 1 .5 a n d 1 .6 d es ­ c r ib e a s im p le a n d p r a ctica l s o lu tio n to th es e m ea s u r em en t p r o b lem s . B eca u s e th e m a th ­ em a tics a re n e w it m ig h t s eem th a t u s in g th e R a s ch m o d e l w ill ta k e u s fa r a w a y fr o m th e tr a d itio n a l ite m s ta tis tics w ith w h ic h w e a re fa m ilia r . T h is is n o t s o.

A p p ly in g th e

R a s ch m o d e l in te s t d e v e lo p m e n t g ives u s n e w ver s ion s o f th e o ld

s ta tis tics . T h e s e n e w s ta tis tics c o n ta in a ll o f th e o ld fa m ilia r in fo r m a tio n , b u t in a fo r m w h ic h

s olves

m o s t o f th e m ea s u r em en t p r o b le m s

th a t h a ve a lw a ys

b es et tr a d itio n a l

te s t c o n s tr u c tio n . T o s h o w th is w e w ill e x a m in e th e th r ee m o s t c o m m o n tr a d itio n a l ite m a n d p er s on s ta tis tics a n d s ee h o w c lo s e ly th e y r e la te to th e ir c o r r es p o n d in g R a s ch m ea ­ s u r em en t s ta tis tics .

THE M EASUR EM ENT M ODEL

25

T h e I t e m P - V a lu e

T h e m o s t fa m ilia r tr a d itio n a l ite m s ta tis tic is th e ite m “ p - va lu e .” T h is is th e p r o p o r ­ tio n

o f p er s o n s in a s p e c ifie d

e q u a tio n

s a m p le w h o g e t th a t ite m c o r r e c t. T h e P R O X e s tim a tio n

( 1 . 6 . 2 ) g ives u s a c o n v e n ie n t w a y to

fo r m u la te th e r e la tio n s h ip

b e tw e e n

th e

tr a d itio n a l ite m p - va lu e a n d R a s ch ite m d iffic u lty . I f th e p - va lu e fo r ite m i is e x p r es s ed as

P t = s( / N

in w h ic h st is th e n u m b e r o f p er s o n s in th e s a m p le o f N p er s o n s w h o a n s w er ed ite m i c o r r e c tly , th e n th e P R O X e s tim a te d R a s ch ite m d iffic u lty is

d t = M + (1 + o 2 / 2 . 8 9 ) ’/ ’ Cn [ ( 1 - P l) / p t]

E q u a tio n

[1 .7 .1 ]

1 .7 .1 s h o w s th a t th e R a s ch ite m d iffic u lty d t is in a o n e - to - o n e r e la tio n

w ith th e ite m p - va lu e r e p r e s e n te d b y p( . I t a ls o s h o w s th a t th is o n e - to - o n e r e la tio n is c u r vi­ lin e a r a n d in v o lv e s th e a b ility m ea n M a n d va r ia n c e a 2 o f th e c a lib r a tin g s a m p le.

W h a t th e R a s ch m o d e l d o e s is to u s e th e lo g it fu n c tio n

Cn [ ( 1 - P t ) / P t l

to tr a n s fo r m th e ite m p - va lu e w h ic h is n o t lin e a r in th e im p lie d va r ia b le in to a n e w va lu e w h ic h

is. T h is n e w

lo g it va lu e ex p r es s es th e ite m

a n d m a k es th e s u b s e q u e n t c o r r e c tio n

d iffic u lty o n a n e q u a l in te r va l s ca le

o f th e ite m ’s p - va lu e f o r th e a b ility m ea n M a n d

va r ia n c e a 2 o f th e c a lib r a tin g s a m p le e a s y to a c c o m p lis h .

T h is c o r r e c tio n is m a d e b y s ca lin g th e l o g it t o r e m o v e th e e ffe c ts o f s a m p le va r ia n ce a 2 a n d tr a n s la tin g th is s ca led lo g it to r e m o v e th e e ffe c ts o f s a m p le m e a n M . T h e r es u ltin g R a s ch ite m of

d iffic u ltie s a r e n o t o n ly o n a n e q u a l in te r va l s ca le b u t th e y a r e a ls o fr e e d

th e o b s e r ve d a b ility m ea n a n d va r ia n c e o f th e c a lib r a tin g s a m p le. Ju s t as th e ite m

p - va lu e P ( h a s a b in o m ia l s ta n d a r d e r r o r o f

S E ( p t) = [ p t (1 -

p [ ) / N ] 'A

[1 .7 .2 ]

s o th e P R O X ite m d iffic u lty d t h a s its o w n c lo s e ly r e la te d s ta n d a r d e r r o r o f

S E ( d ( ) = (1 + ct 2 / 2 . 8 9 ) ' /2 [ 1 / N p t (1 -

pt) ] *

.

[1 .7 .3 ]

B u t th e r e a r e t w o im p o r ta n t d iffe r e n c e s b e tw e e n E q u a tio n s 1 .7 .2 a n d 1 .7 .3 . U n lik e th e p - va lu e s ta n d a r d e r r o r in E q u a tio n 1 .7 .2 , th e R a s ch s ta n d a r d e r r o r in E q u a tio n 1 .7 .3 is c o r r e c te d

f o r th e a b ility va r ia n c e a 2 o f th e c a lib r a tin g s a m p le. T h e s ec o n d d iffe r e n c e

b e tw e e n th es e t w o fo r m u la tio n s is m o r e s u b tle , b u t e ve n m o r e im p o r ta n t.

T h e tr a d itio n a l ite m p - va lu e s ta n d a r d er r o r s in E q u a tio n 1 .7 .2 a r e m a x im u m in th e m id d le a t p - va lu es n ea r o n e - h a lf a n d z e r o a t th e e x tr e m e s a t p - va lu es o f ze r o o r o n e. T h is m a k es it a p p e a r th a t w e k n o w th e m o s t a b o u t a n ite m , th a t is h a ve th e s m a lles t e r r o r f o r its

s ta n d a rd

p - va lu e w h e n , in fa c t, w e a c tu a lly k n o w th e lea s t. T h is is b eca u s e th e ite m

p - va lu e is fo c u s e d

o n th e c a lib r a tin g s a m p le as w e ll as o n th e ite m . A s th e s a m p le g oes

o f f ta r g e t f o r th e ite m , th e ite m

p - va lu e n ea rs z e r o o r o n e a n d its s ta n d a rd e r r o r n ea rs

z e r o . T h is a s s u res u s th a t th e ite m

p - va lu e f o r th is p a r ticu la r s a m p le is e x tr e m e b u t it

BEST T EST D E SIGN

26

tells u s n o th in g els e a b o u t th e ite m . T h u s even th o u g h o u r k n o w le d g e o f th e ite m ’s pva lu e is in cr ea s in g o u r in fo r m a tio n c o n c e r n in g th e a ctu a l d iffic u lty o f th e ite m is d e ­ crea s in g. W h en ite m p - va lu es a re ze r o o r o n e, th e ca lib r a tin g s a m p le w h ic h w a s in ten d ed to te ll u s h o w th a t ite m w o r k s is s h o w n to b e to o a b le o r t o o u n a b le to in te r a c t w ith th e item . W e k n o w e x a c tly in w h ic h d ir e c tio n to lo o k fo r th e ite m d iffic u lty , b u t w e h a ve n o in fo r m a tio n as to w h e r e in th a t d ir e c tio n it m ig h t b e.

In con tr a s t, th e R a s ch s ta n d a rd e r r o r fo r d t va ries in a m o r e r ea s on a b le m a n n er. T h e ex p r es s io n p t (1 - p t) w h ic h g oes to ze r o as p t g oes to ze r o o r o n e , a p p ea r s in th e d e n o m i­ n a to r o f E q u a tio n 1 .7 .3 in s tea d o f in th e n u m er a to r , as it d o e s in E q u a tio n 1 .7 .2 . T h e r e ­ fo r e , th e R a s ch s ta n d a rd e r r o r is s m a lles t a t p t = .5 , w h er e th e s a m p le is c en te r e d o n th e ite m

a n d th u s gives

us th e m o s t in fo r m a tio n a b o u t h o w th a t ite m

fu n ctio n s . A t th e

ex tr e m es , h o w e ve r , w h e r e w e h a ve th e lea s t in fo r m a tio n , th e R a s ch s ta n d a rd e r r o r g o es to in fin ity

r e m in d in g us th a t w e h a ve lea r n ed a lm o s t n o th in g a b o u t th a t ite m

fr o m

th is

s a m p le.

T h e It e m P o in t-B is e ria l

T h e s eco n d m o s t w id e ly u s ed tr a d itio n a l ite m b e tw e e n

th e s a m p led

p er s on s ’ d ic h o to m o u s

s ta tis tic is th e p o in t b is eria l c o r r e la tio n

res p on s es to an ite m

a n d th e ir to ta l tes t

s cores . T h e ite m p oin t- b is er ia l h a s tw o ch a r a cter is tics w h ic h in te r fe r e w ith its u s efu ln es s as a n in d e x o f h o w w e ll a n ite m fits w ith th e s et o f ite m s in w h ic h it a p p ea rs . F ir s t, th er e is n o clea r b asis fo r

d e te r m in in g w h a t m a g n itu d e ite m p oin t- b is er ia l es ta b lis h es ite m

a c c e p ta b ility. R e je c tin g th e s ta tis tica l h yp o th es is th a t a n ite m p oin t- b is er ia l is ze r o d o es n o t p r o d u c e a s a tis fa c to r y s ta tis tica l c r ite r io n fo r va lid a tin g a n ite m . T h e s eco n d in te r ­ fe r in g ch a r a cter is tic is th a t th e m a g n itu d e o f th e p oin t- b is er ia l is s u b s ta n tia lly in flu e n c e d b y th e s core d is tr ib u tio n o f th e ca lib r a tin g s a m p le. A g iven ite m ’s p o in t- b is er ia l is 'la r g es t w h e n th e p ers on s in th e s a m p le a re s p rea d o u t in s cores a n d c en te r e d o n th a t ite m . C o n ­ ver s e ly as th e va r ia n ce in p er s on s cores d ecrea s es o r th e s a m p le le ve l m o ves a w a y fr o m th e ite m le ve l, s o th a t th e p - va lu e a p p r oa ch es ze r o o r o n e , th e p oin t- b is er ia l d ecrea s es to ze r o rega r d les s o f th e q u a lity o f th e ite m .

Th e

R a s ch

s ta tis tic th a t cor r es p on d s in m ea n in g to th e ite m p oin t- b is er ia l is th e

ite m ’s m ea n s qu a re res id u a l g iven in E q u a tio n 1 .6 .2 6 . T h is m ea n s qu a re res id u a l is n o t o n ly s en s itive to item s w h ic h fa il to c o r r e la te w ith th e te s t s cor e, b u t a ls o to ite m p o in tb is eria ls w h ich a re u n e x p e c te d ly la r ge. T h is h a p p en s , fo r e x a m p le , w h en a n a d d itio n a l a n d u n m o d e lle d va r ia b le p r o d u ces a lo c a l in te r a c tio n b e tw e e n a u n iq u e fea tu r e o f th e ite m

in

q u es tio n

a n d a c o r r es p o n d in g id io s yn cr a s y a m o n g s om e m em b er s o f th e ca li­

b r a tin g s a m p le.

In c o n tr a s t w ith th e p oin t- b is er ia l, th e R a s ch ite m m ea n s qu a re res id u a l h a s a u s efu l s ta tis tica l r e fe r e n c e d is tr ib u tio n . T h e r e fe r e n c e va lu e fo r te s tin g th e s ta tis tica l h yp o th es is th a t a n ite m b elo n g s in th e te s t is a m ea n s qu a re o f o n e w ith a s ta n d a rd e r r o r o f ( 2 / f) 54 fo r f d eg r ees o f fr e e d o m . Th u s th e e x te n t to w h ic h a n o b s e r ve d m ea n s qu a re e x c e e d s th e e x p e c te d va lu e o f o n e ca n b e tes ted fo r its s ta tis tica l s ig n ifica n ce a t w h a te ve r s ig n ifica n ce le ve l is c on s id er ed u s efu l.

T h e R a s ch ite m m ea n s qu a re is a ls o v e r y n ea r ly in d iffe r e n t to th e a b ility d is tr ib u tio n o f th e c a lib r a tin g s a m p le. T h is p r o vid es a te s t o f ite m f it w h ich is fo c u s e d o n ju s t th os e s a m p le a n d ite m

ch a ra cteris tics w h ic h rem a in w h en th e m o d e lle d va lu es fo r ite m d iff i­

c u lty a n d p er s on a b ilities a re r e m o ve d .

THE M EASUR EM ENT M ODEL

27

T h e P erso n T e s t S c o re

T h e m o s t fa m ilia r tr a d itio n a l p er s o n s ta tis tic is te s t s c o r e , th e n u m b e r o f c o r r e c t a n s w er s th e p e r s o n ea rn s o n th e te s t ta k e n . O n c e a g a in w e ca n u s e th e P R O X e s tim a tio n p roced u re

to s h o w

a b ility . U s in g th e

th e c o n n e c tio n b e tw e e n th e tr a d itio n a l te s t s co r e a n d R a s ch p er s o n

P R O X e s tim a tio n e q u a tio n ( 1 . 6 . 1 ) w e h a ve

b „ = H + (1 + w 2 / 2 . 8 9 ) * in [ r „ / ( L - r „ ) ]

[1 .7 .4 ]

w ith a s ta n d a r d e r r o r o f

S E ( b j ,) = (1 + w 2 / 2 . 8 9 ) *

[ L / r „ { 1 - r „ ) ] 14

[ 1 .7 .5 ]

in w h ic h

rv

=

th e te s t s c o r e o f p e r s o n v,

=

th e a ve r a g e d iffic u lty le v e l o f th e te s t a n d

L H

=

th e n u m b e r o f ite m s in th e te s t,

th e va r ia n c e in d iffic u ltie s o f th e te s t item s .

A s w ith th e ite m p - va lu es w e s ee th e lo g it fu n c tio n tr a n s fo r m in g th e p er s o n s cores w h ic h

a r e n o t lin e a r in th e va r ia b le th e y im p ly in to a n a p p r o x im a te ly lin e a r m e tr ic . W e

a ls o s ee th is l o g it b e in g s ca led f o r te s t w id th , w h ic h is r e p r e s e n te d in E q u a tio n 1 .7 .4 b y th e ite m d iffic u lty va r ia n c e oj 2 , a n d th e n b e in g s h ifte d to a d ju s t f o r te s t d iffic u lty le ve l H s o th a t th e r e s u ltin g e s tim a te d p e r s o n a b ility is fr e e d fr o m th e lo c a l e ffe c ts o f th e te s t a n d b e c o m e s a te s t- fr e e m ea s u r e.

T h e s ta n d a r d e r r o r o f th is m ea s u r e is m in im u m a t s co r es n ea r 5 0 p e r c e n t c o r r e c t, w h e r e w e h a ve th e m o s t in fo r m a tio n a b o u t th e p e r s o n , a n d g o es to in fin ity a t s cor es o f z e r o a n d 1 0 0 p e r c e n t, w h e r e w e h a ve th e le a s t in fo r m a tio n a b o u t th e p er s o n .

W h ile tr a d itio n a l te s t p r a c tic e s a lm o s t a lw a y s e m p h a s ize th e a n a lys is o f ite m v a lid ity , h a r d ly a n y a tte n tio n is e v e r g ive n t o th e v a lid ity o f th e p a tte r n o f r es p on s es le a d in g to a p e r s o n s co r e . A s fa r as w e k n o w n o o n e c a lc u la tes a p er s o n p o in t- b is e r ia l c o e ffic ie n t in ord er

to d e te r m in e th e r e la tio n s h ip b e tw e e n th e r es p on s es th a t p er s o n g ives to ea ch ite m

a n d th e s u p p o s e d ly r e le v a n t ite m p - va lu es . T h is w o u ld b e a r e a s o n a b le w a y to a p p ly th e tr a d itio n a l p o in t- b is e r ia l c o r r e la tio n c o e ffic ie n t t o th e s u p er vis io n o f p er s o n s co r e v a lid ity .

T h e R a s ch a p p r o a c h to p e r s o n s c o r e v a lid ity is o u tlin e d in E q u a tio n s 1 .6 .1 9 th r o u g h 1 .6 .2 3 a n d d is cu s s ed a n d illu s tr a te d a t le n g th in C h a p ter s 4 a n d 7 .

Th ere

a r e o th e r

c o n n e c tio n s th a t ca n b e m a d e b e tw e e n tr a d itio n a l te s t s ta tis tics

a n d R a s ch s ta tis tics . W e c o u ld r e v ie w h e r e th e va r io u s w a y s th a t tr a d itio n a l te s t r e lia b ility a n d v a lid ity , n o r m r e fe r e n c in g , c r ite r io n r e fe r e n c in g , fo r m e q u a tin g a n d m a s te r y tes tin g a r e h a n d le d in R a s ch m e a s u r e m e n t. B u t ea ch o f th es e to p ic s d es er ves a th o r o u g h d is ­ cu s s io n a n d th a t, in fa c t, is th e p u r p o s e o f th e c h a p ter s w h ic h fo llo w . O u r n e x t s tep n o w is to s ee h o w c o n s tr u c tio n .

th e P R O X

e s tim a tio n p r o c e d u r e w o r k s to s o lve a s im p le p r o b le m in tes t

2

2 .1

I T E M C A L I B R A T IO N B Y H A N D

IN T R O D U C T IO N

T h is c h a p te r d es cr ib es a n d illu s tr a tes in d e ta il a n e x tr e m e ly s im p le p r o c e d u r e fo r th e R a s ch c a lib r a tio n o f tes t item s . T h e p r o ce d u r e , c a lle d P R O X , a p p r o x im a te s th e res u lts o b ta in e d

b y m ore

e la b o r a te a n d

h en ce m o r e a ccu r a te p r o ced u r es

e x tr e m e ly

w e ll. It

a ch ieves th e b a s ic a im s o f R a s ch ite m a n a lys is , n a m e ly lin e a r iza tio n o f th e la te n t s ca le a n d a d ju s tm en t fo r

th e lo c a l e ffe c ts o f s a m p le a b ility d is tr ib u tio n . T h e a s s u m p tion w h ic h

m a k es P R O X s im p le is th a t th e e ffe c ts o n ite m c a lib r a tio n o f s a m p le a b ility d is tr ib u tio n ca n b e a d e q u a te ly a c c o u n te d fo r b y ju s t a m ea n a n d s ta n d a rd d e via tio n . T h is a s s u m p tion m a k es P R O X s o s im p le th a t it ca n ea s ily b e a p p lied b y h a n d . In p r a c tic e, it w ill o fte n b e c o n v e n ie n t to le t ite m c a lib r a tio n b e d o n e b y c o m p u te r . H o w e ve r , d eta il,

PRO X

th e r e b y

p r o vid es

an o p p o r tu n ity

to

illu s tr a te R a s ch ite m

a n a lys is in m in u te

e x p o s in g to c o m p le te c o m p r e h e n s io n th e p r oces s in vo lve d , a n d , w h e r e

c o m p u tin g fa c ilitie s a re r e m o te o r it is u r g en t to c h e c k c o m p u te r o u tp u t fo r p la u s ib ility, th en P R O X p r o vid es a m e th o d fo r ca lib r a tin g item s w h ic h re q u ir es n o th in g m o r e th a n th e o b s er ved

d is tr ib u tio n s

o f item

s co r e a n d

p er s on

s cor e,

a hand

c a lc u la to r ( o r a d d in g

m a ch in e) a n d p a p er a n d p en cil. T h e d a ta fo r illu s tr a tin g P R O X c o m e fr o m th e a d m in is tr a tio n o f th e 1 8 - item . K n o x C u b e T e s t, a s u b tes t o f th e A r th u r P o in t S ca le (A r th u r , 1 9 4 7 ) to 3 5 s tu d en ts in G r a d es 2 to

7. O u r a n a lys is

m a n a g in g

n ot

o n ly

o f th es e d a ta th e

s h ow s

c o n s tr u c tio n

h ow

of

R a s ch

n a tio n a l

ite m

ite m

a n a lys is ca n

b a n ks

but

a ls o

b e u s efu l fo r th e s m a lles t

im a g in a b le m ea s u r em en t p r o b le m , i.e., o n e s h o r t tes t g iven to o n e r o o m fu l o f ex a m in ees . U s in g s tu d en t c o r r e c t/ in c o r r e c t res p on s es to ea ch ite m o f th e tes t, w e w o r k o u t in d e ta il ea ch

s tep

o f th e p r o ce d u r e fo r

PRO X

ite m

a n a lys is . T h e n , C h a p te r 3 r e vie w s

c o m p a r a b le c o m p u te r a n a lys es o f th e s a m e d a ta b y b o th th e P R O X p r o c e d u r e a n d th e m o r e a ccu r a te U C O N p r o ce d u r e u s ed in m o s t c o m p u te r p r o g r a m s fo r R a s ch ite m a n a lys is . T h e s e d e ta ile d s tep s o ffe r a s ys tem a tic illu s tr a tio n o f th e ite m a n a lys is p r o c e d u r e w ith w h ich to c o m p a r e a n d b y w h ic h to u n d er s ta n d c o m p u te r o u tp u ts . T h e y a ls o d em o n s tr a te th e ea s e o f h a n d c o m p u ta tio n s u s in g P R O X

( P R O X is d e r ive d a n d d es cr ib e d a t len g th in

W r ig h t a n d D ou g la s , 1 9 7 5 b , 1 9 7 6 , 1 9 7 7 a ; C o h en , 1 9 7 6 , a n d W r ig h t, 1 9 7 7 ). F in a lly , th e y illu s tr a te th e e m p ir ica l d e v e lo p m e n t o f a la ten t tr a it o r va r ia b le. E a ch s tep m o ve s fr o m th e

o b s e r ve d

d a ta

to w a r d

th e

in fe r r e d

va r ia b le ,

fr o m

th e c o n fin e s

o f th e o b s er ved

tes t- b ou n d s cores to th e rea ch es o f th e in fe r r e d te s t- fr ee m ea s u rem en ts .

2 .2 .

THE K N O X CUBE TEST

W h ile th e A r th u r P o in t S ca le c o ver s a va r ie ty o f m en ta l ta sk s, th e K n o x C u b e T e s t im p lies

a s in g le la ten t tr a it. S u cces s o n th is s u b tes t r eq u ir es th e a p p lic a tio n o f vis u a l

a tte n tio n a n d s h o r t- ter m m e m o r y to a s im p le s eq u en cin g ta s k . It a p p ea rs to b e fr e e fr o m s ch o o l- r ela ted ta s ks a n d h en c e to b e a n in d ic a to r o f n o n ver b a l in te llec tu a l c a p a c ity.

28

IT E M C A L IB R A T IO N B Y H A N D

Th e

Kn ox

29

C u b e T e s t u ses fiv e o n e- in c h cu b es . F o u r o f th e cu b es a r e fix e d

tw o

in ch es a p a r t o n a b o a r d , a n d th e fifth c u b e is u s ed to ta p a s eries o n th e o th e r fo u r . T h e fo u r a tta c h e d cu b es w ill b e r e fe r r e d to , fr o m l e ft to r ig h t, as “ 1 , ” “ 2 , ” “ 3 ” a n d “ 4 ” to a v o id c o n fu s io n w h e n s p e c ify in g a n y p a r tic u la r s eries to b e ta p p e d . In th e o r ig in a l ver s io n o f th e te s t u s ed f o r th is e x a m p le , th e r e a r e 1 8 s u ch s eries g o in g fr o m th e tw o - s te p s eq u en ces ( 1 - 4 ) a n d ( 2 - 3 ) to th e s even - s tep s e q u e n c e (4 - 1 - 3 - 4 - 2 - 1 - 4 ). U s u a lly, a s u b je c t is a d m in is te r ed th is te s t tw ic e w ith a n o th e r s u b tes t fr o m th e b a tte r y in te r ve n in g . H o w e v e r , w e n eed u se o n ly th e fir s t a d m in is tr a tio n fo r o u r a n a lys is . T h e 1 8 s eries a r e g iv e n in F ig u r e 2 .2 .1 . T h e s e a r e th e 1 8 “ ite m s ” o f th e tes t. N o te th a t Ite m s 1 a n d 2 r e q u ir e a tw o - s te p s e q u e n c e ; Ite m s 3 th r o u g h 6 , a th r ee- s tep s eq u e n c e ; Ite m s 7 th r o u g h

1 0 , a fo u r - s te p s e q u e n c e ; Ite m s 1 1 th r o u g h

1 3 , a five - s te p s eq u en ce,

Ite m s 1 4 th r o u g h 1 7 , a s ix - s tep s e q u e n c e ; a n d Ite m 1 8 , a s even - s tep s eq u en ce.

F I G U R E 2 .2 . 1

I T E M N A M E A N D T A P P IN G O R D E R F O R T H E K N O X CUBE TEST

IT E M N A M E

2 .3

T A P P IN G O R D E R

1

1

4

2

2

3

3

1

2

4

4

1

3

4

5

2

1

4

6

3

4

1

7

1

4

3

2

8

1

4

2

3 4

9

1

3

2

10

2

4

3

1

11

1

3

1

2

4

12

1

3

2

4

3

13

1

4

3

2

4

14

1

4

2

3

4

1

4

1

3

3

1

4

15

1

3

2

16

1

4

2

17

1

4

3

1

2

4

18

4

1

3

4

2

1

T H E D A T A F O R IT E M A N A L Y S IS

T h e r es p on s es o f 3 5 s tu d en ts to a s in g le a d m in is tr a tio n o f th e 1 8 ite m K n o x C u b e T e s t a r e g ive n m a tr ix . A

in

T a b le

2 .3 .1 . T h e s e r es p on s es a re a r r a n g ed in a p er s o n - b y- item

d a ta

c o r r e c t r es p o n s e b y a s tu d e n t to a n ite m is r e c o r d e d as a 1, a n d a n in c o r r e c t

r es p o n s e as a 0 . T h e ite m s h a ve b ee n lis te d a cros s th e to p in th e o r d e r o f a d m in is tr a tio n .

BEST T EST D ESIGN

30

S tu d e n t s cores , th e n u m b er o f c o r r e c t res p on s es a ch ieved b y ea ch s tu d en t, a re given a t th e en d o f ea ch r o w in th e la s t c o lu m n o n th e rig h t. Ite m s cores , th e to ta l n u m b er o f c o r r e c t res p on s es to ea ch ite m , a re g iven a t th e b o tto m o f ea ch colu m n . In s p e ctio n o f T a b le 2 .3 .1 s h ow s th a t th e o r d e r o f a d m in is tr a tio n is v e r y c lo s e to th e o r d e r o f d iffic u lty .

Item s 1, 2 a n d 3 a r e a n s w er ed c o r r e c tly b y a ll s tu d en ts . A s econ d ,

s lig h tly g rea ter , le ve l o f d iffic u lty is o b s er ved in Item s 4 th r o u g h 9. T h e n Item s 1 0 a n d 11 s h ow a s h a rp in cr ea s e in d iffic u lty . Item s 1 2 th r o u g h 1 7 a re a n s w er ed c o r r e c tly b y o n ly a fe w s tu d en ts , a n d n o s tu d en t s u cceed s o n Ite m 18. O n ly 1 2 s tu d en ts s co r e s u cces s fu lly a t lea s t o n c e o n Item s 1 2 th r o u g h 1 7 , a n d o n ly fiv e o f th es e s tu d en ts d o o n e o r m o r e o f th e s ix- ta p item s s u cces s fu lly.

2 .4

C A L I B R A T I N G IT E M S A N D M E A S U R I N G P E R S O N S

Th e

g en er a l p la n fo r a cco m p lis h in g ite m

a n a lys is b eg in s w ith e d itin g th e d a ta in

T a b le 2 .3 .1 to r e m o ve p ers on s a n d item s fo r w h ic h

n o d e fin ite es tim a tes o f a b ility o r

d iffic u lty ca n b e m a d e, i.e., th o s e w ith a ll c o r r e c t o r a ll in c o r r e c t res p on s es . T h is m ea n s th a t P er s on 3 5 a n d Item s 1, 2, 3 a n d 1 8 m u s t b e s et a s id e, le a vin g 3 4 p ers on s a n d 14 item s

fo r

a n a lys is .

Th en

th e

r em a in in g

in fo r m a tio n

a b ou t

p ers on s

and

ite m s

is

s u m m a r ized in to a d is tr ib u tio n o f p er s on s cores a n d a d is tr ib u tio n o f ite m s cores . N e x t th es e s core d is tr ib u tio n s a re r e n d er ed as p r o p o r tio n s o f th e ir m a x im u m p os s ib le va lu e a n d th e ir fr e q u e n c y o f o ccu r r en ce is r e c o r d e d . T h e p r o p o r tio n s a re th en c o n ve r te d to lo g od d s , o r lo g its , b y ta k in g fo r item s th e n a tu ra l lo g o f th e p r o p o r tio n in c o r r e c t d ivid e d b y th e p r o p o r tio n c o r r e c t a n d fo r p er s on s th e n a tu ra l lo g o f th e p r o p o r tio n o f su ccess es d ivid e d

b y th e p r o p o r tio n

o f fa ilu r es . T h is c on ver ts p r o p o r tio n s , w h ic h a re

b o u n d e d b y 0 a n d 1 , to a n e w s ca le w h ic h e x te n d s fr o m -

° ° to + 00 a n d is lin eq r in th e

u n d e r lyin g va r ia b le. For

“ ite m

d iffic u lty ”

th is

va r ia b le

in crea s es

w ith

th e

p r o p o r tio n

o f in c o r r e c t

res p on s es . F o r “ p er s on a b ility ” it in crea s es w ith th e p r o p o r tio n o f c o r r e c t res p on s es . T h e m ea n a n d va r ia n ce fo r ea ch d is tr ib u tio n o f lo g its a re th en c o m p u te d , a n d th e m ea n ite m lo g it is lu sed to c e n te r th e ite m lo g its a t ze r o . T h is c h o ic e o f o r ig in fo r th e n e w s ca le is in e vita b ly a r b itr a r y b u t m u s t b e m a d e. B a s in g it o n item s r a th er th a n o n p ers on s a n d p la cin g it in th e c e n te r o f th e c u r r e n t item s is n a tu ra l a n d c o n ve n ie n t. T h e ite m lo g it a n d p er s on lo g it va ria n ces a r e u s ed to ca lcu la te tw o ex p a n s io n fa ctor s , on e

fo r

item s

and

on e

fo r

p ers on s .

Th es e

fa c to r s

a re

u s ed

to

ca lcu la te

th e fin a l

s a m p le- fr ee ite m d iffic u ltie s a n d te s t- fr ee p er s on a b ilities . T h e y a re n eed ed b eca u s e th e a p p a r en t r e la tive d iffic u ltie s

o f th e item s

d ep en d

u p on

h ow

d is p er s ed

in a b ility th e

s a m p le o f p er s on s is. T h e m o r e d is p er s ed th e p ers on s , th e m o r e s im ila r in d iffic u lty w ill item s a p p ea r . T h is is a ls o tr u e fo r a p p a r en t a b ility. T h e m o r e d is p er s ed th e te s t in ite m d iffic u lty , th e m o r e s im ila r in a b ility w ill p er s on s a p p ea r. T h e s e e ffe c ts o f s a m p le s p rea d a n d tes t w id th m u s t b e r e m o ve d fr o m th e es tim a tes o f ite m d iffic u lty a n d p er s on a b ility , if th es e es tim a tes a r e to b e m a d e s a m p le- fr ee a n d tes t- fr ee. F in a lly , th e s ta n d a rd er r or s o f th es e es tim a tes a re ca lcu la ted . T h e s ta n d a rd errors a re n ee d ed to assess th e p r ecis ion o f th e es tim a tes . T h e y d e p e n d o n th e s a m e ex p a n s io n fa c to r s p lu s th e e x te n t to w h ic h th e ite m d iffic u lty is c e n te r e d a m o n g th e p er s on a b ilities a n d th e p er s on a b ility is c en te r e d a m o n g th e ite m d iffic u ltie s . T h e m o r e th a t item s o r p ers on s a r e c en te r e d o n ta r g et, th e m o r e p r ecis e a re th e ir es tim a tes a n d h en c e th e s m a ller th e ir s ta n d a rd errors .

IT E M C A L IB R A T IO N B Y H A N D

31

Z ui ° cc

a £

8 8

|> . 0

0

«0

0

0

^

0

0

» - 0 0 0 0 0 « - ( w> 0 0 > ,-

0> » - C N ( M C N ^ if) O N 0 0 0 1 0 * - (

o o o o o o o o o o o o o o o o o o o c o o o o o o o o o o o o o o

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ^ - 0 0 0 0 0 0 0 0 0 0

TEST

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ^ - 0 0 0 0 0 0 0 0 0 0

C UB E

0 0 0 0 0 0 ^ - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

KNOX

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

*

-

0

0

0

0

0

»

-

0

0

0

0

r

-

THE

0 0 0 0 0 0 « - 0 0 0 0 0 0 0 « - 0 0 » - 0 ’- » - 0 » - » - 0 0 0 0 0 0 0 0 0 0

o o o o o o

—o o » - o o « - « - « - « - o o o o « - « - o * - o o o « - « - o o » - - o o

O

CM

O

r-

o

o

O

CM

UI 2 < z

2

LU

h-

O R IG IN A L

RESPONSES

OF

35

PERSONS

TO

18

IT E M S

ON

0 0«- 0 00*- 00000*- 0»- 000000«- 00000000000'

O “

co 5 cc
»

c o

co L 05

a

z o

05

JS

hCC

O Q.

co r*^ oo 05 o r^. r-» oo

«—

cn

co

lo

co oo co co co

CO N CO 05 O CO CO CO CO 05

r- CM CO in 05 05 05 05 05

05

co co 05 05 05 05 05

JS

o oc

3 JS Eh

o

o

O

o

co

cn

O

r ;

o

o o o

co o r-

CM

co

CN

CD

r-

CM CM CO CO ^ 0 6 0 6 0

LT5 05 CO CO CN 10 Ifl CO 0 0 0 0 0

CO o 10 CO 00 00 0 0 0 0 0

a

o 0 5 in 05 05 05 O

6 6 0

2

^

cd 09

05 £2

05

JS

>»

L O G IT S

FROM

P R O P O R T IO N S

£2

-a o a o

z

!2

-Q

H

]>

•a

CC

o a. o

05

>>

o in

cn co ^ in in in

in in

CO N 00 O) O 10 in

lo

10 co

*— cn co in co co co co co

CO CO 05 O co co co co

1-

cn co .

'O

05 T3

i" *

oc

CL

a

c o a

o a o

m 0 5 ^ o in O 05 05 05 CO

c

0 0 0 I

I

I

in «- co cn cq cq 0 0 0 0 0 I I I I I o

CO

CO 00 05 LO Tin in ^ ^

0 0 0 0 0 1 1 l 1 1

CO CO

CN00 ^ O COCN CN CN

0

0 0 0

I

II

I

0

I

CO CN 00 ^ o

o

o a

O

o

6 0 6 6 0 1 1 1 I 1 05

£) 2 z

g H

OC

o

a.

05

JO

3

CO S 00 05 O CN CN CN CN CO

1—

cn co ^ in CO CO CO CO CO

CO CO 05 o CO CO CO CO ^

«- cn co ^

in

^

co rs oo 05 o

^

'T

in

O

c 05 JS

3

C 05

-C 1 -4 -> .

05

CO ■ 05

2

OC

2

o

(5 05

I

C5 * 05 ^

£2

fl e x

la .2 O

a o

05 CO CO ] 54

ite m ca lib r a tion s .

S tep 7 . In

C o rrec tin g Person Measures f o r th e E ffe c t o f T e s t W id th

T a b le 2 .4 .6 w e o b ta in th e fin a l c o r r e c te d p er s on m ea s u res a n d th e ir s ta n d a rd

err ors fr o m th e te s t w id th ex p a n s io n fa c to r X .

E X P L A N A T I O N O F T A B L E 2 .4 .6

N O T A T IO N A N D F O R M U L A E

C o lu m n 1 g ives a ll p os s ib le s cor es b eca u s e w e w a n t to

h a ve

p os s ib le

m ea s u res tes t

a va ila b le

s co r e

fr o m

fo r

1 to

r = 1, L -1

e ve r y L — 1,

w h a te ve r s cores w e r e a c tu a lly o b s er ved . C o lu m n 2

r ep ea ts th e in itia l p er s on m ea s u res fr o m

b?

= y r

C o lu m n 7 o f T a b le 2 .4 .4 .

C o lu m n 3

is th e p er s on

a b ility ex p a n s io n fa c to r

X = 2 .0 9 d u e to te s t w id th . C o lu m n 4

is

th e

c o r r e c te d

p er s on

m ea s u res

ob ­

b r = X b °=

Xyr

ta in ed b y m u ltip ly in g ea ch in itia l va lu e in C o lu m n 2 b y th e ex p a n s io n fa c to r o f 2 .0 9 .

C o lu m n 5

is th e s ta n d a rd

e r r o r o f th e c o r r e c te d

SE ( b r ) = X [ L / r ( L - r) ]

p er s on m ea s u res .

2 .5

D IS C U S S I O N

T h e P R O X ite m a n a lys is p r o ce d u r e has b een c a r e fu lly d es cr ib ed n o t o n ly b eca u s e it a ccom p lis h es

ite m

c a lib r a tio n

and

h en ce

p er s on

m ea s u r em en t

but

a ls o

b eca u s e

it

e m b o d ie s in a lo g ic a l a n d s tr a ig h tfo r w a r d m a n n er th e s im p les t p o s s ib le a n a lys is o f th e in te r a c tio n b e tw e e n item s a n d p ers on s . T h e d ec is ive id ea o n w h ic h th is a n a lys is is b a s ed is th a t th e p r o b a b ility o f s u ccess is d o m in a te d b y th e p e r s o n ’s a b ility a n d th e ite m ’s d iffi­ c u lty . A m o r e a b le p er s on is s u p p os ed a lw a ys to h a ve a g r e a ter ch a n ce o f s u ccess o n a n y ite m th a n is a les s a b le p er s on . A n y p a r ticu la r p er s on is s u p p os ed a lw a ys to h a ve a b e tte r ch a n ce o f s u ccess o n a n ea s y ite m th a n o n a d iffic u lty o n e. T o th e e x te n t th is is th e cas e th e p r o b a b ility o f a n y p e r s o n ’s s u ccess o n a n y ite m ca n b e s p e c ifie d as th e con s eq u en ce of

th e d iffe r e n c e

p o s itio n

on

b e tw e e n

th e

p er s o n ’s p o s itio n

th a t s a m e va r ia b le. T h a t is th e

on

R a s ch

a s in g le va r ia b le a n d m od el

fo r

ite m

th e ite m ’s

a n a lys is a n d tes t

c o n s tr u c tio n a n d , in d eed , th e fu n d a m en ta l m o d e l im p lic it in th e ite m a n a lys is o f a ll th os e w h o w o r k w ith u n w eig h ted s cores (A n d e r s e n , 1 9 7 7 ).

IT E M

C A L IB R A T IO N

A ll a b ility ,

th e is

d iffic u lty ,

in fo r m a tio n

a s s u m ed

s u m m a r ize d

B Y H A N D

to

45

o b s e r ve d

be

a b o u t a p e r s o n ’s p o s itio n

exp res s ed

in

h is

r es p o n s es

to

th e

o n th e va r ia b le , e .g ., h is s et o f ite m s h e ta k es as

in th e u n w e ig h te d c o u n t o f th e n u m b e r o f ite m s h e g ets c o r r e c t. F o r ite m

th e

in fo r m a tio n

o b s e r ve d

is

a s s u m ed

to

b e c o m p le te ly

c o n ta in e d

in

th e

u n w e ig h te d c o u n t o f p er s o n s in th e s a m p le w h o r e s p o n d e d c o r r e c tly to th a t ite m .

O f

cou rs e,

th is

m o d e lin g

of

th e

in te r a c tio n

b e tw e e n

p er s o n

and

ite m

is

an

id e a liz a tio n a n d ca n o n ly a p p r o x im a te w h a te v e r a c tu a lly h a p p en s . A l l id ea s , h o w e v e r , a re, in th e e n d , o n ly a p p r o x im a tio n s o f, o r a b s tr a c tio n s fr o m , e x p e r ie n c e . T h e ir va lu e ca n o n ly b e ju d g e d in te r m s o f th e ir u s efu ln es s , th a t is, th e ir d e m o n s tr a b le r e le va n c e to th e s itu a tio n u n d e r s tu d y , a n d th e ir s im p lic ity . T h is c h a p te r h a s illu s tr a te d th e s im p lic ity a n d p o te n tia l c o n v e n ie n c e o f R a s ch

ite m

a n a lys is . Its u tility is te s tifie d

to b y h u n d r ed s o f

a p p lic a tio n s . O u r n e x t ta s k is to s h o w h o w th e s a m e d a ta ju s t a n a ly ze d b y h a n d c o m e o u t w h e n th e P R O X p r o c e d u r e a n d its m o r e e la b o r a te a n d a c c u r a te p a r e n t p r o c e d u r e , U C O N , a r e a p p lie d to th e m b y c o m p u te r .

3

3 .1

I T E M C A L IB R A T IO N B Y C O M P U T E R

IN T R O D U C T IO N

In th is c h a p ter w e d is p la y a n d d es cr ib e c o m p u te r o u tp u t fo r R a s ch ite m c a lib r a tio n u s in g th e e s tim a tio n p r oced u r es P R O X a n d U C O N

(W r ig h t a n d P a n ch a p a k es a n , 1 9 6 9 ;

W r ig h t a n d D ou g la s , 1 9 7 5 b , 1 9 7 7 a , 1 9 7 7 b ). T h e K n o x C u b e T e s t d a ta a n a lyze d b y h a n d in C h a p ter 2 a re u s ed fo r illu s tr a tio n . T h e e s tim a tio n p r o ced u r es a re p e r fo r m e d b y th e c o m p u te r p r o g r a m

B IC A L

(W r ig h t a n d M ea d , 1 9 7 6 ). T h e a ccu r a cy a n d u tility o f th e

h a n d c a lib r a tio n d es cr ib ed in C h a p ter 2 a re eva lu a ted b y c o m p a r in g th e “ h a n d ” es tim a tes w ith th o s e p r o d u c e d b y a c o m p u te r a n a lys is o f th e s a m e d a ta . K n o w in g th e s tep s b y w h ic h th es e p r o ced u r es ca n b e a p p lie d b y h a n d s h ou ld fa c ilita te u n d er s ta n d in g a n d u s in g th e c o m p u te r o u tp u t. W e w ill m o ve th r o u g h th e c o m p u te r o u tp u t s tep b y s tep in o r d e r to b r in g o u t its o r g a n iza tio n

and

u se. B IC A L

p r o d u ces th e o u tp u t g iven

in T a b le s 3 .2 .1 to 3 .2 .9 . A

co m p a r is o n o f th e ite m a n d p er s on s ta tis tics fr o m P R O X b y h a n d w ith th o s e fr o m P R O X b y c o m p u te r is g iven in T a b le s 3 .3 .1 th r o u g h 3 .3 .5 . 3 .2

B IC A L O U T P U T F O R A P R O X A N A L Y S IS O F T H E K N O X C U B E T E S T D A T A

T h e fir s t p a g e o f th e o u tp u t, in T a b le 3 .2 .1 , reca p s th e c o n tr o l s p e c ific a tio n s n eces ­ s a ry to a p p ly th e 1 9 7 6 ver s io n o f B IC A L to th is c a lib r a tio n jo b ( f o r d eta ils c o n s u lt th e m a n u a l th a t g o es w ith y o u r B IC A L p r o g r a m ). A t th e to p w e b eg in w ith th e j o b title , th e va r iou s c o n tr o l p a r a m eter s , th e r e c o r d ( o r c a r d ) colu m n s r ea d , th e te s t s co r in g k e y a n d a c o p y o f th e fir s t p er s o n r e c o r d rea d . F in a lly , th e o u tp u t r e p o r ts th a t 1 8 item s a n d 3 4 p er ­ s on s w e n t in to th is a n a lys is . __________________|

--------------------------

T A B L E 3 .2 .1

|_ __ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

P R O G R A M C O N T R O L S P E C IF IC A T IO N S

-----------------------

K N O X CUBE TEST CO NTRO L PARAM ETERS N IT E M

NGROP

M l NSC

MAXSC

LREC

KCAB

SCORE

18

10

1

17

21

1

0

COLUM NS SELECTED

1 1

*

2

3

4

. . .. . .. . .. . .. . .. . .. . ..

111111111111111111 KEY

111111111111111111 F IR S T S U B J E C T 001111111100000000000

N U M B E R O F IT E M S

18

N U M B E R O F SUB JT

34

46

IT E M

C A L IB R A T IO N

B Y C O M PUTER

47

T h e c o n tr o l p a r a m eter s f o r th is j o b w e r e : 1.

N u m b e r o f ite m s ( N I T E M ) :

18

T h e r e w e r e o r ig in a lly 1 8 ite m s in th e K C T .

2.

S m a lle s t s u b g r o u p s ize ( N G R O P ) :

10

S u b g r o u p s o f a t lea s t 1 0 p er s o n s a r e to b e

3.

M in im u m s c o r e ( M IN S C ) :

4.

M a x im u m s c o r e ( M A X S C ) :

17

T h e m a x im u m s c o r e to b e u s ed is 1 7 .

5.

R e c o r d le n g th t o b e r e a d ( L R E C ) :

21

T h e d a ta c o m e s fr o m th e fir s t 2 1 p o s itio n s

fo r m e d f o r a n a ly zin g ite m fit. 1

T h e m in im u m s c o r e t o b e u s ed is 1.

o f ea ch p er s o n r e c o r d . [ T h e c o lu m n s e le c t c a r d (lis te d in T a b le 3 .2 .1 u n d e r “ C O L U M N S E LE C TE D ” )

s p e c ifie s

th e

18

c o lu m n s

th a t c o n ta in th e s e te s t r e s p o n s es .] 6.

C a lib r a tio n p r o c e d u r e ( K C A B ) :

The

c a lib r a tio n

p roced u re

to

be

u s ed

is

PRO X. [T h e

s e le c tio n

cod e

is:

1 =

PRO X,

2 =

UCON] 7.

S c o r in g o p tio n ( S C O R E ) :

T h e d a ta a r e a lr e a d y s co r e d . [ T h e fu ll c o n tr o l c o d e is : 0 = d a ta to b e s cored

d ic h o to m o u s ly

s u p p lie d ;

a c c o r d in g

to

k ey

1 = d a ta a r e s u cces s ive in teg er s ;

2 = s c o r e d a ta “ c o r r e c t” , i f r es p o n s e va lu e equ a l

to

or

les s

“ in c o r r e c t” ; 3

=

th a n

k ey

s c o r e d a ta

s u p p lie d , els e “ c o r r e c t” , i f

r e s p o n s e va lu e e q u a l t o o r g r e a te r th a n k e y s u p p lie d , els e “ in c o r r e c t” .] T a b le 3 .2 .2 g ives ea c h ite m ’s r e s p o n s e fr e q u e n c ie s f o r ea ch r es p o n s e va lu e. T h is ta b le ca n a c c o m m o d a te u p to fiv e r es p o n s e va lu es as s p e c ifie d b y th e u s er. A n “ u n k n o w n ” v a lu e c o lu m n r e c o r d s th e c o u n t o f a ll o th e r va lu es e n c o u n te r e d . T h e fin a l c o lu m n is fo r th e k e y . T h e k e y m a r k s th e va lu e s p e c ifie d as c o r r e c t w h e n th e d a ta is s till to b e s co r e d . A s T a b le 3 .2 .2 s h o w s th e K C T d a ta w a s e n te r e d in s c o r e d fo r m . T h e a p p r o p r ia te k e y , th e r e fo r e , is th e v e c to r o f ‘ l ’s s h o w n in T a b le s 3 .2 .1 a n d 3 .2 .2 . E a ch ite m is id e n tifie d o n th e l e f t b y its s e q u e n c e n u m b e r in th e o r ig in a l o r d e r o f te s t ite m s as r e a d in to B IC A L . A fo u r - c h a r a c te r ite m n a m e ca n a ls o b e u s ed to id e n tify te s t ite m s . F o r th e K C T w e h a ve n a m e d th e ite m s b y th e n u m b e r o f ta p s r e q u ir e d . T a b le 3 .2 .2 e n a b le s u s t o e x a m in e th e o b s e r ve d r es p o n s es f o r o b v io u s d is tu r b a n ces t o o u r te s t p la n a n d w ill o fte n s u g g es t p o s s ib le e x p la n a tio n s f o r g ros s m is fits . T h e d is tr i­ b u tio n o f r es p o n s es o v e r m u ltip le - c h o ic e d is tr a c to r s , f o r e x a m p le , ca n r e ve a l th e u n d u e in flu e n c e o f p a r tic u la r d is tr a c to r s . T h e e ffe c ts o f in s u ffic ie n t tim e s h o w u p in th e p ilin g u p o f r es p o n s es in th e U N K N c o lu m n to w a r d th e e n d o f th e tes t. T h e e ffe c ts o f w id e ­ s p r ea d in e x p e r ie n c e in te s t ta k in g s h o w u p in th e p ile- u p o f U N K N r es p on s es in th e fir s t o n e o r t w o ite m s o f th e tes t. W e s ee a g a in in T a b le 3 .2 .2 w h a t w e a lr e a d y le a r n e d fr o m T a b le 2 .3 .1 , n a m e ly th a t th e fir s t th r e e ite m s a r e a n s w e r e d c o r r e c tly b y a ll 3 4 p er s on s , th a t Ite m 1 8 w a s n o t a n ­ s w e r e d c o r r e c tly b y a n y o n e a n d th a t th e r e is a r a p id s h ift fr o m la r g e ly c o r r e c t res p on s es to

la r g e ly

in c o r r e c t r es p on s es

b e tw e e n Ite m s 9 a n d 1 1 . S in c e IT E M

NAM E

g ives th e

n u m b e r o f ta p s in th e s eries , w e s ee th a t th is s h ift o ccu r s w h e n th e ta s k m o ve s fr o m a s eries o f fo u r ta p s u p t o fiv e ta p s .

48

B E S T T E S T D E S IG N

T A B L E 3.2.2

R E S P O N S E F R E Q U E N C IE S F O R E A C H R E S P O N S E A L T E R N A T I V E

A L T E R N A T I V E R E S P O N S E F R E Q U E N C IE S 0

SEQ

IT E M

NUM

NAME

1

UNKN

KEY

1

2

0

34

0

0

0

0

1

2

2

0

34

0

0

0

0

1

3

3

0

34

0

0

0

0

1

4

3

2

32

0

0

0

0

1 1

5

3

3

31

0

0

0

0

6

3

4

30

0

0

0

0

1

7

4

3

31

0

0

0

0

1

8

4

7

27

0

0

0

0

1

9

4

4

30

0

0

0

0

1

10

4

10

24

0

0

0

0

1

11

5

22

12

0

0

0

0

1

12

5

28

6

0

0

0

0

1

13

5

27

7

0

0

0

0

1

14

6

31

3

0

0

0

0

1

15

6

33

1

0

0

0

0

1

16

6

33

1

0

0

0

0

1

17

6

33

1

0

0

0

0

1

18

6

34

0

0

0

0

0

1

T a b le 3 .2 .3 r e p o r ts th e e d itin g p roces s . I t s u m m a r izes th e w o r k o f th e e d itin g r o u ­ tin e w h ich

s u cces s ively r e m o ve s p er s o n r e co r d s w ith ze r o o r p e r fe c t s cores a n d item s

c o r r e c tly a n s w er ed b y a ll p er s on s o r n o t a n s w er ed c o r r e c tly b y a n y p er s on s , u n til a ll s u ch p ers on s

o r item s

a re d e te c te d a n d s et a s id e. T h e e d itin g p r oces s d eter m in e s th e fin a l

m a tr ix o f ite m - b y- p er s o n res p on s es th a t is a n a lyze d . T a b le 3 .2 .3 s h ow s th a t in itia lly th er e w e r e n o p er s on s w ith p e r fe c t o r ze r o s cores , a n d th a t 1 8 ite m s e n te r e d th e ru n , w ith n o p er s o n s co r in g b e lo w 1 o r a b o ve 1 7 , le a vin g 3 4 p er s on s fo r c a lib r a tio n

( T h e 3 5 th p er s o n a p p ea r in g in T a b le 2 .3 .1 h a d a lr ea d y b een

r e m o ve d fr o m th e d a ta d e c k b y h a n d .). Ite m s 1 , 2 a n d 3 a re th en r e m o ve d b y th e e d itin g p ro ces s b eca u s e th e y w e r e a n s w er ed c o r r e c tly b y a ll s u b jects a n d Ite m

1 8 is r e m o ve d

b eca u s e n o o n e a n s w er ed it c o r r e c tly . A ft e r th is e d itin g th e c a lib r a tio n s a m p le s till c o n ­ s ists o f 3 4 s u b jects , b u t n o w o n ly th e 1 4 item s w h ic h ca n b e c a lib r a ted r em a in , w ith th e s a m e m in im u m s co r e o f 1 a n d a n e w m a x im u m s cor e o f 1 3 . T a b le 3 .2 .4 s h ow s th e d is tr ib u tio n o f p er s on s o v e r th e K C T s cores . T h e h is to g r a m is s ca led a c c o r d in g to th e s ca le fa c to r p r in te d b e lo w th e gra p h . T h e d is tr ib u tio n o f p er s on s cores gives a p ic tu r e o f h o w th is s a m p le r e s p o n d e d to th es e item s . I t s h ow s h o w w e ll th e item s w e r e ta r g e te d o n th e p ers on s a n d h o w r e le va n t th e p er s on s s ele cted w e r e fo r th is c a lib r a tio n . F o r th e b es t c a lib r a tio n , p ers on s s h ou ld b e m o r e o r less e ve n ly d is tr ib u ted o v e r a ra n g e o f s cores , a r ou n d a n d a b o ve th e c e n te r o f th e tes t. In o u r s a m p le w e s ee a s ym m e tr ic a l d is tr ib u tio n a r ou n d a m o d a l s co r e o f 7.

IT E M

C A L IB R A T IO N

B Y C O M PU TER

49

T A B L E 3 .2 .3

T H E E D IT IN G P R O C E S S

K NO X CUBE TEST N UM BER O F ZER O SCORES

0

N UM BER OF PER FEC T SCORES

0

N U M B E R O F IT E M S S E L E C T E D

18

N U M B E R O F IT E M S N A M E D

18

SUBJECTS B ELO W

1

0

SUBJECTS A B O V E

17

0

S U B J E C T S R E M A IN IN G

34

T O T A L SUBJECTS

34

R E J E C T E D IT E M S IT E M

IT E M

ANSW ERED

NUMBER

NAME

CORRECTLY

1

2

34

H IG H S C O R E

2

2

34

H IG H S C O R E

3

3

34

H IG H S C O R E

18

6

0

LOW SCORE

SUB JECTS D E L E T E D

=

0

S U B J E C T S R E M A IN IN G

=

34

IT E M S D E L E T E D

=

4

IT E M S R E M A IN IN G

=

14

M IN IM U M S C O R E

=

1

M A X IM U M S C O R E

=

13

T A B L E 3 .2 .4

S A M P L E P E R S O N A B IL IT Y D IS T R IB U T IO N

SCORE

D IS T R IB U T IO N O F A B IL IT Y COUNT

2

P R O P O R T IO N

1

0

0 .0

2

1

0 .0 3

X

3

2

0 .0 6

XX

4

2

0 .0 6

XX

5

2

0 .0 6

XX

6

3

0 .0 9

XXX

7

12

0 .3 6

xxxxxxxxxxxx

8

5

0 .1 5

xxxxx

9

4

0 .1 2

xxxx

10

1

0 .0 3

X

11

2

0 .0 6

XX

12

0

0 .0

13

0

0 .0

14

0

0 .0 E A C H X = 2 .9 4 P E R C E N T

B E S T T E S T D E S IG N

50

T a b le 3 .2 .5 s h ow s th e d is tr ib u tio n o f ite m

eas ines s. T h e s ca le is g iven b e lo w th e

gra p h . Item s 4 th r o u g h 9 a re s een to b e fa ir ly ea s y, w ith Ite m 8 th e m o s t d iffic u lt a m o n g th em . Ite m

1 0 is s lig h tly m o r e d iffic u lt. Ite m

1 1 is m u ch m o r e d iffic u lt, fo llo w e d b y

Item s 1 2 a n d 1 3 , m o r e d iffic u lt s till. F in a lly , item s 1 5 , 1 6 a n d 1 7 a re s o d iffic u lt th a t o n ly o n e p er s on a n s w er ed th es e ite m s c o r r e c tly ._ _ _ _ _ _ _ _ T A B L E 3 . 2 . 5 ___ __ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

T E S T IT E M E A S IN E S S D I S T R I B U T IO N

IT E M

D I S T R I B U T I O N O F E A S IN E S S 2

4

6

8

COUNT

P R O P O R T IO N

4

32

0 .9 4

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

5

31

0 .9 1

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

6

30

0 .8 8

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

7

31

8

27

0 .9 1 0 .7 9

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

9

30 24

0 .8 8

10

0 .7 1

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

11

12

0 .3 5

xxxxxxxxxxxxxxxxxx

12

6

0 .1 8

xxxxxxxxx

13 14

7

0 .2 1

xxxxxxxxxx

0 .0 9

xxxx

15

3 1

0 .0 3

X

16

1

0 .0 3

X

17

1

0 .0 3

X

10

E A C H X = 2 .0 0 P E R C E N T

T a b le 3 .2 .6 gives th e e s tim a tio n in fo r m a tio n . A t th e to p a r e th e P R O X d iffic u lty a n d a b ility e x p a n s io n fa c to r s . N o tic e th a t th es e va lu es a re id e n tic a l to th o s e w e o b ta in e d b y h a n d in C h a p ter 2. W ith in th e ta b le , th e fir s t fo u r c o lu m n s g ive th e ite m s eq u en ce n u m b er , ite m n a m e, ite m d iffic u lty a n d s ta n d a r d er r or . ___________________

T A B L E 3 .2 .6

C A L IB R A T IO N B Y P R O X

D IF F I C U L T Y E X P A N S IO N F A C T O R

1 .3 1

A B I L I T Y E X P A N S IO N F A C T O R

2 .1 0

SEQUENCE

IT E M

IT E M

STANDARD

NUMBER

NAME

D IF F IC U L T Y

ERROR

4

3

-3 .8 6 5

0 .8 3 3

5

3

-3 .2 9 4

0 .6 9 1

6 7

3

-2 .8 7 6

0 .6 0 8

4

-3 .2 9 4

0 .6 9 1

8

4

-2 .0 0 7

0 .4 8 5

9

4 4

-2 .8 7 6 -1 .3 8 8

0 .6 0 8 0 .4 3 0

5

0 .5 4 7

0 .4 1 0

5 5

1 .7 6 7

0 .5 1 4

1 .5 1 8

0 .4 8 5 0 .6 9 1

10 11 12 13 14 15

6 6

16 17

6 6

2 .8 0 5 4 .3 2 1 4 .3 2 1 4 .3 2 1

1 .1 6 0 1 .1 6 0 1 .1 6 0

IT E M

C A L IB R A T IO N

T a b le

3 .2 .7

BY CO M PU TER

51

g ives th e lo g it a b ility m ea s u r e a n d its s ta n d a r d e r r o r f o r ea ch s co r e o n

th e K C T a n d th e n u m b e r o f p er s o n s in th e s a m p le o b ta in in g ea ch s co r e . F o r ea ch r a w s c o r e w e ca n s ee th e s a m p le fr e q u e n c y a t th a t s c o r e a n d th e a b ility a n d s ta n d a r d e r r o r im p lie d

b y th a t s c o r e . T h e s a m p le a b ility m ea n a n d s ta n d a r d d e v ia tio n a r e g ive n a t th e

b o tto m o f th e ta b le .

T A B L E 3 .2 .7

M EA SU R EM EN T BY PROX C O M P L E T E S C O R E E Q U IV A L E N C E T A B L E RAW SCORE

COUNT

PERSON

STANDARD

A B IL IT Y

ERROR

13

0

5 .4 0

12

0

3 .7 7

1 .1 1

11

2

2 .7 3

0 .9 4

1 .5 1

10

1

1 .9 3

0 .8 6

9

4

1 .2 4

0 .8 1

8

5

0 .6 1

0 .7 8

7

12

0 .0 0

0 .7 8

6

3

-0 .6 1

0 .7 8

5

2

-1 .2 4

0 .8 1

4

2

-1 .9 3

0 .8 6

3

2

-2 .7 3

0 .9 4

2

1

-3 .7 7

1 .1 1

1

0

-5 .4 0

1 .5 1

M E A N A B IL IT Y = - 0 .0 6 S D O F A B IL IT Y =

T a b le

3 .2 .8

in c lu d e a d iv is io n

p r o v id e s

ite m

1 .1 4

c h a r a c te r is tic cu r ves a n d f i t s ta tis tics . T h e tes ts o f f it

o f th e c a lib r a tio n s a m p le in to a b ility s u b g r o u p s b y s co r e le ve l. T h r e e

g r o u p s h a ve b e e n m a d e o u t o f th e K C T s a m p le, th e 1 0 p er s o n s w ith s co r es fr o m 1 to 6 , th e 1 2 p er s o n s a t s c o r e 7 a n d th e 1 2 p er s o n s w ith

s co r es fr o m

8 to 1 3 . C o n tr o l o v e r

g r o u p s ize a n d h e n c e o v e r th e n u m b e r o f g r o u p s u s ed is a s s er ted th r o u g h th e c o n tr o l p a r a m e te r N G R O P . A n e va lu a tio n o f ite m d iffic u lty in va r ia n c e o v e r th es e a b ility g r o u p s is m a d e b y c o m p a r in g f o r ea ch ite m its d iffic u lty e s tim a te s o v e r th e d iffe r e n t g r ou p s . T h e tes ts o f f i t a r e th u s s a m p le - d e p e n d e n t. H o w e v e r , i f th e d iffic u lty e s tim a tes th e y u s e pass th e s e tes ts , th e n th o s e e s tim a te s a r e s a m p le - fr e e as fa r as th a t s a m p le is c o n c e r n e d . O f c o u r s e , s u cces s fu l ite m f i t in o n e s a m p le d o e s n o t g u a r a n tee f i t in a n o th e r . H o w e v e r , as th e a b ility g r o u p s w ith in a g ive n s a m p le a r e a r r a n g ed b y s co r es , w e d o o b ta in in fo r m a tio n a b o u t th e s ta b ility o f ite m d iffic u ltie s o v e r va r io u s a b ilitie s a n d th e r e fo r e ca n s ee w h e th e r ou r

ite m s

are

d is p la y in g s u ffic ie n t in va r ia n c e

over

th es e

p a r ticu la r a b ility

q u a lify th e ite m s f o r u s e as in s tr u m en ts o f o b je c tiv e m ea s u r em en t.

grou p s

to

52

B E S T T E S T D E S IG N

i- oc Z ID C L

O

o

CN

o

CO CN

05 CD

d

d

d

d

d

,_

CN

p

LO

l o

d

d

d

00 i LO

o

d

'

O CN

CO

CO

p

p

CO | co 1

d

d

d

d

X

C/J Q Q Z

o

LO

r>» o

oo\

o

05

1

1 1 1

CQ

p

o7

o 00

05

r— 00

CO

CO

p

p

co n

d

d

1

i

.

o CN r-

/ 00 \ LO i q 1 It-’/ o

CN

CO

' oo\

CO

o

o

o

i* ■

CD CO

LO LO

o

f°A

o

CM

CN o

o *-

CO *-

r—

00 r^.

q

o

o

o

CO

CD

CO

o

o

LO

o

CM

o

O

t—

LO

CD

CN 00 o

o r^*

o

IO

CO O) o

o

o

col *fr /

CM CN t-’

o

o

CN 00 o

CO

co t—’

00 o

CO *— o

00 o

00 1 o j i- ,

CO «— o

CO t— o

CN

o

o

o

o

i 1

o

o

o

1

o> CO

»

--

0) *-* 00 CN

> 00 (A

©

c

M k 0)

© 0

©

©

0 «*)

to

(in

0L

n n / *^

G® 0

© 0

©3

©0

G®

G®

(3 ®

0

s

(2 )

©

©

o

in

r

0© G®

0 cvj 1

0 ro 1

ro

I ^

g

( N

--

0©

Q t

I H

I

Z

o h in a> in a> > 4) a

a

h

a> E rtJ

Z

z

H (/)

E a>

D (/)

*


0 n

c

8

V)

Q

lii

Nur

E w

3.

z

(ta p s m e d ia n

V A R IA B L E KCT THE

0

•™

E

o

D O C U M E N T IN G

] 1/4

1 =

1(1

“ P v i J / P v i ] 54

= e x p [ ( d j - bv ) / 2 ]

z 2, = e x p Id , - bv )

C O N S T R U C T IN G A

T a b le

V A R IA B L E

123

5 .1 1 .2 g ives a s u m m a r y o f th e u n e x p e c te d r es p on s es o b s e r ve d in th e K C T B

d a ta . C o lu m n

1 g ives th e ra n g e o f a b s o lu te d iffe r e n c e b e tw e e n p er s o n a b ility a n d ite m

d iffic u lty . C o lu m n 2 ex p r es s es th is d iffe r e n c e as z 2 = e x p (I b -

d |) a n d C o lu m n 3 c o n ­

ve r ts z 2 to th e r es p o n s e im p r o b a b ility [ 1 / (1 + z 2 ) ] it im p lies .

_ __ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |

T A B L E 5 .1 1 .2

|_ ____________________

S U M M A R Y O F U N E X P E C TE D RESPONSES O N KCTB 101 P E R S O N S B Y 2 3 IT E M S

A b ility D iffic u lty

I m p r o b a b i l it y

P o s s ib le

E x p e c te d

O bs erved

It e m

P e rs on

D iffe r e n c e

z2

1 /(1 + z 2 )

Count

Count

Count

N am es

Nam es

O v e r 4 .6

O ver 9 9

U n d e r .01

226

2

3

3 .9 - 4 .6

4 9 -9 9

. 0 2 - .0 1

133

5

3

8,

2 .9 - 3 .8

1 9 -4 9

.0 5 - .0 2

184

20

8

H 3 (3 ]

4

. 0 10.

9

C 4 9 M ^ )6 8 F . 12

79M

5

12M ,

6 . ® 1 1

18,

19

83F,

93F 95M

13M ,

27F

4 7 M .C 4 9 M ^ > 8 2 F

95m|

10F

W e h a ve c o u n te d th e n u m b e r o f ite m - b y - p e r s o n in te r a c tio n s w h ic h c o u ld fa ll w ith in ea ch r o w

o f T a b le 5 .1 1 .2 a n d m u ltip lie d

th is “ p o s s ib le ” c o u n t b y its im p r o b a b ility to

e s tim a te th e c o u n t w e m ig h t e x p e c t i f th es e d a ta f i t th e m o d e l. T h is w a s d o n e b y m u lti­ p ly in g ( 2 2 6 ) x

.0 1 s

2, (2 2 6

+

133) x

.0 2 s

( 2 + 5 ) a n d (2 2 6 + 1 3 3 + 1 8 4 ) x .0 5 s

( 2 + 5 + 2 0 ). T h e a c tu a l c o u n ts o b s e r ve d in th e d a ta a r e g ive n in C o lu m n 6 . T h u s w h en (b

-

d ) is o v e r 4 .6 lo g its w e e x p e c t a b o u t tw o im p r o b a b le r es p on s es a n d w e o b s e r ve

th r e e . W h en

(b

-

d ) is b e tw e e n 3 .9 a n d 4 .6 w e e x p e c t a b o u t fiv e im p r o b a b le res p on s es

a n d a g a in w e o b s e r ve th r e e . F in a lly w h e n ( b - d ) is b e tw e e n 2 .9 a n d 3 .8 w e e x p e c t a b o u t tw e n ty im p r o b a b le r es p o n s es b u t o b s e r ve o n ly e ig h t. T h e s e d a ta s eem to f it th e m o d e l r a th e r w e ll. W h e n w e s ca n th e 1 4 m o s t u n e x p e c te d ite m

a n d p er s o n r es p on s es g iven in T a b le

5 .1 1 .2 , w e s ee th a t th e y a r e w e ll d is p er s e d o v e r ite m s a n d p er s on s . O n ly Ite m s 3 a n d 7 a n d P er s o n s 4 9 M a n d 9 5 M a p p ea r tw ic e a n d th e s ex es a r e e q u a lly r e p r es en ted . W e m u s t c o n c lu d e th a t n o c le a r s ign o f s ys te m a tic m is fit h a s b ee n d e te c te d in th es e d a ta . N e ve r th e le s s , in o r d e r to u s e th e K C T B q u a lity

c o n tr o l,

we

w ill

p roceed

w ith

e x a m p le to s h o w th e a p p lic a tio n o f ite m

a fu r th e r

a n a lys is o f th e s ix m o s t u n e x p e c te d

r es p on s es . T h e s e r es p o n s es o f P er s o n s 4 9 M , 6 8 F , 7 9 M , 8 3 F , 9 3 F a n d 9 5 M to Ite m s 4 , 7, 8 , 9 , 1 0 a n d 1 2 a r e g ive n in T a b le 5 .1 1 .3 . F o r ea ch o f th es e u n e x p e c te d in c o r r e c t r e s p o n ­ s es, g ive n b y a b le p er s o n s o n ea s y ite m s , w e h a ve e n te r e d th e a p p r o p r ia te ( b v -

d j). W e

h a ve a ls o g ive n f o r ea ch ite m its c h a r a c te r is tic s o n th e K C T va r ia b le , n a m e ly its n u m b er o f ta p s , r ever s es a n d d is ta n c e a n d th e d e m o g r a p h ic ch a r a cter is tics o f s ex , a ge a n d g r a d e f o r ea ch p er s o n .

124

B E S T T E S T D E S IG N

T A R I F R 11 3

T H E S IX M O S T U N E X P E C T E D R E S P O N S ES O N KC TB (1 0 1 Persons B y 2 3 Item s)

P e rs o n A b ility

P ers on C h a ra c te ris tic s

It e m D i f f i c u l t y

d; bv

-4 .3

-4 .1

0 .4

0 (4 .7 ) *

1

1 .4

1 1 1 1 1

0 (5 .5 )

1 .9 2 .4 3 .6 4 .3

-

2.6 1 1

1 1 1 1

0 (4 .5 )

1 1 1

-

2.1

-

1 1 1 1

2.1

-

1 1 1 0 (4 .5 )

0 (5 .7 )

1

1 1

0 .1

N am e

Sex

A ge

G rade

1 1 1 1 1

49

M

16+

68

F

16+

79

M

16+

83

F

16+

93

F

16+

0 (4 .4 )

95

M

16+

12 + 12 + 12 + 12 + 12 + 12 +

Ite m C h a r a c te ris tic s N am e

#7

#4

#8

J9

# 10

# 12 4

Taps

4

3

4

4

4

R everses

0

0

1

2

2

D is ta n c e

3

3

5

2 6

5

7

* ( b - d ) = ( 0 . 4 ) - (-4 .3 ) = 4 .7

T h e d iffic u lty ch a r a cter is tics o f th e item s in revers es a n d d is ta n ce s h o w th e in cr ea s e w e w o u ld e x p e c t as th e item s b e c o m e m o r e d iffic u lt. A l l s ix ite m s a re o n th e ea s y e n d o f th e va r ia b le. T h e s ix p ers on s , o n th e o th e r h a n d , a re a ll r e la tive ly a b le a d u lts . T h is s u g­ gests th a t, i f a s ys tem a tic s ou r ce o f m is fit h a s b een d e te c te d h er e, it c o u ld o n ly b e a s lig h t te n d e n c y to w a r d s ca reles s n es s , o r la p s es o f a tte n tio n , a m o n g s o m e o ld e r p ers on s w o r k in g o n ite m s r a th er to o ea s y fo r th em . F it a n a lys is m a tr ices , lik e T a b le 5 .1 1 .3 , w h ic h b r in g to g e th e r th e p er s o n a n d ite m ch a r a cter is tic s

o f th e m o s t u n e x p e c te d

res p on s es , a re c o n ve n ie n t fo r s u p er vis in g th e

q u a lity o f ite m fu n c tio n in g . T h es e m a tr ices id e n tify a n d s u gg es t c o r r e c tio n s fo r th e s ys ­ te m a tic s ou rces o f ite m fa ilu r e s h o w n in th e d a ta . T h e c a lcu la tio n s n eces s a ry to eva lu a te u n e x p e c te d res p on s es ca n b e a c co m p lis h ed in th r e e w a ys . T h e fir s t tw o a re U C O N

b y c o m p u te r a n d th e h a n d m e th o d e x p la in e d in

C h a p ter 4. T h e th ir d w a y is a cr u d e, b u t q u ick , m e th o d w h ic h o fte n s u ffic es in p r a ctic a l w ork . T h is cr u d e m e th o d o f f i t a n a lys is con s is ts o f id e n tify in g a n d ca lcu la tin g o n ly th e fe w la r g es t z 2 ’s o b s er ved o n a n ite m a n d th en a d d in g to th em a 1 fo r ea ch o th e r p er s on ta k in g th a t ite m . T h is as s u m es th a t a ll o f th e d is tu rb a n ce o b s er ved in th a t ite m is d u e to its o u ts ta n d in g res id u a ls a n d th a t th e r es t o f th e p a tter n is m o r e o r les s as e x p e c te d .

C O N S T R U C T IN G A V A R IA B L E

125

T a b le 5 .1 1 .4 g ives a n illu s tr a tio n o f th is m e th o d . T h e r e w e h a ve ta k e n fr o m T a b le 5 .1 1 .3 ju s t th e s in g le la r g es t z v i2 o b s e r ve d in o u r K C T B d a ta a n d a d d e d to it a 1 fo r ea ch o th e r p er s o n ta k in g th a t ite m , in th is ca s e 1 0 0 . T h is g ives z v i2 + 1 0 0 = X 2 as th e ch i- s q u a re f o r th a t ite m a n d vj = X 2 / 1 0 0 as th e ite m m ea n s q u a re.

T o s ee w h e th e r th is c r u d e m e th o d ca n b e u s efu l, w e w ill c o m p a r e it w ith U C O N a n d th e h a n d m e th o d d e s c r ib e d in C h a p te r 4 , b u t n o w a p p lie d to th es e K C T B d a ta . In th o s e p r o c e d u r e s w e s u m a ll 1 0 1 a ctu a l z 2 ’s to m a k e o u r ite m f i t a n a lys is a n d th e n d iv id e th is s u m o f s q u a res b y its 1 0 0 d eg r ee s o f fr e e d o m to g e t th e m ea n s qu a res s h o w n in T a b le 5 .1 1 .5 .

I

_________________________

T A B L E 5 .1 1 .4

l

C R U D E F IT A N A L Y S IS F O R S I X K C T B IT E M S

I_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I

S in g le

C r u d e F i t S ta tis tic s

Ite m

A b i l i t y m in u s D i f f i c u l t y

It e m

C h i-S q u a r e

M e a n S q u a re

Nam e

D if f e r e n c e

z2

X2= lz2 + 100)

w = X2/100

9

5 .7

299

399

4 .0

4

5 .5

245

345

3 .5

7

4 .7

110

210

2 .1

10

4 .5

90

190

1 .9

8

4 .5

90

190

1 .9

12

4 .4

81

181

1 .8

____ ______ _______ ______ __ |

T A B L E 5 .1 1 .5

|_______________________

A C O M P A R IS O N O F I T E M Q U A L I T Y C O N T R O L M E T H O D S A P P L IE D T O K C T B

Ite m

UCON

H a n d F it

C ru d e F it

Nam e

M e a n S q u a re

M e a n S q u a re

M e a n S q u a re

9

3 .4 2

3 .1 6

4 .0

4

2 .6 4

2 .5 6

3 .5

7

1 .4 8

1 .4 0

2 .1

10

1 .4 6

1 .2 0

1 .9

8

1 .4 3

1 .1 2

1 .9

12

1 .3 6

0 .9 8

1 .8

T h e U C O N a n d h a n d f i t m e th o d s a p p r o x im a te o n e a n o th e r r a th e r c lo s e ly . A lth o u g h th e c r u d e f i t m ea n s qu a res a r e s o m e w h a t la r g er in m a g n itu d e , th e ir o r d e r is id e n tic a l to th e o th e r m e th o d s a n d th e ir va lu es a r e s u ffic ie n tly c lo s e to g e t a c le a r id e a c o n c e r n in g th e r e la tiv e f i t o f th es e s ix ite m s . T a b le 5 .1 1 .5 s u gges ts th a t th e c r u d e m e th o d ca n b e u s efu l f o r th e q u ic k a n a lys is o f ite m fu n c tio n in g .

126

B E S T T E S T D E S IG N

5.12 N O R M R E F E R E N C IN G T H E K C T V A R IA B L E W h ile n or m s a re n o m o r e fu n d a m en ta l to th e c a lib r a tio n o f ite m

b a n ks th a n a re

d is tr ib u tio n s o f p er s on h eig h ts to th e r u lin g o f ya r d s tic k s , it is u s u a lly u s efu l to k n o w va r iou s d em o g r a p h ic ch a r a cter is tics o f a va r ia b le d e fin e d b y a n ite m b a n k . S o m e o f th es e d em o g r a p h ic

ch a ra cteris tics

may

even

h a ve

n o r m a tive

im p lica tio n s

u n d er

p a rticu la r

cir cu m s ta n ces . B eca u s e o f a s h ift in em p h a s is , n o r m in g a va r ia b le in th e R a s ch a p p ro a ch ta k es m u ch les s d a ta th a n n o r m in g a tes t. W e n ee d o n ly u se en ou g h item s to es tim a te th e d es ir ed “ n o r m in g ” s ta tis tics . O n ce th e va r ia b le is n o r m e d , th en a ll p os s ib le s cores fr o m a ll p os s ib le tes ts d r a w n fr o m th e ca lib r a ted b a n k a re a u to m a tic a lly n o r m - r e fe r e n ce d th r ou g h th e va r ia b le. O fte n w e a r e s a tis fied w ith a m ea n a n d s ta n d a rd d e via tio n fo r ea ch c e ll in o u r n o r m ­ a tive s a m p lin g p la n . T h es e tw o s ta tis tics c o u ld b e es tim a ted fr o m a r a n d o m s a m p le o f 1 0 0 o r s o p er s on s ta k in g a n o r m in g te s t o f o n ly tw o item s . O f cou r s e, a s o m ew h a t lo n g e r tes t o f 1 0 o r 1 5 item s w ill d o a b e tte r jo b . N o t o n ly w ill th e es tim a tes b e b e tte r b u t th e e x tr a item s w ill y ie ld s ta n d a r d err ors a r ou n d th e n o r m in g s ta tis tics a n d th u s a te s t o f f it fo r th e p la u s ib ility o f th e d a ta . M o r e th a n 1 5 item s in a n o r m in g tes t, h o w e ve r , w ill s eld o m b e n eces s a r y. T h is m ea n s th a t w e c o u ld n o r m s ix d iffe r e n t va r ia b les s im u lta n eo u s ly b y a llo ­ c a tin g 1 5 item s to ea ch o f s ix s u b tes ts a d m in is ter ed as o n e 9 0 - item c o m p o s ite tes t. W e ca n es tim a te q u ic k n or m s fr o m fr e q u e n c y d a ta o n b a n k c a lib r a ted item s w ith o u t s co r in g o r m ea s u rin g th e in d ivid u a l p ers on s . T h is m a y b e u s efu l w h e n tr im m in g s a m p le d a ta is u n d es ira b le. I f w e s eek a p r o b a b ility s a m p le fr o m a p o p u la tio n , fo r e x a m p le , w e w o u ld r a th er n o t d is to r t th e s a m p le ’s s ta tu s b y e lim in a tin g s om e o f th e p ers on s s a m p led b eca u s e th e y ea r n ed ze r o o r p e r fe c t s cores . T h is n o r m in g p r o ce d u r e ca n b e a c c o m p lis h ed b y w o r k in g d ir e c tly fr o m th e m o d e l a n d th e o b s er ved n u m b er o f r ig h t a n s w ers to ea ch ca lib r a ted ite m .

,

1. F o r ea ch s a m p lin g c e ll in th e n o r m in g s tu d y, s e le c t fr o m th e ite m b a n k a s u ita b le s et o f K ca lib r a ted item s s u ffic ie n tly s p a ced in d iffic u lty d ; to c o v e r th e e x p e c te d a b ility d is p er s io n o f th a t p a r ticu la r s a m p lin g c ell. N o te th a t ea ch s a m p lin g cell, in p r in cip le, h a s its o w n in d ivid u a lly ta ilo r e d n o r m in g tes t. 2. A d m in is te r th is te s t o f K item s to a r a n d o m s a m p le o f N p ers on s fr o m th e s p eci­ fie d cell. 3. O b s er ve th e n u m b er o f p ers on s s; s u cceed in g o n ea ch item . 4 . C a lcu la te th e n a tu ra l lo g od d s h; o f th es e c o r r e c t a n s w ers Sj fo r ea ch ite m hj = fin [ S j / ( N - s ,) l

[ 5 .1 2 .1 J

i = 1 ,K

5. R eg r es s th es e lo g o d d s h; o n th e a s s ocia ted ite m d iffic u ltie s d ; o ve r th e K item s to o b ta in th e in te r c e p t A a n d s lo p e C o f th e lea s t s qu a res s tr a ig h t lin e. 6 . E s tim a te th e p o p u la tio n m ea n M a n d s ta n d a rd d e via tio n S D o f th a t c e ll’s a b ilities as M = - A /C S D = 1 .7 [(1 - C 2 ) / C 2 ] *

[5 .1 2 .2 ] [5 .1 2 .3 ]

W e w ill a p p ly th es e p r o ced u r es to th e K C T B s a m p le o f 1 0 1 p er s on s to s ee h o w w e ll th e y

r e c o ve r th e s a m p le m ea n a n d s ta n d a rd d e via tio n th a t w e h a ve a lr ea d y es tim a ted

fr o m th e m ea s u rem en ts o f ea ch o f th e 1 0 1 p ers on s to b e M * = 0 .1 9 a n d S D ' = 2 .4 4 .

C O N S T R U C T IN G A

V A R IA B L E

127

W e h a ve th e ite m d iffic u ltie s d ; f o r Ite m s 3 th r o u g h 2 5 a n d s o n e e d o n ly to c o m p u te th e n a tu r a l lo g o d d s h j o f o b s e r ve d c o r r e c t a n s w ers to ea ch o f th es e item s . T h e va lu es o f h;

a r e g ive n

in C o lu m n

3 o f T a b le 5 .1 2 .1

w ith

th e c o r r e s p o n d in g ite m

d iffic u ltie s in

C o lu m n 4 . R e g r e s s in g th es e lo g o d d s r ig h t a n s w ers o n th e ite m d iffic u ltie s o v e r th e 2 3 ite m s g ives u s a n in te r c e p t o f A = 0 .0 7 a n d a s lo p e o f C = - 0 . 5 6 . E q u a tio n 5 .1 2 .2 e s tim a te s th e s a m p le m ea n as M

=

-A /C

=

-0 .0 7 /-0 .5 6

=

0 .1 3 .

J

T A B L E 5 .1 2 .1

I

K C TB LO G O D D S C O R R E C T A N SW ER S A N D IT E M D IF F IC U L T IE S

1

2

_3_

4

P e rs o n s

Log O dds

It e m

It e m

S u c c e e d in g

C o rre c t

D iff ic u lty

N am e

si

h s= C n [S j/N - S j)]

di

6.20

3

98

3 .4 9

-

4

91

2 .2 1

-4 .1 1

5

82

1 .4 6

-2 .5 8

6

83

1 .5 3

-2 .7 6

7

92

2 .3 2

-4 .3 4

8

82

1 .4 6

-2 .5 8

9

78

-2 .0 6

10 11 12

78

1.2 2 1.2 2

68

0 .7 2

-1 .0 3

57

0 .2 6

-

13

66

0 .6 3

-0 .8 5

14

62

0 .4 6

-0 .5 2

15

73

0 .9 6

-1 .5 1

16

65

0 .5 9

-0 .7 7

0 .8 6

1 .9 3

17 18

-2 .0 6

0 .12

30

-

37

-0 .5 5

1 .3 6

19

29

-0 .9 1

20 21 22

20

-1 .4 0

2.0 1 2.8 8

16

-1 .6 7

3 .3 3

16

-1 .6 7

3 .3 3

23

-2 .4 5

4 .5 2

24

8 2

-3 .9 0

6 .2 7

25

3

-3 .4 9

5 .8 1

N = 101

M ean

0 .0 0

S ta n d a r d D e v ia t io n

3 .3 2

128

B E S T T E S T D E S IG N

E q u a tio n 5 .1 2 .3 es tim a tes th e s a m p le s ta n d a r d d e via tio n as SD

=

1 .7 [(1 - C 2 ) /C 2 ] *

=

1 .7 [(1 - 0 .3 1 1 / 0 .3 1 ] *

=

2 .5 4 .

T h e s e q u ic k n o r m reg res s io n es tim a tes o f 0 .1 3 fo r th e m ea n a n d 2 .5 4 fo r th e s ta n ­ d a rd

d e via tio n

com p a re

s a tis fa c to r ily w ith th e va lu es o f 0 .1 9 a n d 2 .4 4 c o m p u te d b y

m ea s u rin g ea ch o f th e 1 0 1 p er s on s a n d th en ca lcu la tin g th e ir m ea n a n d s ta n d a rd d e via tio n in th e u s u a l w a y. T h e p lo t o f th e lo g o d d s c o r r e c t a n s w er s h j a g a in s t th e ite m d iffic u ltie s d , in F ig u r e 5 .1 2 .1

s h ow s h o w w e ll th es e n o r m in g d a ta f i t th e s tr a ig h t lin e e x p e c te d b y th e m o d e l. F I G U R E 5 .1 2 . 1

T H E Q U I C K N O R M IN G M E T H O D O F E S T IM A T IN G S A M P L E M E A N S A N D S T A N D A R D D E V IA T IO N S A P P L IE D T O K C T B

LOG O D D S CORRECT ANSW ERS

+ 5

+ 4

®

- 6

- 5

- 4

- 3

- 2

- 1

©

+1

+2

+3

+4

+5

+6 IT E M D IF F IC U L T Y

- 1

-2

- 3 In t e r c e p t = S lo p e

0 .0 7

= -0 .5 6 - 4

- 5

6

6 .1

D E S IG N IN G T E S T S

IN T R O D U C T IO N

In C h a p te r 5 w e h a ve s h o w n h o w to es ta b lis h th e o p e r a tio n a l d e fin itio n o f a va r ia b le b y m ea n s o f a c a lib r a te d b a n k o f ite m s . T h e n e x t s tep is to fin d o u t h o w t o u s e th es e c a li­ b r a te d ite m s t o m a k e m ea s u r es . T o d o th is w e m u s t c o n s id e r tw o r e la te d q u es tio n s . F ir s t, w e n e e d t o fin d o u t h o w t o m a k e th e b e s t p o s s ib le s e le c tio n o f c a lib r a te d ite m s fr o m o u r bank

in

o r d e r to m a k e a n y p a r tic u la r m ea s u r em en ts w e h a ve in m in d m o s t e ffe c tiv e .

S e c o n d , g iv e n s u ch a s e le c tio n o f ite m s a n d a n o b s e r ve d p a tte r n o f r es p on s es to th e m , w e n e e d t o fin d o u t h o w to e va lu a te th e q u a lity o f th is o b s e r va tio n a n d , i f it is va lid , h o w to e x tr a c t fr o m it th e m ea s u r e w e s eek , to g e th e r w ith its s ta n d a r d e r r o r . I t is th e fir s t q u es ­ tio n ,

b e s t te s t d es ig n , w h ic h is th e m a jo r to p ic o f th is c h a p ter . C h a p te r 7 d ea ls w ith

m a k in g m ea s u res .

6 .2

THE M EASUREM ENT TARGET

W h en

w e p la n a m e a s u r e m e n t, th e r e m u s t b e a ta r g e t p er s o n o r g r o u p o f p er s on s

a b o u t w h o m w e w a n t to k n o w m o r e th a n w e a lr e a d y k n o w . I f w e c a r e a b o u t th e q u a lity o f o u r p r o p o s e d m e a s u r e m e n ts , th e n w e w ill w a n t to c o n s tr u c t o u r m ea s u r in g in s tr u m en t w ith th e s p e c ific s o f th is ta r g e t in m in d . In o r d e r to d o th is s ys te m a tic a lly w e m u s t b eg in b y s e ttin g o u t as c le a r ly as w e ca n w h a t w e e x p e c t o f o u r ta r g e t. W h er e d o w e s u p p os e it is lo c a te d o n th e va r ia b le ? H o w u n c er ta in a r e w e o f th a t a p p r o x im a te lo c a tio n ? W h a t is th e lo w e s t a b ility w e im a g in e th e ta r g e t c o u ld h a ve? W h a t is th e h ig h es t? H o w a r e o th e r p o s s ib le va lu es d is tr ib u te d in b e tw e e n ? S o m e tim e s w e h a ve e x p lic it p r io r k n o w le d g e a b o u t o u r ta r g et. W e, o r o th e r s , h a ve m e a s u r e d it b e fo r e a n d s o w e ca n s u g g es t its p r o b a b le lo c a tio n a n d d is p er s io n d ir e c tly in te r m s o f th e s e p r io r m ea s u r es o n th e va r ia b le a n d th e ir s ta n d a r d er r or s . S o m e tim e s w e ca n u s e ite m s c a lib r a te d a lo n g th e va r ia b le , s o m e o f w h ic h w e b e lie v e a r e p r o b a b ly ju s t r ig h t f o r th e ta r g e t, s o m e o f w h ic h a r e n e a r ly t o o h a rd a n d s o m e o f w h ic h a re n e a r ly t o o ea s y. T h e n w e ca n ta k e fr o m th e d iffic u ltie s o f th es e r e fe r e n c e ite m s r o u g h in d ic a tio n s o f th e p r o b a b le c e n te r a n d b o u n d a r ie s o f o u r ta r g e t. O n e w a y o r a n o th e r w e a s s em b le a n d c la r ify o u r s u p p o s itio n s a b o u t o u r ta r g e t as w e ll as w e ca n s o th a t w e ca n d e r iv e fr o m th e m th e te s t d es ig n w h ic h h a s th e b es t ch a n ce o f m o s t in c r e a s in g o u r k n o w le d g e . O b v io u s ly i f w e k n o w e ve r th in g w e w a n t to k n o w a b o u t o u r ta r g e t, th en w e w o u ld n o t h a ve to m ea s u r e it in th e fir s t p la c e. H o w e v e r , n o m a tte r h o w little w e k n o w , w e a lw a y s h a ve s o m e id e a o f w h e r e o u r ta r g e t is. B e in g as c le a r as p o s s ib le a b o u t th a t p r io r k n o w le d g e is es s en tia l f o r th e d es ig n o f th e b es t p o s s ib le tes t.

G ra h a m A . D ou g la s c o lla b o r a te d in th e p r ep a r a tio n o f p a rts o f th is ch a p ter . S ee W rig h t a n d D ou gla s , 1975a.

129

130

B E S T T E S T D E S IG N

A

ta r g e t s p e c ific a tio n is a s ta tem en t a b o u t w h er e o n th e va r ia b le w e s u p p os e th e

ta r g e t to b e. W e exp res s o u r b es t gu ess b y s p e c ify in g th e ta r g e t’s s u p p os ed cen ter , its s u p p os ed d is p er s ion a n d p erh a p s its s u p p os ed s h a p e o r d is tr ib u tio n . I f w e le t M

= o u r b es t gu ess as to ta r g e t lo c a tio n ,

S

= o u r b es t gu ess as to ta r g e t d is p er s io n ,

D

= o u r b es t gu es s as to ta r g e t d is tr ib u tio n ,

th en w e ca n d es cr ib e a ta r g e t G b y th e ex p r es s io n G (M , S , D ) a n d w e ca n s u m m a r ize o u r p r io r k n o w le d g e , a n d h en ce o u r m ea s u r em en t r eq u ir em en ts fo r a n y ta r g e t w e w is h to m ea s u re, b y gu es s in g, as w e ll as w e ca n , va lu es fo r th e th r e e ta r g e t p a r a m eter s M ,S a n d D . A p ictu r e o f a ta r g e t is g iven in F ig u r e 6 .2 .1 . G u es s in g th e s u p p os ed lo c a tio n M o f a ta r g e t is p e r fe c tly s tr a ig h tfo r w a r d . H o w e v e r , gu es s in g th e d is p er s ion S a n d th e d is tr ib u ­ tio n D fo r c e s u s to th in k th r o u g h th e d iffe r e n c e b e tw e e n a ta r g e t w h ic h is a s in g le p er s on a n d o n e w h ich is a g r o u p . F o r th e s in g le p er s o n , S ca n d es cr ib e th e e x te n t o f o u r u n cer­ ta in ty a b o u t w h e r e th a t p er s on is lo c a te d . T h e la r g er o u r u n c er ta in ty, th e la r g er S. I f w e ca n s p e c ify b ou n d a r ies w ith in w h ic h w e fe e l fa ir ly s u re th a t th e p er s on w ill b e fo u n d , w e ca n s et S s o th a t M ± k S d e fin e s th es e b ou n d a ries . T h e n , even i f w e h a ve n o clea r id ea a t a ll a b o u t th e d is tr ib u tio n D o f o u r u n c er ta in ty b e tw e e n th es e b ou n d a r ies , w e ca n n ever th eles s e x p e c t th a t a t lea s t (1 - 1 / k 2 ) o f th e p os s ib le m ea s u res w ill fa ll w ith in M ± k S . I f w e g o fu r th e r a n d e x p e c t th a t th e m ea s u res w e th in k p os s ib le fo r th e p er s on w ill p ile u p n ea r M , th en w e m a y e ve n b e w illin g to ta k e a n o r m a l d is tr ib u tio n as a u s efu l w a y to d es cr ib e th e s h a p e o f o u r u n c er ta in ty. In th a t ca s e w e ca n e x p e c t .9 5 o f th e p os s ib le m ea s u res to fa ll w ith in M ± 2 S a n d vir tu a lly a ll o f th e m to fa ll w ith in M ± 3 S . W e w ill r e fe r to th es e tw o ta r g e t d is tr ib u tio n s as th e T c h e b y c h e ff interv al a n d th e

norm al. W e m ig h t c o n s id e r o th e r ta r g e t d is tr ib u tio n s , b u t th es e tw o s eem to c o v e r a ll r ea s on a b le ta r g e t s h a p es r a th er w e ll. F o r e x a m p le , i f w e fe e l u n h a p p y a b o u t th in k in g o f o u r ta r g e t as a p p r o x im a te ly n o r m a l, th en it is u n lik e ly th a t w e w ill h a ve a n y d e fin ite a lter n a tive c le a r ly in m in d . Th u s , th e m o s t lik e ly a lter n a tive to a n o r m a l ta r g e t is o n e o f u n k n o w n d is tr ib u tio n , b es t ca p tu r ed b y a T c h e b y c h e ff in ter va l. T h is r e a liza tio n th a t a ll p os s ib le ta r g e t s h a pes ca n b e s a tis fa c to r ily r ep r es en ted b y ju s t tw o r ea s on a b le a lter n a tives is im p o r ta n t b eca u s e it m a k es a u n iq u e s o lu tio n to th e p r o b le m o f b es t te s t d es ign n o t o n ly p os s ib le b u t even p r a ctic a l. I f th e ta r g e t is a g r o u p r a th er th a n a n in d ivid u a l, th en w e m a y ta k e S a n d D to b e o u r b es t gu ess as to th e s ta n d a rd d e via tio n a n d d is tr ib u tio n o f th a t g r o u p . I f w e th in k th e g r o u p has a m o r e o r less n o r m a l d is tr ib u tio n , th en w e w ill ta k e th a t as o u r b es t gu ess fo r D . O th e r w is e w e ca n a lw a ys fa ll b a ck o n th e T c h e b y c h e ff in ter va l. F in a lly , w e m u s t b e e x p lic it a b o u t h o w p r ecis e w e w a n t o u r m ea s u r em en t to b e. A fte r a ll, th is is o u r m o tiv e fo r m ea s u rin g. I t is ju s t b eca u s e o u r p r es en t k n o w le d g e a b o u t o u r ta r g e t is to o a p p r o x im a te to s u it u s th a t w e w a n t to k n o w m o r e p r e c is e ly w h e r e o u r ta r g e t is a n d , i f it is a g r o u p r a th er th a n a n in d ivid u a l, m o r e p r ec is ely a b o u t its d is p er s ion . H o w e ve r , w h e th e r th e ta r g e t is a n in d ivid u a l o r a g r o u p , o u r d ec is io n a b o u t th e d es ired s ta n d a r d e r r o r o f m ea s u r em en t S E M w ill b e m a d e in ter m s o f in d ivid u a ls , fo r th a t, in th e en d , is w h a t w e a c tu a lly m ea s u re. In th e ca s e o f a on e- p er s on ta r g et, w e w a n t th e S E M to b e en o u g h s m a ller th a n S to r ew a r d o u r m ea s u r em en t e ffo r ts w ith a u s efu l in crea s e in th e p r ecis io n o f o u r k n o w le d g e a b o u t w h e r e th a t ta r g e t p er s on is lo c a te d . In th e ca s e o f a g r o u p ta r g e t w e w a n t to a ch ieve

D E S IG N IN G T E S T S

131

a n im p r o v e d e s tim a te n o t o n ly o f M , th e c e n te r o f th e g r o u p , b u t a ls o o f S , its d is p er s io n . T h e o b s e r va b le va r ia n c e o f m ea s u res o v e r th e g r o u p e s tim a tes n o t o n ly th e u n d e r lyin g va r ia n c e in a b ility S 2 b u t a ls o th e m e a s u r e m e n t e r r o r va r ia n c e S E M 2 . O u r a b ility to s ee th e d is p e r s io n o f o u r ta r g e t a g a in s t th e b a c k g r o u n d o f m e a s u r e m e n t e r r o r d e p e n d s o n o u r a b ility to d is tin g u is h b e tw e e n

th es e tw o c o m p o n e n ts o f va r ia n ce. S in ce th e y e n te r in to

th e o b s e r va b le va r ia n c e o f e s tim a te d m ea s u res e q u a lly , th e s m a lle r S E M 2 is w ith r e s p e c t to S 2 , th e m o r e c le a r ly w e ca n id e n tify a n d e s tim a te S 2 , th e c o m p o n e n t d u e to ta r g e t d is ­ p e r s io n . T h u s , f o r a ll ta r g e ts w e s e e k a n S E M c o n s id e r a b ly s m a lle r th a n S.

__ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ I

--------------------------

F IG U R E 6 .2 .1

I_ _ _ __ __ __ __ __ __ __ __

T H E P IC T U R E O F A T A R G E T

-------------------------

R E L A T IV E FREQUENCY P

4S

SHAPE D

6 .3

M ±2S

M ±3S

In t e r v a l

.7 5 +

.8 9 +

N o rm a l

.9 5

.9 9

T H E M E A S U R IN G T E S T

A te s t is a s e t o f s u ita b ly c a lib r a te d ite m s c h o s en to g o to g e th e r to fo r m a m ea s u rin g in s tr u m e n t. T h e c o m p le te s p e c ific a tio n o f a te s t is th e s e t o f a ll p a r a m eter s w h ic h ch a r a c­ te r iz e th e s e ite m s . B u t w h e n w e e x a m in e a p ic tu r e o f h o w a te s t w o r k s to tr a n s fo r m o b ­ s e r ve d s co r e s in to e s tim a te d m ea s u res , w e s ee th a t th e o p e r a tin g cu r ve is r a th e r s im p le a n d le n d s its e lf to s p e c ific a tio n th r o u g h ju s t a fe w te s t p a r a m eter s . W h en th e w a y o u r ite m s o p e r a te fits th e R a s ch m o d e l, th e n w e k n o w th a t th e o n ly ite m p a r a m eter s w h ic h w e n eed t o c o n s id e r in o r d e r to d e te r m in e th e o p e r a tin g ch a r a cter is tics o f a te s t a r e its ite m d iff i­ c u ltie s . W h en w e im p o s e a r e a s o n a b le fix e d

d is tr ib u tio n o n th es e d iffic u ltie s , th en n o

m a tte r h o w m a n y ite m s w e u s e, w e ca n r e d u c e th e n u m b e r o f te s t p a r a m eter s to o n ly th ree.

132

B E S T T E S T D E S IG N

F IG U R E 6.3.1

[

T H E O P E R A T IO N O F A T E S T

R E L A T IV E SCORE f = r /L

A B IL IT Y M EASURE . b

LM D —

db gf L O D = C f/L

In F ig u r e 6 .3 .1 w e ca n s ee fr o m th e s h a p e o f th e te s t o p e r a tin g cu r ve th a t its tw o o u ts ta n d in g fea tu r es a re its p o s itio n a lo n g th e va r ia b le, w h ic h w e w ill ca ll te s t height, a n d th e ra n g e o f a b ilitie s o v e r w h ic h th e te s t ca n m ea s u re m o r e o r les s a ccu r a tely, a ch a ra cter­ is tic ca u s ed p r im a r ily b y th e d is p er s io n o f th e ite m d iffic u ltie s , w h ic h w e w ill ca ll tes t

w idth. B u t h e ig h t a n d w id th d o n o t c o m p le te th e c h a r a c te r iza tio n o f a tes t. W h en w e lo o k m o r e c lo s e ly a t th e w a y th e te s t cu r ve tr a n s fo r m s o b s er ved s cores in to in fe r r e d m ea s u res w e s ee th a t th er e is a d is c o n tin u ity in o b s er va b le s cores w h ic h is g o in g to d eter m in e th e s m a lles t in c r e m e n t in a b ility w e ca n m ea s u re w ith a n y p a r ticu la r tes t. T h is lea s t m ea s u r­ a b le d iffe r e n c e L M D d ep en d s o n th e te s t’s lea s t o b s er va b le d iffe r e n c e L O D . S in ce th e lea s t ch a n ge p os s ib le in a te s t s cor e is o n e, th e L O D in r e la tive s co r e f = r/ L, m u s t b e 1 / L. In S e c tio n 6 .5 w e w ill fin d th a t th e s ta n d a rd e r r o r o f m ea s u rem en t, o r lea s t b elieva b le d if­ fe r e n c e , S E M a ls o d ep en d s o n th e n u m b er o f item s in th e tes t. In d e e d S E M = L M D 14. S o in o r d e r to fin is h c h a r a c te r izin g a te s t w e m u s t a ls o s p e c ify its length.

D E S IG N IN G T E S T S

133

F r o m th is w e s ee th a t a n y te s t d es ig n ca n b e d e fin e d m o r e o r les s c o m p le te ly ju s t b y s p e c ify in g th e th r e e te s t c h a r a c te r is tic s ; h e ig h t, w id th a n d le n g th . I f w e le t H

=

th e h e ig h t o f th e te s t o n th e va r ia b le , th a t is ,

th e

a ver a g e

d iffic u lty

o f its

s e le c te d

ite m s , W

=

th e

w id th

o f th e te s t in ite m d iffic u ltie s ,

th a t is, th e ra n g e o f its ite m

d iffic u ltie s

and L

=

th e

le n g th o f th e te s t in n u m b e r o f item s ,

th e n w e ca n d e s c r ib e a te s t d es ig n T b y th e c o n v e n ie n t e x p r e s s io n T ( H , W , L ) . In th e p r a c tic a l a p p lic a tio n o f b e s t te s t d es ig n , h o w e v e r , w e w ill h a ve t o a p p r o x im a te o u r b e s t d e s ig n T f o r a ta r g e t G fr o m a fin ite p o o l o f e x is tin g item s . In o r d e r to d is c r im ­ in a te in o u r th in k in g b e tw e e n th e b e s t te s t d es ig n T ( H , W , L ) a n d its a p p r o x im a te r e a liza ­ tio n in p r a c tic e , w e w ill d e s c r ib e a n a c tu a l te s t as t ( h , w , L ) w h e r e h

=

th e

a ve r a g e d iffic u lty o f its a ctu a l item s ,

and w

6 .4

=

a n e s tim a te o f th e ir a ctu a l d iffic u lty ra n g e.

TH E SHAPE O F A BEST TEST

A b e s t te s t is o n e w h ic h m ea s u r es b e s t in th e r e g io n w ith in w h ic h m ea s u r em en ts a r e e x p e c te d

to o c c u r . * M e a s u r in g b e s t m ea n s m ea s u r in g m o s t p r e c is e ly . A b es t te s t d es ig n

T ( H , W , L ) is o n e w ith th e s m a lle s t e r r o r o f m e a s u r e m e n t S E M o v e r th e ta r g e t

G (M ,S ,D )

f o r g ive n le n g th L ( o r , w h a t is e q u iv a le n t, w ith th e s m a lle s t L f o r a g ive n va lu e o f S E M ). “ O v e r th e ta r g e t” im p lie s th e m in im iza tio n o f a d is tr ib u tio n o f p o s s ib le S E M s . T h u s , a p o s itio n

w ith

r e s p e c t to th e m o s t lik e ly ta r g e t d is tr ib u tio n m u s t b e ta k e n b e fo r e th e

m in im iz a tio n o f S E M ca n p r o c e e d . W e b r in g th e p r o fu s io n o f p o s s ib le ta r g e t s h a p es u n d e r c o n tr o l b y fo c u s in g o n th e t w o e x tr e m e s — in te r v a l a n d n o r m a l. H o w s h a ll m in im iza tio n b e s p e c ifie d in ea ch ca s e? F o r a n o r m a l ta r g e t it s eem s r e a s o n a b le to m a x im iz e a ver a g e p r e c is io n , th a t is , to m in im ize a ve r a g e S E M , o v e r th e w h o le ta r g e t. T o d e c id e w h a t to d o f o r a n in te r va l ta r g e t, w e n e e d to k n o w h o w th e S E M va ries o v e r p o s s ib le te s t s co r es . W h en w e d e r iv e a n e x a c t fo r m f o r th e p r e c is io n o f m ea s u r em en t, w e fin d th a t f o r o r d in a r y tes ts w ith less th a n th r e e lo g its b e tw e e n a d ja c e n t item s , p r e ­ c is io n is a m a x im u m f o r m e a s u r e m e n ts m a d e a t th e c e n te r o f th e te s t a n d d ecrea s es

as

te s t a n d ta r g e t a r e in c r e a s in g ly o ff- c e n te r w ith r e s p e c t to o n e a n o th e r . F o r tes ts c en te r e d o n th e ir ta r g e ts th is m ea n s th a t m a x im iz in g p r e c is io n a t th e b o u n d a r ies o f a n in te r va l ta r g e t is a g o o d w a y t o m a x im iz e p r e c is io n o v e r th e ta r g e t in te r va l. S o f o r in te r va l ta r g ets w e w ill m a x im iz e p r e c is io n a t th e ta r g e t b ou n d a r ies . W h e n w e d e r iv e th e S E M 2 fr o m o u r r es p o n s e m o d e l w e w ill d is c o ve r th a t it is th e r e­ c ip r o c a l o f th e in fo r m a tio n

a b o u t a b ility s u p p lie d b y ea ch ite m r es p o n s e a ver a g ed o v e r

th e tes t. S in c e th e m o s t in fo r m a tiv e ite m s a r e th o s e n ea r es t th e a b ility b e in g m ea s u red

♦ A tte m p ts to m e e t th is r e q u ir em en t h a ve b een m a d e b y B ir n b a u m (1 9 6 8 , p p . 4 6 5 - 4 7 1 ). O u r idea s a re c o n s is ten t w ith h is e ffo r ts , b u t w e h a ve ta k en th em to th e ir lo g ica l a n d p r a ctica l con clu s ion .

134

B E S T T E S T D E S IG N

a n d th e lea s t in fo r m a tiv e a re th o s e fa r th e s t a w a y, th e p r ecis ion o ve r th e ta r g e t w ill d e­ p en d n o t o n ly o n th e d is tr ib u tio n o f th e ta r g e t b u t a ls o o n th e s h a pe o f th e tes t. Th u s , th e q u es tio n o f w h a t is a b es t tes t a ls o d ep en d s o n o u r ta k in g a p o s itio n w ith r e s p e ct to th e b es t d is tr ib u tio n o f te s t ite m d iffic u ltie s . W h a t a re th e r ea s on a b le p os s ib ilities ? I f w e w a n t to m ea s u re a n o r m a l ta r g et, th en a te s t m a d e u p o f n o r m a lly d is tr ib u te d ite m d iffic u ltie s o u g h t to p r o d u c e th e b es t m a x i­ m iza tio n o f p r ec is io n o v e r th e ta r g et. T h is is th e c o n c lu s io n im p lie d in B im b a u m ’s a n a ly­ sis o f in fo r m a tio n m a x im iza tio n (B im b a u m , 1 9 6 8 , p . 4 6 7 ). H o w e ve r , n o r m a l tes ts a re clu m s y to c o m p o s e . N o r m a l o r d e r s ta tis tics ca n b e u s ed to d e fin e a s et o f ite m d iffic u ltie s , b u t th is is ted iou s . M o r e p r o b le m a tic is th e o d d c o n c e p ­ tio n o f m ea s u rin g im p lie d b y an in s tr u m en t c o m p o s e d o f n o r m a lly d is tr ib u te d m ea s u rin g elem en ts . A n o r m a l te s t w o u ld b e lik e a ya r d s tic k w ith ru lin g s b u n ch ed in th e m id d le a n d s p rea d a t th e en d s . M ea s u rin g w ith s u ch a n ir r e g u la r ly r u led ya r d s tic k w o u ld b e a w k w a r d . In th e lo n g ru n , even fo r n o r m a l ta r g ets , o u r in te r e s t b ec o m es s p rea d o u t e ve n ly o v e r a ll th e a b ilities w h ich m ig h t b e m ea s u red b y a tes t. E q u a lly s p a ced item s a re th e te s t s h a p e w h ic h s erves th a t in te r e s t b es t. T h a t is th e w a y w e c o n s tr u c t ya r d s tick s . T h e te s t d es ign c o r r e s p o n d in g to an e ve n ly r u led y a r d s tic k is th e u n ifo r m te s t in w h ic h item s a re e ve n ly s p a ced fr o m ea s ies t to h a r d es t (B im b a u m , 1 9 6 8 , p . 4 6 6 ). T w o ta r g e t d is tr ib u tio n s , n o r m a l a n d in ter va l, a n d tw o te s t s h a pes, n o r m a l a n d u n i­ fo r m , p r o d u c e fo u r p os s ib le c o m b in a tio n s o f ta r g e t a n d tes t. W r ig h t a n d D ou g la s (1 9 7 5 a ) in ves tig a ted a ll fo u r c o m b in a tio n s r a th er e x te n s ive ly a n d fo u n d th e n o r m a l te s t to w o r k b es t o n th e n o r m a l ta r g e t a n d th e u n ifo r m te s t to w o r k b es t o n th e in te r va l ta r g et. W h en th e y c o m p a r e d th e n o r m a l a n d u n ifo r m tes ts o n n o r m a l ta rg ets , h o w e ve r , th es e tw o tes t s h a p es d iffe r e d s o little in th e ir m ea s u rin g p r ecis io n as to a p p ea r e q u iva le n t fo r a ll p r a c­ tic a l p u rp os es . T h u s th e b es t a ll p u r p os e te s t s h a p e is th e u n ifo r m tes t.

6 .5

T H E P R E C IS IO N O F A B E S T T E S T

N o w w e tu rn to th e r es p on s e m o d e l fo r m u la tio n o f th e s ta n d a rd e r r o r o f m ea s u re­ m e n t S E M s o th a t w e ca n b e c o m e e x p lic it a b o u t w h ic h te s t d es ign s m a x im ize p r ecis ion b y m in im izin g S E M . W e m u s t fin d o u t h o w th e te s t d es ig n T ( H , W , L ) in flu en ces S E M a n d h o w w e ca n v a r y th e te s t ch a r a cter is tics o f H , W a n d L in res p on s e to a ta r g e t s p e cific a ­ tio n G (M , S , D ) in o r d e r to m in im ize S E M o v e r th a t ta rg et. T h e r es p on s e m o d e l s p ecifies p fj

= e x p (b f - d | ) / [ 1 + e x p (b f - d j ) ]

[6 .5 .1 ]

w h er e

P fj

=

th e p r o b a b ility o f a c o r r e c t res p on s e a t f a n d i,

bf

=

th e a b ility es tim a te a t r e la tive s co r e f = r/ L ,

dj

=

th e ca lib r a ted d iffic u lty o f ite m i.

T h e m ea s u re b f is es tim a ted fr o m a te s t o f len g th L w ith item s | d;| fo r i = 1 , L th r ou g h th e e q u a tio n ( f o r d eta ils s ee S ectio n s 1 .5 a n d 3 .7 ) L

f= Z p f,/L , i

fo r f = 1 /L ,(L -

1 )/L

[6 .5 .2 ]

D E S IG N IN G T E S T S

135

w ith a s y m p to tic va r ia n c e

1 / 2 p f i (1 -

pfI) = S EM

f2

[ 6 .5 .3 ]

i

T h is is th e s q u a r e o f th e s ta n d a r d e r r o r o f m e a s u r e m e n t a t r e la tive s c o r e f .

W e s ee th a t S E M f d e p e n d s o n th e s u m o f p fj (1 -

p f i ) o v e r i. T h u s it is a fu n c tio n o f

b f a n d a ll th e d j. H o w e v e r , flu c tu a tio n s in p (1 - p ) a r e r a th e r m ild fo r p b e tw e e n 0 .2 a n d 0 .8 . T o e x p e d ite in s ig h t in to th e m a k e - u p o f S E M f w e ca n r e fo r m u la te

it s o th a t th e a ve r ­

a g e va lu e o f P fj (1 - P f j ) o v e r i is o n e c o m p o n e n t a n d te s t le n g th L is th e o th e r .

S E M f = | l / [ 2 p f i (1 -

P f. ) ] ( % ( 1 / L

),/2 =

( C f / L ) y*

[6 .5 .4 ]

i

in w h ic h

C f = [ 2 Pfi (1 ~ Pfl )/ L ] - 1 In th is e x p r e s s io n w e fa c to r te s t le n g th L o u t o f S E M in o r d e r to fin d a le n g th - fr e e e r r o r c o e ffic ie n t C f .

R e s u m in g o u r s tu d y o f th e o p e r a tin g cu r ve o f a te s t g ive n in F ig u r e 6 .3 .1 w e s ee th a t th e le a s t m ea s u r a b le d iffe r e n c e in a b ility L M D is ( “i y - ) L O D . S in ce th e le a s t o b s e r va b le in c r e m e n t in r e la tiv e s c o r e is 1 / L , a ll w e n e e d to c o m p le te th e fo r m u la tio n o f th e L M D is th e d e r iv a tiv e o f b w ith r e s p e c t to f w h ic h fr o m E q u a tio n s 6 .5 .1 a n d 6 .5 .2 is dh —

L = [ 2 p f i (1 - p f i ) / L

]_1

[6 .5 .5 .]

B u t th is is o u r e r r o r c o e ffic ie n t C f , th u s th e le a s t m ea s u r a b le d iffe r e n c e a t r e la tive s co r e f is

L M D f — Cf/L

[6 .5 .6 ]

and

S E M f = L M D f ’/a -

W ith S E M f in th is fo r m

( C f / L ) 1/a

w e n o te th a t, as fa r as te s t s h a p e is c o n c e r n e d , it is C f w h ic h r e ­

q u ir e s m in im iza tio n . T h is w ill b e tr u e w h e th e r w e u s e C min to m in im ize S E M f g iven L o r to m in im iz e L g ive n S E M f .

6 .6

T H E E R R O R C O E F F IC IE N T

N o w w e n e e d to k n o w m o r e a b o u t th is e r r o r c o e ffic ie n t C f. T h e es s en tia l in g r e d ie n t o f C f is th e e x p r e s s io n P fj (1 s p on s e

to

ite m

i

w ith

P fj) . T h is is th e in fo r m a tio n If; o n b f c o n ta in e d in a r e ­

d iffic u lty d j (B ir n b a u m , 1 9 6 8 , p. 4 6 0 - 6 8 ). Its a ver a g e va lu e

l #. = S l f l / L = C f - >

I

o v e r th e ite m s o n a te s t is th e a ver a g e in fo r m a tio n a b o u t b f p e r ite m p r o v id e d b y th a t te s t. T h u s C f is th e r e c ip r o c a l o f a ve r a g e te s t in fo r m a tio n . T h e g r e a te r th e in fo r m a tio n o b ­ ta in e d b y a te s t th e s m a lle r C f a n d h e n c e th e s m a lle r S E M f a n d s o th e g r e a te r th e p r ecis ion .

136

B E S T T E S T D E S IG N

W h a t va lu es ca n w e e x p e c t C f to ta k e? W e ca n a p p r o a ch th is q u es tio n in tw o w a ys : in ter m s o f th e in flu e n c e o f r ea s on a b le va lu es o f ( b f - d j) o n p fi a n d , fo r u n ifo r m tes ts , in te r m s o f te s t w id th W a n d th e b o u n d a r y p r o b a b ilitie s p f l fo r i = 1 , th e ea s ies t ite m , a n d p fL fo r i = L , th e h a r d es t ite m . T h e p r o b a b ility p fi is d e fin e d in E q u a tio n 6 .5 .1 . B eg in n in g w ith r ea s on a b le va lu es o f ( b f d iffe r e n c e is ze r o , th en p f i = 1/2, p f i (1 th en p f j = 1/ 8, p f i (1 -

d j) , w e s ee th a t w h en b f = d j a n d th e ir

p f i ) = 1/4 a n d C f = 4 , b u t w h en ( b f -

p f i ) = 1/9 a n d C f = 9 . ( N o tic e th a t C f = 9 w h en ( b f -

d ;) = - 2

d j) = + 2 a n d

P fi = 7/8 a ls o ). S in ce a n a vera g e ca n n e ve r b e g r e a ter th a n its m a x im u m e le m e n t n o r less th a n its m in im u m , w e ca n u se th es e fig u r es as b ou n d s fo r C f .

When

- 2 < ( b f - d|) < + 2

then

1 /8 ]

.

T h is U C O N p r o c e d u r e r e q u ir e s 3 o r 4 ite r a tio n s a n d a c o n v e r g e n c e c r ite r io n f o r s u cces s ive va lu es o f b r' s u ch as

I b ri+1 - b ri |
c_

E

k.

o

>

i—

(3

Q_

QSl

CO E

w

a. CN

D "D
*

o o

UJ

cq

co

r**

o o o

00 0) O o

o'

CN

i-

c

CD £

00 00 ff 5

w e r e je c t th e m ea s u r e as it s ta n d s a n d ta k e w h a te v e r s tep s w e ca n to e x tr a c t a “ c o r r e c te d ” m ea s u r e fr o m a n a c c e p ta b le s e g m e n t o f th e r e s p o n s e r e c o r d , i f o n e ex is ts .

T h e d e ta ile d s tu d y o f p e r s o n m is fit o f co u r s e d e p e n d s o n a d e ta ile d s tu d y o f th e a p p r o x i­ m a te n o r m a l d e via te s

zvi =

(2 xvi - 1) | exp [ ( 2xvi - 1)(d j - bv)/2 ] j

in th e r e s p o n s e r e c o r d in o r d e r t o tr a c k d o w n th e p o s s ib le s ou r ces o f ir r e g u la r ity.

S in c e th o s e p o r tio n s o f th e 2

z 2 w h ic h c o n tr ib u te m o s t to t a re th e la r g e p o s itive

te r m s , w e ca n s tr e a m lin e th e d e te r m in a tio n o f r e c o r d v a lid ity b y fo r m in g a q u ic k s ta tis tic fo c u s e d o n th e m o s t s u r p r is in g r es p on s es . T a b le 4 .3 .3 (a ls o g ive n as A p p e n d ix T a b le C ) s h o w s th a t th e d iffe r e n c e b e tw e e n p e r s o n m ea s u r e b a n d ite m d iffic u lty d m u s t b e o f th e o r d e r o f ± 2 .0 b e fo r e z 2 g r o w s la r g e r th a n 7 o r its p r o b a b ility b e c o m e s less th a n 0 .1 2 . T o r e a c h a p r o b a b ility f o r a g ive n r es p o n s e o f .0 5 o r les s w e m u s t r e la x o u r s ta n d a rd to a (d -

b ) d iffe r e n c e o f ± 3 p r o d u c in g a z 2 o f 2 0 .

I f w e c o n c e n tr a te o u r a tte n tio n o n s u r p r is in g r es p on s es f o r w h ic h |d - b| >

3 , th en

th e a c tu a l z 2 ’s m a y b e lo o k e d u p in T a b le 4 .3 .3 ( o r A p p e n d ix T a b le C ) a n d c o m b in e d w ith a n a ve r a g e v a lu e o f 1 f o r a ll th e r e m a in in g ite m s in th e r es p o n s e s e g m e n t to p r o d u c e a c r u d e 2 z 2 f o r w h ic h a c r u d e t ca n b e c a lc u la te d .

F o r e x a m p le , o v e r th e 9 r es p o n s es o f P er s o n 1 2 M , th e r e is o n ly o n e s u rp ris e. T h is is w h e r e ( d -

b ) = - 3 . 4 a n d z 2 = 3 0 . C o m b in in g th is va lu e o f 3 0 w ith e ig h t l ’s fo r th e

r e m a in in g e ig h t ite m s o f th e te s t g ives u s a c r u d e 2 z 2 = 3 8 a n d a cr u d e t = 5 .3 . T h is va lu e f o r t is n o t fa r fr o m th e m o r e e x a c t 4 .9 w e c a lc u la te d in T a b le 7 .8 .2 a n d lea d s u s to th e s a m e c o n c lu s io n o f a s ig n ific a n t m is fit.

In T a b le 7 .8 .4 w e s u m m a r ize th e r es id u a l a n a lys is f o r a ll n in e p er s on s . F o r ea ch p e r ­ s o n w e g iv e th e a b ility m ea s u r e a n d s ta n d a r d e r r o r fr o m

th e ir s e lf- ta ilo r e d s eg m en t o f

ite m s . N e x t w e g ive th e s u m o f s qu a res , d eg r ee s o f fr e e d o m , m ea n s q u a re a n d f i t s ta tis tic f o r ea ch p e r s o n ’s r e c o r d . F o r e ig h t ca s es w e fin d n o e vid e n c e o f m is fit a n d s o w e ta k e th e ir m ea s u r es as p la u s ib le. O n ly th e s e lf- ta ilo r e d s e g m e n t o f P er s o n 1 2 M ’s r e c o r d s h ow s a s ig n ific a n t m is fit. A s w e s a w in T a b le 7 .8 .2 , th is m is fit is d u e e n tir e ly to h is in c o r r e c t r e s p o n s e o n th e fir s t a n d ea s ies t ite m in h is r e c o r d . T h e re a s on f o r th is in c o r r e c t res p on s e m ig h t b e a fa ilu r e in te s t ta k in g o r a la p s e in fu n c tio n in g . In e ith e r ca s e w e a r e s till in te r ­ e s te d in th e b e s t p o s s ib le e s tim a te o f P er s o n 1 2 M ’s a b ility . T h e p r o b le m o f e x tr a c tin g th e b e s t p o s s ib le m ea s u r e fr o m a fla w e d r e c o r d w ill b e d is cu s s ed in S e c tio n 7 .1 0 .

170

B E S T T E S T D E S IG N

____________

T A B L E 7 .8 .4

__ _ _ _ _ _ _ _ _ _ _

F I T A N A L Y S I S O F T H E N IN E P E R S O N S M E A S U R E D B Y S E L F - T A IL O R E D T E S T S

S e lf- T a ilo r e d P erson G ro u p

A b ility

E rro r

N am e b

P re sc h o o l

P rim a ry

A d u lt

s

R e s id u a l A n a ly s is Sum o f

D egree s o f

M ean

F it

S q u ares

F re e d o m

S q u a re

S ta tis tic

lz 2

(L -1 )

V

t

3M

-5 .7

1 .3

1.1

3

0 .4

-1 .0

6F

-3 .8

0 .9

2 .4

5

0 .5

-1 .0

12M

-2 .8

0 .8

3 5 .0

8

4 .4

4 .9 *

29M

-0 .8

0 .7

1 7 .6

16

1.1

0 .3

35F

-0 .3

0 .7

1 6 .8

14

1 .2

0 .5

69M

1 .4

0 .6

2 0 .0

14

1 .4

1 .0

-0 .6

88M

3 .2

0 .9

7 .3

10

0 .7

98F

4 .4

0 .9

2.1

5

0 .4

-1 .1

101F

5 .2

1.1

1 .7

4

0 .4

-1 .0 * M i s f i t S ig n al

7 .9

D IA G N O S IN G M IS F IT

C o n s id e r a ga in th e 1 0 ite m te s t w ith ite m s in o r d e r o f in cr ea s in g d iffic u lty im a g in ed fo r T a b le 7 .8 .1 . W er e w e to e n c o u n te r th e p a tter n p r o d u c e d b y P er s on E , n a m e ly S co r e 0 0 0 1 0 1 1 0 1 1

5

w e w o u ld b e p u zzle d a n d w o n d e r h o w th is p er s o n c o u ld a n s w er th e h a r d q u es tio n s c o r ­ r e c tly , w h ile g e ttin g th e fir s t th r e e ea s ies t q u es tio n s in c o r r e c t. W er e th e y “ s lee p in g ” o n th e ea s y p o r tio n o f th e tes t? O n th e o th e r h a n d w e r e w e to e n c o u n te r th e res p on s e p a tter n S co r e 1 0 1 0 0 0 0 1 1 1

5

o u r s u rp ris e w o u ld b e as g r ea t, b u t n o w w e m ig h t b e in clin e d to e x p la in th e ir r e g u la r ity as th e r es u lt o f lu c k y “ gu es s in g” o n th e th r ee h a r d es t item s . B o th th e p r o b a b ilis tic n a tu re o f th e m o d e l a n d o u r e ve r y d a y e x p e r ie n c e w ith ty p ic a l res p on s e p a tter n s lea d s u s to e x p e c t p a tter n s w h ic h h a ve a c e n te r r e g io n o f m ix e d c o r r e c t a n d in c o r r e c t res p on s es . W h en w e e n c o u n te r a p a tter n lik e S co r e

1 1 1 1 1 0 0 0 0 0

5

M A K IN G

M EASURES

it th e r e fo r e

171

s tr ik es u s as “ t o o

good

to

b e tr u e . ” T h is u n e x p e c te d ly r e g u la r p a tte r n is

s o m e tim e s p r o d u c e d b y p er s o n s w h o w o r k v e r y s lo w ly a n d c a r e fu lly r e fu s in g to p r o c e e d t o th e n e x t ite m u n til th e y h a ve d o n e e v e r y th in g p o s s ib le to a n s w e r th e p r e s e n t ite m c o r ­ r e c tly . W e w ill r e fe r to th is p a tte r n as “ p lo d d in g . ” F in a lly , w e ca n a ls o id e n tify a s p ecia l fo r m c a lle d

o f “ s le e p in g ” w h ic h m ig h t b e tte r b e

“ fu m b lin g ” in w h ic h th e in c o r r e c t r es p on s es a re b u n ch ed a t th e b e g in n in g o f th e

te s t s u g g e s tin g th a t th e p er s o n h a d tr o u b le g e ttin g s ta r ted . T o s u m m a r ize , w e id e n tify th e fo llo w in g k in d s o f r es p o n s e p a tter n s : S core

1 1 1 1 0 1 0 0 0 0

5

1 1 1 0 1 0 1 0 0 0

5

“ fu m b lin g ”

0 0 0 1 0 1 1 0 1 1

5

“ g u es s in g ”

1 0 1 0 0 0 0 1 1 1

5

“ p lo d d in g ”

1 1 1 1 1 0 0 0 0 0

5

“ n o r m a l”

“ s le e p in g ” o r

In S e c tio n 7 .8 w e id e n tifie d a m is fittin g r es p o n s e p a tte r n f o r P er s o n 1 2 M . N o w w e w ill in ve s tig a te m is fittin g r e c o r d s , s u ch as th a t o f P er s o n 1 2 M , to s ee h o w th e d ia g n os is o f ir r e g u la r r e s p o n s e p a tte r n s m ig h t b e a c c o m p lis h e d . T h e s e lf- ta ilo r e d r es p o n s e p a tte r n fo r P e r s o n 1 2 M , w ith ite m s in o r d e r o f in cr ea s in g d iffic u lty , is S co r e 0 1 1 1 1 1 0 0 0

5

T h e e va lu a tio n o f th is r es p o n s e p a tte r n in T a b le 7 .8 .3 s h o w s a s ig n ific a n t m is fit, t = 4 .9 . In T a b le 7 .9 .1 w e s h o w th e r e s p o n s e p a tte r n f o r P er s o n 1 2 M a g a in a n d a d d fo r ea ch r e ­ s p o n s e th e p r o b a b ility p o f its o c c u r r e n c e u n d e r th e m o d e l. W e a ls o g ive h is r es p on s e p a tte r n in te r m s o f z ’s in a d d itio n to th e z 2 ’s. W h en w e p lo t th e z ’s fo r P er s o n 1 2 M in F ig u r e 7 .9 .1

w e s ee w h a t a “ s le e p in g ” o r “ fu m b lin g ” r es p o n s e p a tte r n lo o k s lik e . T h is

fig u r e d is p la ys th e s e g m e n t o f ite m s r e s p o n d e d to . E a ch ite m is s p a ced h o r izo n ta lly a lo n g th e

KCT

va r ia b le

a c c o r d in g to its d iffic u lty o n th e lo g it s ca le. Its ve r tic a l p o s itio n is

d e te r m in e d b y th e p e r s o n ’s s ta n d a r d r es id u a l z p r o d u c e d in r es p o n s e to th a t ite m . Th e

o b s e r ve d

r es p o n s e

p a tte r n

o f P er s o n

12M

in F ig u r e 7 .9 .1 s h ow s h o w

th e z

s ta tis tic in d ic a te s m is fit. Ite m 3 h a s a z = - 5 . 5 w h ile th e o th e r ite m s h a ve z ’s n ea r th e ir e x p e c te d va lu e o f ze r o . T h e e ff e c t o f Ite m 3 u p o n th e r es p o n s e p a tte r n o f P er s o n 1 2 M ca n b e h ig h lig h te d b y c o n s id e r in g th e t w o a lte r n a tiv e p a tter n s g ive n in T a b le 7 .9 .1 a n d F ig u r e 7 .9 .1 . In - a lte r n a tiv e p a tte r n A w e r e ta in a s c o r e o f fiv e b y e x c h a n g in g th e c o r r e c t r es p on s e o f “ 1 ” to Ite m 8 , a r e la tiv e ly h a r d ite m , w ith th e in c o r r e c t r es p o n s e o f “ 0 ” to Ite m 3 , th e ea s ies t ite m a tte m p te d . N o w w e h a ve th e p a tte r n S co r e

1 1 1 1 1 0 0 0 0

5

172

B E S T T E S T D E S IG N

in

in

5

7

in

in

00 «CN

-Q I *o

rr

in

cn |

o

co

o

r* o

i

i

X CN a

x

a> in CM I

o ’ I

co

o

n

o i

x

CN

a>

~o o >

CD

*5 o

CN I

CN

00

CD ID

O* o I

O)

’ o’ I

■ o

cn

co

o

o ’

i

T3 C CO

t—

a>

00

CO in

o ) o

i

oo

T—

CN I

E

d

^

E

0)

D IA G N O S IN G

" S L E E P IN G

c

co co

in

N

o

d

I

co T-‘

jn

co O

o

n a

in

«•?

*

co

cn

oo

T-* o

T

in

I

o

co o ’

I

8 o

c «3

O

*2

a 'v co (0 0) CO

I

^ f—

CN N

I X CN

o O)

o

CN

co o ’ i

I 2

-Q I 2

X

co

oo

o’

i

.Q 2

cn

in

o

i

CN CD

in

‘ o’

I

t—

I X CN

(M

N

IX r—

(N N

o> O

CN o ’

-Q I ■ o

I X CN X CN

a x CN

n co 'Z

O

ii

CN

u -

O

00 C

i

£

i

II

i—

a>

CN

M A K IN G

M EASURES

173

I

F I G U R E 7 .9 . 1

|

D I A G N O S I N G " S L E E P IN G '

b = -2 .8

P e rs o n

Q

12M (t= 4 .9 )

2

A lt e r n a t iv e B ( t = 0 .1 )

L o g it S c a le

-2

174

B E S T T E S T D E S IG N

T A B L E 7 .9 .2

R E S ID U A L A N A L Y S IS F O R " S L E E P IN G " P A T T E R N O F P E R S O N

12M

Case

Sum o f

M ean

F it

D e s c rip tio n

S q u are s

S q u a re

S ta tis tic

£ z 2

V

t

3 5 .0

4 .4

A lt e r n a t iv e A

4 .6

0 .6

-1 .0

A lt e r n a t iv e B

8 .5

1.1

0 .1

P ers o n 1 2 M

4 .9 *

(b == - 2 . 8 )

* M is f it S ig n al

z2

=

e xp [(2 x -

V

=

£ z 2 / ( L — 1)

t

1 ) (d — b>]

[Cn (v ) + (v - 1)1 [ L - 1 ) / 8 ] *

T h e m is fit s ta tis tics f o r th es e th r ee p a tter n s a re s u m m a r ized in T a b le 7 .9 .2 . T h e r e w e s ee th a t A lte r n a tiv e A has a t = - 1 . 0 in s tea d o f 1 2 M ’s t = 4 .9 . In A lte r n a tiv e p a tter n B w e e x ch a n g e th e c o r r e c t res p on s e o f “ 1 ” to Ite m 5 w ith th e in c o r r e c t res p on s e o f “ 0 ” to Ite m 1 5 , th e h a r d es t ite m in th e s eg m en t. T h is p r o d u ces th e a lter n a te r es p on s e p a tter n S co r e 1 1 1 1 0 0 0 0 1

5

In te r e s tin g ly en ou g h , th e m is fit fo r th e ex ch a n g e in p a tter n B is s m a ll, o n ly t = 0 .1 . T h is is b eca u s e Ite m

1 5 , w ith d iffic u lty d = - 1 . 5 , is n o t as h a rd in r e la tio n to P ers on 1 2 M ’s

a b ility o f b = - 2 . 8 as Ite m 3 , w ith d iffic u lty - 6 . 2 , is t o o ea s y. In T a b le s 7 .9 .3 a n d 7 .9 .4 a n d F ig u r e 7 .9 .2 w e illu s tr a te “ s lee p in g ” a n d “ gu es s in g” res p on s e

p a tter n s u s in g th e

o b s er ved

r e co r d

o f P er s on

8 8 M . T o ch a n ge h is res p on s e

p a tter n to a s leep in g p a tter n w e r ep la ce h is c o r r e c t res p on s es to tw o ea s y item s w ith in c o r r e c t res p on s es a n d s h ift th es e tw o c o r r e c t res p on s es to Item s 1 7 a n d 2 1 , th u s k eep ­ in g th e s cor e r = 6 . N o w w e h a ve th e res p on s e p a tter n S cor e

0 0 1 1 1 1 1 1 0 0 0 w h ic h is ch a r a cter is tic o f s leep in g . T h is p a tter n ea rn s t = 9 .1 in T a b le 7 .9 .4 .

6

M EASURES

175

O)

?

d

oo

q

«-t q

2

«

cm

o

co

in

l

«-

CO ‘

O

CO O)

I

n

q

q

o

cm

d

i

I

CN

o

CO

CO

q

*

I

co o

l

o

w * *

co

q

in

T" O

in

q

o