Chapter 5 SUBSET SELECTION OF THE T BEST POPULATIONS 5.1 Introduction In Chapter 4 we discussed the problenl of selecting the t "good" populations by choosing those t populations with the largest observed location parameters (and smallest observed scale paralneters). Lower bounds were obtained on the difference between the mininlum selected and maximum non-selected parameters, and upper and lower bounds were obtained on all parameters. As indicated in that chapter, inlproved bounds nlay be obtained if attention is restricted to a particular selection problem. This is investigated in the present chapter. Suppose that we wish to select a (random size) subset containing the t good populations, with a prespecified probability of correct selection (PCS). This selec­ tion goal has been previously investigated by Carroll, Gupta and Huang (1975) for the location and seale parameter eases, and by Gupta and Sobel (1962) for the particular case of nonnal variances. Carroll, Gupta and Huang's lower bound on the PCS for the location paranleter case is discussed in Section 5.2 and is compared with an alternative approach taken by Bofinger and Mengersen (1986). It is the latter approach that is studied in detail in this chapter. Although the new results still fail to provide an exact solution, the corresponding lower bound on the PCS is 80 CHAPTER 5. SUBSET SELECTION 81 shown to be less conservative than that of Carroll, Gupta and Huang. Special results are obtained for the case of normal means with common unknown variance and the relevant tables are presented. Bofinger and Mengersen investigated the conservativeness of their bound on the PCS for this particular case using sim­ ulation and, based on these results, conjectured a new lower bound for sufficiently large PCS. Carroll, Gupta and Huang's results for the scale parallleter case are compared in Section 5.3 with those obtained by Bofinger and Mengersen. Particular reference is lllade to the problem of selecting the t normal populations with the smallest variances and the necessary tables are produced. As for the location parallleter case, Bofinger and Mengersen's results are shown to be superior to those of Carroll, Gupta and Huang's. Gupta and Sobel's approach is also discussed; their conjecture (Conjecture 2 in their paper) is analagous to that made by Bofinger and Mengersen for the location parameter case and, if true, would result in a less conservative bound on the PCS than those of both Carroll, Gupta and Huang, and Bofinger and l'vlengersen. In Section 5.4, a related selection problem is considered. This involves selection of only good populations (rather than all good populations) and was investigated by Bofinger and Mengersen. An example illustrating the results of Section 5.2 (for the location parameter case) is given in Section 5.5. 5.2 Selection of all t good populations - location parameter case Consider k randolll variables ."'Ki' i = 1, .. , k. frolll populations 7ri wit.h continuous distribution functions F( x - OJ. As usual, we order the 8i and ."'Ki values as follows: ."'KR(l) < -"'KR (2) < ... < XR(k) wit.h p( .) and R(·) parametric and randolll permutation functions respectively. CHAPTER 5. SUBSET SELECTION 82 5.2.1 A previous approach The goal considered by Carroll, Gupta and Huang (1975) is to select a subset of populations, based on the Xi values, of size at least t, containing the t good populations, that is, those 1ri corresponding to the t largest ()i values. The selection goal, then, is: Goal G1: To select populations 1ri Vi E rt, where rt = {p(k - t + l),p(k - t + 2), .. ,p(k)} . (5.1 ) The selection procedure R3 considered by Carroll, Gupta and Huang may be written as: Procedure PI: Select. a subset of populations 1ri Vi E G( d), where G( d) = {i : Xi > XR(k-t+l) - d} (5.2) with d a positive constant. A correct. selection, CS (referred to by Carroll, Gupta and Huang as a correct decision, CD) is defined to be t.he event CS = {rt ~ G(d)} , (5.3) that is, that. all t populations in rt are selected. The problenl, then, is to determine the smallest value of d such that the PCS is at least. a specified value, so that. PCS = p(){cs : PI} = p(){rt ~ G(d)} 2:: P* ( 5.4) - - Lower bound on the pes Under their procedure R3 (given by PI here)~ Carroll, Gupta and Huang were unable to find an explicit. expression for the LFC of Ineans (at which the PCS attains its infimul11), so instead they derived a lower bound on the PCS by considering the event E = {_Y i - () i > X j - () j - d \j i E It, j rt It} (5.5 ) CHAPTER 5. SUBSET SELECTION 83 with '"'It defined in (5.1). Since the event that these bounds are correct is a subset of the event CS of correct selection, the probability of the event E is a lower bound for the PCS under procedure Pl. As discussed in Chapter 4, E has probability given by (5.6) and hence Carroll, Gupta and Huang showed that (5.7) A Monte Carlo study conducted by Carroll, Gupta and Huang for the case of normal means with equal known variance, however ~ showed that t.his lower bound is quite conservative, due to the use of the "catch-all" event E. Upper bound on the pes An upper bound on the infimum of the PCS was also given by Carroll, Gupta and Huang in t.heir Lemma 3.2. Under the configuration the aut.hors showed that where /(1- F(x))j(l - F(x + b))t-j-l(F(x) - F(x - d))t-j-l (F(x + 8)l-2t+ j+ldF(x) /(1 -F(x - 8))j(F(x - 8) - F(x - 8 - d))t- j (1 - F(X)Y-j-l(F(x)l-2i+ jdF(x) . ( 5.8) CHAPTER 5. SUBSET SELECTION 84 Critical values satisfying (5.8) for given P* were tabulated by Carroll, Gupta and Huang (in their Table III) for the normal means case. This upper bound on the infimum of the PC S was improved by Bofinger and Mengersen (198 b), as discussed later in this chapter. 5.2.2 A new approach Lower bound on the pes The problelll of selecting the t best populations was considered using a different approach by Bofinger and Mengersen (1986). These authors generalised a result of Hsu (1984b) and although, like Carroll, Gupta and Huang, they too could not derive a LFC of llleans,the lower bound on the PCS obtained under their method was shown to be substantially less conservative than that of Carroll, Gupta and Huang. As discussed in Chapter 2, for the particular case of multiple comparisons with­ the best (t = 1), Hsu (1984) used Fabian's notion of a pivotal event to construct an upper bound for of max(O~ Xi - XR(k) + d) Vi -I R(k) . Bofinger and Mengersen generalised this approach and constructed strict upper bounds for of lllax(O, "'Yi - ... YR(k-tli) + d) Vi = 1, .. , k where ~:rR(k-tli) is the tth largest of so that { ~Y~(k-t) if 'l.· =. R_( J..~ - t +, 1), .,.' R( k) XR(k-tli) = ,~ , _X R (J..-t+l) If 2 - R(lL .. ~R(k - t) . (5.9 ) ( 5.10) (5.11) CHAPTER 5. SUBSET SELECTION 85 The event that these upper bounds are correct may be written as CU B = {Oi - 0 p(k-t+l) < max( 0, -¥i - XR(k-tli) + d) 'Vi = 1, .. , k .} (5.12) Bofinger and Mengersen proposed the following two lemmas to: a) denlonstrate that the event CUB is a. subset of the event CS of correct selections under PI, and b) derive a lower bound on P!l{ CU B} and hence derived a theorem (Theorem 5.1 below) which shows that the lower bound for PO{ CUB} is a lower bound for PO{ C S I PI}. - - Lelllllla 5.1 Using the definitions (5.3) and (5.12) for CS and CUB respectively, CUB~CS. Proof Since Oi - Op(k-t+l) < 0 'Vi ~ It , the upper bound (5.10) for 0i - Op(k-t+l) is correct 'Vi ~ It. Sil1ularly, Hence the expression (5.12) may be written as CUB {Oi - Op(k-t+l) < -Xi - -X"R(k-tli) + d 'Vi E It} C {O < -Xi - .. :rR(J,,-tli) + d 'Vi E rt} . Since -'Yi - _XR(J,,-tii) > 0 whenever _XR(J,,-tli) = )tR(k-t) (that is, i = R( It- - t + 1 L .. , R( It:)), then CUB c {O < "Yi - _YR (J,,-t+l) + d 'Vi E'Tt} {J: i > -YR(k-t+l) - d Vi E It} cs. (5.13) CHAPTER 5. SUBSET SELECTION 86 Alternatively, notice that procedure PI involves selecting all of those populations with positive upper bounds, as given by (5.10). Lelnma 5.2 P8{CUB} ~ (k - t + 1)Pk- 1,t-l(d) - (k - t)Pk,t-l(d) + Pk- t+1,1(d)-1 (5.14) where Pk,t( d) is defined in (5.6). Proof The proof may be divided into three parts. In part a) the LFC of fl is derived; in part b) the variable XR(k-tli) is considered and in part c) the distribution functions of int.erest. are investigated~ using results frolll parts a) and b). a) First the LFC of fl is derived, being the configuration of nleans which min­ imises Pfl{CU B}. The expression for CUB (given by (5.12)) nlay be rearranged to obt.ain Notice that i) .¥i - 8i has the saIne distribution for all values of fl, and ii) XR(k-tli) has the saIne distribution for all values of 8i . (5.15) Consider 8p(k-t+l) fixed. Now XR(k-tli) is stochastically increasing in 8j Vj #- i, so if any 8j is increased, XR(k-tli) tends to increase and hence the probability in (5.15) is decreased. By the ordering of the () values, so the LFC of fl is given by (5.16) where ,; = {p( 1), p( 2 L .. , p( A: - t + I)} . CHAPTER 5. SUBSET SELECTION 87 b) Consider now XR(k-tli), i E '"'It (since i ~ '"'It may be ignored for the purpose of investigating (5.15)). Under the LFC, X a.s . d * i ---+ 00, l ~ '"'It . Hence, if i = p(k - t + 1), by considering (5.11) we can say X a.s. X R(k-tli) = ~ax j. J ..tYR(k-tli) - Op(k-t+l) - d Vi E '"'It} P{Yi > 2nd largest 1j - d Vi rt '"'I; jE'Yt" and Y~p(k-t+l) > ~ax Ij - d} J tt'Yt > P{}~ > 2nd l~rgest 1Tj - d Vi rf:. ,;} JE'Yt" + P{1Tp(k_t+l) > l~ax l'-j - d} - 1 . J rt"rt CHAPTER 5. SUBSET SELECTION Now P{Yi > 2nd l~rgest Yj - d Vi rt I;} JEr; (k - t + l)(k - t) J(l - F(y - d))t-l(l - F(y))(F(y))k-t-1dF(y) (k - t + l)(k - t) J(l - F(y - d))t-l(F(y))k-t-1dF(y) -(k - t + l)(k - t) J(l - F(y - d)r-1(F(y))k-tdF(y) (k - t + 1)Pk-1,t-dd) - (A~ - t)Pk,t-l(d) where Pk,t(d) is defined in (5.6). Also, P{l'"p(k-t+l) > ~ax Yj - d} - 1 J~rt Pk-t+1,1 (d) . Renee, at the LFC, the result follows. Theorem 5.1 88 Po{CS I PI} ~ (k - t + 1)Pk - 1 ,t-l(d) - (k - t)Pk ,t-l(d) + Pk-t+1,1(d) -1 . (5.18) Proof The proof follows from Lenlnlas 5.1 and 5.2. Relnark For t = 1, the right hand side of expression (5.18) reduees to whieh is Gupta's (1956) result for subset selection of the (one) best. population. CHAPTER 5. SUBSET SELECTION 89 Upper bound on the infimum of the PCS Under the LFC (5.16) for the event CUB and by virtue of (5.17), it is clear that the pes reduces to Pfl{ C S I PI} = Pk-t+l,l( d) . (5.19) This, then, is an upper bound for the infiInuIn of Pfl{ C S I PI} (and hence a lower bound on the value of d such that the right hand side of (5.18) is equal to P*.) For the nonnal means case, comparison of Bechhofer's (1954) tabulated values of Pk - t +1.1(d) and Carroll, Gupta and Huang's Table III establishes the superiority of (5.19) over the latter authors' upper bound. For exaInple, with P* = p(}{CS I PI}, for {P*, k, t} = {0.90, 10, 2}, the d value satisfying (5.19) for the normal means case is 2.98, whereas the corresponding value satisfying (5.8) is 2.74. Silnilarly, for {0.95, 6, 3}, t.he new upper bound produces t.he value d = 2.92, whereas Carroll, Gupta and Huang's bound produces d = 2.41. 5.2.3 Normal means case Suppose that F(.) is the normal distribution with 111ean () and common unknown variance (J"2. Let Xi be the mean of n independent observations froIn 7ri and let S2 be the usual pooled estimator of (J"2, so that vS2 / (J"2 has a chisquared distribution on v degrees of freedom. Lower bound on the PCS With goal G1 given in Section 5.2.1, we consider the selection procedure: Procedure P2: Select a subset of populations 7ri, i E G(d), where G( d) = {i : Xi > XR(k-t+l) - dSn- I / 2 } (5.20 ) with d a positive constant. As before, correct selection is defined to be the event (5.3), with 1t and G( d) defined by (5.1) and (5.2) respectively, and CUB becomes the event CUB = {(}i - Bp(k-t+l) < max(O,Xi - XR("~-tli) + dSn- 1 / 2 ) Vi = 1, .. ,k}. (5.21) CHAPTER 5. SUBSET SELECTION 90 U sing these definitions, Bo:finger and Mengersen proved the following theorem. Theorem 5.2 ( 5.22) where (5.23 ) and (5.24 ) with <1>( • ) the standard normal distribution function and G v (' ) the distribution func­ tion for S / a . Proof The proof follows that given for Theorem 5.1, working conditionally on 5 and then taking the expectation over 5. Discussion of the lower bound The sharpness of PL as a lower bound for Po{ C 5 I P2} was investigated by Bofinger and Mengersen using simulation. A Monte Carlo experiment was conducted in which the NAG routines G05DHF and G05DDF were used to simulate, respectively, a sample variance 52 and k sanlple means Xi distributed N(8 i , l.0), i = 1, .. , k. The saIne set. of randolll variables were used for each {P*, k, t, v} combination, wit.h each cOlnbination considered a separate experiment. Three P* values (0.90~0.50~0.10) and two v values (30,00) were considered. For each of these six cOlnbinations of P* and v, five (k, t) combinat.ions were considered ((k~t) = (6,2), (6,5L (10,2L (10,3),10,8)), giving thirty {P\k,t~v} combinations in all. Each experinlent was repeated 100,000 times, giving a standard error for each simulated probability of less than 9.5E-4 for P* = 0.90,0.10 and 1.6E-3 for P* = 0.50 . CHAPTER 5. SUBSET SELECTION 91 Table 5.1: Comparison of the PCS under configurations C1 and C2 for v = 00. (P*, k, t) (0.90,10,8) (0.50,10,3) (0.10,10,8) c C1 C2 C1 C2 C1 C2 00 0.958 0.958 0.738 0.738 0.679 0.679 3.0 0.961 0.971 0.749 0.760 0.684 0.704 1.0 0.980 0.997 0.813 0.851 0.728 0.826 0.5 0.984 0.998 0.807 0.838 0.725 0.818 0.1 0.982 0.993 o ,..,'"':~ ., . ..., 0.795 0.699 0.792 0.0 0.981 0.981 0.757 0.757 0.689 0.689 In light of the LFC given by (5.16), two configurations of fi were considered: C1: 8; = I ~ C2: 8; = { ~ with 0 ::=; C ::=; 00. i::=;k-t+1 i=k-t+2 i>k-t+2 i::=;k-t+l i'2k-t+2 (5.25 ) (5.26) As expected, for all {P*,v,k,t} combinations considered, both Pfi{CUB} and P.Q{ C S I P2} were larger under configuration C2 than under configuration C1, for all values of c. This difference is illustrated in Table 5.1 for p(){ C S I P2}, v = 00 and three {P*, k, t} cOlllbinations. Hence, only configuration C1 was considered further. For the thirty {P*. k~, t, v} cOlllbinations, the silllulated behaviour of p(){ C S I P2} and Po{ CU B} under configuration C1 was investigated over the range of c values (0 ::=; c ::=; coL using values of d such that P* = PL. As expected, the values of P.Q{ CU B} consistently increased in c and were less conservative than the corresponding values of Pfi{ C S I P2}. The behaviour of Pfi{ C S ! P2} for v = 00 is illustrated in Figure 5.1 for P* = 0.90,0.10. The sallle behaviour was observed for v = 30 and P* = 0.50. CHAPTER 5. SUBSET SELECTION 0.980 0.970 0.960 N n. iii 0.950 o CL 0.940 0.930 p* 0.90 ., .......... - ... jI< " 0.920 L-_......L. __ ..l.-_---L __ .....L-__ L--_......L. __ ..l.-_--' 0.0 0.65 0.60 N 0.55 I Q. iii o Ii:' 0.50 O . .s I I I 0.5 , jI< , / 1.0 1.5 p* --'-----+- ,"""- ' .... 2.0 c 0.10 ;I/' ...... 2.5 ... 3.0 3.5 {k,t) -16,21 +- -+ 10,21 ~ -* 10,3} 4.0 - - . O . .w 1L-_-.l.... __ ..L-_---L __ -L. __ L-_-.l.... __ -L-_---l 0.0 O.S 1.0 1.5 2.0 c 2.5 3.0 3.S Figure 5.1: pes at configuration C1 with v 4.0 ex:> and P* 92 0.90,0.10 CHAPTER 5. SUBSET SELECTION 93 As demonstrated by Figure 5.1, the probability appeared to have a single max­ imum over the values of c, for all {P*, k, t} conlbinations (and for both v values considered). The minimum probability appeared to depend on the value of P*; for small P* (certainly less than 0.5 according to Figure 5.1), it was at c = 0 while for larger P* this minimum was at c = 00. The P* value for which equal minima were found at c = 0 and c = 00 was dependent on the {v, k, t} combination. This seems to indicate that the LFC for Po{ C 5 I P2} is given by (5.25), with c = 0 for small P* and c = 00 for larger P* values. Hence Bofinger and Mengersen conjectured that, for sufficiently large P* values, the LFC for Po{ CU B}, given by (5.16), is also the LFC for Po{ C S I P2}. This conjecture is discussed in lllore detail later in this section. Bofinger and Mengersen also numerically compared their lower bound PL with that of Carroll, Gupta and Huang and with the siluulated values of Pft{ CU B}, for P* = 0.90, v = 30,00 and various {k, t} combinations, under configuration (5.16). These cOlllparisons are detailed in Table 5.2. The value of Po{ C S I P2} is given by P~~~+l,l (d) using the appropriate value of d satisfying PL ;; 0.90. Carroll, Gupta and Huang's probability of correct selection, PCGH { C S} say, is given by P~~)t+l,l (d) using the value of d such that p~~) (d) = 0.90. The latter values were cOluputed using the package RS-MCB developed by Gupta and Hsu (1984). Table 5.2 shows that Bofinger and Mengersen's lower bound for Po{ CU B} is not too far above the nOlllinal value of 0.90. However, the lower bound for Po{ C 5 I P2}, while an ilnprovelnent on that of Carroll, Gupta. and Huang, is still quite conserva­ tive, especially as h' and t increase and as P* decreases. For P* = 0.10 in particular, both lower bounds on Po{ CUB} and Po{ C S : P2} are quite unsatisfactory. For- - - tuna.tely, such probabilities are rarely used in practice (although this is no reason for not requiring further iluprOVell1ent in the bound on the PCS.) Conjecture regarding the LFC Based on the sill1ulation results discussed above, Bofinger and 11engersen conjectured that, for large P*, the infimum of the pes is given by P~~)t+l,l (d). They noticed, however, that this conjecture does not hold for sI11all P*, as shown by the silTIulation results, since the LFC is not given CHAPTER 5. SUBSET SELECTION t k 2 4 8 10 3 6 10 l;~~ 10 8 10 Table 5.2: Simulated and calculated results under configuration Cl P{ CU B} and P{ C S I P2} use values of d satisfying PL = 0.90 . PCGH { C S} uses values of d satisfying p~~)( d) = 0.90 . -,' v = 00 ! I v = 30 P{CUB} P{CS} PCGH{CS} t k P{CUB} P{CS} PCGH{CS} 0.900 0.922 0.944 2 4 0.903 0.923 0.943 0.900 0.924 0.946 8 0.903 0.925 0.944 0.902 0.925 0.946 10 0.903 0.926 0.944 0.902 0.935 0.961 3 6 0.904 0.936 0.960 0.903 0.937 0.962 10 0.904 0.939 0.961 0.900 0.947 0.972 5 6 0.900 0.948 0.971 0.903 0.950 0.975 10 0.906 0.951 0.974 0.903 0.958 0.982 8 10 0.904 0.959 0.981 94 by (5.16) in these cases. For large P*, however, it was pointed out that if the conjecture were true, the percentage points d would be reduced. For example, for {P*,v,k,t} = {0.75,oo,10,5}, the value of d = 2.63 satisfying PL = 0.75 would be replaced by 1.97 (satisfying P~~~+1.1(d) = 0.75) and, for {0.95,30,6,3}, d would de­ crease fronl 3.30 to 3.03. (Notice that the values of d appropriate to Carroll, Gupta and Huang's procedure for these combinations would be 2.94 and 3.66 respectively.) Bofinger (1988) has shown that this conjecture does hold for t = k - 1. Hence, for (k, k - 1) combinations, the infimuIll of the PCS is given by pi~) (d). Tables for the lower bound on the pes - normal means case COlllPutation of the expression for P~~)(d), given by (5.24), was discussed in Chapter 4. As indicated in that chapter, a satisfactory approximation to p~~) (d) may be obtained by writing (5.24) as and using Bechhofer~s (1954) tables to approxilllate Pk.t(dS/a), the inner integral of (5.24). This approach is llluch faster than alternatives based on Gaussian quadra­ ture and gives percentage points accurate to one in the second decilllal place. The CHAPTER 5. SUBSET SELECTION 95 method was again elnployed to compute values of d such that PL is within a small amount 5 = 2.0E - 5 of P* = 0.75,0.90,0.95. The results are presented in Table B.3 for selected values of v, k and t. (This table is located in Appendix B.) Us­ ing Bofinger's (1988) results, the values of d in Table B.3 for t = k - 1 are those satisfying P* = pJ~) (d). As an overall check on the consistency of the values presented in Table B.3, a response surface was fitted over k and v to each (P*, t) combination, as described in Chapter 4. The outliers were verified by computing the percentage points of p~~)( d) using the routine Pktnud described in Chapter 4. None of the tabulated values, when compared with the new percentage points, differed by Inore than 0.01 and always the tabulated values were the more conservative. Interpolation For values of v not included in Table B.3, and for P* values be­ tween 0.75 and 0.95, the interpolation methods suggested for Table B.1 ( described in Chapter 4) apply. The behaviour of these methods is similar to that described in Chapter 4 for Table B.1. 5.3 Selection of all t good populations - scale Pa­ rameter Case Consider h' independent randon1 variables Yi ~ i = 1, .. , k, fron} populations 7ri with continuous distribution functions F( y / 1/'d. 'V'le order the 'If'i and Yi as follows: The selection goal is now to select a subset, based on the Ii values, of "good" populations, with a good populat.ion now identified as possessing one of the t slnall ­ est 1/'i values. Goal G3: With pes at least P*, select. all populations 7fi, i E Ttl, where '"ye = {p( 1), p(2), .. , p( t)} . ( 5.27) CHAPTER 5. SUBSET SELECTION 96 This problem of selecting populations based on scale parameters has been con­ sidered under the subset selection approach by Gupta and Sobel (1962) for the normal variances case, and by Carroll, Gupta and Huang (1975) and Mengersen (1987) for more general distributions. In the latter paper, Mengersen extended in an obvious way the results obtained by Bofinger and Mengersen for the location paranleter case, described in Section 5.2. Mengersen's results are detailed in this section. In all three papers, the selection procedure is: Procedure P3: To select all populations 7ri ~i E G(c), \Ii E G(c), where G ( c) = {i : Ii < c -llR( t)} , with 0 < c < 1 constant. (This procedure is considered in Section 7.2 of Gupta and Sobel's paper and corresponds to procedures R4 of Carroll, Gupta and Huang and Q1 of Mengersen.) A correct selection then becomes the event CS = {,t l ~ G(c)} which, again, is the event that all t good populations are selected. 5.3.1 A previous approach For the scale parameter case, Carroll, Gupta and Huang adopted the same approach as that taken for the location parameter problenl. Since they were unable to find a LFC for £ (such that, P 1}1 { C 5 I P3} reaches an infilllunl), Carroll, Gupta and Huang proposed the following lower bound: where (5.28 ) Gupta and Sobel (1962) also proposed this bound for the nonnal variances case. CHAPTER 5. SUBSET SELECTION 97 5.3.2 A new approach Bofinger and Mengersen's procedure for the selection of the t best populations in the location parameter case was extended by Mengersen to the scale parameter problem. Strict lower bounds on Vi = 1, .. , k of (5.29 ) were constructed, where lR(tli) is the tth sl11allest of Then the selection procedure P3 is equivalent to selecting all populations with lower bounds (5.29) less than 1. Theorem 5.3 where Q~l/j( c) is defined in (5.28). , Proof Let _Y = -log Y and d = -log c. Then the distribution functions become location-type functions F( x - Bi ) with Also, procedure P3 is equivalent to selecting 7ri if - log}i > -log l~R(t) + log c , that is, if -'Yi > tth largest _\ - d ~ which is procedure PI described in Section 5.2. Hence we l11ay apply the proofs of Lennnas 5.1 and 5.2 and Theorenl 5.1 to obtain the required result.. CHAPTER 5. SUBSET SELECTION 98 5.3.3 Normal variances case Let Ii be the sample variance based on n observations from each of the normally distributed populations Jri, i = 1, .. , k. Then the distribution function F(y/,l/'i) is that of a chisquared variable on v = n - 1 degrees of freedom and,VYi is distributed ~ as chisquared on v degrees of freedom. a For this particular case, the values of c such that the right hand side of expression (5.30) is within a slnall value b = 2.0E - 5 of P* = 0.75,0.90,0.95 were presented by Mengersen (1987) and are displayed in Table B.4 (see Appendix B) for various {v, k, t} combinations. The lnethod of conlputing expression (5.28) is described in Section 4.3. It is expected that the tabulated values are correct to within one in the second decinlal place. By comparing the values of c in Table B.4 with those of Table B.2 (for Qtl( c)), for given {P*, v, k, t} it can be seen that the new procedure is an improvement over that of Carroll, Gupta and Huang (since larger c values are obtained in the former case). For example, the value of c in Table B.4 corresponding to {P*, v, k, t} = {O. 75,8,5, 2} is 0.346, whereas the corresponding value under Carroll, Gupta and Huang's procedure (froln Table B.2) is 0.319. Sinlilarly, Mengersen's value of c for {0.95, 60, 10~ 3} is 0.520 whereas Carroll, Gupta and Huang's value is 0.488. The overall consistency of the tabulated values was checked in the same way as for Table B.3, by fitting a response surface and examining residuals. Those values corresponding t.o large residuals were recolnputed and, where possible, were cOlnpared wit.h Carroll, Gupt.a and Huang's or Gupta and Sobel's tabulated values. Interpolation The methods of interpolation suggested for Table B.2 apply to Table B.4, for Zi values not induded in the table and for P* values between 0.75 and 0.95. Results for large degrees of freedom For large degrees of freedoln, Mengersen considered the use of the normal approximation to the chisquared distribution, to derive the required values of c using the corresponding values of d in Table B.3 (for the location paralneter case). Pairs of values of c (obtained from Table B.4) CHAPTER 5. SUBSET SELECTION P* 0.75 I ! I I i I 0.95 I v 10 e 30 1 ~ I C 60 Ie 120 I ~ I 10 1 ~ I I I 30 e C 60 e Table 5.3: Comparison of c and c c is value satisfying PL = P* c is value satisfying normal approximation. (k, t) ( 4,2) ( 7,2) (10,2) (4,3) (7,3) (10,3) (7,5) 0.443 0.336 0.293 I 0.518 0.338 0.287 0.389 0.412 0.311 0.272 1 0.475 0.302 0.256 0.340 0.632 0.545 0.509 0.686 0.544 0.500 0.583 0.623 0.540 0.505 1 0.671 0.534 0.492 0.566 0.724 0.655 0.624 0.766 0.653 0.616 0.683 0.719 0.654 0.624 0.759 0.648 0.613 0.674 0.796 0.743 0.719 0.828 0.740 0.711 0.763 0.794 0.743 0.721 0.824 0.737 0.710 0.760 0.263 0.205 0.181 0.312 0.212 0.183 0.247 0.215 0.161 0.139 0.255 0.160 0.134 0.184 0.475 0.418 0.393 I 0.518 0.424 0.393 0.456 0.460 0.402 0.377 0.500 0.404 0.374 0.430 0.594 0.544 0.521 0.629 0.548 0.520 0.575 (10,5) 0.296 0.252 0.504 0.490 0.618 0.611 0.712 0.709 0.193 0.134 0.401 0.375 0.527 1120 1 ~ 1 ~:~~~ 1 ~::~~ 1 ~::~~ 1 ~:~~~ 1 ~::~: 1 ~::!~ 1 ~::~! 1 ~::!~ I I c I 0.690 I 0.652 I 0.631 I 0.717 I 0.651 I 0.629 I 0.671 I 0.631 (10,7) 0.330 0.277 0.531 0.512 0.640 0.630 0.729 0.724 0.216 0.361 0.423 0.394 0.546 0.531 0.652 0.645 99 and C (obtained using the nonnal approxinlation) for various {P*, v, k, t} combina­ tions are given in Table 5.3 for illustration. As expected, the deficiencies of this approximation that were noticed in Chapter 4 were more apparent for the results of the present chapter. It appears that the usual normal approximation may not give satisfactory results for this selection goal for t > 1. As noticed in Chapter 4, a better method of approxinlating the c values for large degrees of freedom, for g{'neral values of t, renlains to be investigated. A conjectured lower bound For the normal variances case, Gupta and Sobel (1962) conjectured that the LFC of '!E is given by { 0 i=l, .. ,s-l 7/'p(i) = , 1f'p(k) 'i = s~ .. , k - 1 CHAPTER 5. SUBSET SELECTION 100 where s(l :::; s :::; t) is a non-decreasing function of P*. Hence, for large P*, s is set equal to t and the required value of c is that value satisfying Q~~t+l,l(C) = P*. This is analagous to the conjecture proposed by Bofinger and Mengersen (1986) for the location parameter case, which is described in Section 5.2. Gupta and Sobel proved their conjecture for the special case {v, k, t} = {2, 3, 2}. For this case, the exact value of c, for P* between 1/3 and 2/3, is shown to be c = (3(1 - P*) + J{(l - P*)(17 - P*)} )/(4(1 + P*)) and~ for P* between 2/3 and 1, the value is given by c = (1 - P*)/ P* . If this conjecture were true for all values of v, k and t, then the critical val­ ues of c appropriate to Procedure P3 could be considerably reduced. For exam­ ple, for {P*, v, k, t} = {0.75, 8, 5, 2} the value of c would be 0.398, compared with Mengersen's value of 0.346 (from Table B.4) and Carroll, Gupta and Huang's value 0.319 (from Table B.2). For {0.95, 60, 10, 3}, c would be 0.537, compared with 0.520 from Table BA and 00488 from Table B.2. 5.4 Selecting only good populations 5.4.1 Location parameter case As well as selecting a subset containing all good populations with a prespecified pes, Bofinger and Mengersen considered the problem of selecting a subset contain­ ing only good populations. The goal in this case becomes: Goal G4: Select only those 1ri, i E rb where rt is defined by (5.1). The corre- sponding procedure proposed by Bofinger and Mengersen is: Procedure P4: Select all1ri' i E G*( d), where G*( d) = {i : -~i > -~R(k-t) + d} (5.31) with d a positive constant satisfying the probability requirenlent ( 5.32) CHAPTER 5. SUBSET SELECTION 101 Taking an approach similar to that described In Section 5.2, Bofinger and Mengersen constructed strict lower bounds for of min(O, Xi - XR(k-tli) - d) Vi = 1, .. , k with .XR(k-tli) defined in (5.11). The event that these lower bounds are correct is given by CLB = {OJ - Op(k-t) > max(O,Xi - XR(k-tli) - d) Vi = 1, .. ,k.} As in Section 5.2, a lower bound is obtained on Pf!{ C 5 I P4} by: (5.33 ) (5.34) i) establishing that the event CLB is a subset of the event CS of correct selections, and ii) finding a lower bound for PO{CLB}. i) With the definitions ofCS and CLB given by (5.3) and (5.34) respectively, an obvious modification of the proof of Leillma 5.1 may be made t.o show that CLB~CS. This may alternatively be demonstrated by noticing that Procedure P4 involves select.ing all populations with non-negative lower bounds, as given by (5.33). ii) Selection of only "good" populations (those 7ri, i E rd is equivalent to rejection of all (J...~ - t) "bad" populations (those 7ri, i rf rt). Hence, a lower bound for PO{ C LB} n~ay be obt,ained by replacing t with (k - t) in the right. hand side of expression (5.18). Bofinger and Iviengersen thus showed t.hat. (5.35 ) For the case of nonnal means with COllunon unknown variance, these values lllay be obtained for a part.icular {P*, v, k, t} cOlllbination froin Table B.3, using the value corresponding to {P*, v, k, k - t}. CHAPTER 5. SUBSET SELECTION 102 5.4.2 Scale parameter case Selection of a subset containing only good populations, with a good population 7ri in this case possessing one of the t smalle~t scale parameters ~'i, was considered by Mengersen (1987) using an extension of Bofillger and Mengersell's (1986) results. The selection goal considered in this case is: Goal G5: Select all 7ri, i E tt', where III is given by (5.27) and the corresponding procedure is, Vi = 1, .. , k: Procedure P5: To select those 7ri, i E G*(c)~ where G*(c) = {i: li < c- 1YR (i+1) } with probability of correct selection satisfying P~{CS I P5} = P{G*(c) E ttl} . Strict upper bounds l11ay be constructed for Vi = 1, .. , k of ll1.ax( 1, c Yi / YR(t+lli)) with YR(t+lli) defined as the (t + 1 )th slllallest of The following points may be ll1.ade: i) The procedure P 5 corresponds to selecting those populations with upper bounds equal to 1. Hence the event that these upper bounds are correct is a subset of the event of correct selection. ii) A lower bound on P~, {C 5 I P4} is given by p~/,{CS i P4} ~ (t + l)Qr~l,t(C) - tQtl+l(c) + Q~~)l.l(C)-l As for the location paral11eter case, this lower bound is obtained by replacing t wi th (h' - t) in expression (5.')0) (which is a lower bound 011 correct selection of all t good populations and hence only (l.- -- t) bad populations). CHAPTER 5. SUBSET SELECTION 103 5.5 Example Consider the example, given in Chapter 3, of eight candidates (in this case, lecturers) evaluated by 21 judges (students) for particular awards, with two of these candidates eliminated at an earlier stage. The final scores for the six remaining candidates are given by: Carole Debbie Wayne Wendy Russell Janice 134.7 125.2 118.7 111.5 89.5 70.2 with S2 equal to 107.04 on 140 degrees offreedolll. As proposed in Chapt.er 3, these students lllay wish to select two "best" lecturers, t.hat is~ the lecturers with the largest true scores fJp (6) and fJp (5). Suppose that they wish to achieve this with 90% confidence of correct selection. Froln Table B.3, with {P*, v, k, t} = {0.90, 140, 6, 2}, d = 2.81. Hence dsn- 1j2 = 6.35 and the students 11lay select Carole and Debbie, with 90o/c) confidence that these are indeed the best lecturers with respect to the specified attribute . At the 95o/cl confidence level. however, the value of d froln Table B.3 is 3.22 and the set of lecturers {Carole, Debbie, Wayne} must be chosen in order to be 95o/c) confident that the two best lecturers are included. If the students decided instead t.o select only good lecturers, then at the 90% level, Carole and Debbie would again be chosen. At the 95% level, however, only Carole could be chosen. It is int.eresting t.o compare these selection decisions based on Procedure P2 with those resulting frolll Carroll, Gupta and Huang's procedure R3 and t.hose possible if the conjecture in the previous section were true ( procedure Peonj , say). Selections under each of these procedures are illustrated in Table 5.4. It can be seen that Bofinger and l\1engersen's procedure in this exalllple out per­ fonns Carroll. Gupta and Huang's procedure. but could be 111uch ilnproved if the fonner authors' conjecture were proven to be true for all k. CHAPTER 5. SUBSET SELECTION 104 Table 5.4: COlnparison of selections under Procedures R 3 , P2 and Pconj . P* Procedure Select all good Select only good led,urers led,urers 0.90 R3 Carole,Debbie,Wayne Carole I P2 Carole,Debbie I Carole,Debbie i Pconj Carole,Debbie , Carole,Debbie 1 0 . 95 R3 Carole,Debbie,Wayne I Carole P2 Carole,Debbie,Wayne , Carole I i Pconj Carole,Debbie I Carole,Debbie Chapter 6 SIMULTANEOUS COMPARISONS WITH A CONTROL AND WITH THE BEST 6.1 Introduction In the previous two chapters, emphasis has been on selecting "best" populations, with respect to a particular population paranleter. In In any experinlents, however, comparisons are not Inerely required with the best population, but also with sonle known or unknown control. The experiinenter in this situation may wish to use such comparisons in order to select populations which are demonstrably better than the control and not. demonstrably worse than the (unknown) "best" population. If no such population exists, t.he experiinenter Inay wish to select only the control. In any case, comparisons are required which provide useful infoflnation about the relative performance of the experimental populat.ions cOll1pared with the control and with the best. A possible approach that the experinlenter lnay take is to include the control population with the experinlental populations and apply Hsu's (1984) confidence 105 CHAPTER 6. SIMULTANEOUS COMPARISONS 106 bound procedure for comparison of all populations with the (unknown) best. These intervals may then be used to select a subset containing the best population. If the control is included in the selected subset, however, the only conclusion that the experimenter may make is that no population is demonstrably better than the control. If the control is not in the selected subset, it Inay be concluded that at least one of the selected populations is demonstrably better than the control, but the experilllenter will be unable to identify the particular populations. Alternatively, Dunnett's (1955,1964) one- and two-sided confidence intervals may be employed to COlllpare all the experilIlental populations with the control and hence to select a subset of populations which are delnonstrably better than the control. This approach, however, reveals no infonnation about the best popula­ tion. Brostrom's (1981 ) step-down procedures are also useful for cOlnparisons with a control, but again yield no comparison with the best. Simultaneous cOlllparison of populations with a control and with the best has received attention recently from such authors as Turnbull (1976), Bechhofer and Turnbull (1978) and Bristol and Desu (1985). In all of these papers, the indifference zone approach to selection was employed. For the location parameter case, in which a population 7ri is declared to be "better" than 7rj if the corresponding location parameter ei is larger than ej , all three methods require the the experimenter to specify five constants {bl , b2 , ll3' Po, PI} such that: (i) the probability of selecting no population as deillonstrably better than the control is at. least Po whenever Bp(k) - eo < b}, where Bp(k) is the paralne­ tel' corresponding to the (unknown) best population and eo is the paralIleter corresponding to the control, and (ii) the probability of correctly selecting the best population is at least PI when­ ever ()p(J.~) - eo ~ b2 and ep(J.~) - ()p(k-I) ~ b3 • All three lllethods further assume that the parallleter corresponding to the con­ trol population is fixed and known. Bechhofer and Turnbull (1978) considered only the nonnal 11leans case, whereas Bristol and Desu's (1986) Inethod is designed for the case of cOlllparing guarantee tilIles of two-paralnet.er exponential distributions. CHAPTER 6. SIMULTANEOUS COMPARISONS 107 If the scale parameter is common and known, both sets of authors proposed a one­ stage procedure, but if the scale parameter is common and unknown, two-stage procedures were recommended. Finally, if no populations are found to be better than the control, the only conclusion that may be made under these rules is that "no populations are selected". There are a nUlnber of disadvantages with these methods. - Often the experilnenter cannoL or is reluctant to, specify the five paranleters required for ilnplelnentatioll of the procedures. - In l11any cases t.he controlillean l11ay be unknown. - The scale parailleters of the populat.ions lllay be unknown but a two- st.age experilllent. nlay be ilnpract.ical. - More ilnportantly. however, the experilnenter Inay require Inore detailed in­ fonnation about the cOlnparative perfonnance of the populations. Results expressed as nUlllerical cOlllparisons between experilnental populations and the cont.rol, and between experimental populat.ions and the best., Inay be re­ quired rather than merely the final selection stateillent. If a population is of interest for other reasons (such as cost efficiency) and can be shown to be "not too far from best" or "not much worse than the control", this may be useful to the experilnenter. An alternative approach to this probleln has been proposed by Bofinger and lVlengersen (1988). Using a confidence bound approach, bounds were obtained on cOl11parisons of all experinlental populations with the control. For those populations found to be del110nstrably better than the control. intervals were constructed for corllparisons with the (unknown) best population. It is these results that are dis­ cussed in detail in this chapter. The bounds proposed by Bofinger and Mengersen are detailed in Section 6.2 and in Section 6.3 yarious selection decisions based on t.he cOlllparisons are considered. Three cases are identified: (i) cOlllparisons with the control are required to be as sensitive as possible. CHAPTER 6. SIMULTANEOUS COAfPARISONS 108 (ii) comparisons with the best are required to be as sensitive as possible, and (iii) comparisons with the control and the best are required to be equally sensitive. Bofinger and Mengersen considered in detail the special case of normal means with conlnlon unknown variance and produced appropriate tables for each of the three cases above. These results are presented in Section 6.4, with an example in Section 6.6. In Section 6.5 two related problenls are considered: (i) simultaneous cOlllparisons with the control and with the t best and (ii) selection of the best provided that its mean does not lie in a prespecified "undesirable" region. 6.2 Confidence Bounds 6.2.1 Notation Let ~~i' i = 1, .. , p, be random variables froin populations ?Ti with continuous distri ­ bution functions (cdf) Fl(X - ed. Let Xo be a randoln variable from ?To , the control population with which the other p populations are to cOlnpared, with cdf Fo(x - ( 0 ), 80 unknown. If 80 is known (so that a known standard is employed instead of a con­ trol), then Fo( x - ( 0 ) is replaced by a degenerate distribution concentrated at the known 80 , Let fl = (8o, 8J , ... , 8p ) and let n be the adlnissible space for fl. The () and _'-"" yalues are ordered as follows: (6.1 ) (6.2 ) where p( . ) and R( .) are paralnetric and randoIIl pennutation fUIlctions respectively. The populat.ion IT p(p) is then defined as the true best population (excluding the control ). The set of populations which are truly better t han the control is denoted by {IIi},isy, where r = {i : f}/ - eo > O} (6.3 ) CHAPTER 6. SIMULTANEOUS COMPARISONS 109 with #, = 9 and with the complementary set given by l' = {1,2, .. ,p} -,. 6.2.2 Construction of bounds Bofinger and Mengersen (1988) identified those populations 7ri, i = 1, .. ,p which are observed to be demonstrably better than the control by the set: G = {i : Xi - Xo > c}, c > ° ( 6.4) with G = {1,2, ... ,p} - G the complementary set. These populations may also be identified by constructing lower bounds on Vi = 1, .. ,p of Li=min(O,Xi-Xo-c) . (6.5) Those 7ri,i E G, will have zero (rather than negative) lower bounds and those 7ri, i E G, will have negative lower bounds. In Bofinger and Mengersen's procedure, the latter populations are assullled to be of little interest to the experimenter, so are not cOll1pared with the best population. For 7ri, i E G, intervals for e· - 111ax e· 1 j#i,O J of Ii = [l111n( 0, Xi - I?-l~X ~"¥j - b), max( 0, ~"¥i - .max. Xj + b)] (6.6) )#1.0 JEG,#z were const.ructed by Bofinger and Mengersen, with b another positive constant. No upper bound for the interval Ii is defined if #G = 1 (since InaxjEG,#i Xj is undefined). Bofinger and Mengersen noted that if such an upper bound is at ­ tempted, perhaps by using lllaXj#O,i ~"¥j instead of lllaXjEG4i Xj, a smaller event than Eb (given below by (6.9)) lllust be considered, resulting in reduced confidence if the same c and b values are used. CHAPTER 6. SIMULTANEOUS COMPARISONS 110 The joint confidence for the bounds given by (6.5) and (6.6) is given in Bofinger and Mengersen's Theorem 2.1, which is detailed below in Theorem 6.1. Theorem 6.1 With G, Li and Ii defined in (6.4), (6.5) and (6.6) respectively, the joint confi­ dence for the bounds Li and Ii is given by inf P{Oi - 00 > Li Vi = 1, .. ,p and Oi - max OJ E Ii Vi E G} fiE 0 . j:ji,O min l~-r+l (c )Ur (b) O~r~p (6.7) with ( 6.8) Proof Let independent events Ec and Eb be expressed as: {-~i - Oi < -yo - 00 + c Vi E ;;y} } {_Yi - Oi < .o¥ p(p) - 0 p(p) + b Vi E " f- p(p)} (6.9) so that (6.10) Notice that Oi - 00 > 0 Vi E , . Hence, if the event Ec occurs, then for all p populations, Li is a correct lower bound for Oi - 00 ; that is, We need, then, to show that (6.11) Notice the following: CHAPTER 6. SIMULTANEOUS COMPARISONS (i) B p(p) - maxj::pp(p),O Bj > 0 and henee the lower bound given by Ii is eorreet for i = p(p). (ii) Bi-Bp(p) <0 Vii-p(p),O and henee the upper bounds given by Ii are eorreet for i f- p(p). ( iii) E e ::::} {G s: I } and hence Be nEb ~> {B1 - Bp(p) > Xi - Xp(p) - b Vi E G, i- p(p)} =:::> {Bi - lllax B· > .Yi - lllax X· - b Vi E G f- p(p)} . j#i,O J j#i.O J , (iv) If #G > 1, then with X M = InaxjEG,::pp(p) .Yj , Ee nEb ::::} {Bp(p) - Bj < Xp(p) - Xj + b Vj E G, i- p(p)} ::::} {B p(p) - BM < Xp(p) - .YM + b} =? { B p(p) - B p(p-l) < X p(p) - X M + b} . Froln (6.12) and (6.13) we find that P{ Bi - Bo > Li Vi = 1, .. ,p and Bi - max B· E Ii Vi E G} j::pi,O J > P{Ee nEb} l/~_g+l (c )Ug ( b) > rnin l;-r+I(C)Ur(b) • O::;r::;p Now consider a eonfiguration of fL given by Bp(p-r) < Bo, r ~ 2 (Jp(p-r+l) Bp(p_I)--"t(Jp(p)-HX) • 111 (6.12) (6.13) CHAPTER 6. SIMULTANEOUS COMPARISONS Under this configuration, Then conditionally on G a~. f' since T > 2. P{Bi - rp~Xej E Ii Vi E G I Bi - Bo > Li Vi = 1, .. ,p} J'i::.t,O -+ P{O> Xi - ?-Uax.Xj - b Vi E ), #- p(p) and JE-y,#l o < .. \"p(p) - .lnax Xj + b} JE')'#p(p) Taking the second part of this expression, { X p(p) > . max .. \" j - b} JE-y,#p(p) { .. \" i - b < X p(p) Vi E r, f p( p) } =? {Xi - b < ?-Uax. Xj Vi E r, i- p(p)} JE-y,::j::.t and hence the conditional probability above approaches P{Xp(p) > .lnax Xj - b} JE')'#p(p) Ur ( b) . 112 (6.14) For T ~ 2, then, the result is proven. If the Inininlunl in (6.7) in fact occurs at r = 0, t.he infinlulll is attained at a configuration in which The minimum cannot occur at r = 1 since which is the value at. T = O. CHAPTER 6. SI]}!ULTANEOUS COMPARISONS 113 Remark Bofinger and Mengersen noted that, for i E G and #G > 1, the intervals Ii are given by Hsu (1984) for i = 1, .. ,p. Hsu's intervals are preferred if no comparisons with the control are required, since the confidence is generally larger with the saIne values of band c or, alternatively, smaller (b, c) combinations may be used with the same confidence. Also, more comparisons are lnade in this case. It is shown later, however, that if no comparisons with the control are required, Bofinger and Mengersen's procedure uses the InininIUlll possible value of b, which is the sanle as Hsu's value and which results in the saIne confidence as that of Hsu. Bofinger and Mengersen gave the following lenuna and theorem for use in cal­ culating the probability (6.7). (The proofs for these may be found in the paper by Bofinger and Mengersen (1988).) LelTIlTIa 6.1 The expression is strictly convex in r, Vr E [2,p - 1]. Also Theorem 6.2 a) If 3rM E [2, p - 1 J such that then the mininlulll value of Qr (c, b) (with respect to r) is given by b) If no such rM exists then the nIinill1Um of Qr( c, b) is given by lnin( Qo( c, b), Qp(c, b)) lnin( l~+ 1 ( c) , Up ( b)) . (6.15) (6.16) (6.17) (6.18) CHAPTER 6. SIMULTANEOUS COMPARISONS 114 Remark In practice, T M may be guessed (and checked) in order to find (6.17). Alterna­ tively, it may be guessed that no such r M exists, which may be checked by finding that either or Choice of c and b Suppose that. an experinlenter wishes t.o const.ruct. both bounds (6.5) and (6.6) for cOlllparison with the control and the best, with specified sinlultaneous confidence P*. Then P* is equal to the right. hand side of (6.7) and, for particular Fo(') and FI (.) distributions, appropriate values of c and b, say c* and b*, may be calculated. The values of c* and b* satisfying P* are not unique, however, since b* may be decreased at t.he expense of increasing c* and vice versa. With a smaller value of b*, nlon' sensitive cOInparisons with the best are achieved but comparisons with t.he control are less sensitive. This luay be useful if an experimenter is reasonably sure that SOllIe of the populations are better than the control. Similarly, a smaller value of c* is appropriate if the bounds (6.5) (for cOlnparison with the control) are required to be nlore sensitive than the intervals (6.6) (for comparison with the best). Three cases of special interest. were considered by Bofinger and Mengersen: (i) COluparisons with the control are required to be as sensitive as possible. In this case, the l1llnimUlll possible value of c* is taken, with (ii) Comparisons with the best are required to be as sensitive as possible. In this case, the InininluIlI possibe value of b* is taken, with (iii) Conlparisons with the control and with the best are required to be equally sensitive. In this case, b* = c*. CHAPTER 6. SIMULTANEOUS COMPARISONS 115 Case (i) The value Ce is defined so that (6.19) and hence be is found to satisfy Case (ii) The value bB is defined so that (6.20 ) and hence CB is found t.o sat.isfy (6.21 ) Case (iii) The value Cc B is required to satisfy ( 6.22) In order to solve (6.19), Theorenl 6.2 and the reluarks following it may be useful. In order to solve (6.20), Bofinger and ~\'iengersen 's Theorelll 2.3 Inay be used, which is given below in Theorenl 6.3. Using this theorem, we can take as CB the value satisfying (6.20) for T = P - 1, if t.his value is larger than that satisfying (6.20) for T = P - 2 and T = O. Similarly, (6.21) Inay be solved with the aid of their Theorems 2.4 and 2.5, which are given below as Theorenls 6.4 and 6.5. (Again, the proofs for the following theore1l1s lllay be found in the paper by Bofinger and Mengersen (1987).) Theorem 6.3 If c(' is defined by (6.19) and if C(B) satisfies ( 6.23) then (6.24 ) CHAPTER 6. SIMULTANEOUS COMPARISONS 116 Theorem 6.4 a) If 3CM and TM E [2,p - 1] such that then where Cc is defined by (6.19). b) If no such TM and CM exist then CCB = luax( Cc, bB ) with bB defined by (6.20). Relnark By Theorem 6.4, we can "guess" a value of T between 2 and p - 1, find the corresponding value of CM and test if this is less than the value of eM for T - 1 and T + 1. Notice, however, that if CM and TM exist. but CM < Ce, it would be more efficient to use case (i) instead of case (iii). That is, instead of requiring equally sensitive cOluparisons wit.h the control and the best, the experinlenter should obtain the more sensit.ive comparisons wit.h t.he best. Sinularly, if CM and TM do not exist and if Cc > bB, case (i) should again be used, while if Ce < bB case (ii) should be applied. Bofinger and ivlengersen ~s Theoreul 2.5 is useful for the case in which Fo(·) FI ( .) (so that only location paralneters differ between the control and t.reatment distributions). This t.heoreul is given below in Theoreu16.5 (wit.h proof omitted). Theorem 6.5 If Fo(·) = F I (·), t.hen t.he required value of C = b sat.isfies CHAPTER 6. SIMULTANEOUS COMPARISONS 117 6.3 Selection Decisions The bounds (6.5) and (6.6) may be used for selection decisions. With joint confi­ dence given by (6.7), the following assertions may be made: a) All populations 7ri, i E G, are demonstrably better than the control population b) If R(p) E G (that is, G is not eInpty) and XR(p) 2: max Xj + b j:f:O,R(p) t.hen we can assert (as well as (a)) that R(p) = p(p). That. is, 7rR(p) is better than all of the ot.her populations, including the control. Otherwise (for G non-empty) we can assert that 7rR(p) is better than 7ro and that BR(p) is within max Xj + b - XR(p) j:f:O,R(p) of B p(p)' If this difference is sufficiently sinall, the experilllenter may still be interested In 7rR(p). c) If SOllle other population 7fi, i EGis of interest, then an impression of its perfonnance cOlllpared with the best population lllay be gained by examining the value If this value is not too slllall, the population 11lay still be worthy of consideration due to other factors such as cost efficiency or availability. Notice, however, that it is the bounds that are provided to the experilllenter, and it is the experimenter who Inay Inake appropriate selection decisions based on this infonnation. d) If G is e111pty (so that no population is found to be denl0nstrably better than the control), then again the bounds (6.5) are useful in considering the magnitude of the differences between the populations and the control. e) If the experinlenter wishes to select a subset containing all populations which are delilonstrably better than the control and not demonstrably worse than the best population, the subset would include all 7fi for which CHAPTER 6. SIMULTANEOUS COMPARISONS 118 That is, all populations with Li = 0 and positive upper bound given by Ii would be selected. Bofinger and Mengersen's met.hod may be favourably compared with t.he Indif­ ference Zone-type approaches proposed by Turnbull (1976), Bechhofer and Turnbull (1978) and Bristol and Desu (1985) with respect to the following: • The control may be unknown and 111ay have a different distribution from that of t.he other experiinental populations. The three alternative procedures above require a fixed (known) standard. • The five constants which 1l1ust be specified by the experilnenter under the Indifference Zone-type procedures, which define the indifference zones and probabilities of correct selection, are not required under the confidence bound approach. Instead, it is necessary to prescribe P* and which of the three cases is of particular interest. • Specifying the confidence bounds leaves the onus of selection on the exper­ inlenter. ]\1ore infornlation about the cOinparative perfornlance of the pop­ ulations with respect to the control and to the best is available to the ex­ periinenter under the confidence bound approach than Inerely the selection statements provided by the Indifference Zone-type approaches. 6.4 Special Results for the Normal Means Case with Common Unknown Variance Let -"\i be the sainple Inean based on 11 1 independent observations frolll 7ri, dis­ tributed nonnally with unknown variance 0- 2 , 'i = 1, .. ,p. Let -Yo be the sanlple I11ean based on no independent observations froin the control population 7ro, with the saIne (unknown) variance (r2. TheIL with <1>(.) denoting the standard nonnal distribut.ion, CHAPTER 6. SIMULTANEOUS COMPARISONS 119 and The variance (T2 is estimated by the usual 52, such that v52 I (T2 has a chi-squared distribution on l/ degrees of freedom. Under this formulation, we have: G - { . . "lr X 5( -1 + -1 )1/2} - z .... '1.i - 0 > c no n 1 and L - . (0 X 1.r 5( -1· -1)1/2) i-mIn , i - -'1.0 - C no + n 1 Ii = [min(O, Xi - InaXj:f:i,O Xj - b521/2n~1/2), Inax(O, Xi - maXjEG4i Xj + b521/2n~1/2)] . (6.25 ) (6.26 ) ( 6.27) The joint confidence of the bounds (6.26) and (6.27) obtained by Bofinger and Mengersen is given in Theorem 6.6. Theorem 6.6 With G,Li and Ii defined by (6.25), (6.26) and (6.27) respectively, we have infP{Oi-Oo>Li Vi=l, .. ,p and ()i-~~xOjEIi ViEG} (lEO )'1-1,0 min ES{Pp-r+1(c51(T,P),Pr(bSI(T, 1/2)} (6.28) O~r~p where Es{·} indicates expectation over the distribution of 5, and Proof Let (6.29 ) (6.30) CHAPTER 6. SIMULTANEOUS COMPARISONS 120 Then, conditionally on S, the left. hand side of (6.28), with the infimum over ft E Or instead of all 0, is shown to be equal to v: (cS(n -1 + 17. -1 )1/2)U (bS2 1/2n -1/2) p-r+1 0 '1 r 1 where Vp_r+1(cS(n~1 + n~1)1/2) j( 0.5, the values of Cc (and hence CB) nlay be found froIll CHAPTER 6. SIMULTANEOUS COMPARISONS 122 Gupta, Panchapakesan and Sohn's (1985) tables. Krishnaiah and Armitage (1966) gi ve some values of Cc for p = O. Case (iii) Comparisons with the control and with the best are required to be equally sensitive. With C - b ?1/2(1 _ p)1/2 CB - CB-- , values of CCB are computed as the solution to For various {P*, p, v, p} c0111binatiolls, these CCB values are presented in Table B.7 (in Appendix B). Notice, however, that some of the values in this table have been replaced by the notation (i) or (ii). This indicates that for this particular {P*, p, v, p} combination, it is more efficient t.o use one of the other two cases. For example, when it is preferable to use Case (ii), since a sn1aller value of C may be used with bB with the same confidence. When this occurs, the value in Table B.7 is replaced by the notation (ii). Sillularly, when CCB = Cc the value in Table B.7 is replaced by the notation (i), indicating that it is preferable to use Case (i), since a slnaller value of b Inay be used with Cc with the same confidence. Where possible, the values in Tables B.5, B.6 and B.7 were cOlnpared with values in other tables, such as those of Gupta, Panchapakesan and Sohn (1985). Response surfaces described in Chapter 4 were fitted to a nUlnber of sections of the tables and outliers (identified by large R11S values) were verified by cOlnputing theln separately and comparing different values of r in detail. CHAPTER 6. SIMULTANEOUS COMPARISONS 123 6.4.1 Upper bound on the joint confidence Bofinger and Mengersen noticed that (6.33) where is the probability that (p - r) multivariate Student t variables, with degrees of free­ dom v and common correlation p, are all less than c. This expression is straight­ forward to compute and has been tabulated by a nUlllber of authors, as discussed in Chapter 4. It may be used to find useful upper bounds for bc, CB and CCB. For Cases (i) and (ii), Table 6.1 presents values of be and CB (depending on the case) such that expression (6.33) equals 0.75,0.95 for various p, v and p. (Case (iii) was olnitted because Cases (i) and (ii) are preferable for many of the values.) The super­ script accompanying each value is the amount by which the upper bound exceeds the corresponding value in Tables B.5 and B.6 (as appropriate). In Case (i), the r value which minimises (6.28) was also found to minimise (6.33) for most {P*, p, p, v} conlbinations, but for larger v, p and p the required r value for (6.33) may exceed that for (6.28). (All r values tend to fall between 2 and 5.) The bounds become luore satisfactory for small p and larger v. For case (ii), the r value which minimises (6.33) was also found to minimise (6.28). For large v these bounds are not too conservative. 6.5 Related Problem.s 6.5.1 Simultaneous comparison with a control and with the t best A generalisation of the results of this chapter may be luade by considering the problelll of simultaneously comparing populations with a control and with the t best (0 ::; t ::; p). This may be useful in practice for a nUlnber of reasons. For example, CHAPTER 6. SIMULTANEOUS COMPARISONS 124 Table 6.1: Upper bounds for various combinations of {p·,p,p,v} Upper bounds for be for case (i). p=4 p=9 p = 16 P = 20 v P 0.75 0.95 0.75 0.95 0.75 0.95 0.75 0.95 10 0.10 1.28.02 2.35.01 1.86.07 2.88.05 2.27·14 3.28.11 2.43.17 3.43.13 0.30 1.31.01 2.36.01 1.95.08 2.94.06 2.37.13 3.35.10 2.53.16 2.51.12 0.50 1.36.01 2.39.01 2.06.08 3.01.06 2.49.12 3.43.08 2.66.15 3.62.12 40 0.10 1.22.01 2.12.00 1. 71.02 2.51.01 2.02.03 2.76.02 2.14.04 2.86.02 0.30 1.25.01 2.12.00 1.78.02 2.53.00 2.11.03 2.81.02 2.23.03 2.90.01 0.50 1.29.00 2.15.01 1.87.02 2.58.00 2.21.02 2.86.01 2.33.03 2.97.02 120 0.10 1.20.00 ! 2.09.01 1.68.01 2.43.00 1.97.01 12.67.01 2.08.01 2.75.00 0.30 1.23.00 2.09.01 1.74.00 2.46.00 2.05.00 2.70.00 2.17.01 2.79.00 0.50 1.28.00 2.10.00 1.83.00 2.50.00 2.15.00 /2.75.00 2.26.00 2.85.00 Upper bounds for CB for case (ii). ! p=4 p=9 p = 16 P = 20 v 0.75 0.95 0.75 0.95 0.75 0.95 0.75 0.95 10 1.63.04 2.59.04 2.34.11 3.28.09 2.78.16 3.75.14 2.94.18 3.93.15 40 1.51.01 2.28.01 2.08.02 2.74.01 2.40.03 3.02.01 2.52.04 3.13.01 120 1.49.00 2.22.01 2.03.00 2.64.00 2.33.00 2.90.01 2.44.01 2.99.00 CHAPTER 6. SIMULTANEOUS COMPARISOl'-lS 125 an experimenter may indeed wish to identify more than one good population, with confidence that all of those identified perform better than some control or standard. In another situation, more than one good population may be identified and, from these, a single population may be finally selected on some other basis, such as cost or availability. In this case, the () values are assumed to be ordered as follows: Those populations which are truly better than the control are included in the set I, defined by (6.3) and those which are observed to be delllontrably better than the control are included in the set G, defined by (6.4). The t populations (excluding the control) which are truly best are those 7ri, 'i E 1t, where It = {p(k - t + 1), ... ,p(k)} . The lower bounds Li on ()i - ()o Vi = 1, .. ,p, given by (6.5), are again constructed. Provided #G ~ t, intervals are considered for Vi E G of (6.34 ) where _XR (p-t+ll i ) is the tth largest ){j, j = 1, .. ,p, t- i and _XR (p-t+ll i ),G is the tth largest _Xj, j E G,t- i. ~Then #G < t, no upper or lower bound for the interval Itt is defined. When # G = t, only the lower bound is defined. Theorenl 6.7 With G,Li and Iti defined by (6.4), ( 6.5) and (6.34) respectively, we have inf P{()i - 80 > Li Vi = 1, .. ,p and ()i - ()p(p-t+l) E Iti Vi E G} {lEO nun {l'~-r+ 1 ( c) Ur,t( b)} (6.35 ) t:Sr:Sp CHAPTER 6. SIMULTANEOUS COMPARISONS 126 where Ur,t(b) = 1, r = O, .. ,t -1 ; Ur,t(b) = t f F{-t(x + b)(l - F1(x)r-1dF1(x) , t:::; r :::; p and Proof Replace Eb in the proof of Theorenl 6.1 by E ~ = {_¥ j - () j < _¥ i - () i + b Vi E '"'f t ,j E , n "1 t } and let 9 = #,. Then and the event Ec is defined in (6.9) with By a direct extension of the proof of Theorem 6.1, then, the result follows. As detailed earlier, selection decisions may be made based on the bounds Li and I ti • With joint confidence given by (6.35), we Inay luake the following statements: a) All populations 7ri ,i E G are deillonstrably better than the control population b) If R(p - t + 1) E G and -¥R(p-t+l) ? rnaxj~G40 _Xj + b, then we can assert both (a) and that {p(p-f+1), .. ,p(p)} = {R(p-t+l), .. ,R(p)}, that is, that 7rR(p-t+l), .. , 7rR(p) are indeed better than all the other populations, including the control. c) For those tri ,i E G, the value CHAPTER 6. SIMULTANEOUS COMPARISONS 127 gives some information about the performance of 7ri compared with the tth best population. The population may still be worthy of the attention of the experinlenter if this value is not too far below zero .. d) If #G < t, so that there are not t populations which are demonstrably better than the control, the experimenter may wish to reduce the value of t or the joint confidence. In any case, the performance of the experinlental populations with respect to the control Inay be assessed using the llunlerical comparisons L 1 ,. e) A subset of populations which are denl011strably better than the control and not demonstrably worse than the t best populations includes all 7ri for which Xi 2:: Inax(Xo + c, XR(P-t+l) - b) . For the normallneans case with comnlon unknown variance, the joint confidence is given by A lower bound on this joint confidence is given by Conservative values of c and b statisfying this lower bound for the nornlal nleans case Inay be found by interpolation in tables such as Gupta, Panchapakesan and Sohn (1985) ( for c values) and Mengersell and Bofinger (1987) (for b values; see Table B.l). Satisfad,ory interpolation rn.ethods for the latter tables are described in Chapter 4. 6.5.2 Simultaneous comparisons with restrictions on the "best" The approach taken in this chapter Inay be used to develop solutions to other prob­ leIlls. For exaIllple~ Chen (1985) uses an Indifference Zone approach to select t.he CHAPTER 6. SIMULTANEOUS COMPARISONS 128 best normal population provided that its mean does not lie in a prespecified "unde­ sirable" region of the indifference zone (as discussed in Chapter 2). By the nature of the selection procedure, it is also possible that no population is selected if XR(p) and XR(p-l) differ by too little. That is, even if () p(p) lies outside the "undesirable" region, no selection will be made if 7r p(p) is not distinctly best. An alternative approach nlay be for the experilnenter to specify a nlinimum value 00 such that, if Oi fronl any population 7ri is less than 00 , that population should not be selected. (This corresponds to Chen's "undesirable" region.) This value 00 lnay be used as a known standard in the procedure described in this chapter. In this way, with a specified joint confidence, only those populations which are denlonstrably better than the standard are considered and all those populations which are not deulonstrably worse than the best are identified. The confidence bound approach gives nUlnerical infornlation about the perfonnallce of the populations compared with the control and with the best, rather than only the selection decisions obtained under Chen's procedure. 6.6 Example Consider the exaulple given in Chapter 3, in which 21 first year students each judge 8 lecturers on the basis of a nUlnber of specified attributes. After early exclusion of hvo of the lecturers, the relnaining six lecturers are to be conlpared in order to select the best. lecturer. Suppose, however, that it is decided to award the prize only if the chosen lecturer is shown to be denlonstrably better than the lecturer who was a"warded the prize in the previous year. That is, the st.udents wish to select the best lecturer, provided that he or she is better than a control (the previous year's ,;vinner). Suppose that the control lecturer in this case is Wendy. (Not.ice that. the st.udents are taking \Vendy t.o be a cont.rol~ rather than taking her previous year's score to be a (known) standard. This is Ulore appealing because, for exalnple, different students are involved in the judging procedure this year.) Fronl Chapter 3, the observed final scores for each of the lecturers are as follows: CHAPTER 6. SIMULTANEOUS COMPARISONS Carole Debbie Wayne Wendy Russell Janice 134.7 125.2 118.7 111.5 89.5 70.2 with S2 = 107.04 on 140 degrees of freedom. In this case, p = 5, s( no1 + nIl )1/2 = 3.19 and p = 0.5. Consider the following cases of interest: i) Comparisons with Wendy are required to be as sensitive as possible. 129 ii) COlllparisons with the best lecturer (other than Wendy) are required to be as sensitive as possible. iii) C01l1parisons with Wendy and the best (other) lecturer are required to be equally sensitive. Suppose that the students decide to investigate the decisions that can be made at both the 75% and 95% confidence levels. The appropriate critical values found from Tables B.5, B.6 and B.7 for each of these cases are as follows: Case P* = .75 P* = .95 (i) Cc 1.40 2.26 be 1.45 2.21 (ii) C(B) 1.66 2.34 bB 1.31 2.18 (iii) use case (i) In this situation, it is lllore useful to use Case (i) rather than consider equally sensitive cOlnparisons with the control and the best. Lower confidence bounds on Bi - Bo, where Bo is the true score for Wendy and ()i is the true score for the other lecturers Ci, i = 1, .. ,5, are constructed as follows: Case i P* I Carole i Debbie \\Tayne Russell Janice I i I (i) 0.75 0 I 0 0 -26.47 -45.77 0.95 0 0 -0.01 -29.21 -48.51 I I I (ii) 0.75 0 01 -27.30 -46.60 ~I -0.261 0.95 0 -29.46 -48.77 CHAPTER 6. SIMULTANEOUS COMPARISONS 130 Hence, for both Cases (i) and (ii), at the 75% confidence level, the students would select Carole, Debbie and Wayne as dernonstrably better than Wendy. At the 95% level, however, they would only include in this subset Carole and Debbie, although by investigating the bounds, they may consider including Wayne for case (i). (Notice that this could not be considered under the subset selection or indifference zone approaches. ) In order to compare these lecturers with the best, intervals Ii were constructed: Case P* Carole Debbie Wayne (i) 0.75 I (0,14.13) (-14.13,0) ( -20.63,0) 0.95 I (0,16.55) (-16.55,0) [ (ii) 0.75 (0,13.68) (-13.68,0) (-20.18,0) i 0.95 (0,16.45) (-16.45,0) Hence, for both P* values and for all three cases, Carole may be asserted to be both demonstrably best and demonstrably better than Wendy. Chapter 7 ROBUSTNESS TO NORMALITY OF A SELECTION RULE 7.1 Introduction In the previous three chapters, a special case on which luuch emphasis has been placed is t.hat of independent nornlal distributions, with COlllnlon unknown vari­ ance and equal numbers of observations frolll each experiIllent.al population. For t.his case~ particular results were derived and t.ables of percent.age points satisfying probability requireillent.s were presented. A quest.ion that. now arises is how robust the procedures are to deviations froill the assullIPtions on which this special case is based. As discussed in Chapters 2 and 4, a number of authors have considered different correlat.ion structures for the probleIlls of selecting the one best. population (general degrees of freedolu) and selecting f best populations (infinite degrees of freedom). For exaluple. Gu~~niRaP~Mi~a(l985) have produced extensive tables of percentage ,." points satisfying 131 CHAPTER 7. ROBUSTNESS TO NORMALITlr 132 for different correlation values p. SOlne indication of the effect of non-independence on the required percentage point or resulting probability of correct selection may be gained by examining these tables. If a COllllllon variance cannot be assumed, lllodifications of the methods described In the previous chapt.ers may be made. These may take the form of two-stage procedures, such as those proposed by Dudewicz and Dalal (1975), Rinott (1978) and Santner and Talllhane (1985) for selecting the best population, or Bofinger and Lewis' (1987) (confidence bound) approach for cOlllparing populations with a control and with the best. The problelll of unequainulllbers of observations may silllilarly require a modifi ­ cation of t.he probability expressions. Results have been obtained for the case t = 1, SOllle of which were discussed in Chapter 2 and which were detailed by Gupta and Panchapakesan (1979, pp.22-25, 233-237, 244-245, 417-420). For t = 1 under the confidence bound approach, Hsu (1984) has proposed a methodology for unbalanced designs which follows that established for balanced designs, with the exception that a vector of d values is used instead of a single value. This was also detailed in Chapter 2. Although Dunnett (1980a,1980b,1982),Tamhane (1979) and others have inves­ tigated the effects of non-norlllality on procedures for simultaneous comparisons, the robustness of ranking and selection procedures to non-normality has only re­ cently been considered. As discussed in Chapter 2, Dudewicz and Mishra (1985) COlll pared the perforlllance of Bechhofer' s (1984 ) procedure for selecting the t best nornlal means (with known variance) under a nUlllber of non-norlllal distributions. Exact results were given for the unifonn distribution and silllulated results for the i-distribution on 3 degrees of freedom. Across a range of (k, t) cOlnbinations and for two parailleter configurations~ the authors established that Bechhofer's procedure is indeed quite robust to such deviations froln nonnality. These results beg further questions, however, such as how skewness affects the procedure and whether the subset. selection and confidence bound approaches enjoy such robustness. CHAPTER 7. ROBUSTNESS TO NORMALITy" 133 In this chapter, robustness to non-normality is considered in detail for the pro­ cedure described in Chapter 5, which aims to select from k populations a sub­ set containing the t populations with the largest location parameters. Using a method suggested by Gupta and Sohn (1985), Tukey's generalised lambda distribu­ tion (G LD) is used to approximate various continuous distributions and to evaluate the probability of correct selection (PCS) arising under these distributions. This is achieved by considering the perforrnance of the lower bound on the PCS that was obtained in Chapter 5. If the bound perforrns satisfactorily, the PCS will also perfonn satisfactorily and hence the procedure Inay be asserted to be robust for that particular case. If, however, the bound performs poorly, this is not sufficient indication that the PCS is also unsatisfactorily small (since it may be only that the bound is extrenlely conservative in these cases). Relnedies to this problem are suggested. This chapter indicates preliminary investigations into the question of non-normality and is not intended to answer all of the issues that arise fronl these considerations. \Vhere appropriate, there will be Inention of these issues and of further investiga­ tions that are at present being undertaken. The next section details the particular selection probleul to be considered in this chapter. The generalised lalnbda distribution is investigated in Section 7.3 and is used in Section 7.4 to cOlnpute lower bounds for the probability of correct selection under the selection procedure. A Monte Carlo experiInent is also undertaken to investigate the true probability of correct selection for cases in which the lower bound is unsatisfactory. A Inore general approach to the robustness problem is considered in Section 7.5 and an exaInple of this approach is discussed in Section 7.6. 7.2 The Selection Problem Consider the probleul of selecting the f best of k populations, based on 11 indepen­ dellt observations froIll each population. The selection goal investigated in Chapter 5 is: CHAPTER 7. ROBUSTNESS TO NORMALITY 134 Goal G: Select 7fi, Vi E It, where Ii = {p(k - t + l),p(k - t + 2), .. ,p(k)} . The procedure proposed in Chapter 5 to satisfy this goal is: Procedure P: Select 7fi, Vi E G(d), where G( d) = {i : Xi > -Y"R(k-t+l) - dO"} (7.1 ) with d satisfying ( 7.2) where P* is the prespecified probabilit.y of correct selection and Pk,t( d) is given by (7.4) It is assulned that. independence exists between all observations, and that all k populat.ions have the same dist.ribution with comnl0n known variance 0"2 = 1, but it is not assunled that the fonn of the distribution is nonnal. Using the percentage point. satisfying (7.2), with F(·) taken to be the nonnal distribution, we wish to exalnine the value of P* using non-nonnal F(·). That is, if Procedure P were used with d values tabulated in Table B.3 (in Appendix B), we wish to exanline the effect on the resulting probabilit.y of correct selection if the underlying distribution is in fad non-nonnal. In order t.o investigat.e t.his question, use of the G LD was investigated. This distribution, introduced by Tukey in 1960 for synunet.ric distributions and gener­ alised by Ralnberg and Schlneisser (1972,1974) to include asymnletric dist.ribut.ions, ,vas used originally to generate randOll1 variables. Recently, as discussed in Chap­ ter 2, Gupta and Sohn (1985) have used this distribution to develop and evaluate a select.ion procedure based on sanlple Inedians. The latter aut.hors showed t.hat satisfactory approximat.ions to a very wide class of distributions may be obtained CHAPTER 7. ROBUSTNESS TO NORl\IALITY· 135 using the GLD, by using different combinations of the distribution's parameters. Gupta and Sohn also showed that the GLD was useful in obtaining approximations to the percentage points of (7.4) for t = 1 and suggested that the G LD could be used to test robustness of selection procedures t.o non-nornlality. 7.3 The Generalised Lambda Distribution The G LD may be expressed as follows: (7.5 ) where F- 1(.) is the inverse of the distribution function F(·) of the GLD, p is a uni­ form (0,1) random variable and where Al and A2 are location and scale parameters respectively, and A3 and A4 are measures of skewness and kurtosis. (Notice that, although A2 is described as a scale paralneter, it is not necessarily positive.) For Al = 0, Ramberg and Schmeiser showed that the even monlents of the GLD, w hen they exist, are given by where f3 denotes the beta function. The kth nloment exists if and only if -1/k < min( A3 , A4) (so that the arguinents of the beta function are nonnegative). Ramberg and Schmeisser (1974) identified t.he four regions in which the GLD is a legitimate probability distribution, that is, where the density function f( x) is nonnegative for all x and J~ov f( x )dx = 1. For sYllllnetric distributions, '\3 is equal to A4 and the sign corresponding to .\3 must. be t.he saIne as t.he sign corresponding to A2. For these distributions, all odd central nlOlllents are zero and both the Illean and Inedian equal AI. The standardised second and fourth central nlOlnents ((T2, (t4) are given by (12 = 2{1/{2.\3 + 1) - 13(.\3 + 1,.\3 + 1))/'\; , 1/(4'\3 + 1) - 4/1(A3 + 1,3A3 + 1) + 3/1(2'\3 + 1,2'\3 + 1) 2[1/(2,\2 + 1) - ;3('\3 + 1''\3 + 1))2 CHAPTER 7. ROBUSTNESS TO NORMALITY· 136 7.3.1 Generating random variables J\1any symmetric and asymlnetric distributions may be approximated in the form of t.he GLD. Given the first four nlOlnents (in tenns of the parameters of the known distribution), Ramberg and Schmeisser (1972,1974) illustrated how the appropri­ ate paralneter combinations (A}, A2, A3, A4) for the G LD which approxinlates the particular distribution may be obtained. If the first four moments do not exist (A3 ::; -0.25), the parameter conlbinations may be obtained by using the per­ centile points of the distribution. Table 7.1 lists combinations of paranleters for the GLD, approximating a nUluber of COlUluon symmetric and asymmetric distributions, which were tabulated by Ralnberg and Schllleisser. For exalllple, for the nonual distribution with luean () and variance (J2, 0:4 = 3.0 and, frolll Table 7.1, (AbA2,A3) = ((),0.1975/o-,0.135). Hence, the inverse GLD which approxilnates the inverse normal distribution is given by x = () + o-(pO.135 - (1 - p )0.135) /0.1975 , (7.6) ° < p < 1. This lnay be used to generate approximately normal random variables. 7.3.2 Approximating empirical distributions Ralnberg and Schllleisser suggested that empirical distributions may be approx­ ilnated by using estilllates of the lllOlnents based on the randoll1 salnple Xi, i = 1, .. , n. For a wide range of distributions, they generated randolll variables, obtained estilnates of the parallleter values (A} ~ A2, A3, A4) and cOlnpared thenl with the exact paralneter values frolll the G LD lllost closely approxilllating the exact distribut.ion. There was eillphasis given to cOlllparing the tails of the distributions, since this is the lllost critical region in which the two distributions will differ. Even though the approxin1ations are based on fourth luon1ents, the estilllated parailleters for the G LD were shown to be quite close to the corresponding parallleters of the original (exact) distribution. For both synlnletric and aSYlllmetric distributions, the authors outlined the steps necessary to obtain estiInates of A}, A2, A3 and A4 frolll a randolll saluple. To assist CHAPTER 7. ROBUSTNESS TO NORMALITY· 137 Table 7.1: Characteristics of Given Distributions. Reproduced from Ramberg and Schrneisser (1972,1974) I Distribution Symnletric Nonnal (N) 0.0 .1975 .1349 .1349 t on 5 df (t 5 ) 0.0 -.3202 -.1359 -.1359 t on 10 df (tIO) 0.0 .0261 .0148 .0148 t on 34 df (t34) 0.0 .1563 .1016 .1016 Uniform (U) 0.0 1.0000 .5774 .5774 Logistic (Lo) 0.0 -6.59E-4 -3.63E-4 -3.63E-4 Laplace (La) 0.0 -.1686 -.0802 -.0802 Cauchy (C) 0.0 -3.0674 -1.0000 -1.0000 Asymrnetric Gamma Distribution: F( x) = ;30 x o - l e- f3x /f( 0:) x > 0,0: > 1,;3 > 0) I I I I 0: = 1,;3 = .5 (G 11 ) ! .0008 .0002 .0000 .0004 0: = 1,;3 = 1.0 (G12) .0004 .0002 .0000 .0004 0: = 5,;3 = .5 (G21) 7.1747 3.5874 .0252 .0939 0: = 5,~1 = 1.0 (G22) .0216 .0434 .0252 .0939 0: = 10,1' = .5 (G31) 16.7094 .0207 .0442 .1243 0: = 10,,8 = 1.0 (G32) 8.3547 .1243 .0442 .1243 0: = 50,,8 = .5 (G41) 95.8320 .0126 .0869 I .1525 Q = 50,;1 = 1.0 (G42) 47.9160 .0251 .0869 .1525 I 0 = 200, i1 = .5 (G51) 395.4852 .0068 .1103 .1501 0: = 200, f3 = 1.0 (G52) 197.7426 .0136 .1103 .1501 CHAPTER 7. ROBUSTNESS TO NORMALIT"Y 138 in the computations, appropriate tables were produced. For symmetric distributions, Ramberg and Schlneisser (1972) detailed the fol­ lowing steps to calculate estimates of the required ,\ combinations, given a random sample Xi, i = 1, .. , n. 1. Compute n iJ = X = L Xi/n i=1 n 0-2 = L(Xi - X)2 /n, i=1 n 0:4 = L(-¥i - _¥)4 /n0-4 1=1 fronl the data. 2. Find,\2 and '\3 satisfying the expressions for the central monlents (or corre­ sponding to the specified 0:4 value in Ranlberg and Schmeisser's (1972) Table I). 7.3.3 Ranking and selection using the GLD Gupta and Sohn (1985) also used the GLD to examine the perfornlance of ranking and selection procedures for selecting the population corresponding to the largest nledian. They, too, found this distribution gives satisfactory approximations to the exact distribution and showed that, with the lambda combination corresponding to the nonnal distribution (see Table 7.1), values of d obtained with the GLD satisfying Pk,1 (d) = 0.90,0.95,0.99 agreed to at least two decilnal places with the value of d obtained using the exact nonnal distribution. Because of these satisfactory results, the problenl to be considered in this chapter nlay be approached with the aid of the GLD. Using the value of d satisfying (7.2) for the nonnal distribution (tabulated in Table B.3) and assulning a C011unon variance of unity, we wish to find the probability (7.3) for a non-nornlal distribution F(x - 8) which has been expressed in the form of a GLD. That is, we wish to investigate CHAPTER 7. ROBUSTNESS TO NORMALITY 139 how much variation can be tolerated in the lambda values of the GLD before the selection procedure P fails to give a satisfactory pes. As indicated in the introduction, this question is investigated by considering the lower bound PL on the pes given by (7.3). Notice that if PL is satisfactorily close to the prespecified probability (P*. say), the pes may also be asserted to be close to (or larger than) P*. If PL is much larger than P*, the pes may be asserted to be even larger than P*. If however, PL is lnuch smaller than P*, it cannot be deternlined whether the bound is conservative (and the pes is still satisfactory) or the pes is also small. This problem is also discussed in a later section. 7.4 Computation of the lower bound for given distributions To evaluate the expression Pk,t(d) given by (7.4) for a given (p·,k,t,A},A2,A3,A4) combination, the values of X satisfying (7.4) for p values of {.OOO( .001 ).005, .01( .01 ).99, .995( .001)1.00} were obtained. For each elelnent of p, the value of F( x + d) was then obtained by linear interpolation in the table of calculated values of X. The required probability PL was approxinlated by replacing Pk.t(d) in the right hand side of (7.3) by (where ~ indicates forward differences L with sunlnlation over the values of p. Using the GLD approximation to the nonnal distribution (given by (7.6), the Pk,t( d) values were compared with the corresponding exact Pk,t( d) values tabulated by Bechhofer (1954) (also given in Table B.1 for 7/ = (0). In no case did the two figures differ by more than 0.001. This agrees with Gupta and Sohn's (1985) conclusions. U sing the value of d corresponding to P* = 0.75,0.95 for the nonnal distribution case (tabulated in Table B.3), the GLD representing other conllllon distributions CHAPTER 7. ROBUSTNESS TO NORMALITY· 140 (with parameters given in Table 7.1) was substituted for F(·) in (7.3) and the re­ sulting value of PL was obtained. This was done for a number of (k, t) combinations. For most asymmetric distributions, the values of PL were very low. (The only exception to this was G22, for which the values of PL were quite close to the specified probability for all (P*, k, t) combinations.) Many of the values were below the minimunl possible PCS (using random selection of populations), indicating that it is the bound PL that is extremely conservative in these cases. It is not possible from this investigation to infer whether the PCS is also low for these distributions (and hence whether Procedure P is robust to asymnletry.) For 11lOSt symnletric distributions, the PL values were quite close to the specified probability P*, for all (k, t) combinations. The exception to this was the Cauchy distribution. The PL values obtained for sYlnmetric distributions are given in Table 7.2, with the names of the distributions defined in Table 7.1. Discussion oCTable 7.2 For sonle reason, the conlbination (k, t) = (2,1) appears to give results in Table 7.2 which are inconsistent with the general trends observed over the other (k, t) cOlnbinations. Although it is difficult to ascertain the reason for this, two factors may be main contributors. The first arises from the various approximations used in computing the tabulated d values (see Chapter 5), which are only accurate to ±0.01 and in using the GLD instead of the exact distribution. The second consideration is the small size of the d value for this (k, t) conlbination; a change of 0.01 in this value may result in a fluctuation in PL which is larger than that observed for the saIne change in a larger value of d. This case is not included in the following discussion on the trends observed in Table 7.2. For the t distribution, the lower bound PL (and hence the PCS) appeared to be quite close to P*. Generally, there was much nlore variation in the PL values at P* = 0.75 compared with P* = 0.95. For P* = 0.75, all (k, t) combinations considered achieved PL values which, when rounded to two decimal places, were greater than or equal to 0.75. (The sillallest probability was 0.748.) For P* = 0.95, Inore of the PL were sinaller than the specified P*, with the smallest value being 0.92. Overall, for both P* values. the procedure seeined to achieve the largest PL CHAPTER 7. ROBUSTNESS TO NORAfALITY 141 Table 7.2: Values of FL for selected symmetric distributions I (k,t) I P* II t5 I tlO I t34 I U I Lo \ La I C I (2,1 ) .75 11. 72 .76 .75 .74 .76 .77 .66 (4,1 ) .79 .76 .75 .72 .77 .78 .52 (7,1 ) I .78 .76 .75 .73 .76 .77 .40 (10,1) .76 .75 .75 .75 .76 .76 .30 ( 4,2) .80 .77 .75 .71 .77 .79 .42 (7,2) .79 .76 .75 .71 .77 .78 .19 (10,2) .77 .75 .75 .74 .76 .77 I .00 (7,5 ) .79 .76 .75 .72 .76 .77 .19 (10,5 ) .78 .75 .75 .75 .76 ,...,,..., .00 .1 I (10,8 ) .77 .75 .75 .75 .75 .76 .06 I (2~1 ) .95 Ii .95 I .95 .95 I .95 .95 .95 .79 (4,1 ) .95 .95 .95 .97 .95 .95 I .65 (7,1 ) .93 .94 .95 .99 .94 .94 .52 (10,1 ) .93 .94 .95 1.00 .94 .93 .42 ( 4,2) .94 .94 .95 .96 .95 .94 .59 (7,2 ) .93 .94 .95 .99 .94 .93 .37 (10,2 ) .92 .93 .95 1.00 .94 .93 .19 (7,5 ) 1·93 .94 .95 .98 .94 .93 .37 (10,5 ) .92 .93 .94 1.00 .93 .92 .14 (10,8 ) .92 .93 .94 .99 .93 .92 .23 CHAPTER 7. ROBUSTNESS TO NORMALITY 142 values for small k. These results tend to indicate that Procedure P is quite robust to heavier tails and in most cases gives a larger pes (since this is always larger than PL ) than that required. For the uniform distribution, the performance of the bound PL appears to de­ pend on the value of P*. For P* = 0.75, the values of FL tended to be smaller than P* (except for k = 10), whereas for P* = 0.95, they were generally larger than P* (except for k = 2). For larger P*, then, it appears that Procedure P gives a much larger pcs than that required. For smaller P* (such as 0.75), it can only be noticed that the PCS is at least greater than FL. The logistic and Laplace distributions exhibited a quite different performance frOlll that of the unifonn, with most conservative results achieved for small k and t values. For P* = 0.75, all observed PL values were greater than 0.75, but for P* == 0.95, all FL values (with the exception of k = 2) were less than 0.95. The pes obtained under these distributions, then, are likely to be greater than the required probability for P* = 0.75 and not more than 0.03 below (and possibly greater than) the required probability for P* = 0.95. Frolll Table 7.2, the Cauchy distribution performed very badly. Many of the FL values were well below the minimum possible probability (if random selection of populations took place), indicating that the bound is extremely conservative in this situation. This investigation gives little information, then, about the true value of the pes obtained for the Cauchy distribution. 7.4.1 Alternatives if PL is small As noticed above, if the PL values are sillall, little may be asserted about the true PCS using the above approach. In such cases, two alternatives may be considered. Firstly, alternative bounds, such as Pk,t( d) proposed by Carroll, Gupta and Huang (1975) or Pk-t+1,1 (d) conjectured by Bofinger and Mengersen (1986) and proved by Bofinger (1988) for t = k - 1, lllay be considered. It lllay be that these give larger bounds than PL for the part.icular distribution. Certainly, they should for III lower and upper bounds, respectively, for the pes. A 1110re appealing approach may be to silnulate the true PCS, using the GLD to generate randolll variables froln the CHAPTER 7. ROBUSTNESS TO NORMALITY 143 particular distribution, as described in Section 7.2.2. To illustrate the silllulation approach, Monte Carlo experiments were carried out for the following cases: uniform distribution, P* = 0.75, (k,t) = (4,1),(7,5); Cauchy distribution, P* = 0.95, (k,t) = (4,2),(7,5). Using an equal means config­ uration, the GLD approximating the particular distribution was used to simulate k random variables and the event of correct selection under Procedure P, using the appropriate value of d from Table B.3, was determined. This was repeated 500 times for P* = 0.75 and 100 times for P* = 0.95, giving observed values of the probability of correct selection (PC S, say) with standard errors of 0.02. The resulting values of PC S for the four cases considered are given below. Distribution P* i (k, t) PL PCS Unifonll .75 (4,22 ) .71 .88 (7,5 ) .72 .73 Cauchy .95 (4,2 ) .72 .87 (7,5 ) .37 .68 7.5 General results The results of Section 7.4 are particularly useful if the experinlenter believes that the given data are not normal but that they do follow one of the distributions considered in that section. In this case, it is possible to determine if the use of the d values tabulated for the normal distribution will indeed give a satisfactory PCS-requirement even with the non- nornlal data. Although this method of assessing robustness to non-nonnality by examining particular distributions is COlnnlon in the literature, it has a disadvantage of not giving any infonnation about the perfonllance of the procedure when the data do not follow one of the examined distributions. Even if the data are "close" to one of the distributions studied, no real statenlent regarding the expected confidence level can be lllade. CHAPTER 7. ROBUSTNESS TO NORAfALITY 144 Through use of the GLD, a more general assessment of robustness to non­ normality may be made. By varying the parameters of the GLD, the FL values may be calculated for the new A combination, drawing a "picture" of the perfor­ mance of the lower bound over the space of A values. If only two A parameters are allowed to vary, the results may be represented as a two-dil11ensional contour plot of probabilities or by three-dimensional perspective plots. For sYl11metric distributions (A3 = A4), the behaviour of the bound as (A2' A3) vary was investigated in this way. For various (k, t) combinations, the value of d corresponding to a specified P* value (given by 7.2 with F(·) the normal distribution function) was found froln Table B.3. This (P* ~ k, t, d) combination was then used in the calculation of FL given by (7.3) with F(·) the G LD approxiluation to the particular distribution function under consideration). This was computed for each combination of (':\2, A3)' A2 = -1.0( .05 )1.0, A3 = -1.0( .05 )1.0, such that the sign of .:\2 is the same as the sign of .A:~ (since it is only in these two quadrants that the GLD is a valid density function). A selection of the resulting contour plots are displayed in Figure C.1 (located in Appendix C). These plots were generated using the statistical package S. Discussion of Figure e.l The sets of surfaces depicted in Figure C.1 include (P*, k, 1) = (0.75,6,2), (0.95,6,2), (0.75,4,2), (0.95,4,2), (0.75,10,2), (0.75,10,5) . The following overall observations may be made: - For fixed k and t, as P* increases the "cliffs" hecon1e lllore steep, and the two regions of zero probability (the surface "floor") and the valley connecting theln (passing through the point (0,0)) hecolne snlaller. Consequently, the "ceiling" of the surface, representing FL values close to unity, decreases in slze. - As k increases for fixed P* and t, t.he surface "floor" area increases. The observed perfoflnance of particular (synulletric) distributions, illustrated in Table 7.1, lllay be verified in these plots, by noting the "height" of the surface (the P value) for the appropriate ().2, ).3) COlllbina.tioll. Notice also that for the ().2, A3 ) CHAPTER 7. ROBUSTNESS TO NORAfALITY 145 combination corresponding to the nor111al distribution, each of the plots gives a PL val ue (al1110st) equal to P*, as expected. 7.5.1 Empirical assessment of the effects of non-normality For symmetric distributions, Figure C.1 111ay be used to assess the effect of non­ normality on Procedure P for a particular data set. For aSYl1unetric distributions, a series of similar plots may be generated, but since three paralneters (':\2, ':\3, .:\4) 111USt be considered, it may be easier to COlnpute the approxirnate probability directly by using the estinlated .:\ values in the GLD and substituting this distribution in (7.4). A proposed lnethod of using Figure C.1 is outlined belo,,\;'. 1. Test for sYl1ul1etry in the data, by such nlethods as a stenl-and-leaf display or normal probability plot.. If there is an indication of aSYln111etry, consider possible transformations or stop and use an alternative method. If there is no aSYl11111etry apparent, continue. 2. Estilnate the values (.:\2'.:\3) using the lllethods suggested by Ramberg and Schmeisser (1972), described in Section 7.2. 3. Consider the plot in Figure C.1 corresponding to the particular (P* , k, t) com­ bination. Detennine froln the plot the value of PL pertaining to the partie-ular (':\2, .:\3) cOll1bination. If this probability is dose to P* and not too conservative, use Procedure P with the d values corresponding to the nonnal distribution case, since the PCS will be close to (or greater than) P* in this particular case (so t.hat Procedure P is asserted to be robust to any non-nonnality present in the data). If the probability PL is too srnall, a nUlnber of options are available. Firstly, consult the plots for the sa111e (k,t) conlbination with sInaller P* value (Pz, say). Consider the P value corresponding to (k~, t) from this plot. If this is satisfactorily close to P*, use Procedure P with the value of d corresponding to P* = P2 • This will result in a selection procedure which has PCS at least CHAPTER 7. ROBUSTNESS TO NORMALIT1r 146 P*. Secondly, if the PL value is still too slllall even for very large P2 , consider the options discussed earlier. These include evaluating alternative bounds to obtain more information about the size of the pes or silllulating the true pes. Thirdly, it Inay be more desirable to consider another approach. For exalnple, procedures designed for specific non-normal distributions (such as binomial) or nonparametric Inethods may be examined. If the PL values are too large, use the first option described for conservative values, but consult larger P* values. Alternatively, consider other approaches, as discussed above. The above methodology has the advantage of not requiring that the particular dist.ribution be one of a slllal1null1ber of conlIllonly investigated distributions. Even a slight deviation frol11 nonnality Inay be assessed for its ilnpact on the performance of the selection procedure P. Also, there is no requirelnent for tables of d values for distributions other than the nonnal; the d values for the normal means case may be applied for aln10st any cont.inuous distribut.ion. This inconvenience is replaced by another, though, of requiring contour plots for various (P*, It: , t) combinations. It l11ay be possible to reduce the ntunber of such plots required by investigating relationships between PL values as P*, J..., and t change (as was touched on in Section 7.3) and '"interpolating" between plots. This is currently under consideration. If this is found to be feasible, it l11ay be 1110re useful to generate contour plots of the pes values (rather than of the lower bound), obtained frol11 sil11ulation results. This would be practical only if it is possible to effectively reduce the nUll1ber of contour plots required, since the COlllputing time involved in sil11ulat.ing the pes for each ( A2, A3) cOlnbillation for each (P*, k, t) would be lllllCh greater than that required to obtain lower bounds on the pes. Notice also that this ll1ethod Inay be extended to other selection procedures. The use of the GLD in the way described above enables quite general examination of the robustness of a wide range of procedures and the corresponding probability bounds. CHAPTER 7. ROBUSTNESS TO NORMALITY 147 7.6 Example Consider the problem posed in Chapter 3, in which two lecturers are to be chosen as "best" from eight (two of whom were earlier excluded by other means). The selection is to be based on the scores of 21 students. From Chapter 3, the six remaining lecturers' final scores are: Carole 134.7 Debbie 125.2 Wayne 118.7 Wendy 111.5 Russell 89.5 with 8 2 = 107.04 on 140 degrees of freedoill. Janice 70.2 Suppose that the students wish to select, with at least 75% confidence, the two best lecturers. FrOlll Table B.3, the value of d corresponding to (P*, v, k, t) = (0.75,140,6,2) is 2.16. Hence, those lecturers with scores larger than X R(5) - ds/ fo = 120.3 are included in the selected subset; that is, Carole and Debbie are selected. The above results were obtained under the assuInption that the students' scores are independent and normally distributed with COllUllon (unknown) variance. Using the lllethods described in this chapter, we wish t.o investigate the validity of these assuInptions and what effect any deviation frolll norlllality has on the confidence of correct selection. Although only the six remaining candidates will be considered in the final selection, the entire set of eight candidates' scores Illay be used in assessing the above assulllptions. Step 1 Using the residuals obtained after taking out lecturer and student effects from the set of scores -~uv, u = 1, .. ,6, v = 1, .. ,21 given in Chapter 3, we may test for hOlllogeneity of variance, independence and sYlllmetry in the distribution of the data. In a t.wo-way analysis of variance, the judge (in this case, student) effect is not. significant (F < 1). Due t.o the large degrees of freedolll for the estilllated variance, we Illay assunle that cr 2 is effectively known, but. we lllUSt still ascertain hOlllogeneity of variance. The estiinated variances 87 of the residuals of scores awarded to each lecturer Ci,i = 1, .. ,8 are COlllputed t.o be: David Carole Debbie vVayne Wendy Chris Russell Janice 109.4 110.6 81.8 109.1 98.4 75.7 92.4 71.9. CHAPTER 7. ROBUSTNESS TO NORMALITY 148 The estimated variance S2 cOlnputed from t.he combined set of residuals (frolll all eight. lecturers) is 89.7. The chi-square statistic for a test of homogeneity of variance is 0.34410, which is non-significant. The range of correlations between lecturers is (-0.660,0.295), with 23 correlations negative and 5 positive. Steill-and-leaf plots of the residuals for each lecturer, and the combined set of residuals, do not indicate asymmetry. Similarly, a plot of normal scores for the cOlllbined set of residuals is satisfactorily linear. For the combined data, a sign test for a Inedian of zero gives a non-significant P-value. Estimates of skewness and kurtosis for the combined data are 0.48 and -1.49 respectively. The range of estilllates of skewness for the individual lecturers is (-0.40,0.74) and of kurtosis is (-1.28,0.35 ). Overall, then, the asstllllptions of honlogeneity of variance and sYll1.1uetry may be assul1led to hold, but the aSSulllPtion of independence Inay be in doubt. There is a suggestion of negative correlation between lecturers, but aSSUllle for the present that the data are independent so that we can proceed. Step 2 The following lllay now be cOluputed using the residuals: 168 8(X) = L-~i = 0.0 1=1 168 o-2(X) = I)Xi - _~)2 /168 = 89.2 i=1 168 Q4(X) == 'L:)Xi - X)4/(No-4) = 2.47 . i=1 In Procedure P, the means of21 observations are used. Hence we need to consider the distribution of these 111eanS, rather than that of the individual observations. The 11.1011lents of the standardised variables z=~t/Cr are cOlnputed as follows: e ( z) = e ( _\"") = 0. ° CHAPTER 7. ROBUSTNESS TO NORAfALITY 149 Interpolating in Ramberg and Schmeisser's Table I, we find that (A2' A3) - (.20, .14). (The values of ("\2, ..\3) for the nornlal distribution are (.198,.135).) Step 3 Consulting the contour plots in Figure C.1 at the point (..\2'..\3) for (P·,k,t) = (.75,6,2), we find that, if the percentage point corresponding to the latter combination in Table B.3 is used in Procedure P above, the lower bound FL will be (as expected) very close to 0.75. The students Inay then choose Carole and Debbie as the two best lecturers, with close to (if not great.er than) 75 o/c. confidence of correct select.ion. They Illay be satisfied t.hat this confidence holds even if t.heir data are not exactly nonllally distributed. (The siIllulated value of the pes in this case, using the methods described in Section 7.4.1), was found to be 0.85). Notice that in the above approach, and for ranking and selection procedures in general, the tests of symlnetry and homogeneity of variance applied above may not be the Iuost suitable for this particular problem. Perhaps traditional tests of these assulnptions are not entirely applicable for ranking and selection problenls, since only a nUlnber of the populations are of interest. Alternatively, we could apply Raluberg and SchIlleisser's (1974) steps for aSYlumetric distributions and compute the values of A3 and A4 separately. SOlne test of equality n"lay then be applied to assess synl111etry, which would help to detennine the most appropriate approach. The developnlent of suitable tests for sYlnlnetry and homogeneity for ranking and selection procedures requires further investigation. It was also noticed above that there was SOlne indication of negative correlation In the data. The largest correlations, however, were found to be between David and other lecturers. Ren1elnberillg that David was excluded before any selection took place, it is difficult to assess the inlpact of such correlation. This question of the effect of correlation on ranking and selection procedures, and appropriate tests for correlation which Inay have adverse results on the procedures, requires further investigation and is at present under consideration. Lastly. the Ineans used in the above selection procedure were, as expected, Iuuch CHAPTER 7. ROBUSTNESS TO NORMALITY' 150 more normally distributed than the individual observations. From the central limit theorem, it is reasonable to expect that, regardless of the underlying distribution, if a large enough sample is taken the nleans will be normally distributed. The size of such a sample is another problem requiring further consideration. It would also be interesting to assess the impact of this on the performance of asymmetric distributions. Chapter 8 CONCLUSION As stated in the introduction, this thesis attempts to provide answers to a number of selection problems. The goals considered in this thesis are those of comparison and selection of the t best populations, and simultaneous comparison with a control and the best. Although these occupy but a very small niche in the now quite large field of ranking and selection, they are quite fundamental problems. The Inethods developed to satisfy the goals enlploy the confidence bound approach, which provides an appealing alternative to the more t.radit.ional indifference zone and subset selection formulations. Although the procedures developed to satisfy these ranking and selection goals have been demonstrated to satisfy the specified probability requirements and to be in some ways superior to existing methods, it is important not to ignore further questions that may be asked about the results. A real concern, for example, is that of the robustness of the procedures to deviation from any of the stated assump­ tions. One approach to such a question for the problem of selection of the t best Ineans, if the assumption of normality is relaxed, was considered in Chapter 7 of this thesis. This approach Inay be extended to the Inethods developed in the other chapters to gain information about their robustness to nOflnalit.y. It renlains, how­ ever, to investigate more fully the effect of relaxing any of t.he st.ated assumptions of the various selection procedures, including COllunon variances and equal sample sizes and cOlnbinations of these, before the true practical story of t hest' procedures' 151 CHAPTER 8. CONCLUSION 152 robustness is fully understood. It is the practical use of these methods that must be borne in mind, since ranking and selection, like all statistics, is primarily a tool to be used by practitioners. The theory of ranking and selection, while presently undergoing rapid development, has long ago matured to a level at which it is potentially relevant and applicable in practice. Although this is recognised by many researchers in the ranking and selection field, and although practitioners have been requesting effective solutions to t.he very questions that. ranking and selection theory addresses, the combination of theory and application has not eventuated on nearly the scale expected. Many reasons may be forwarded for this~ although none of theln are new. Lack of education of applied statisticians in the field of ranking and selection, both at univer­ sity level and "on the job" , has retarded the acceptance of the "new" methodology. This is exacerbated by the reluctance of researchers to publish relevant ranking and selection developments in journals COllllllonly consulted by the applied practitioner, rather than in journals devot.ed solely t.o st.atist.ics. Often, however, even if practi ­ tioners are eager t.o embrace the nlethodology, the esoteric nature of t.he literature on the particular problelll, the lack of relevant. tables or, more commonly, the lack of cOlnputer soft.ware discourages the potential users. This does not illlply that. no attelllpt has been lnade to apply ranking and selec­ tion t.heory in practice. In a large nUl11ber of papers, real life or contrived problems are used to illustrate the application of the particular procedure. SOllle of these solut.ions are then presented in applied science journals. As discussed in Chapter 2, several reviews of ranking and selection have appeared in applied science journals and useful examples have been distributed throughout several books devoted to ranking and selection. Only recently, however, has the practical application been taken a step further, to el11brace an entire practical problel11 in such a way that a user l11ay follow each step of the lllethodology frolll the posing of a relevant. question to the interpretation of the corresponding results. In this thesis, such an attempt has been described. In the field of personnel selection, such goals as comparison, selection and ranking of candidat.es, goals that forlll t.he backbone of ranking and selection lit.erat.ure, are pertinent. In Chapter 3, a CHAPTER 8. CONCLUSION 153 ranking and selection methodology designed to be directly applicable to ranking and selection theory was detailed. Since it was not adequate to develop only the ranking and selection tools without first devising a quantitative scoring rule on which to base comparison between candidates, the methodology begins with the development of such a rule. Since it was also not adequate to expect all practitioners to manually execute the methodology, a computer package, PERSEL, was developed. PERSEL incorporates the goals and procedures of Chapters 4 and 5 of the thesis, as well as a number of other relevant ranking and selection goals. As indicated, there are many potential improvelnents to be made and outstand­ ing concerns about the selection and cOlnparison procedures developed in this thesis. Bounds on the probability statelnents 11lay be inlproved; questions such as robust­ ness relllain to be further investigated and application of the results to other related problems may also be made. So, too, there are l1lany criticislllS that nlay be nlade of the proposed application to personnel selection, including the scoring methodology, the advocated goals and procedures, and the package PERSEL. More sophisticated scoring ll1ethods 111ay be introduced; a wider variety of goals and alternative pro­ cedures Inay be considered, and such improvements as graphical enhancement may be useful in PERSEL. All of these are currently under investigation. The development of the theory of ranking and selection is undoubtedly most important, since it is on this solid base that practical application of the results may be nlade. The latter, however, is equally as ill1portant. It is from the continued interaction of researcher and practitioner that the field of ranking and selection will expand, with respect to the type of problell1s posed for solution, the scientific fields to which the theory is applied and the variety of people making useful contributions to the literature. In this light, it is not as inlportant that the personnel selection nlethodology proposed in this thesis is not "best" in all respects, as the fact that it has been developed to directly answer specific questions in a specific field. From such a hurnble beginning, it is hoped that inlprovements and extensions will eventuate fronl both users and researchers, resulting in a statistically sophisticat.ed, useful package. It is also hoped that the field of personnel selection is not the only practical CHAPTER 8. CONCLUSION 154 area to benefit from the theoretical and practical developments presented in this thesis, but that the methodology may be extended to other related fields, such as market research, quality control and biometrics. With continued interaction between researchers and practitioners this may, in the not too distant future, be realised.