Free Trial

Safari Books Online is a digital library providing on-demand subscription access to thousands of learning resources.

Share this Page URL
Help

7 What Sample Sizes Do We Need?: Part 2:... > Chapter Review Questions - Pg. 180

180 CHAPTER 7 What Sample Sizes Do We Need? Part 2 CHAPTER REVIEW QUESTIONS 1. Assume you need to conduct a single-shot (not iterative) formative usability study that can detect about 85% of the problems that have a probability of occurrence of 0.25 for the specific participants and tasks used in the study (in other words, not 85% of all possible usability problems, but 85% of the problems discoverable with your specific method). How many participants should you plan to run? 2. Suppose you decide that you will maintain your goal of 85% discovery, but need to set the target value of p to 0.2. Now how many participants do you need? 3. You just ran a formative usability study with four participants. What percentage of the problems of p = 0.5 are you likely to have discovered? What about p = 0.01, p = 0.9, or p = 0.25? 4. Table 7.11 shows the results of a formative usability evaluation of an interactive voice response application (Lewis, 2008) in which six participants completed four tasks, with the discovery of 12 distinct usability problems. For this matrix, what is the observed value of p across these problems and participants? 5. Continuing with the data in Table 7.11, what is the adjusted value of p? 6. Using the adjusted value of p, what is the estimated total number of the problems available for discovery with these tasks and types of participants? What is the estimated number of undiscovered problems? How confident should you be in this estimate? Should you run more participants, or is it reasonable to stop? Answers 1. From Table 7.1, when p = 0.25, you need to run seven participants to achieve the discovery goal of 85% (P(x 1) = 0.85). Alternatively, you could search the row in Table 7.2 for p = 0.25 until you find the sample size at which the value in the cell first exceeds 0.85, which is at n = 7. 2. Tables 7.1 and 7.2 do not have entries for p = 0.2, so you need to use the following formula, which indicates a sample size requirement of 9 (8.5 rounded up). ln 1 - pðx 1Þ lnð1 - 0:85Þ lnð0:15Þ -1:897 = = = = 8:5 n = lnð1 - 0:20Þ lnð0:80Þ lnð1 - pÞ -0:223 3. Table 7.2 shows that the expected percentage of discovery when n = 4 and p = 0.5 is 94%. For p = 0.01, it's 4% expected discovery; for p = 0.9, it's 100%; for p = 0.25, it's 68%. 4. For the results shown in Table 7.11, the observed average value of p is 0.28. You can get this by averaging the average values across the six participants (shown in the table) or the average values across the 12 problems (not shown in the table), or dividing the number of filled cells by the total number of cells (20/(6 × 12) = 20/72 = 0.28). 5. To compute the adjusted value of p, use the following formula. The deflation component is (0.28 - 1/6)(1 - 1/6) = 0.11(0.83) = 0.09. Because there were 12 distinct problems, 8 of which occurred once, the Good-Turing component is 0.28/(1 + 8/12) = 0.28/1.67 = 0.17. The average of these two components--the adjusted value of p--is 0.13. ! h i p est 1 p 1 1 1 1 - + p adj = est - 2 n n 2 ð1 + GT adj Þ