Masters Project - Usability Testing Using Think-Aloud Protocol

Tasks
Task 1: You have just received your coaching assignment and have logged onto the EPSS. Without leaving this page, think aloud about your thoughts on the homepage. What would you click next?
Task 2: You are in the Plan phase with your performer. You want to find information about the second step in the Plan phase. Locate the page with the second step in the Plan phase. Remember to think aloud as you work.
Task 3: You are on the fourth step in the Check phase with your performer. Locate this page. Find the forms available to you at this step. How would you access these forms? Remember to think aloud as you work.
Task 4: You need to print out extra copies of the “Participant Feedback Form.” Find this form. Remember to think aloud as you work.

See the task sheet given to users

Usability Study
The usability study was conducted after an off-site meeting for Teacher Coaches in Warner-Robins, GA. I went down there with my laptop and scheduled times following the meeting with the participants. The first two participants used the application at the meeting site; participants 3, 6, 7, and 8 tested in the lobby or breakfast room of their hotel; participants 4 and 5 tested it in their hotel room. Using a variety of locations in testing gives more ecological validity to the results since users will be using the tool at the home, school, or office. Also, participants 4, 5, 6, and 7 tested the software in pairs, in more of a co-discovery method style. This choice was made in hopes that the conversation between the two participants would provide additional insight into the tool's usefulness and ease of use. The participants that tested in pairs did seem to be slightly freer in their comments.

Notes from study

Subject 1
Familiar with Plan-Do-Check-Act

T1. “Plan-Do-Check-Act looks familiar”

“May need instructions – would not know what to do with coaching if I wasn’t familiar”

Used mouse to explore page. Said he would do that first to see what was active. Said he would tend to navigate using the graphic because he was visual. Would start with “Pre-Coaching”

T2. Click Plan from home page. Moves mouse around the page a little. Sees the boxes lighting up. Did not know P-2 was step 2. He recommended losing the “P-“ part. Likes the graphic on the upper right. It helps orient him. Likes the way the screen is laid out. Easy to follow.

T3. Make the forms help text easier. Doesn’t quite get the form across.

T4. I would search the site. I would type the form name in.

Overall, I really like the way it is set up. I like the tabs. No improvements except the numbering

Subject 2
First heard of LPPC program that day

T1. “I would Interact with the diagram/flowchart more than the text, but I am mathematical” Hovers around page and then clicked pre-coaching.

T2. Clicked Plan from PC-1 page. (NTS: Make entire tab hot). No underline is confusing. The color change is too faint. I asked about the How Tos…he responded they may be too small. He said he liked that he could see everything on one page.

T3. Clicked Check from P-2 page. Then, he clicked C-4. Talked about the mouse over functions of the links. “Usually the underline doesn’t disappear when you hover over it. You did something different.”

T4. Wasn’t sure how to find the form. When I pointed out the forms library link he said he didn’t even consider that section as part of the site.

Overall he said there was too many fades. The left menu wasn’t making an impact. “Side section isn’t bold.” Also, he said for some people there may be too many boxes. “Boxes inside of boxes.” But, he said it was okay for him because he was a math person.

Prefers C-4 to C-5, but didn’t really notice the difference at first. Colors were a little dull.

Subject 3
Was on the Design Team for LPPC

T1. Moved mouse around to discover you could click on the boxes. Likes the different options provided for entering the site. “When I get more used to it, I can go straight to the step. I like that.”

T2. Flow chart makes it easy to keep track of your place. I like that the tabs are always there. “The forms are always in the same spot. That’s easy.”

T3. Skipped this task because she had already gone through the phases and mentioned the forms.

T4. Found the forms library. Suggested I should reorder the left side menu.

Overall, she like the orienting side graphic. She liked the diagram. “Clean, easy to use & understand.” Let’s you know what the flow is and how many steps there are. She said she thought this was a really good way to present the information.

Subjects 4 & 5
That day was the first time they had heard of LPPC

T1. Liked the visual. Thought it was overall a little monotone & dull. It isn’t very eye-catching.

T2. Very self-explanatory. The display is very clear. liked the layout and tabs. Commented that they like being able to get back where they started. They didn’t feel lost.

T3. Recommended making the instruction text look different. Perhaps putting it on top. They thought it was a link. Also recommended that I be very specific with the instruction text. The order of words is important.

T4. Did not find the forms library. Blamed it on the white on blue text. Too easy to overlook. Said they might use the search function, but it might return too many options.

Overall, the colors are the problem. They are just dull and things run together. Make sure the really important things stand out. Mentioned the need for specificity again. They mentioned text may not be big enough.

Subjects 6 & 7
Familiar with LPPC

T1. Notices the steps text on the diagram. They like the different ways of entering the site, but mentioned that the step text at the bottom may be too small.

T2. Clicked on Plan and looked over the page. Commented that it is very clear. They like the flow chart. Mentioned that P-1 terminology might be confusing, meaning the terms used for performer, sponsor, etc. They mentioned that a definition might be useful. Liked the instruction text used to guide them on what to do with the forms. Linda mentioned that in other cases she can really get confused on what to do with all of the forms.

T3. Said they liked the diagram and the highlighting. They also liked the diagram on the upper right. I helps remind them where they are.

T4. Didn’t find the library, but said some kind of repository would be helpful. When I pointed it out, they said they really liked it.

Overall, they were very positive. They said the colors were pleasing. They didn’t want a site that was too catchy visually. Can be too distracting. Only criticism was the terminology. Should define those at least in the Pre-Coaching phase.

They also like that it told you where you are on the site in several different places. All yellow.

Subject 8
No experience with LPPC

She just ignored the tasks and went through the entire site making comments along the way.

Click Pre-coaching using the graphic. Began on PC-1 and moved along the line. She liked that the forms were always in the same spot. Said she would always know where to look. NTS: Make the entire tab active!

She thought the P-1 box in the Step Detail was a link. The “How To” was apparent to her. She commented on the before, during, and after text. She liked having that reminder.

Her overall comments were that it was a little dull. She recommended color coding the different phases. She likes the way the IBM Change Toolkit is.

She liked the additional links to the next step on the bottom. PC also has the previous step linked, which she liked that.



All participants immediately picked up on how the navigate through the site. They quickly got the number of steps on each page and that the step detail box would change when they clicked a new step box.