Masters Project - Analysis of Existing Systems

The systems included in the Evaluation of Existing Systems are all highly regarded as either excellent examples of performance-centered design systems or systems designed to support a long-term process goal. Three of the systems evaluated were recommended by Gloria Gery, the pioneer of performance-centered design, as notable implementations of electronic performance support systems. The fourth system, the IBM Change Toolkit developed by IBM and Rosabeth Moss Canter, a specialist in the field of change management, was chosen because it is an example of a system that scaffolds, supports, and encourages a long-term process, in this case change, in educational institutions. The support website shares commonalities with both types of systems.

In her article, “Ten Years Later: A New Introduction to Attributes and Behaviors and the State of Performance-Centered Systems” Gloria Gery lists ten online systems currently available to the general public as strong examples of EPSS (Gery 2003). Among her selections are Expedia.com, Do It Yourself, and TurboTax, which I evaluated against Gery’s 26 attributes and behaviors for performance-centered design (Gery 2000) and Jakob Nielsen’s usability heuristics (Nielsen 2004).

The entire analysis available here. Below is the portion describing the impact on the design of the support system.

Impact on Support Tool Design

Though there are many lessons to be learned from the evaluation of Expedia.com, DIYOnline.com, TurboTax, and the IBM Change Toolkit, none of the systems shared many similarities to the requirements for the Support Tool. The overall goal of an EPSS is to provide the knowledge and assistance the user needs to perform a task with speed and efficiency and without the assistance of another person. Expedia.com, DIYOnline.com, and TurboTax are all clearly designed to meet this goal. The IBM Change Toolkit, is not a traditional EPSS, but does aim to provide the user with the necessary information and tools to accomplish a task. The difference being the Change Toolkit supports a task that includes other people, occurs outside of the application, and has a long term goal extending over multiple sessions with the application. Of all the systems I evaluated, however, this system was most similar in nature to the Support Tool. It also happened to be the one meeting the least number of criteria.

The evaluation revealed the Change Toolkit meets very few of the standards set forth by Gloria Gery and Jakob Nielsen. The Toolkit doesn’t claim to be an EPSS, so failing to meet the standards of an EPSS is understandable. However, it is an online system, so failing nearly every usability heuristic is not acceptable. The lessons learned from the evaluation are considered in terms of the similarities between the Toolkit and the Support Tool, which are the organization of large amounts of information, establishing a shared vision of the objective, and providing resources to support change leaders.

Large amounts of data
The Toolkit’s method of organizing large amounts of information is to provide equal access to everything without nudging the user down the correct path for his needs. The site buries a link to the first action a new user would want to take, creating a new project, in the third slot under the heading “Get Started.” After creating a project, the site provides not direction for the next step. The lesson learned is to anticipate the goals of the user and make the path to achieving those goals obvious. The toolkit’s text-heavy “New User Introduction” is uninviting and excessive. The lesson learned is to create layers of information, starting general and moving towards more detail. This approach gives the power to the user to retrieve as much or as little information as he desires.

Establishing a shared vision of the objective
The Toolkit uses a number of pages to sell the vision of change as embraced by IBM and Rosabeth Moss Kanter, but again unwisely chooses to employ vast amounts of text on pages that require scrolling to accomplish much of it. The lesson learned from this features is described above. However, the Toolkit does have an interactive presentation (that took the user to another site that required re-logging in) that well explains its methodology, but it is placed under the heading “Guided Tours and Lectures” rather than incorporated into the text pages as an alternative mode of communication. The Toolkit is really strong in two aspects of creating a shared vision. One is extensive cross-referencing that explicitly link concepts together for the user to ensure the user’s mental model of the toolkit matches the actual toolkit. Another is the use of vignettes for each topic to better communicate the effectiveness of the presented tool by relating a real-life account of its use. The Support Tool would highly benefit from heavy cross-linking in order to reduce the time the user spends searching for related documents and resources. Vignettes of success stories would also be an asset in the Support Tool for inspiring excitement and commitment to the program.

Providing resources The Toolkit supports change leaders by institutionalizing five tools that tailored to each topic. The first is an overview of the topic that crosslinks three other topics, the second is the background section that provides more information, the third is the diagnostic tool with helps users identify or confirm a problem they may have within their organization, the fourth is the action tools that aid in planning next steps, and the fifth are the vignettes. The virtues of crosslinking and vignettes were discussed above, but the background, diagnostic, and action tools all support the user in various ways. The background tool provides detailed information about the topic and motivation for action. The diagnostic tool provides a mechanism for needs assessment that can reinforce the case for change the user is trying to make. The action tool provides concrete suggestions on how to proceed from the current point. The lesson learned is that the coach likely will face pushback from one or more members of the collaborative at some time during the coaching engagement, therefore creating an opportunity for the Support Tool to meet a need. The Support Tool should create topics on the subject of each point of potential pushback that include problem identification, detailed information, motivation to act, and steps to overcome.

The similarities between the Support Tool and Expedia.com, DIYOnline.com, and TurboTax were surprisingly few, but worth discussing. Of Gery’s 26 attributes and behaviors, the Support Tool will aim to implement 13 of them: representing the context, establishing a work context, aiding in establishing a goal, structuring a work process, reinforcing task to business strategy, institutionalizing current best approach, using metaphors and direct manipulation, provide alternative views of support resources, provide contextual feedback, provide support resources within the context, provide alternative navigation mechanisms, provide obvious next steps, and maintain consistency. In regards to these 13 attributes, the Support Tool can benefit from an examination of how the three systems did or did not meet Gery’s standards. All three systems chose to use task-related language rather than computer language such as “Select a hotel” rather than “Click the item in the list below.” Also, direct manipulation was heavily used, especially in the deck designer. The Support Tool will not have a need for drag-and-drop manipulation, but can benefit from the three system’s heavy used of radio buttons, list boxes, and sliders. DIYOnline and TurboTax both used a progress bar to orient users to where they are in the work process. Expedia did not use a progress bar because making reservations requires far fewer steps, but it keeps users oriented with a large title bar announcing the task (“Create your package from Atlanta to Miami (and vicinity”). The systems help the user choose an appropriate path by asking questions or using links that identify a goal (“Design Your Deck Now” button), which the Support Tool will use to help guide coaches through the system. In order to structure the work process, the Support Tool will employ a technique most similar to TurboTax’s progress bar. TurboxTax displays a graphic throughout the user’s interaction that shows the total number of steps and which step the user is on. Unlike the deck designer or Expedia, the Support Tool will not force the user to make a decision before proceeding to the next step. Despite a few unclear messages, the systems also provided effective contextual feedback to lead users to correcting and avoiding mistakes. Contextual feedback will be essential to the Support Tool because users, especially the inexperienced, quickly become frustrated when error messages are too vague. The Support Tool will also provide alternative means of navigating the site, such as top and side menus, site maps, and a search mechanism. Like the three systems each do so well, the Support Tool needs to provide the user with obvious next steps to reduce his cognitive load, which is one of the Support Tool’s primary goals.

An examination of the implementation of performance-centered design and usability guidelines in each of these systems helps to inform the design of the Support Tool by providing examples and the outcomes of design decisions. It revealed which of Gery’s 26 attributes would be applicable in the Support Tool as well the importance of following Nielsen’s usability heuristics in every online application.

References

Gery, G. (2000). "Attributes and Behavior of Performance Centered Systems Chart." Retrieved July 26, 2005, from http://gloriagery.com/articles/attributesandb.

Gery, G. (2003). Ten Years Later: A New Introduction to Attributes and Behaviors and the State of Performance-Centered Design. EPSS Revisited: A Lifecycle for Developing Performance-Centered Systems. G. J. Dickelman. Silver Spring, MD, International Society for Performance Improvement: 1 - 3.

Nielsen, J. (2004). "Ten Usability Heuristics." Retrieved July 27, 2005, from http://www.useit.com/papers/heuristic/heuristic_list.html.