“Here we go again. . . ”
Do you ever find yourself saying that as you roll up your sleeves and get ready for the barrage of phone calls from executives and managers that can’t seem to remember how they accessed their online Performance Review forms? Worse yet are the managers that don’t seem to remember how to fill them out once they access the form?
Every year, there are companies that lose valuable productivity because of ineffective annual performance appraisal training. Managers have 12 months between reviews and because they don’t re-enforce the skills learned on a weekly or sometimes even monthly basis, they often forget.
Human resource trainers frequently have to provide a refresher’s course or spend valuable one on one time with corporate executives, walking them personally through the process once again.
Some companies have turned to computer based training methods as a convenient solution. Online performance review forms and training can be purchased in neatly packaged “off the shelf” or cloud based performance review systems. These products are quite convenient but can be very costly for large organizations.
One such corporation had already invested in developing their own “in-house” performance review system. They just needed a more effective way to provide support for their management team.
Entertainment and Resorts (A fictitious name has been provided for corporate privacy), consulted with the learning technologies department at Harrisburg University of Science and Technology in Harrisburg, PA to find such a solution. I was the graduate student selected for the task to transition corporate classes into effective e-learning modules.
The project was the capstone for my graduation requirements. The last 6 credits of my degree were to be fulfilled with an experiential practicum. Although the internship itself was 80 paid hours, the process spanned over 6 months due to the rigorous graduation requirements of the practicum itself. I was blessed to have a department head (Andy Petroski), that truly believes in a high quality education and makes his students step up to the plate before issuing us our Masters of Science in Learning Technologies.
Executive Summary
A Brand Name “Entertainment and Resorts” Corporation seeks an online training solution for their performance review. Currently training takes place in the classroom. “E & R” would like to convert the performance review course online so that managers may access the training as needed. The training will be a hybrid of performance review best practices and data entry procedures for the in-house IBM Lotus Notes database software.
Project Goal
The goal of the project is to:
- Successfully launch a custom performance review e-training module by summer 2011
- Utilize visuals of the Lotus Notes database to explain how the software tools support the efforts of the performance review process
- Produce an effective learning module that will integrate performance appraisal standards with online learning requirements, subject matter and learning objectives
Project Objectives
By spring 2011, an instructional technologist will
- Identify course subject, instructor and materials
- Identify online learning requirements
- Identify course objectives
- Design flow chart
- Design storyboard
- Implement course prototype
- Evaluate prototype
Project Milestones
- Research
- Flowchart
- Storyboard
- Design document
- Create media
- Develop
- Evaluation
Project Conflicts
The project must fit into Lotus Notes software which is not a Learning Management System. The course must be built remotely which may have a different Lotus Notes interface then some employees. The help desk reported that there are several different Lotus Notes interfaces and that working remotely may not really make a difference anyhow.
Course selected
Performance Review was the course selected for the project. Currently, the performance review training takes place in the classroom in the late third quarter. The new training will allow the large quantities of employees to review content quickly and efficiently.
Objectives
- Provide information about the Employee Performance Review database and available tools for creating a meaningful and effective environment for supporting, tracking and evaluating employee performance on an ongoing basis
- Review the performance review process impact as it relates to employees, managers and the corporate model as a whole
- Demonstrate the specific steps to an effective performance management process such as Goals, Self-Appraisal, Performance Logs, Review, Ratings and Submitting
Timeline
The finished tutorial will be approximately 45 to 60 minutes in length. Opportunities for chapter review are provide at the end of each chapter.
Course Materials
E & R strives to be as paperless as possible. However, hourly employees that do not have access to Lotus accounts will need to have a paper version printed out by their supervisors to fill out. The tutorial will address the steps to this procedure.
Profiles
Entertainment & Resorts (E & R, web.2011) is a privately held company founded in 1920’s when the founder separated his manufacturing operations from his other businesses. Entertainment & Resorts is a hospitality company that employs about 1,500 full-time employees and 7,500 seasonal and part-time employees.
E & R Mission (E & R, web.2011)
Be a leader in the hospitality and entertainment industries by delivering excellence to our guests and employees, while enhancing our relationships with the company entities and the community, and sustaining financial integrity.
Online Learner Profile
Most of the learners will be in the supervisor or management category. Most hourly employees do not have Lotus Notes accounts and will have a written performance review. The learning module will address both salary and hourly training.
Technical Environment
E & R does not currently have a Learning Management System. Lotus Notes hosts an in-house performance database software system managing employee performance reviews. The training department is able to track assessment scores through captivate and Lotus notes. However, Human Resources is not concerned about keeping score from the performance review assessments The e-training module will display video screen capture while demonstrating preferred input methods of performance review data. Each section will provide best practice tips to maximize the training. At the end of each chapter, a short quiz will be used to reinforce skills.
508 Requirements
The graphics, text and voice-overs are designed to provide the necessary information to successfully navigate through the Performance Review database for both Salaried and Hourly employees. Computers must have speakers, the speakers must be turned on and the volume turned up. A scripted version will be created for employees with hearing disabilities. Employees and Managers may refer back to the tutorial at any time for a refresher.
Company ValuesThese are the values that are important to the strengthening of the E & R company and community. Employees may create up to three goals for each value, but are cautioned to keep goals in check, as they will be held accountable for them if they are not met. Currently, a help icon may be clicked for an example of each goal. Professional Development was a required value this year and a minimum personal goal in this category must be met. |
||||||||||||||||||
|
(Entertainment and Resorts Corporation, web.2011)
|
Performance Review Research
The research component of the project explores current trends in performance review and best practices for transitioning classroom based training into effective e-learning. A Gantt chart was penciled out to establish a tentative framework to realize project scope.
Research Goals
Research questions were established in order to:
- Find an example of Performance Review training software
- Consider “Best practices for transitioning to online learning”
- Explore current trends in Online Performance Review training
Research Methods
Research methods included questionnaires, interviews, phone calls, e-mails, internet searches, social networking and forums. Comparable classes were challenging to find as the course uses common key words in searches and often lead to advertisements and unrelated material.
Authoring Tool
Captivate 5 is the tool selected for the project as E & R has recently purchased several licenses for their training department. E & R is not concerned about keeping score from the assessments. Assessments at the end of the chapter will serve the purpose of reinforcing the material.
Current Industry Practices
Each research component provides valuable guidelines and considerations for the performance review project. Video screen capture and software simulations can be authored easily using the adobe captivate authoring tool and can bring a visual representation to the learner. Unbiased qualitative performance review feedback is essential to an effective review and was examined for ideas on best practice approaches throughout the module.
Slide Conversion
A power point to online performance training module was researched for comparison. The University of Utah, Division of Human Resources (Utah, web. 2011) provided an interesting example of a simple employee performance review e-learning module. The module focuses on knowledge and attitude, showcasing the purpose of the employee review process. The form to fill the module is a PDF printable rather than an interactive in-house software e-form. Directions are simply a link to the form.
The University of Utah Online (Utah, web. 2011) employee performance appraisal reveals a string of linear slides repurposed for the web with limited interactive features. Role-play illustrations and simple assessments are assigned at chapter ends. Class participants are instructed to print and fax answers to human resource personnel for proof of course. Links are cleverly highlighted for additional expansion in gray content areas. A list of resources is provided for the learner to further expand their understanding at the end of the module.
Video Screen Capture
Research has demonstrated that the concept of using visual representation along with text assists learners with high visual spatial aptitudes (Clark and Lyons, 2004). The current trend toward learning occurs when visuals are relevant, simple, clean and not distracting.
A third party performance review demonstration (promantek.com, 2011) utilizes video screen capture by panning in and out as an arrow clicks through the software. The screen darkens momentarily while points of interest such as a button or a check box are highlighted. A similar demo (silkroad.com, 2011) manipulates an arrow around the screen while screen shot stills transition in and out of the frame. The designer cleverly creates a video capture effect and reduces unnecessary movement in the process, giving it a cleaner, more professional appearance. Overall video screen capture and possibly total file size may actually be reduced. Areas of interest are colorfully shaded and eye-catching photography stills are incorporated throughout.
Storyline
An adult learner pedagogy model (MSU, web. 2011) suggests that learner relevant course content and lesson times be clearly identified. Course content will include a relevant storyline to interest the learner. Storylines show the learner how a person might apply a performance review in a variety of scenarios.
Every lesson activity will give the learner an approximate time to complete. Learners that complete the task before the time expiration will not be penalized. However, the nature of the Cisco Server may time out if left idling. Learners should set aside an uninterrupted hour to complete the course.
Finally, the storyline component focuses on a more personal feel. A good storyline uses realistic scenarios to get the learner thinking about how they may draw upon skills they have already used in real life settings. For example, a learner may understand the term software tool to a word document or typewriter. Another learner may understand that performance logs are also used as documentation for writing positive and negative comment about employee performance and habits, similar to keeping a diary or a journal.
Rating Bias
E & R takes a cutting edge approach to employee evaluations. Management does not assign quantitative numbers in their performance reviews. The Performance Appraisal Question and Answer book (Grote, Richard 2002) supports this philosophy. Dick Grote argues that the best form of evaluation is not numerical, but rather a descriptive summary of specific behaviors and how employee behaviors impact the organization. Employees may feel that their evaluation has more personalization with a descriptive summary, rather than the traditional numerical scores. Therefore, the supervisors that provide the evaluation must put more thought in the appraisal and word it in such a way that demonstrates objectivity.
The IT manager’s handbook (Holtsnider, Bill, and Jaffe, Brian 2007) stresses the importance of keeping performance reviews objective rather than subjective in order to avoid wrongful termination lawsuits. The key areas of evaluation with most employees are typically; work quality, flexibility, creativity, communication, timeliness, accountability and interpersonal skills. Workers that go above and beyond the requirements of the job and access self-training are more likely to get better reviews.
Performance Review Feedback
Effective feedback is both objective and specific (DOCHROC, 2010). Performance feedback that is expressed positively and accurately is more relevant to employees. Each evaluated behavior should be measurable as it relates to the employees. After performance reviews have been noted in the system, the review should be given to the employee in a personal conference. Employees should be prepared for this meeting by listing self-accomplishments and future goals for performance improvement.
Negative Reviews
Negative reviews can be especially touchy and even unpredictable. It is best to inform human resources if the review is going to be negative and maintain professionalism throughout (DOCHROC, 2010). Performance behaviors, rather than personality traits, should be addressed with specific details, such as the date and time, to maintain clarity. Sensitivity to privacy and tone of voice can make the process go more smoothly.
Implementation
Nick Shackleton Jones from the Aconventional.com blog recommends considerations with implementation strategies for course completion during the design phase. An excellent e-learning module is not going to be very proficient if only five percent of the users are actually completing the course. For this reason, the course design will propose that learners walk through each of the subcategories before allowing them to go through sub-menus for specific review.
Research Conclusion
In conclusion, corporations should set realistic standards for employees to be measured against. Opportunities for re-training should be provided whenever possible. Performance review is an opportunity to remind employees how their role plays a part in the workplace operations and how their behaviors can shape corporate success.
The flowchart (Appendixes A-C) was the most challenging in this particular project as it merged skills related content with knowledge and attitude material for the performance review process. Previously, the courses were taught separately or back to back in the classroom. The final product is still to be determined but will likely include pop up boxes with best practice tips during skills training.
Flow Chart (Attachment A)
The most valuable lessons that were realized in the research process were the question and answer forum boards on Linked-In. I was able to pose a question about research tips when I felt as though I had exhausted resource leads. The Linked-In forum allows the question to be posed to your friends list by personal invite. I received a significant response and was really grateful for by my colleague’s suggestions.
Course Design
The course design was created in three sections. Flow charts were used to get an understanding of how the course would flow. A design document further expanded the flow charts for content clarification. Finally, the storyboards were created to provide the developer the necessary details for executing the design.
Flow Chart (Attachment B)
Scope of Project
Performance review classroom training will be converted into an online learning course. The project will include repurposed content from Power Points and classroom materials. The course will not include Lotus Notes training and does not teach learners management and documentation skills. The training will fall under the assumption that the learner has some familiarity with the Lotus Notes Database. It is also assumed that learners have previous experience effectively documenting employees and creating performance reviews through the classroom and other mediums. Best practice tips, however, are provided throughout the course to expand on these skills.
Course Goal
The goal of the course is to answer any questions learners have about how performance reviews are done at E & R. Visuals of the tools are used to cover specific steps and explain how they are there to support team members in implementing the annual performance management process.
The primary audience of the course will be corporate managers and directors. A variety of age, sex and races will be represented. The delivery strategy will be interactive e-learning. The e-learning course is designed for managers to take the course at their convenience through the Lotus Notes Database. Learners must already be fluent with navigating through the database before taking the course. Learners may refer back to specific modules as needed. The Entertainment and Resorts Performance Review Tutorial will be mainstreamed for management use by October 2011.
Course Objectives
By the end of the course, team members will be able to:
- Explain the steps to effectively create a performance review
- Differentiate between salary and hourly performance reviews
- Design a goals document
- Maintain employee performance logs
- Create a self appraisal
- Assemble a performance review
- Properly submit documentation through Lotus Notes
Resources are outlined for learners during the course. A help page has been created and frequently asked questions are available on the Lotus Notes Database.
E-learning modules:
The course was broke down into five modules. An introductory module explained to learners the performance review course navigation. The three subsequent modules identified the steps to provide necessary attachments for the performance review. The fifth module was the performance review itself. Learners are walked through the automated process of assembling attachments into the performance review and properly submitting the review to the Human Resource Department.
Module 1: Introduction
Learners will explain the steps to effectively create a performance review. An e-learning module will be delivered with the definition of the performance review process as it relates to productivity and examples will be shown. This e-learning module distinguishes between salary and hourly performance reviews.
Module 2: Goals Document
The goals document is the first of three attachments that must be created before authoring the performance review. Learners will identify the steps to design a goals document for their employees and themselves. The goal document starts the process.
There are three sections in the performance review documents. Supporting Our Values defines the four core organizational values. Supporting Our Mission defines five key areas by which E & R measures their business. Professional Development allows learners to define specific, professional growth goals. At least one goal must be in the Professional Development section this year. An effective goal will be in line with Entertainment and Resorts values and mission.
Module 3: Performance Logs
Performance logs are used to document, track, make notes and add comments throughout the year on employee performance. Both hourly and salary performance reviews will incorporate this important step.
Managers and employees create one performance log for each relevant category. All of the categrories do not need to be used. Throughout the year, managers and employees may edit any of the nine possible logs categories.
Module 4: Self Appraisal
The self-appraisal is optional but provides managers and employees valuable information for rich dialogue about performance. Self Appraisal is virtually a clone of the performance review tool. Employees complete the self-appraisal by clicking the appropriate rating for each behavior. Items cannot be skipped. Performance log comments are automatically loaded under the comments section. Additional comments can be entered by the employee.
Module 5: Performance Review
This module takes the learner through the automated steps of incorporating the previous attachments into the performance review. Instructions for the rating scale are defined. Self appraisals and performance logs are automatically detected by the Lotus Notes Database. Any performance log comments employees add to their self evaluation are included. Steps are provided for learners to seamlessly incorporate attachment documentation throughout the review. Learners will identify the differ features for the performance review and links to supporting documentation throughout. Best practice tips are provided for the learner to administer an effective interview.
Outline
Explain the steps to effectively create a performance review
- Access the directions for the course
- List course objectives
- Access course modules
- List the functions of the main menu
- Identify the sequence of the course modules
- Explain how the performance review process increases productivity in business
- Access tool features to assemble performance tools more efficiently
- Access a hard copy from HR
- Access the help menu
Differentiate between salary and hourly performance reviews
- List the steps of the salary performance review process
- Determine whether a goals document is required
- Access a hard copy from HR
Design a goals document
- Access and open goals document
- Understand the purpose of creating a goals document
- Add goals to a goals document
- Define goals and best practices for authoring them
- Submit goals and Check status
Maintain employee performance logs
- Defining the purpose of performance logs
- Creating performance logs
- Adding performance logs
Create a self appraisal
- Access self appraisal
- Importing a performance log
- Define rating
- View goals on a self appraisal
- Submit and acknowledge a self appraisal
- Re-import and check the status of a self appraisal
Assemble a performance review
- Start a Performance Review
- Understand the process of attaching performance logs to a review
- Accessing Managers performance Logs
- Scoring an overall rating
- Incorporating best rating practices on a performance review
Properly submit documentation through Lotus Notes
- Submit a performance review
- Understanding an approving managers role during a review
- List procedures after the review
- Use an electronic signature
- Check the status of a submitted performance review
- Utilize the functions of the help menu as a resource for further questions
Design Attachments
The following five attachements are the instructional design documents for the e-learning course. Attachments A-C are the flow charts. They are used to see how the course will flow together. Attachment D is a design document. It details the learning componants of the course. The Attachment E is the storyboard. The storyboard provides the developer all of the necessary details to make an effective learning experience for the learner.
Flow Charts
The flow charts turned out to be more difficult than originally anticipated. An appropriate design was selected after several drafts were authored. Flowchart.com was the authoring tool that the flow charts were authored in. Attachment A is an overview for the performance review course. Attachment B breaks down the components of the course modules. Attachment C provides a diagram of the performance review process.
Flow Chart(Attachment C)
Design Document(Attachment D)
Because of the complexity of the flow charts, a design document was used to further clarify and break down the components of the course. The design document provides a more detailed look then the flow chart or outline, but without the intimate breakdown of the storyboards.
Storyboards
The storyboards provide detailed information about the breakdown of individual slides for the purposes of passing the design to a developer. Each slide is set up as a stage, with graphics entering and exiting the scene. The first column lists the slide numbers and titles. The second two columns provide information about the slide in context to the learning objectives. Column four lists all of the text that will be visible to the learner on the stage. Column five is the verbal script that will be read to the learner, as the learner watches the graphics enter and exit the stage as visual reinforcements. Column six describes the activity that will take place on the stage. Column seven identifies the graphic titles and numbers that will be used during the scene. Column 8 refers to the amount of time that the learner should expect to view or hear activity on the stage during the scene.
Storyboards (Attachment E)
The first page of the storyboards shows the main menu layout and how the background should look by providing an example for the developer. The buttons to the main and help menus are listed along with buttons and graphic hot spots linking to corresponding slides.
There are 17 total pages of storyboards for 99 slides on captivate. The following screen shots were selected for further examples:
Design Conclusion
A wiki was used to provide a sandbox for the design through wikispaces. The wiki provided a place for project supervisors to make their comments and view the project progress. The flowchart embedded cleanly into the wiki; however any editing had to occur directly from flowchart.com. Unfortunately, the glitches in the flowchart provided for several rewrites and caused the project to be delayed from the original timeline. The timeline was restructured to provide for additional time during the summer. The design experience, however, was interrupted when the client requested a new deadline. The new deadline was abruptly pushed back, sending the project promptly into the development phase.
All learning objectives and sub-objectives were designed before development. The bulk of the text and scripting were designed during the development stage. Most of the scripting came from materials provided by E & R from the classroom course Power Point slides. Script was consolidated and re-purposed for the e-learning course. The script appeared to have different authors and used second (you) and third person (he/she) terms interchangeably. I rewrote the script to be read from a neutral third person narration. Regrettably, I should have consulted with the client as they purposefully designed the script to be narrated in second person. They took the changes and put them back into second person as they wanted to create a more intimate feel. Since this occurrence, I have noticed that several other e-learning modules follow the second person narration suit (The English Teacher in me cringes!)
Over half of the design was completed during the development phase. The sections left blank were the activities and graphics column. I had to come back in and describe stage movement and name the graphics.
The graphics were not all named because of the sudden immediacy to get the job done. The great thing about captivate is that it assigns a number to files that do not have names and organizes the files automatically. This proved to be especially helpful as I grasped to think of the most effective way to describe design files for a developer.
On a personal note, I have struggled finding a storyboard that really helps me flow and freely author. In the past I story-boarded directly on PowerPoint but I felt that the screen was too confining.
During the storyboarding, one of the processes that seemed to take a bulk of time was formatting all of the wording in the activities and media columns to look uniform. I spent several hours using bold, italics and adjusting upper and lower case errors. I also spent time re-arranging the graphic and activity content in a particular order of appearance as to be less confusing to a developer.
The E & R storyboards in the previous examples are on a word doc. I still felt confined to a box during the project and In the future, I will take those considerations to generate a more design friendly storyboard template to help streamline the process. If possible, I plan on designing a spreadsheet storyboard template on my next project.
Development
Introduction Information
|
Title: |
The Performance Review Process |
|
Created by: |
Angela Rupert in conjunction with E & R |
|
Supervised by: |
Andy Petroski, Harrisburg University of Science and Technology |
|
Location of Program: |
Entertainment and Resorts |
|
Number of Participants: |
Ongoing class for corporate management. |
Topics to be Covered
- Performance Management Process
- Goals Documents
- Performance Logs
- Self Appraisals
- Performance Review
Media Requirements
Power Point and Captivate will be used to create an e-learning course. Graphics, screen capture and media sources will be used from Microsoft and Adobe licenses. Users will be required to use their speakers or a headset during the course in order to hear the narrative.
Media and Tools
|
Media: |
Screen capture, simulations, photography, graphics, and animation, text, text to voice, audio files |
|
Tools: |
Captivate, Power Point, IBM Lotus Notes Database, Word, Excel |
|
Virtual Tools: |
Google Docs, Flowchart.com, Wikispaces, Virtual Server |
Delivery Methods
|
Objective |
Skills Taught |
KSA |
Delivery Method |
| Explain the performance review cycle |
|
K |
Provide learner with an overview of the performance review process |
| Differentiate between salary and hourly performance reviews |
|
K |
Distinguish between different types of employees by using case studies as examples in a narrativeEngage learner by incorporating hot spot formative assessment questions |
| Create a Goals Document |
|
S |
Simulations will provide the learner a step by step self paced tutorial to open the Lotus Notes Database |
|
A |
List SMART goals and provide case studies as examples in a narrative | |
|
S |
Screen capture will enter and exit the stage as key features are highlighted | |
| Maintain Performance Logs |
|
A |
A narrator will explain the performance logs purpose as a means to document behavior |
|
S |
A true/false formative assessment question will engage learner | |
| Create Self Appraisals |
|
S |
Screen capture will enter and exit the stage as key features are highlighted |
|
K |
Animation runs in the background while image is highlighted | |
| Assemble the performance review |
|
S |
Screen capture will enter and exit the stage as key features are highlighted |
|
S |
A true/false formative assessment question will engage learner | |
|
A |
Text will enter the stage as key features are narrated | |
|
S |
Screen capture will enter and exit the stage as key features are highlighted |
Formative Assessments
- Learners will be asked true and false questions occasionally during the course
- Learners will unlock the different modules as they progress
- Hot spot questions will allow learners to recall prior knowledge and select an appropriate response.
Trainee Evaluation
Learners will use a mouse to forward slides at their own pace. This will not only keep them engaged, but allow for pause if using a simulation or staged tutorial to walk them through the steps of the performance review in a separate window. Human Resources will be able to assess the effectiveness of the tutorial by the number of phone call requests received by the help desk for assistance with the Performance Review Process.
Development Conclusion
The development of the project was rushed before the storyboards were completed. A crude draft was put together for the new deadline. The client appeared satisfied with the course progress and requested to personally voice over all audio files. All of the graphics and media were somewhat timed to demonstrate their order of appearance on stage. Final graphics and text stage appearances’ would need to be correspond with the narration.
Captivate has a feature that allows the author to reduce file size conveniently. Files that have been abandoned or unused can be deleted from the library with the push of a button. The same image is recycled and captivate provides an unlimited amount of instances without greatly increasing overall file size. The file size was still too large to send through e-mail so, E & R requested a jump drive as the deliver method of the e-learning course.
Because of the rush deadline rush, I was unable to identify media and author interactions before the development stage. There are pros and cons to developing before storyboarding. On the surface it seems there is a certain amount of freedom to move graphics around if you are developing on the fly. Sometimes the vision of the graphics and how they will move together and interact appears when I see them on the stage. However, by creating the storyboards after the development stage, I still have to still make significant adjustments to the project to keep it in line with objectives. Media and text appear to be more uniform as well. In the end, I think that planning and organization from storyboarding in the beginning saves more time.
Assessment and Evaluation
The assessment and evaluation phase is an integral part of instructional design. The Performance Review course provides the learner with three separate tiers of evaluation; formative assessment, final assessment and a course evaluation. Formative assessment questions are administered throughout the Performance Review Course. Final assessments were created to measure skills and reinforce learning. Finally, course evaluation gives learners the opportunity to provide feedback for the course.
The assessment and evaluation will be designed directly onto a Google form so that learners can take the course and evaluations without any assistance. A small group of five or more learners have been selected by the Human Resource Department to pilot the course, assessment and final evaluation.
Formative Assessment:
In e-learning, assessments are designed to create interactivity by providing formative questions throughout the course. Self directed asynchronous e-learning can be enhanced by pop up questions and hot spots to make learning more engaging. A learner that misses a formative question may choose to review material before progressing to the next level, therefore, increasing skills retention. In the Performance Review Course, formative true/false and multiple choice slides were authored to synthesize learning material. Real case scenarios were added along with questions to afford practice opportunities to make correct decisions.
Final Assessment:
A 15 question final assessment was created to measure learner success at the end of the course. Questions were also designed to reinforce skills. Learning theories such as blooms taxonomy were used to roll back questions into the course objectives. Multiple choice, fill in the blank, true/false and checkboxes were used to provide learners a variety of methods to demonstrate their knowledge, skills and attitude. Higher level blooms text based questions such as short answer and essay response were avoided do to the high volume of managers anticipated to take the course. Some of the formative questions from the e-learning course were repurposed for the final assessment. The final assessment was designed not only to measure learning outcomes, but reinforce skills. Learners are not required to retake the course if they do not pass the skills assessment.
Evaluation
An 11 question evaluation was constructed to provide learner feedback. Slides or assessment questions can be adjusted accordingly, especially if errors or problematic patterns are detected. Research was conducted to evaluate best practices for authoring e-learning assessments. All evaluation and assessment research searches were catalogued. Learning theory, best practice tips and e-learning assessment examples were carefully considered before authoring questions for the final assessment and evaluation sheet.
Evaluation Theories
Donald Kirkpatrick
The same techniques are commonly used to develop e-learning course evaluations as a traditional course. Kirkpatrick’s classic model of training is still commonly used to differentiate levels of training evaluation (Kirkpatrick, 1994).
- Level one evaluations measure the course as an effective learning tool
- Level two evaluations measure content retention
- Level three evaluations measure behavior on the job
- Level four evaluations measure the results of business outcome
The ten question evaluation will be used at a Kirkpatrick level one to provide feedback about the course. The fifteen question assessment will be administered to learners at the end of the course on a Kirkpatrick level two.
Level one evaluations measure learner reaction to the training and how they feel about their experience. It is important to gage whether learners feel that the material is relevant to their job. Some may argue that the only thing level one evaluations measure is if they like the course. (Strother, 2002)
| Evaluation Level | Description and Characteristics | Examples of Evaluation Tools and Methods | Relevance and Practicability |
| 1. Reaction | Reaction evaluation is how the delegates felt about the training or learning experience. | ‘Happy sheets’, feedback forms.Verbal reaction, post-training surveys or questionnaires. | Quick and very easy to obtain.Not expensive to gather or to analyze. |
| 2. Learning | Learning evaluation is the measurement of the increase in knowledge – before and after. | Typically assessments or tests before and after the training.Interview or observation can also be used. | Relatively simple to set up; clear-cut for quantifiable skills.Less easy for complex learning. |
| 3. Behavior | Behavior evaluation is the extent of applied learning back on the job – implementation. | Observation and interview over time are required to assess change, relevance of change, and sustainability of change. | Measurement of behavior change typically requires cooperation and skill of line-managers. |
| 4. Results | Results evaluation is the effect on the business or environment by the trainee. | Measures are already in place via normal management systems and reporting – the challenge is to relate to the trainee. | Individually not difficult; unlike whole organization.Process must attribute clear accountabilities. |
|
© Alan Chapman. A free resource from www.businessballs.com |
|||
Fitzpatrick, Sanders and Worthen
In the publication, Evaluating e-learning; a Guide to the Evaluation of e-learning (Attwell, 2006; Fitzpatrick, Sanders, and Worthen, 1997), Graham Attwell merges two learning theories to provide a broader spectrum of evaluation approaches. Atwell identifies with five of Fitzpatrick, Sanders, and Worthen‘s evaluation approaches:
- Objectives-orientated evaluation approach
- Management-orientated evaluation approach
- Consumer-orientated evaluation approach
- Expertise-orientated evaluation approach
- Participant-orientated evaluation approach
Objectives-orientated evaluation approach
Objectives-orientated evaluations are considered goals driven and systematic. This type of evaluation is often used by companies that need to justify costs by demonstrating a correlation with performance measurement objectives. Sometimes other results that are not objective oriented are overlooked. The questions selected in the final course assessment roll back up to the objectives.
Management-orientated evaluation approach
The management-orientated approach uses evaluation information for effective decision making. Flow charts and decision outcome models focus on the evaluation needs of managers. Other stakeholders can sometimes be overlooked in the process. The level one evaluation component will evaluate whether the needs of the managers were met by the e-learning course.
Consumer-orientated approach
The consumer orientated approach uses evaluation to improve service or products for the end user. The focus is summative rather than predictive and relies heavily on criteria referenced benchmarking. A small portion of the evaluation will ask end users how the Performance Review course can be improved. Both the assessment and evaluation will generate statistics into a spread sheet for the Human Resource department. Human Resources can keep track of the answers and evaluate questions that need to be changed or the delivery of information if too many learners are missing the same question.
Expertise-orientated approach
Expertise orientated evaluation is based on the subjective professional opinion of experts. The peer review process is most commonly associated with this approach. Expert opinions may show bias and lack the reliability of an instrument based evaluation. The peer review process, however, is one of the oldest forms of evaluation. Despite the drawbacks, peer review works well in situations where subject pools are small and personal feedback is welcomed. Experts from the Human Resource department were invited to voluntarily participate in the course evaluation process.
Participant-orientated evaluation approach
The participant-orientated approach assesses a wider audience of project stakeholders then traditional evaluations. By gathering data from a variety of sources, evaluators can more readily identify problematic patterns and make adjustments where necessary. This type of evaluation is often expensive and difficult to administer due to time constraints. Participants from outside of Entertainment and Resorts were not invited to participate in course evaluation.
Peter Van der Knaap
Learning-orientated evaluation approaches
Atwell expands Fitzpatrick‘s program evaluation guidelines further by adding an additional learning-orientated approach by Peter Van der Knaap. Van der Knaap’s ‘Learning Oriented Approach’ is a relatively new classification (Attwell, 2006; Knaap, 2004). The purpose of the learning oriented evaluation is to contribute to an overall learning experience. Formative evaluation is emphasized and questions are deigned to extend learning. This approach was incorporated in the assessment components of the performance review. Formative questions during the process get the subject to think about how they will use the new skill on the job. Learning oriented assessment questions at the end of the e-learning performance were not only designed to measure skills but reinforce them.
Criteria for Consideration
Research indicates that there are many different criteria that can indicate the effectiveness of course design. Research indicates that e-learning inventories should be designed primarily to account for the course’s ability to support business objectives, evaluate course design and provide for cultural sensitivity (corpU, 2007).
Business Objectives:
The Performance Review evaluation sheet will enable learners to indicate that the course provided the necessary knowledge, skills and attitude to achieve the company’s desired results. Strategic questions will ask learners about course objectives and company goals. Employees can explain what they will need to accomplish these.
Course Design
Questions for a course evaluation should account for the course design. Ask evaluators to compare the e-learning course to the previously held classroom version. Did learners find that the formative questions provided for thought provoking real-case scenarios? An essay response could appropriately reflect how the learner plans to use the new skill on the job.
Course Structure
An effective learning course structure is adult friendly. The course should flow together. Learners need to be able to navigate through the course with ease. The course should be broken down into modules that make sense and a simple interface design should be selected.
Practical evaluations ask the learner if course material was presented in a logical format. The interface should be intuitive, the menu uncluttered and easy to understand. Was the learner able to navigate through all of the buttons, find the directions and help page? Learners should be provided opportunities to respond on the course structure.
Globalization
Ethical considerations are often passed over on a level one evaluation. A large company size of Entertainment and Resorts has revolving supply of diverse employment. Seasonal work attracts a global audience and cultural sensitivity is of primary importance. The level one evaluation asks learners to offer their opinions about graphics, scenarios, local phrases and humor in the e-learning course that could be misinterpreted as culturally insensitive.
Evaluation Comparative
The following is a comparative level one evaluation sample commonly used by training departments. The actual evaluation and assessment spread sheets that were used in the performance review course are illustrated further down.
|
Training Evaluation & Feedback |
Course Title & Date | ||||
|
Rating Options: |
a lot |
some |
little |
none |
Specific Highlights and/or Suggested Improvements? |
| Enjoyment: Did I enjoy the course? |
|
|
|
|
|
| New knowledge and ideas: Did I learn what I needed to, and did I get some new ideas? |
|
|
|
|
|
| Applying the learning: Will I use the information and ideas? |
|
|
|
|
|
| Effect on results: Do I think that the ideas and information will improve my effectiveness and my results? |
|
|
|
|
|
| Any other comments? | |||||
|
© Alan Chapman. A free resource from www.businessballs.com. Not to be sold or published 2011 |
|||||
Focus Group
A small focus group of Human Resource employees participated in the pilot course. The group viewed the course from a learner’s perspective. After the computer based training, the subjects were administered a course assessment and evaluation. Six subjects participated in the assessment and five subjects participated in the evaluation.
Learner Demographics
|
Profile |
Range |
Average |
|
Gender |
Female Only |
Female |
|
Age |
30-50 |
38 |
|
Years of Service |
0-25 |
8 |
|
# of Direct Reports |
0-10 |
2 |
Assessment Summary
|
Results |
Samples |
Percent Correct |
Actual Score |
|
Subjects scored an average of 83%. Question 13 was not averaged into the score (Highlighted in yellow below). Inconstancies in answers 2, 3 and 5 prompted the authoring of additional slides to further clarify the difference between the hourly and salary performance review during the introduction module.
|
Total Possible: |
100% |
28 |
|
Subject 1 |
79% |
22 |
|
|
Subject 2 |
75% |
21 |
|
|
Subject 3 |
93% |
26 |
|
|
Subject 4 |
79% |
22 |
|
|
Subject 5 |
86% |
24 |
|
|
Subject 6 |
86% |
24 |
|
|
*Mean Score is 23 |
|||
| Assessment Test | |||
| Question: | Answer: (Correct answer is bold) | Recommendations: | |
| 1. Which steps are optional in the Hourly Performance Review? |
|
(69% correct) The information that I received on the performance logs was either incorrect or incomplete. Either way, the performance logs section should be re-evaluated by the Human Resource department staff for accuracy. Staff should make corrections and add additional information to the performance logs section. | |
|
|||
|
|||
|
|||
|
|||
| 2. What is the first step of the salary performance review cycle? |
|
(66% correct) Continue to monitor the assessment question; however, “Open a salary review” might be a good answer to toss. | |
|
|||
|
|||
|
|||
|
|||
| 3. A department manager has hourly employees, some are full time and others are part time. They are not part of the union, work regular hours and are expected to meet specific standards but that’s about it. Which of the following two digital forms should this manager start with? |
|
(66% correct) I added some additional slides clarifying the salary and hourly performance review tutorials. | |
|
|||
|
|||
|
|||
|
|||
| 4. Employees create a digital performance review on the Lotus Notes Database. |
|
(100% correct) Continue to monitor question for relevance. | |
|
|||
| 5. A supervisor has a part time employee. She works about 25 hours a week. Because her role in the department, they discuss how her work contributes to the overall team goals. What type of review should the supervisor author? |
|
(55% correct) I added some additional slides clarifying the salary and hourly performance review tutorials. | |
|
|||
|
|||
|
|||
|
|||
| 6. Which of the following five options are not available in the Overall Performance Rating? |
|
(66% correct) I noticed that the check box was not correctly aligned to the answer on this slide in the course. I corrected the answer and magnified it. | |
|
|||
|
|||
|
|||
|
|||
| 7. Managers create a new performance log every time they document their employee’s behavior. |
|
(100% correct) Continue to monitor question | |
|
|||
| 8. __________________ is a quick and easy method to refresh your memory on what training you may have accomplished that may impact yourself appraisal. |
|
(100% correct) Continue to monitor question, the question may be too easy. | |
|
|||
|
|||
|
|||
|
|||
| 9. Performance Logs appear in a different section in the Performance Review then the Self Appraisal. |
|
(83% correct) The performance logs section should be re-evaluated by the Human Resource department staff for accuracy. Staff should make corrections and add additional information to the performance logs section. | |
|
|||
| 10. Which of the following is NOT considered a best practice when rating a performance review? |
|
(84% correct) Continue to monitor question | |
|
|||
|
|||
|
|||
|
|||
| 11. Once inside the Lotus Notes Data base, which are some of the following steps do you need to take when opening a new performance review? |
|
(98% correct) This is the question that I designed to encourage the learner to reach back to the tutorial and double check.The purpose of this question was to encourage learners to access the tutorial before answering the question. | |
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
| 12. Managers do not have to wait until their employees have completed their self-appraisals in order to begin the performance review. |
|
(100% correct) Continue to monitor question | |
|
|||
| 13. After administering the performance review to your employees, which of the following steps must still be performed before submitting electronic documentation to HR? |
|
(92% correct) Toss this question, this question was not calculated in the final scores of the assessment* Results not calculated in final score. | |
|
|||
|
|||
|
|||
|
|||
| 14. For employees with Lotus Notes Accounts, an automatic e-mail is generated to the employee with a link to the performance review. Employees are asked to examine the review for errors and click on the ACKNOWLEDGE button in lieu of a signature. |
|
(100% correct) Continue to monitor question | |
|
|||
| 15. After acknowledging the performance review, the _____________ checks the status by opening the performance review database, clicks on the employee’s name and checks the status and dates of all documents. |
|
(100% correct) Continue to monitor question, the question may be too easy. | |
|
|||
|
|||
|
|||
|
|||
Evaluation Results
|
Question |
Profiled Response* |
Summary of Findings |
|||
|
*Profiled Response is the typical, consolidated or significant responses selected from the evaluation |
|||||
| 1. How much will the Performance Review Course improve your ability to perform your job more efficiently? |
Range |
Average |
This was on a scale of one to ten. The number 8 would indicate a strong improvement for Human Resource workers to perform more jobs effectively. | ||
|
5-9 |
8 |
||||
| 2. How will you apply the new skills, knowledge and attitude from the performance review course to your current position? | “An excellent resource for managers to refer back to, as questions arise throughout the year.” | Responses favorably indicated that the tutorial would be an excellent resource for supervisors to refer back to throughout the year. | |||
| 3. Select the following objectives do you feel need additional coverage after viewing the course. | “Maintaining Performance Logs” | The performance logs section will need to be revisited. The performance logs may have had some missing information that I didn’t receive when I created the tutorial. | |||
| 4. The Performance Review Course was designed to provide management the necessary information for effective decision making. (On a scale of 1-10, please evaluate whether the needs of the managers were met by the e-learning course.) |
Range |
Average |
Results indicate that the course provides HER supervisors the necessary information for effective decision making. | ||
|
6 -10 |
8 |
||||
| 5. How do you feel that the course could be improved? | “Change the voice over and add more on screen movement” | It is quite apparent that the voice makeover is the number one concern expressed by four of the five individuals. I will submit the final copy with a new voice over. Pop up and zoom boxes with additional key words and magnifiers will be added. | |||
| 6. What did you take from the course to align behaviors and decisions with the organizational brand, mission and values? | “The performance review system is now systematic and leaves little room for misinterpretation. It will get all of the departments utilizing the same review methods.” | I found a similar question to this when I was researching effective evaluation questions. The performance review is closely tied to the E & R Mission and Values. | |||
| 7. On a scale of 1-10, did you find that the formative questions provided for thought provoking real-case scenarios? |
Range |
Average |
A comment at the end of the survey expresses favorably for the scenarios. | ||
|
6 – 10 |
8 |
||||
| 8. Which of the following areas (if any) did you feel needed improvement with the course structure? | “Navigation/Buttons/ Interface design” | The initial plan for the module was to lock the subjects into each slide so they can’t skip to the next slide. I think that it will be too cumbersome not to allow the adult learner to have the freedom to navigate through the modules more fluidly. Learners will still unlock levels but I will add more navigational friendly buttons for the final copy. | |||
| 9. Which of the following menu features did you find helpful during the course? | “Main Menu” | The main menu buttons worked out well. Drop boxes had been considered but not selected due to the client’s wishes that learners watch entire module sections. | |||
| 10. Were there any graphics, scenarios, local phrases or humor in the e-learning course that could be misinterpreted as culturally insensitive? | “No need for in this particular evaluation form to “state your ethnicity”. | One response applauded the sensitivity to ask the question. Another stated that there was no need to ask employees to state ethnicity it in the survey.Interestingly, not one person volunteered their ethnicity in the demographics section. | |||
| 10. Name three things that you like the most about this course. | “The step-by-step process,” “Formative Assessments,” and “Easy to follow/use.” | This section indicated that the module was user friendly. | |||
| 11. Please place any additional comments in this section. | “Include a certificate of completion at the end.” | A window to a pdf certificate can be constructed quite easily and accessed through captivate at the end of the course. | |||
Evaluation Conclusion
Performance Logs
Performance Logs were mentioned as an area that could use further defining. Based on the response to question one in the assessment (five incorrect), five of the six employees believe that Performance Logs are optional. The tutorial indicates that performance logs are not optional. The employee with the least amount of time at E & R is also under this impression. Are performance logs optional? Should the tutorial be changed to reflect that the performance logs are optional, or continue the tutorial in order to encourage managers to maintain performance logs? Either way, a consistent policy should be established to keep managers and Human Resources from having misunderstandings.
Practice Assessment
One subject suggested providing learning links at the end of each assessment question. This would enable learners to refer back to the specific section of the tutorial in which the topic was addressed. Since E & R does not use a LMS, the assessment and evaluation were not authored inside Captivate, (due to the inability to capture results). A practice assessment may be beneficial if added into the tutorial. Practice questions should be further designed to encourage the learner to reach back into the tutorial and by providing a link as a reference. Not only would this enable the learner to access the tutorial to answer their own questions, it would get them comfortable practicing how to access the information for themselves, rather than leaning on the Human Resource Department. Final evaluation and assessment directions should indicate that learners double check their responses in the tutorial.
Course Adjustments
As a result of the feedback, I will make the following critical changes:
- Pop up boxes with key words will be added where magnification is needed
- In some of the slides, zoom features will be added
- Three additional slides will be added to clarify the difference between hourly and salary performance reviews
- The final course will feature a voice over
- Navigation buttons will be added back in
Further Recommendations
The Staff can continue to add or take away features from the performance review as they see fit. Here are some possible additional recommendations to improve the course:
- Add, clarify, and adjust additional instruction to the performance logs section
- Change out any weak or problematic graphics with fresh or original images
- Create a small window on the screen for key words and questions, preferably in the same spot on all the slides, something similar to a chalkboard
- Create a practice assessment with links to related material for each of the questions on Captivate at the end of the course
- Create a pdf window on the final slide for a printable certificate
- Continue to monitor assessment questions for validity
Project Summary
The Performance Review Project turned into a much richer learning experience then I originally anticipated. The practicum requirements were challenging, but cleverly designed to provide me with advanced instructional design skills as an end result. I was quite fortunate to get an opportunity to work with Entertainment and Resorts. The opportunity to practice the skills I learned in the LTMS program in a supervised workplace setting was really the experience that I desired to complete my studies at Harrisburg University.
The evaluation component was very rewarding. Even though I only had a small focus group, the feedback that I received gave me valuable insight. I am grateful for the opportunities that were presented to me and the mentors that went above and beyond their job descriptions to provide me with a valuable learning experience.
Works Cited
- Clark, Ruth Colvin and Lyons, Chopeta. Graphics for Learning. 2004. Print.
- “Factors to Consider.” VuDAT (Virtual University Design & Technology). Web. 18 Nov. 2010.http://vudat.msu.edu/design_factors/
- Fitzpatrick, Jody L., James R. Sanders, and Blaine R. Worthen. Program Evaluation: Alternative Approaches and Practical Guidelines. 2nd Edition. New York: Longman, 1997. Print.
- Graham Attwell (ed). Evaluating E-learning A Guide to the Evaluation of E-learning Evaluate Europe Handbook Series Volume 2. Bremen, Germany: Graham Attwell (ed.). http://www.pontydysgu.org/. Printer: Perspektiven-Offset-Druck, Bremen, Germany, 2006. Web. 24 July 2011. http://www.pontydysgu.org/wp-content/uploads/2007/11/eva_europe_vol2_prefinal.pdf
- Grote, Richard C. The Performance Appraisal Question and Answer Book: a Survival Guide for Managers. New York City: AMACON, 2002. Print.
- Henry, Judith W. “Center for the Learning and Leadership Development, Hershey Entertainment and Resorts.” Interview. 23 Feb. 2011.
- “Hershey, PA Who Is Hershey Entertainment & Resorts?” The Sweetest Place On Earth. Web. 03 Apr. 2011. http://www.pa.com/about_/mission_statement.php
- Holtsnider, Bill, and Jaffe, Brian D. IT Manager’s Handbook: Getting Your New Job Done. 2nd ed. San Francisco: Morgan Kaufmann, 2007. Print.
- “How to Evaluate E-Learning.” CorpU Weekly. Corporate University Xchange, 15 Apr. 2008. Web. 24 July 2011. http://www.corpu.com/weekly/article/how-evaluate-e-learning/
- Kirkpatrick, Donald L. Evaluating Training Programs: the Four Levels. 1st ed. San Francisco: Berrett-Koehler, 1994. Print.
- Kirkpatrick, Donald L. “Donald Kirkpatrick’s Learning Evaluation Theory – a Training and Learning Measurement, Evaluations and Assessments Model.” http://www.businessballs.com. Alan Chapman, 2009. Web. 24 July 2011. http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm
- Lorz, Terri Jo. Employee Performance Reviews. Program documentation. Division of Human Resources. The University of Utah. Web. 26 Mar. 2011. http:/www.hr.utah.edu/training/management-essentials/Employee-Performance-Reviews.pdf
- “Performance Appraisal Software.” YouTube – Broadcast Yourself. TrakStar
- Professional Software (promantek.com). Web. 26 Feb. 2011. http://www.youtube.com/watch?v=_oonUupNXBw
- Shackleton-Jones, Nick. “Top Ten E-learning Mistakes.” Aconventional Blog. 9 Mar.
- 2011. Web. 21 Mar. 2011. http://www.aconventional.com/2011/03/top-ten-e-learning-mistakes.html
- Strother, Judith B. “An Assessment of the Effectiveness of E-learning in Corporate Training Programs.” IRRODL Vol 3, No 1. The International Review and Research in Open and Distance Learning, 2002. Web. 24 July 2011. http://www.irrodl.org/index.php/irrodl/article/viewArticle/83
- United States. DOCHROC. Office of Workforce Development & Performance
- Management. Mid-Year Performance Review Training For Supervisors. By Kelly Spence, Sandy Conway, Ursula Snead, and Nancy Williams. Department of Comerce, Apr. 2010. Web. 26 Feb. 2011. http://hr.commerce.gov/s/groups/public/@doc/@cfoasa/@ohrm/documents/content/prod01_009100.pdf
- Van Der Knaap, Peter. Theory-based Evaluation and Learning: Possibilities and
- Challenges. Diss. Netherlands Court of Audits, The Hague, The Netherlands. (London, Thousand Oaks and New Delhi): SAGE Publications, 2004. Flemish Evaluation Platform. SAGE Publications (London, Thousand Oaks and New Delhi). Web. 25 July 2011. http://www.evaluatieplatform.be/doc/Van der Knaap – 2004 – TBE.pdf
- “WingSpan Video Demo – Employee Performance Management.” Talent Management
- Software – SilkRoad Technology HR Software. Web. 03 Apr. 2011. http://www.silkroad.com/WingSpan/controller.html
I read this piece of writing fully on the topic of the
comparison of latest and earlier technologies, it’s awesome article.