Assessments in the Hot Spot

Testing Interface Design
Ethical considerations with assessment accommodations should be further researched with individual variations as a valid and reliable alternative to mainstream testing.

Metric variables can affect the validity of the assessment phase in evaluation. Intrinsic factors reflect the individual. The amount of sleep, health, emotional state, learning preference and cultural diversity can generate varied results in testing.

An open accommodation e-test would be valid and reliable if all persons shared the same opportunities to access accommodations. Students would be encouraged to access accommodations as a test taking strategy and e-tests would record accessed accommodations. Students that relied heavily on accessing accommodations would have the opportunity to receive remediation in study skills if necessary. Many disabled students would be able to take their tests alongside their peers without being pulled out to have the tests read to them by special education teachers.

The following assessment strategy outlines a proposal to open accommodations to all students. A sample assessment prototype was constructed to further illustrate this purpose. (Screen shots from the sample prototype are incorporated throughout the article.)  Hot spots are incorporated in the example prototype.

The following screen shots illustrate the various lay-outs that the test-taker can select for using hot spots rather than multiple choice, by clicking directly on the most appropriate answer.  Hot Spots eliminate the need for alpha-linear answers that can be distracting to the test-taker.

Background

School districts can lose government funding or worse, administrative control due to poor test scores. Federal law mandates that schools provide opportunities for testing accommodations to students with disabilities. With so much at stake, school districts are spending a bulk of the school year teaching multiple choice test strategies to students in order to increase test scores.

As educators, we have an ethical obligation to teach students how to access their individual learning style rather than funnel all student learning into linear, text based, multiple choice thinking. Studies of mainstreamed assessments with testing accommodations for all populations should be conducted with the highest of priorities.

Methodology

An example exam has been constructed to demonstrate what open-accommodation standardized e-testing should be modeled after. Online assessments should be purposefully planned using proven learning models to reduce metric variables among learners. Learner perception is influenced by an intuitive interface. A web interface is selected by the learner before the test to optimize the performance of the test. Web interface selection and all optional accommodations selected by the learner are recorded to give educators more information about the learner. National standardized tests should be repurposed for their original intent; as tools to get a clearer picture of the learner’s profile.

Linear testing strategies are emphasized even though studies show that human brains are not linear in structure (U.S. Department of Education, 1997). Design considerations of the following e-test prototype provide the learner with an optional testing interface. The learner can select linear or non-linear test interface strategies. The linear test interface would be considered traditional in design; the test author selects the order of the sub-categories for the learner to test in.

Emotional well-being, test anxiety and stress can greatly factor metric variables. The e-test illustrates improvements for visual learners that prefer information organized and presented in a non-linear, central graphic format.

The tester can select subject subsections according to their individual preference. For the learner that prefers to tackle more familiar questions at start, this can help reduce stress.

A person who experiences test anxiety can have relaxing music played on a headset. A mental relaxation tutorial can be accessed before the test at the option of the user. With authoring technology, a person who is losing concentration can choose to take breaks with stimulating computer brain games between test sections.

The following e-test offers a variety of accommodation options for learners with visual or auditory processing difficulties. Test takers can listen for auditory responses to hot spots through a headset.

A dyslexic learner can choose a cursive font to help prevent mistakes due to letter reversal during testing.

Inclusion of English language learners on large scale assessments is a critical issue nationwide (Butler, F. A. and Stevens, R 1997). Examples, prompts, questions and directions can be accessed through a computer generated voice in English or in the student’s native language.

An accommodation that allows English language learners to select key test words in their native tongue can easily be constructed.

Such assessments can further be authored to track the amount of times that the English language learner relied on reverting to their language of origin in order to measure English confidence proficiency.

Test questions that specifically measure text reading ability should be offered in both traditional linear style and with context cues to measure the differences in reader comprehension.

In other words, how often does the reader rely on context cues in order to comprehend?

The e-test proposal incorporated content on three grade levels in three different subjects.

Reading content is illustrated in the elementary grade levels.

Math content was chosen to demonstrate e-testing in the middle school level.

Content from a psychology 101 course demonstrates e-testing at the adult education level.

Test questions in each subject and age group have been further broke down into a Bloom’s Taxonomy grid to demonstrate testing on each level. An example from each of the Bloom’s levels has been made into a question on the test in each subject. Each of the three categories incorporates hot spots on the first five levels of blooms.

The sixth level of blooms, the evaluation phase, addresses an opportunity for students to provide written responses to demonstrate higher cognitive skills.

Data Collection and Analysis

Instructional technologists will collaborate with educators and state employees to develop a more comprehensive prototype. Measurable objectives for standardized testing are to be identified from state educational departments. Potential eligible questions may be modified from previously authored tests. Quantitative data will be collected for analysis from the assessments at pilot schools across the nation. Qualitative responses will also be included in the pilot findings.

Program Costs
An analysis of program costs is requested. If a committee approves the test prototype for further research, a grant committee can be selected to analyze costs and request appropriate educational funding opportunities from the government and private institutions for the development of future studies on open-accommodation style, standardized e-testing.

Conclusions and Recommendations
The open-accommodation e-test proposal should be studied in an appropriate assessment research environment for further evaluation. Currently, K-12 assessment accommodations are only available for students with Individualized Education Plans. Students that are eligible for Individual Education Plans must demonstrate a significant discrepancy between mainstream students in order to qualify for accommodations.

Test accommodation strategies should be taught instead of funneling and re-wiring brains to be linear and standardized. Individuals should be directed towards personal learning style accommodations to tackle tests. The viewpoint that testing must be as identical as possible to be reliable and fair should be challenged, all individuals should have access to test accommodations. Currently, educators teach our children multiple choice strategies throughout the school year. Rather, our focus should teach students how to study based on their individual learning preferences and provide testing in their individualized style. If everyone has the same access to these accommodations, then it is still valid and reliable.

Further test developments could provide testers with individualized levels and alternate accommodation views on a “per question” basis. For example, if a tester finds difficulty comprehending a particular question, the tester can select an alternate view such as a non-linear mind-map visual or auditory reading of the question. By tracking the tester’s accommodation choices and frequency during testing, educators have an opportunity to pin-point specific learning strategies that may need to be addressed for remediation. Entire school districts that rely heavily on certain accommodations when testing can receive funding for retraining staff to provide more appropriate teaching strategies.

In conclusion, keep in mind that assessments in general should always be treated as a tool to provide a snapshot into how a learner’s mind assimilates and retains information. The future of test reliability should be considered for research at the evaluation phase as test analysts need more clearly defined metric interpretation methods.

HUST 8/11/10 by Angela Rupert, Graduate Student, LTMS 520 Assessment, Dr. Gerry Post, Instructor, Andy Petroski, Department Chair

References

Shaw, S. (2008). Essay Marking On-Screen: Implications for Assessment Validity. E-Learning, 5(3), 256-274. Retrieved July 10, 2010, from http://www.wwwords.co.uk/rss/abstract.asp?j=elea&aid=3382
Butler, F. A. and Stevens, R (1997). Accommodation Strategies for English Language Learners on Large-Scale Assessments: Student Characteristics and Other Considerations. CSE Technical Report 448. Retrieved July 10, 2010, from http://research.cse.ucla.edu/Reports/TECH448.pdf
Benson, P. J. (2001). The More Things Change. . .Paper Is Still With Us The Journal of Electronic Publishing vol. 7, no. 2, Dec., 2001. Retrieved July 10, 2010, from http://quod.lib.umich.edu/cgi/t/text/text- idx?c=jep;view=text;rgn=main;idno=3336451.0007.205
Kordel, R. (2008). Information Presentation for Effective E-Learning EDUCAUSE Quarterly, vol. 31, no. 4 (October–December 2008) Retrieved July 10, 2010, from http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolu m/InformationPresentationforEffe/163438
Gregg Corr and Ruth Ryder (RIM 2007) Presentation: Monitoring, TA and Enforcement U.S. Department of Education, Office of Special Education Programs 3.29.07 Retrieved July 10, 2010, from http://idea.ed.gov/explore/view/p/%2Croot%2Cdynamic%2CPresentation%2C24%2C
Building the Legacy: IDEA (2004) Idea Partnership. Web. Retrieved July 10, 2010, from http://www.ideapartnership.org/index.php?option=com_content&view=article&id=1344 &oseppage=1
Making Connections: How Children Learn. Read With Me – A Guide for Student Volunteers Starting Early Childhood Literacy Programs (1997) U.S. Department of Education. Web. Retrieved July 10, 2010, from http://www2.ed.gov/pubs/ReadWithMe/makconn.html

Advertisements
Posted in Assessment, Development, education, Human Learning, Learning Technologies, Media Design, Online Learning, standardized testing, Uncategorized, Visual Learning | Tagged , , , , , , , , , , , | Leave a comment

Needs Analysis Report Case Study

Angela Rupert

The Plagiarism Pledge

Instructional eLearning Design Case Study on Attitude Change


Section 1 – Need Analysis Report

Executive Summary: The Harrisburg University of Science and Technology is an academic institution of higher education offering academics in mathematics, science, and technology. The university requires copy write and plagiarism awareness training for entry level students at both the undergraduate and graduate level. With internet and technology rapidly changing, Harrisburg University seeks to familiarize students with the proper way to reference writing and media in essays, blogs, podcasts, wikis and other mediums.

Business Need: Harrisburg University seeks a learning solution for student understanding of the ethical implications of plagiarism in order to reduce academic instances of plagiarism.

Learning Opportunity: A learning module outlining the consequences of plagiarism as a career and reputation killer will teach students to take responsibility for their own work.

Expected Benefits: Students will take more care to cite sources when borrowing ideas inspired from other authors. HU professors will reduce the number of confrontations when verifying student sources for authenticity.

Audience Analysis: A typical student at HU is an undergraduate in their early twenties. A secondary audience is the graduate student, average age in their forties. Males and Females are nearly evenly split in both categories.  Older students may not be familiar with referencing media in blogs, podcasts, wikis and other mediums that were not available to them years ago. The upcoming generation is more technology savvy, having quickly caught on to the copy and paste features of word processing. On occasion, incoming freshman may have slipped through high school with teachers too busy to verify student sources.

Project Design: An asynchronous e-learning module will illustrate to incoming students what plagiarism is and why they need to be more conscientious writers. The video will be shown to incoming students during writing class and plagiarism training with the school librarian. An e-learning module will incorporate open-source media and digital imaging on a power point presentation.

Project Success Measures: The campus librarian will track the number of student plagiarism responses before and after the training.

Out of Scope: The e-learning module will be short in length and will not get into the specifics as to how to avoid copyright infringement and any rules about what constitutes fair use laws.

Estimated Project Costs: Project costs should not exceed the price of any e-learning module or software that will be needed to be purchased for the final portfolio.

Posted in Human Learning, Instructional Design, Learning Technologies, Online Learning, Uncategorized | Tagged | Leave a comment

LCMS selection for the small business

Professional Development, Inc. LMS Analysis

lms_capture

Section 1 – Needs Analysis

Executive Summary
Professional Development, Inc., a local professional communication and professional development business, seeks to launch their new subdivision of online learning.  Currently training takes place in the classroom.  The online training initiative will be a hybrid of asynchronous e-learning and synchronous webinars.  A research strategy will analyze current practices for transitioning classroom to online learning and to compare costs to ROI. An LMS or similar wrapper is needed to host the new course series.  A technology selection analysis will be used to assist the client in choosing an LMS.  In addition, further options for blended learning will be addressed. The project was framed in Google docs, forms and spreadsheets in order to get ballpark costs and organize research analysis outcomes.

Business Need
Professional Development, Inc. would like to convert their courses online so that they can expand their audience and increase the revenue in their business.

Expected Benefits
Asynchronous Learning:  Students may access the training as needed and may be more likely to select course offerings.  Students that miss will have the option to make up the class with asynchronous learning.  Courses will be accessible at any time of the day.
Synchronous Learning: Live webinars allow the business to have a wider audience, where students can practice with real examples of participants’ conversations simultaneously over the internet.  It is expected that the business will receive additional revenues.

Learning Opportunity
Asynchronous Learning: Students will be able to access Professional Development, Inc. video courses online.  A library of skills has already been recorded.  For example, if the learner chooses to review specific skills before critical business meetings, they may access the lessons at their convenience.
Synchronous Learning:  Live webinars can attract a wider audience not limited to the greater central Pennsylvania area. In addition, webinars can be recorded and placed into an LMS so that students can review what they learned at any time.

As live webinar presentations become more mainstreamed in business, participants will have more options to apply their new skills in a variety of platforms.  Instructors and peers can assist participants as they evaluate and provide feedback of how their communication styles come across media technology.  Students will appreciate the values and interests of their audience from the client’s perspective.  Participants will practice using the latest in webinar communication etiquette such as polling, raising hands, chatting, forums and emails.  Professional Development, Inc. instructors will expand these activities while they build upon these exercises with style, content, and delivery of their verbal presentations.

A blended approach allows webinars to take place during regular courses. A producer runs the webinar and assists the online audience during the live course.
The online expansion will allow students to further examine communication habits. Learners will break down and strengthen crucial communication elements in a broader setting.  Students learn to adjust their personal style of communication by participating in a series of like courses through webinars, classroom and asynchronous learning, skills will be reinforced.

Online Learner Profile
Most of the learners will be in the sales or business management fields.  Age ranges are anticipated to be consistent with the work force with a wide range of technology skills.

Section 2 – Project Outline
Project Goals

  • Select a Learning Management System or appropriate wrapper to host courses
  • Discuss options for blended learning platforms

Project Objectives

  • Analyze technology selection needs vs. wants
  • Assist client in choosing an appropriate LMS based on his unique business needs
  • Compare costs to ROI
  • Discuss further options for blended learning

Project Conflicts

  • Time and expenses related to the purchase of a functioning LMS and web conferencing host will need to be sorted out prior to the project.
  • The instructional designer will provide some guidance to the selection of these platforms but ultimately, the client must make the final purchasing decision.
  • The instructional designer will work on this project on a part time basis. (A Gantt chart will be created after technology selection).

Out of Scope
The following items are not considered to be within the scope of the project:

  • Providing, designing or coding a website
  • Designing and formatting a platform or LMS for more than 20 unpaid hours total
  • Coding an LMS
  • Analyzing, designing, developing or teaching a course
  • Assisting with the production of live webinars for more than 6 unpaid hours
  • Maintaining files, updates, media or courses after the end of the project
  • Additional training or evaluation after project completion

Clients Responsibilities
The client is ultimately responsible to provide an LMS or similar platform for the instructional designer to work on at his expense.  The client shall provide a web conferencing host for the webinar courses at his expense.  All additional maintenance and expenses related to platforms and upkeep are the responsibility of the client.

Project Success Measures

  • An LMS is selected and purchased
  • A webinar is discussed and evaluated

Project Milestones

  • Analysis of LMS needs vs. wants
  • A total of six LMS will be considered and evaluated
  • Meeting with Harrisburg University and Avalon Foster
  • A consultation of the preliminary results
  • Schedule vendor demonstrations
  • Establish Budget
  • Design a Return for Proposal (RFP) for LMS Bids
  • Selection of LMS

Timeline and Resources
A Gantt chart will be created after the selection of an LMS

Project Design Summary
An embedded questionnaire was sent to vendors selected to contact for an RFI.  30 vendors responded.

Research Methods:

  • Questionnaire
  • Phone calls/ e-mails
  • Google Docs
  • Estimated Project Costs
  • Establish Budget
  • ROI

Section 3 – Business Profile

Business Goal
Professional Development, Inc. seeks to improve communication skills in individuals and organizations by receiving the benefit of an ideal development environment – fun, interactive sessions that focus on the most powerful tools and techniques.

Business Objectives

Business Philosophy and Values

Business Mission
Improve communication skills by experimenting in a safe and risk-free environment.

Vision
Professional Development, Inc. business is founded on a “Classroom” concept.  “Workouts” are designed for ‘delivering confidence through competence’ by drill and practice. Members are able to try new methods, styles and techniques where feedback can be provided.  Online learning provides greater flexibility and increased potential for revenue.

Projected Growth
By expanding into online learning, students will be able to access and review courses through the internet.  Webinars provide additional opportunities to expand the audience beyond the parameters of the Harrisburg Metropolitan area.

Research Methods:

  • Interview
  • e-mails
  • Phone calls
  • CG Web Site
  • YouTube Videos
  • Attend a CG Course

Section 4 – Research

Research Goals
Research effective online learning models that take into account the subject matter, online learning requirements, course materials, learning objectives

Research Objectives
By September 2012, the client will be able to:

Research Methods

Initial LMS e-mail price and feature inquiries  (See Appendix B for list of LMS recipients and responses)
Survey Request for Information (RFI) (See Appendix C)
A Request for Information was created after the initial e-mail inquiries in the form of an embeddable survey.  This was sent to over 100 LMS vendors.  The survey had 32 responses and several vendors responded to the e-mail but declined to participate in the survey for various reasons.  Based on the e-mail responses, it appeared that some of the LMS vendors preferred to receive a more formal Request for Proposal (RFP) from their potential customers to bid on.

The survey was helpful in that it confirmed the initial selection as to be in alignment with the needs and wants of the client.  Price breakdowns were varied per business model and the survey uncovered some creative ways for LMS companies to charge customers for additional services.  The survey allowed scoring of specific features to be more accurate and made the phone calls to the sales representatives go more smoothly.

LMS Websites
Product websites were researched and reviewed for several LMS companies including those on the final list.

LMS Free Trials
Some of the LMS companies that were reviewed offered free trials.  The free trials were used as a sandbox to experiment with navigational features.

Phone calls to LMS representatives
Most of the companies were contacted and interviewed by telephone in the final round.  Gaps of information were covered and price structures were verified.

Virtual LMS Demonstrations
Only one LMS was selected for a live virtual demonstration.  efront was chosen because of the high score that it received and its remote location in Grease.  A second demonstration is available upon request.  Vendors from the final LMS list are eager to demonstrate their products at the client’s convenience!

Research Conclusion
A LMS Decision Selection Analysis was created to rate the features and costs of the following selected LMS systems.  The rating criteria were based on the needs and wants of the client.  The following scores are based on the combined costs and features and ranked accordingly.

LMS Short List(See Appendix A  for extended results)

LMS Final List:

Approx Price

Name

Score

Notes

$1000/mo Fugu (Comparative) Total: 3246Rank: 6 Difficult to calculate (See Notes)
$300/mo WestNet MLP Total: 3946Rank: 4 Free storage for first 50 courses
$320/mo Efront Total: 4457Rank: 1 3 month minimum
$399/mo iSpring Online Total: 3261Rank: n/a Disqualified due to limited registration and shopping cart capabilities
$475/mo Accord Interzoic Media Total: 4174Rank: 3 Storage Fees extra Each 5gb storage is $20
$495/mo  iLMS Total: 4190Rank:2 up to 100 courses only
$300-$500/mo TeraLearn LCMS 3796Rank: 5 Additional Storage Fees, negotiable

It is recommended to schedule demonstrations for each LMS moving forward.  A Return for Proposal should be created to start the bidding process.  The formal RFP would allow the client to state their maximum budget. The client should plan for a $300 – $500/mo budget with initial bidding around or under $300/month.  If the $300/month bid does not produce favorable results, a new bid can be submitted.  The instructional technologist would like to create the RFP for the client as the next step in the process and schedule the first round of LMS demonstrations.

Section 5 – Development and Implementation

Development Plan
How many hours from the development team will be put in the project?
TBD

Implementation Resources

  • Discussion with Harrisburg University about using Adobe Connect as a platform for webinars.
  • Avalon Foster may be available to assist with marketing

Available Technology

  • You Tube videos
  • CG Video Camcorder

Comparative LMS
Fugu (Gracie University) was the comparable LMS selected due to its slick interface design, usability and client likability.  Fugu was contacted about its services as a potential LMS.  Unfortunately the business model would not fit well with Professional Development, Inc. and the price would be too costly to make it worth it.  Fugu wanted to retain its branding and receive 35 percent of all revenues.  Fugu would be unable to serve blended learning management for the classroom side of Professional Development, Inc..

LMS Interface Design Considerations:
Considerations should be made to keep the interface more intuitive. Currently in the classroom, Professional Development, Inc. curriculum takes participants through four levels of competency. New skills are tested as students learn the fundamentals of each level. As participants progress through the various levels and display competence, they receive insignia pins to indicate their new level. (Insignia pins are similar to that in martial arts and coded sequentially in white, yellow, blue, silver, and black.) This process allows participants to move at their pace. The interface design should correspond to the classroom competency levels so that participants that utilize both classroom and online learning can more easily navigate through the interface.

Uniformity:
Uniformity should be a consideration to align the new online learning with the classroom and business model.

Individual Services:
Business coaching and programs are set to 90 minute classroom workouts.  Current classroom membership allows you to attend up to two communication “workouts” per week.  Online courses should be designed to correspond with classroom courses for uniformity. An interface should allow users to access free and member based videos; these can be the video shorts that were already created on ‘You Tube.’  Members can access two live or recorded webinars per week.  Additional courses should be extra.

Corporate Services:
Each Communication Competency Module (CCM) is 8 hours of coached sessions scheduled to meet the needs of individual companies.  Professional Development, Inc. can up sell corporate packages.  Companies can have their sessions recorded by webinar and presented as a live blended option accessible later as an asynchronous learning CD or hosted on the LMS for a specific time period. Sessions can be produced on or off site. Currently the optimum classroom format is four two-hour sessions spaced one or two weeks apart. Five basic CCM’s are available and custom courses can be designed for an additional fee.

Limitations:
The ideal LMS will be one that Professional Development, Inc. will be able to easily maintain without hiring a developer.  Videos should be easy to upload and maintain.  The LMS should have a shopping cart feature, utilize assessments and generate scores.  Maintenance and bandwidth will need to be analyzed further.  Plug-ins such as captivate should be compatible with the LMS.  Webinars that can automatically be loaded would be a bonus.  An LMS should be considered for a small business budget.

Posted in Development, education, Instructional Design, Learning Management System, Learning Technologies, Low Cost Technology Solutions!, Online Learning | Tagged , , , , , , , , , , , , , , , , , , , | Leave a comment

Instructional Design Phase Estimate

design phase example

design phase example 2

http://www.slideshare.net/bchapman_utah/how-long-does-it-take-to-create-learning?from=ss_embed

Posted in Estimation, Instructional Design | Leave a comment

Instructional Technologist Framework

Instructional Technologist Framework

Executive Summary:
The Widget Factory seeks an online training solution for their Human Resource Department. Currently, training takes place in the classroom. The Widget Factory would like to put their classroom training online so that managers make access the training as needed.

Adobe e-learning Suite 2 is the tool selected for the project.  Assessments at the end of each chapter will serve the purpose of reinforcing the material and scores will be generated automatically through Captivate 5.

Project Goal:
Launch a custom online training module that will effectively model subject matter, online learning requirements, course materials and learning objectives.

Project Objectives:
By spring 2011, an instructional technologist will
Identify course subject, instructor and materials
Identify online learning requirements
Identify course objectives
Design flow chart
Design storyboard
Develop course prototype
Implement course prototype
Evaluate prototype

Research Goals:
Find an examples of similar training
Consider “Best practices for transitioning to online learning”
Create a research Gantt chart
List research resources

Table of Contents:

Project Identification
o Identify project goal
o Define project objectives
o Identify project sponsorship
o Identify stakeholders impacted by project
o Analyze development resources
o Work Breakdown Structure
o Utilize a project management tool
o Incorporate Gantt charts and time lines

Course Identification:
o Identify course subject
o Identify stakeholders impacted by project
o Identify online learning requirements
o Identify course materials
o Identify course learning objectives

Project Analysis
o Analyze resources
o Identify constraints or project limitations
o Analyze subject matter
o Research existing solutions
o Identify time constraints
o Define Project Schedule
o Identify costs associated with the project
o Identify internal and external resources required
o Summarize conclusions related to objectives

Project Design
o Obtain an instance of Adobe e-learning module
o Design flow chart
o Break down learning activities
o Create learning objectives for each of the activities
o Incorporate technology into the assignments
o Design interactive learning lessons
o Design course storyboard

Project Development
o Develop course prototype
o Media selection
o Organize media assets
o Edit media using appropriate tools
o Develop assessments
o Develop instructors guide (paperless)
o Prepare assets identified in storyboard

Project Implementation
o Identify launch options
o Stage a “Kick-Off Meeting”
o Implement course prototype
o Document progress
o Use graphic representations for task analysis
o Identify project milestones
o Provide updates to the management team
o Facilitate focus groups

Project Evaluation
o Obtain feedback
o Catalog Feedback
o Revise learning solution
o List additional accommodations
o Summarize progress
o Identify follow up needs
o Write a closeout report

Posted in Human Resources, Instructional Design, Learning Technologies, Uncategorized | Tagged | Leave a comment

“Here we go again. . .” Corporate Annual Performance Review Training: Transitioning to Online Learning with Captivate 5

“Here we go again. . . ”

Do you ever find yourself saying that as you roll up your sleeves and get ready for the barrage of phone calls from executives and managers that can’t seem to remember how they accessed their online Performance Review forms?  Worse yet are the managers that don’t seem to remember how to fill them out once they access the form?

Every year, there are companies that lose valuable productivity because of ineffective annual performance appraisal training.  Managers have 12 months between reviews and because they don’t re-enforce the skills learned on a weekly or sometimes even monthly basis, they often forget.

Human resource trainers frequently have to provide a refresher’s course or spend valuable one on one time with corporate executives, walking them personally through the process once again.

Some companies have turned to computer based training methods as a convenient solution.  Online performance review forms and training can be purchased in neatly packaged “off the shelf” or cloud based performance review systems.  These products are quite convenient but can be very costly for large organizations.

One such corporation had already invested in developing their own “in-house” performance review system.  They just needed a more effective way to provide support for their management team.

Entertainment and Resorts (A fictitious name has been provided for corporate privacy), consulted with the learning technologies department at Harrisburg University of Science and Technology in Harrisburg, PA to find such a solution.  I was the graduate student selected for the task to transition corporate classes into effective e-learning modules.

The project was the capstone for my graduation requirements.  The last 6 credits of my degree were to be fulfilled with an experiential practicum.  Although the internship itself was 80 paid hours, the process spanned over 6 months due to the rigorous graduation requirements of the practicum itself.  I was blessed to have a department head (Andy Petroski), that truly believes in a high quality education and makes his students step up to the plate before issuing us our Masters of Science in Learning Technologies.

Executive Summary

A Brand Name “Entertainment and Resorts” Corporation seeks an online training solution for their performance review.  Currently training takes place in the classroom.  “E & R” would like to convert the performance review course online so that managers may access the training as needed.  The training will be a hybrid of performance review best practices and data entry procedures for the in-house IBM Lotus Notes database software.

Project Goal

The goal of the project is to:

  • Successfully launch a custom performance review e-training module by summer 2011
  • Utilize visuals of the Lotus Notes database to explain how the software tools support the efforts of the performance review process
  • Produce an effective learning module that will integrate performance appraisal standards with online learning requirements, subject matter and learning objectives

Project Objectives

By spring 2011, an instructional technologist will

  • Identify course subject, instructor and materials
  • Identify online learning requirements
  • Identify course objectives
  • Design flow chart
  • Design storyboard
  • Implement course prototype
  • Evaluate prototype

Project Milestones

  • Research
  • Flowchart
  • Storyboard
  • Design document
  • Create media
  • Develop
  • Evaluation

Project Conflicts

The project must fit into Lotus Notes software which is not a Learning Management System.  The course must be built remotely which may have a different Lotus Notes interface then some employees.  The help desk reported that there are several different Lotus Notes interfaces and that working remotely may not really make a difference anyhow.

Course selected

Performance Review was the course selected for the project.  Currently, the performance review training takes place in the classroom in the late third quarter.  The new training will allow the large quantities of employees to review content quickly and efficiently.

Objectives

  • Provide information about the Employee Performance Review database and available tools for creating a meaningful and effective environment for supporting, tracking and evaluating employee performance on an ongoing basis
  • Review the performance review process impact as it relates to employees, managers and the corporate model as a whole
  • Demonstrate the specific steps to an effective performance management process such as Goals, Self-Appraisal, Performance Logs, Review, Ratings and Submitting

Timeline

The finished tutorial will be approximately 45 to 60 minutes in length.  Opportunities for chapter review are provide at the end of each chapter.

Course Materials

E & R strives to be as paperless as possible.  However, hourly employees that do not have access to Lotus accounts will need to have a paper version printed out by their supervisors to fill out.  The tutorial will address the steps to this procedure.

Profiles

Entertainment & Resorts (E & R, web.2011) is a privately held company founded in 1920’s when the founder separated his manufacturing operations from his other businesses. Entertainment & Resorts is a hospitality company that employs about 1,500 full-time employees and 7,500 seasonal and part-time employees.

E & R Mission (E & R, web.2011)

Be a leader in the hospitality and entertainment industries by delivering excellence to our guests and employees, while enhancing our relationships with the company entities and the community, and sustaining financial integrity.

Online Learner Profile

Most of the learners will be in the supervisor or management category.  Most hourly employees do not have Lotus Notes accounts and will have a written performance review.  The learning module will address both salary and hourly training.

Technical Environment

E & R does not currently have a Learning Management System.  Lotus Notes hosts an in-house performance database software system managing employee performance reviews. The training department is able to track assessment scores through captivate and Lotus notes.  However, Human Resources is not concerned about keeping score from the performance review assessments   The e-training module will display video screen capture while demonstrating preferred input methods of performance review data.  Each section will provide best practice tips to maximize the training.  At the end of each chapter, a short quiz will be used to reinforce skills.

508 Requirements

The graphics, text and voice-overs are designed to provide the necessary information to successfully navigate through the Performance Review database for both Salaried and Hourly employees.  Computers must have speakers, the speakers must be turned on and the volume turned up.  A scripted version will be created for employees with hearing disabilities.  Employees and Managers may refer back to the tutorial at any time for a refresher.

Company Values

These are the values that are important to the strengthening of the E & R company and community.  Employees may create up to three goals for each value, but are cautioned to keep goals in check, as they will be held accountable for them if they are not met.  Currently, a help icon may be clicked for an example of each goal.  Professional Development was a required value this year and a minimum personal goal in this category must be met.

(Entertainment and Resorts Corporation, web.2011)

Integrity

    Acting in a manner that reflects dedication and integrity
  • Communicates the company’s vision, history, values, and strategy with conviction.
  • Makes efforts to serve, support and collaborate with community partners.
  • Demonstrates understanding of how job role and responsibilities support integrity.
  • Aligns behaviors and decisions with the organizational brand, mission and values.
  • Presents favorable image of the organization and destination to others
Team Focused

    Supporting one another as we work toward common goals and earning each other’s trust.
  • Works collaboratively with other teams across the organization to achieve business objectives and solve business problems.
  • Effectively communicates with all members of the group or team to achieve team goals and objectives.
  • Draws upon team members’ strengths to identify problems and develop solutions, ensuring that all voices are heard in the decision-making process, when appropriate.
  • Attends activities and/or interacts with members of diverse groups to increase cultural competence, which facilitates effective team performance.
  • Shows respect for the roles and contributions of other team members.
Service:

    Serving our employees and their families, our guests, and community and environment
  • Own: Demonstrates ownership and enthusiasm for serving others.
  • Anticipate: Takes initiative to share knowledge, experience, resources and information to meet the needs of others.
  • Delight: Demonstrates behavior that creates a welcoming environment while reinforcing the value of inclusion and respect for all employees and guests without regard to personal reward.
  • Inspire: Performs role and responsibilities in a manner that serves as a source of inspiration to others.
Respectful of Others:

    Treating all people with dignity, while respecting their differences and ideas
  • Demonstrates a willingness to accept the experiences, backgrounds, ideas and perspectives of others.
  • Shows support for fostering a workplace of civility; one that is free from aggressive or discriminatory behaviors towards others.
  • Treats people the same regardless of race, religion, gender, age, country of origin or other characteristic while avoiding the use of stereotyping.
  • Shows consideration for and is sensitive to the difference between intent and impact of decisions and comments on others.
  • Maintains composure, behaves appropriately and seeks win-win outcomes when faced with adversity with guests or co-workers.
 Employee Experience

    (Management Only)
  • Effectively introduces and assimilates new employees to organizational culture and team.
  • Able to identify knowledge, skills and abilities necessary to fulfill employee’s current or future job/role responsibilities effectively and provides resources, coaching and training opportunities as appropriate.
  • Holds employees accountable for behaviors/results by providing regular feedback in a manner that best develops the individual.
  • Delegates responsibilities appropriately and effectively.
  • Encourages ideas and suggestions from employees.
Business Strategy & Growth

    (Management Only)
  • Translates company strategies into meaningful goals and priorities for self and/or department.
  • Takes into account long-term vision under present-day pressures.
  • Demonstrates ability to be visionary and future oriented.
  • Continuously learns and demonstrates an understanding of the competitive environment, trends in the industry and technology that may impact the business.
  • Encourages and values creative thinking.
Quality Standards & Practices

    (Management Only)
  • Demonstrates commitment to establishing and delivering quality standards / processes with consistency and excellence.
  • Able to track trends and identify needs to improve processes, standards, systems or services that impact internal / external customers.
  • Achieves assigned projects / tasks according to established targets and deadlines.
  • Processes transactions in accordance with daily operations including but not limited to: employment / hiring practices, wage / salary actions, performance reviews, and expense accounts.
  • Firmly adheres to codes of conduct and ethical principles by honoring confidentiality and complying with legal, and safety requirements.
Financial

    (Management Only)
  • Monitors costs and revenues in line with budget.
  • Budgets according to business demands and organizational goals.
  • Adheres to revenue and accounting procedures to ensure all financial transactions and reporting requirements are handled in an accurate and timely manner.
  • Meets specific financial goals as defined by job expectations and departmental objectives.
  • Able to plan and allocate resources to ensure financial stability of the business unit, department or organization.
Guest Experience

    (Service and Management Positions)
  • Communicates clearly with internal / external customers.
  • Explores ways to enhance the internal / external customer experience.
  • Adjusts the application of policies and procedures appropriately to accommodate the internal / external customer needs within the best interests of customer and organization.
  • Gives full attention to every customer service interaction and responds to internal / external customer requests and issues in a timely manner.
  • Seeks to anticipate and understand internal and external customer needs / expectations with attention to cultural-specific values (communication, gender, space, etc.).

Performance Review Research

The research component of the project explores current trends in performance review and best practices for transitioning classroom based training into effective e-learning.  A Gantt chart was penciled out to establish a tentative framework to realize project scope.

Research Goals

Research questions were established in order to:

  • Find an example of Performance Review training software
  • Consider “Best practices for transitioning to online learning”
  • Explore current trends in Online Performance Review training

Research Methods

Research methods included questionnaires, interviews, phone calls, e-mails, internet searches, social networking and forums.   Comparable classes were challenging to find as the course uses common key words in searches and often lead to advertisements and unrelated material.

Authoring Tool

Captivate 5 is the tool selected for the project as E & R has recently purchased several licenses for their training department.  E & R is not concerned about keeping score from the assessments.  Assessments at the end of the chapter will serve the purpose of reinforcing the material.

Current Industry Practices

Each research component provides valuable guidelines and considerations for the performance review project.  Video screen capture and software simulations can be authored easily using the adobe captivate authoring tool and can bring a visual representation to the learner.   Unbiased qualitative performance review feedback is essential to an effective review and was examined for ideas on best practice approaches throughout the module.

Slide Conversion

A power point to online performance training module was researched for comparison.  The University of Utah, Division of Human Resources (Utah, web. 2011) provided an interesting example of a simple employee performance review e-learning module.  The module focuses on knowledge and attitude, showcasing the purpose of the employee review process.  The form to fill the module is a PDF printable rather than an interactive in-house software e-form.  Directions are simply a link to the form.

The University of Utah Online (Utah, web. 2011) employee performance appraisal reveals a string of linear slides repurposed for the web with limited interactive features.  Role-play illustrations and simple assessments are assigned at chapter ends.  Class participants are instructed to print and fax answers to human resource personnel for proof of course.  Links are cleverly highlighted for additional expansion in gray content areas.  A list of resources is provided for the learner to further expand their understanding at the end of the module.

Video Screen Capture

Research has demonstrated that the concept of using visual representation along with text assists learners with high visual spatial aptitudes (Clark and Lyons, 2004).  The current trend toward learning occurs when visuals are relevant, simple, clean and not distracting.

A third party performance review demonstration (promantek.com, 2011) utilizes video screen capture by panning in and out as an arrow clicks through the software.  The screen darkens momentarily while points of interest such as a button or a check box are highlighted.  A similar demo (silkroad.com, 2011) manipulates an arrow around the screen while screen shot stills transition in and out of the frame. The designer cleverly creates a video capture effect and reduces unnecessary movement in the process, giving it a cleaner, more professional appearance.  Overall video screen capture and possibly total file size may actually be reduced.  Areas of interest are colorfully shaded and eye-catching photography stills are incorporated throughout.

Storyline

An adult learner pedagogy model (MSU, web. 2011) suggests that learner relevant course content and lesson times be clearly identified.  Course content will include a relevant storyline to interest the learner.  Storylines show the learner how a person might apply a performance review in a variety of scenarios.

Every lesson activity will give the learner an approximate time to complete.  Learners that complete the task before the time expiration will not be penalized.  However, the nature of the Cisco Server may time out if left idling.  Learners should set aside an uninterrupted hour to complete the course.

Finally, the storyline component focuses on a more personal feel.  A good storyline uses realistic scenarios to get the learner thinking about how they may draw upon skills they have already used in real life settings.  For example, a learner may understand the term software tool to a word document or typewriter.  Another learner may understand that performance logs are also used as documentation for writing positive and negative comment about employee performance and habits, similar to keeping a diary or a journal.

Rating Bias

E & R takes a cutting edge approach to employee evaluations.  Management does not assign quantitative numbers in their performance reviews.  The Performance Appraisal Question and Answer book (Grote, Richard 2002) supports this philosophy.  Dick Grote argues that the best form of evaluation is not numerical, but rather a descriptive summary of specific behaviors and how employee behaviors impact the organization.  Employees may feel that their evaluation has more personalization with a descriptive summary, rather than the traditional numerical scores.  Therefore, the supervisors that provide the evaluation must put more thought in the appraisal and word it in such a way that demonstrates objectivity.

The IT manager’s handbook (Holtsnider, Bill, and Jaffe, Brian 2007) stresses the importance of keeping performance reviews objective rather than subjective in order to avoid wrongful termination lawsuits.  The key areas of evaluation with most employees are typically; work quality, flexibility, creativity, communication, timeliness, accountability and interpersonal skills.  Workers that go above and beyond the requirements of the job and access self-training are more likely to get better reviews.

Performance Review Feedback

Effective feedback is both objective and specific (DOCHROC, 2010).  Performance feedback that is expressed positively and accurately is more relevant to employees.  Each evaluated behavior should be measurable as it relates to the employees.  After performance reviews have been noted in the system, the review should be given to the employee in a personal conference.  Employees should be prepared for this meeting by listing self-accomplishments and future goals for performance improvement.

Negative Reviews

Negative reviews can be especially touchy and even unpredictable.  It is best to inform human resources if the review is going to be negative and maintain professionalism throughout (DOCHROC, 2010).  Performance behaviors, rather than personality traits, should be addressed with specific details, such as the date and time, to maintain clarity.   Sensitivity to privacy and tone of voice can make the process go more smoothly.

Implementation

Nick Shackleton Jones from the Aconventional.com blog recommends considerations with implementation strategies for course completion during the design phase.  An excellent e-learning module is not going to be very proficient if only five percent of the users are actually completing the course.  For this reason, the course design will propose that learners walk through each of the subcategories before allowing them to go through sub-menus for specific review.

Research Conclusion

In conclusion, corporations should set realistic standards for employees to be measured against.  Opportunities for re-training should be provided whenever possible.  Performance review is an opportunity to remind employees how their role plays a part in the workplace operations and how their behaviors can shape corporate success.

The flowchart (Appendixes A-C) was the most challenging in this particular project as it merged skills related content with knowledge and attitude material for the performance review process.  Previously, the courses were taught separately or back to back in the classroom.   The final product is still to be determined but will likely include pop up boxes with best practice tips during skills training.

Flow Chart (Attachment A)

The most valuable lessons that were realized in the research process were the question and answer forum boards on Linked-In.  I was able to pose a question about research tips when I felt as though I had exhausted resource leads.  The Linked-In forum allows the question to be posed to your friends list by personal invite.  I received a significant response and was really grateful for by my colleague’s suggestions.

Course Design

The course design was created in three sections.  Flow charts were used to get an understanding of how the course would flow.  A design document further expanded the flow charts for content clarification.  Finally, the storyboards were created to provide the developer the necessary details for executing the design.

Flow Chart (Attachment B)

Scope of Project

Performance review classroom training will be converted into an online learning course.  The project will include repurposed content from Power Points and classroom materials.  The course will not include Lotus Notes training and does not teach learners management and documentation skills.  The training will fall under the assumption that the learner has some familiarity with the Lotus Notes Database.  It is also assumed that learners have previous experience effectively documenting employees and creating performance reviews through the classroom and other mediums.  Best practice tips, however, are provided throughout the course to expand on these skills.

Course Goal

The goal of the course is to answer any questions learners have about how performance reviews are done at E & R.  Visuals of the tools are used to cover specific steps and explain how they are there to support team members in implementing the annual performance management process.

The primary audience of the course will be corporate managers and directors.  A variety of age, sex and races will be represented.  The delivery strategy will be interactive e-learning.  The e-learning course is designed for managers to take the course at their convenience through the Lotus Notes Database.  Learners must already be fluent with navigating through the database before taking the course.  Learners may refer back to specific modules as needed.  The  Entertainment and Resorts Performance Review Tutorial will be mainstreamed for management use by October 2011.

Course Objectives

By the end of the course, team members will be able to:

  • Explain the steps to effectively create a performance review
  • Differentiate between salary and hourly performance reviews
  • Design a goals document
  • Maintain employee performance logs
  • Create a self appraisal
  • Assemble a performance review
  • Properly submit documentation through Lotus Notes

Resources are outlined for learners during the course.  A help page has been created and frequently asked questions are available on the Lotus Notes Database.

E-learning modules:

The course was broke down into five modules.  An introductory module explained to learners the performance review course navigation.  The three subsequent modules identified the steps to provide necessary attachments for the performance review.  The fifth module was the performance review itself.  Learners are walked through the automated process of assembling attachments into the performance review and properly submitting the review to the Human Resource Department.

Module 1: Introduction

Learners will explain the steps to effectively create a performance review.  An e-learning module will be delivered with the definition of the performance review process as it relates to productivity and examples will be shown. This e-learning module distinguishes between salary and hourly performance reviews.

Module 2: Goals Document

The goals document is the first of three attachments that must be created before authoring the performance review.  Learners will identify the steps to design a goals document for their employees and themselves.  The goal document starts the process.

There are three sections in the performance review documents.  Supporting Our Values defines the four core organizational values.   Supporting Our Mission defines five key areas by which E & R measures their business.   Professional Development allows learners to define specific, professional growth goals.  At least one goal must be in the Professional Development section this year.  An effective goal will be in line with  Entertainment and Resorts values and mission.

Module 3: Performance Logs

Performance logs are used to document, track, make notes and add comments throughout the year on employee performance.  Both hourly and salary performance reviews will incorporate this important step.

Managers and employees create one performance log for each relevant category. All of the categrories do not need to be used. Throughout the year, managers and employees may edit any of the nine possible logs categories.

Module 4: Self Appraisal

The self-appraisal is optional but provides managers and employees valuable information for rich dialogue about performance.   Self Appraisal is virtually a clone of the performance review tool.  Employees complete the self-appraisal by clicking the appropriate rating for each behavior. Items cannot be skipped.  Performance log comments are automatically loaded under the comments section.  Additional comments can be entered by the employee.

Module 5: Performance Review

This module takes the learner through the automated steps of incorporating the previous attachments into the performance review.  Instructions for the rating scale are defined.  Self appraisals and performance logs are automatically detected by the Lotus Notes Database.  Any performance log comments employees add to their self evaluation are included.   Steps are provided for learners to seamlessly incorporate attachment documentation throughout the review. Learners will identify the differ features for the performance review and links to supporting documentation throughout.  Best practice tips are provided for the learner to administer an effective interview.

Outline

Explain the steps to effectively create a performance review

  • Access the directions for the course
  • List course objectives
  • Access course modules
  • List the functions of the main menu
  • Identify the sequence of the course modules
  • Explain how the performance review process increases productivity in business
  • Access tool features to assemble performance tools more efficiently
  • Access a hard copy from HR
  • Access the help menu

Differentiate between salary and hourly performance reviews

  • List the steps of the salary performance review process
  • Determine whether a goals document is required
  • Access a hard copy from HR

Design a goals document

  • Access and open goals document
  • Understand the purpose of creating a goals document
  • Add goals to a goals document
  • Define goals and best practices for authoring them
  • Submit goals and Check status

Maintain employee performance logs

  • Defining the purpose of performance logs
  • Creating performance logs
  • Adding performance logs

Create a self appraisal

  • Access self appraisal
  • Importing a performance log
  • Define rating
  • View goals on a self appraisal
  • Submit and acknowledge a self appraisal
  • Re-import and check the status of a self appraisal

Assemble a performance review

  • Start a Performance Review
  • Understand the process of attaching performance logs to a review
  • Accessing Managers performance Logs
  • Scoring an overall rating
  • Incorporating best rating practices on a performance review

Properly submit documentation through Lotus Notes

  • Submit a performance review
  • Understanding an approving managers role during a review
  • List procedures after the review
  • Use an electronic signature
  • Check the status of a submitted performance review
  • Utilize the functions of the help menu as a resource for further questions

Design Attachments

The following five attachements are the instructional design documents for the e-learning course.   Attachments A-C are the flow charts.  They are used to see how the course will flow together.  Attachment D is a design document.  It details the learning componants of the course.  The Attachment E is the storyboard.  The storyboard provides the developer all of the necessary details to make an effective learning experience for the learner.

Flow Charts

The flow charts turned out to be more difficult than originally anticipated.  An appropriate design was selected after several drafts were authored.  Flowchart.com was the authoring tool that the flow charts were authored in.  Attachment A is an overview for the performance review course.  Attachment B breaks down the components of the course modules.  Attachment C provides a diagram of the performance review process.

Flow Chart(Attachment C)

 

Design Document(Attachment D)

Because of the complexity of the flow charts, a design document was used to further clarify and break down the components of the course.  The design document provides a more detailed look then the flow chart or outline, but without the intimate breakdown of the storyboards.



Storyboards

The storyboards provide detailed information about the breakdown of individual slides for the purposes of passing the design to a developer.  Each slide is set up as a stage, with graphics entering and exiting the scene.  The first column lists the slide numbers and titles.  The second two columns provide information about the slide in context to the learning objectives.  Column four lists all of the text that will be visible to the learner on the stage.  Column five is the verbal script that will be read to the learner, as the learner watches the graphics enter and exit the stage as visual reinforcements. Column six describes the activity that will take place on the stage.  Column seven identifies the graphic titles and numbers that will be used during the scene.  Column 8 refers to the amount of time that the learner should expect to view or hear activity on the stage during the scene.

Storyboards (Attachment E)

The first page of the storyboards shows the main menu layout and how the background should look by providing an example for the developer.  The buttons to the main and help menus are listed along with buttons and graphic hot spots linking to corresponding slides.

There are 17 total pages of storyboards for 99 slides on captivate.  The following screen shots were selected for further examples:

 


 

 Design Conclusion

A wiki was used to provide a sandbox for the design through wikispaces.  The wiki provided a place for project supervisors to make their comments and view the project progress.  The flowchart embedded cleanly into the wiki; however any editing had to occur directly from flowchart.com.  Unfortunately, the glitches in the flowchart provided for several rewrites and caused the project to be delayed from the original timeline.  The timeline was restructured to provide for additional time during the summer.  The design experience, however, was interrupted when the client requested a new deadline.  The new deadline was abruptly pushed back, sending the project promptly into the development phase.

All learning objectives and sub-objectives were designed before development.  The bulk of the text and scripting were designed during the development stage.  Most of the scripting came from materials provided by E & R from the classroom course Power Point slides.  Script was consolidated and re-purposed for the e-learning course.  The script appeared to have different authors and used second (you) and third person (he/she) terms interchangeably.  I rewrote the script to be read from a neutral third person narration.  Regrettably, I should have consulted with the client as they purposefully designed the script to be narrated in second person.  They took the changes and put them back into second person as they wanted to create a more intimate feel.  Since this occurrence, I have noticed that several other e-learning modules follow the second person narration suit (The English Teacher in me cringes!)

Over half of the design was completed during the development phase. The sections left blank were the activities and graphics column.  I had to come back in and describe stage movement and name the graphics.

The graphics were not all named because of the sudden immediacy to get the job done.  The great thing about captivate is that it assigns a number to files that do not have names and organizes the files automatically.  This proved to be especially helpful as I grasped to think of the most effective way to describe design files for a developer.

On a personal note, I have struggled finding a storyboard that really helps me flow and freely author. In the past I story-boarded directly on PowerPoint but I felt that the screen was too confining.

During the storyboarding, one of the processes that seemed to take a bulk of time was formatting all of the wording in the activities and media columns to look uniform.  I spent several hours using bold, italics and adjusting upper and lower case errors.  I also spent time re-arranging the graphic and activity content in a particular order of appearance as to be less confusing to a developer.

The E & R storyboards in the previous examples are on a word doc.   I still felt confined to a box during the project and  In the future, I will take those considerations to generate a more design friendly storyboard template to help streamline the process. If possible, I plan on designing a spreadsheet storyboard template on my next project.

Development

Introduction Information

Title:

The Performance Review Process

Created by:

Angela Rupert in conjunction with E & R

Supervised by:

Andy Petroski, Harrisburg University of Science and Technology

Location of Program:

Entertainment and Resorts

Number of Participants:

Ongoing class for corporate management.

Topics to be Covered

  • Performance Management Process
  • Goals Documents
  • Performance Logs
  • Self Appraisals
  • Performance Review

Media Requirements

Power Point and Captivate will be used to create an e-learning course. Graphics, screen capture and media sources will be used from Microsoft and Adobe licenses.  Users will be required to use their speakers or a headset during the course in order to hear the narrative.

Media and Tools

Media:

Screen capture, simulations, photography, graphics, and animation, text, text to voice, audio files

Tools:

Captivate, Power Point, IBM Lotus Notes Database, Word, Excel

Virtual Tools:

Google Docs, Flowchart.com, Wikispaces, Virtual Server

Delivery Methods

Objective

Skills Taught

KSA

Delivery Method

Explain the performance review cycle
  • Performance Management Process

K

Provide learner with an overview of the performance review process
Differentiate between salary and hourly performance reviews
  • Distinguish between salary and hourly performance reviews

K

Distinguish between different types of employees by using case studies as examples in a narrativeEngage learner by incorporating hot spot formative assessment questions
Create a Goals Document
  • Open the performance review tool on the Lotus Notes Database

S

Simulations will provide the learner a step by step self paced tutorial to open the Lotus Notes Database
  • Best practices for authoring goals

A

List SMART goals and provide case studies as examples in a narrative
  • Creating goals using the Lotus Notes Database form

S

Screen capture will enter and exit the stage as key features are highlighted
Maintain Performance Logs
  • Best practices for maintaining performance logs throughout the years

A

A narrator will explain the performance logs purpose as a means to document behavior
  • Creating performance logs on the Lotus Notes template

S

A true/false formative assessment question will engage learner
Create Self Appraisals
  • Automatically incorporating goals and logs into the form

S

Screen capture will enter and exit the stage as key features are highlighted
  • Identify rating options

K

Animation runs in the background while image is highlighted
Assemble the performance review
  • Automatically incorporating goals and logs and self appraisal  into the form

S

Screen capture will enter and exit the stage as key features are highlighted
  • Identify and distinguish the different components of the performance review from the self appraisal

S

A true/false formative assessment question will engage learner
  • Incorporate best practices during the interview

A

Text will enter the stage as key features are narrated
  • Submit the performance review

S

Screen capture will enter and exit the stage as key features are highlighted

Formative Assessments

  • Learners will be asked true and false questions occasionally during the course
  • Learners will unlock the different modules as they progress
  • Hot spot questions will allow learners to recall prior knowledge and select an appropriate response.

Trainee Evaluation

Learners will use a mouse to forward slides at their own pace.  This will not only keep them engaged, but allow for pause if using a simulation or staged tutorial to walk them through the steps of the performance review in a separate window.  Human Resources will be able to assess the effectiveness of the tutorial by the number of phone call requests received by the help desk for assistance with the Performance Review Process.

Development Conclusion

The development of the project was rushed before the storyboards were completed.  A crude draft was put together for the new deadline.  The client appeared satisfied with the course progress and requested to personally voice over all audio files.  All of the graphics and media were somewhat timed to demonstrate their order of appearance on stage.  Final graphics and text stage appearances’ would need to be correspond with the narration.

Captivate has a feature that allows the author to reduce file size conveniently.  Files that have been abandoned or unused can be deleted from the library with the push of a button.  The same image is recycled and captivate provides an unlimited amount of instances without greatly increasing overall file size. The file size was still too large to send through e-mail so, E & R requested a jump drive as the deliver method of the e-learning course.

Because of the rush deadline rush, I was unable to identify media and author interactions before the development stage.  There are pros and cons to developing before storyboarding.  On the surface it seems there is a certain amount of freedom to move graphics around if you are developing on the fly.  Sometimes the vision of the graphics and how they will move together and interact appears when I see them on the stage.  However, by creating the storyboards after the development stage, I still have to still make significant adjustments to the project to keep it in line with objectives.  Media and text appear to be more uniform as well.  In the end, I think that planning and organization from storyboarding in the beginning saves more time.

Assessment and Evaluation

The assessment and evaluation phase is an integral part of instructional design.  The Performance Review course provides the learner with three separate tiers of evaluation; formative assessment, final assessment and a course evaluation.  Formative assessment questions are administered throughout the Performance Review Course.   Final assessments were created to measure skills and reinforce learning.   Finally, course evaluation gives learners the opportunity to provide feedback for the course.

The assessment and evaluation will be designed directly onto a Go­­ogle form so that learners can take the course and evaluations without any assistance.  A small group of five or more learners have been selected by the Human Resource Department to pilot the course, assessment and final evaluation.

Formative Assessment:

In e-learning, assessments are designed to create interactivity by providing formative questions throughout the course.  Self directed asynchronous e-learning can be enhanced by pop up questions and hot spots to make learning more engaging.  A learner that misses a formative question may choose to review material before progressing to the next level, therefore, increasing skills retention.  In the Performance Review Course, formative true/false and multiple choice slides were authored to synthesize learning material.  Real case scenarios were added along with questions to afford practice opportunities to make correct decisions.

Final Assessment:

A 15 question final assessment was created to measure learner success at the end of the course.  Questions were also designed to reinforce skills.  Learning theories such as blooms taxonomy were used to roll back questions into the course objectives.  Multiple choice, fill in the blank, true/false and checkboxes were used to provide learners a variety of methods to demonstrate their knowledge, skills and attitude.  Higher level blooms text based questions such as short answer and essay response were avoided do to the high volume of managers anticipated to take the course.  Some of the formative questions from the e-learning course were repurposed for the final assessment.  The final assessment was designed not only to measure learning outcomes, but reinforce skills.  Learners are not required to retake the course if they do not pass the skills assessment.

Evaluation

An 11 question evaluation was constructed to provide learner feedback.  Slides or assessment questions can be adjusted accordingly, especially if errors or problematic patterns are detected.  Research was conducted to evaluate best practices for authoring e-learning assessments.  All evaluation and assessment research searches were catalogued.  Learning theory, best practice tips and e-learning assessment examples were carefully considered before authoring questions for the final assessment and evaluation sheet.

Evaluation Theories

Donald Kirkpatrick

The same techniques are commonly used to develop e-learning course evaluations as a traditional course.  Kirkpatrick’s classic model of training is still commonly used to differentiate levels of training evaluation (Kirkpatrick, 1994).

  • Level one evaluations measure the course as an effective learning tool
  • Level two evaluations measure content retention
  • Level three evaluations measure behavior on the job
  • Level four evaluations measure the results of business outcome

The ten question evaluation will be used at a Kirkpatrick level one to provide feedback about the course.  The fifteen question assessment will be administered to learners at the end of the course on a Kirkpatrick level two.

Level one evaluations measure learner reaction to the training and how they feel about their experience.  It is important to gage whether learners feel that the material is relevant to their job.  Some may argue that the only thing level one evaluations measure is if they like the course. (Strother, 2002)

Evaluation Level Description and Characteristics Examples of Evaluation Tools and Methods Relevance and Practicability
1. Reaction Reaction evaluation is how the delegates felt about the training or learning experience. ‘Happy sheets’, feedback forms.Verbal reaction, post-training surveys or questionnaires. Quick and very easy to obtain.Not expensive to gather or to analyze.
2. Learning Learning evaluation is the measurement of the increase in knowledge – before and after. Typically assessments or tests before and after the training.Interview or observation can also be used. Relatively simple to set up; clear-cut for quantifiable skills.Less easy for complex learning.
3. Behavior Behavior evaluation is the extent of applied learning back on the job – implementation. Observation and interview over time are required to assess change, relevance of change, and sustainability of change. Measurement of behavior change typically requires cooperation and skill of line-managers.
4. Results  Results evaluation is the effect on the business or environment by the trainee. Measures are already in place via normal management systems and reporting – the challenge is to relate to the trainee. Individually not difficult; unlike whole organization.Process must attribute clear accountabilities.

© Alan Chapman. A free resource from www.businessballs.com

Fitzpatrick, Sanders and Worthen

In the publication, Evaluating e-learning; a Guide to the Evaluation of e-learning (Attwell, 2006; Fitzpatrick, Sanders, and Worthen, 1997), Graham Attwell merges two learning theories to provide a broader spectrum of evaluation approaches.  Atwell identifies with five of Fitzpatrick, Sanders, and Worthen‘s evaluation approaches:

  • Objectives-orientated evaluation approach
  • Management-orientated evaluation approach
  • Consumer-orientated evaluation approach
  • Expertise-orientated evaluation approach
  • Participant-orientated evaluation approach

Objectives-orientated evaluation approach

Objectives-orientated evaluations are considered goals driven and systematic.  This type of evaluation is often used by companies that need to justify costs by demonstrating a correlation with performance measurement objectives.  Sometimes other results that are not objective oriented are overlooked.  The questions selected in the final course assessment roll back up to the objectives.

Management-orientated evaluation approach

The management-orientated approach uses evaluation information for effective decision making.  Flow charts and decision outcome models focus on the evaluation needs of managers.  Other stakeholders can sometimes be overlooked in the process.  The level one evaluation component will evaluate whether the needs of the managers were met by the e-learning course.

Consumer-orientated approach

The consumer orientated approach uses evaluation to improve service or products for the end user.   The focus is summative rather than predictive and relies heavily on criteria referenced benchmarking.  A small portion of the evaluation will ask end users how the Performance Review course can be improved.  Both the assessment and evaluation will generate statistics into a spread sheet for the Human Resource department.  Human Resources can keep track of the answers and evaluate questions that need to be changed or the delivery of information if too many learners are missing the same question.

Expertise-orientated approach

Expertise orientated evaluation is based on the subjective professional opinion of experts.  The peer review process is most commonly associated with this approach.  Expert opinions may show bias and lack the reliability of an instrument based evaluation.  The peer review process, however, is one of the oldest forms of evaluation.  Despite the drawbacks, peer review works well in situations where subject pools are small and personal feedback is welcomed.  Experts from the Human Resource department were invited to voluntarily participate in the course evaluation process.

Participant-orientated evaluation approach

The participant-orientated approach assesses a wider audience of project stakeholders then traditional evaluations. By gathering data from a variety of sources, evaluators can more readily identify problematic patterns and make adjustments where necessary.  This type of evaluation is often expensive and difficult to administer due to time constraints.  Participants from outside of Entertainment and Resorts were not invited to participate in course evaluation.

Peter Van der Knaap

Learning-orientated evaluation approaches

Atwell expands Fitzpatrick‘s program evaluation guidelines further by adding an additional learning-orientated approach by Peter Van der Knaap.  Van der Knaap’s ‘Learning Oriented Approach’ is a relatively new classification (Attwell, 2006; Knaap, 2004).  The purpose of the learning oriented evaluation is to contribute to an overall learning experience.  Formative evaluation is emphasized and questions are deigned to extend learning.   This approach was incorporated in the assessment components of the performance review.  Formative questions during the process get the subject to think about how they will use the new skill on the job.  Learning oriented assessment questions at the end of the e-learning performance were not only designed to measure skills but reinforce them.

Criteria for Consideration

Research indicates that there are many different criteria that can indicate the effectiveness of course design.  Research indicates that e-learning inventories should be designed primarily to account for the course’s ability to support business objectives, evaluate course design and provide for cultural sensitivity (corpU, 2007).

Business Objectives:

The Performance Review evaluation sheet will enable learners to indicate that the course provided the necessary knowledge, skills and attitude to achieve the company’s desired results.  Strategic questions will ask learners about course objectives and company goals.  Employees can explain what they will need to accomplish these.

Course Design

Questions for a course evaluation should account for the course design.  Ask evaluators to compare the e-learning course to the previously held classroom version.  Did learners find that the formative questions provided for thought provoking real-case scenarios?   An essay response could appropriately reflect how the learner plans to use the new skill on the job.

Course Structure

An effective learning course structure is adult friendly. The course should flow together.  Learners need to be able to navigate through the course with ease.  The course should be broken down into modules that make sense and a simple interface design should be selected.

Practical evaluations ask the learner if course material was presented in a logical format. The interface should be intuitive, the menu uncluttered and easy to understand.  Was the learner able to navigate through all of the buttons, find the directions and help page?  Learners should be provided opportunities to respond on the course structure.

Globalization

Ethical considerations are often passed over on a level one evaluation.  A large company size of Entertainment and Resorts has revolving supply of diverse employment.  Seasonal work attracts a global audience and cultural sensitivity is of primary importance.  The level one evaluation asks learners to offer their opinions about graphics, scenarios, local phrases and humor in the e-learning course that could be misinterpreted as culturally insensitive.

Evaluation Comparative

The following is a comparative level one evaluation sample commonly used by training departments.  The actual evaluation and assessment spread sheets that were used in the performance review course are illustrated further down.

Training Evaluation & Feedback

Course Title & Date 

Rating Options:

a lot

some

little

none

Specific Highlights and/or Suggested Improvements?
Enjoyment: Did I enjoy the course?

New knowledge and ideas: Did I learn what I needed to, and did I get some new ideas?

Applying the learning:  Will I use the information and ideas?

Effect on results:            Do I think that the ideas and information will improve my effectiveness and my results?

Any other comments? 

© Alan Chapman. A free resource from www.businessballs.com. Not to be sold or published 2011

Focus Group

A small focus group of Human Resource employees participated in the pilot course.  The group viewed the course from a learner’s perspective.  After the computer based training, the subjects were administered a course assessment and evaluation.  Six subjects participated in the assessment and five subjects participated in the evaluation.

Learner Demographics

Profile

Range

Average

Gender

Female Only

Female

Age

30-50

38

Years of Service

0-25

8

# of Direct Reports

0-10

2

Assessment Summary

Results

Samples

Percent Correct

Actual Score

Subjects scored an average of 83%.  Question 13 was not averaged into the score (Highlighted in yellow below).

Inconstancies in answers 2, 3 and 5 prompted the authoring of additional slides to further clarify the difference between the hourly and salary performance review during the introduction module.

Total Possible:

100%

28

Subject 1

79%

22

Subject 2

75%

21

Subject 3

93%

26

Subject 4

79%

22

Subject 5

86%

24

Subject 6

86%

24

*Mean Score is 23

Assessment Test              
Question: Answer: (Correct answer is bold) Recommendations:
1. Which steps are optional in the Hourly Performance Review?
  • Goals Document
(69% correct) The information that I received on the performance logs was either incorrect or incomplete.  Either way, the performance logs section should be re-evaluated by the  Human Resource department staff for accuracy.  Staff should make corrections and add additional information to the performance logs section.
  • Performance Review
  • Interim Review
  • Performance Logs
  • Self Appraisal
2. What is the first step of the salary performance review cycle?
  • Open a Goals Document
(66% correct) Continue to monitor the assessment question; however, “Open a salary review” might be a good answer to toss.
  • Open Hourly Performance Review
  • Open Salary Performance Review
  • Establish Performance Logs
  • Create Self Appraisal
3. A department manager has hourly employees, some are full time and others are part time.  They are not part of the union, work regular hours and are expected to meet specific standards but that’s about it.  Which of the following two digital forms should this manager start with?
  • Establish performance logs and then create a self appraisal
(66% correct) I added some additional slides clarifying the salary and hourly performance review tutorials.
  • Establish performance logs and then create an hourly performance review
  • Create a goals document and then create a salary performance review
  • Create a goals document and then create a self appraisal
  • Create a self appraisal and the create an hourly performance review
4. Employees create a digital performance review on the Lotus Notes Database.
  • TRUE
(100% correct) Continue to monitor question for relevance.
  • FALSE
5. A supervisor has a part time employee. She works about 25 hours a week.  Because her role in the department, they discuss how her work contributes to the overall team goals.  What type of review should the supervisor author?
  • Salary
(55% correct) I added some additional slides clarifying the salary and hourly performance review tutorials.
  • Goals Based
  • Hourly
  • Non-Goals Based
  • Interim Review
6. Which of the following five options are not available in the Overall Performance Rating?
  • Exceeds Expectations
(66% correct) I noticed that the check box was not correctly aligned to the answer on this slide in the course.  I corrected the answer and magnified it.
  • Achieves Expectations
  • Not Meeting Expectations
  • Exceptional Performer
  • Not Rated
7. Managers create a new performance log every time they document their employee’s behavior.
  • TRUE
(100% correct) Continue to monitor question
  • FALSE
8. __________________ is a quick and easy method to refresh your memory on what training you may have accomplished that may impact yourself appraisal.
  • Display Training Record
(100% correct) Continue to monitor question, the question may be too easy.
  • View Customized Goals
  • Insert Training Level
  • Scroll Department Records
  • Select Memory Level
9. Performance Logs appear in a different section in the Performance Review then the Self Appraisal.
  • TRUE
(83% correct) The performance logs section should be re-evaluated by the  Human Resource department staff for accuracy.  Staff should make corrections and add additional information to the performance logs section.
  • FALSE
10. Which of the following is NOT considered a best practice when rating a performance review?
  • Documenting personality traits based on the opinions of fellow team players
(84% correct) Continue to monitor question
  • Viewing the Self Appraisal
  • Citing specific example of observable behavior
  • Avoiding discriminatory statements
  • Performance Log Documentation
11. Once inside the Lotus Notes Data base, which are some of the following steps do you need to take when opening a new performance review?
  • Select the File Menu button
(98% correct) This is the question that I designed to encourage the learner to reach back to the tutorial and double check.The purpose of this question was to encourage learners to access the tutorial before answering the question.
  • View Certification Log
  • Select the Lotus Notes Application Menu
  • Select the Notes/400  Item
  • Select Cluster Directory
  • Select HR
  • Select Salary or Hourly Performance Review
12. Managers do not have to wait until their employees have completed their self-appraisals in order to begin the performance review.
  • TRUE
(100% correct) Continue to monitor question
  • FALSE
13. After administering the performance review to your employees, which of the following steps must still be performed before submitting electronic documentation to HR?
  • Personal Reflection
(92% correct) Toss this question, this question was not calculated in the final scores of the assessment* Results not calculated in final score.
  • New Self Appraisal for the following year
  • Goals Discussion
  • Documentation of how feedback was received
  • Everybody signs
14. For employees with Lotus Notes Accounts, an automatic e-mail is generated to the employee with a link to the performance review.  Employees are asked to examine the review for errors and click on the ACKNOWLEDGE button in lieu of a signature.
  • TRUE
(100% correct) Continue to monitor question
  • FALSE
15. After acknowledging the performance review, the _____________ checks the status by opening the performance review database, clicks on the employee’s name and checks the status and dates of all documents.
  • Employee
(100% correct) Continue to monitor question, the question may be too easy.
  • Supervisor
  • Human Resource Department
 
  • Administrative Assistant
 
  • None of the Above
 

 

Evaluation Results

Question

Profiled Response*

Summary of Findings

*Profiled Response is the typical, consolidated or significant responses selected from the evaluation

1. How much will the Performance Review Course improve your ability to perform your job more efficiently?

Range

Average

This was on a scale of one to ten.  The number 8 would indicate a strong improvement for Human Resource workers to perform more jobs effectively.

5-9

8

2. How will you apply the new skills, knowledge and attitude from the performance review course to your current position? “An excellent resource for managers to refer back to, as questions arise throughout the year.” Responses favorably indicated that the tutorial would be an excellent resource for supervisors to refer back to throughout the year.
3. Select the following objectives do you feel need additional coverage after viewing the course. “Maintaining Performance Logs” The performance logs section will need to be revisited.  The performance logs may have had some missing information that I didn’t receive when I created the tutorial.
4. The Performance Review Course was designed to provide management the necessary information for effective decision making. (On a scale of 1-10, please evaluate whether the needs of the managers were met by the e-learning course.)

Range

Average

Results indicate that the course provides HER supervisors the necessary information for effective decision making.

6 -10

8

5. How do you feel that the course could be improved? “Change the voice over and add more on screen movement” It is quite apparent that the voice makeover is the number one concern expressed by four of the five individuals.  I will submit the final copy with a new voice over. Pop up and zoom boxes with additional key words and magnifiers will be added.
6. What did you take from the course to align behaviors and decisions with the organizational brand, mission and values? “The performance review system is now systematic and leaves little room for misinterpretation.  It will get all of the departments utilizing the same review methods.” I found a similar question to this when I was researching effective evaluation questions.  The performance review is closely tied to the  E & R Mission and Values.
7. On a scale of 1-10, did you find that the formative questions provided for thought provoking real-case scenarios?

Range

Average

A comment at the end of the survey expresses favorably for the scenarios.

6 – 10

8

8. Which of the following areas (if any) did you feel needed improvement with the course structure? “Navigation/Buttons/ Interface design” The initial plan for the module was to lock the subjects into each slide so they can’t skip to the next slide.  I think that it will be too cumbersome not to allow the adult learner to have the freedom to navigate through the modules more fluidly.  Learners will still unlock levels but I will add more navigational friendly buttons for the final copy.
9. Which of the following menu features did you find helpful during the course? “Main Menu” The main menu buttons worked out well.  Drop boxes had been considered but not selected due to the client’s wishes that learners watch entire module sections.
10. Were there any graphics, scenarios, local phrases or humor in the e-learning course that could be misinterpreted as culturally insensitive? “No need for in this particular evaluation form to “state your ethnicity”. One response applauded the sensitivity to ask the question.  Another stated that there was no need to ask employees to state ethnicity it in the survey.Interestingly, not one person volunteered their ethnicity in the demographics section.
10. Name three things that you like the most about this course. “The step-by-step process,”  “Formative Assessments,” and  “Easy to follow/use.”                                      This section indicated that the module was user friendly.
11. Please place any additional comments in this section. “Include a certificate of completion at the end.” A window to a pdf certificate can be constructed quite easily and accessed through captivate at the end of the course.

Evaluation Conclusion

Performance Logs

Performance Logs were mentioned as an area that could use further defining.  Based on the response to question one in the assessment (five incorrect), five of the six employees believe that Performance Logs are optional.  The tutorial indicates that performance logs are not optional.  The employee with the least amount of time at  E & R is also under this impression.  Are performance logs optional?  Should the tutorial be changed to reflect that the performance logs are optional, or continue the tutorial in order to encourage managers to maintain performance logs?  Either way, a consistent policy should be established to keep managers and Human Resources from having misunderstandings.

Practice Assessment

One subject suggested providing learning links at the end of each assessment question.  This would enable learners to refer back to the specific section of the tutorial in which the topic was addressed.  Since E & R does not use a LMS, the assessment and evaluation were not authored inside Captivate, (due to the inability to capture results).  A practice assessment may be beneficial if added into the tutorial.  Practice questions should be further designed to encourage the learner to reach back into the tutorial and by providing a link as a reference.  Not only would this enable the learner to access the tutorial to answer their own questions, it would get them comfortable practicing how to access the information for themselves, rather than leaning on the Human Resource Department.  Final evaluation and assessment directions should indicate that learners double check their responses in the tutorial.

Course Adjustments

As a result of the feedback, I will make the following critical changes:

  • Pop up boxes with key words will be added where magnification is needed
  • In some of the slides, zoom features will be added
  • Three additional slides will be added to clarify the difference between hourly and salary performance reviews
  • The final course will feature a voice over
  • Navigation buttons will be added back in

Further Recommendations

The Staff can continue to add or take away features from the performance review as they see fit.  Here are some possible additional recommendations to improve the course:

  • Add, clarify, and adjust additional instruction to the performance logs section
  • Change out any weak or problematic graphics with fresh or original images
  • Create a small window on the screen for key words and questions, preferably in the same spot on all the slides, something similar to a chalkboard
  • Create a practice assessment with links to related material for each of the questions on Captivate at the end of the course
  • Create a pdf window on the final slide for a printable certificate
  • Continue to monitor assessment questions for validity

Project Summary

The Performance Review Project turned into a much richer learning experience then I originally anticipated.  The practicum requirements were challenging, but cleverly designed to provide me with advanced instructional design skills as an end result.  I was quite fortunate to get an opportunity to work with Entertainment and Resorts.  The opportunity to practice the skills I learned in the LTMS program in a supervised workplace setting was really the experience that I desired to complete my studies at Harrisburg University.

The evaluation component was very rewarding.  Even though I only had a small focus group, the feedback that I received gave me valuable insight.  I am grateful for the opportunities that were presented to me and the mentors that went above and beyond their job descriptions to provide me with a valuable learning experience.

Works Cited

Posted in Assessment, authoring tools, Development, education, Human Learning, Human Resources, Instructional Design, Learning Technologies, Low Cost Technology Solutions!, Online Learning, Uncategorized | Tagged , , , , , , , , , , , , , , | 1 Comment

Pod Cast! How to teach a child to read.

Posted in Uncategorized | Leave a comment