Goal, process and outcomes based evaluation

For this project elements of all 3 types of evaluation will be included:

1. Goal based evaluation - this focuses on how the learning experiences address the business problem. Does this learning fill the gap on current practice in reading, interpreting, reporting and reviewing non-18FDG scans? Does it equip radiologists with the skills needed to read and report on PSMA?

2. Process based evaluation - each phase of ADDIE from analysis to implementation will be evaluated. Feedback from all stakeholders about Analysis, Design, Development and Implementation will be collected with a view of seeing what can be improved for future development projects.

3. Outcomes based evaluation - through assessing learning objectives and looking at application of skills on the job, the impact of the digital learning experience on the learner's knowledge, skills and behaviours can be evaluated. 

Model of evaluation

Although there are other models of evaluation, for this project the Kirkpatrick Model will be used, incorporating the 4 stages:

Level 1: Reaction

Level 1 measures learner experience and engagement with the course, it is reviewed on an ongoing basis as learners flow through the experience:

• Learning data analytics from LearnDash LMS will provide data on

  • • Course enrolment numbers

    • Course completion data (What percentage of learners have completed? How long did the learning take? Were there parts of the course that took significantly longer than others? If they have not completed, where did they stop within the learning experience?)

•   Learner feedback survey

  • • A feedback survey will be embedded into the learning platform and become available to the learner upon completion of the course

    • Survey: https://www.smartsurvey.co.uk/s/PSMAPETCT/

Level 2: Learning

This level is an integral part of the training experience and evaluates and measures whether the course achieved the intended learning outcomes. It evaluates the transfer of learning of the participants and measures: knowledge, skills, expertise and is done using the end of course assessment but is only effective if the questions in the assessment align perfectly with the course content and learning outcomes.  Again, the LearnDash platform will provide the tools to do this, and this evaluation will occur on an ongoing basis as learners access and complete the course:

• LMS report on quiz completion, number of attempts to pass and average scores

Level 3: Behaviour

Level 3 evaluation will take place a few months after the rollout of the training course. Behaviour changes take a while to take effect within an organisation, and so level 3 evaluations are typically conducted 3-6 months after the training. Evaluation is designed to determine if the newly acquired skills, knowledge, or expertise are being used in the every day environment of the learner. To be most effective, the level 3 evaluation approach will be a bi-yearly review of relevant metrics and performance review data. Evaluation will be in the form of:

• Manager's survey

• Performance reviews - are the numbers of radiologists reporting more accurately on PSMA PET-CTs?

Level 4: Results

Level 4 data measures how the training program impacts and contributes to the success of the organization as a whole. For this project the leading indicators of business impact are:

1. Improved patient experiences and better clinical outcomes for patients.  

2. An increase in the number of radiologists skilled and equipped to report on PSMA PET-CT.

3. An increase in the number of PSMA PET-CT scans reported.

Useful 

The  commentary from Part Two Screen Plan and Storyboard goes into specifics and answers the following:

• Has the course been designed to minimise cognitive overload? (Chunking of content has been considered and content has been separated into lessons and modules)

• Have Meyer's multimedia principles been addressed? (The course contains images/interactions/examples of text formatting to bring the design to life. Content has been thought through in line with Mayer's principles.) 

Desirable

This incorporates visceral, behavioural, and reflective design:

• The course has been designed using Rise 360 templates to contain images and text layouts to visually appeal to the learner

• Real case-based scenarios are used throughout the course to connect the learner with the material

Accessible

The course is designed with Universal Design Principles in mind:

• Multiple means of engagement through different learning activities and the extent to which the course is accessible for all learners (see UDL section)

Usable and Findable

• The learning materials are easy to find, all interactions work and have been tested

• Assessments are easy to find and cross-referenced with scan images that need to be accessed through the internal PACS system. They have been fully tested and instructions are clearly set out for learners

Credible

• All content has been validated by leading experts in the field, any research papers that the content refers to have also been validated as peer-reviewed and credible

• Instructional design principles have been fully adhered to:

  • Learning outcomes clearly defined and articulated, content and assessments are aligned with outcomes, a discussion forum is included for learning reflections

• Quality control processes are in place:

  • Content edit and QA / peer review to ensure images/text are referenced correctly, spelling and grammar is checked, consistency standards

The Digital Learning Institute's UCDL Honeycomb Model Checklist (see Appendix 1) is a tool to measure and evaluate based on the segments of the honeycomb listed above. The team will use this tool to evaluate the course and put a plan in place for any improvements needed. It will be the job of the Learning Developer/ID to complete the form. As this course is asynchronous eLearning some of the parts of the checklist related to Communication and Connections and Building a Community are not fully applicable. The course does contain a discussion forum and it is hoped that in future course development projects we can use more of the points on the checklist as a guide for building better communication, connections, and community. As we look to include more social learning and networking opportunities within our course offerings in our LMS, our Marketing & Events Coordinator will be responsible for evaluating against these 2 areas of the checklist.

Data collected will help inform such questions as: are there areas of the course where learners are stopping and not moving forward? Are there stumbling blocks to course completion? Are learners successful in the assessments? How many attempts at the assessments are learners taking? Are there specific quiz questions that are causing problems for the learner? For example:

• Learners are stopping and not moving forward/ stumbling blocks to completion.

  • Do we need to change the course expectations set out at the beginning of the learning? Are learners running out of time because we have anticipated the learning taking less time than it is in reality, and therefore need to adjust the time requirements for completing the course? Do we need to implement protected study time for our learners?

  • Is there a particular module where learners are stopping? If so, we need to look at the content and check that it is relevant, that the outcomes for this part of the learning are achievable.

• Unsuccessful assessments.

  • If there is a high rate of failure, have we set the assessment at the appropriate level? Do the assessment questions refer back to the learning content and the learning outcomes?

Accessing and analysing this data will help promote data driven design and identify areas of improvements to the course, through a cyclical continuous process of monitoring, gaining insights, and taking action.