July 14th, 2015

Implementing Digital Learning KPIs

The importance of implementing digital learning KPIs

Implementing Digital Learning KPIs


There is a great deal of truth in the maxim ‘if you can’t measure something, you can’t manage it’. As part of the digital learning fellowship at PROCAT, I have introduced a balanced scorecard to measure critical success factors. I’ve made this really easy so that data on each measure is added to a worksheet each month and will automatically autopopulate and update the balanced scorecard.

I also keep a weekly report which tracks milestones, activities for the week and the following week and monitors risks and issues.

Implementing Digital Learning KPIs

Here are some things to consider when implementing key performance indicators:

KPIs can:

• Tell you whether you are meeting objectives
• Reveal problems that bias, emotion and time can cover up
• Through Key Performance Indicators (KPIs) and Performance Indicators (PIs), provide a common language for planning and reporting throughout the institution and on the benefits realised through change
• Provide an early warning of problems and an indication of how effective the responses to them have been
• Give you more time to prepare for fluctuations in demand or workflow, personnel changes and other resource issues
• Inform management decisions
• Give vital feedback to staff, students, stakeholders on the quality of products and services
• By making achievements and failures visible, make effective accountability possible.
• Show what you are aiming for
• What you have to do to meet your objectives
• Show you how to measure progress towards your objectives
• Detect performance problems and remedy them
• Determine whether you have the right objectives and when to change them

The questions we will be answering are:

• Are we moving towards our high-level objectives?
• Are we moving towards them quickly enough?
• Are we delivering the outcomes that we should?
• What should we be measuring and how?
• How can we set targets and provide incentives that will improve quality and not distract from key objectives?
• How can we do better?
• Are we getting value for money?
• What is an acceptable level of quality for the services we provide?
• How can we ensure that we meet our business objectives?
• Are we meeting our targets?
• Are we attaining that desired level of quality?
• Could we do more work in the same time (improving efficiency)?
• Could we do our work better (improve effectiveness)?
• Could we save money (improving economy)?
• How are individual staff performing?
• What are we trying to achieve? What constitutes success?
• What processes and outputs do we need to know about?
• What will be measured?
• What levels of performance are acceptable?
• Who will collect the data and how?
• Monitor and measure – what is happening now?
• Have we met our targets? How can we improve?
• Review – did we find out what we needed to know? Should we redefine success? What can we learn from this? Did we realise the benefits we wanted?
• Which of your processes are critical?
• What goals are we trying to achieve?
• What are our core products and services that are helping us or could help us achieve our goals?
• Who are our customers? Do we want more or different customers?
• What are our main business processes and what supports them in their operation?
• What flows of material and information are involved in those processes?

Measuring effectiveness is about determining whether all critical processes are working together to achieve what you set out to do. Effectiveness measures focus on the relationship between outputs and outcomes: how well the outputs of processes or activities within the business are contributing to its overall success.

Examples of inputs:

• Running costs, overheads, consumables
• Expertise
• Staff training
• Financial investment
• Policies, procedures and standards

Less intangible inputs

• Management time / commitment
• Motivation and enthusiasm
• Positive, proactive or sympathetic management attitudes
• Positive business culture

Examples of Outputs:

• Products made
• Projects completed
• Enquiries dealt with
• Proportion of time a service is available
• Turnover and profit

Examples of outcomes include:

• Increasing market share; moving into new market areas
• Improving quality of service to students
• Providing services or marketing products in new ways (online via the VLE, for example)
• Far reaching business change
• Reappraisal of where we are heading
• Enhancing the reputation and public image

Effectiveness measures aim to assess how well you are doing such things, or to what extent you have achieved them.

As well as ‘how much’ or ‘how many’, we also need to measure ‘how good’ – the quality of the outputs or processes. Attributes that could be measured to assess quality include:-

• Usability or usefulness for the student
• Attractiveness – eg. student feedback on the VLE saying it is user friendly.
• Accessibility
• Flexibility – producing a range of study material in talking book form; PDF; print form etc
• Accuracy – downtime of websites;
• Availability of VLE

Hard, numeric measures are always the ideal, but some quality aspects can be very difficult to quantify and their measurement has to be subjective. Usability and flexibility are good examples; by definition, an assessment of these attributes is going to be partly or completely subjective. Time also lends perspective. As data accumulates, subjective questions can become more objective simply because of the weight of evidence.

We mustn’t neglect subjective aspects simply because mathematical techniques cannot be applied to them. Other techniques such as student surveys, will provide data that can be analysed. It is a question of gathering information and analysing it with as much objectivity as possible. For the framework to be balanced, some imprecision or even uncertainty in the framework may be inevitable. This does not mean it gives an unreliable or misleading picture. It would be wrong to imply that all aspects of performance are empirically and immediately ‘knowable’.

A balanced performance framework should include all relevant quality measures; what ‘relevant’ means is specific to each institution.

We don’t want to have too many measures!

There should be no more performance measures than are necessary. Questions to consider are:-

• Is it generally agreed that this activity or process needs to be watched closely, and action for improvement taken if necessary?
• Should it be continuously monitored for improvement?
• Does the activity relate to customer satisfaction and the quality of our service, or is it purely an internal one?
• Is the benefit of measuring performance in this area worth the cost of taking the measurement?
• Can the impact of the activity be determined or directly attributed?

Our KPIs need to give a balanced overall picture. If we focus on one area, we are likely to be led to unmeasured activities being neglected. Balance is achieved by all stakeholders being involved.

Keep information focused, flexible and meaningful.

What makes good KPIs?

• Well defined
• Clear and unambiguous to avoid misinterpretation and ensure consistency
• Easy to understand by everyone who uses them
• Credible enough to be communicated to others, within and beyond the business
• Focused
• Linked to strategy
• Appropriate and useful for the people who will use them and developed with their involvement
• Able to suggest actions for improvement
• Accurate, reliable and verifiable
• Open to comparisons and analysis
• Timely; producing data regularly enough to make it possible to track progress and take action

Why have targets?

Targets are quantified objectives. They express the goals of the business and provide the basis for identifying problems and moving towards solutions. Defining a target answers the question ‘what are we aiming for?’

Targets are more precise and mathematical at lower levels of the process chain (economy, efficiency) and become more subjective but no less real at the higher levels (effectiveness).

Targets should be both realistic and challenging. Out of reach but not out of sight!

Some examples of targets:-

• Service available
• Product introduced
• Changes to the business carried out
• Standards complied with
• Costs reduced to a certain level or by a percentage
• Staff retention
• Specified number of queries answers
• Specified number of staff trained
• Specified number of student feedback
• Service available for a specified proportion of time
• Queries answered within a specified time
• Number of complaints below a specific level
• New systems improve efficiency of internal processes
• Internal business processes transformed
• Culture of the business transformed or improved in specific areas
• Moving into new areas
• Significant growth in business volume
• Public image improved

You and your team must be able to relate to your targets and feel that you have control over achieving them. They can provide a common language within your team and in your institution as a whole.

Choose the right number of targets is important. Too many can be counterproductive and increase the risk of conflicting targets. Too few can also lead to lack of motivation and drive.

Good targets will be SMART – specific, measurable, achievable, relevant and timed

It is important to have measures that have meaning in both ‘before’ and ‘after’ states to show the benefits of change. You will need to discuss and agree exactly what levels of performance it is fair to say are being delivered in the ‘before’ state to set a baseline.

Key questions for setting targets

• Have targets been selected for key activities?
• Are the targets SMART – specific, measurable, achievable, relevant, timed
• Are the targets spread over a range of performance measures, so that you get a balanced picture of what is being achieved?
• Can you be sure that overall performance will not be subverted by the focus on particular targets?
• Are the targets challenging? Do they encourage continuous improvement?
• Are requirements for continuous improvement realistic? Do they take account of possible or likely future events?
• Will targets be revised when the performance framework is reviewed?
• Have the consequences of not meeting targets been discussed and considered?

Visit PROCAT Prospects College of Advanced Technology

This PROCAT Digital website is produced to report on activity from the Digital Learning Fellowship at PROCAT Prospects College of Advanced Technology, funded by The Education and Training Foundation.