Exam Success rate – a measurement of training effectiveness?

How do we measure the effectiveness of trainings/trainer? Rather how should we measure the effectiveness?

I will explore the question in an ITIL ® context here, though the impact of the question and its answer(s) are equally relevant for any trainings.

The effectiveness should measured through the outcome delivered by the training.

Of course it depends on the objective of the client and/or students – but generally the outcome can be measured through a couple of things

• How much the training enabled the students to enhance their knowledge/skill which they are able to apply into practice.
• If it is a certification based training, how much the training enabled the students to achieve certification

Here, the first parameter is difficult to measure (if you consider the last part of the measurement: “ able to apply into practice”) Here, there is a definite factor of capabilities, profile etc of each student.

So training organizations tend to measure that a step earlier – at the end of the training program, through training feedback. A good metric considering the constraints, but not the best obviously. Most definitely, the organizations where the students belong, would want to measure the later parameter – of enhancement of knowledge and enhance in their performance/value to the organization.

In this post, I want to focus a little more on the second measure: Success rate in certifications.  Most of the training organizations do use this, and I have seen more than a handful organizations where that is a major (and many-a-time,  ‘the major’) metric for trainers’ performance.

In an ideal context, this gives a perfect idea about the effectiveness of the training (like saying ‘under ideal testing conditions’). In the world we are in, I have a few concerns on the same, which need to be addressed by the training organizations, when they want to measure the effectiveness using the ‘success rate’.

Let me list some of those concerns and scenarios at the top of my mind:

 

ITIL® Student of the year (UK)

Saw the press release yesterday from official ITIL site:

‘Donovan Pearce, Joint Winner of itSMF’s 2009 Student of the Year Award’ :  Read the press release on the Official ITIL Site.

First of all, personally I was not aware of such an award. So after reading the press release, I decided to update my info-base.

This is specifically by itSMf UK and the basic criteria is you should have taken multiple certifications in UK.  (Of course, the official site press release doesn’t mention that)

This is what is there officially at the itSMf website, if you are among those like me who didn’t know: itSMf UK awards 2009

Here are some details about the award, available on the itSMf UK site:

 

The ‘evergreen’ story of Incident and Problem!

There are at least two types of topics that people never have enough of:

a) Those topics people are big fans of and

b) Those where people never reach a conclusion!

Some thing like the ‘chicken & egg’ one? Yeah some thing like that!

We have such a ever-green topic in ITIL® : Topic on Incidents and Problems!

Isn’t really amazing that it is such a hot topic of debate, even after ITIL framework is in its 3rd version (and undergoing the next revision!) and been in wide use for more than two decades?

There are a few current hot discussions going on the same topic – just to point a few  : a) one on LinkedIn, another one at the ITskeptic site as well as one on the Pink Elephant blog. The unbelievable number of responses and replies on all those discussions (and the fact that most of those discussions are continuing without conclusion) substantiate my topic here.

Some of the key confusions that still seem to prevail are:

1. When to initiate a problem ticket (though some still ask this question intentionally or by mistake in the notorious ‘When does Incident become problem’ way –and responses come flooding – oh no.. never….!) and most of the time one has to conclude the ever-effective ITIL response: ‘It depends’ . As I mentioned in the ITSkeptic discussion, it could be the reason, ITIL could never been very clear and prescriptive on that!
2. Should problem investigation and resolution should happen in parallel to the incident handling or should it be post the service restoration? (One obvious factor here is whether you have different function/team for Incident and Problem management or not).

The factor of concern though, is:

The points I high-lighted above looks very basic and core factor in implementation of Incident and Problem management.  Even after such lengthened discussion across the forums, the ITIL documentation is not really addressing those basic considerations. Even a high-level guidance on the same with a couple of possibilities (as the extension or examples to the ‘it depends’ factor) would have helped in getting some kind of streamlined understanding of those in the industry.

The currently undertaken  ITIL v3 revision doesn’t seem to be addressing this – as it is not apparently one area identified in revision (as far as I could find anywhere). Hence we might have to wait for ITIL v4 to expect some kind of light at the end of the tunnel!

 

ITIL® is a Registered Trade Mark of the Office of Government Commerce in the United Kingdom and other countries