Assuring a consistent approach to assessment

In Ireland the National VET providers merged to create Education and Training Boards (ETBs). Before the merger each VET provider had their own individual processes and procedures (agreed with the national Quality Assurance agency) to ensure the fair and consistent assessment of learners. Following the creation of the ETBs, these different practices became obvious and this meant learners whose work was similar could get a different ‘grade classification’ depending on which centre provided the training.

A central aspect of VET provision is the quality assurance of the assessment decisions. There are strict rules and national standards set by the Irish regulator for qualifications, and qualifications are awarded with one of three grades - Pass, Merit or Distinction. Each VET provider decides on which grade to award and uses an internal and an independent external moderation process based on the national standards. This case study looks at how to assure the consistency of awarding these assessment grades across the 34 centres in one ETB.

The Limerick and Clare ETB was created from the merger of nine VET providers which operated in 34 assessment centres in 300 separate locations. The newly formed ETB was responsible for providing (and assessing) 100 programmes and quality assuring the grading decisions of 800 staff. The ETB’s quality assurance support service began by analysing the existing data on grades – this identified very significant differences in practice between the centres e.g. in one centre, more than 70% of learners had received a ‘Distinction’ grade over a five year period. In another centre 40% of learners had received the grade of ‘Distinction’ in the same five years. The other 32 centres were between these two extremes. Figure 1 shows the percentage of learners receiving a grade of Distinction in each of the 34 centres from 2013-2015.

Figure 1

Ireland Figure 1

The quality assurance support service also looked at individual modules taken by learners to see if there was a similar pattern. Figure 2 (which considers a module on communications) confirms that there was a similar range of grades being awarded by the 34 centres.

Figure 2 – the green lines shows the percentage of learners awarded a Distinction in each of the 34 centre

Ireland Figure 2

This analysis highlighted the need to provide more support to staff on the quality assurance of their assessment decisions. In advance of working with individual members of staff, the quality assurance support service organised a seminar for managers and quality assurance staff in each centre. The focus of the seminar was to present the findings and discuss what could be happening. The discussions focused on:

  • which factors could potentially be influencing the number of Distinction grades being awarded across the ETB centres?
  • what initiatives could help to improve consistency of assessment across the centres?
  • what are the challenges the ETB faces in implementing improvement plans, and what solutions could be recommended to overcome these challenges?
  • are there other indicators which could be used to evaluate the consistency and accuracy of the assessment decisions in the 34 centres?

The discussions also considered the national regulator’s statutory guidelines on quality assurance and the regulator’s emphasis on the governance and management of quality at the provider level.

The seminar discussions led to the establishment of a central Quality Assurance Support Service with three members of staff, a new governance structure for quality assurance with Steering Groups and Implementation Groups. In addition a new programme of staff support for quality assurance was developed.

This organisational review of assessment practices included self-evaluation and an analysis of the effectiveness of the ETB’s monitoring processes. As a result there will be strategic and operational changes as new initiatives and interventions are established.

What problems were encountered and overcome in using thes EQAVET+ indicative descriptors?

Prior to the creation of the ETBs, each VET provider operated their own quality assurance system, with local decisions and governance systems. After the merger, in order to consider the effectiveness of quality assurance provision and validity of assessment in multi-centre provision, each centre has to prove that the methods used to assess learners are valid, accurate and reliable across the entire organisation.

Making comparisons between each centre was difficult as the data had to be refined and analysed until a clear picture of the assessment grades became apparent. As each centre had different groups of learners, the quality assurance support service needed to find a way of making a fair comparison between different learner populations. The information which was revealed through the analysis of the raw data raised significant issues for the ETB – evidence of inconsistences between centres. To understand the cause(s) of the problem required analysis and input from colleagues in each centre.

Many potential causes were identified and many potential solutions have been put in place - the following table highlights some of these.

Potential Causes

Solutions put in place

The centres were using different versions of the module descriptors

Common versions of module descriptors put in place

The centres were using different versions of the assessment documents

Development of common assessment documentation for all VET centres

The amount of continuing professional development (CPD) which had been provided for tutors

Development and delivery of common CPD on Quality Assurance to all staff – with an initial focus on assessment standards

The centres used different mechanisms to set standards in relation to assessment

Establishment of Communities of Practice - tutors from common fields developed materials to show good practice in assessment

Communication and instructions provided by each centre to their external moderators

Development of a common communication system to recruit all external moderators. The Quality Assurance Support Service managed the recruitment and allocation of external moderators

The quality of the feedback from external moderators

Development of a consistent approach (and standards) to feedback from the external moderators to tutors

Different approaches to quality assure a proposal to change a centre’s provision

The Limerick and Clare ETB (LCETB) Programme Group was  established - this managed centre approvals, identified and managed quality assurance risks

The centres used different mechanisms to ensure the reliability, integrity and results of learner assessment. There were different ways to protect the integrity of academic processes and standards

The LCETB Quality Assurance Steering Group reviewed the academic results to identify trends and developed a set of themes which emerged from the external moderators’ reports and centres’ examination panels

The centres used different approaches to communicate with centre managers, quality assurance and assessment staff

The LCETB’s Quality Assurance Implementation Groups established formal communications between the Quality Assurance Support Service and centre staff. This included regular updates on national and local quality assurance issues

These changes required new systems to review the effectiveness of the improvements. At this stage in the implementation it is too early to comment on whether the solutions have been a success.

What lessons have been learnt by using these EQAVET+ indicative descriptors?  

The use of data in evaluation and review has resulted in significant organisational learning. The ETB knew that, after the merger, it would have to assure the validity of assessment in terms of consistency across the organisation. It is now evident that significant changes and support is required if the organisation wishes to guarantee the validity of assessment of learning outcomes in each centre.

The process of using valid, accurate and reliable assessment processes to measure learner outcomes is critical to the quality of learners’ experience, employer confidence and staff satisfaction. Effective processes increase transparency in the ETB. Open engagement with the data and a focus on quality factors support staff at the programme and organisational level. The process of collecting data; collating information and analysing the results has resulted in significant organisational learning and increased awareness and understanding of provider-owned quality assurance at all levels in the organisation.

© European Quality Assurance in Vocational Education and Training

This project has been funded with support from the European Commission. This Website reflects only the views of EQAVET and the Commission cannot be help responsible for any use which may be made of the information contained therein

Designed and developed by Arekibo