Actueel

Toespraken & Artikelen

Speech by Guy Aelterman, board member ENQA and NVAO, at the 16th AEEE Conference, Ghent Belgium, August 23rd-26th 2006

Only the spoken word counts.

The accreditation process in Europe
The central topic of this paper is accreditation, and, more generally, quality assurance, in education and training.

A further introductory remark concerns the distinction between “academic” accreditation and “professional” accreditation. The former is concerned with the assessment of a degree programme or an institution that offers education or training, while the latter is concerned with access to the professional field. It should be said that the approach of academic and professional accreditation in the British/American education system differs considerably from the approach in the continental European education system, whereby the division of powers between the EU (read EC) and the various national member states plays an important part in Europe. 

The last introductory remark I wish to make is that this paper is based predominantly on experiences in higher education, but the principles can be applied to other levels of education as well.

Let us first discuss quality assurance in education, without specifically mentioning accreditation as yet.

Most quality systems known in education, but also in the social profit and even the profit sector consist of two levels, namely quality assurance in and by the organisation, the company or the educational institution itself, and external quality assurance by third parties. The former is generally called internal quality assurance, the latter external quality assurance, external assessment, or external evaluation.  Internal quality assurance is to a large extent the responsibility of the organisation, the company, or the institution itself, whereas external evaluations are carried out by an “independent” third party, usually a quality agency, whereby the notion of quality agency should be interpreted in the broadest sense.  

The factors that determine the choice for one or the other evaluation system include the understanding of the notion of quality for and by the institution itself; the aims the institution intends to achieve; purposes of the external evaluation; the manner in which the external evaluation is expected to be conducted; the institution’s partners; the persons who order the external evaluation to be carried out, etc.

In addition, each quality assurance system is faced with the duality of the control and accountability function versus the improvement function, the tension between consultations of partners versus confrontations between opponents, of trust versus distrust, etc.

Quality (definition)
The notion of quality is difficult to define. There are several definitions, with each definition referring to a specific context within which or on the basis of which quality should be defined, as well as to specific views on standards and indicators.

Gravin (1988) has made an attempt to classify the definitions of Quality in five categories:
- Transcendental: these definitions are subjective and personal, based on notions such as beauty etc.;
- Product-related: quality can be measured, based on objective product characteristics;
- User-related: this definition takes user satisfaction as a starting point and is therefore not always particularly objective;
- Production-related: in conformity with certain requirements or specifications in terms of production;
- Value-related: quality is defined in relation to costs.

In the public and services sectors, the non-profit sector, and in education, the notion of Quality is usually bound up with particular basic premises or approaches. Green (1994) has tried to describe the various concepts of quality based on six approaches:
- Excelling = the classic academic approach, cf. centres of excellence
- Meeting (basic) standards
- Fitness for purpose = fulfilling set objectives
- Value for money= value-related
- Consumer satisfaction= consumer-oriented
- Quality of transformation (transformation and innovation processes)

Van Damme (2004) reduces the various approaches of Quality to two dimensions, namely level and relativity (quality is relative or absolute). Relativity itself is concerned with internal or external points of reference. This eventually results in four definitions:
- Excellence = seeking to achieve high quality by applying the highest possible standards. The main emphasis is usually on strict input requirements. Excellence is often associated with an elitist and rather limited society-oriented or society-driven attitude;
- Fitness for purpose = based on internal and relative values. In effect, the institution sets the level itself. There are a number of attractive aspects to this approach, which allows for diversity, flexibility and improvement. The disadvantage of this approach is that the standards are at times set too low and that, frequently, little consideration is given to external factors and the expectations of society;
- Fulfilling basic standards = this approach provides certain guarantees, although these are usually minimal. It strongly tends towards an absolute value-approach, which poses the danger of little flexibility and less attention to the improvement function.  It is the opposite of seeking excellence;
- Consumer satisfaction = aimed at external and relative values. This approach is as it were the opposite of fitness for purpose. It is gaining popularity in a market-driven environment.

The four approaches outlined above all appear in the education sector. The predominance of one over the other depends on the spirit of the times. In actual fact, the truth lies somewhere in the middle of these four approaches, with shifts in emphasis as a result of the effect of action and reaction to which the education sector is susceptible as well. Quality is therefore a multi-dimensional and multi-level concept, consisting of four components:
- Guaranteed achievement of basic standards;
- Capacity to determine objectives in a differentiated context, and to achieve them by means of input and process variables;
- Ability to meet the requirements and fulfil the expectations of the stakeholders;
- Striving for excellence in functioning.
The interaction of these components is influenced by the times and societal views.

Systems of external evaluation
Generally speaking, four models of external evaluation can be distinguished in education: the audit, the assessment or peer review, accreditation, and benchmarking:
 - An audit is in essence an analysis of the strengths and weaknesses of an institution’s quality system. When an audit is conducted, the institution is evaluated on a number of aspects, i.e. the effectiveness of its internal quality assurance; the accuracy, comprehensiveness, and reliability of the data published by the institution with regard to the institution and its programmes; and the implementation of the quality assurance system at all levels, down to the degree programme level. An example of an audit agency is the Quality Assurance Agency in the UK that carries out audits of British higher education institutions;
- Assessment is the traditional system of peer review, a visit or examination, and is usually conducted at degree programme level. This is the system that has been applied in most European countries in the past decades, and which forms, to a considerable extent, also the basis for most accreditation decisions;
- Accreditation includes that an independent body formally and publicly pronounces (yes/no) on the quality of a degree programme or institution, after a quality assessment has taken place that is based on previously agreed standards that need to be met. With accreditation, the emphasis is on independent, public, and previously agreed standards or criteria;   
- Benchmarking includes that various partners (institutions, degree programme providers) simply exchange experiences and develop good practices.

Object of evaluation
An important criterion for the choice of a quality system is to know what exactly should be evaluated. In most cases, the main point of interest is the quality of the institution or degree programme. This is a rather logical approach from the point of view of stakeholders such as students or the professional field. If the purpose is to determine whether institutions should be granted basic funding or awarded scholarships, or to arrive at international classifications of institutions, the preferred option will be to evaluate institutions. Another option is to evaluate the output, as happens with certain professional accreditations (access to the profession); or even staff is subjected to an assessment. The latter is rare in Western Europe, if it occurs at all.

Focus
The topics that are focused on in the event of an external evaluation can frequently be directly related to the objectives of the evaluation or what is to be evaluated. There is a wide range of possible topics to focus on: input (curriculum design, content, staff, infrastructure), education processes, student support and guidance, outcomes, infrastructure, financial aspects and viability, administration, qualification, organisational processes, strategic management, etc.

Usually, a combination of topics is evaluated. In case of the accreditation of higher education in Flanders and the Netherlands, these topics are the aims and objectives, the course programme, staff, facilities, quality, and results.

A striking fact is that there is a perceived hesitance to report on the topic of results or outcomes in the evaluations, and that it often meets with resistance. In addition, the results are too much viewed as an aim in themselves rather than a policy instrument or indicator.

The topic of quality would seem evident, yet was not made an explicit point of attention in external evaluations until in the last decade. Prior to this, the institutions used to have some or another form of quality assurance in place, but this was not normally the object of structural thought or active development.

Purposes of external evaluation
The traditional purposes of external evaluation are well-known: accountability, improvement of quality, funding, degree awarding power, recognition of diplomas and degree programmes, access to the profession. Less frequent or noticeable are evaluations that involve an investigation into the system or methodology itself. The purpose of such evaluations usually is to adapt an existing quality system, or they take place in the context of mutual recognition of agencies and/or degree programmes.

Methodology
There is relatively much variety in methodology, yet this does not necessarily lead to widely divergent conclusions.

- Each external evaluation starts with a self evaluation by the degree programme team or the institution. In the traditional assessment or accreditation systems, the findings are laid down in a report, while a considerable number of audit systems prescribe that documents and data are supplied in a structured manner.  

- The assessment can take very different forms. It usually concerns a peer review accompanied by a site visit, but there are variants to this method. The assessment may be limited to the analysis of data and documents. Peers are traditionally associated with experts in the field, but this is often not the case. They may be audit experts, education experts, representatives of stakeholders or persons ordering the external assessment, and there may be students among them.

However, it is clear that the quality of an external evaluation depends entirely on the quality of the panel.

My organisation, the NVAO, always presupposes that the panel must have expertise with regard to quality assurance, the relevant academic or professional domain, international comparability, educational development (programme development, didactics, methodology), the higher education system of the country concerned, management and administration (expert knowledge of the latter is required for evaluation of institutions). Also, it is important that the panel possesses international expertise and that it contains a student.

- The panel report may consist of a final decision on the basis of the external evaluation process (in the traditional visit or assessment), but it may also constitute the document on which the evaluation agency/body eventually bases its decision. For instance, in the accreditation system for higher education in the Netherlands and Flanders, the assessment (including the element of improvement) is kept separate from the actual accreditation.

- The involvement of the stakeholders in the entire evaluation process may vary considerably, as does the follow-up process.

In view of the above, it may be concluded that there is a considerable degree of variation or choice in the way the various quality systems are elaborated. On a European scale, we have to conclude that increasing numbers of countries are taking the step towards accreditation.

What informs the choice for accreditation? The motives to choose for accreditation vary, but in general the reasons tend to be:
- demonstrating that a particular basic quality is in place;
- accountability towards (national) governments with regard to allocated funds and autonomy;
- a public verdict (yes/no) given by an independent organisation, on the quality of the relevant degree programme/institution;
- the possibility to impose sanctions in the event no accreditation is obtained;
- promoting mobility, or international recognition of degree programmes or institutions.

Accreditation does not take place with a view to ranking, and it may have the disadvantage that at times the aspect of accountability becomes more important than the aspect of improvement. 

The choice for accreditation is not just a matter of the national level, but also, and in my view more importantly, of the international level. After all, in Europe, quality assurance in higher education is an intrinsic part of the Bologna process.

In the Bologna Declaration, which was itself preceded by the Sorbonne Declaration, a number of significant objectives were formulated to reinforce and clarify the position of European higher education. It was the European countries’ response to a number of global and also individual, internal challenges.

Via the Bologna Declaration, Europe intends to:
- introduce a recognisable Bachelor’s/Master’s degree structure;
- encourage and stimulate the mobility of students, academic staff and graduates;
- increase the transparency, compatibility and comparability of programmes and diplomas;
all of which, of course, in order to continue developing a strong knowledge society and economy.

In order to support and achieve this aim, we need the ‘Promotion of a European co-operation in quality assurance with a view to develop comparable criteria and methodologies’ (Bologna Declaration).

Tackling the need on the basis of substantially extended quality assurance in higher education is a theme that has returned in all the follow-up conferences. The strongest, and certainly the most explicit, reference was made in the Bergen Communiqué, which was drawn up in 2005. This memorandum mentions not just an overarching qualification framework, standards and guidelines for Quality Assurance and a European Register for Quality Assurance Agencies, but also ‘Mutual Recognition’.
‘We underline the importance of co-operation between nationally recognised agencies with a view to enhancing the mutual recognition of accreditation or quality assurance decisions’ (Bergen communiqué, 2005).

This is both an enormous challenge and the key to the mobility that is in all quarters. In addition, the communiqué did not (yet) mention mutual recognition of diplomas or grades, but of accreditation or quality assurance decisions instead. The recognition of programmes and diplomas is, moreover, not a matter of immediate concern, although it is facilitated by the recognition of accreditation or quality assurance decisions.

The question is: how should we take up the challenge of the Bergen Communiqué?

The ECA, the European Consortium for Accreditation, a network and project, took up the challenge as long ago as in 2003 and intends to achieve mutual recognition of accreditation decisions in around 2007. The ECA includes 16 accreditation agencies, spread over 10 countries, but the method developed by the ECA members is transferable to other partnerships. In other words, the principle of mutual recognition developed within the framework of this project can be used just as much for the recognition of decisions from other evaluation agencies.

In effect, mutual recognition means ‘If I, as agency A, were to implement the accreditation that agency B has implemented, I would achieve the same result.’

The step-by-step process developed by ECA to realise mutual recognition can be summarised as follows:
(a) Analysis of mutual standards and procedures;
(b) Recognition of mutual standards and procedures;
(c) Recognition of results of assessments;
(d) Recognition of accreditation decision.

A process like this is certain to succeed provided:
- there is mutual trust;
- people accept a certain diversity of criteria and procedures;
- people accept mutual verification of data and results.

This assumes that:
- where necessary, national regulations must be amended;
- exchange and cooperation projects between the different accreditation or evaluation agencies will be set up; 
 - networks of accreditation or evaluation agencies will be developed.

We hope to realise the mutual recognition of accreditation decisions between the 16 ECA-partners before the end of 2007. Recently, a number of ECA countries even made a next significant step. They signed with the ENIC/NARICs a “Joint Declaration concerning the automatic recognition of qualifications’’ (ENIC: European National Information Centres & NARIC: National Academic Recognition Information Centres).

The aim of the Joint Declaration is the automatic recognition of qualifications (diplomas) on the basis of mutual recognition of accreditations. For example, if NVAO accredits the Master of Financial Economics at the University of Leuven, the Austrian ENIC/NARIC should not only recognise the accreditation decision in Austria, but also this diploma this qualification. Should, because it will take time and effort. The reality is somewhat more complex than it looks.

Let me try to explain what could be the significance of the mutual recognitions.

The mutual recognition of accreditation decisions readily allows a statement on the level of a programme (Bachelor’s or Master’s) to be made. A statement regarding the orientation of the programme (more or less research-oriented) and regarding the domain will be more difficult to make. Both the Bachelor’s and Master’s levels are clearly described in a widely accepted manner in the Dublin descriptors or in the overarching qualification framework approved in Bergen last year.

The following steps are more difficult to take.

First of all, there is the aspect of programme orientation, i.e. the orientation of the Bachelor’s and Master’s programme in terms of a professional or academic or scientific orientation. The approaches of the different countries, usually implemented in their legislation, differ considerably. This has to do with the historical and/or cultural background, or the (national) frame of reference within which higher education is organised.

Limiting myself for the time being to Master’s degrees, it is apparent that one side of the spectrum is purely professional, with the research context being minimal, while the other side is academic or scientific. In the case of, for example, the research Master’s, the research aspect is obviously at its maximum.

Various countries, such as the Netherlands and Flanders, have anchored this difference in their legislation. Some countries screen it off by using the suffixes ‘of sciences’ or ‘of arts’, while others translate the different orientation into the programme profiles, final competences or the access to professions. One has to conclude that the division into professional and academic Master’s degrees as such is becoming fairly untenable. The focus is often on mixtures of professional and academic degrees, with the concept of research also being translated into a spectrum which goes from fundamental, via applied, to practice-based research. The term research Master’s degree gives a clear picture of the importance of research in the programme as a whole. In the more professionally oriented degrees, a certain variability of the research component or impact can be observed.

For example, quite a number of management or accountancy programmes at Master’s level are very professionally oriented, although the required initial competency is a Master’s degree, preferably with a scientific orientation. In other words, some research competency has to be present at intake. Other professional Master’s programmes then, of course, demand a Bachelor’s level as an admission requirement, as well as some considerable professional experience. A striking fact is that knowledge development centres or teaching staff with research experience are often linked to what used to be professional Master’s programmes, where they perform on the professional-innovative interface, further explore the knowledge domain, and translate this into innovation in their own fields of work.

How do we then deal with the orientation in international comparisons and in mutual recognitions? The answer is to lay this down in competences to be acquired within the programme, validated by means of quality accreditation or evaluation, and possibly confirmed in a diploma supplement.

Incidentally, the same applies to mutual recognition of domain-specific aspects. This can only be expressed through competences which have been acquired in a validated way. The AACSB- or the EQUIS- accreditation are examples of accreditation whereby certain domain-specific competences are laid down in a structured way or are agreed for validation.

Terug