Actueel

Nieuws Nieuw accreditatiestelsel Agenda NVAO Congressen Toespraken & Artikelen 2007 2006 2005 2004 2003 Correspondentie Vraag en antwoord Nieuwsbrief RSS Feeds

Toespraken & Artikelen

Speech 'Sets of standards for QA agencies: a comparison' by Guy Aelterman, board member ENQA and NVAO, at INQAAHE Workshop, May 17th-19th 2006

Only the spoken word counts.

I. Introduction

A survey of the international area of quality assurance agencies shows that there are several networks and therefore also several codes of practice and member regulations. In the context of this workshop we will investigate whether a degree of transparency and comparability can be found in and between the standards and criteria that are applied by the different networks. The central question is: are there strong similarities or essential differences and do these in turn relate to specific aims or characteristics of the networks?

Not only is it possible for different networks to apply different standards and criteria, but the interpretation given to each of these standards may differ as well. Depending on the network, the same standard can be interpreted or implemented differently. The differences may be minimal and fractional, but in some cases they are relatively fundamental. It also frequently occurs that the same standards are interpreted and implemented differently within the same network. However, it would be a utopian dream and rather unrealistic if all standards and criteria would have to be defined and applied in the same manner in all networks and by all users.

A third remark or finding is that a degree of standardisation in the central terminology is missing: Standards, criteria, guidelines, references, subjects, items, facets, principles, … it seems a bit of a mishmash.

For the purpose of comparing the different networks we have taken the guidelines for good practice of INQAAHE as a starting point. Noblesse oblige: INQAAHE is the organiser of this workshop. However, there are other reasons to do this as well, including the extensive experience of this network, its global nature, and the not unimportant fact that these guidelines served as a model for other networks. On the other hand, this cannot be a reason not to view the guidelines of INQAAHE with some critical distance and, when necessary, suggest improvements or additions.

The codes that will be compared are: INQAAHE’s code, the standards and guidelines for QA in the European Higher Education Area which we will for easy reference call the ENQA code (approved by the Bologna signatories in Bergen last year), the code of the European Consortium for Accreditation ECA, the criteria for membership of the Asia-Pacific Quality Network APQN , the code of good practice of the Association of Specialized and professional Accreditors ASPA and finally the OECD-UNESCO guidelines for quality provision in cross-border HE. In view of the highly specific approach and objective of the OECD guidelines, we will notice that these do not systematically fit within our comparison, but rather constitute an addition in the specific situation of cross-border HE.

As regards the ENQA-code we first focused on the standards for ‘’QA agencies’’, and secondly on the standards for the “External Evaluation’’ ( The ENQA-code has standards on tree levels: standards for the Internal QA of the HEI, for the external evaluation and thirdly for the QA agencies self.)

Finally, there is the notion or definition of QA agency itself, which INQAAHE calls EQA agency (External Quality Assurance Agency). This term is usually defined in the different codes and used in the various standards.

II. Standards listed in the INQAAHE code

The mission statement (1)

Findings

This item can be found in the INQAAHE (1), ENQA (3.3,3.5), ECA (1,13), ASPA (1) and APQN (1,2) code.

Considerations

A “written’’, explicit statement of goals and objectives is requested.
In addition, INQAAHE and ENQA state that when drawing up the mission statement, the cultural and historic context within which the quality assurance agency has to work has to be taken into account.
INQAAHE and ECA clearly state that QA has to be the main activity of the agency.
INQAAHE refers to a policy and strategic plan that has to be drawn up.

Conclusion

Relatively comparable description of this item in each of the codes that has been studied.
There is little or no dispute or difference of opinion as regards this standard.
Point of interest
The requirement that QA is to be the main activity of the organisation may be a topic for debate for professional organisations that are, in addition to external evaluation and accreditation, engaged in other activities. Such organisations will have to give particular attention to the structural position of their QA activities, as well as the extent to which they meet the other standards of the network codes.

The relation between the QAA and the HEI (2)

Findings

INQAAHE has given a clear description of this item.
In the ENQA regulations it is implicitly included in standards 2.2 and 2.6 and explicitly mentioned in the explanations.
ECA does not mention it as a separate item either, but it is implicitly included in the criteria or references (16, consultative stakeholders, Q improvement, etc.).
APQN refers to it in standard 2, ASPA in standards 1 and 2.

Considerations

QA is the primary responsibility of the HEI.
The QAA have to respect the autonomy of the institutions.
Regular consultations of stakeholders are explicitly mentioned.
The eventual result of the QA has to lead to an attitude of accountability and improvement.

Conclusions

This item only occurs as a standard in its own right in the INQAAHE code; the other networks mention it as an element of reference or include it in the guidelines.
The translatability of this item will in principle not give any problems. The standard is generally accepted.
The way of implementation of the QAA code, the procedures that have been developed, and the criteria that are applied to external evaluation can be a point of disagreement and discussion with the HEI. The activities of the QAA are sometimes perceived as patronising, bureaucratic and not particularly useful or efficient. The HEI feel that rules are very much imposed on them.
Some HEI organisations are of the opinion that the consultations of the stakeholders may not be limited to advice with or without obligation, but have to lead to talks that result in binding agreements between HEI and QAA.

Point of interest

Despite the fact that, in general, the principles listed under this item do not give rise to disputes, only an attitude of openness, communication and mutual respect can effect that the perception remains in line with the principle.

Decision-making (3)

Findings

This item can easily be found in the INQAAHE-code , in the ECA code under standards 3, 4 and 14 , ENQA lists it under 3.6 and 3.7, APQN under 7 and ASPA under 4 and 6.

Considerations

Criteria and procedures have been laid down beforehand and made public.
The SER (Self Evaluation Report) and the predefined terms of reference, available to the public, form the basis of the external evaluation.
Independence in decision-making and the assessment of the panel are primordial.
The decisions will be consistent.

Conclusions

This item is generally accepted and can be found in all codes.

Standards.

Point of interest

Despite the fact that the need for “independence” is considered self-evident by all parties, the notion itself is understood and implemented in different ways: independent as regards whom (which stakeholders)?; independence as regards the evaluation itself and/or the eventual decision?; who appoints for instance the board and how does this organ guarantee its independence ?

The external committee (4)

Findings

This item is mentioned as a standard by INQAAHE and ECA (16), ENQA discusses the item in the context of various standards (2.4, 3.6 and 3.8) without an explicit standard ‘external committee’ being listed in the code. In the APON and ASPA- codes we find it in respectively 4 and 7.

Considerations

The composition of the panels is in agreement with the specific terms of reference and the aims of the external evaluation.
Most codes request that parties lay down explicit rules for the appointment of panel members.
Attention for the formulation of “no conflict of interest’’.
The principle of independence of panels throughout the evaluation process is mentioned in all codes.

Conclusions

There is little or no dispute about the principle and the rules.
There are different views on the proper composition of a panel, but this is very much related to the aims and items for consideration of the external evaluation. This has at times consequences for the balance between quality expertise, educational expertise and professional peers.

Point of interest

The idea of the ‘independence’ of panel members: “the more independent, the less expertise’’.
The presence of students in the panels or the involvement of students in the whole external evaluation process have become a more or less generally accepted requirement or condition in Europe.

The public face (5)

Findings

This item is mentioned in the INQAAHE code (5), the ECA code (standards 8 and 9), by ENQA (3.7), APQN (7), APSA (9) and OECD (20.4).

Considerations

Documents, criteria, procedures as well as processes, review decisions and outcomes have to be made public.

Point of interest

All codes formally state that the decisions have to be made public, but this gives rise to the question what this should be understood to mean. Does it mean that the review report, or a summary of this report, is to be published as well, or would this not be necessary? This question is related to the issue of the aim of the external evaluation and the consequences of such an evaluation. Does the evaluation aim at accountability or improvement, or a combination of both? What are the target groups? Does the report contain confidential information, etc.? See the interesting views of INQAAHE on this matter.

Documentation (6)

Findings

This item is only mentioned as an explicit standard by INQAAHE. The other networks mention this point under various standards, but overall the same basic principles are adhered to.

Considerations

Documents as regards criteria , procedures and processes have to be available.
Sufficiently comprehensive description of what the SER has to consist of, with clear listing of expectations and requirements.
If the assessment leads to accreditation, the rules of both processes have to be made transparent.

Conclusion

This item is not in dispute at all and is considered as entirely self-evident.

Resources (7)

Findings

Resources are discussed as standards or items in every code (INQAAHE 7, ENQA 3.4, ECA 5, APQN 6 and APSA 6).

Considerations

There have to be sufficient means to develop the activities freely and independently. Such means also have to be freely available.
The term ‘means’ should be interpreted as both human and financial resources.

Conclusions

Resources are a generally accepted criterion.
Resources are a basic condition for independent, adequate and efficient operations.

System of appeal (8)

Findings

This item is discussed in quite every code, albeit not always as a standard: INQAAHE 8, ENQA 3.7, ECA 10 and APSA 6.

Considerations

A system of appeal has to be in place and all parties must know it.

Conclusion

This item is mentioned in quite every code, is generally accepted and not in dispute.
This item is a principal condition for independent and, more importantly, objective and correct approach within (internationally) accepted legal rules.

QA of the agency (9)

Findings

This item is discussed in every code (INQAAHE 9, ENQA 2.8 and 3.8, ECA 6 and 7, APQN *, APSA 2 and 5, OECD 20.6).
An internal QA system simply has to be present at each agency.

Considerations

The system has to be continuous.
The evaluation of the QAA has to be based on a SER and an external evaluation.
The external evaluation is cyclical.
The QA policy has to be made known to the public and feedback on the evaluation to stakeholders is essential.

Conclusions

Item is mentioned in every code.
ENQA is most explicit about the course of the external evaluation of the agency.

Point of interest

Who conducts the evaluations (composition of the panel; international or national review)?
What will be the consequences of a negative assessment in the event of an external evaluation?

Collaboration with other agencies (10)

Findings

Collaboration is clearly mentioned as a standard by INQAAHE (10), ECA (11) and APSA (2) but not so much explicit by ENQA.
This topic also receives much attention in the OECD code (20.2 and 20.3).

Considerations

Collaboration between QAA, exchange of good practice or experts, inform each other about evaluation decisions.
Allow succession of international providers.
OECD promotes the building of networks and exchange of information, panel members and experience in the context of cross-border education.

Conclusions

The aspect of networking is most strongly represented by ECA, where it is discussed in the context of mutual recognition, and most weakly in ENQA, although the latter is a network in its own right and organises regular workshops for members.

Point of interest

Attention for information regarding diploma and accreditation mills.

III. Topics not included in the INQAAHE code

Official status (11)

Findings

This item is mentioned in the ENQA code (3.2) and in the ECA code (2).
Specific aspect of the Bologna policy as implemented by the European ministers of education, i.e. quality assurance within higher education for each of the member states that take part in the Bologna process, with a view to the realisation of the EHEA. To a large extent this topic is also related to the principle of sovereignty of each of the countries involved as regards education and the subsidiarity principle within Europe and its member states.

Considerations

Formally recognised by the competent public authorities in the EHEA as an agency responsible for external quality assurance.
Established on a legal basis.
Complying with any requirements set by the legislative jurisdictions.

Conclusions

The official status has been accepted as a more or less obvious standard in Europe, without regard to a number of organisations that have operated in Europe for several years and deliver good work.

Point of interest

This clause underlines a number of basic problems:
Some QAA are recognized by delegated QA umbrella organisations or by accreditation agencies which have the delegated power to recognize other agencies.
There are a number of internationally operating, renowned organisations that deliver work of very high quality, but have not been recognised by the “public authorities in the EHEA’’ and, consequently, cannot possibly meet this standard.

Elements of external review of QAA (12)

The ENQA code not only sets out the standards and guidelines that the QAA have to meet, but also states that the agencies have to be subjected to an external review at least once every five years.
The nationally recognised agencies can be reviewed on a national basis (“external quality assurance agencies established and officially recognised as national agencies by a Bologna signatory state should normally be reviewed on a national basis”). This means that the national authorities organise the review, but that, in addition to any “national” criteria and standards, the QA conditions as laid down in the S&G have to be met.
The panel has to have an international dimension (this point is not elaborated in the ENQA-code).
The results of the review have to be incorporated in an evaluation report that is available to the public.

It is striking that most codes lay down few or no explicit rules that the external reviews have to comply with. For this reason, a number of items for consideration have to be underlined:

The review is an external evaluation of the agency, whereby, in addition to the quality requirements laid down in the code of the network or the umbrella organisation, a number of specific (national, regional) requirements can be set;
If a review is carried out by national authorities, it must in any event be ensured that the evaluation panel possesses sufficient international expertise and an international dimension;
The reviews have to take place periodically, with a reasonable interval between two reviews;
The evaluation report has to be available to the public.

Issues specifically relating to accreditation (13)

The networks that focus on accreditation usually impose “specific” standards or points of reference in addition to the standards and criteria mentioned above. These obviously do not conflict with the general conditions, but rather specify them.

IV. Conclusion

There is a high degree of comparability between the codes of the various QA networks. A certain room for interpretation and implementation of the codes, both between and within the networks, can be justified on the basis of the fact that the original objective was first and foremost to arrive at a number of mutual agreements to meet a certain minimum of quality requirements. Agencies that are qualitatively comparable and subscribe to a code will in turn guarantee that the institutions or courses they have evaluated will also be comparable in terms of quality, which is one of the main aims: to give a clear indication to stakeholders, governments, financers, partners and the public at large about the various course providers and the level of education they offer.

Taking into account that a degree of diversity within and between networks has to be accepted, that mutual trust is a prerequisite for developing reliable quality systems and networks, and that reliable systems and networks function within a framework of very clear and verifiable agreements, we may conclude that there is considerable transparency and comparability between the codes of the major international networks of QAA.

Terug