Actueel

Downloads bij deze pagina

Toespraken & Artikelen

‘Considerations on Publications: Advantages and Disadvantages of Publications of Accreditation Decisions and Report’, speech by NVAO-chairman Karl Dittrich at ENQA-Conference, 14-15 February 2008, Oslo

1. Introduction

Our society is becoming increasingly transparent. The decrees and decisions of governments and government bodies have to be made public and must be based on sound lines of argumentation and reasoning. Citizens, organizations and companies are seeking their "rights" and are understandably no longer satisfied with administrative bureaucratic decisions based purely on power. And if the parties themselves do not look for the reasons for and backgrounds to decisions, there is always the press that can call on public access rules to uncover the truth. Moreover, organizations that protect the rights of consumers and citizens are increasingly better informed and equipped to play an important role in the publication of decrees and decisions.
The time in which decisions could largely be made in back rooms lies in the far distant past. And the time in which decisions were made as simple announcements with no explanation of the content or accountability is scarcely imaginable.

As quality assurance agencies we are all facing this problem and state of affairs. Whether we evaluate, audit or accredit, the duty to make well-founded decisions and to make these public is one we must all fulfil. This is, however, not simple, because - quite rightly - in a civilised society the privacy of individuals and organisations is also protected under the law. Therefore, we sometimes have to wrestle with conflicting interests and if the national legislature has not set down clear rules, we ourselves have to ensure the necessary transparency and weigh the interests involved. I would like to talk to you about this in the next thirty minutes. The questions to be dealt with are: "How do quality assurance agencies handle the publication of their decisions, which considerations play a role in this and, to line up with the subject of this seminar, what effects can the publication of decisions have on the position and procedures of the assessing experts?"

I do not pretend that my treatment will be exhaustive and I will not be making any value judgements; I would simply like to try and describe the complexity of this problem. I am doing this because of the importance of our work: the assessment and promotion of the quality of higher education.

2. The practice in Europe

In recent weeks, I tried to find out from 16 organisations, all members of ENQA, about their publication policy on decisions regarding the assessment of the quality of institutions and/or programmes. This was not easy because my command of quite a few languages falls somewhat short. It is true that virtually every organisation has a website in English, but the content of these websites is usually much more limited than the native language version. In addition, I made use of the questionnaire that NOKUT sent out to the participants of this seminar. Furthermore, I was able to use secondary sources as a basis, namely the bilateral comparisons of content that the members of the European Consortium for Accreditation have drawn up in recent months and years, as well as a number of publications. This notwithstanding, my analysis could still contain errors, and if this is the case, I am entirely accountable! I would also like to emphasise that not all the organisations make "decisions". They may also conduct evaluations and the need for publication is then much less evident, as we will see later. The EUA and the quality assurance agency of Baden-Württemberg (Evalag) do not publish their evaluations but rather leave this to the evaluated institutions themselves.

If my interpretation and analysis is correct, the decisions of our organisations are widely published, but in varying forms:
- In the most comprehensive form, all positive and negative decisions are published, including the underlying reports of the panels: this appears to be the case for all Scandinavian agencies except NOKUT; the Accreditation Organisation of the Netherlands and Flanders (NVAO) and the agencies that carry out the assessments within this system; the British QAA and the organisations from Estonia and Lithuania;
- A less comprehensive form is that in which all decisions are published but without the underlying panel reports: to the best of my knowledge, this is the practice of the Austrian ÖAR, the Polish PKA, the Norwegian NOKUT, the Bulgarian NEAA and the French CTI;
- An even less comprehensive form is that in which only positive decisions are published. There are three variants in this category: decisions together with the panel reports (the Spanish ANECA?), decisions together with the panel reports if the applicant consents (the Austrian agencies FHR and AQA, and the Swiss OAQ) and summaries of positive decisions and panel reports (the German agencies ACQUIN, AHPGS, AQAS, FIBAA, ASIIN, ZEVA and the Akkreditierungsrat).

As I mentioned, it is quite likely that my classification of these categories is not refined enough and that my interpretation of what I read is not entirely accurate. We will have time to go into this in more detail in the discussion to follow.

3. Backgrounds and explanation

A number of different - sometimes conflicting - arguments play a role in the discussion of the way in which the assessments of quality assurance agencies are published, and in particular the scope of the publications. In this paragraph I will discuss several of these arguments, once again with certain reservations and no pretence of being exhaustive.

a. The importance of transparency
In a number of countries, the "transparency" of the actions of the government or of bodies authorised by the government to make decisions, is literally set down in law, legislation or administrative practices. In particular in Scandinavia, England, Belgium and the Netherlands, transparency weighs so heavily in the balance that all documents that play a role in the making of decisions must be automatically and unreservedly published. The party being assessed is aware of this and can therefore take it into account. Needless to say, rules also apply in those countries where almost total transparency is the case that oblige the assessor to act "properly" (rules of "good governance") and the assessed party has all kinds of rights to hear and be heard, to object and appeal, but ultimately every decision is made completely public. This transparency is naturally to the benefit of the "public", whether you define this in terms of students, the labour market, the political arena or society in its broadest sense.

b. The importance of accountability
From the perspective of government responsibility for access to and the funding and quality of higher educations, it can be expected that there is accountability for the assessed quality of programmes. This is something those politically responsible, the labour market and students and their parents (as part of society as a whole) are entitled to. Government regulations, for example, about the publishing of decisions, are often supported by arguments about the large financial contribution the government makes to the funding of academic universities and universities of professional education. The government virtually always derives "rights" from this funding, that have to do with the public accountability of higher education institutions. Needless to say, the degree of autonomy of the higher education institutions plays a role in this, and thus the relationship between the government and the institutions. But even in countries where institutions have degree-awarding power, the accountability obligation is by and large not subject to discussion.
An interesting and special type of accountability can be present in countries with an extremely open education system, like those of the Netherlands and Austria, where all kinds of private providers can enter the education sector. The relationship between these providers and the government is not so much expressed in terms of funding, but rather in the awarding of degrees and qualifications for which the government desires quality guarantees. In that respect, these private providers are obliged to "comply" with government regulations.

c. The importance of consumer protection and information provision
The publishing of decisions contributes to the information available to "consumers" of education and protects them from qualitatively poor or weak providers. This objective can be fulfilled in an active or in a passive sense. Fulfilling in an active sense entails publishing both positive and negative decisions; if it is fulfilled in a passive sense, then consumers must deduce the possible existence of a negative decision about the quality of a programme from the absence of a positive decision. This demands a certain amount of interpretation, but if people are familiar with the regulations in a country it does not have to lead to any problems. It is striking that the form in which information is made available to "consumers" is not very well developed yet. In a number of systems, the commitment and perseverance of prospective students is severely tested. The information made public by the organisations is so large in scope that finding relevant information becomes extraordinarily complex! The full scale of this problem became clear in the ECA Qrossroads project aimed at realising a single European information tool for accreditation decisions.

d. The importance of the right to privacy and the protection of "sensitive" information
In several legal systems, mainly those based on the German legal system, a great deal of emphasis is placed on the protection of the interests of the assessed parties. The basis of this approach is to be found in a "dislike" of an overly powerful state and the protection of the individual (a person as well as an organization) in respect of the state. Not making negative decisions public, for example, is based on privacy legislation and breaching this in general leads to severe penalties. Moreover, account must naturally be taken of considerations regarding competition. Institutions and programmes find themselves in a situation in which there is competition for students and lecturers. From this perspective, it is therefore not a matter of course that all information and decisions are made public.

e. The prevention of the improper use of decisions
In several systems, in addition to the arguments above, another argument is used to limit the publication of reports and decisions: the prevention of the improper use of this information. This argument is mainly used to discourage ranking. In journalistic practice, you can see with a certain amount of regularity that the assessments of panels are rated in a particular way and then added up to arrive at rankings. This is considered undesirable for a number of reasons: the quality assessments of panels are not by definition comparable due to the different composition of the panels; not all of the subjects assessed are of equal significance for the quality of a programme; the comments, often aimed at improvement, are taken as absolute judgements whereby a distorted image of the quality of a programme is created, etc.

f. The importance of legal "tenability"
If the decision is made for the complete publication of decisions, lines of reasoning and assessments, people should be prepared for possible debates and legal proceedings as a result of unequal assessment, a possible lack of consistency between different assessments and for the extremely careful consideration and substantiation of assessments. In some systems this would lead to a very heavy set of formal requirements for decision-making that could have the consequence of highly bureaucratizing tendencies. In practice, this could result in sizeable, difficult to read and relatively veiled reports.

g. The protection of panels and their members
As a final consideration, it can be pointed out that in practice, although in a number of systems decisions are published, the underlying panel reports are not made public. This pertains mainly to those systems in which recourse to the courts is very easy. In some systems, we see that the panels (or even persons on the panels, particularly the chair) are threatened with legal proceedings by individuals or institutions. This is, of course, extremely undesirable within the framework of peer reviews and it is understandable that panels are protected against such practices. Legal proceedings should then be instigated against the quality assurance agency rather than the panels.

Those who are responsible for the policy regarding making the results and the underlying reports of the panels public, therefore have to take into account a large number of different interests. Some of these interests are in opposition to one another, as is the case for the complete publication in favour, for example, of a rational decision-making by students about their choice of study versus the right to privacy of the programme, the institution and individuals. The considerations made and to be made are often expressions of the political culture in the country concerned. These cultures have great differences, varying from total and virtually unrestricted transparency to forms of transparency that take individual interests relatively strongly into account.

4. Panels and publications

Yesterday we heard how important experts are in assessing the quality of education and research. Panels must therefore be compiled with the utmost care because the independence, expert knowledge and authority of the panel are the deciding factors for the legitimacy of the assessment of a programme or institution. This legitimacy is, of course, also for the outside world that has to be able to trust the assessment of the experts. But this legitimacy is even more important for the party being assessed: people must know they are being treated fairly, they have to be able to trust the assessment of the experts and, moreover, have the feeling that the implicit or explicit recommendations set down in the panel report are to the point.
Certainly when quality assurance agencies have to make momentous decisions about the existing or absent quality of a programme or institution, the legitimacy of the panels must be above reproach. This applies even more strongly if sanctions are imposed on the basis of the decisions of quality assurance agencies, sanctions that may have consequences for funding, the right to award degrees, or the enrolment of new students.

The panels already bear what is in itself a heavy responsibility. If, on top of this, the reports of the panels were to be published, this would demand the careful supervision and training of the panels and their secretaries. In general, the panel reports are sufficiently diligent and balanced. The panels are aware that they must comply with criteria, and in addition they are usually quite capable of finding the right balance between assessing the basic quality to be evaluated and making recommendations for improving the quality of the programmes and institutions. Diligence and the balance between assessments and recommendations ensure the legitimacy of the panels and the large majority of our assessment procedures do not seem to present any problems in this respect. But we have to remain alert! The publication of the reports may have a number of possible effects that influence the behaviour of the panels. I would like to mention three of these:

a. The "macho" panel
It is imaginable that panels will adopt an extra forceful attitude to show that they "dare". For example, in the Dutch context we have experienced that extremely authoritative professors now desire to be involved as panel members in the accreditation system, while they were unwilling to do so under the "softer" assessment system. The reason given was literally that the fact that their assessments could have an impact would make their efforts worthwhile. In their eyes, the former assessment system was too non-committal and this repelled this very powerful and authoritative group more than the possible public consequences of their assessments in the accreditation system.
We should immediately heed the warning here. The game has to be played well and fairly. In Flanders, the chair of a panel was recently threatened with legal proceedings by an assessed professor after the panel had made such critical comments about the weight of a particular subject in the programme that the programme - in accordance with the Flemish basic rules regarding accreditation applications - could not be accredited; the situation became even more threatening when the institution involved also appeared to be going to instigate legal proceedings against the panel. If that would have happened, the legitimacy of a system with peer reviews would be gone. However, to me, the risk that a much more procedurally oriented assessment system will be imposed seems greater than the risk that that a government would be satisfied with a "softer" evaluation system.

b. The "old school tie" panel
A very different effect of the publication of their reports occurs in panels that allow their own role during an assessment to evolve from that of assessor into that of advisor. You can often read of such struggles in panel reports: the arguments to grant an assessment of "satisfactory" are not supported by the facts observed by the panel but rather by the willingness of the assessed colleagues to make amendments, implement or change policy, or to improve behaviour. The assessment of an actual situation is then converted into an "assessment of a plan", which is of course something else entirely. This type of procedure can often be tracked back to the personalities of the panel members: there is mutual respect and in the worst cases it is an example of the non-intervention principle: "this panel will not be too hard on you if you won't be too hard on us later". The formal, proper independence of a panel cannot resolve this type of behaviour. However, it could often help if true outsiders were added to the panels, especially if they were to come from abroad. But even then, such behaviour cannot always be prevented, certainly not in the smaller disciplines or specialisations.

c. "Veiled" language
In assessment practice, you can see with a certain regularity that during site visits, panels are more forceful than they are later in writing. In itself this is understandable because the written report has to provide careful assessments and lines of argument. This often involves a certain degree of caution because an assessment is seldom black or white or yes or no. Usually, assessments are shades of grey and demand the weighing of various views before a judgement can be made.
NVAO was confronted with this in the earliest phase of its existence when it chose to be highly pro-active in the assessment of the panel reports, and negative comments and statements became blown up out of proportion as it were, whereby the positive assessment of the panel was brought into doubt. This had in part to do with a learning curve. In an accreditation system, the assessments of a panel obviously have to be stronger as well as better substantiated than in a system aimed at improvements. Like a master and as the body ultimately responsible for the assessments, NVAO therefore had to adopt an accurate and exact position. This position, however, appeared to have the reverse effect on the quality of the reports because panels became reluctant to include negative comments in their reports. It took a long time before the panels dared to do so again. Only when NVAO had publicly admitted it had "gone too far" in filtering the reports did the panels have enough confidence to again base their "public" considerations in the reports on positive as well as less positive findings. Incidentally, in the Netherlands, there are still institutions that ask panels to send their recommendations as non-public management letters to avoid a situation in which NVAO would not follow a positive assessment of a panel or to prevent competitor institutions and programmes from receiving too much information.

5. Conclusion

The publishing of information regarding the quality of institutions and programmes is not a matter of course. As we have seen, there are different perspectives from which the importance of publication can be viewed. Moreover, publication can have an effect on the procedures of the panels. It is therefore understandable that a lot of thought is being given to the advantages and disadvantages and the possible positive and negative effects of publication. Furthermore, the political culture and particularly the prevailing views about the position of the state play an important role in the shaping of a broad or reserved publication policy.

It is therefore not easy to draw a conclusion from what I have presented to you this morning, but I will try to do so nevertheless. To this end, in the first place I will choose for the perspective of the higher education institutions themselves. And I will assume that we are dealing with institutions with certain ambitions: institutions that have thought about and continue to think about their mission, that have based their vision and goals on this mission, and are continuing to develop it further. Such institutions offer something to both society and students: a curriculum with a particular content that may also have been developed on the basis of an educational model. Such institutions promise something to the labour market and to their students and on the basis of this, they try to attract students and the interest of the labour market. They are actively engaged in marketing and public relations. Quality assurance agencies often assess whether an institution or a programme delivers on its promises at an internationally acceptable level for a bachelor's or master's degree programme. Promises must be kept: if an institution or a programme is found not to deliver on its promises, this should be made known. And this is an argument for as great as possible transparency in decisions. That is my first position.

My second approach is from the perspective of what I believe to be the necessary internationalisation of the European Higher Education Area. Most European countries, education institutions, students and a large proportion of the economically most important branches of industry benefit from high student mobility, expressed in a considerable amount of foreign experience. If this viewpoint is shared, it is evident that sufficient and reliable information should be available about the quality of institutions and programmes. Ideally, this information would be provided by the institutions themselves. But precisely for the sake of objectivity and to prevent one-sided marketing and public relations, I believe that the assessments of quality assurance agencies constitute an additional guarantee.

I am well aware that I am a product of my own political system and that from that perspective I probably stress the benefits of the publication of independent assessments more strongly than the disadvantages, but I am also aware that I am consistent. As a former chair of a university (Maastricht University) that has chosen an educational approach that is entirely its own (Problem-Based Learning) and that has moreover chosen for strong internationalisation with full conviction, I have always been a supporter of as much transparency as possible about quality assessments. This position obliged my university to be constantly alert to possible drops in quality and to always strive to deliver on our promises. And this consistent attitude is ultimately the most successful!
That is why I am convinced that the publication of decisions and the reports of the expert panels will ultimately have a positive effect on the quality of institutions and programmes.

Thank you for your attention!

Dutch version

Terug