Analysing assessment: the OECD report

A good foundation for evaluation and assessment is in place but risks being distorted by transfer tests. Peter Cheney summarises the OECD review and the progress made over the last year.

Northern Ireland has a robust system for measuring pupils’ progress, according to a major OECD review, but the unregulated transfer tests are creating unnecessary stress and duplication.

Based in Paris, the Organisation for Economic Co-operation and Development comprises 35 member states and has a broad interest in improving policy. The review does not put forward an official OECD line and instead has three stated purposes: provide insights and advice to the Northern Ireland authorities; help other OECD countries understand Northern Ireland’s approach; and improve international knowledge on evaluation and assessment policies.

The review was written by Claire Shewbridge, Marian Hulshof, Deborah Nusche and Lars Stenius Staehr. Shewbridge and Nusche work in the OECD’s secretariat. Hulshof manages the R&D programme within the Dutch Inspectorate of Education. Staehr is a project manager at Danish healthcare company Novo Nordisk and a testing consultant for Denmark’s Ministry of Education. They visited Northern Ireland in February and March 2013 and published their report last November.

Five aspects of the system were considered:

1. the evaluation and assessment framework;

2. student assessment;

3. teacher appraisal;

4. school evaluation; and

5. system evaluation.

The overall framework is focused on improving how students learn and the outcomes of that learning. It also recognises the importance of improving results for pupils from poorer backgrounds and has the potential to improve the quality of support offered to schools.

However, the report also found some examples of duplication and inconsistency. There is a “rich documentation” of pupil progress at primary level but many post-primary schools do not use this information once a pupil has been accepted.

A “polarised political debate” over post-primary transfer continued to hold up the effective implementation of pupil assessment. “This is penalising pupils and is reportedly creating unnecessary stress and duplication of work for teachers in many primary schools,” the authors commented.

Implementation can also be “tokenistic” as teachers have reservations about the level of progression used for assessment. Schools like the “functionality” of commercial tests “although these may not be aligned to curriculum” and narrow the focus of teaching.

Student assessment procedures – up to and including GCSEs – have a “strong official focus” on formative assessment and teachers’ professional judgement. Communication with parents is good. Information systems for tracking a student’s progress are used effectively in many cases.

That said, there was a tension between the two purposes of teacher assessments of students at the end of each key stage. The first is to document student learning progress and inform teaching and learning in the classroom. The second is to measure the school system’s performance i.e. the literacy and numeracy element.

Some educationalists warned that the collection of information for school performance would compromise the reliability of teachers’ judgements. There would be an understandable focus on driving up the school’s literacy and numeracy results, perhaps at the expense of the pupil’s individual progression.

A level of progression can represent up to two years of learning. Some teachers feel that the gaps between the levels were “too large to be used effectively for formative assessment.” Teachers were also frustrated by the technical faults arising from computer-based assessments and thought that “pupils would not be able to demonstrate their full ability” when tested in this way.

Transfer pressure

Primary schools reported “pressure from parents to ignore official policy and spend teaching time on preparing their pupils for the unregulated transfer tests” thus disrupting the curriculum. Many parents did not support academic selection but felt that this was the only way that they could get their child into a high performing school.

Many pupils sit both transfer tests – to ensure a wider range of options – and children were spending “up to four weekends in unfamiliar environments to take these tests.” Many parents were more interested in the transfer test results than in the key stage assessments which have measured their child’s progress through primary school.

Teachers are seen as respected and trusted professionals. Their appraisal system is “comprehensive and thoughtfully designed”, based on the merit principle and “firmly rooted in classroom observations”.

However, the competence standards are not consistently used, the process “may not focus sufficiently” on individual needs, and there is no external validation. Teachers and principals also felt that there were not enough opportunities for professional development.

Departmental policy promoted “effective self-evaluation” of schools. External evaluations were “broad and based in quality assurance” and focused on improvement. The ability to self-evaluate varied among school leaders and boards of governors and teachers were nervous about the implications of formal intervention – “too much focus on measurement, labelling and ultimate threat of closure.”

Context is always important but is often ignored. The authors noted that the Belfast Telegraph publishes unofficial school league tables but does not clearly explain the factors affecting school performance. This approach “may lead to misinterpretation” and may result in pupils from disadvantaged backgrounds not being accepted by certain schools.

At a system level, the department holds schools to account and expects an improvement in performance by pupils from disadvantaged backgrounds.

The province has also started to participate in international comparative studies (TIMSS and PIRLS) and researchers conduct periodic qualitative surveys e.g. on bullying.

There was, though, an “urgent need” to build trust in the new assessment system among teachers and the “reporting burden” for schools also needed to be minimised. Performance targets only use a narrow set of measures.

DUP education spokesman Mervyn Storey reaffirmed his call for a single official transfer test system: “This would lessen the need and uptake of private tuition as all pupils would have equal access in the classroom, which is the most appropriate forum for preparation to take place.” However, he did not propose how many tests should be carried out and said that the details should be drawn up by educationalists.

The OECD report is available at www.deni.gov.uk/assessment

Executive response

A Department of Education spokesman told agendaNi that the report “will inform the ongoing development and refinement of a broad range of education policies” affecting pupil assessment, teacher appraisal, school leader appraisal, school evaluation and system evaluation and how these help to improve pupils’ learning.

Minister O’Dowd has set up a programme board inside the department which will work on the various parts of the education system on the different aspects of the report. The spokesman emphasised: “It is not a report just for the Minister and department but for all involved in children’s education.”