George Bush famously asked, ‘Is our children learning?’. That’s also the question by Uwezo, a coalition of NGOs working in Kenya, Uganda and Tanzania. Their report published today makes dismal reading about the quality of schools.
First, a word about the report. This is not a study by the World Bank, or a group of donors. It is a study by Uwezo, an East African initiative hosted by three NGO networks: TEN/MET in Tanzania, WERK in Kenya and UNNGOF in Uganda, with overall quality assurance and management support from Twaweza. They conducted their own survey (standardized across the countries) to test the literacy and numeracy of more than 100,000 children, the largest ever survey of its kind in the region. When citizens themselves are telling us about whether their public services work, we should be paying attention.
There has been a remarkable increase in the number of children in school in East Africa since Uganda reintroduced free primary education in 1997 (followed a few years later by Tanzania and Kenya). Over the ten years from 1999 to 2009 net primary school enrolment has risen in Kenya from 62% to 83%, and in Tanzania from 49% to 96%.
But the Uwezo report finds that the quality of education that those children receive is ‘very poor’. According to the Uwezo tests, most of the children in Standard 3 had not reached the Standard 2 levels of literacy and numeracy. Only by the time they reach Standard 7 are most children able to read and write at the levels expected in Standard 2.
Kenya’s pupils did best, followed by Tanzania and then Uganda, and there are large variations within as well as between countries. The report has interesting things to say about the apparent reasons for the differences:
an important finding given the enormous resources invested in recent years in improving school infrastructure, was that school quality was weakly associated with literacy and numeracy levels. Children in areas with better school infrastructure did not perform better than in lower quality schools or more crowded classrooms.
The report finds grounds for optimism in the variations within countries; the fact that some places do better than others suggests that improvements are possible within those contexts. The report speculates that quality is driven by non-observed factors such as the quality of teaching, practical accountability and teacher motivation.
The poor quality of education, despite considerable investment in school infrastructure, may perhaps be the result of what Lant Pritchett, Michael Woolcock and Matt Andrews call “isomporphic mimicry”. The idea is borrowed from sociologists of organisations, and it describes the way in which organisations can sustain their legitimacy by the adoption of the forms of effective organisations in a way which camouflages a persistent lack of functionality. This behavour may be partly a consequence of the way that donors work to increase service delivery in developing countries. Pritchett et al observe:
Development agencies, both multi-lateral and bi-lateral, have very strong tendencies towards promoting isomorphic mimicry— encouraging governments to adopt the right policies and organization charts and to pursue best practice reforms—without actually creating the conditions in which true novelty can emerge, be evaluated, and scaled.
What can be done to improve education in Kenya, Tanzania and Uganda? The Uwezo report says:
Differences in performance among districts within each of the three countries, and between public and private schools, suggest that certain schools have ‘figured out’ how to achieve better results within existing constraints. Investigating why certain districts, and within districts certain schools, do so much better than others could provide important clues about what matters most for improved learning.
The report also suggests that Cash on Delivery aid may be a good way to create the incentives to improve quality:
One such approach, which is still untested but whose design has been informed by careful review of the evidence of what works, is called Cash on Delivery. The core idea here is that instead of funding inputs, a mechanism is created by which payments are made against the achievement of a specified and independently verified outcome, such as $50 per student who completes Standard 2 with 80% literacy and numeracy competencies. This approach has been original designed for improving the effectiveness of aid given to national governments, but the approach may be even more useful for how national governments create incentives to improve performance at district and school levels.
(By the way, I’m on the board of Twaweza, though I can claim no credit for this report.)
21 responses to “In school not learning”
[…] was intrigued to come across the term ‘isomorphic mimicry’ (thanks to the reference in Owen Abroad), used by Lant Pritchett to explain the process of “replicating existing forms and processes […]
I think improving education is also about looking at the bigger picture, particularly labour market opportunities for teachers and students alike.
I sat in three urban Zambian classrooms for three months last year. There was a big problem of low commitment and absenteeism on the part of teachers. Out of a scheduled 6 lessons, students usually only had 2 lessons.
In Zambia’s Copperbelt, there are very few jobs. So many people go into teaching, without a passion for teaching. And, because of low pay many concentrate on other sources of income generation. As one student explained:
‘you know these teachers it’s not like ‘I want to be a teacher when I grow up’, it’s just circumstances that force them to become teachers. Like maybe you didn’t have a lot of money to pay for a course that you’ve always wanted to do, so they pay for teaching, that is a lesser cost. So when they go into teaching, now that they’ve got their own money, you know what they will do, they will start going to school again, since they are already on payroll. Some have a business, some have gone back to school, like maybe they wanted to be an accountant. So when you are doing two things at one time you either concentrate more on the other thing and less on the other, so they concentrate more on their business’.
Similarly for the students, limited labour market opportunities seem to have thwarted commitment to their own schooling. What’s the point of focusing on books (often perceived as boring, and requiring a long-term focus in an insecure environment), when you can get money right now through having boyfriends? Living in a Copperbelt township, where most sources of income were being a domestic worker, selling at the market, or working as a mechanic or casual labourer in the mine. There were few role models inspiring young people, showing them what was possible with education.
Improving incentives through ‘cash on delivery’ aid might help. But I’m not sure if it can get students more interested in education in a context where employment is not guaranteed through education.
Also, I’m concerned about corruption. I previously worked in The Gambia, where teachers routinely bought exam papers, so they could teach to them, ensure their classes do better and thereby protect their own jobs, in an tricky job market. If the school exam results really matter, as would be the case under cash on delivery, I worry this practice might increase.
Those sound depressingly like the stats you get from public education in the US.
I think Alice makes some excellent points, reflecting a number of my own thoughts on this subject – this includes her points about Cash on Delivery of Aid.
If I’m honest I do sometimes struggle with the fact that Education and Health systems get so much support from national governments – I think this stems from a concern that the countries involved are cajoled into spending all this money on these sectors because its what they believe the international donors want them to spend it on. I am not against spending on these sectors per se but sometimes wonder whether these systems are being put in place before there is an absolute need for them (i.e. if children go to school they will get better jobs – this only works if there are better jobs to be had or if economic growth is occuring). There is also my ongoing concern about how sustainable these systems are – we all know that none of these countries are in a position to fund their ongoing administrative running costs so shouldn’t we be trying to keep overall spends on these sector to more manageable levels i.e. certain percentage of GDP? Additionally would it not be better to redirect some of the funds going into education and health programmes to programmes designed to increase the tax base or improve tax collection systems – or indeed for that matter into programmes that are designed to stimulate economic growth? My personal view on all this is that in order for education and health systems to be really valued and of sufficiently high quality you need to generate a significant number of ‘middle class’ aspirational citizens who will in fact drive and demand the development and provision of better schools and health care – this internally nationally driven development will ultimately be much more sustainable than the currently rather international driven way we approach these sectors. It seems to me that if you put comprehensive or improved health and education systems in place prior to the formation of these middle classes you could, as evidenced by the Uwezo report, you could end up supporting an ineffective and inappropriate use of aid. Of course as with all things its about finding an appropriate balance and treading a fine line with regards to the question about what comes first the chicken or the egg. However I suspect that in the case of schools and clinics its traditionally been much easier for donors to fund these than to fund SME development programmes or Agricultural service programmes or work on attracting private sector investment into a country. The result is that we have an inbalance of aid flows and its ultimately hurting as opposed to helping – sometimes less is more.
Moving back to Owen’s post the thing I didn’t get a feel for was what were the concrete suggestions from the report to turn around or improve upon this rather disappointing state of affairs. The report would appear to have done an excellent and useful job on evaluating and highlighting these issues but I guess I’d like to know more about those areas that were having success and exactly how this had been brought about – was it for example that those areas that were having success were located in areas where economic growth was slso occuring and where communities had an increased interest in the quality and provision of schooling?
Yes, I think the points made are spot on, but what can we do? Having worked as an Educational Psychologist in the UK and in Australia, I could never understand how we seemed to know how to get almost all children reading with skill and interest, so that if you had a project to do that you could, but it succeeded only within that project. It is a strange contradiction in that it is not about money, or even great expertise, it is about belief, it is about passion, and commitment to succeed and then following any one of a number of clear step by step processes to achieve the goals. My gut feel would be to set up a programme which targeted a range of schools and paid them by results, with challenging targets but very solid payment as investment processes. So the target should be really 99% but promoted as 100%. And the payment? Well, a creative payment system would be to get the teachers to say what they need to be paid to achieve those results! And a team of ‘motivators’ would be needed to go round the schools, making sure they start on target, that they have the resources they need and that the key elements are in place, which are belief, commitment, energy, passion.
I could tell a few stories about this, having been involved in Adult Literacy and Numeracy and in generating resources for learning to read and write. But I will just mention one. One school I covered I visited not too often. One day in the Head Teacher’s room I was looking at the reading skills chart which was on the wall for every class. I noticed that from 9 years onwards every child could read (had a reading age over 8).
I asked the Head how they did this, noting that they never referred reading problems to me. She said, if they cannot read by 8 then I teach them! And this was not about techniques, it was about total faith everyone had, children, parents, and other teachers, that every child would learn to read properly, real books with interest, by the time they were nine. The Head hardly had to do any teaching, not for long anyway, magic happened.
This is completely unsurprising, and it’s is systematic of an international education system which confuses schooling with actual learning. The more of these studies emerge the better, especially given the rhetoric surrounding the MDG goal, and the enormous success in getting kids into school. This, and the ASER studies in South Asia (Uwezo is a spin-off of the amazing work Rukmini Banerji and collegues have been doing at the ASER centre in India (disclaimer: I do work with them occassionally)) highlight that even when kids are in school, they are not learning. The question is why this is the case?
There have been numerous studies evaluating the link between resources and outcomes, which invariably show that inputs don’t matter. Even when they do matter, the sheer gulf in learning between developing countries and OECD dwarfs any gains from an input based approach. See Lant Pritchetts new book (preliminary draft on his website) for a more eloquent exposition of this argument. The evidence from RCT’s etc, show that it is the governance of schooling which matters (contract teachers, performance related pay (with caveats), monitoring).
The key differences between schools which work and the schools which don’t are teachers, but not in terms of inputs such as PTR’s or formal qualifications. It is the teaching process which drives differences. At a simple level, this is highlighted by absence and time-on-task (which are very high and low respectively). At a more complex level it is the lack of accountability or career structure (wages are linked to experience, not performance).
The overwhelming share of education finance is allocated to teacher based inputs (such as lower PTR or higher salaries), encompassing 90% + of all spending in *most* countries. This is inconsistent with a system which maximises the marginal inputs and costs across all inputs. It is (as Filmer and Pritchett argue in their 1997 paper), consistent with a system whereby teacher utility also enters the maximising function. Where moves to reform the system are met with hostility by the teacher unions and politicians, no increase in inputs will change the status quo.
In many developing countries the political economy of the sector is simply not acknowledged. Governments take a very broad, input based approach to quality, which is fueled by expansions in EMIS systems. This is useful in one sense, but also may be used to mask issues if the prevailing view is inputs allow for enrollments ; enrollments = schooling ; schooling = learning. The last link is broken. Whether or not it is feasible to fix it is the true question.
Owen, if you are ever in london we should go for a pint. Or an orange juice in case you have a 50 mile run or something. You should be able to get my e-mail from this post (if not i’ll e-mail it).
My word.
You mean government supplied services aren’t very good?
Shocker!
Never seen it happen anywhere else, have we?
@Tim – I would say it is less a question of ownership (government, private sector etc) and more a question of accountability and whether there is pressure to do better. Private sector monopolies are not much good either.
[…] This post originally appeared on Owen Abroad. […]
[…] you’ve got the kids into school. What happens if they don’t learn much? Owen Barder discusses a new report on education quality in East Africa, written and published (for once) by East […]
[…] donor focus dictates policy focus, and focussing strictly on enrolment means that students are in school, but not learning. The second because teaching is a lucrative career that includes a nice steady paycheque and […]
Interesting post.
Forgive me for taking this discussion from an international context to a domestic US one, but I was struck by the fact that the Uwezo report mentioned cash on delivery as a solution for under-performing schools. The following is a very superficial analysis of the proposal for cash on delivery in education, but it is something I was wondering. At a VERY basic level, isn’t the premise of No Child Left Behind (i.e., the receipt of funding based on outcomes) quite similar to cash on delivery in the education system of countries abroad?
Critics have vilified NCLB–and not just the high-stakes testing part–and pointed out how it grossly increases inequality. So I guess my question is would this be different in a developing country setting?
Could this mean that our traditional model of “going to school” for education be ineffective in Africa? Is there another way to structure education in different parts of the world to make education more effective? Or is this just another result of a poor development project and lack of access to quality resources?
Ultimately I think this plays back to the age-old adage “It is qulaity, not quantity that matters.” If education improvements had originally been measured with the results of this standardized test rather than enrollment rates would the system failure have been realized sooner? Would anything have been done about it? Can you actually measure things such as “quality of teaching, practical accountability and teacher motivation” be measured?
I don’t mean to be pedantic, but I think the performance order should be Kenya, Uganda and then Tanzania, as Uganda performed better than Tanzania on most indicators.
That being said, I’ve been working with Ugandan primary schools for the past three years and the poor quality of primary education definitely needs to be addressed. Uwezo is a great initiative.
Though the report admits its results offer “limited guidance,” it does suggest trying “new approaches, with rigorous experimental designs, would be one way to elicit how to improve learning in schools.”
There are any number of possible interventions which could address this issue. I agree that school infrastructure is relatively ineffective in most cases, but I don’t think we should conclude “inputs don’t matter.” In Uganda language policy is extremely important. Local languages are supposed to be the medium of instruction for the first three years of primary school and then in P4 English transitions from being a subject to becoming the medium of instruction.
Very few local languages have teaching materials and none of the teachers’ colleges include any instruction as to how to teach in local languages. RTI and Makerere University recently did an Early Grade Reading Assessment which had similar findings to Uwezo. They suggest increasing local language materials and training could greatly improve reading levels in primary schools.
This is just one suggestion. Contract teachers, in-service teacher training, performance based pay … there are many possibilities, to implement with rigorous evaluation (though I’m biased as I’m working on an RCT.)
One final comment is to not forget about drop out rates. Though enrollment rates in early primary are very high, the majority of children (in Uganda at least) drop out before completing Primary 7. It might be interesting to see if this changes when quality improves (though my experiences identifies financial concerns as the determining factor.)
[…] and Uganda report by local NGO coalition (not the World Bank this time). Commented by Owen Barder here. 3. State of taxation in the Philippines featured in The Economist. 4. Of intelligence services, […]
[…] http://www.foreignpolicy.com/articles/2011/07/18/the_price_is_right In school not learning – https://owen.org/blog/4715 Can aid work? – https://owen.org/blog/4738 Make learning part of your brand – […]
Excellent analysis of Uwezo’s report. I’m working in the Northern Region of Ghana in Education right now with Engineers Without Borders Canada (though my opinions don’t necessarily reflect theirs) on Evidence-Based Decision-Making and most of what is in the report and what you’re saying here is relevant in this country, if not in all of West Africa.
Access to education is such a big focus in Ghana that data to calculate indicators of quality isn’t even collected through the Education Management Information System (EMIS)(donors include USAID, DFID, and UNICEF). As a result, it’s hard to immediately tell how Ghana’s students are faring, even at the basic level. There is also a perception that pupil-teacher ratio (PTR) is one of the most important indicators in terms of describing quality of education, despite the fact that the data collected over the years reveals absolutely no correlation between performance on a standardized exam and PTR. i.e. it doesn’t matter how many students a teacher has; if s/he is good, then the students will learn.
I spent a week as a student at one of the high schools here and it really reinforced many of the things that Paul Wigan mentioned above. Teachers were routinely absent and nobody was there to hold them accountable to their posts. The internal accountability system at the school was ineffective and the government accountability structure in the area is plagued with corruption. I sat with the students for two whole days of class, waiting for a teacher to show up. They were definitely in school, but not learning.
In this case, management really is the issue, but I don’t think COD is the silver bullet solution. One of my colleagues at the Regional Education Office pointed out that motivation is driven by cash, but that at the same time pay ought to be related to performance. It’s a chicken-and-egg problem: do you pay teachers more to motivate them to do better, or tell them that you’ll pay them more if they perform better, knowing that they’re satisfied with the low pay they already get?
Beyond that, a government needs resources in order to pay based on performance. Each teacher needs to get a basic salary (see point about motivation above), but if you’re also going to scale pay based on performance, then you’ve got to have the money to pay out to awesome teachers. Ghana already spends the greatest proportion of its GDP on Education and the budget is tight; there’s no room to pay teachers more. Also, in Ghana at least, teachers don’t last more than 5-10 years on average. After that, they usually have saved enough to go back to school to get a better career. Thus, scaling pay based on time served is essentially meaningless since nobody is there long enough to get a significant raise in pay. Shifting the pay scaling policies won’t have any effect in this country.
There’s another big hole in the COD plot: how do you measure teacher performance? The easiest way is to look at student performance, but this isn’t a rigorous method because there are many ways for teachers to inflate student performance in order to get paid more once they’ve realized that better student performance means higher pay for them.
Okay, so then you assess the teachers, right? Well, who does the assessment? If the management is already broken, how can you trust it to evaluate its employees objectively? Besides that, where’s the money to run a country-wide teacher evaluation program? Is a donor going to run that project? For how long? I really love the idea of assessing teachers, but it’s a very complex task to undertake and I think the idea of COD alone ignores that complexity.
In my opinion, a better solution is one that addresses the issue holistically. First, we need to realize that teachers work in incredibly demotivating environments. That needs to change! Broken management needs to be addressed before teacher performance can even be considered. I’m not a management consultant, so I don’t really know what that would look like, unfortunately. I imagine, though, that it would involve a lot of reciprocal accountability AND support from school administrations to teachers.
I don’t know about the attitude around teaching in the countries listed, but in Ghana it is seen as a pretty nice job with nearly infinite job security, steady (if low) pay, and no accountability for performance. It isn’t seen as a profession and few take it seriously. That, too has to change. It should be difficult to get into the teaching profession and all teachers should be paid accordingly. I come from a country where teachers don’t get paid enough, so it feels somewhat hypocritical to make that statement. Still, it’s true regardless of your country: teachers generally ought to be paid more than they are.
Also, when teachers aren’t doing well, the management should be able to fire them. Doctors and lawyers are fired all the time for not doing their jobs, so why not teachers? In Ghana at least, there are many educated individuals who are qualified to teach and who are passionate about impacting the lives of children positively, but there aren’t often any open spaces because it’s nearly impossible to fire a teacher here. Assessing the teacher’s performance doesn’t even factor in at this stage; if Ghana were only to fire all of its chronically absent and late teachers, I bet it would get rid of 20-30% of teaching staff. A teacher’s performance doesn’t matter, after all, if s/he is sitting at home instead of teaching.
Finally, I think the “education is the key to a good life” sensitization rhetoric that comes from nearly every NGO involved in education needs to be turned on its head. Education is NOT the key to a good life, but rather one thing that increases one’s chances at getting there. Being smart, working hard to learn, and being entrepreneurial are much more important factors contributing to getting out of poverty. It’s paternal to tell families in a rural village that education = a good life. They deserve to know the truth; that 6/10 of them won’t make it past the basic level (stats from Northern Region in Ghana). Teach parents instead to question the education their children are getting, and I believe they can be great contributors to solving the issue of quality education.
I know my contribution is about two weeks late, but I hope it gets heard.
-C
Hi Owen, given your interest in CoD, I thought you might find this evaluation of a new project in New York schools interesting:
http://blogs.worldbank.org/impactevaluations/node/610
Alice
@Alice
Sorry for the delay in replying. Thanks for the link to the summary findings showing that paying teachers bonuses did not, in the New York context, improve outcomes.
I don’t want to make assumptions about your views, but I think you are perhaps suggesting that this counts against the idea of Cash on Delivery aid; whereas I think this counts in its favour.
Cash on Delivery makes no claim about, and no assumption about, the value of using teacher bonuses to improve education. Under Cash on Delivery the payments by donors are paid to the government, not to schools or teachers. It is for governments to decide (and experiment and learn) to find out what works in their context to improve education. Perhaps it is providing de-worming tablets. Perhaps it is tackling teacher absenteeism. Perhaps it is latrines for girls, or more text books. Perhaps it is some combination of these and other interventions. Governments can, if they wish, experiment with paying bonuses to teachers and schools. I’m willing to believe that this might work in some contexts, even if it does not work in New York. (Cash on Delivery aid makes this possible – because governments can enter into back to back commitments; but this approach is in no way a requirement.)
The evidence you quote demonstrates that we often don’t know what will work. New York was smart to test the idea of bonus payments, and (especially) smart to cancel the programme when it proved to be unsuccessful. We don’t do enough of that in aid. Donors come along with ideas about what will work, and then write that into logframes for long term programmes. With Cash on Delivery aid, by contrast, it is open to the government to decide for itself what to do (drawing on expertise of donors and others when and how it chooses), and it can experiment as New York has done to find out what works in its context.
That’s why I think the evidence you cite suggests that Cash on Delivery Aid is a good idea: because it shows the benefits of trying ideas and learning about what works. Cash on Delivery aid is not based on the idea that making bonus payments to schools or teachers will improve education: it is based on the idea that donors should provide finance to countries in a way which is flexible enough for countries to be able to manage their public services in a way which they believe will work in their context, including the ability to test ideas, and scale up their successes.
Owen
[…] Dedicated and talented teachers who have the knowledge, skills, and passion to support and encourage their students are some of the most monumental game-changers in education and development, and yet they work in an environment bereft of accountability and lack the resources to help them do their jobs well. Not only does this put teachers at a disadvantage, it means students are essentially abandoned in classrooms. Both parties are in school, not learning. […]
[…] Dedicated and talented teachers who have the knowledge, skills, and passion to support and encourage their students are some of the most monumental game-changers in education and development, and yet they work in an environment bereft of accountability and lack the resources to help them do their jobs well. Not only does this put teachers at a disadvantage, it means students are essentially abandoned in classrooms. Both parties arein school, not learning. […]