Appendix B. How I ended up in a scientific spat about migration figures and what I learned from it

By Maite Vermeulen

Note: This is an abridged extract of the original article published in the now defunct publication The Correspondent. The full text can still be accessed here.

I have to tell you how the debunking of an important theory about migration was itself debunked. You probably had to read that sentence twice, and I get that…I learned a lot from this experience. About how science works, and how we as journalists contend with that. About what expertise actually is, and why it is so limited. And about certainty, doubt and being right. So buckle in and brace yourself for a story about that time I said I was wrong – and turned out to be mistaken.

How it all started: the migration hump

It all started a few months ago when I read a new study about the migration hump. I was immediately interested, since “the hump” is a well-known, very influential theory about the relationship between migration and development. Basically, the theory states that as poor countries become richer, outward migration increases rather than decreases. This may seem counter-intuitive: we might expect that when countries get richer, reasons to leave will diminish because life there is better now, right? But the migration hump shows that this is only the case above a certain income level, starting from about USD 7,000 to USD 10,000 per person per year.

Many poor countries are a long way away from that, which means that economic development in those countries will lead to more migration, not less. That’s because migration costs money, and when people who were previously very poor have some, they are more likely to leave. Come up with a graph comparing income and emigration, and you’ll see a more or less hill-shaped curve showing the lowest rate of emigration in poor countries, the highest rates in middle income countries, and falling rates for rich countries: the migration hump.

I frequently reference the migration hump in my articles, especially to criticize European migration policy. And there’s a reason for that: the European Union is spending more and more money on development aid to reduce migration. But the migration hump shows that this policy is based on a misconception: if more aid leads to more development in poor countries, that funding will cause net migration to increase, not decrease. And then that new study came across my desk, released under the MEDAM research project. The researchers were quite blunt: their analysis of migration data showed that the migration hump was an oversimplification. In actual fact, their models produced opposite results. They calculated that when a poor country becomes richer, emigration to rich countries goes down. Their explanation was that their method was different: instead of comparing emigration in poor and rich countries, they compared countries with themselves, over time. Why? Because a comparison between poor and rich countries overlooks the differences between those countries: differences that can affect income as well as migration.

I had colleagues and migration experts with more knowledge of econometrics take a look at the new paper; I spoke to the researchers, and then decided to write an update. The research looked convincing, and I wanted to hold myself accountable, because a theory I had often cited in my pieces did not seem to hold up. I thought that was the end of my hump saga. But then I was tagged in a Twitter thread by Michael Clemens, a leading development economist at the Center for Global Development. The new research, he tweeted, was based on a statistical error.

Clemens and his calculations

There was nothing wrong with my article as such, Clemens told me in a private message. “The problem is with the research itself.” All very friendly, of course. But I wasn’t so sure. Could I have seen this coming? Should I have done something differently? What could I learn from this?

I took another in-depth look at the paper, and delved into Clemens’s criticism. I looked at his charts, tables, formulas. The only slight problem was I didn’t understand any of it. This wasn’t really all that strange, because Clemens’s criticism targets researchers’ statistical methods. If you don’t have a degree in econometrics, the analysis is almost impossible to follow. In fact, it’s almost impossible for people who have studied advanced statistics. My colleague Sanne Blauw – with a PhD in econometrics – called me after spending three hours analysing both papers: “I think I more or less understand Clemens’s criticism.”

I asked more experts for assistance: professors and PhD students who could explain the statistics to me, who had experience with time series and cross-sectional panel data, who knew more about spurious regressions and nonstationary variables. I had long phone calls with Michael Clemens and Claas Schneiderheinze, one of the researchers who authored the original MEDAM paper. I can’t say I’ve completely mastered the maths. But here’s what I now understand of the discussion.

What I learned from this

Whether or not this paper is based on a statistical error (this discussion will probably be settled in academic journals in the next few months), all this commotion makes me wonder about my relationship with science as a journalist: what it is – or what it should be. Every single person – including a journalist – has a limited framework that shapes their ability to understand something. I went to university, but I never took advanced statistics. Nor do I understand topics like the nitrogen cycle, Japanese grammar or the mathematics behind climate models. There is simply so much more that we don’t know than what we do.

Sometimes that doesn’t matter. I don’t have to understand Newton to say something meaningful about poverty alleviation. But often it does matter, even if we don’t realize it. As journalists, when our own knowledge and skills fall short, we rely on experts to fill in the gaps. But even for those experts, what they don’t know extends far beyond what they do know. Especially when it comes to statistics. Many biologists, medical professionals, psychologists, economists or social scientists hire specialized colleagues to run their statistical analyses. And those specialists design models that are so complicated that only a handful of people can really understand them, or provide critical commentary. The mathematical calculations behind the models are so far removed from reality that results pop out like a rabbit out of a top hat: we have no idea how it works, but the outcome is self-evident.

Who knows how the statistical stage magic actually works? We can draw an obvious parallel with the epidemiological models being used to predict the course of the coronavirus pandemic: who has any idea exactly how those models work?

And that’s how a journalist – or policymaker – can end up in a tricky situation when two experts are making contradictory claims. Can you place two non-stationary variables on one side of a panel data regression without losing the long-term trend? Yes you can; no you can’t! How on earth can a journalist possibly figure out who is right? The only solution seems to be cumulative knowledge: asking all the smart people you can find to give it their best shot too. At its very best, that’s how science should work.

And when that happens, it often turns out not to be about what’s true or false. Instead, it’s about which question we want to answer. The MEDAM paper answers an interesting question – just not the question of whether or not the migration hump holds true. And maybe the researchers subconsciously fell into a pitfall that science has created for itself: contentious studies that debunk something major are considered more prestigious than studies that confirm the prevailing assumptions. Just think about it: this was a study that I (a journalist) decided to focus attention on. I probably wouldn’t have taken such a close look if their model had once again supported the famous migration hump.

This discussion shows that the best thing we can do is to keep being critical: constantly doubting, questioning and admitting that what we know – and what experts know – is limited. Had I dug deeper I might have been able to raise some questions about the data set used in the MEDAM paper. But then again: there is no such thing as an unproblematic data set when it involves something as complicated as migration figures. And the concept that two non-stationary variables cannot be regressed if you are controlling for a cointegrated third variable – that’s not a question I could even have imagined asking in the context of this paper. And neither have many, many scientists, because the MEDAM paper has been read and widely acclaimed by lots of other smart people.

Actually, I’ve started thinking that journalists, scientists and policymakers are all in the same boat here: we would love for the world to be simpler than it can be. We want to be able to capture it in a nice, neat model, and then wrap it all up in a nice, neat article. But reality is so much more capricious and complex than any model can capture.

Seeing more shades of grey is also a way to understand the world better – but it’s not quite as simple to put into a pithy headline. It’s easier to just say: I was right after all.