The failures of models

Many years ago I started as a macroeconomic modeller and forecaster. After a few years, I became disillusioned with the tool and methodologies followed, which were often naive and pedantic. In particular I came to the conclusion that forecasting was a ridiculous waste of time, adding nothing to human welfare. As for modelling, it can be a useful tool, but has been systematically misused with spurious results convincing gullible politicians and journalists of the merits of a policy action which had no basis in reality. In short, a model became a tool to persuade, with the assumptions being opaque, and the results tweaked to give the outcome desired.

Models, of course, have long been used. In the modern era of warfare (particularly in the 18th and 19th centuries), for example, careful models were made of the landscape to plan engagements. These were highly formalised and almost ritualistic encounters, so it is no surprise that a good general (such as Napoleon) could achieve decisive results by showing flexibility and reading the terrain.

But those physical models were well understood, including their limitations.

Unfortunately the modern computable general equilibrium model is the antithesis of transparency. I think the world would be a better place if they did not exist (despite my comment above that they can be a useful tool). While a useful tool, the tendency for models to be misused dominates the advantages they bestow.

There has been much criticism (rightly) of the use of various technical models in financial institutions which led them to take excessive risks at the ultimate cost to the taxpayer. The mathematical rigour and numerical precision of risk-management and asset-pricing tools tends to conceal the weaknesses of models and assumptions to those who have not developed them.

Researchers, too, have failed in their ethical duty to warn of the limitations of their models. Overall, the dynamic general-equilibrium models not only have weak microeconomic foundations,  but their empirical performance has been remarkably poor, often worse than useless.

The models used in financial firms failed to assist management (and regulators) to assess the risks borne. The models used in governments (finance ministries and international organisations) failed to provide any useful information when it came to the tensions and imbalances in the world economy in the lead up to the financial crisis. All were in a state of shock when the crisis exploded, leading them to flap around in a panic throwing untold amounts of taxpayers’ money at a problem which they only exacerbated.

Is it such a stretch, therefore, to assert that the model caused the crisis? I think not.

While today many are looking with skeptical eyes at the macroeconomic model, the same cannot be said of climate change models which share exactly the same weaknesses as the macroeconomic model.

Climate models, too, are worse than useless, and certainly do not prove that the climate is changing.

Let’s start relying once more on empirical evidence and measurement, not complex computer models that even their creators do not understand. It is hubris to think we can model a whole economy, much less the climate.

Models can have their place: an architectural drawing, engineering specifications, etc have a long and valued part in society. But their foundations are well understood, and their results are predictable and have been validated.

Beyond this, when models become overly complicated, and their results unclear they should be rejected as a piece of propaganda.

About Samuel J

Samuel J has an economics background and is a part-time consultant
This entry was posted in Economics and economy, Treasury. Bookmark the permalink.

31 Responses to The failures of models

  1. Blogstrop

    It’s very telling that the same people who have great faith in the climate alarmist’s models have unshakeable faith in the Labor Party, despite all the evidence of several rounds of federal rule and numerous state misadventures during our lifetimes.
    The views of the doubly deluded or completely complicit should by now be a laughing stock or be ignored. But they keep on cropping up here, adding massively to the threads as they are engaged by the indefatigable resident commenters.
    They also keep appearing in the media generally, which is a poor indication of the state of the nation, and it’s likely to continue to warp election results regardless of whether we have compulsory or casual drive-by voters.

  2. The issue is not the models per se, but the refusal or inability of those using them to reject a cherished hypothesis.

  3. Elizabeth (Lizzie) B.

    When models don’t agree with empirical realities it is a good idea to check the assumptions in the models. Models are useful for showing how wrong models can be.

  4. Poor Old Rafe

    To see how the models were fiddled to get the result the climate alarmists wanted, see the account that Garth Paltridge provided, especially the nonb-briefing of Garnaut.

  5. mundi

    financial modelling is laughable. if people claim to be able to do it, then why havent they made billions in wise investments.

    remember all those models claiming the gfc meltdown was a 6 to 8 sigma provability event…

  6. Ellen of Tasmania

    The problem here is that, in science, models are an hyptohesis. But users tend to treat them as a conclusion. Models are a great way of combining a complicated set of ideas and interactions and then seeing how they relate to real world observations. The result tells us something about the state of our knowledge. They are a great research tool. The problem comes when we use their predictions as reality without any attempt at validation, as is done in climate science.

  7. Skuter

    The models used in governments (finance ministries and international organisations) failed to provide any useful information when it came to the tensions and imbalances in the world economy in the lead up to the financial crisis. All were in a state of shock when the crisis exploded, leading them to flap around in a panic throwing untold amounts of taxpayers’ money at a problem which they only exacerbated.

    Samuel, it is worse than this. Let’s not forget how models were used by Treasury to justify stimulus expenditure. Claims such as: ‘the nation building and jobs package saved or created X thousand jobs’ are all used as post hoc justification for the spending and are an artefact of Treasury’s TRYM model. They have no basis in reality.

  8. dd

    The problem really is with models. It’s demand driven. People want concrete results, pretty charts, and solid predictions.
    To build any model of the world you need to put in numerous working assumptions and simplifications which frequently, you just dont’ know for sure if they’re right. often you know they’re wrong but they fall into the category of ‘probably doesn’t matter.’ Many assumptions are implicit – they’re part of how the code was set up, rather than explicitly part of the design.

    For this reason, it’s not uncommon for the computer programmer to be the team’s expert on the workings of the model. Seriously, go to a lab or research team that are running models or simulations, ask some curly questions, and see which door everyone’s index fingers start commanding you to visit.

  9. Poor Old Rafe

    The waste of intellectual and financial resources is in economic models of the wrong kind is matched (though not in quantity) by the philosophical investment in positivism and logical empiricism. This zombie philosophy cast a blanket of boredom and futility over parts of the profession from the 1930s to the 1960s when the students turned to more exciting but equally useless fads and fashions. Don’t get me started!

  10. Poor Old Rafe

    For a useless example of model building, look at my efforts in the 1970s. But you have to admit that until you realise that it is not working, running these models is as much fun as you can have with your clothes on.

  11. johanna

    I’ve worked with models and modellers quite a bit, and the good modellers are genuinely super-cautious and humble about what is possible. Basic things like – the more variables there are, the less accurate the outcome will be – are simply not understood by both users and citers of models to an alarming extent.

    Dynamic systems models intended for predictive purposes are mostly voodoo masquerading as science, as has been demonstrated time and time again.

    However, there are very nifty and useful models where the variables are minimal and the design has been thoroughly tested against empirical evidence. For example, the models used to design aircraft are a wonder.

    But, you still don’t put the public up in a model-designed plane until that puppy has been thoroughly tested under real conditions.

  12. .

    Models are useful. Good ones are bloody hard to build ones that are useful to external users are generally not methodologically rigourous.

    They need to be taken with a grain of Austrian school salt.

  13. Nuke Gray

    You are wrong- Elle Mcpherson is an older model that still works! And the Naomi campbell model still makes appearances.

  14. Entropy


    For this reason, it’s not uncommon for the computer programmer to be the team’s expert on the workings of the model. Seriously, go to a lab or research team that are running models or simulations, ask some curly questions, and see which door everyone’s index fingers start commanding you to visit.


    This is so, so true!

  15. Quite so Samuel J.

    Computers are terrific calculators which save time in calculation and in the manipulation of heaps of data.

    Major problems are born when lazy “management” expects an enthusiastic IT Department, using the latest computer technology, supported by a gaggle of pointy heads to manage the entire process. IT becomes the leader and the client get exactly what it deserves.

    What’s the current status IT? Sir, we have just assessed progress and we will finish in two weeks. One year, two years from now the answer will be the same.

  16. Frank

    For example, the models used to design aircraft are a wonder.

    Has a visitor once that worked for Pratt & Whitney – a real life rocket scientist. Quizzing her on the software she used (Matlab) led her to pointing out that they have a pretty good understanding of airflow on the outside of the engine. But the inside of the engines, not so much.

  17. Greg

    I used to work in the oil industry where computer models are commonly used however, risk management was at the centre of everything we did, from geological to economic to political risk. Our assumptions were explicitly stated along with the uncertainties for all key parameters which ultimately resulted in an economic value, plus a “proven” level (90% confidence) plus a “possible” level, the upside. These procedures have been used for decades in this industry to great success. Perhaps economists and other users of modelling (such as climate scientists) should learn from this industry where good risk management is essential to avoid failure.

  18. .

    That’s a tad condescending. I’m sure there are oil firms that failed to model properly and ultimately failed as businesses.

  19. Louis Hissink

    The principal fault with models is their inability to cope with catastrophic and unpredictable events.

    The inability of climate models to hindcast the MWP and the LIA is well known – but the error is made in assuming these events were the result of continuing predictable processes. It is more likely that the LIA was initiated with a chance encounter of a meteorite swarm with the earth – Korean data is documented in their annals of the Choson dynasty, for example. That event also saw the collapse of circum pacific civilisations, the extinction of the NZ moas (Maoris reckon a sky fire killed the animals) and Ted Bryant of Wollongong University suggests that a meteorite impact occurred during the 16th century beween NZ and Australia, leaving a trail of tsunami deposits along the eastern seaboard of Australia, and as well in the Kimberley, apparently. That event also terminated the Chinese Ming Dynasty, and set the scene for the Europeans to fill the gap left by that catastrophe – a catastrophe that also affected Europe but not to as great an extent as the Pacific region.

    This type of “close encounter” between the earth and some other cosmic body is totally unpredictable, and it’s a scenario that is totally ignored in climate modelling. That’s why the models fail – and will always fail.

  20. Louis Hissink

    If you want a graphical representation of a catastrophe, study the price chart of Whitehaven coal this week – the precipitous drop in share price can be likened as a catastrophic event, and the subsequent rise back to previous levels as the reaction. So too with the earth except that the ignorati insist we are causing the temperature rise back to previous levels.

  21. There’s a paper doing the rounds purporting to show that algorithmic forecasting is impossible (“incomputable”).

  22. And by “doing the rounds” I mean it was written 6 and a half years ago and I only heard about it this week while chatting with a fellow about universal models of … well … anything.

  23. Actually, I didn’t state that correctly.

    The argument the paper makes is that there aren’t computable simple models for complex problems:

    What we have shown here is that there does not exist an elegant constructive theory of prediction for computable sequences, even if we assume unbounded computational resources, unbounded data and learning time, and place moderate bounds on the Kolmogorov complexity of the sequences to be predicted. Very powerful computable predictors are therefore necessarily complex.

    So complex problems require complex models. But as the complexity of the model approaches the complexity of the real world, you’ve lost the utility of a model, which is that it must be simple enough for humans to understand and verify.

  24. DrBeauGan

    The soothsayers are always with us. Now they come armed with computers.

  25. Frank

    The principal fault with models is their inability to cope with catastrophic and unpredictable events.

    Yes. Determinism is easier to spot in hindsight, less so to account for in advance. Or perhaps sometimes systems only appear deterministic in retrospect.

    The same thing is a worry when you see calculations on the basis of what has been lost due to some event. For example; economic loss due to cyclone and the attendant loss of crops. How can you count that which was never to be as a loss, based on what should have been if everything had gone according to expectations. Or worse, imagining a situation and then defining some metric to allow measurements of the discrepancy between how it should be and how it is, then putting a dollar value on it and bemoaning the loss. Like sick days lost to hangovers versus the ideal of an alcohol free population. Then again the operations research guys might beg to differ on this line of thought.

    So complex problems require complex models. But as the complexity of the model approaches the complexity of the real world, you’ve lost the utility of a model, which is that it must be simple enough for humans to understand and verify.

    That is the whole point of modelling though, to throw out whatever variables you can consider as negligible and still be able to solve it. Otherwise the answers are such lumbering beasts with too many moving parts to make sense of. A case of having a worthwhile model negating the point of having a model.

  26. Samuel J

    I think CGE models are used as an alternative to thinking. Lazy economics really.

  27. Elizabeth (Lizzie) B.

    So complex problems require complex models. But as the complexity of the model approaches the complexity of the real world, you’ve lost the utility of a model, which is that it must be simple enough for humans to understand and verify.

    That’s really neat, Jacques. Various sorts of systems theorists have been speculating on modelling aspects of ‘social existence/interaction’ ever since any sort of modelling moved into complex computational spheres, but they have always produced nonsense if they tried – and always will, due to complexity.

  28. Frank;

    The same thing is a worry when you see calculations on the basis of what has been lost due to some event.

    I believe economists call this a “counterfactual”. In other places, it’s known as “alternate history” and is correctly categorised as a sub-genre of science fiction.

    That is the whole point of modelling though, to throw out whatever variables you can consider as negligible and still be able to solve it

    The point being that you can either have simplicity or you can have fidelity. The paper I linked above shows that you cannot have both.

    In particular, lots of interesting systems have catastrophic shifts somewhere in their phases space; with sensitivity to variations in starting conditions that are below the threshold of detection. They’re chaotic: strictly speaking deterministic but in practice unpredictable. The further into the future the model runs, the wronger it gets.

    I mean you can do stuff like regression modelling to get the general case, but every once in a while … well, to put it charitably, every once in while you’re going to need to tinker with the model.

  29. Elizabeth;

    Various sorts of systems theorists have been speculating on modelling aspects of ‘social existence/interaction’…

    The funny thing is that the systems theorists are several hundred years late to the emergent-phenomena party. Mandeville beat everyone to the punch with his Fable of the Bees.

    …but they have always produced nonsense if they tried – and always will, due to complexity.

    Agreed: trying to precisely model social systems is a nonsense. Any predictive model approaches the complexity of the original, and then you’re still fucked because you can hardly go around measuring people’s neural activity on a millisecond-by-millisecond basis.

    You can definitely have accurate-but-imprecise models. I can say “when prices fall, demand rises” with a great deal of confidence. By how much will demand rise? I don’t know. What about odd cases like status goods? I don’t know. What if there’s a big advertising campaign? I don’t know.

    But would I steadily get richer betting on this model against someone prepared to bet against it?

    Yes. Yes I would.

  30. Mick Gold Coast QLD

    “These procedures have been used for decades in this industry to great success. … Perhaps economists and other users of modelling (such as climate scientists) should learn from this (oil) industry where good risk management is essential to avoid failure.”

    That is the oil industry I knew too Greg. I arrived armed with my experience of DCFs and the like, learned in property, project design and banking where the fanciful cash flows work backwards from the result the top man expects to see.

    They do Brisbane toll roads and tunnels using the same method, and that is why they fail. You should have seen the chicanery surrounding the emergence of the property trusts, and the amazing assumptions in the models used to justify their abyssmal investments.

    I found these blokes to be disciplined, reflective, not at all frantic and quite unconcerned by what they thought the CEO or the board wanted to hear. Indeed it was made crystal clear I’d been brought in to provide dispassionate independent expert advice.

    It was rare to find anyone in the executive who wasn’t first a chemical engineer (schooled in objective professional methods) and the mood was collegiate not internecine. From the head man down people wanted to be there. If the best men working together on the proposed lube blending plant, coal loader, freeway mega-retail outlet, joint venture or refinery investment found it wouldn’t work it was not a case of then contriving viability.

    They are little interested in the eternal search for the guilty, not-being-me-the-investigator, which infects so many industries. Australian management is simply appalling in that regard.

    “That’s a tad condescending. I’m sure there are oil firms that failed to model properly and ultimately failed as businesses.”

    How is that condescending Dot? What experience makes you so sure?

    The industry is an outstanding example of how to do things well, superior by any measure to any of the several industries to whom I have sold my body. It doesn’t come up with bright ideas, construction finance them with other peoples’ money and depart after flogging them to unsuspecting rich dills – they hold and operate their investments for decades, which introduces a whole ‘nother element.

  31. Frank

    Jacques:

    had a look at that paper. The general gist is straightforward but the details beyond me.

    The point being that you can either have simplicity or you can have fidelity.

    Yes. But from the perspective of the person charged with having to concoct and use one of these things then simplicity is probably the more attractive choice? Chances are you wind up with something intractable or some oiled wriggling octopus of dependant possibilities that makes it hard to say anything useful otherwise.

    lots of interesting systems have catastrophic shifts somewhere in their phases space; with sensitivity to variations in starting conditions that are below the threshold of detection. They’re chaotic: strictly speaking deterministic but in practice unpredictable. The further into the future the model runs, the wronger it gets.

    Hit them with a hammer, Dirac delta in other words. You can try that and it may bring out harmonics. They would be a function of the model though. Sensitive dependence upon initial conditions and deteriminism are (partly) defining characteristics for chaotic systems. But deterministic they are even if they operate below some measurement threshold. What I’m thinking of is more fundamental, as in how many dimensions are you looking at and can it ever be agreed on beforehand.

    If extra variables keep needing to be added as someone gets a better understanding of a situation then that is one thing. A one off event that has consequences flowing smoothly from it, but in no way predictable beforehand is another. Is it still deterministic if you can keep throwing things at it from nowhere. Analogously you can have locally chaotic systems that are globally stable but can you have short term component-wise deterministic systems that long term aren’t? I suspect you can from looking at the world, from the point of modelling it would surely make things tricky.

Comments are closed.