Snake Oil Economic Models

October 29th, 2011

Remember, this isn’t physics.

Gold Eyes Biggest 3-Day Fall In 28 Years

This is hilarious. It reminds me of my experiences in trying to develop black box trading systems. It’s relatively easy, for example, to program an autotrading system for EURUSD that does extremely well on some finite chuck of historical data. Initially, I thought that increasing the size of the dataset would help to produce better results over a variety of conditions. It turns out that the size of the initial dataset didn’t really matter because everything I tried resulted in overtraining my algorithm to that particular dataset. Once calibrated, what do you think happened as more data was added? That’s right, the %Profitable trades decreased. Eventually, it became clear that I was developing ornate coin tossing systems.

Anyone who works with trading systems should print this out, frame it and hang it on the wall. It’s from the article below:

That financial models are plagued by calibration problems is no surprise to Wilmott–he notes that it has become routine for modelers in finance to simply keep recalibrating their models over and over again as the models continue to turn out bad predictions. “When you have to keep recalibrating a model, something is wrong with it,” he says.

Yeah, buddy.

One more thing: The headline on the piece below is, “Why Economic Models Are Always Wrong.” I would argue that if something was always wrong, it would be easy to win! You could simply go the other way. In my experience, over time and given enough data, everything I’ve done nauseatingly approaches a coin toss. “Always wrong,” in a financial model would be a winning lottery ticket.

Via: Scientific American:

When it comes to assigning blame for the current economic doldrums, the quants who build the complicated mathematic financial risk models, and the traders who rely on them, deserve their share of the blame. [See “A Formula For Economic Calamity” in the November 2011 issue]. But what if there were a way to come up with simpler models that perfectly reflected reality? And what if we had perfect financial data to plug into them?

Incredibly, even under those utterly unrealizable conditions, we’d still get bad predictions from models.

The reason is that current methods used to “calibrate” models often render them inaccurate.

Research Credit: minerva

2 Responses to “Snake Oil Economic Models”

  1. dagobaz says:

    mm reversal levels from the daily chart of the indu: 87.21% success of reversal within es stop tolerance of 2.0 points (approximately 11,500 iterations)

    max # of sequential losers: 12
    number of times expected in 1k sample: 1.08

    with just about anything else I have done, my results mirror yours.

    i do believe any curve-fitting exercise is going to revert to the mean eventually.

    – c

  2. Kevin says:

    You know how I was trying to get my friend to help me modify that MM indicator to work as a strategy… He never did, but I finally figured it out. So I’m able to backtest with it now.

    I just have daily data now, because I’ve gone back to my bottom feeding stuff, but I’ll run MM against INDU daily bars and see what I can see.

Leave a Reply

You must be logged in to post a comment.