Musings on Stacked Machine Learning Models

I recently worked with 3 machine learning models for FOREX prediction. One was ARIMA, the next multivariable linear regression, and the last one a stacked model that used the outputs of the first two as learners for input.

Here are the results from each:

While the regression model has an R2 value of 0.9714, the ARIMA beat it with a far better R2 value of 0.9976. That did not strike me odd, as in econometric time series analysis usually proves far more precise. But the stacked model only fared slightly better with an R2 value of 0.9978. I was expecting a little more precision out of the stacked model as it should have preprocessed and worked out the wrinkles and inefficiencies of the first two.

The topic of study is the TRM, Colombia’s official exchange rate for the US dollar. Economists and financiers have been trying to predict FOREX values since ever (or at least a very long time…) I was not looking for a break-through formula, but just something that would improve our company’s cost accounting accuracy and our ability to hedge currency volatility through forwards.

But I digress. The musing here is the additional overhead of the stacked model vs. a simple ARIMA forecast (using the excellent library by Robert Hyndmann here.) The accuracy is welcomed because when dealing with money every bit of a penny counts. But the production model is complicated at least. In my company, there is no such thing as a machine learning production model. Our IT staff is A+, but not very math-savvy. Any machine learning code is done by me, with little hope of moving it to production (and in my company it makes sense, we are small.) If a bigger enterprise how much more precision is worth the cost of additional computing cycles? The application of machine learning to ARIMA is not necessarily complicated. But having to run multiple models to ensemble does require additional computing power.

In a practical world, the difference between models is only 0.002. If the TRM is at 3,450 pesos to a dollar, the difference is 0.69 cents per dollar or 690,000 pesos in a one-million-dollar contract. That is the equivalent of 200 US dollars in one-million, which is money none the less but not a lot.

There is no right or wrong here. Since I cannot deploy my fancy algorithms to production I have not an easy way of estimating costs. But the musing is still interesting, and the simplicity of the model formulation has an incredible amount of complexity when brought to life.