The curse of dimensionality

By | 21 January 2015

sale of sheepDevelopment of the AgOptimizer program is continuing and we are averaging around 400 million daily iterations of the system on a single machine.

The AgOptimizer system undertakes an exhaustive search. This is a problem-solving technique that consists of systematically enumerating all possible candidates for the solution and checking whether each candidate satisfies the problem’s statement.

While a brute-force search will always find a solution if it exists, the time required to find the answer is proportional to the number of candidate solutions – which in many practical problems tends to grow very quickly as the size of the problem increases.  A classic example of this is that our AgOptimizer for Sheep is currently focused on finding the optimal number and type of sheep to sell following shearing, however if we also factor in that the lambing rates for the client are expected to increase in future years with supplementary feeding then the number of possible solutions to evaluate increases exponentially. As even a simplified version of our model needs to search through more than 2.5 billion possible solutions, and exponential increase in possible solutions will much more challenging to manage.

We are finding that the use of an exhaustive search for AgOptimizer is a great way to test the model and is also useful as a baseline method when benchmarking other algorithms.

Today we are testing the sale of livestock in groups of 50 as a means to simplify and counter-act the curse of dimensionality!