AWS Forecast - What does the parameter "backtest windows" really mean?

Question

I am new to "AWS Forecast" but I am training a model on retail data using the AWS UI. Two of the fields to setup the model, have optional input for: "Number of backtest windows" and "Backtest window offset".

Although, these are optional I want to understand what they mean and the AWS description does not help me understand it. Would anyone be able to provide an example with explicit numbers and dates, so that it makes more sense in how it is applied?

Thanks so much.

Answer 1

Amazon Forecast uses backtesting. This means that during the CreatePredictor operation, Amazon Forecast evaluates a predictor by splitting a dataset into two: training and testing. It uses the training dataset to train the model, and the testing dataset to evaluate the model’s predictive accuracy.

With the Number of backtest windows you specify how many times the algorithm will perform the split into training and validation. Best practice is to have more than 1 backtest window to minimise selection bias as in this case you assess the overall model accuracy from multiple windows. With the backtest window offset, you specify where you want to split the dataset for training and evaluation.

This example on GitHub has a lot of information and graphs on this.