'fable vs. forecast ETS model estimation performance
I'm seeing a large difference in the performance of the forecast ets function and the fable ETS function when estimating multiple models, surprisingly in favor of forecast. Here's a simple example using the "tourism" data set from fpp3. The inelegant loop in the forecast section is intentional. Apologies for the rest of the code in advance:
# libraries ---------------------------------------------------------------
library(fpp3)
library(forecast)
# fable functions ---------------------------------------------------------
start_time <- Sys.time()
fable_fit <- tourism %>%
model(
ets = ETS(Trips)
)
fable_out <- fable_fit %>%
forecast(h=4) %>%
hilo(c(95,80)) %>%
unpack_hilo(c('95%','80%')) %>%
as_tibble() %>%
unite(Key_FC,Region, State, Purpose, Quarter, sep = '@', remove = FALSE)
end_time <- Sys.time()
fable_time <- end_time - start_time
cat("Time for `fable` estimation & forecast:", round(as.numeric(fable_time,units = "secs"),2), "sec\n")
# forecast functions ------------------------------------------------------
tourism_aug <- tourism %>%
unite(Key,Region, State, Purpose, sep = '@', remove = TRUE)
keys <- unique(tourism_aug$Key)
forecast_out <- tibble()
start_time <- Sys.time()
for(i in 1:length(keys)) {
ts_single <- tourism_aug %>%
filter(Key == keys[i]) %>%
select(-Key) %>%
ts(frequency=4, start=c(1998,1))
forecast_new <- forecast(ets(ts_single[,2]),h=4) %>%
as_tibble(rownames="index") %>%
mutate(Key = keys[i])
forecast_out <- forecast_out %>% bind_rows(forecast_new)
}
end_time <- Sys.time()
forecast_time <- end_time - start_time
cat("Time for `forecast` estimation & forecast:", round(as.numeric(forecast_time,units = "secs"),2), "sec\n")
# Comparison --------------------------------------------------------------
comparison_data <- forecast_out %>%
unite(Key_FC, Key, index, sep="@", remove = FALSE) %>%
left_join(fable_out,
by = c("Key_FC" = "Key_FC"),
suffix = c(".forecast",".fable")) %>%
mutate(mean_2 = (`Point Forecast` - .mean)^2) %>%
mutate(`80%_lower_2` = (`80%_lower`-`Lo 80`)^2) %>%
mutate(`80%_upper_2` = (`80%_upper`-`Hi 80`)^2) %>%
mutate(`95%_lower_2` = (`95%_lower`-`Lo 95`)^2) %>%
mutate(`95%_upper_2` = (`95%_upper`-`Hi 95`)^2) %>%
arrange(desc(mean_2))
comparison_summary <- comparison_data %>%
select(Key, Quarter,
mean_2, `80%_lower_2`, `80%_upper_2`,
`95%_lower_2`,`95%_upper_2`) %>%
group_by(Key) %>%
summarise(MSE_mean = mean(mean_2),
`MSE_80%_lower` = mean(`80%_lower_2`),
`MSE_80%_upper` = mean(`80%_upper_2`),
`MSE_95%_lower` = mean(`95%_lower_2`),
`MSE_95%_upper` = mean(`95%_upper_2`),
max_err = max(MSE_mean,
`MSE_80%_lower`,
`MSE_80%_upper`,
`MSE_95%_lower`,
`MSE_95%_upper`
)
) %>%
arrange(desc(max_err))
On my machine, my output is
Time for `fable` estimation & forecast: 106.93 sec
Time for `forecast` estimation & forecast: 37.03 sec
The comparison_summary table seems to indicate that the models are giving the same forecasts, only fable is taking almost 3 times as long as using forecast in a simple loop.
Am I using either of the functions incorrectly? My first thought was that fable ETS is searching a much larger set of models, but on default settings the search space for both algorithms should be the same.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
