'Loading Kears Model after crash: Do keras callbacks such as EarlyStopping consider the pre-crash training history?
I am training some models on a remote machine and I lost my ssh connection. I have been saving the models at the end of each epoch with the following call back.
cp = k.callbacks.ModelCheckpoint(
filepath=checkPointOutPutFolderPath,
verbose=0,
monitor="val_loss",
save_best_only=True,
save_weights_only=False,
mode="auto",
save_freq="epoch", )
I also had an early stopping call back and reduce lr call back
es = k.callbacks.EarlyStopping(monitor='val_loss', restore_best_weights=True, patience=7)
rlronp = k.callbacks.ReduceLROnPlateau(monitor='loss', factor=.5, patience=3)
My question is what happens when I reload the model with respect to the early stopping and reduce learning rate call backs? If the validation loss did not improve for 6 iterations before the crash does the model checkpoint save that information? Will the callback fire if the first iteration also does not improve validation loss? Or will I risk overtraining if I restart it? I think It was getting close... If the callback won't fire correctly does anyone know how I could fix it?
model = k.models.load_model(checkPointOutPutFolderPath)
cbs = [es, rlronp, cp]
history = model.fit(trainGen,
epochs=max_epoch,
batch_size=batchSize,
validation_data=valGen,
shuffle=True,
callbacks=cbs,
workers=16,
max_queue_size=64,
validation_freq=1,
)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
