'Continue LM pretraining with Huggingface - loss function clarification
I'm trying to use Huggingface's tensorflow run_mlm.py script to continue pretraining a bert model, and didn't understand the following: in the above script, the model is loaded using from_pretrained and then compiled with a dummy_loss function before running model.fit(...). The dummy_loss function defined in the script ignores y_true and simply returns the mean of y_pred. Is this loss function overridden somehow? I can't understand how the script actually continues pretraining if this is the only loss function used.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
