Training Transformer for Text Summarization Train and Validation loss decreasing until nearing 0 but the validation R1 score is bad

Question

I am training a Transformer model for text summarization task with transformer seq2seq, the training loss and validation loss keep decreasing until 0.000~ but the R1 score of the validation only 0.1-0.3 and won't increase. I tried to change the model parameter, decreasing layer, increasing layer, increasing/decreasing dimension, increasing/decreasing feed forward dimension but model can't generalize good enough with R-1 score, but validation loss is very low nearing 0.(0.0001~). Any suggestion? Thanks!