Question Answering with pre-trained model T5

Question

I want to use the pre-trained T5 model https://huggingface.co/docs/transformers/model_doc/t5 on the task of Question Answering on the https://huggingface.co/datasets/boolq knowing that my inputs will be the passage and the question and the output is the boolean true or false that is the answer for the question.

I have seen some people tuning the model to this specific task. But, I want to know if there is a way to do it with pre-trained model to get some outputs and then compare them with the model after tuning.

Thanks!

Answer 1

Wasn't the T5 model also trained on BoolQ which would make this difficult and kind of fishy to test/evaluate because the later test data would not really be unseen data for the model? You can see it listed in the model card on huggingface as well as Google's original paper.

What I do find strange is that giving the pretrained T5-base a question from the dataset does not yield the expected answer or answer format. There is a fine-tuned version of t5 for BoolQ which gives a more acceptable answer. Same problem with the pretrained model for Question answering in the SQuAD format even when using the exact example and format from the paper.

Which leads me to think the fine-tuning on question answering is unlike some other tasks not actually included in the released version of the model or at least does not seem to have enough of an effect for the model to remember how the task works. In which case fine-tuning on it (again/more) would make sense again.