How to use the DeBERTa model by He et al. (2022) on Spyder?

Question

I have recently successfully analyzed text-based data using sentence transformers based on the BERT model. Inspired by the book by Kulkarni et al. (2022), my code looked like this:

# Import SentenceTransformer
from sentence_transformers import SentenceTransformer
# use paraphrase-MiniLM-L12-v2 pre trained model
sbert_model = SentenceTransformer('paraphrase-MiniLM-L12-v2')
# My text
x='The cat cought the mouse'
# get embeddings for each question
sentence_embeddings_BERT= sbert_model.encode(x)

I would like to do the same using the DeBERTa model but can't get it running. I managed to load the model, but how to apply it?

import transformers 

from transformers import DebertaTokenizer, AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")
model = AutoModel.from_pretrained("microsoft/deberta-v3-base")                    

sentence_embeddings_deBERTa= model(x)

The last line does not run, error message is:

AttributeError: 'str' object has no attribute 'size'

Any experienced DeBERTa users out there?

Thanks Pat

Answer 1

Welcome to SO ;) When you call encode() method it would tokenize the input then encode it to the tensors a transformer model expects, then pass it through model architecture. When you're using transformers you must do the steps manually.

from transformers import DebertaTokenizer, DebertaModel
import torch
# downloading the models
tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
model = DebertaModel.from_pretrained("microsoft/deberta-base")
# tokenizing the input text and converting it into pytorch tensors
inputs = tokenizer(["The cat cought the mouse", "This is the second sentence"], return_tensors="pt", padding=True)
# pass through the model 
outputs = model(**inputs)

print(outputs.last_hidden_state.shape)

Lastly, you must know what kind of output you're supposed to work with.