'How to replace a value of smaller length than one with bigger length in pandas dataframe?

I have a dataset which I read as an pandas dataframe. I wanna tokenize one column of the dataset. My tokenized value is a list with bigger length than my original cell value.

Error message:

ValueError: Replacement lists must match in length. Expecting 1 got 72

How can I do this replacement in my dataframe?

Code where I tried to iterate over and replace my values

trainData = pd.read_csv("data/train.csv")
test = trainData[0:100]
CODEBERTA_LANGUAGE_ID = "huggingface/CodeBERTa-language-id"
tokenizer = RobertaTokenizer.from_pretrained(CODEBERTA_LANGUAGE_ID)
for i in test["code"]:
    test["code"] = test["code"].replace([i], tokenizer.encode(i))

Image example of my dataframe

Dataframe



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source