using nn.Cross entropy between outputs and target label

Question

I use this code

function to train the model

def train():
  
  model.train()

  total_loss, total_accuracy = 0, 0
  
  # empty list to save model predictions
  total_preds=[]
  
  # iterate over batches
  for step,batch in enumerate(train_dataloader):
    
    # progress update after every 50 batches.
    if step % 50 == 0 and not step == 0:
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

    # push the batch to gpu
    #batch = [r for r in batch]
 
    sent_id, mask, labels = batch['input_ids'],batch['attention_mask'],batch['labels']
    print(6)
    print(sent_id)
    print(mask)
    print(labels)
    print(batch['input_ids'].shape)
    print(batch['attention_mask'].shape)
    print(batch['labels'].shape)

    # clear previously calculated gradients 
    model.zero_grad() 
    print(7)       

    # get model predictions for the current batch
    preds = model(sent_id, mask)
    print(8)
    print(len(preds))
    print(len(labels))
    print(preds.size())
    
   
    preds =torch.argmax(preds, dim=1)
    preds =torch.argmax(preds, dim=1)
    print(preds)
    print(labels)

    # compute the loss between actual and predicted values
    loss = loss_fn(preds, labels)
    print(9)

    # add on to the total loss
    total_loss = total_loss + loss.item()
    print(10)

    # backward pass to calculate the gradients
    loss.backward()

    # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
  # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # update parameters
    optimizer.step()

    # model predictions are stored on GPU. So, push it to CPU
    preds=preds.numpy()

    # append the model predictions
    total_preds.append(preds)

  # compute the training loss of the epoch
  avg_loss = total_loss / len(train_dataloader)
  
  # predictions are in the form of (no. of batches, size of batch, no. of classes).
  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  #returns the loss and predictions
  return avg_loss, total_preds

import torch.nn as nn

loss_fn=nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)

i got error in calculation of loss preds tensor([5, 1, 1, 1, 0, 2, 1, 4, 2, 3, 0, 2, 0, 1, 0, 3, 5, 3, 1, 2, 0, 2, 2, 1, 0, 1, 4, 0, 5, 5, 4, 5, 0, 2, 0, 1, 4, 0, 0, 3, 5, 1, 1, 1, 4, 4, 4, 1, 2, 1, 3, 3, 2, 1, 0, 2, 0, 4, 4, 4, 3, 2, 0, 5])

labels tensor([0, 0, 1, 2, 3, 0, 0, 0, 0, 1, 1, 0, 0, 0, 4, 0, 0, 2, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 2, 1, 3, 2, 0, 3, 4, 0, 1, 0, 0, 0, 0, 0, 0, 5, 0, 0, 3, 0, 0, 1, 0, 0, 0, 2, 0, 0, 2, 0, 0, 2, 0, 0, 0]) i used them in loss=loss_fn(preds,labels) error:

in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: Expected floating point type for target with class probabilities, got Long

Answer 1

The problem is that you passing the wrong value of preds (tensor) to the loss_fn function. Look well and you will notice that you are passing the output of preds = torch.argmax(preds, dim=1) while you should pass the output of preds = model(sent_id, mask). Doing so you are passing two tensors of dtype int64 to the loss function. However, the loss function (CrossEntropyLoss) expects a tensor of dtype float32 as its first parameter (i.e. the input parameter) -- see in 'Examples' at https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html. Then you get a error: "Expected floating point type..."

To solve the problem you can compute the loss before changing the value of preds (i.e. before preds = torch.argmax(preds, dim=1)), as I do bellow. Or you could give another name to the output of model(), for instance outp, and pass it to your loss function, like: loss_fn(outp, labels).

# get model predictions for the current batch

preds = model(sent_id, mask)

# compute the loss between actual and predicted values

loss = loss_fn(preds, labels)
preds =torch.argmax(preds, dim=1)