Pytorch classification gives wrong prediction

Question

After training my classification model I get an accuracy of 94% on my Test data. I am working with TIFF images. To load the data and feed it into the classification model I am using the Dataloader from Pytorch.

My Dataloader function looks like this:

def dataload(self,train_path,batch_train,test_path,batch_train_val):

    #Transforms
    transformer=transforms.Compose([
        #transforms.Resize((450,450)),
        transforms.Resize((150,150)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),  #0-255 to 0-1, numpy to tensors
        transforms.Normalize([0.5,0.5,0.5], # 0-1 to [-1,1]
                             [0.5,0.5,0.5])])

    train_loader=DataLoader(
        torchvision.datasets.ImageFolder(train_path,transform=transformer),
        #torchvision.datasets.ImageFolder(train_path,transform=get_transformer()),
        batch_size=batch_train, shuffle=True
    )
    test_loader=DataLoader(
        torchvision.datasets.ImageFolder(test_path,transform=transformer),
        #torchvision.datasets.ImageFolder(train_path,transform=get_transformer()),
        batch_size=batch_train_val, shuffle=True
    )
    return [train_loader,test_loader]

Dataloader manages Tiff images and converts them somehow automatically into a three layer image, because a TIFF image has 4 layers but my model need a three layer image as an input.

When I finally tried to to use my saved model I got several problems. Since I am loading each image separately to predict the label I don't use Dataloader from Pytorch anymore.

My code looks like this:

  all_images = [f for f in listdir(pred_path) if isfile(join(pred_path, f))]
    for i in all_images:
        transformer=transforms.Compose([
        transforms.Resize((150,150)),
        transforms.ToTensor(),
        transforms.Normalize([0.5,0.5,0.5], # 0-1 to [-1,1]
                             [0.5,0.5,0.5])])
        
        image=Image.open(pred_path+"/"+i).convert('RGB')
        image_tensor=transformer(image).float()
        image_tensor=image_tensor.unsqueeze_(0)
        input=Variable(image_tensor)
        output=model(input)            
        index=output.data.numpy().argmax()

Since Tiff images have 4 layers and my model expects a three layer image I get an error. However when I manually convert the Tiff to a JPG image or convert them directly to RGB in my code, the model always predicts the same label for every image.

The strange part is that I only get all these problems when using the Efficientnet B7 model. When I use a small custom model, everything works fine I get neither of the above problems.