'Multilabel Multiclass image classification with tensorflow

i try to train a model that classified frames from my garden camera to detect different animals. My dataset has 8000 images, 2000 per class of squirrels, pigeons, cats and dogs.

If only one class is visible in the frames the prediction from the model is good with 99.5% accuracy.

but:

Problem #1

If more then one class is shown the predicted probability of one of the classes is correct classified, the others are more ignored

Problem #2

if no class can classified, the prediction values vary alot.

I use the following code for training

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Model
from keras.layers import GlobalAveragePooling2D, BatchNormalization, Dropout, Dense
from keras.applications.mobilenet_v2 import MobileNetV2, preprocess_input
from keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau

base_model = MobileNetV2(include_top=False, input_shape=(224, 224, 3), weights='imagenet')
base_model.trainable = False

x = base_model.output
output = GlobalAveragePooling2D() (x)
output = BatchNormalization() (output)
output = Dense(128, activation='relu') (output)
output = Dropout(0.2) (output)
output = Dense(64, activation='relu') (output)
output = Dropout(0.2) (output)
output = Dense(4, activation='sigmoid') (output)
model = Model(inputs=base_model.input, outputs=output)

model.compile(
    loss='binary_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

train_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.4,
    horizontal_flip=True,
    validation_split=0.2,
)

train_generator = train_datagen.flow_from_directory(
    data_path,
    target_size=(224, 224),
    class_mode='categorical',
    shuffle=True,
    subset='training',
    seed=42
)

validation_datagen = ImageDataGenerator(
    preprocessing_function=preprocess_input,
    validation_split=0.2,
)

validation_generator = validation_datagen.flow_from_directory(
    data_path,
    target_size=(224, 224),
    class_mode='categorical',
    shuffle=False,
    subset='validation',
    seed=42
)


early = EarlyStopping(monitor='val_loss', mode='min', patience=5, verbose=1, restore_best_weights=True)
checkpoint = ModelCheckpoint("/content/checkpoints/", monitor='val_loss', mode='min', save_best_only=True, verbose=1)
reduceLR = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3)
callbacks = [
             reduceLR,
             early,
             checkpoint,
             ]

hist = model.fit(train_generator, validation_data=validation_generator, epochs=20, batch_size=32, callbacks=callbacks, verbose=1)

To test the model i use CV2 to feed the model with the frames

import cv2
import numpy as np
from keras.models import load_model

model = load_model("mobilenetv2_last_model.h5")

cap = cv2.VideoCapture("video.mp4")

while True:
    _, frame = cap.read()
    frame = cv2.resize(frame, None, fx=0.5, fy=0.5)
    image = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    image = cv2.resize(image, (224, 224))
    image = image / 127.5 - 1
    prediction = model.predict(np.expand_dims(image, axis=0))[0]
    print(prediction)

have anyone experience with this or knows how I can solve the problems



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source