Can you give advice on suitable algorithms for Image anomaly detector?

Question

We want to detect anomalies in images. We have an image stream that is taken by a stationary camera placed outdoors filming an area / landscape with no cars or pedestrians. The images will change due to sunlight and temperature but such change is most often normal (think of ice or snow melting or the image becoming bright due to strong sunlight). We have an idea of using a form of neural network that takes images and sensor data as inputs. The neural network is first trained on normal image data and normal temperature and sun measurements. When there is an abnormality in the images not explained by sun or temperature the area of the image where the anomaly is should be output. One example could be a starting movement of soil or a small landslide or water in not common areas of the picture. What technique / algorithms can be used for this?