Calculating processing time of a deep learning model

Question

My model deals with videos, and I want to calculate how fast it can process frames as in frames per second or processing time for 1 frame.

I have made a single function to get predictions, it takes in raw frames as input, does all the preprocessing, and returns the classification. One of the preprocessing steps is sampling the frames from the video, basically, it reduces the number of frames which go into the deep learning model by 1/5. Without all the preprocessing, the model won't perform as expected.

So my question is, should I consider the preprocessing time aswell? And, most importantly, is this processing time for all frames or just for the frames the model actually sees?

Sample code structure as below:

start = time.time()
prediction = main(data)
end = time.time
print("Time for 1 frame=",(end-start)/n_frames) # lets say n_frames = 50

Inside the main function:

preprocessed = preprocess(data) # resizing, sampling down from 50 to 10 frames
prediction = model.predict(preprocessed)
return prediction

Example: Input is of 50 frames, and total time taken to preprocess them and make predictions is 1 second. (Note that the model only sees 10 preprocessed frames) So, the processing time for 1 frame is 1/50seconds. OR Should it be 1/10seconds, as model only gets to process 10 frames, others simply get skipped in preprocessing. And where should I put the start time and end time frame?

Which way is the standard way or the right way?

Answer 1

There is no standard way, it depends on what exactly you're going to use the results for.

If you're trying to ONLY demonstrate the time used up by the deep learning model, don't include the preprocessing steps.

If you want to time the end to end process, include the entire pipeline.

An even better solution would be to profile you're code. This will give you a breakdown of how long each part of your code takes so you don't have to pick on or the other.

In your case, since you want to put the time in terms of the number of frames, I don't care about how many frames your model sees or any of the in workings of your pipeline. As a user, all I care about is if I put X number of frames in, how long will it take. So go with 50 frames / second.