CNTK - Faster RCNN Train with My Own Labels Data Set Can not Train More Than 20 Images

Question

I'm working with CNTK Faster RCNN object detection and now I have been facing with problem. To make you understand the problem, I will start with explain my work process from started.

First I follow by https://docs.microsoft.com/en-us/cognitive-toolkit/object-detection-using-faster-r-cnn to install all of need package. I successful in the step. Then I try with grocery data set which is contain 20 images train (I'm using base model as AlexNet). And the results is done. everything look work at this point.

Then I use VoTT to labels my dataset and I put it into data set folder of CNTK. I also use annotations_helper.py to generate other input files for prepare model training step.

After I create My_DataSet_config.py and change some configuration. I realize that I can not train my data set more than 20 image. Let's say if I train 30 images programs will error like gt_boxes is empty (it's really empty but with some specific images training number it's no longer empty). So I try to follow some instruction I found on GitHub like the problem is image and annotation files, try to delete the image and run again.

I really done that but it's not solution on my case. If the number of data set for train still not 20 images, I will find the error again with any image. Please take a look. Thank you

Python 3.5 Windows CNTK 2.7

Here is my data set configuration file. enter image description here

Here is my model configuration file. enter image description here