Very low IoU for semantic Segmentation on Pascal VOC data using DeeplabV3 model in Tensorflow

Question

I am trying to train a model in TensorFlow for semantic segmentation on the Pascal VOC dataset and I am not able to attain more than a 0.3 IoU score on the validation set. My implementation is similar to https://github.com/rishizek/tensorflow-deeplab-v3. I have used a batch size of 10, with SGD and momentum=0.9, and running on GPU for distributed training but use a static learning rate of 0.003 without the batch normalization decay as suggested in the original deeplabv3 paper https://arxiv.org/pdf/1706.05587.pdf. Can anyone provide any suggestions of how this might be improved? I know the suggestions mentioned in the paper should help but even without them, they achieve around 0.6 mIoU