Merging two datasets with different image dimensions for segmentation

Question

I am working on the task of Semantic segmentation wherein, I want to combine two datasets namely Freiburg forest dataset (with varying image resolution, let's say I fixed it to 896x480) and Yamaha CMU (YCOR) dataset with fixed (i.e. 1024x544) image resolution. I have two questions:

1. Should the smaller image dataset be resized to the bigger one or vice-versa i.e. 1024x544 to 896x480 ?
2. What other things do I have to take care when merging multiple datasets in the context of semantic segmentation ?

Thanks in advance.