How to shuffle tensorflow dataset without buffer?

Question

I meet problem when I try to shuffle the ds from the code below:

#ds_1 length=20000, ds_2 length=2000
ds_1 = tf.data.Dataset.zip((image_ds1, label_ds1))
ds_2 = tf.data.Dataset.zip((image_ds2, label_ds2))  #labels in label_ds2 are all the same
ds = ds_1.concatenate(ds_2)

Using ds.shuffle(buffer_size) with a large buffer size could accomplish this task but it takes large RAM at the same time. The ds.shuffle() would always shuffle the ds when ds is called which is not needed for me. I just want to shuffle ds once and save RAM. Thanks for helps in advance.