'Data Preprocessing with tf.data.service not working

I use TPU in colab:

try: # detect TPUs
  tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() # TPU detection
  strategy = tf.distribute.TPUStrategy(tpu)
except ValueError: # detect GPUs
  strategy = tf.distribute.MirroredStrategy() # for GPU or multi-GPU machines

Then start DispatchServer and WorkerServer:

d_config = tf.data.experimental.service.DispatcherConfig(port=5000)
dispatcher = tf.data.experimental.service.DispatchServer()  #d_config
#w_config = tf.data.experimental.service.WorkerConfig(port=5001,dispatcher_address=dispatcher.target.split("://")[1], worker_address='localhost:5001')
w_config =tf.data.experimental.service.WorkerConfig(dispatcher_address=dispatcher.target.split("://")[1])
worker = tf.data.experimental.service.WorkerServer(w_config)
dispatcher.start()

And use it in apply in dataset loaded through TFRecordDataset:

return augmented.repeat().apply(tf.data.experimental.service.distribute(processing_mode="parallel_epochs", service=dispatcher.target)).batch(batch_size).prefetch(AUTO) #repeat().shuffle(2048) #augmented

But, when create model with strategy.scope() it hangs in an infinite loop:

with strategy.scope(): # creating the model in the TPUStrategy scope places the model on the TPU
   model = create_model()
model.summary()

What am i doing wrong? I want to prepare a dataset outside the TPU in 40 CPUs in colab.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source