How can I handle TensorFlow sessions to train multiple Keras models at the same time?

Question

I need to train multiple Keras models at the same time. I'm using TensorFlow backend. Problem is, when I try to train, say, two models at the same time, I get Attempting to use uninitialized value.

The error is not really relevant, the main problem seems to be that Keras is forcing the two models to be created in the same session with the same graph so it conflicts.

I am a newbie in TensorFlow but my gut feeling is that the answer is pretty straightforward : you would have to create a different session for each Keras model and train them in their own session. Could someone explain me how it would be done ?

I really hope it is possible to solve this problem while still using Keras and not coding everything in pure TensorFlow. Any workaround would be appreciated too.

Answer 1

You are right, Keras automatically works with the default session. You could use tf.compat.v1.keras.backend.get_session() or tf.compat.v1.keras.backend.set_session(sess) to manually set the global Keras session (see documentation).

For instance:

sess1 = tf.Session()
tf.compat.v1.keras.backend.set_session(sess1)
# Train your first Keras model here ...

sess2 = tf.Session()
tf.compat.v1.keras.backend.set_session(sess2)
# Train your second Keras model here ...

Answer 2

I train multiple models in parallel by using pythons multiprocessing, https://docs.python.org/3.4/library/multiprocessing.html.

I have a function that takes two parameters, an input queue and an output queue, this function runs in each process. The function has the following structure:

def worker(in_queue, out_queue):
    import keras

    while True:
        parameters = in_queue.get()
        network_parameters = parameters[0]
        train_inputs = parameters[1]
        train_outputs = parameters[2]
        test_inputs = parameters[3]
        test_outputs = parameters[4]

        build the network based on the given parameters

        train the network

        test the network if required

        out_queue.put(result)

From the main python script start as many processes (and create as many in and out queues) as required. Add jobs to a worker by calling put on its in queue and get the results by calling get on its out queue.