InternalError: Failed copying input tensor from device:CPU:0 to device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized

Question

linux_user@server_machine:~/spatial/tuning_neighbors_7$ python3 *hyper*.py
Traceback (most recent call last):
  File "tuner_hyperband.py", line 209, in <module>
    load_data_k(
  File "tuner_hyperband.py", line 127, in load_data_k
    tz = tf.convert_to_tensor(data_z, np.float32)
  File "/home/linux_user/.local/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/linux_user/.local/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 106, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.
linux_user@server_machine:~/spatial/tuning_neighbors_7$

What is the main reason for this error?

• Is it because of low disk space?

• Is it because of low RAM memory?

• Is it because of an engaged GPU?

How to resolve this?

Answer 1

It is because of GPU RAM memory is full (at least for my case). Is that the full message? Because in my case it was quite longer (and clear it is a memory issue). See here.

Some debugging tips:

1. You can use tf.config.experimental.get_memory_info('GPU:0')['current'] / 10**9 to see the memory usage until a point.
2. At some point, tensorflow should tell you your available memory in a message like Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9058 MB memory
3. If you are using numpy arrays that are converted into tensors or used in fit, evaluate or equivalent functions you can use npa.size * npa.itemsize / 10**9 to know the size needed to create the new tensor.