'problem with dimensions when using keras text_dataset_from_directory i get (None,) and can't lode it in to a model
I am using keras.utils.text_dataset_from_directory (see code). when I reach model.fit I get a warning about input dimentions and a error (that I think has something to do with the output).
code:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import Rescaling
MAX_VAL = 0.5
num_classes = 2
train_ds = keras.utils.text_dataset_from_directory(
directory='.../training_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
validation_ds = keras.utils.text_dataset_from_directory(
directory='.../validation_data/',
labels='inferred',
label_mode='categorical',
class_names=None,
batch_size=32,
max_length=None,
shuffle=True,
seed=None,
validation_split=None,
subset=None,
follow_links=False)
inputs = keras.Input(shape=(None,))
x = layers.Reshape((-1, 1))(inputs)
x = Rescaling(scale=1.0 / MAX_VAL)(x)
x = layers.Dense(32, activation="softmax")(x)
outputs = layers.Dense(num_classes, activation="softmax")(x)
model = keras.Model(inputs=inputs, outputs=outputs)
model.summary()
keras.utils.plot_model(model, "my_first_model.png")
model.compile(optimizer='Adam', loss='categorical_crossentropy')
history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
output:
Epoch 1/10
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (None,).
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-3656f32a72a9> in <module>
1 model.compile(optimizer='Adam', loss='categorical_crossentropy')
----> 2 history = model.fit(train_ds, epochs=10, validation_data=validation_ds)
~\AppData\Roaming\Python\Python38\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\func_graph.py in autograph_handler(*args, **kwargs)
1145 except Exception as e: # pylint:disable=broad-except
1146 if hasattr(e, "ag_error_metadata"):
-> 1147 raise e.ag_error_metadata.to_exception(e)
1148 else:
1149 raise
ValueError: in user code:
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1021, in train_function *
return step_function(self, iterator)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1010, in step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "...\Python\Python38\site-packages\keras\engine\training.py", line 1000, in run_step **
outputs = model.train_step(data)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 860, in train_step
loss = self.compute_loss(x, y, y_pred, sample_weight)
File "...\Python\Python38\site-packages\keras\engine\training.py", line 918, in compute_loss
return self.compiled_loss(
File "...\Python\Python38\site-packages\keras\engine\compile_utils.py", line 201, in __call__
loss_value = loss_obj(y_t, y_p, sample_weight=sw)
File "...\Python\Python38\site-packages\keras\losses.py", line 141, in __call__
losses = call_fn(y_true, y_pred)
File "...\Python\Python38\site-packages\keras\losses.py", line 245, in call **
return ag_fn(y_true, y_pred, **self._fn_kwargs)
File "...\Python\Python38\site-packages\keras\losses.py", line 1789, in categorical_crossentropy
return backend.categorical_crossentropy(
File "...\Python\Python38\site-packages\keras\backend.py", line 5083, in categorical_crossentropy
target.shape.assert_is_compatible_with(output.shape)
ValueError: Shapes (None, 2) and (None, 1, 2) are incompatible
The inputs to the model are 1d vectors (of length ~27000) saved in .txt files:
0.101471743,0.099917953,0.103334975,0.099364908,0.099035715,...,0.097369999,0.099680934
readin dataset frome dir formated as:
/training_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
/validation_data/
...class_a/
......a_text_1.txt
......a_text_2.txt
...class_b/
......b_text_1.txt
......b_text_2.txt
How can I get the dimensions right?
EDIT:
I saved the data as a .jpg file and loaded it using image_dataset_from_directory. this fixed the issue. but I would still like to understand why I cant get the data from .txt files properly. (I lose a lot of data transferring from float data to 8bit int in .jpg. and using image_dataset_from_directory requires all img size to be the same length, I wand my data to be different sizes).
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
