'How to fix cardinality error in my CNN model
I am currently working on a CNN model where my inputs are CSV's with 1080 rows and 4 attributes (columns). I have all the CSVs under their own categories directory, for example /Category A/..., /Category B/... etc.
At start I have created two arrays:
X = []
Y = []
and then in a for loop going thru all the directories I read the contents of the CSV in a 1080x4 shape and put it in my X array like this:
(I have confirmed the values are correctly read and in correct shape)
X.append(pandas.read_csv(itemPath).values)
and I add the category of the item in my Y array right after, so the order of X items are aligned with order of Y items (categories of X values).
Y.append(cat)
Then here is my model, albeit I got it mostly from a Kaggle example, I just wanted to see if it works at all, but I think the values should not give me errors.
model = Sequential()
model.add(Convolution2D(108,4,4, padding= 'valid', data_format='channels_last',input_shape=(1080, 4, 1),activation= 'relu' ))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128, activation= 'relu' ))
model.add(Dense(50, activation= 'relu' ))
model.add(Dense(10, activation= 'softmax' ))
model.compile(loss= 'categorical_crossentropy' , optimizer= 'adam' , metrics=[ 'accuracy' ])
and here is where I train my model:
model.fit(X, Y,
epochs=20,
batch_size= 20)
score = model.evaluate(X, Y, batch_size=128)
I get the following error:
ValueError Traceback (most recent call last)
Input In [103], in <cell line: 1>()
----> 1 model.fit(X, Y,
2 epochs=20,
3 batch_size= 20)
4 score = model.evaluate(X, Y, batch_size=128)
File d:\github\bitirmecnn\venv\lib\site-packages\keras\utils\traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
65 except Exception as e: # pylint: disable=broad-except
66 filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67 raise e.with_traceback(filtered_tb) from None
68 finally:
69 del filtered_tb
File d:\github\bitirmecnn\venv\lib\site-packages\keras\engine\data_adapter.py:1655, in _check_data_cardinality(data)
1651 msg += " {} sizes: {}\n".format(
1652 label, ", ".join(str(i.shape[0])
1653 for i in tf.nest.flatten(single_data)))
1654 msg += "Make sure all arrays contain the same number of samples."
-> 1655 raise ValueError(msg)
ValueError: Data cardinality is ambiguous:
x sizes: 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080, 1080
y sizes: 510
Make sure all arrays contain the same number of samples.
Here is the model summary:
Model: "sequential_12"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_14 (Conv2D) (None, 270, 1, 108) 1836
dropout_6 (Dropout) (None, 270, 1, 108) 0
flatten_6 (Flatten) (None, 29160) 0
dense_18 (Dense) (None, 128) 3732608
dense_19 (Dense) (None, 50) 6450
dense_20 (Dense) (None, 10) 510
=================================================================
Total params: 3,741,404
Trainable params: 3,741,404
Non-trainable params: 0
_________________________________________________________________
Where am I going wrong? The X clearly has 510 entries with dimensions of 1080x4, what should I do to fix this?
Also do you guys have any advice on the model values? I am pretty new into this, I have 510 items and 6 categories
Solution 1:[1]
I used the following code to generate random data and was able to replicate the issue with it.
# Generating random X and y values
x=tf.random.uniform((1080,4,1))
X=[]
for i in range(510):
X.append(x)
y=tf.random.uniform((510,))
Reshape X to overcome the error as follows:
# Generating random X and y values
x=tf.random.uniform((1080,4,1))
X=[]
for i in range(510):
X.append(x)
# Reshaping X to overcome the error
X=tf.reshape(X,(510,1080,4,1))
y=tf.random.uniform((510,))
The units in the output layer must be 6, as your data has 6 categories. Replace the following line
model.add(Dense(10, activation= 'softmax' ))
with this:
model.add(Dense(6, activation= 'softmax' ))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |