Simple neural network - how to store weights?

Question

I recently started learning Python and am trying to implement my first neural network. My goal is to write a function that generates a neural net with a variable amount of layers and nodes. All necessary information for that is stored in layerStructure (e.g.: First layer has four nodes, third layer has three nodes).

import numpy as np

#Vector of input layer
input = np.array([1,2,3,4])

#Amount of nodes in each layer
layerStructure = np.array([len(input),2,3])

#Generating empty weight matrix container
weightMatrix_arr = np.array([])

#Initialsing random weights matrices
for ii in range(len(layerStructure[0:-1])):
    randmatrix = np.random.rand(layerStructure[ii+1],layerStructure[ii])
    print(randmatrix)

The code above generates the following output:

[[0.6067148  0.66445212 0.54061231 0.19334004]
 [0.22385007 0.8391435  0.73625366 0.86343394]]
[[0.61794333 0.9114799 ]
 [0.10626486 0.95307027]
 [0.50567023 0.57246852]]

My first attempt was to store each random weight matrix in a container array called weightMatrix_arr. However, since the shape of individual matrices varies, I cannot use np.append() to store them all in the matrix container. How can I save these matrices in order to access them during the backpropagation?

Answer 1

You can use a list instead of an np.array:

#Generating empty weight LIST container
weightMatrixes = []

#Initialsing random weights matrices
for ii in range(len(layerStructure[0:-1])):
    randmatrix = np.random.rand(layerStructure[ii+1],layerStructure[ii])
    weightMatrixes.append(randmatrix)
    print(randmatrix)

Otherwise you can set the weightMatrix_arr dtype to object: :

#Generating empty weight LIST container
weightMatrixes = np.array([], dtype=object)

#Initialsing random weights matrices
for ii in range(len(layerStructure[0:-1])):
   randmatrix = np.random.rand(layerStructure[ii+1],layerStructure[ii])
   weightMatrixes = np.append(weightMatrixes, randmatrix)

Note both ways you can't access the inner layer indexes without accessing the layer matrix:

weightMatrixes[layer, 0, 3] # ERROR
weightMatrixes[layer][0, 3] # OK

Answer 2

If memory consumption is not a problem, you can shape all layers as a longest one, and just ignore extra cells according to a layerStructure value.

Answer 3

I used a python dictionary to store the weights for each hidden layer with layer number as a key to the dictionary, 
so that while retrieval is easy to access the weights I,e simple and clean use the dictionary to store the model weights, 
its doesn't matter the shape of weights. below is a snippet of code.


"""def generate_weights(layers):
    Weights={}
    for i  in range(1,len(layers)):
        w0=2*np.random.random((layers[i-1],layers[i]))-1
        Weights[i-1] = w0
    return Weights

generate_weights([3,4,2])"""