Why am I getting a copy instead of a reference when flattening a list of ndarrays?

Question

And how do I get references to each element in the ndarrays?

The goal is to take a list of variable shaped ndarrays containing primitives, use ravel() on all the ndarrays and merge them into a list who elements are all references to the elements in the list of ndarrays.

Here is my attempt at this. To be clear, weights is a list of ndarrays, and I want weights_unraveled to be a one dimensional list containing references to every element in each ndarray.

def unravel_weights(weights):
    weights_unraveled = []
    for w in weights:
        weights_unraveled.extend(w.ravel())
    return weights_unraveled

I know that w.ravel() returns a one dimensional list with references to elements of w and assigning it to another variable does not make a copy. I have tested it. Does passing the value to a function make a copy? Does the returning of a value from a function make a copy? I doubt it because assignment did not.

If extend() is making copies, how do I stop it, or how do I merge the list of references returned by w.ravel() into a single list containing references to every element in each ndarray?

This is the code that manifests the undesired behavior.

    flat_child_weights = unravel_weights(child_weights)
    flat_parent_weights = unravel_weights(weights1) 
    for i in range(len(flat_child_weights)):
        change = random.choice([True, False])
        if change:
            flat_child_weights[i] = flat_parent_weights[i]

Is a copy being made when assigning the return value of unravel_weights? I also doubt it.

Debugging has verified that flat_child_weights and flat_parent_weights are indeed one dimensional lists, but they are copies of child_weights and weights1 respectively. To be clear, flat_child_weights[i] = flat_parent_weights[i] does change the elements of flat_child_weights, but the changes are not reflected in child_weights.

How can I get the elements of child_weights to reflect changes in flat_child_weights?

Answer 1

You get references instead of copies from your numpy calls because they return views and not copies, unlike the array.extend. Here is a method that will do the same, though it requires some upfront work. You deleted your old question, so I cannot reference that code.

If I recall, you started with a list of ndarrays, which could not be directly turned into a 2D ndarray because the ndarrays are different lengths. This will replace your unravel_weights.

1. Create a 1D ndarray with all the weights flattened:

flat_child_weights = np.concatenate([weights.ravel() for weights in child_weights)

2. Key step to link your output to the work on the flattened weights: recreate child_weights from views of flat_child_weights, something like:

child_weights_new = []
index = 0
for weights in child_weights:
    child_weights_new.append(flat_child_weights[index:index+weights.size()].reshape(weights.shape))
    index += weights.size()

Now you can run through your work on the flat_child_weights, and child_weights_new will be updated (but not child_weights), so return child_weights_new.