Why is '...->nil' forbidden as np.vectorize signature?

Question

When you want to write a function that does only side effect, and don't want to actually return a result array (to print something or call back methods or function on the stored objects), you can't use np.vectorize(f, signature='...->nil') (where ... is the input signature depending on the function).

Example:

def f(x, y):
  print(f'{x=} {y=}')
  return
F = np.vectorize(f, signature='(m),(m)->()') # Works but returns an array containing references to None...
F = np.vectorize(f, signature='(m),(m)->nil') # Won't work
a = np.arange(2*10).reshape((10,2))
F(a, [0,0])

Why such a limitation?

Answer 1

While I've answered a few SO about using signature in np.vectorize, I haven't paid attention to the https://numpy.org/doc/stable/reference/c-api/generalized-ufuncs.html, generalized-ufuncs link. My impression was that the signature was added to vectorize before the full development of the gufuncs standard. And I agree with @Michael that the use of nil in that documentation is unclear.

The error message when using nil is:

In [12]: F = np.vectorize(f, signature='(m),(m)->nil')
Traceback (most recent call last):
  Input In [12] in <cell line: 1>
    F = np.vectorize(f, signature='(m),(m)->nil')
  File /usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:2272 in __init__
    self._in_and_out_core_dims = _parse_gufunc_signature(signature)
  File /usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:2000 in _parse_gufunc_signature
    raise ValueError(
ValueError: not a valid gufunc signature: (m),(m)->nil

np.lib.function_base._parse_gufunc_signature is a python function, that uses re to parse the string. The parsing string is:

In [19]: np.lib.function_base._SIGNATURE
Out[19]: '^\\((?:\\w+(?:,\\w+)*)?\\)(?:,\\((?:\\w+(?:,\\w+)*)?\\))*->\\((?:\\w+(?:,\\w+)*)?\\)(?:,\\((?:\\w+(?:,\\w+)*)?\\))*$'

No 'nil' in that.

For the valid case, the parsed result is:

In [20]: np.lib.function_base._parse_gufunc_signature('(m),(m)->()')
Out[20]: ([('m',), ('m',)], [()])

In Python all functions return something, even if it is only None. That's true for print (which in Py2 was command/statement, not a function). None is a Python object, special, but not the same as nil/null in C.

So to me it's logical that F should return an array full of None, with the shape determined the broadcasted dimensions (minus the trailing m one)

So a (3,1) with a (2,) broadcasts to (3,2), and returns a (3,), along with 3 calls to f:

In [22]: F(np.array([[5],[9],[8]]),np.array([8.9]))
x=array([5]) y=array([8.9])
x=array([9]) y=array([8.9])
x=array([8]) y=array([8.9])
Out[22]: array([None, None, None], dtype=object)

The array returned by F can be ignored.

From the vectorize signatured docs:

If provided, pyfunc will be called with (and expected to return) arrays with shapes given by the size of corresponding core dimensions.

f returns something, a scalar None. In terms of signature shape that's no different than if it returned a scalar string or number.

vectorize is Python code, so can be studied. Over the years that I've followed it it's gotten more complex. Originally I found that it was always slower than the equivalent python iteration or list comprehension. In newer tests I've found that it scales better, so for very large inputs, it can be faster. It is still nowhere like true, compiled, numpy vectorization.

In simple cases vectorize actually uses frompyfunc, which returns an object dtype array. It then converts that output to the otypes output. frompyfunc is compiled (though it does not compile your pyfunc), and is faster, by itself, than vectorize.

With a signature, vectorize takes a different route. It's docs warn that this is slower.

The core of _vectorize_call_with_signature is a ndindex iteration:

for index in np.ndindex(*broadcast_shape):
    results = func(*(arg[index] for arg in args))

In short, np.vectorize is a not a performance tool, especially with signature. But it still may be useful, especially when you have several input arguments, and want the full broadcasting power.