Recursive lookup problem in Puppet and Hiera

Question

Puppet agent fails with the following error:

    Error: Could not retrieve catalog from remote server: 
    Error 500 on SERVER: Server Error: Lookup of key 'my.net.hosts.agent01' failed: 
    Recursive lookup detected in [my.net.needs_proxy, my.net.hosts.agent01] on node agent01

My Hiera setup is as follows. Node-specific file:

# environments/test/data/nodes/agent01.yaml
---
my:
  net:
    host:
      ip_address: "%{lookup('my.net.hosts.agent01')}"
      
    needs_proxy: false

and the "subsystem" specific file:

# environments/test/data/subsystems/main.yaml
---
my:
  net:
    hosts:
      agent01: "192.168.0.1"
      agent02: "192.168.0.2"
      agent03: "192.168.0.3"

    needs_proxy: false

hiera.yaml is the following:

# environments/test/hiera.yaml
---
version: 5
hierarchy:
  - name: "Per-node data"
    path: "nodes/%{::trusted.certname}.yaml"
  - name: "Per-subsystem data"
    path: "subsystems/%{::subsystem}.yaml"
  - name: "Common and fallback data"
    path: "common.yaml"

AFAICT it is correctly set-up and both files above are considered in the hierarchy for node agent01.

The expected behaviour is that the lookup interpolation %{lookup('my.net.hosts.agent01')} in the node-specific file (agent1.yaml) picks up the value as found in the subsystem-specific file (main.yaml). But that does not happen and the above error is produced.

I can't see any obvious mistake, perhaps I am missing something. Any help is appreciated.

(edited to add hiera.yaml)

Answer 1

AFAICT it is correctly set-up and both files above are considered in the hierarchy for node agent01.

You need to appreciate that Hiera is a flat key / value store that happens to support values with structured types. The key / subkey syntax that you demonstrate using with the lookup interpolation function might lead a person to mistake the data model for some kind of tree, but such a view leads to incorrect expectations.

In this case, the Hiera data presented defines two values for key my. The structure of those values is an altogether secondary consideration. You having not overridden the default lookup strategy for that key, you get a priority lookup for the whole value of that key. There are then two issues with your data definition:

1. The highest-priority of those values, which is the only one that matters in this case, does not provide the data that you are trying to retrieve via the lookup function.

2. As the diagnostic says, you have a recursive lookup: in order to load the value associated with key my, the lookup function within requires Hiera to first load the value of key my.

You could resolve (1) by specifying deep merging for my, but I do not expect that to resolve (2).

You need to restructure the data, bearing in mind Hiera's flat(-ish) data model. It's no particular problem for one of the items to be or contain a lookup table of IP addresses, but that can be referenced only externally or from the values of different items.