Salt Error while bringing up minion for multi-master

Question

I'm getting some errors when I start salt minions:

    Error while bringing up minion for multi-master. Is master at salt-server00001.nix.us.companyworld.company.com responding?
2019-10-16 11:57:27,935 [salt.minion      ][ERROR   ][34679] Error while bringing up minion for multi-master. Is master at usazrcncm10286.nix.us.kworld.kpmg.com responding?
2019-10-16 11:57:37,990 [salt.crypt       ][ERROR   ][34679] The masters sent its public-key signature, but signature verification is not enabled on the minion. Either enable signature verification on the minion or disable signing the public key on the master!
2019-10-16 11:57:37,990 [salt.crypt       ][CRITICAL][34679] The Salt Master server's public key did not authenticate!
The master may need to be updated if it is a version of Salt lower than 2017.7.2, or
If you are confident that you are connecting to a valid Salt Master, then remove the master public key and restart the Salt Minion.

I am NOT setting up multi-master.

I have accepted the key for the server on the salt master.

I've opened up the salt ports on the firewall.

But the salt server still isn't responding.

This is my salt versions report:

salt --versions-report
Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Mar 30 2018, 15:31:59)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.3.1

System Versions:
           dist: redhat 6.10 Santiago
         locale: UTF-8
        machine: x86_64
        release: 2.6.32-754.2.1.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.10 Santiago

Why am I seeing this problem and how do I correct it?

Answer 1

Look for errors in the /var/log/salt/minion

Then, issue salt '*' service.status and check if the master is running.

Do a check of the salt-master service systemctl status salt-master

In my case, the salt-master failed to start:

? Salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/salt-master.service.d
           ??salt-master-opts-managed.conf
   Active: failed (Result: timeout) since Wed 2020-07-08 10:59:22 -03; 54min ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
  Process: 5780 ExecStart=/usr/bin/salt-master (code=killed, signal=TERM)
 Main PID: 5780 (code=killed, signal=TERM)

Jul 08 10:57:52 v321p601 systemd[1]: Starting The Salt Master Server...
Jul 08 10:58:26 v321p601 salt-master[5780]: [INFO    ] Setting up the Salt Master
Jul 08 10:59:22 v321p601 systemd[1]: salt-master.service start operation timed out. Terminating.
Jul 08 10:59:22 v321p601 systemd[1]: Failed to start The Salt Master Server.
Jul 08 10:59:22 v321p601 systemd[1]: Unit salt-master.service entered failed state.
Jul 08 10:59:22 v321p601 systemd[1]: salt-master.service failed.

Try restarting the service with systemctl start salt-master. If the error persists, run netstat -lnp on the master and see if the Salt ports are there and listening.

In my case, I've reissued the command to start the salt-master service after a while and it went online:

? salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/salt-master.service.d
           ??salt-master-opts-managed.conf
   Active: active (running) since Wed 2020-07-08 12:19:31 -03; 4min 55s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 27609 (salt-master)
   CGroup: /system.slice/salt-master.service
           ??27609 /usr/bin/python /usr/bin/salt-master
           ??27618 /usr/bin/python /usr/bin/salt-master
           ??27820 /usr/bin/python /usr/bin/salt-master
           ??27824 /usr/bin/python /usr/bin/salt-master
           ??27839 /usr/bin/python /usr/bin/salt-master
           ??27840 /usr/bin/python /usr/bin/salt-master
           ??27841 /usr/bin/python /usr/bin/salt-master
           ??27842 /usr/bin/python /usr/bin/salt-master
           ??27849 /usr/bin/python /usr/bin/salt-master
           ??27850 /usr/bin/python /usr/bin/salt-master
           ??27851 /usr/bin/python /usr/bin/salt-master
           ??27852 /usr/bin/python /usr/bin/salt-master
           ??27853 /usr/bin/python /usr/bin/salt-master

Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Authentication accepted from 4b11df09-8c57-4745-ac5a-1ab3da854566
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Authentication request from eeea44c1-5d2e-430d-8217-50f2517e587b
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Authentication accepted from eeea44c1-5d2e-430d-8217-50f2517e587b
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Authentication request from 48892d9e-78b1-43c2-a0d4-479ac4907432
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Authentication accepted from 48892d9e-78b1-43c2-a0d4-479ac4907432
Jul 08 12:21:14 v321p601 salt-master[27609]: [INFO    ] Got return from 70a62d5b-31df-403d-b0f5-3e90d1b9fd7d for job 20200708122114319236
Jul 08 12:21:18 v321p601 salt-master[27609]: [INFO    ] Authentication request from 39f50b14-89d4-427b-bd46-39b11cada7da
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO    ] Authentication request from 81f9b89c-a7d3-4264-b156-f2b0470590c3
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO    ] Authentication accepted from 39f50b14-89d4-427b-bd46-39b11cada7da
Jul 08 12:21:27 v321p601 salt-master[27609]: [INFO    ] Authentication accepted from 81f9b89c-a7d3-4264-b156-f2b0470590c3

References: saltstack cheatsheet saltmaster not running properly?