'Ansible SSH user change during playbook execution causes freeze

I have a playbook that is launched from AWX with machine credentials, ie. with ssh_user_A. In this playbook, i need to perform a couple of tasks with another SSH user, ie. ssh_user_B.

In terms of credentials, I have:

  • machine credentials for ssh_user_A that are OK, I can do whatever I need
  • a public/private keys couple for ssh_user_B:
    • the public key is present on the remote host in the ~ssh_user_B/.ssh/authorized_keys file
    • if I test this public/private key with a temp machine credentials (not possible in final target) and a dummy playbook or an ad-hoc module call, it works I can do whatever I need

I change the SSH user in the playbook with these instructions:

- name: Switch to ssh_user_B user
  ansible.builtin.set_fact:
    ansible_user: "ssh_user_B"
    ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
    ansible_ssh_private_key_file: "{{file_that_contains_the_private_key}}"
  delegate_to: localhost

And the user change is triggered by a simple wait_for_connection:

- name: Just here to validate the user change
  ansible.builtin.wait_for_connection:
    timeout: 10

But unfortunately, the only thing that I get is a behavior that I do not understand:

  • with valid public/private keys, the task that is supposed to trigger the user change seems frozen at the first SSH connection attempt, nothing more happens, and nothing in the target remote host SSH logs (in these logs, I have the connection for the ssh_user_A, but absolutely nothing for the ssh_user_B one)

AWX output:

PLAYBOOK: test.yml *************************************************************
Positional arguments: test.yml
verbosity: 4
remote_user: ssh_user_A
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/awx_14019_pqypcw0i/tmpm_98ffss',)
subset: target_remote_host
extra_vars: ('@/tmp/awx_14019_pqypcw0i/tmp7493qmoq', '@/tmp/awx_14019_pqypcw0i/env/extravars')
forks: 5
1 plays in test.yml
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'

PLAY [MultiUserTest] ***********************************************************
Read vars_file './vars/main.yml'

TASK [Gathering Facts] *********************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:2
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_A

... Connection OK

TASK [Another task OK] ***************************************************
...

TASK [Task done on localhost] ********************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:57
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root

... Connection OK

TASK [Another locahost task OK] ***************************************************
...

TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
    "ansible_facts": {
        "ansible_ssh_pass": "dummy_passphrase",
        "ansible_ssh_private_key_file": ".dummy_pk_random_name",
        "ansible_user": "ssh_user_B"
    },
    "changed": false
}
Read vars_file './vars/main.yml'

TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14019_pqypcw0i/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''

... Here it's frozen, nothing more in the AWX output, nothing in the remote host SSH logs...
  • with an invalid public/private keys (ie. with additional characters, no LF at the end of the private key file, etc.), the task that is supposed to trigger the user works and ends in error. I can see that every SSH connection method is tested, and it ends in error:
TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
    "ansible_facts": {
        "ansible_ssh_pass": "dummy_passphrase",
        "ansible_ssh_private_key_file": ".dummy_pk_random_name",
        "ansible_user": "ssh_user_B"
    },
    "changed": false
}
Read vars_file './vars/main.yml'

TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14021_ypwngn_h/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14021_ypwngn_h/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''
<target_remote_host> (5, b'', b'OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021\
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched \\'final\\'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched \\'final\\'
debug2: match found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/awx_14021_ypwngn_h/cp/bc63f1e109" does not exist
debug2: resolving "target_remote_host" port 22
debug2: ssh_connect_direct
debug1: Connecting to target_remote_host [<IP>] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9984 ms remain after connect
debug1: identity file .dummy_pk_random_name type -1
debug1: identity file .dummy_pk_random_name-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to target_remote_host:22 as \\'ssh_user_B\\'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: order_hostkeyalgs: have matching best-preference key type [email protected], using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: [email protected],zlib,none
debug2: compression stoc: [email protected],zlib,none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,aes256-cbc,[email protected],aes128-ctr,aes128-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha1,[email protected],hmac-sha2-512
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: [email protected]
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: [email protected]
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:YYY/ZZZ
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <IP>
debug1: Host \\'target_remote_host\\' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug1: Will attempt key: .dummy_pk_random_name  explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner

###################################################################
# SERVER MESSAGE.                                                 #
###################################################################

debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: .dummy_pk_random_name
Load key ".dummy_pk_random_name": invalid format
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can\\'t open /dev/tty: No such device or address
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
ssh_user_B@target_remote_host: Permission denied (publickey,password).
')

... Other SSH connection method attempt

I also see the connection attempts in the remote host SSH logs, they all look like this:

Feb 28 14:25:10 target_remote_host sshd[4108095]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<IP>  user=ssh_user_B
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Connection closed by authenticating user ssh_user_B <IP> port 12345 [preauth]

What I've checked/done/excluded:

  • as said previously, the public/private key couple is OK, tested though AWX on a separate credentials machine
  • I've tested to copy the private key file on the remote folder in order to ckeck its content and see if it was well formed, and it is (here $ represent the LF):
-----BEGIN OPENSSH PRIVATE KEY-----$
...
-----END OPENSSH PRIVATE KEY-----$
  • I've created a new public/private key couple with a passphrase that only contains letters and digits, in order to exlude a problem from a special character in the passphrase (like a `), it does not change anything
  • If I do not provide a passphrase, I get a no passphrase given error from the remote host as expected

Regarding these elements, I assume that my SSH connection parameters are OK but something stucks the playbook, but I really do not understand what... Any clue/help would be appreciated.

Note that I do not (and won't) have access on the host where AWX is installed and the playbook runs.

Part of the playbook that contain the credentials change:

- name: Main private key block
  block:

  - name: Creating temp filenames
    ansible.builtin.set_fact:
      filename_private_key: ".dummy_pk_random_name"
      # Just for testing purposes, target is a key in a Vault
      temp_copy_file: "./files/temp_pk_file.txt"

  - name: Sub part 1
    block:
    ##########
    - name: Creating container file for private key
      ansible.builtin.file:
        path: "{{filename_private_key}}"
        state: touch
        mode: '600'
      changed_when: false

    - name : Putting private key into needed file
      ansible.builtin.copy:
        src: "{{temp_copy_file}}"
        dest: "{{filename_private_key}}"
        mode: '600'
      changed_when: false
    ##########
    delegate_to: localhost      

  - name: Sub part 2
    block:
    ##########
    - name: Switch to ssh_user_B user
      ansible.builtin.set_fact:
        ansible_user: "ssh_user_B"
        # The passphrase comes from a Vault
        ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
        ansible_ssh_private_key_file: "{{filename_private_key}}"
      delegate_to: localhost

    - name: Just here to validate the user change
      ansible.builtin.wait_for_connection:
        timeout: 10
    ##########

  ##########
  always:
    - name: Ensure private key is not present anymore
      file:
        path: "{{ filename_private_key }}"
        state: absent
      changed_when: false
      delegate_to: localhost
  ##########
sshansibleansible-awx


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions

VS Code / Bitbucket / SSH - Permission denied (publickey)

Jsch java ssh client is not getting disconnected after command gets executed

why password less ssh not working?

How to generate ssh keys (for github)

SSH: Connection closed by remote server

How I can directly login to rails console through ssh?

Git's famous "ERROR: Permission to .git denied to user"

Is there any way of having maven scp wagon work consistently on linux/mac/windows platform?

SSH Key - Still asking for password and passphrase

ssh-keygen - how to set an rsa key with a certain username

Forward SQL Server ODBC over SSH via putty

git pushes with wrong user from terminal

How to check if ssh-agent is already running in bash?

create jenkins ssh username with private key credential via rest xml api

ssh connect Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)