hashicorp same vault binary started on different linux fails to work with same etcd

Question

Having fight with weird issue. I one of my env vault is not able to work stable with etcd as storage.

So here is story. I have etcd server 3.5 version installed. Works perfectly with etcdctl tool

When I run on one system. Ubuntu 20.04.2 LTS

I am having issues with JWT token. In vault logs I am having

{"level":"warn","ts":"2022-03-29T16:23:27.614Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002b7500/#initially=[https://etcd-server:2379]","attempt":99,"error":"rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"}

But some records vault is able to read. So sometimes it thinks JWT is OK.

When I copy same binary on Fedora 35 and run there I do not have an issue.

From etcd logs I can extract JWT token in both cases. And can verify it using JWT tools. Both correct and signature is OK as well.

Etcd token is runnning with

name: ETCD_AUTH_TOKEN
value: jwt,priv-key=jwt-token.pem,sign-method=RS256,ttl=10m

Interesting thing that if I will run same on other Fedora 35 box I am having JWT issue as well.

If I am setting ETCD_AUTH_TOKEN into 'simple' then as expected all OS starts working without issues.

So I am really lost why first of it does not work with JWT on all. And second why it really works only on one system. Binary of vault is static and downloaded from Hashicorp site as iti is. So does not depends on system libs. Time is synced on all systems.

Will be appreciate any help and ideas. Thank you