'Hyperledger Go SDK Client error connection is in TRANSIENT_FAILURE remote error: tls: bad certificate

I know there are several errors like this one, but I see most of them are with the test-network, using a local environment and my case is different.

I have a blockchain running using k8s it used to be running in V1.4 and we migrated it to V2.3 so the certificates were generated without a SAN but to make them work with the new version we are using the env variable GODEBUG="x509ignoreCN=0" I can connect to it using a console client to do changes in the channel and configuration, and I can connect with a node js client and execute contracts, I want to connect using a Go client for a different app but I'm having trouble with the config.

I'm getting a TRANSIENT_FAILURE error in the Go Client and a tls: bad certificate in the peer but the certificates are the same that work for the other environments, the config.yaml I'm using is based on my configuration from the test-network that's also working.

Something weird though, is that the configPath doesn't seem to work and I need to add the users individualy to make it run.

So this is my config.yaml

---
name: network-company
version: 1.0.0
client:
  organization: company
  logging:
    level: debug
  connection:
    timeout:
      peer:
        endorser: "300"
  cryptoconfig:
    path: /home/company

  credentialStore:
    path: /home/company/users
    cryptoStore:
      path: /home/company/users/[email protected]/msp

  BCCSP:
    security:
      enabled: true
      default:
        provider: "SW"
      hashAlgorithm: "SHA2"
      softVerify: true
      level: 256

  tlsCerts:
    systemCertPool: false

    client:
      cert:
        path: /home/company/users/[email protected]/tls/cert.pem
      key:
        path: /home/company/users/[email protected]/tls/key.pem

organizations:
  company:
    mspid: company
    cryptoPath: /home/company/users/[email protected]/msp
    users:
      admin:
        cert:
          path: /home/company/users/[email protected]/msp/signcerts/cert.pem
        key:
          path: /home/company/users/[email protected]/msp/keystore/key.pem
    peers:
      - peer1.mycompany.com
      - peer2.mycompany.com

peers:
  peer1.mycompany.com:
    url: grpcs://peer1.mycompany.com:443
    tlsCACerts:
      path: /home/company/users/[email protected]/tls/ca.pem

    grpcOptions:
      ssl-target-name-override: peer1.mycompany.com
      hostnameOverride: peer1.mycompany.com
      keep-alive-time: 10s
      keep-alive-timeout: 20s
      keep-alive-permit: false
      fail-fast: false

    httpOptions:
      verify: false

  peer2.mycompany.com:
    url: grpcs://peer2.mycompany.com:443
    tlsCACerts:
      path: /home/company/users/[email protected]/tls/ca.pem

    grpcOptions:
      ssl-target-name-override: peer2.mycompany.com
      hostnameOverride: peer2.mycompany.com
      keep-alive-time: 10s
      keep-alive-timeout: 20s
      keep-alive-permit: false
      fail-fast: false
      allow-insecure: true

    httpOptions:
      verify: false

channels:
  mychannel:
    peers:
      peer1.mycompany.com:
        endorsingPeer: true
        chaincodeQuery: true
        ledgerQuery: true
        eventSource: true
      peer2.mycompany.com:
        endorsingPeer: true
        chaincodeQuery: true
        ledgerQuery: true
        eventSource: true

    policies:
      queryChannelConfig:
        minResponses: 1
        maxTargets: 1
        retryOpts:
          attempts: 5
          initialBackoff: 500ms
          maxBackoff: 5s
          backoffFactor: 2.0

entityMatchers:
  peer:
    - pattern: peer1.mycompany.com
      urlSubstitutionExp: peer1.mycompany.com:443
      sslTargetOverrideUrlSubstitutionExp: peer1.mycompany.com
      mappedHost: peer1.mycompany.com
    - pattern: peer2.mycompany.com
      urlSubstitutionExp: peer2.mycompany.com:443
      sslTargetOverrideUrlSubstitutionExp: peer2.mycompany.com
      mappedHost: peer2.mycompany.com

These are my certificates paths:

/home/company/users/[email protected]
    + msp
        + admincerts/<empty>
        + signcerts/cert.pem
        + cacerts/ca.pem
        + intermediatecerts/ca.pem
        + tlscacerts/ca.pem
        + tlsintermediatecerts/ca.pem
        + keystore/key.pem
    + tls
        + ca.pem
        + cert.pem
        + key.pem

These are the logs from the client:

[fabsdk/core] 2022/01/04 14:34:09 UTC - sw.getOptsByConfig -> DEBU Initialized SW cryptosuite
[fabsdk/fab] 2022/01/04 14:34:09 UTC - n/a -> DEBU KeyStore opened at [/home/company/users/[email protected]/msp/keystore]...done
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU Client is: {Organization:company TLSCerts:{Client:{Key:{Path:/home/company/users/[email protected]/tls/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/[email protected]/tls/cert.pem Pem: bytes:[]}}}}
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU channels are: map[mychannel:{Orderers:[] Peers:map[peer1.mycompany.com:{EndorsingPeer:true ChaincodeQuery:true LedgerQuery:true EventSource:true} peer2.mycompany.com:{EndorsingPeer:true ChaincodeQuery:true LedgerQuery:true EventSource:true}] Policies:{QueryChannelConfig:{MinResponses:1 MaxTargets:1 RetryOpts:{Attempts:5 InitialBackoff:500ms MaxBackoff:5s BackoffFactor:2 RetryableCodes:map[]}} Discovery:{MinResponses:0 MaxTargets:0 RetryOpts:{Attempts:0 InitialBackoff:0s MaxBackoff:0s BackoffFactor:0 RetryableCodes:map[]}} Selection:{SortingStrategy: Balancer: BlockHeightLagThreshold:0} EventService:{ResolverStrategy: MinBlockHeightResolverMode: Balancer: BlockHeightLagThreshold:0 PeerMonitor: ReconnectBlockHeightLagThreshold:0 PeerMonitorPeriod:0s}}}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU organizations are: map[company:{MSPID:company CryptoPath:/home/company/users/[email protected]/msp Users:map[admin:{Key:{Path:/home/company/users/[email protected]/msp/keystore/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/[email protected]/msp/signcerts/cert.pem Pem: bytes:[]}}] Peers:[peer1.mycompany.com peer2.mycompany.com] CertificateAuthorities:[]}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU orderers are: map[]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadEndpointConfiguration -> DEBU peers are: map[peer1.mycompany.com:{URL:grpcs://peer1.mycompany.com:443 GRPCOptions:map[allow-insecure:true fail-fast:false hostnameoverride:peer1.mycompany.com keep-alive-permit:false keep-alive-time:10s keep-alive-timeout:20s ssl-target-name-override:peer1.mycompany.com] TLSCACerts:{Path:/home/company/users/[email protected]/tls/ca.pem Pem: bytes:[]}} peer2.mycompany.com:{URL:grpcs://peer2.mycompany.com:443 GRPCOptions:map[allow-insecure:true fail-fast:false hostnameoverride:peer2.mycompany.com keep-alive-permit:false keep-alive-time:10s keep-alive-timeout:20s ssl-target-name-override:peer2.mycompany.com] TLSCACerts:{Path:/home/company/users/[email protected]/tls/ca.pem Pem: bytes:[]}}]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).compileMatchers -> DEBU Matchers are: {matchers:map[]}
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).compileMatchers -> DEBU Entity matchers are not configured
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadTLSClientCerts -> DEBU Reading pk from config, unable to retrieve from cert: Could not find matching key for SKI: Failed getting key for SKI [[91 7 147 74 144 157 11 46 234 18 159 199 54 134 208 140 16 147 185 225 158 7 112 91 182 170 49 46 139 190 194 99]]: key with SKI 5b07934a909d0b2eea129fc73686d08c1093b9e19e07705bb6aa312e8bbec263 not found in /home/company/users/[email protected]/msp/keystore
[fabsdk/fab] 2022/01/04 14:34:09 UTC - fab.(*EndpointConfig).loadPrivateKeyFromConfig -> DEBU pk read from config successfully
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU Client is: {Organization:company Logging:{Level:debug} CryptoConfig:{Path:/home/company} TLSCerts:{Client:{Key:{Path:/home/company/users/[email protected]/tls/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/[email protected]/tls/cert.pem Pem: bytes:[]}} SystemCertPool:false} CredentialStore:{Path:/home/company/users CryptoStore:{Path:/home/company/users/[email protected]/msp}}}
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU organizations are: map[company:{MSPID:company CryptoPath:/home/company/users/[email protected]/msp Users:map[admin:{Key:{Path:/home/company/users/[email protected]/msp/keystore/key.pem Pem: bytes:[]} Cert:{Path:/home/company/users/[email protected]/msp/signcerts/cert.pem Pem: bytes:[]}}] Peers:[peer1.mycompany.com peer2.mycompany.com] CertificateAuthorities:[]}]
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).loadIdentityConfigEntities -> DEBU certificateAuthorities are: map[]
[fabsdk/msp] 2022/01/04 14:34:09 UTC - msp.(*IdentityConfig).compileMatchers -> DEBU Matchers are: {matchers:map[]}
[fabsdk] 2022/01/04 14:34:09 UTC - fabsdk.initSDK -> DEBU SDK initialized successfully
SDK created
resourceManagerClient created
[fabsdk/fab] 2022/01/04 14:34:09 UTC - chconfig.(*params).SetChConfigRefreshInterval -> DEBU RefreshInterval: 1m30s
[fabsdk/util] 2022/01/04 14:34:09 UTC - lazyref.WithRefreshInterval.func1 -> DEBU Checking refreshIntervalSetter
[fabsdk/fab] 2022/01/04 14:34:09 UTC - peer.(*peerEndorser).ProcessTransactionProposal -> DEBU Processing proposal using endorser: peer2.mycompany.com:443
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).DialContext -> DEBU DialContext: peer2.mycompany.com:443
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).createConn -> DEBU creating connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).createConn -> DEBU storing connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - comm.(*CachingConnector).removeConn -> DEBU removing connection [peer2.mycompany.com:443]
[fabsdk/fab] 2022/01/04 14:34:09 UTC - txn.SendProposal.func1 -> DEBU Received error response from txn proposal processing: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE
[fabsdk/common] 2022/01/04 14:34:09 UTC - retry.(*RetryableInvoker).Invoke -> DEBU Failed with err [queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE] on attempt #1. Checking if retry is warranted...
[fabsdk/common] 2022/01/04 14:34:09 UTC - retry.(*RetryableInvoker).Invoke -> DEBU ... retry for err [queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE] is NOT warranted after 1 attempt(s).
2022/01/04 14:34:09 error inicializando el SDK: error creando el cliente del canal: event service creation failed: could not get chConfig cache reference: QueryBlockConfig failed: QueryBlockConfig failed: queryChaincode failed: Transaction processing for endorser [peer2.mycompany.com:443]: Endorser Client Status Code: (2) CONNECTION_FAILED. Description: dialing connection on target [peer2.mycompany.com:443]: connection is in TRANSIENT_FAILURE

These are the logs from the peer:

2022-01-04 15:23:56.740 UTC [core.comm] ServerHandshake -> ERRO 335 Server TLS handshake failed in 7.869808ms with error remote error: tls: bad certificate server=PeerServer remoteaddress=10.250.37.211:33666
2022-01-04 15:23:56.740 UTC [grpc] WarningDepth -> DEBU 336 [core]grpc: Server.Serve failed to complete security handshake from "10.250.37.211:33666": remote error: tls: bad certificate
  • A couple of weird things: I need to add the users to the organization, some how it's not able to extract the users from the configPath and I get an error user not found.
  • It's looking for the tls private key in the msp/keystore folder and of course that isn't there
  • I tried adding the GODEBUG="x509ignoreCN=0" but it doesn't change the result
  • I generated an extra TLS certificates just for the client adding the SAN to them as "email:[email protected]" and mapped it to the configuration with the same result.
  • I have tried adding and removing the entityMatchers with the same result.

I also did a tcp dump on it server answers to the Client hello with Server hello and then fails with a 400 bad request error.

I have used the configurations referenced from here and here but everything looks good to me.

I'm running out of ideas, is my configuration right? why do I need to force the clients and configPath doesn't work? why is it looking for the tls key in the msp keystore? but mostly why do I get a tls bad certificate if the certificates are good and pass an openssl verify against the ca.pem?

Thanks

UPDATE

This is my certificate info using openssl x509 -in certificate.pem -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            56:00:01:ba:18:d9:cf:ed:a7:00:b9:af:b3:00:01:00:01:ba:18
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = CO, L = Medellin, O = mycompany, CN = CA My Company, emailAddress = [email protected]
        Validity
            Not Before: Jan  5 12:36:08 2022 GMT
            Not After : Jan  5 12:36:08 2023 GMT
        Subject: C = CO, ST = ANTIOQUIA, L = MEDELLIN, O = mycompany, OU = admin, CN = [email protected]
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:d0:c8:3b:20:2f:8f:3b:91:72:f8:71:27:30:cf:
                    b5:6e:38:e3:5f:e6:1b:42:77:fd:00:f2:8c:1b:ae:
                    6c:2e:7d:50:03:75:d5:03:db:72:d5:33:27:f3:65:
                    c1:2a:09:72:6a:d5:cc:c5:45:ca:e5:e5:8c:65:4b:
                    f2:51:dc:dd:ac
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: 
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, Microsoft Encrypted File System, E-mail Protection, TLS Web Client Authentication, Any Extended Key Usage
            X509v3 Subject Alternative Name: 
                DNS:mycompany.com
            X509v3 Subject Key Identifier: 
                9E:C5:4B:E8:67:B1:84:BF:67:E9:BE:A3:DE:B5:EB:E3:3B:4E:D3:D0
            X509v3 Authority Key Identifier: 
                keyid:40:54:B4:93:16:92:20:E3:DB:AA:7A:A3:13:06:EC:65:4F:83:B3:58

            X509v3 CRL Distribution Points: 

                Full Name:
                URI:http://crl.epm.com.co/CRL/CAEPMClaseI.crl
                URI:http://crl2.epm.com.co/CRL/CAEPMClaseI.crl

            Authority Information Access: 
                OCSP - URI:http://ocsp.epm.com.co/OCSP
                CA Issuers - URI:http://crl.epm.com.co/CRL/CAEPMClaseI.crt

            1.3.6.1.4.1.311.21.7: 
                0-.%+.....7.....-...K..=...>...k8...,...A..d.. 
            S/MIME Capabilities: 
......0...+....0050...*.H..
..*.H..
    Signature Algorithm: sha256WithRSAEncryption
        6f:d6:85:b1:08:cd:07:78:3d:92:50:85:14:3a:f0:2e:1f:84:
        10:7c:1d:15:10:0f:9f:66:e7:1a:41:63:12:83:0c:7c:04:c7:
        c8:f4:8b:97:d9:6e:28:78:cb:9b:af:89:6b:1a:f3:06:40:1e:
        6f:90:81:8f:f9:03:ff:92:5a:de:8c:7c:35:f9:21:00:fd:85:
        c0:73:48:71:c0:82:7a:17:6d:09:ce:b6:03:5a:c5:1f:4f:47:
        6d:95:94:a3:e2:cd:19:03:3d:85:a0:7e:5e:cf:a5:55:32:9f:
        28:cf:d4:10:c8:42:57:a1:9b:cc:9a:4b:82:75:86:15:92:b9:
        ef:d2:b2:1c:5d:a9:fe:ff:d4:f2:83:9d:31:46:50:26:ea:78:
        88:9c:72:0a:1e:81:42:88:7b:04:4b:4b:2b:86:3d:ed:f9:4d:
        8f:5e:47:1b:0b:8e:a9:8a:ed:ed:8a:09:92:58:6b:14:6e:aa:
        50:c1:ae:2f:75:41:5b:e9:28:2b:85:9f:70:0a:d3:27:64:44:
        f4:f6:61:99:a0:db:90:05:8c:af:09:fa:36:a5:67:3a:b8:f5:
        b3:06:fe:ba:50:aa:1f:2e:43:af:a6:3a:81:01:d1:26:cc:61:
        3b:c3:d9:b5:50:d7:27:61:f6:3e:a3:1d:99:0f:f1:37:20:d4:
        ea:6d:59:a9:56:09:09:74:90:d4:52:64:69:b5:ad:09:b8:67:
        74:ca:52:76:0a:37:0e:29:e0:5b:5a:bc:24:42:a1:14:77:1f:
        72:f5:e9:f8:64:17:2e:fc:57:e1:d3:74:3a:1d:a5:fb:f4:c3:
        a3:1e:97:f3:ba:d6:33:6a:24:d5:79:d7:9c:07:b4:9d:d0:52:
        24:dc:0d:d5:a1:37:42:25:09:3b:96:a5:91:19:6c:7e:58:71:
        99:ff:04:14:4d:ea:25:57:23:58:2b:44:84:f9:c3:c4:f6:a6:
        ed:81:75:0a:07:e4:ab:bf:74:ec:e6:8c:9e:b8:75:82:47:20:
        75:64:94:45:93:2f:ec:8b:8f:2c:19:c8:b0:29:a0:4c:f7:05:
        6d:c8:3d:54:78:9e:ac:17:1c:19:5e:3c:1a:ba:93:db:1c:41:
        ab:09:e2:d7:e3:11:87:eb:b7:38:e6:cf:85:1e:8a:75:9d:8c:
        68:44:5d:d6:60:e7:91:52:ec:08:5b:26:7b:a4:42:d6:49:fc:
        3c:74:6c:96:23:99:86:6b:6b:57:89:94:ad:08:98:35:1d:38:
        07:09:ae:c3:8c:1f:3c:36:94:a9:da:74:73:ff:5e:c1:0d:f7:
        45:a6:85:94:1d:34:46:56:fd:82:3f:8d:c0:d0:a1:20:1b:76:
        d3:2e:f8:a7:6c:3f:25:f4
hyperledger-fabrictls1.2hyperledger-fabric-sdk-go


Solution 1:[1]

So the issue seemed to be with the TLS certificates from the peers, they where generated before go 1.16 and didn't had the SANs in them, and the error seemed to be getting reflected to the go SDK. @yacovm thank you for your help figuring it out.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions

How to serve a Vue application over HTTPS for local development

TLS 1.2 - Alert Level - Fatal - Description Protocol Version

MsSQL driver which could not establish a secure connection by using Secure Sockets Layer (SSL)?

TLS 1.2 + Java 1.6 + BouncyCastle

Enabling TLS-1.2 on embedded Jetty

Set up Google cloud certificates for Postgresql database ( hostname mismatch )

Java HttpsURLConnectionImpl.getOutputStream() error : java.net.SocketException: recv() failed, errno = 104 Connection reset by peer

Sending Email using Outlook / Office365 with Laravel - Failed to authenticate on SMTP server

Forcing .NET application to use TLS 1.2 or later

How to enable TLS on Teradata Express

"SSL.CERTIFICATE_VERIFY_FAILED: Self signed certificate in certificate chain" in python with PROTOCOL_TLS_CLIENT

NuGet Unable to load the service index for source Authentication failed

Certificate Chain does not work in mockserver

how does an SSL certificate chain bundle arranged?

Hyperledger channel creation on orderer Error