'How to connect to Hyperledger Fabric Gateway Service (new in HF 2.4) with TLS enabled?

I have a Hyperlegder Fabric network set-up which is operating fine as long as I don't use new Fabric-Gateway SDK (https://hyperledger-fabric.readthedocs.io/en/release-2.4/gateway.html). I upgraded my network from 2.3.1 to 2.4.1 and wanted to try the new SDK, but cannot connect to the Peer. Below I give some details of my configuration.

Peer-base docker service:

  peer-base:
    image: hyperledger/fabric-peer
    environment:
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=${COMPOSE_PROJECT_NAME}_basic
      - FABRIC_LOGGING_SPEC=info:gateway,comm,comm.grpc,comm.grpc.server=debug
      - CORE_CHAINCODE_LOGGING_LEVEL=info
      - CORE_PEER_LISTENADDRESS=0.0.0.0:7051
      - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
      - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/peer/msp
      - CORE_PEER_GOSSIP_USELEADERELECTION=true
      - CORE_PEER_GOSSIP_ORGLEADER=false
      - CORE_PEER_PROFILE_ENABLED=true
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/peer/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/peer/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/peer/tls/ca.crt
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=***
      - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=***
      - CORE_METRICS_PROVIDER=prometheus
      - CORE_OPERATIONS_LISTENADDRESS=0.0.0.0:7055
      - CORE_PEER_GATEWAY_ENABLED=true
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: peer node start
    volumes:
        - ./config:/etc/hyperledger/configtx
        - /var/run/:/host/var/run/
    networks:
      - basic
    restart: always

After migrating to 2.4.1, I added CORE_PEER_GATEWAY_ENABLED=true.

The peer docker service, which extends the peer-base:

  peer0.org1.tcash.com:
    container_name: peer0.org1.tcash.com
    extends:
      file: docker-compose-org1-base.yaml
      service: peer-base
    environment:
      - CORE_PEER_ID=peer0.org1.tcash.com
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_ADDRESS=peer0.org1.tcash.com:7051
      - CORE_PEER_CHAINCODEADDRESS=peer0.org1.tcash.com:7052
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=test2.tcash.sigmacomp.pl:7051
      - CORE_PEER_GOSSIP_ENDPOINT=test2.tcash.sigmacomp.pl:7051
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb0.org1.tcash.com:5984
    ports:
      - 7051:7051
      - 7053:7053
      - 7055:7055
    volumes:
      - ./crypto-config/peerOrganizations/org1.tcash.com/peers/peer0.org1.tcash.com:/etc/hyperledger/peer
      - ./persistence/peer0.org1.tcash.com/:/var/hyperledger/production
    depends_on:
      - couchdb0.org1.tcash.com
    extra_hosts:
      - orderer0.tcash.com:146.59.17.169
      - orderer1.tcash.com:146.59.17.169
      - orderer2.tcash.com:146.59.17.169
      - orderer3.tcash.com:146.59.17.169
      - orderer4.tcash.com:146.59.17.169
      - peer2.org1.tcash.com:51.195.202.90
      - peer3.org1.tcash.com:51.195.202.90
      - peer4.org1.tcash.com:51.68.172.244
      - peer5.org1.tcash.com:51.68.172.244

No changes have been made here during migration to 2.4.1.

I can see in the Peer logs, that new gateway service has been started:

2022-01-21 12:34:09.177 UTC 0023 INFO [nodeCmd] serve -> Starting peer with Gateway enabled
2022-01-21 12:34:09.177 UTC 0024 INFO [nodeCmd] serve -> Starting peer with ID=[peer0.org1.tcash.com], network ID=[dev], address=[peer0.org1.tcash.com:7051]
2022-01-21 12:34:09.177 UTC 0025 INFO [nodeCmd] func7 -> Starting profiling server with listenAddress = 0.0.0.0:6060
2022-01-21 12:34:09.177 UTC 0026 INFO [nodeCmd] serve -> Started peer with ID=[peer0.org1.tcash.com], network ID=[dev], address=[peer0.org1.tcash.com:7051]

After deploying the network, I try to run the transaction with the following code (NodeJS):

'use strict';

const fs = require('fs');
const crypto = require('crypto');
const grpc =require('@grpc/grpc-js');
const { connect, signers } = require('@hyperledger/fabric-gateway');


async function main() {
    // Main try/catch block
    try {
        const credentials = fs.readFileSync('walletOffline/user.cert.pem');
        const identity = { mspId: 'Org1MSP', credentials };

        const privateKeyPem = fs.readFileSync('walletOffline/user.key.pem');
        const privateKey = crypto.createPrivateKey(privateKeyPem);
        const signer = signers.newPrivateKeySigner(privateKey);

        const ccpJSON = fs.readFileSync('connection.json');
        const ccp = JSON.parse(ccpJSON);
        const peerName = ccp.organizations.org1.peers[0];
        const peerAddress = ccp.peers[peerName].url.replace('grpcs://', '');
        const tlsCACert = ccp.peers[peerName].tlsCACerts.pem;
        const grpcOptions = ccp.peers[peerName].grpcOptions;

        const tlsRootCert = Buffer.from(tlsCACert);
        const tlsCredentials = grpc.credentials.createSsl(tlsRootCert);

        const client = new grpc.Client(peerAddress, tlsCredentials, grpcOptions);
        
        const gateway = connect({identity, signer, client});
        const network = gateway.getNetwork('tcashchannel');
        const contract = network.getContract('tcash');
        const result = await contract.evaluateTransaction('queryAccountState', '100', '');
        console.log('result: ' + result);
    } catch (error) {
        console.log('Error: ' + error);
        console.log(error.stack);
    }
}

main();

As you can see, I am extracting connection parameters from the JSON connection profile. This connection profile I use with the 'old' HF Node SDK and it's working without issues. However running this code gives me the following error from contract.evaluateTransaction() after 120 seconds timeout:

GatewayError: 14 UNAVAILABLE: failed to create new connection: context deadline exceeded
    at newGatewayError (/Users/michaliwanicki/git/tcash/tcash-application/node_modules/@hyperledger/fabric-gateway/dist/gatewayerror.js:40:12)
    at Object.callback (/Users/michaliwanicki/git/tcash/tcash-application/node_modules/@hyperledger/fabric-gateway/dist/client.js:81:67)
    at Object.onReceiveStatus (/Users/michaliwanicki/git/tcash/tcash-application/node_modules/@grpc/grpc-js/build/src/client.js:180:36)
    at Object.onReceiveStatus (/Users/michaliwanicki/git/tcash/tcash-application/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)
    at Object.onReceiveStatus (/Users/michaliwanicki/git/tcash/tcash-application/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /Users/michaliwanicki/git/tcash/tcash-application/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78
    at processTicksAndRejections (internal/process/task_queues.js:77:11)

I can also see the corresponding entry in the peer logs:

2022-01-21 14:24:14.961 UTC 007e INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Evaluate grpc.peer_address=178.183.68.178:54151 error="rpc error: code = Unavailable desc = failed to create new connection: context deadline exceeded" grpc.code=Unavailable grpc.call_duration=2m0.00087636s

There are no errors or warnings in the peer log.

EDIT: After switching logging level to DEBUG and filtering it out, I came across the following part:

2022-01-27 13:38:19.217 UTC 67af DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 69.892651ms server=PeerServer remoteaddress=178.183.68.178:58755
2022-01-27 13:38:19.356 UTC 67b0 DEBU [lockbasedtxmgr] newQueryExecutor -> constructing new query executor txid = [407898ef-0004-4f25-be10-b603a2aaf919]
2022-01-27 13:38:19.357 UTC 67b1 DEBU [statecouchdb] GetState -> GetState(). ns=, key=CHANNEL_CONFIG_ENV_BYTES
2022-01-27 13:38:19.358 UTC 67b2 DEBU [lockbasedtxmgr] Done -> Done with transaction simulation / query execution [407898ef-0004-4f25-be10-b603a2aaf919]
2022-01-27 13:38:19.358 UTC [grpc] WarningDepth -> DEBU 02f [core]Adjusting keepalive ping interval to minimum period of 10s
2022-01-27 13:38:19.359 UTC [grpc] InfoDepth -> DEBU 030 [core]parsed scheme: ""
2022-01-27 13:38:19.359 UTC [grpc] InfoDepth -> DEBU 031 [core]scheme "" not registered, fallback to default scheme
2022-01-27 13:38:19.359 UTC [grpc] InfoDepth -> DEBU 032 [core]ccResolverWrapper: sending update to cc: {[{test2.tcash.sigmacomp.pl:8051  <nil> 0 <nil>}] <nil> <nil>}
2022-01-27 13:38:19.360 UTC [grpc] InfoDepth -> DEBU 033 [core]ClientConn switching balancer to "pick_first"
2022-01-27 13:38:19.360 UTC [grpc] InfoDepth -> DEBU 034 [core]Channel switches to new LB policy "pick_first"
2022-01-27 13:38:19.360 UTC [grpc] InfoDepth -> DEBU 035 [core]Subchannel Connectivity change to CONNECTING
2022-01-27 13:38:19.360 UTC [grpc] InfoDepth -> DEBU 036 [core]pickfirstBalancer: UpdateSubConnState: 0xc002ed2b30, {CONNECTING <nil>}
2022-01-27 13:38:19.361 UTC [grpc] InfoDepth -> DEBU 037 [core]Channel Connectivity change to CONNECTING
2022-01-27 13:38:19.360 UTC [grpc] InfoDepth -> DEBU 038 [core]Subchannel picks a new address "test2.tcash.sigmacomp.pl:8051" to connect
2022-01-27 13:38:19.370 UTC [grpc] InfoDepth -> DEBU 039 [core]Subchannel Connectivity change to TRANSIENT_FAILURE
2022-01-27 13:38:19.370 UTC [grpc] InfoDepth -> DEBU 03a [core]pickfirstBalancer: UpdateSubConnState: 0xc002ed2b30, {TRANSIENT_FAILURE connection closed}
2022-01-27 13:38:19.370 UTC [grpc] InfoDepth -> DEBU 03b [core]Channel Connectivity change to TRANSIENT_FAILURE
2022-01-27 13:38:19.370 UTC [grpc] InfoDepth -> DEBU 03c [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"

EDIT 2: I noticed that there are some errors in peer logs belonging to the other peers in the network (not the one which is called by the client application and running the Gateway service). It seems that there is a problem with establishment of TLS between peers when using Gateway SDK:

2022-02-10 14:36:24.934 UTC 24b0 DEBU [gossip.comm] func1 -> Got message: GossipMessage: Channel: , nonce: 0, tag: CHAN_OR_ORG state_info_pull_req: Channel MAC:23b92135be842b052b823a7c87853436fb579040416405d4fdfd0b6db0aa02d9, Envelope: 39 bytes, Signature: 0 bytes
2022-02-10 14:36:24.934 UTC 24b1 DEBU [gossip.gossip] handleMessage -> Entering, 54.37.226.59:7051 5c2af6d536100ada4e7f1829978c7f0163a6589f47f44207aa51a84987fe6a5b sent us GossipMessage: Channel: , nonce: 0, tag: CHAN_OR_ORG state_info_pull_req: Channel MAC:23b92135be842b052b823a7c87853436fb579040416405d4fdfd0b6db0aa02d9, Envelope: 39 bytes, Signature: 0 bytes
2022-02-10 14:36:24.935 UTC 24b2 DEBU [gossip.gossip] handleMessage -> Exiting
2022-02-10 14:36:24.942 UTC 24b3 ERRO [core.comm] ServerHandshake -> Server TLS handshake failed in 15.541µs with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.24.0.1:36394
2022-02-10 14:36:24.942 UTC [grpc] WarningDepth -> DEBU 04e [core]grpc: Server.Serve failed to complete security handshake from "172.24.0.1:36394": tls: first record does not look like a TLS handshake

I suspect that there is some piece of configuration which is required for this feature to work, which I am missing. I will appreciate if anyone can help me find it.

node.jshyperledger-fabrichyperledger


Solution 1:[1]

It looks like the gateway peer is failing to connect to another endorsing peer in the network. Are you seeing any gossip communication between the peers in the logs? Try reducing the dialTimeout to something less than the endorsementTimeout in the core.yaml and see if it connects to the other peers.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Andrew Coleman

Related Questions