How to send gzip compressed body

Question

We want to send compressed body by gzip HTTP request in chai-http in mocha.

let chai = require('chai');
let chaiHttp = require('chai-http');
const zlib = require('zlib');

chai.use(chaiHttp);


object = {
    "content": {
        "data": {
            "key": "testval"
        }
    }
};

const objStr = JSON.stringify(object);
const objBuf = Buffer.from(objStr, "utf-8");
const bodyContent = zlib.gzipSync(objBuf);
const bodyLen = Buffer.byteLength(bodyContent, 'utf-8');

chai.request("http://serverurl")
    .post('/path')
    .set('Content-Type', 'application/json')
    .set('Content-Encoding', 'gzip')
    .set('Content-Length', bodyLen) 
    .set('Accept-Encoding', 'gzip')
    .send(bodyContent)
    .end((err, res) => {
        expect(err).to.be.null;
        expect(res).to.have.status(200);

        done();
    });

However, we met the error Error: incorrect header check at Zlib.zlibOnError on the server-side. Is there anything am I missing?

Answer 1

Looks like it's superagent, it's breaking the encoding, because it stringifies everything that is not a string and has application/json, so instead of a Buffer:

<Buffer 1f 8b 08 00 00 00 

it's sending a JSON:

{"type":"Buffer","data":[31,139,8,0,...

breaking the encoding, causing the error.

It happens here:

https://github.com/visionmedia/superagent/blob/2f4af99bdee91b9fc63c6563abae367a4b3f1a8e/src/node/index.js#L988-L996

Adding !Buffer.isBuffer(data) or something there might work.

Until then, try a workaround, for example with http:

const http = require('http')


const options = {
  hostname: 'serverurl',
  port: 443,
  path: '/path',
  method: 'POST',
  headers: {
    'Content-Encoding': 'gzip',
    'Content-Type': 'application/json',
    'Accept-Encoding': 'gzip',
    'Content-Length': bodyLen
  }
};

const req = http.request(options, (res) => {

  //res.setEncoding('utf8');
  //res.on('data', function(chunk) {
  //  console.log('data', chunk);
  //});

  expect(res).to.have.status(200);
  done();

});
req.on('error', function(e) {
  console.log('err: ' + e.message);
  //expect(err).to.be.null;
  process.exit();
});

req.write(bodyContent);
req.end();

Answer 2

I was able to make this function for superagent by overriding the serialize function when application/json is in play. If the buffer is identified as a gzip buffer, then let it pass through the serialization unmodified.

var isgzipBuffer = require('@stdlib/assert-is-gzip-buffer')

superagent.serialize['application/json'] = function(...args) {
    if (isgzipBuffer(args[0]) === false) {
        return JSON.stringify(...args)
    }
    return args[0]
}

I also added this to update the Content-Encoding:

       if (isgzipBuffer(bodyParam) === true) {
            headerParams['Content-Encoding'] = 'gzip'
        }

This was done within the context of a codegen javascript client so exact syntax for setting headers may vary.