Servlet ignores Content-Length and uses Transfer-Encoding: chunked based on User-Agent

Question

I want to compress response body in javax.servlet.Filter. Here is my code

byte[] bytes =  // compressing response body
response.addHeader("Content-Encoding", "gzip");
response.addHeader("Content-Length", String.valueOf(bytes.length));
response.setContentLength(bytes.length);
response.setBufferSize(bytes.length * 2);
ServletOutputStream output = response.getOutputStream();
output.write(bytes);
output.flush();
output.close();

But actual response I see in Chrome Dev Tool is

Accept-Ranges: bytes
Cache-Control: max-age=2592000
Content-Type: application/javascript;charset=UTF-8
Date: Fri, 14 Dec 2018 15:34:25 GMT
Last-Modified: Tue, 09 Oct 2018 13:42:54 GMT
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked

I do not expect Transfer-Encoding: chunked, because I declare "Content-Length". I wrote a simple test on java

URLConnection connection = new URL("http://127.0.0.1:8081/js/ads.js").openConnection();
connection.addRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
connection.addRequestProperty("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7");
connection.addRequestProperty("Cache-Control", "no-cache");
connection.addRequestProperty("Connection", "keep-alive");
connection.addRequestProperty("Host", "127.0.0.1:8081");
connection.addRequestProperty("Pragma", "no-cache");
connection.addRequestProperty("Upgrade-Insecure-Requests", "1");
connection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"); 
connection.connect();
connection.getHeaderFields().forEach((s, strings) ->
        System.out.println(s + ":" + String.join(",", strings)));

and here is what I found:

• if I comment setting "User-Agent" header or change "User-Agent" to any other value then I get response with "Content-Length"
• if "User-Agent" points on Chrome then I get Transfer-Encoding: chunked.

I debugged up to sun.nio.ch.SocketChannel#write method and it gets correct ByteBuffers with Content-Length header values.

I cant undestand where is this magic transormation to chunked happening?

Update

The strange thing is that I write gziped bytes into Socket (I'm sure as I debugged up to call of native method write in SocketChannel implementation). But URLConnection returns my unzipped byte array with Chrome's User-Agent and correct gziped byte array if I dont specify User-Agent header or put some random string. SO seems like magic happening somewhere in Windows socket implementation.

Answer 1

Shown Code

I would assume that your code shown works and that the problem is somewhere else.

Setup

• Windows 10
• Tomcat 7.0.92
• Chrome 71.0.3578.98

Testcase

I tried to create a small filter example to be able to try out your test code.

By the way, a compression filter more suitable for productive use can be found in the examples supplied with Tomcat (webapps\examples\WEB-INF\classes\compressionFilters).

import java.io.*;
import java.util.zip.GZIPOutputStream;
import javax.servlet.*;
import javax.servlet.http.*;

public class CompressionFilter  implements Filter {

    public void init(FilterConfig filterConfig) { }
    public void destroy() { }

    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
        final HttpServletRequest request = (HttpServletRequest) servletRequest;
        final HttpServletResponse response = (HttpServletResponse) servletResponse;

        ResponseWrapper wrapper = new ResponseWrapper(response);
        filterChain.doFilter(request, wrapper);
        byte[] uncompressed = wrapper.getBytes();

        byte[] bytes = compress(uncompressed);
        response.addHeader("Content-Encoding", "gzip");
        response.addHeader("Content-Length", String.valueOf(bytes.length));
        response.setContentLength(bytes.length);
        //response.setBufferSize(bytes.length * 2);
        ServletOutputStream output = response.getOutputStream();
        output.write(bytes);
        output.flush();
        output.close();

        System.out.println("request to:" +  request.getServletPath()
                + " size changed from: " + uncompressed.length
                + " to " + bytes.length);
    }

    private byte[] compress(byte[] bytes) throws IOException {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        GZIPOutputStream gzipOutputStream = new GZIPOutputStream(baos);
        gzipOutputStream.write(bytes);
        gzipOutputStream.close();
        return baos.toByteArray();
    }


    public class ResponseWrapper extends HttpServletResponseWrapper {
        private ByteArrayOutputStream output = new ByteArrayOutputStream();
        private PrintWriter printWriter = null;

        ResponseWrapper(HttpServletResponse response) {
            super(response);
        }

        byte[] getBytes() {
            if (printWriter != null)
                printWriter.flush();
            return output.toByteArray();
        }

        public PrintWriter getWriter() {
            if (printWriter == null)
                printWriter = new PrintWriter(output);
            return printWriter;
        }

        public ServletOutputStream getOutputStream() {
            return new ServletOutputStream() {
                private WriteListener writeListener;
                public boolean isReady() { return true; }
                public void setWriteListener(WriteListener writeListener) { this.writeListener  = writeListener; }
                public void write(int b) {
                    output.write(b);
                    if(writeListener != null)
                        writeListener.notify();
                }
            };
        }
    }

}

Result

Three test cases with static html, a JSP generated page and Servlet generated page with some dummy content are shown like this in Chrome's developer tools:

a) using static html

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"108-1545775482914"
Last-Modified: Tue, 25 Dec 2018 22:04:42 GMT
Content-Encoding: gzip
Content-Type: text/html
Content-Length: 97
Date: Tue, 25 Dec 2018 22:34:41 GMT

b) JSP generated

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Encoding: gzip
Content-Type: text/html
Content-Length: 38
Date: Tue, 25 Dec 2018 22:49:17 GMT

c) Servlet generated

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Encoding: gzip
Content-Type: text/html
Content-Length: 65
Date: Tue, 25 Dec 2018 22:49:43 GMT

With this setup there is no Transfer-Encoding: chunked. So maybe the reason for this chunked header can be found somewhere else?