Mocking HTTPS responses in Go

Question

I'm trying to write tests for a package that makes requests to a web service. I'm running into issues probably due to my lack of understanding of TLS.

Currently my test looks something like this:

func TestSimple() {
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(200)
        fmt.Fprintf(w, `{ "fake" : "json data here" }`)
    }))
    transport := &http.Transport{
        Proxy: func(req *http.Request) (*url.URL, error) {
            return url.Parse(server.URL)
        },
    }
    // Client is the type in my package that makes requests
    client := Client{
        c: http.Client{Transport: transport},
    }

    client.DoRequest() // ...
}

My package has a package variable (I'd like for it to be a constant..) for the base address of the web service to query. It is an https URL. The test server I created above is plain HTTP, no TLS.

By default, my test fails with the error "tls: first record does not look like a TLS handshake."

To get this to work, my tests change the package variable to a plain http URL instead of https before making the query.

Is there any way around this? Can I make the package variable a constant (https), and either set up a http.Transport that "downgrades" to unencrypted HTTP, or use httptest.NewTLSServer() instead?

(When I try to use NewTLSServer() I get "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")

Answer 1

The reason you're getting the error http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037 is because https requires a domain name (not an IP Address). Domain names are SSL certificates are assigned to.

Start the httptest server in TLS mode with your own certs

cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
    log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}

server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort

To provide a valid SSL certificate for a connection are the options of:

1. Not supplying a cert and key
2. Supplying a self-signed cert and key
3. Supplying a real valid cert and key

No Cert

If you don't supply your own cert, then an example.com cert is loaded as default.

Self-Signed Cert

To create a testing cert can use the included self-signed cert generator at $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"

You'll get x509: certificate signed by unknown authority warnings because it's self-signed so you'll need to have your client skip those warnings, by adding the following to your http.Transport field:

 TLSClientConfig: &tls.Config{InsecureSkipVerify: true}

Valid Real Cert

Finally, if you're going to use a real cert, then save the valid cert and key where they can be loaded.

---

The key here is to use server.URL = https://sub.domain.com to supply your own domain.

Answer 2

From Go 1.9+ you can use func (s *Server) Client() *http.Client in the httptest package:

Client returns an HTTP client configured for making requests to the server. It is configured to trust the server's TLS test certificate and will close its idle connections on Server.Close.

Example from the package:

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "net/http/httptest"
)

func main() {
    ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()

    client := ts.Client()
    res, err := client.Get(ts.URL)
    if err != nil {
        log.Fatal(err)
    }

    greeting, err := io.ReadAll(res.Body)
    res.Body.Close()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%s", greeting)
}