Golang ssh client timeout not working as expected

Question

I am writing a Golang ssh/sftp client which connects to a sftp server with a slowness in connecting and writing files, using golang.org/x/crypto/ssh package. I need to set Connection timeout and SO timeout (as we do in Java JSCH library).

First to achieve Connection timeout I was using ssh.ClientConfig.Timeout, but only worked for nanosecond and microsecond values, not for milliseconds and above, where I needed to set 5 seconds. According to the API doc also I assume ssh.ClientConfig.Timeout is used only for TCP socket connection creation and ssh handshake is not included there.

So then I tried net.Conn.SetDeadline() and it was for end-to-end connection creation + writing file + connection closing. Since this is also not fine, tried net.Conn.SetWriteDeadline() which looks like SO timeout (applied in TCP packet level) but timeout error is not appeared just after the duration elasped, instead comes out after the server's late reply or subsequent write operation starts.

So can someone please show the correct way of setting Connection timeout and SO timeout in Golang ssh package or tell whether this is supported or not?