getting issue in querying from clickhouse using golang script for max_query_size exceeded

Question

Getting issue in querying from clickhouse getting errror. In one go I am passing more than 200 thousand information but getting error from clickhouse

*Syntax error: failed at position 262140 (''suleuha.net''): 'suleuha.net', 'diagonsys.com', 'scala.com', 'omegahns.com', 'jagrannewmedia.com', '7ty7.com', 'bitseducampus.ac.in', 'creationspo.com', 'mafoirandstod.com', 'c. Max query size exceeded: ''suleuha.net''

func errorLog(err error){
 log.Errorf(err)
}
func connect(host, port) (*sql.DB, error){
connect, err := sql.Open("clickhouse", "tcp://"+host+":"+port+ "?debug=false")
    connect.SetMaxOpenConns(5)
    connect.SetMaxIdleConns(5)
    return connect, err
}

func query() *sql.Rows{
query := fmt.Sprint("SELECT entry FROM db.suppressdomains WHERE entry IN ('" + domains + "') and scat != 'ignore'")
    result, err := clickdb.Query(query)
    errorLog(err)
    return result
}

Answer 1

I faced the same problem on the Java Clickhouse driver. I don't know how is made the Go driver, but it is most likely that you reached the limit of the HTTP protocol.

This is because Clickhouse HTTP interface is reading the request from the URL part of the HTTP request. The header has a maximum size.

In my case, I was trying to INSERT INTO ... (..) with thousands of columns. A fix in the Clickhouse Java driver permitted to put the whole request in the HTTP body instead of the HTTP header. The HTTP POST body has no limit.

In you case, I suggest to change your approach, sending each time all those strings seems a bad design. You can insert those values in a table in Clickhouse.

But also create a ticket on the github of the Go project. The request have to be in the body, not in the header.

Example here : https://clickhouse.com/docs/en/interfaces/http/#:~:text=%24%20echo%20%27SELECT%201%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1%0A%0A%24%20echo%20%27SELECT%201%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%3Fquery%3D%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1%0A%0A%24%20echo%20%271%27%20%7C%20curl%20%27http%3A//localhost%3A8123/%3Fquery%3DSELECT%27%20%2D%2Ddata%2Dbinary%20%40%2D%0A1