'Slow data reading from Google BigTable

Airflow 1.10.14 and composer 1.15.2, google Bigtable in GCP

I'm getting this issue

{taskinstance.py:1152} ERROR - <_MultiThreadedRendezvous of RPC that terminated with
    status = StatusCode.ABORTE
    details = "Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)
    debug_error_string = "{"created":"@1649401290.125577144","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)","grpc_status":10}
>
Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 980, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/home/airflow/gcs/plugins/minutebardata.py", line 55, in execut
    tk.get_data(dt=self.dt
  File "/home/airflow/gcs/plugins/scraper/ticker.py", line 46, in get_dat
    df0 = self.dbhook._load_all_minubebar_tickers(ticker
  File "/home/airflow/gcs/plugins/scraper/db_wrapper.py", line 583, in _load_all_minubebar_ticker
    dfs = [self.process(row, ticker) for row in rows if row is not None
  File "/home/airflow/gcs/plugins/scraper/db_wrapper.py", line 583, in <listcomp
    dfs = [self.process(row, ticker) for row in rows if row is not None
  File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 485, in __iter_
    response = self._read_next_response(
  File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 474, in _read_next_respons
    return self.retry(self._read_next, on_error=self._on_error)(
  File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_fun
    on_error=on_error
  File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_targe
    return target(
  File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 470, in _read_nex
    return six.next(self.response_iterator
  File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 416, in __next_
    return self._next(
  File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 803, in _nex
    raise sel
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with
    status = StatusCode.ABORTE
    details = "Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)
    debug_error_string = "{"created":"@1649401290.125577144","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)","grpc_status":10}

I use the following approach

        row_set = RowSet()
        row_set.add_row_range_from_keys(
            start_key=ticker + b"#" + startdt,
            end_key=ticker + b"#" + enddt)

        rows = table.read_rows(
            row_set=row_set)

This code works properly when I run this locally. However, when I try to run this in GCP I get this issue. Could you give me any hints to find the solution?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source