'Slow data reading from Google BigTable
Airflow 1.10.14 and composer 1.15.2, google Bigtable in GCP
I'm getting this issue
{taskinstance.py:1152} ERROR - <_MultiThreadedRendezvous of RPC that terminated with
status = StatusCode.ABORTE
details = "Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)
debug_error_string = "{"created":"@1649401290.125577144","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)","grpc_status":10}
>
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 980, in _run_raw_tas
result = task_copy.execute(context=context
File "/home/airflow/gcs/plugins/minutebardata.py", line 55, in execut
tk.get_data(dt=self.dt
File "/home/airflow/gcs/plugins/scraper/ticker.py", line 46, in get_dat
df0 = self.dbhook._load_all_minubebar_tickers(ticker
File "/home/airflow/gcs/plugins/scraper/db_wrapper.py", line 583, in _load_all_minubebar_ticker
dfs = [self.process(row, ticker) for row in rows if row is not None
File "/home/airflow/gcs/plugins/scraper/db_wrapper.py", line 583, in <listcomp
dfs = [self.process(row, ticker) for row in rows if row is not None
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 485, in __iter_
response = self._read_next_response(
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 474, in _read_next_respons
return self.retry(self._read_next, on_error=self._on_error)(
File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_fun
on_error=on_error
File "/opt/python3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_targe
return target(
File "/opt/python3.6/lib/python3.6/site-packages/google/cloud/bigtable/row_data.py", line 470, in _read_nex
return six.next(self.response_iterator
File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 416, in __next_
return self._next(
File "/opt/python3.6/lib/python3.6/site-packages/grpc/_channel.py", line 803, in _nex
raise sel
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with
status = StatusCode.ABORTE
details = "Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)
debug_error_string = "{"created":"@1649401290.125577144","description":"Error received from peer ipv4:142.251.6.95:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Error while reading table 'projects/pyproject/instances/bigtable-02/tables/mytable' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)","grpc_status":10}
I use the following approach
row_set = RowSet()
row_set.add_row_range_from_keys(
start_key=ticker + b"#" + startdt,
end_key=ticker + b"#" + enddt)
rows = table.read_rows(
row_set=row_set)
This code works properly when I run this locally. However, when I try to run this in GCP I get this issue. Could you give me any hints to find the solution?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
