'cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})
Currently facing a problem where flask app which is running under uWSGI gives above error after some time.
Exception message which occur is:
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})
Here is the code of the app for wsgi.py
import sys
import logging
sys.stdout = sys.stderr
from cqlengine.connection import (
cluster as cql_cluster, session as cql_session)
from url.settings import CLUSTER
log = logging.getLogger('uwsgi')
try:
from uwsgidecorators import postfork
except ImportError:
# We're not in a uWSGI context, no need to hook Cassandra session
# initialization to the postfork event.
pass
else:
@postfork
def cassandra_init():
""" Initialize a new Cassandra session in the context.
Ensures that a new session is returned for every new request.
"""
if cql_cluster is not None:
cql_cluster.shutdown()
if cql_session is not None:
cql_session.shutdown()
from url.settings import connect_cassandra
connect_cassandra()
log.info('Connection with cassandra completed')
from url.app import app as application
Here is the code the connect_cassandra method.
CLUSTER = ['XX.XX.XX.XX', 'XX.XX.XX.XX']
def connect_cassandra():
# next, setup the connection to your cassandra server(s)...
# see http://datastax.github.io/python-
driver/api/cassandra/cluster.html for options
# the list of hosts will be passed to create a Cluster() instance
from cassandra.cqlengine import connection
from cassandra.cqlengine.management import sync_table
from url.models import Links, LinksAnalytics
connection.setup(CLUSTER, "contentstudio", protocol_version=3)
sync_table(Links)
sync_table(LinksAnalytics)
After restarting app works fine for 20-30 minutes and after that it stops giving this error and sometimes it works which is really frustrating.
Also, in between getting this exception message as well:
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: XX.XX.XX.XX dc1>: ConnectionException('Host has been marked down or removed',)})
UPDATE 1 with nodetool tpstats
node 1
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 1225692 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 853120 0 0
MutationStage 0 0 62573 0 0
MemtableReclaimMemory 0 0 1133 0 0
PendingRangeCalculator 0 0 2 0 0
GossipStage 0 0 4175516 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 64064 0 0
Native-Transport-Requests 0 0 12887762 0 16587
ReadRepairStage 0 0 6887 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 34 0 0
MemtablePostFlush 0 0 1268 0 0
PerDiskMemtableFlushWriter_0 0 0 1123 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 1125 0 0
InternalResponseStage 0 0 45 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
node 2
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 29325 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 407325 0 0
MutationStage 0 0 62573 0 0
MemtableReclaimMemory 0 0 1133 0 0
PendingRangeCalculator 0 0 4 0 0
GossipStage 0 0 4174442 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 0 0 0 0 0
RequestResponseStage 0 0 6845 0 0
Native-Transport-Requests 0 0 989812 0 0
ReadRepairStage 0 0 102 0 0
CounterMutationStage 0 0 0 0 0
MigrationStage 0 0 26 0 0
MemtablePostFlush 0 0 1268 0 0
PerDiskMemtableFlushWriter_0 0 0 1123 0 0
ValidationExecutor 0 0 0 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 1125 0 0
InternalResponseStage 0 0 0 0 0
ViewMutationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
CacheCleanupExecutor 0 0 0 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 0
Solution 1:[1]
I had the same problem and in my case the problem was that uwsgi doesn't enable threads by default, and python-driver has an internal thread pool.
Try enabling threads: https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#a-note-on-python-threads
Solution 2:[2]
Cassandra is a NoSQL distributed DBMS which is written in Java. Java uses a special data structure called HEAP. Java objects reside in the heap area. The heap is created when the JVM (Java Virtual Machine) starts up and may increase or decrease in size while the application runs.
Cassandra uses memory in 4 ways:
- Java heap
- Offheap memory
- OS page cache
- OS TCP/IP stack I/O cache
The error you get, is due to the insuffiecient OS system memory (OS RAM). Probably, in your case, the heap size is not in accord with the memory.
There are some guidelines and recommendations:
HEAP_SIZE is usually between 1/4 and 1/2 of system memory but not larger than 32 GB.
NEW_HEAP_SIZE determines the amount of heap memory allocated to newer objects. The database calculates the default value for this property in MB as the lesser of:
- 100 times the number of cores
- 1/4 of the
MAX_HEAP_SIZE
The cassandra-env.sh automatically configures the min and max size to the same value using the following formula:
Max(Min(1/2 RAM, 1024 Megabytes), Min(1/4 RAM, 32765 Megabytes))
To adjust the JVM heap size, uncomment and set the following parameters in the jvm-server.options file:
Minimum (-Xms)
Maximum (-Xmx)
New generation (-Xmn)
If you use docker for cassandra you can set these parameters in the environment in the cassandra service in the docker-compose.yml file.
For additional information: Cassandra AWS System Memory Guidelines and Chaning heap size parameters
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Gabriel |
| Solution 2 | Mohi |