Websocket Threading

Question

Below is the code to receive live ticks using WebSocket. Each time tick is received callback function on_ticks() is called and it will print ticks.

Can I spawn a single thread in on_ticks() function and call store_ticks() function to store the ticks in the database? if yes can someone please help and show how can it be done? Or is there any other way to call store_ticks() function and store the ticks each time ticks is received?

from kiteconnect import KiteTicker

kws = KiteTicker("your_api_key", "your_access_token")

def on_ticks(ws, ticks):
    print(ticks)

def on_connect(ws, response):
    # Callback on successful connect.
    # Subscribe to a list of instrument_tokens 
    ws.subscribe([738561, 5633])

def store_ticks():
    # Store ticks here

def on_close(ws, code, reason):
    # On connection close stop the main loop
    # Reconnection will not happen after executing `ws.stop()`
    ws.stop()

# Assign the callbacks.
kws.on_ticks = on_ticks
kws.on_connect = on_connect
kws.on_close = on_close


kws.connect()

Answer 1

If the reason you want to spawn a new thread is to avoid delays, I'd say don't be bothered.

I have been using mysql-client (MySQLDB connector) with a MariaDB server, subscribed to 100+ instruments in Full mode, for the past 2 months and there have been no delays in writing the ticks to the DB.

Also, we do not know when and how many ticks we'd receive once we start the ticker.This makes it hard to time/count and close the thread and DB connection. Could end up exhausting the connection limit and the thread # really fast. (DB connection pooling is an overkill here)

The reason I use MySQLDB connector and not pymysql - I've seen an approx 20% increase in write times while using pymsql. This wouldn't be obvious in live ticks . I had cloned a medium sized DB (1 Mill+ rows) , dumped it to a Dataframe in python and then wrote it row by row to another DB and bench marked the result for 10 iterations.

The reason I use MariaDB - all the features of MySQL enterprise edition, without the Oracle fuss.

Just make sure that you set a decent amount of Memory for the DB server you use. This creates a breathing space for the DB's buffer just in case. Avoiding a remote server and sticking on to a local sever also helps to great extent. If you want to back up the data from local to a cloud backup, you can setup a daily job to dump in local, export to cloud and load to DB there

If you are looking for a walkthrough, this page has an example already, along with a code walk through video.

Edit: I just made my code public here

Answer 2

You could modify your store_ticks() function to

def store_ticks(ticks):
    # code to store tick into database

and then modify your on_ticks function to:

def on_ticks(ws, ticks):
    print(ticks)
    store_ticks(ticks)

What goes inside store_ticks(ticks) is dependent on what database you want to use and what info exactly you wish to store in there.

EDIT:
To spawn a new thread for store_ticks(), use the _thread module:

import _thread

def store_ticks(ticks):
    # code to store tick into database

def on_ticks(ticks):
    print(ticks)
    try:
        _thread.start_new_thread(store_ticks, (ticks,))
    except:
        # unable to start the thread, probably want some logging here