'How to monitor Python scripts in real time with alerts
I have quite a few python scripts running on different machines (both PCs and servers). These Python scripts are doing all kind of things (scraping, processing excel files, calling APIs to store data from etc). I have logging implemented on all of these scripts which works nice, but due to the number of these scripts increasing I need a way to monitor their activity and be able to get alerts in case they did not run. All of these scripts are launched automatically either as cronjob on Linux machines or with Windows Task Scheduler. All said scripts are not supposed to run 24/7 but they are only launched when needed and then they exit when they are done or fail.
I came up with an idea of using a regular DB (e.g. MariaDB) to store all runtimes as timestamps from said scripts. Then I would have a second 'steering' table in the same DB and in this table I would set which script is supposed to be launched.
Then I would use a MariaDB view in Grafana to see if scripts ran on time and finished successfully and I would also set up alerts in case they some not launch / finish successfully.
PROS: easy to implement
CONS: MariaDB is not a real time DB, feels kind of heavy for this kind of task. I would need to add a mariaDB connector and define SQL inserts to all my Python scripts.
I feel like there could already exist some kind of solution for this, maybe even for free?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
