'Python Time Series, remove sudden drops in the data (artefacts) during cleaning
I have time series data related to when a device is attached to a human (think ECG).
The quality of the signal is quite bad at points, what has likely happened is that the device has fallen off the human and hence the signal drops to zero, see below:
Using a simple interpolate, we can remove all the drops to zero as shown below:
However, the problem i'm having is that there remains these dips in the data (circled in red in the two images). These are probably points where the device has a 'half-connection' to the human and hence picks up half a trace. I'm trying to distinguish if these points are artefacts, or actual readings.
I'd like to remove them if the span of the drop is under a certain time frame, i.e. if the drops last less than 1 second and returns to its previous level, then its an artefacts and lets remove it. If its over 1 second, then its a real reading and leave it in there.
I've tried several things to no avail such as:
Pandas pct_change - This sort of works but also picks up random points of no change.
I've tried removing signals below a certain SD from the rolling mean, but i'm struggling to then interpolate the signal from the point it drops rather than from the point it crosses the -SD line.
Would appreciate any help / ideas thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|


