Compute rolling z-score in pandas dataframe

Question

Is there a open source function to compute moving z-score like https://turi.com/products/create/docs/generated/graphlab.toolkits.anomaly_detection.moving_zscore.create.html. I have access to pandas rolling_std for computing std, but want to see if it can be extended to compute rolling z scores.

Answer 1

You should use native functions of pandas:

 # Compute rolling zscore for column ="COL" and window=window
 col_mean = df["COL"].rolling(window=window).mean()
 col_std = df["COL"].rolling(window=window).std()

 df["COL_ZSCORE"] = (df["COL"] - col_mean)/col_std

Answer 2

Let us say you have a data frame called data, which looks like this:

enter image description here

then you run the following code,

data_zscore=data.apply(lambda x: (x-x.expanding().mean())/x.expanding().std())

enter image description here Please note that the first row will always have NaN values as it doesn't have a standard deviation.

Answer 3

def zscore(arr, window):
    x = arr.rolling(window = 1).mean()
    u = arr.rolling(window = window).mean()
    o = arr.rolling(window = window).std()

    return (x-u)/o

df['zscore'] = zscore(df['value'],window)

Answer 4

This can be solved in a single line of code. Given that s is the input series and wlen is the window length:

zscore = s.sub(s.rolling(wlen).mean()).div(s.rolling(wlen).std())

If you need to shift the mean and std it can still be done:

zscore = s.sub(s.rolling(wlen).mean().shift()).div(s.rolling(wlen).std().shift())