Best architecture to filter realtime for thousand of users

Question

It's a question for the future, but I'm starting to thinkg about that now.

The Overview

• User navigate in the app to see some realtime data Events -> we don't have no more than 150 realtime Events
• User want to receive a notification when one or more Event of type A have Attr1 > 1 and Attr2 < 2
• User receive background notifications when a realtime Event matches with their filter

Actual behavior

Sending a cron (every 1 minute) to process all User's filters (today almost 8.000) against the realtime data - when match, send to kafka broker. The entire process takes a mean almost 49 seconds.

The Problem

Events are realtime data, that can change in seconds, if we process the filter with a interval of 1 minute, we lost a lot of matches. So, every time that we check Filters against Event we discard more that 95% of filter, but, every filter have one or more match by day.

Questions

What we need to studying for remodeling this? The actual performance is good, we lost some record, but, it's ok. We thinking about future, when we have 2x, 3x, 4x, more users filters.

• I heard about Complex Event Processing and etc, but I'm not sure what I need to studying to solve the futher problem.

Today architecture