Cumulative distinct count

Question

I am working on query to get cumulative distinct count of uids on daily basis.

Example : Say there are 2 uids (100,200) appeared on date 2016-11-01 and they also appeared on next day with new uid 300 (100,200,300) on 2016-11-02 At this point i want store cumulative count to be 3 not 5 as (user id 100 and 200 already appeared on past day ).

Input table:

    date            uid         
2016-11-01          100
2016-11-01          200
2016-11-01          300
2016-11-01          400         
2016-11-02          100
2016-11-02          200                 
2016-11-03          300
2016-11-03          400
2016-11-03          500
2016-11-03          600
2016-11-04          700

Expected query result:

date            daily_cumulative_count
2016-11-01              4   
2016-11-02              4
2016-11-03              6
2016-11-04              7

Till now i am able to get cumulative distinct count per day but it includes previous distinct uids from previous day as well.

SELECT 
  date, 
  SUM(count) OVER (
    ORDER BY date ASC 
    ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
  )
FROM (
  SELECT 
    date, 
    COUNT(DISTINCT uid) AS count
  FROM sample_table
  GROUP by 1
)ORDER BY date DESC;

Any kind of help would be greatly appreciated.

Answer 1

easiest way:

SELECT *, count(*) over (order by fst_date ) cum_uids
  FROM (
SELECT uid, min(date) fst_date FROM t GROUP BY uid
 ) t

or something like this

Answer 2

WITH firstseen AS (
  SELECT uid, MIN(date) date
  FROM sample_table
  GROUP BY 1
)
SELECT DISTINCT date, COUNT(uid) OVER (ORDER BY date) daily_cumulative_count 
FROM firstseen
ORDER BY 1

Using SELECT DISTINCT because (date, COUNT(uid)) will be duplicated many times.

Explanation: for each date dt, it counts uid from the earliest date up to dt, because we are specifying ORDER BY date and it defaults to BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.

Answer 3

You can use exists to check if an id was present on any of the previous dates. Then get the running sum and find the max value for each group which would get you the daily distinct cumulative count.

select dt, max(col) as daily_cumulative_count
from (select t1.*, 
      sum(case when not exists (select 1 from t where t1.dt > dt and id = t1.uid) then 1 else 0 end) over(order by dt) col
      from t t1) x 
group by dt

Answer 4

Please try the following...

SELECT date AS date
       COUNT( uid ) AS daily_cumulative_count
FROM ( SELECT leftTable.date AS date,
              rightTable.uid AS uid
       FROM sample_table AS leftTable
       JOIN sample_table AS rightTable ON leftTable.date >= rightTable.date
       GROUP BY leftTable.date,
                rightTable.uid
     ) AS allUIDSForDateFinder
GROUP BY date;

This statement starts by joining one instance of sample_table to another in such a way that each record in leftTable has associated with it a copy of each record from rightTable that has an earlier or equal date value. This effectively attaches a list to each date of all uid values that have occurred up to and including that date value.

The resulting dataset is refined to unique date and uid combinations through use of GROUP BY.

The refined dataset from the subquery allUIDSForDateFinder is then grouped by date by the main body of the query, and a COUNT() of uid values associated with each group is performed.

If you have any questions or comments, then please feel free to post a Comment accordingly.

Answer 5

A version similar to @stepan-blr but with the final result you are looking for

Version WITH:

WITH t as (
    SELECT uid
           , min(dt) fst_date 
    FROM input_table 
    GROUP BY uid
)
SELECT DISTINCT fst_date
                , count(uid) over (order by fst_date ) daily_cumulative_count
FROM t

Version SELECT FROM SELECT:

SELECT DISTINCT fst_date
                , count(uid) over (order by fst_date ) daily_cumulative_count
FROM (
    SELECT uid
           , min(dt) fst_date 
    FROM input_table 
    GROUP BY uid
     ) t