Quanteda dfm_weight() results in relative frequency > 1

Question

I'm using Quanteda and trying to compute the relative frequencies of specific words in a corpus which is organized by date and party. However, after converting the corpus to a dfm and using dfm_weight(dfmat, scheme = "prop") followed by textstat_frequency, I get scores of bigger than 1.

Here is my code (I also stem and clean my tokens, not here in the code):

corp <- corpus(title_df, text_field = "text", meta = list(title_df[,-4]))
toks <- tokens(corp)
dfmat <- dfm(toks, verbose=TRUE)
dfm_rel_freq <- dfm_weight(dfmat, scheme = "prop") 
rel_freq_all <- quanteda.textstats::textstat_frequency(dfm_rel_freq, groups = year)  
# arrange by max frequency:
rel_freq_all %>% arrange(frequency) %>% tail()

 
 
feature
<chr>
frequency
<dbl>
rank
<dbl>
docfreq
<dbl>
group
<chr>
81093   pension 5.802529    1   117 2004
40971   pension 6.117154    1   97  1998
148372  peopl   6.430454    1   220 2014
65747   pension 6.721089    1   138 2002
53303   pension 7.871011    1   153 2000
74391   pension 8.153381    1   156 2003
6 rows

Answer 1

This is the expected behaviour: quanteda.textstats::textstat_frequency(x, groups = year) will sum the dfm within the year groups. So your proportions from the dfm are being summed, and these can exceed 1.0.

If you wanted a different operation on the groups, for instance mean, then you should not use a groups argument, and then use some dplyr operations such as

library(dplyr)
quanteda.textstats::textstat_frequency(dfm_rel_freq) %>%
    group_by(year) %>%
    summarize(mean_rel_freq = mean(frequency))