split function does not return any observations with large dataset

Question

I have a dataframe like this:

seqnames       pos     strand    nucleotide     count
    id1         12        +          A            13
    id1         13        +          C            25
    id2         24        +          G            10
    id2         25        +          T            25
    id2         26        +          A            10
    id3         10        +          C            5

But it has more than 100,000 rows in total, seqnames has 3138 levels. I would like to split it into lists of dataframes according to seqnames, so I used split function:

data_list <- split(data,data$seqnames)

But it only returns something like this:

List of 3138
 $ id1:'data.frame':    0 obs. of  6 variables:
  ..$ seqnames  : Factor w/ 3138 levels "id1","id2",..: 
  ..$ pos       : int(0) 
  ..$ strand    : Factor w/ 3 levels "+","-","*": 
  ..$ nucleotide: Factor w/ 8 levels "A","C","G","T",..: 
  ..$ count     : int(0) 
  ..$ sample_id : chr(0) 
 $ id2:'data.frame':    0 obs. of  6 variables:
  ..$ seqnames  : Factor w/ 3138 levels "id1","id2",..: 
  ..$ pos       : int(0) 
  ..$ strand    : Factor w/ 3 levels "+","-","*": 
  ..$ nucleotide: Factor w/ 8 levels "A","C","G","T",..: 
  ..$ count     : int(0) 
  ..$ sample_id : chr(0) 

I can't figure out why it is like this because I have used it on a made up dataframe with all numbers (of course, not as many rows as this one) and it works. How can I solve this problem?

Answer 1

It is just that there are many unused levels as the column 'seqnames' is a factor. With split, there is an option to drop (drop = TRUE - by default it is FALSE) to remove those list elements. Otherwise, they will return as data.frame with 0 rows. If we want those elements to be replaced by NULL, then find those elements where the number of rows (nrow) are 0 and assign it to NULL

data_list <- split(data,data$seqnames)
> str(data_list)
List of 5
 $ id1:'data.frame':    2 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 1 1
  ..$ pos       : int [1:2] 12 13
  ..$ strand    : chr [1:2] "+" "+"
  ..$ nucleotide: chr [1:2] "A" "C"
  ..$ count     : int [1:2] 13 25
 $ id2:'data.frame':    3 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 2 2 2
  ..$ pos       : int [1:3] 24 25 26
  ..$ strand    : chr [1:3] "+" "+" "+"
  ..$ nucleotide: chr [1:3] "G" "T" "A"
  ..$ count     : int [1:3] 10 25 10
 $ id3:'data.frame':    1 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 3
  ..$ pos       : int 10
  ..$ strand    : chr "+"
  ..$ nucleotide: chr "C"
  ..$ count     : int 5
 $ id4:'data.frame':    0 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 
  ..$ pos       : int(0) 
  ..$ strand    : chr(0) 
  ..$ nucleotide: chr(0) 
  ..$ count     : int(0) 
 $ id5:'data.frame':    0 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 
  ..$ pos       : int(0) 
  ..$ strand    : chr(0) 
  ..$ nucleotide: chr(0) 
  ..$ count     : int(0) 

Doing the assignment to NULL

data_list[sapply(data_list, nrow) == 0] <- list(NULL)

-check again

> str(data_list)
List of 5
 $ id1:'data.frame':    2 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 1 1
  ..$ pos       : int [1:2] 12 13
  ..$ strand    : chr [1:2] "+" "+"
  ..$ nucleotide: chr [1:2] "A" "C"
  ..$ count     : int [1:2] 13 25
 $ id2:'data.frame':    3 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 2 2 2
  ..$ pos       : int [1:3] 24 25 26
  ..$ strand    : chr [1:3] "+" "+" "+"
  ..$ nucleotide: chr [1:3] "G" "T" "A"
  ..$ count     : int [1:3] 10 25 10
 $ id3:'data.frame':    1 obs. of  5 variables:
  ..$ seqnames  : Factor w/ 5 levels "id1","id2","id3",..: 3
  ..$ pos       : int 10
  ..$ strand    : chr "+"
  ..$ nucleotide: chr "C"
  ..$ count     : int 5
 $ id4: NULL
 $ id5: NULL

data

data <- structure(list(seqnames = structure(c(1L, 1L, 2L, 2L, 2L, 
3L), .Label = c("id1", 
"id2", "id3", "id4", "id5"), class = "factor"), pos = c(12L, 
13L, 24L, 25L, 26L, 10L), strand = c("+", "+", "+", "+", "+", 
"+"), nucleotide = c("A", "C", "G", "T", "A", "C"), count = c(13L, 
25L, 10L, 25L, 10L, 5L)), row.names = c(NA, -6L), class = "data.frame")