How can one extract every payload from warc.wet.gz?

Question

I have been trying to extract the text data from Common Crawl's wet files. I am currently using warc parser by Internet Archieve https://github.com/internetarchive/warc

import warc
w = warc.open(fileName)
for record in w:
  text = record.payload.read()

But this method gives less than half data that is there in payload. Is there any other better method which can give all the data that is there in each of the payload in a file.

Answer 1

The warc library has a bug with its gzip handling which causes warc fail to read the entire WET file. To overcome the bug, you should use Python's gzip library to decompress the file stream on the fly as below:

import gzip
import warc
gzip_fobj = gzip.open(wet_file, "r")
warc_fobj = warc.WARCFile(fileobj=gzip_fobj, compress=False)