ML Kit Barcode Scanning doesn't detect QR codes in the photo of the monitor screen

Question

I am using com.google.mlkit:barcode-scanning:17.0.2 to detect QR codes in the pictures.

After getting URI from the gallery I create InputImage and then process this image with BarcodeScanner to find QR codes. When I select a photo of QR codes on paper code is found. But when I take a photo of the QR code on the monitor screen code is never found. What I should do to be able to detect a QR code in a photo of a monitor screen?

(When I use the same scanner with CameraX to do live QR code detection it finds code on the monitor screen)

val image = InputImage.fromFilePath(context, uri)

val scanOptions =
    BarcodeScannerOptions.Builder()
        .setBarcodeFormats(
            Barcode.FORMAT_QR_CODE,
        )
        .build()

val scanner = BarcodeScanning.getClient(scanOptions)

scanner.process(image)
    .addOnSuccessListener {
        val code = it.getOrNull(0)?.rawValue
        if (code == null) {
            // code NOT found
        } else {
            // code was found
        }
    }

Example of QR code on paper which is found

Example of QR code on the monitor screen which is NOT found

Answer 1

Chances are that you're fighting against Moiré effect. Depending on the QR detection algorithm, the high frequencies introduced by the Moiré effect can throw the detector off its track. Frustratingly, it is often the better QRcode detectors that are defeated by Moiré patterns.

A good workaround is:

• take the picture at the highest resolution you can
• perform a blurring of the picture
• increase contrast to the max, if possible
• (optionally) run a sigma thresholding, or just rewrite all pixels with a luma component below 32 to 0, all those above 224 to 255.

Another way of doing approximately the same operation is

• take the picture at the highest resolution you can
• increase contrast to the max, if possible
• downsample the picture to a resolution which is way lower

The second method gives worse results, but usually can be implemented with device primitives.

Another source of problems with monitors (not in your picture as far as I can see) is the refresh rate. Sometimes, you'll find that the QR code is actually an overexposed QRcode in the upper half of the picture and an underexposed QRcode in the bottom half of the picture. Neither are recognized. This effect is due to the monitor's refresh rate and strategy and is not easy to solve - you can try lowering the monitor's luminosity to increase exposure time, until it exceeds 1/50th or 1/25th of a second, or take the picture from farther away and use digital zooming. Modern monitors have higher refresh rates and actually refresh at more than their own dwell time, so this should not happen; with old analog monitors however it will happen every time.