android tensorflow static image segmentation problems with diff size images

Question

I am trying to detect person using tensorflow segmentation in android app following the demo from github. Demo contains how to detect segmentation in camera but its only in specific size! My requirement is using static image not live preview. So, I used specific image and try to generate output from it.

MainActivity.kt:

class MainActivity : AppCompatActivity() {
    private lateinit var mBinding: ActivityMainBinding
    private lateinit var mContext: Activity
    private val scope = CoroutineScope(Dispatchers.IO)
    private var bitmap: Bitmap? = null

    private val TAG = "ImageDetect>>>"

    private lateinit var imageSegmentationModel: ImageSegmentationModelExecutor

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        mBinding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(mBinding.root)
        mContext = this

        imageSegmentationModel = ImageSegmentationModelExecutor(this, true)

        Glide.with(this)
            .load(R.drawable.sample)
            .skipMemoryCache(true)
            .diskCacheStrategy(DiskCacheStrategy.NONE)
            .listener(object : RequestListener<Drawable> {
                override fun onLoadFailed(e: GlideException?, model: Any?, target: Target<Drawable>?, isFirstResource: Boolean): Boolean {
                    setLoading(false)
                    return false
                }

                override fun onResourceReady(resource: Drawable?, model: Any?, target: Target<Drawable>?, dataSource: DataSource?, isFirstResource: Boolean): Boolean {
                    bitmap = (resource as BitmapDrawable).bitmap
                    setLoading(false)
                    return false
                }
            })
            .into(mBinding.ivSample)
        mBinding.btnDetect.setOnClickListener {
            bitmap?.let {
                setLoading(true)

                try {
                    val bm = ImageUtils.scaleBitmapAndKeepRatio(it, 257, 257)
                    scope.launch {
                        val resultBitmap = imageSegmentationModel.execute(bm)

                        withContext(Dispatchers.Main) {
                            mBinding.ivSample.setImageBitmap(resultBitmap.bitmapMaskOnly)
                            setLoading(false)
                        }
                    }
                } catch (exc: Exception) {
                    setLoading(false)
                    Log.e(TAG, Log.getStackTraceString(exc))
                }
            }
        }
    }

    private fun setLoading(isShow: Boolean) {
        if (isShow) mBinding.pbLoading.visibility = View.VISIBLE
        else mBinding.pbLoading.visibility = View.GONE
    }
}

ImageSegmentationModelExecutor.kt, ImageUtils.kt and ModelExecutionResult.kt are as per github code and there is no change. After that I am executing code Its throws error and catch in ImageSegmentationModelExecutor.kt class something went wrong: y must be < bitmap.height(). So, if we look into code then ImageSegmentationModelExecutor is trying to generate scaledBitmap from original and trying to generate mask from ByteBuffer. So, error must be in these two functions only. I am also understand that Tensorflow only use specific sized input for analyzing image(Here they specified 257) So, I also already converted my original bitmap to 257 size. But still throwing error for bitmap! Is there any solution for these kind of static images with diff sizes?

Answer 1

you can try Imgproc.resize(matSegment, matSegment, new Size(257,257)); generate scale bitmap

Answer 2

I also faced errors in running https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android on my Android device without changes.

Here is my patch I used (I have cropped the image rather than scale the image as it is my belief that the aspect ratio has some information that can be valuable.)

From e7b35c0e9aaa00cd71ef8b9fb4de8ae0b52be539 Mon Sep 17 00:00:00 2001
From: "Mohammad Sheraj (Sheraj)" <[email protected]>
Date: Thu, 31 Mar 2022 12:55:58 +0530
Subject: [PATCH] SHERAJ: Changes to make the code work on device.

---
 .../image_segmentation/android/build.gradle   |  2 +-
 .../gradle/wrapper/gradle-wrapper.properties  |  5 ++-
 .../imagesegmentation/utils/ImageUtils.kt     | 38 +++++++++----------
 3 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/lite/examples/image_segmentation/android/build.gradle b/lite/examples/image_segmentation/android/build.gradle
index 07bcd856..5501018d 100644
--- a/lite/examples/image_segmentation/android/build.gradle
+++ b/lite/examples/image_segmentation/android/build.gradle
@@ -6,7 +6,7 @@ buildscript {
         mavenCentral()
     }
     dependencies {
-        classpath 'com.android.tools.build:gradle:4.0.0'
+        classpath 'com.android.tools.build:gradle:4.2.0'
         classpath 'de.undercouch:gradle-download-task:4.0.2'
         classpath 'org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.71'
         // NOTE: Do not place your application dependencies here; they belong
diff --git a/lite/examples/image_segmentation/android/gradle/wrapper/gradle-wrapper.properties b/lite/examples/image_segmentation/android/gradle/wrapper/gradle-wrapper.properties
index 41dfb879..6be0033b 100644
--- a/lite/examples/image_segmentation/android/gradle/wrapper/gradle-wrapper.properties
+++ b/lite/examples/image_segmentation/android/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,6 @@
+#Thu Mar 31 11:42:10 IST 2022
 distributionBase=GRADLE_USER_HOME
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.7.1-bin.zip
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-bin.zip
-zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
+zipStoreBase=GRADLE_USER_HOME
diff --git a/lite/examples/image_segmentation/android/lib_utils/src/main/java/org/tensorflow/lite/examples/imagesegmentation/utils/ImageUtils.kt b/lite/examples/image_segmentation/android/lib_utils/src/main/java/org/tensorflow/lite/examples/imagesegmentation/utils/ImageUtils.kt
index cc65f3df..4bac8434 100644
--- a/lite/examples/image_segmentation/android/lib_utils/src/main/java/org/tensorflow/lite/examples/imagesegmentation/utils/ImageUtils.kt
+++ b/lite/examples/image_segmentation/android/lib_utils/src/main/java/org/tensorflow/lite/examples/imagesegmentation/utils/ImageUtils.kt
@@ -129,25 +129,25 @@ abstract class ImageUtils {
       if (targetBmp.height == reqHeightInPixels && targetBmp.width == reqWidthInPixels) {
         return targetBmp
       }
-      val matrix = Matrix()
-      matrix.setRectToRect(
-        RectF(
-          0f, 0f,
-          targetBmp.width.toFloat(),
-          targetBmp.width.toFloat()
-        ),
-        RectF(
-          0f, 0f,
-          reqWidthInPixels.toFloat(),
-          reqHeightInPixels.toFloat()
-        ),
-        Matrix.ScaleToFit.FILL
-      )
-      return Bitmap.createBitmap(
-        targetBmp, 0, 0,
-        targetBmp.width,
-        targetBmp.width, matrix, true
-      )
+      val tempBitmap: Bitmap
+      if (targetBmp.width >= targetBmp.height){
+        tempBitmap = Bitmap.createBitmap(
+          targetBmp,
+          targetBmp.width /2 - targetBmp.height /2,
+          0,
+          targetBmp.height,
+          targetBmp.height
+        )
+      }else{
+        tempBitmap = Bitmap.createBitmap(
+          targetBmp,
+          0,
+          targetBmp.height /2 - targetBmp.width /2,
+          targetBmp.width,
+          targetBmp.width
+        )
+      }
+      return Bitmap.createScaledBitmap(tempBitmap, reqWidthInPixels, reqHeightInPixels, true)
     }
 
     fun bitmapToByteBuffer(
-- 
2.29.2.windows.2