'hand-pose model with TensorFlow.js returning `NaN` for scores

Hi there potential helpers, I've been following this guide https://blog.tensorflow.org/2021/11/3D-handpose.html

I seem to have everything setup vaguely correctly

But detector.estimateHands(video) is returning an object with NaN as the score and every coordinate:

[{
  handedness: "Right",
  score: NaN,
  keypoints: [
    {x: NaN, y: NaN, name: 'wrist'},
    {x: NaN, y: NaN, name: 'thumb_cmc'},
    etc...
  ],
  keypoints3d: [
    {x: NaN, y: NaN, z: NaN, name: 'wrist'},
    etc...
  ]
}]

The thing that makes me think it's almost working is that it only returns that object if I actually have my hand in front of the webcam. And if I hold two hands up it returns two of these objects. And then if I hide my hands it returns an empty array.

So it is recognising hands, it's just not returning any usable scores.

Here is the essence of the code:

const model = handPose.SupportedModels.MediaPipeHands;
const config = {
  modelType:"full", 
  runtime: "tfjs"
};
const detector = await handPose.createDetector(model, config);

and then in an requestAnimationFrame loop:

const video = shadowRoot.querySelector('video');
const hands = await detector.estimateHands(video);
console.log(hands);
console.log(hands[0]?.score);

I've got all the mentioned npm modules installed

@tensorflow/tfjs
@tensorflow-models/hand-pose-detection 
@tensorflow/tfjs-core 
@tensorflow/tfjs-converter
@tensorflow/tfjs-backend-webgl

And I've tried playing with config that can be passed into createDetector and estimateHands but with the typescript definitions there's not much choice.

Anyone seen this before and have any idea how to fix it? Thank you 🙇



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source