how to prevent fonts from getting skewed after reloading canvas in fabricjs

Question

We're making a website that has an image editor using fabric js and we have many font options for the users to pick from in the editor. the problem with the fonts only appears when the user refreshes the page or we have to reconstruct the canvas for another reason. We load fonts through FontFaceObserver by using the following code to ensure fonts are loaded beforehand applying them to text on canvas.

import FontFaceObserver from "fontfaceobserver"

export default function loadFontThenDo(
  name: string,
  style: string,
  weight: string,
  callback?: Function,
  failCallback?: Function
) {
  let theFont = new FontFaceObserver(name, {
    weight: weight,
    style: style,
  })
  theFont
    .load()
    .then(function () {
      if (callback) callback()
    })
    .catch(function (e) {
      if (failCallback) failCallback()
    })
}

some images to show the problem: the fonts before refresh on canvas, the fonts after refresh on canvas as you can see after refresh almost all texts get skewed. also as far as i understand the ones that don't get skewed are the basic fonts like arial.

a text is hovered before refresh, a text is hovered after refresh this is the basic problem. although the position on the textbox is correct and it has almost exactly the same values (not 100% same because we use some relative values which might make a difference of very small decimals. I'm sure it doesn't cause the problem though, because I have tested hard coding the values to perfectly match after refresh and the problem was still there)

text content selected before refresh, text content selected after refresh this is also another weird thing that I think causes the above problem of text getting skewed towards the right. because the context box (the blue area) gets smaller the text is naturally pushed towards the right.

code used to initialize texts after refresh

if (isText(obj)) {
          applyDefaultCosmeticAttributes(obj, productConfig)
          let text = obj as unknown as FabricTextbox
          const setText = () => {
            text.set({
              styles: null,
              charSpacing: text.charSpacing,
              editable: true,
              lockRotation: false,
              hasControls: true,
              hasRotatingPoint: true,
              splitByGrapheme: true,
              perPixelTargetFind: false,
              maxWidth: drawingAreaBackgroundData.width * 0.99,
              fontFamily: text.fontFamily,
              fontStyle: text.fontStyle,
              fontWeight: text.fontWeight,
              width: text.width_rel * drawingAreaBackgroundData.width,
              text: text.text,
              fontSize:
                text.font_size_rel * 0.001 * drawingAreaBackgroundData.height,
            })
          }
          loadFontThenDo(
            text.fontFamily!,
            text.fontStyle,
            text.fontWeight,
            setText
          )
          obj.set({
            left:
              obj.x_rel * drawingAreaBackgroundData.width +
              drawingAreaBackgroundData.left,
            top:
              obj.y_rel * drawingAreaBackgroundData.height +
              drawingAreaBackgroundData.top,
          })
        }