Calculating a matrix to invert a "flattened" 3D rendering context

Question

Context:

In my web app I create several groups of elements all positioned relative to each other in 3d space. All elements have transform-style: preserve-3d. A parent element is used to rotate, scale, and move all these elements together in response to user navigation.

Some of these elements are "portals," rectangular windows through which several other elements might be visible. It is important to note that the elements within these "portals" must exist within the same global 3d space (not just 2d) as elements outside their parent portal.

These "portals" have overflow: hidden in order to hide overflowing elements within them.

As per the w3 spec, overflow is one of several css property values that group objects together and creates a 2d “flattened” rendering context; effectively the same result as “transform-style: flat;”

Question:

I must find some way to inverse (cancel-out) the transformation matrix created by transform-style: flat; and create an identical 3D rendering context to one preserved using transform-style: preserve-3d;

All 3D css transforms are represented by a 3d matrix internally. In the case of transform-style: flat the user-agent is doing some mysterious "flattening" math on its own transformation matrix. Then this matrix gets applied to its children, creating the illusion that its children are all flattened in their parent's pane. It should be possible to bypass this effect with a little matrix math.

Unfortunately, the w3 spec makes no specification of what this "flattening" means mathematically; they are rather vague on the subject, simply calling it a "flattened representation of their (the element's) content" that "has no depth."

I can't figure out what matrix math needs to be done to reproduce the "flattening" effect. If the algorithm could be reverse engineered, this "flattening" behavior could be easily negated with an inverted matrix. A secondary element inside each portal element could be used to apply this inverted matrix as a transform, thereby negating the "flattened" transform and correctly restoring 3d position/rotation/perspective.

Shown below is a demo demonstrating that is is absolutely possible to create the impression of depth within a flattened parent. In this case, I naively apply the inverse of the top-most parent's transform. Unfortunately, inverting a "flattened" matrix is not so simple:

Demo and Links:

• JSFiddle demo unchanged for testing
• JSFiddle demo with the naive algorithm demonstration
• W3 spec
• MDN on transform-style

Notes:

• Solutions involving bypassing the problem using masks, clip-paths, other likely won't work since they are also grouping property values
• In addition to the above, clip-paths have rendering issues on chrome and firefox
• Algorithm should work for different parent positions, rotations, scales and perspective. I'm not looking for a magic transform value that will fix my example and my example only.
• I've experimented solving this issue using several independent 3d contexts layered on top of each other. Clip paths were used to cut out portals. However, this resulted in issues when portals need to be obscured by other portals or elements within other layers.
• I am open to other alternative suggestions given the context of my project.
• A similar question has been asked before, however in a much different context where the OP is looking for a css-only solution: CSS: "overflow: hidden" alternative that doesn't break 3D transforms

Edit 1: Why not use three js?

My entire application/website will be built on a model of navigation that exists in 3d space, so why not just use webgl?

1. Webgl cannot be parsed by search engines or screen readers
2. Webgl cannot be statically generated and hydrated client-side
3. My application will include interactable html. This can't be done in webgl:

Technically, there are ways to do hybrid html/webgl websites but it requires css transforms and (for my use case) dynamically calculated clipping, which does not work on chrome and firefox. The only only other way to do clipping is to use overflow: hidden, which lead me to asking this question.

4. Building this in webgl would require a ton of extra work designing and building navigation and lifecycle systems that I can otherwise leverage from existing html frameworks... I'm a one person team.
5. Finally, portals in webgl are not a simple issue to solve, and my requirement of being able to fly "through" portals makes that goal even more lofty... I need to get this working the simplest way possible if my goal is just to create a mvp and get it out to my patiently waiting testers.

As far as my testing/research goes webgl isn't a feasible alternative for what I'm building, (at least for a one person team) however, figuring out a bit of math should be doable.

Edit 2: A partial solution

Thanks to @Markus' answer, I've found that the flattening affect can be achieved by dropping all the values in the column associated with transformations in the z-axis. This can be done by multiplying it's transform by the following matrix:

Take the identity matrix, and change the 3rd item in the 3rd column to a arbitrary small number.

const flatten = identity()
flatten[10] = 1/10000 // arbitrarily small. not zero, since that will invalidate the matrix.

Here's a demo of this: https://jsfiddle.net/aywbe9p7/2/

This seems to suggest that internally, the browser is converting the 3d 4x4 matrix into a 2d 3x3 matrix by dropping the values in it's third column.

So, It would be reasonable to think that inverting this effect would be as simple as re-populating the third column:

// normalize portal transform
const normalizedPortalTransform = normalize(portalTransform)

// try to re-establish perspective in z axis
const final = identity()
final[8] = normalizedPortalTransform[8]
final[9] = normalizedPortalTransform[9]
final[10] = normalizedPortalTransform[10]
final[11] = normalizedPortalTransform[11]

It appears to kind of work, but the perspective is still off:

Here's a demo of this: https://jsfiddle.net/aywbe9p7/3/

I've tried many different combinations of re-populating the matrix, like also including the third row (indexes 2, 6, 10 and 14) and even decomposing the portalTransform's perspective components using the code below, and trying to re-incorporate these values into the final de-flattening matrix. But that doesn't work either.

// Returns the transpose of a 4x4 matrix
function transpose(matrix){
  return Array.from({ length: 16 }, (_, i) => {
    const y = i % 4
    const x = Math.floor(i / 4)
    return matrix[y * 4 + x]
  })
}

// Decompose the perspective component of a 4x4 matrix.
// https://www.w3.org/TR/css-transforms-2/#decomposing-a-3d-matrix
function decompPerspective(matrix){
  // There exists some perspective
  if(matrix[15] != 0 && (matrix[3] != 0 || matrix[7] != 0 || matrix[11] != 0)){
    // Normalize the matrix.
    const m = normalize(matrix)
    
    // Used to solve for perspective
    const perspectiveMatrix = Array.from(m)
    perspectiveMatrix[3] = 0
    perspectiveMatrix[7] = 0
    perspectiveMatrix[11] = 0
    perspectiveMatrix[15] = 1
    
    // The right hand side of the equation.
    const r0 = m[3]
    const r1 = m[7]
    const r2 = m[11]
    const r3 = m[15] // should be 1
    
    // Solve the equation by inverting perspectiveMatrix and multiplying
    // rightHandSide by the inverse.
    const f = transpose(inverse(perspectiveMatrix))
    
    return [
      f[0] * r0 + f[4] * r1 + f[8] * r2 + f[12] * r3,
      f[1] * r0 + f[5] * r1 + f[9] * r2 + f[13] * r3,
      f[2] * r0 + f[6] * r1 + f[10] * r2 + f[14] * r3,
      f[3] * r0 + f[7] * r1 + f[11] * r2 + f[15] * r3, // should be 1
    ]
  }
  // No perspective
  else{
    return [0, 0, 0, 1]
  }
}

Edit 2: Why the math needs to be perfect and not approximative

This might be a little difficult for me to explain without making an animation, but imagine you are looking down on a scene containing a portal. Inside this portal is another scene.

I would like to animate between looking at the top-level scene, and looking into the scene embedded within the portal.

In order to do this, the visual perspective is first animated to "fly" towards the portal element.

Next, when the viewport is filled completely by the portal, the scene containing the portal is deleted. It is replaced with the embedded scene, except that now the portal is gone, so the scene is not inside a portal anymore.

For this transition to work, the perspective of items in a portal must match the perspective of those same items outside a portal perfectly... otherwise this transition will not be smooth.

Let me know if that doesn't make sense. I might be able to make an illustration to demonstrate this better.

Answer 1

If you don't want to use some WebGL-lib like three.js but pure css-transform, the straight forward way to get the desired effect is to dynamically change the perspective-origin of your portal.

Please be aware that Firefox still has some issues with 3D-css. E.g. I recommend not to use any margins or paddings on your objects.

The normal flattening is done by the camera matrix that can be found in the documentation of the perspective parameter (see MDN). It is a function of the perspective value. If the user's view is not orthogonal to the projection screen (which in your case is the portal plane) you can change the perspective-origin accordingly.

Like in your examples I have used Rematrix to adjust the perspective-origin of the portal window.

<!DOCTYPE HTML>
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
<script src="https://unpkg.com/rematrix"></script>
<style>
.rect {
  width:400px;
  height:200px;
  border: 2px solid red;
}
.scene {
  perspective-origin: center;
}
.main {
  position: relative;
  transform-style: preserve-3d;
  display:inline-block;
  vertical-align:top;
}
.inner {
  position: absolute;
}
.portal{
  overflow:hidden;
}
</style>
<body>
</head>
<div id="scene" width=400px>
 <div id="scene1" class="scene rect">
  <div id="obj1" class="main rect">
    <div class="inner rect" style="transform:translateZ(-60px)"></div>
    <div class="inner rect" style="transform:translateZ(-120px)"></div>
    <div class="inner rect" style="transform:translateZ(-180px);background-color:#eee">preserve-3d</div>
  </div>
 </div>
 <div id="scene2" class="scene rect"> 
  <div id="obj2" class="main portal rect">
   <div class="inner rect" style="transform:translateZ(-60px)"></div>
   <div class="inner rect" style="transform:translateZ(-120px)"></div>
   <div class="inner rect" style="transform:translateZ(-180px);background-color:#eee">overflow hidden</div>
  </div>
 </div>
</div>
<script>
let r = Rematrix;
let obj1 = $("#obj1");
let obj2 = $("#obj2");
let scene = $("#scene");
let scene1 = $("#scene1");
let scene2 = $("#scene2");
// locate objects behind center of scene
let tx = (scene1.width() - obj1.width()) / 2;
let ty = (scene1.height() - obj1.height()) / 2;
let tz = -1000;
// set perspective
let p = 2000;
scene1.css("perspective", "" + p + "px");
scene2.css("perspective", "" + p + "px");
// initial camera position
let camPos = [0, 0, p, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
// start rendering
move(0.6, 0.6);
scene.mousemove(function(event){ move(event.clientX / scene.width(), event.clientY / scene.height()); });

function move(x, y) {
  // move objects
  let dx = 120 * (x - 0.5);
  let dy = -120 * (y - 0.5);
  let t = r.toString([r.translate3d(tx, ty, tz), r.rotateX(dy), r.rotateY(dx)].reduce(r.multiply));
  obj1.css("transform", t);
  obj2.css("transform", t);
  // adjust camera perspective of portal object
  let c = [r.rotateY(-dx), r.rotateX(-dy), r.translateZ(-tz), camPos].reduce(r.multiply);
  let px = c[0] + obj2.width() / 2;
  let py = c[1] + obj2.height() / 2;
  let pz = c[2];
  obj2.css("perspective", "" + pz + "px");
  obj2.css("perspectiveOrigin", "" + px + "px " + py + "px");
}
</script>
</body>

Here is a diagram to sketch the idea behind the calculation of the adjusted perspective-origin: