Python's requests_html render() function does not allow to persist CSRF token - cannot login to website

Question

I need to use .render() to get the entire DOM content off of a website managed by a dynamic front-end framework (JS or Ajax e.g.). That website stores a CSRF token in a hidden input field as usual. When I specify use r.html.render(send_cookies_session=True) or r.html.render(), the dynamic part of the DOM is rendered and I capture the CSRF token, but after I logged in and try to crawl a route behind the login page, the CSRF token changes and I am redirected to the login page.

Then when I use r.html.render(reload=False), the dynamic part of the DOM including the hidden div with the CSRF token is not rendered. I can obviously not login as I cannot capture the CSRF token.

I have tried a lot of options for render(), but nothing seems to work.

   def login(self):
        payload = json.loads(self.options.login_params)

        with HTMLSession() as session:
            r = session.get(self.login_url)

            # need to turn off reload or else a new CSRF token is captured when the page is reloaded
            # render the html to capture the CSRF token
            r.html.render(send_cookies_session=True)
            # r.html.render(reload=False)

            html = BeautifulSoup(
                r.html.html, "html.parser")

            # save CSRF token to payload dict
            payload = self.add_csrf_to_payload(html, payload)

            # login with CSRF token from payload
            p = session.post(self.login_url, data=payload)
            html = BeautifulSoup(r.html.html, "html.parser")

            # download HTML from target URL using CSRF => here it breaks down as the CSRF changes due to the render() function (unless set to reload=False)
            r = session.get(self.target_url)
            html = BeautifulSoup(r.html.html, "html.parser")
            payload = self.add_csrf_to_payload(html, payload)

            self.session = session
            self.payload = payload

            return html

Answer 1

To use Basis Universal compression in a glTF file, don't use the .basis format or BasisTextureLoader – the standardized way to do that is with a .ktx2 file. There's an artist guide for updating a glTF asset to use .ktx2 textures available, which includes some details on mipmaps. You'll almost always need mipmaps, unless you know enough about mipmap filtering to be sure you can disable it without causing rendering issues (this is rare).

three.js provides an example of how to load these files, which would look something like this:

const ktx2Loader = new KTX2Loader()
  .setTranscoderPath( 'path/to/basis/transcoder/' )
  .detectSupport( renderer );

const loader = new GLTFLoader();
loader.setKTX2Loader( ktx2Loader );
loader.load( 'path/to/model.glb', function ( gltf ) {

 ...

}, undefined, console.error );