'How to store large files to web local storage?

I am developing ionic 2 app that downloads files from online servers, I want to store those files to the local storage (or caches I don't know).

I know that I can store data to local storage using:

localStorage.setItem('key','value');

But if it is file and specially a large one, how to store it in local storage?

Note: I am using TypeScript.

javascripthtmltypescriptcachinglocal-storage


Solution 1:[1]

For storing relatively large data, I'd recommend using IndexedDB.

Advantages over local storage:

  • Larger size limits
  • Supports other data types than string (native typed arrays!)
  • Asynchronous

Solution 2:[2]

For your consideration in 2022: Origin Private File System

As of early-2022 IndexedDB remains the best overall choice for persisting binary data (e.g. images and videos, user files from <input type="file">, JS File and Blob objects, etc) due to its support for both asynchronous IO and ability to save raw binary data without needing ugly hacks like Base64-encoding for data: URIs.

However, IndexedDB has probably one of the least well-designed API surfaces in HTML5 ("you wire-up event-handlers after you trigger the event? crazy!), and the browser-vendors agree: so Apple's Safari and Google's Chromium both now support a simpler data persistence API named Origin Private File System (OPFS) which was recently introduced in 2021.

  • OPFS is a subset of the larger WHATWG File System API and is also related to the extant WHATWG Storage API (which defines localStorage, indexedDB, etc).

  • Despite OPFS's name containing the term "File System", OPFS does not involve any access to the user's actual local computer file-system: think of it more as an isolated virtual or "pretend" file-system that's scoped to a single website (specifically, scoped to a single origin), so different websites/origins cannot access other websites' OPFS storage.

  • OPFS lacks an initial fixed-size storage quota, whereas localStorage and sessionStorage both have an initial limit of 5MB.

    • However this is not an invitation to store gigabytes of junk on users' computers in OPFS: browsers impose their own soft limits, and if you hit the limit your write operations to OPFS will throw QuotaExceededError which you should be prepared to handle gracefully.

  • Because OPFS doesn't involve the user's actual local filesystem, and because the space is segregated by origin (URI-scheme + hostname + port), it means that OPFS doesn't present a risk of users' privacy or computer security, so browsers won't interrupt your script to show your users a potentially scary-looking security-prompt that the user has to click-through to explicitly grant or deny your website or page's ability to use the OPFS API.

    • Though browsers could still add such a prompt in future if they choose, and of course, browsers can always just disable OPFS and reject all attempts to use it, just as with indexedDB and localStorage today, or even pretend to persist data between page-loads or sessions (e.g. as in Chrome's Incognito mode).

Using OPFS:

Apple have detailed usage examples in the WebKit blog.

As a brief introduction: you use OPFS via navigator.storage and awaiting getDirectory() to get a FileSystemHandle object representing the root directory of your website (origin)'s storage space, and then await methods on the root to create and manipulate files and subdirectories.

For example, supposing you have a <canvas> element with some oh-so fantastic original art your user painted by hand and wants to save locally as a PNG blob in the browser's OPFS storage space, then you'd do something like this:

async function doOpfsDemo() {

    // Open the "root" of the website's (origin's) private filesystem:
    let storageRoot = null;
    try {
        storageRoot = await navigator.storage.getDirectory();
    }
    catch( err ) {
        console.error( err );
        alert( "Couldn't open OPFS. See browser console.\n\n" + err );
        return;
    }

    // Get the <canvas> element from the page DOM:
    const canvasElem = document.getElementById( 'myCanvas' );

    // Save the image:
    await saveCanvasToPngInOriginPrivateFileSystem( storageRoot, canvasElem );

    // (Re-)load the image:
    await loadPngFromOriginPrivateFileSystemIntoCanvas( storageRoot, canvasElem );
}

async function saveCanvasToPngInOriginPrivateFileSystem( storageRoot, canvasElem ) {

    // Save the <canvas>'s image to a PNG file to an in-memory Blob object: (see https://stackoverflow.com/a/57942679/159145 ):
    const imagePngBlob = await new Promise(resolve => canvasElem.toBlob( resolve, 'image/png' ) );

    // Create an empty (zero-byte) file in a new subdirectory: "art/mywaifu.png":
    const newSubDir = await storageRoot.getDirectoryHandle( "art", { "create" : true });
    const newFile   = await newSubDir.getFileHandle( "mywaifu.png", { "create" : true });

    // Open the `mywaifu.png` file as a writable stream ( FileSystemWritableFileStream ):
    const wtr = await newFile.createWritable();
    try {
        // Then write the Blob object directly:
        await wtr.write( imagePngBlob );
    }
    finally {
        // And safely close the file stream writer:
        await wtr.close();
    }
}

async function loadPngFromOriginPrivateFileSystemIntoCanvas( storageRoot, canvasElem ) {
    
    const artSubDir = await storageRoot.getDirectoryHandle( "art" );
    const savedFile = await artSubDir.getFileHandle( "mywaifu.png" ); // Surprisingly there isn't a "fileExists()" function: instead you need to iterate over all files, which is odd... https://wicg.github.io/file-system-access/
    
    // Get the `savedFile` as a DOM `File` object (as opposed to a `FileSystemFileHandle` object):
    const pngFile = await savedFile.getFile();
    
    // Load it into an ImageBitmap object which can be painted directly onto the <canvas>. You don't need to use URL.createObjectURL and <img/> anymore. See https://developer.mozilla.org/en-US/docs/Web/API/createImageBitmap
    // But you *do* still need to `.close()` the ImageBitmap after it's painted otherwise you'll leak memory. Use a try/finally block for that.
    try {
        const loadedBitmap = await createImageBitmap( pngFile ); // `createImageBitmap()` is a global free-function, like `parseInt()`. Which is unusual as most modern JS APIs are designed to not pollute the global scope.
        try {
            const ctx = canvasElem.getContext('2d');
            ctx.clearRect( /*x:*/ 0, /*y:*/ 0, ctx.canvas.width, ctx.canvas.height ); // Clear the canvas before drawing the loaded image.
            ctx.drawImage( loadedBitmap, /*x:*/ 0, /*y:*/ 0 ); // https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage
        }
        finally {
            loadedBitmap.close(); // https://developer.mozilla.org/en-US/docs/Web/API/ImageBitmap/close
        }
    }
    catch( err ) {
        console.error( err );
        alert( "Couldn't load previously saved image into <canvas>. See browser console.\n\n" + err );
        return;
    }
}

  • Remember that OPFS filesystem spaces are not representative of their underlying filesystem, if any, on the user's computer: an extant FileSystemFileHandle in an OPFS space will not necessarily exist in the user's real file-system.

    • In fact, it's more likely they'll be physically persisted the same way as IndexedDB blobs are: as a range of bytes nested inside some other data-store file, e.g. Chromium-based browsers use their own database-engine named LevelDB while Safari seems to use a journaled Sqlite DB for their IndexedDB backing-store. You can see them yourself: On Windows, Chrome/Chromium/Edge stores LevelDB database files for IndexedDB at C:\Users\{you}\AppData\Local\Google\Chrome\User Data\Default\IndexedDB.

  • OPFS is also very simple: while your computer's NTFS, ZFS, or APFS file-system will have features like permissions/ACLs, ownership, low-level (i.e. block-level) access, snapshotting and history, and more, OPFS lacks all of those: OPFS doesn't even have a file-locking mechanism, which is usually considered an essential primitive FS operation.

    • Obviously this keeps OPFS's API surface simpler: making it easier to implement and easier to use than otherwise, not to mention shrinking the potential attack surface, but it does mean that OPFS really does come-across more as a kind-of "local AWS S3" or Azure Blob Storage, which ultimately is still good enough for 95% of applications out there... just don't expect to be able to run a high-performance concurrent write-ahead-logged database server in JavaScript on OPFS.

Comparison of browser-based JavaScript File System APIs:

But don't confuse OPFS with the other filesystem and filesystem-esque APIs:

Solution 3:[3]

Browsers have limitation on how much data you can store in local storage. It varies from browser to browser.

If your file size is under the limit, you could copy the file content as a string and save it as a key value pair of "filename":"file content"

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2
Solution 3 Community

Related Questions