Download all the files from Azure blob storage , zip it and upload the zip file in JAVA

Question

I want to download all the files from Azure blob storage, create a zip file out of these files and upload the zip file back to the blob storage. As the file size can be very large, I dont want to max out the memory. Also this operation needs to be very FAST.

JAVA SDK - azure-storage-blob 12.8.0

EDIT : Code written so far. Not sure how to proceed further with uploading pipedinputstream data parallely.

 String zipFileName = formFileName(exportRequest, requestId);
        final PipedOutputStream pipedOutputStream = new PipedOutputStream();
        final PipedInputStream pipedInputStream = new PipedInputStream(pipedOutputStream);

  AzureObjectStoreService objectStoreService =managedObjectStoreUtils.getObjectStoreService();

            if (filesToZip.size() > 0) {
                System.out.println("Files to zip "+ filesToZip.size());
                CompletableFuture<Boolean> zipCreationFuture = CompletableFuture.runAsync(() -> {
                    LoggerHelper.logInfo(logger, "Inside createZIP file async function");
                    ZipOutputStream zipOutputStream = new ZipOutputStream(pipedOutputStream);
                    try {
                        for (String fileName : filesToZip) {
                            try {
                                BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(fileName);
                                LoggerHelper.logInfo(logger, "Adding zipEntry for file : " + fileName);
                                final ZipEntry zipEntry = new ZipEntry(fileName);
                                zipOutputStream.putNextEntry(zipEntry);
                                byte[] buffer;
                                ByteArrayOutputStream output = new ByteArrayOutputStream();
                                buffer= output.toByteArray();
                                blobClient.getBlockBlobClient().download(output);
                                int len;
                                while ((len = buffer.length) > 0) {
                                    zipOutputStream.write(buffer, 0, len);
                                }
                                zipOutputStream.closeEntry();
                            } catch (SdkClientException e) {
                                LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
                                LoggerHelper.logError(logger, "Failed while getting s3 object");
                            }
                        }
                        zipOutputStream.finish();
                    } catch (IOException ex) {
                        LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) ex);
                        LoggerHelper.logError(logger, "Creating zip file failed");
                    } finally {
                        try {
                            zipOutputStream.close();
                            } catch (IOException e) {
                            LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) e);
                            LoggerHelper.logError(logger, "Failed to close the zip output stream");
                        }
                    }
                    LoggerHelper.logInfo(logger, "Completed createZIP file async function");
        //            return true;
                }).handle((o, exception) -> {
                    LoggerHelper.logExceptionWithMessage(logger, this.getClass().getName(), (Exception) exception);
                    LoggerHelper.logError(logger, "Creating zip file failed");
                    return null;
                });

Answer 1

Was able to do it this way. Please let me know if anyone has a better approach.

CompletableFuture.runAsync(() -> {
                  
                    BlobClient blobClient = objectStoreService.getBlobContainerClient().getBlobClient(zipFileName);
                    BlobOutputStream blobOutputStream = blobClient.getBlockBlobClient().getBlobOutputStream();

                     try {
                            int nextData= pipedInputStream.read();
                            while (nextData!=-1) {
                                blobOutputStream.write(nextData);
                                nextData = pipedInputStream.read();
                            }blobOutputStream.close();
                        } catch (IOException e) {
                            e.printStackTrace();
                        }