Real-Time ZIP: How Servers Instantly Zip and Stream Folders

Have you ever wondered when downloading a GitHub repo or a Google Drive folder: was that .zip file already waiting on the server? The answer is usually no. These files are created in real-time (on-the-fly) the moment you click the download button and are streamed directly to you. This way, no disk space is wasted on the server for millions of possible file combinations, and the user always gets the most up-to-date data.

In this article, we will explore the logic behind this powerful technique and build a simple application with Python (Flask).

🧐 Why “On-the-Fly” Zip?

Compressing and sending files on request has many advantages:

• 💾 Save Disk Space: Creating and storing a pre-made .zip archive for every possible folder or file selection on the server leads to a huge waste of disk space, especially with constantly changing data. With real-time compression, no archive file is ever saved to disk.
• 🔄 Always Up-to-Date Data: When a user downloads an archive, they are sure to get the latest version of the files at that moment. Pre-built archives can quickly become outdated.
• 🔧 Flexibility and Customization: You can offer custom archives by instantly compressing specific files selected by the user or reports generated from a database query.
• ⚡️ Low Memory Usage and Speed: Thanks to the streaming logic, the entire zip file to be created is not loaded into the server’s memory (RAM). Files are read piece by piece, compressed, and sent to the user instantly. This prevents the server from being overloaded even when creating very large archives and allows the download to start faster.

🌊 Flow Diagram

The basic logic of the process is as follows. A request from the client (browser) triggers a streaming process on the server.

flowchart TD
    A["User clicks 'Download' button"] --> B{"Request to /download-zip is sent to the server"}
    B --> C["Server sets HTTP headers<br />Content-Type: application/zip<br />Content-Disposition: attachment"]
    C --> D["Zip stream is initiated"]
    D --> E{"Files are read sequentially"}
    E --> F["File content is added to the zip stream"]
    F --> G["Compressed data chunk is sent to the user"]
    E -- "When all files are done" --> H["Zip stream is finalized"]
    H --> I["HTTP connection is closed"]
    G -- "If there are more files" --> E

🐍 Simple Zip Example with Python (Flask)

In this first example, we will see the most basic way to compress and send files in the server’s memory (RAM). This method is great for small to medium-sized files and is very easy to understand.

Setup and Code

First, let’s install the necessary Flask library for our project:

pip install Flask

Now, let’s create our app.py file:

# app.py
from flask import Flask, Response
import zipfile
import io

app = Flask(__name__)

@app.route('/download-zip')
def download_zip():
    # 1. Create a buffer that acts like a temporary file in memory.
    memory_file = io.BytesIO()

    # 2. Create a zip file using this buffer.
    with zipfile.ZipFile(memory_file, 'w', zipfile.ZIP_DEFLATED) as zf:
        # Define the data to be added to the archive.
        file1_content = b"fr0st"
        file2_content = b"b1rd"

        # Add the data to the archive as files.
        zf.writestr('fr0st.txt', file1_content)
        zf.writestr('b1rd.txt', file2_content)
    
    # 3. Move the buffer's cursor to the beginning to read its content from the start.
    memory_file.seek(0)

    # 4. Set the data and HTTP headers with a Flask Response object.
    return Response(
        memory_file,
        mimetype='application/zip',
        headers={'Content-Disposition': 'attachment; filename="example_archive.zip"'}
    )

if __name__ == '__main__':
    app.run(debug=True)

How to Run

1. Create a file named app.py and paste the code above into it.
2. Run the application: python app.py
3. Test it: Go to http://127.0.0.1:5000/download-zip in your browser. The example_archive.zip file will start downloading.

This method is very simple, but it has a drawback: the created zip file is held entirely in the server’s memory. If you create a 1 GB archive, it will take up 1 GB of your server’s RAM. This is not efficient for very large files. This is where “streaming” comes in.

⚙️ The Engineering Behind It: How Does Streaming Work?

Streaming is a powerful technique that involves processing and transmitting large data in small pieces (chunks) instead of all at once. In real-time zip creation, this is crucial for memory efficiency.

Key Concepts

1. HTTP Chunked Transfer Encoding: Normally, when a server sends a file, it first tells the browser the total size of the file (with the Content-Length header). However, when streaming, the total size is not known at the beginning. This is where “Chunked Transfer Encoding” comes in. The server tells the browser, “I will send you the data in chunks, I don’t know the total size, but I will tell you the size of each chunk.” When the data stream is finished, it sends a final chunk of size “0” to complete the process. The browser combines these chunks to create the original file.

2. Generators in Python (yield): When you use yield instead of return in a function, that function becomes a “generator.” Generators produce values one by one and pause at each yield statement. When the next value is requested, it continues from where it left off. This allows producing data on demand instead of holding all the data in memory as a list. In zip streaming, each compressed data chunk is produced with yield and sent to the client instantly.

3. Efficient Libraries (zipstream-ng): Libraries like zipstream-ng automate this generator logic for us. They take the files, read them in small parts, compress them, and create a data stream by yielding these compressed parts. Flask’s stream_with_context function then takes this stream and turns it into an HTTP response.

🚀 Creating a Streaming Zip with zipstream-ng

zipstream-ng provides a modern, efficient, and flexible API for streaming large files and folders. It is the best choice for web servers and big data processing.

# pip install zipstream-ng
from zipstream import ZipStream, ZIP_DEFLATED

# 1. Instantly zip an entire folder and write it to a file:
zs = ZipStream.from_path("/path/to/files/")
with open("files.zip", "wb") as f:
    f.writelines(zs)

# 2. Advanced API: Comment, compression level, different sources
zs = ZipStream(compress_type=ZIP_DEFLATED, compress_level=9)
zs.comment = b"Contains compressed important files"
zs.add_path("/path/to/files/") # Add a folder
zs.add_path("/path/to/file.txt", "data.txt") # Add a file with a different name

# Add data from a generator
def random_data():
    import random
    for _ in range(10):
        yield random.randbytes(1024)
zs.add(random_data(), "random.bin")

# Add text data
zs.add(b"This is some text", "README.txt")

with open("files.zip", "wb") as f:
    f.writelines(zs)

# 3. Flask streaming integration:
import os.path
from flask import Flask, Response
from zipstream import ZipStream

app = Flask(__name__)

@app.route("/", defaults={"path": "."})
@app.route("/<path:path>")
def stream_zip(path):
    name = os.path.basename(os.path.abspath(path))
    zs = ZipStream.from_path(path)
    return Response(
        zs,
        mimetype="application/zip",
        headers={
            "Content-Disposition": f"attachment; filename={name}.zip",
            "Content-Length": str(len(zs)),
            "Last-Modified": zs.last_modified,
        }
    )

# 4. Adding a manifest (list of archive contents):
import json

def gen_zipfile_with_manifest():
    zs = ZipStream.from_path("/path/to/files")
    yield from zs.all_files() # First, yield the files
    
    # Then, create and add the manifest
    manifest = json.dumps(zs.info_list(), indent=2).encode()
    zs.add(manifest, "manifest.json")
    
    yield from zs.finalize() # Finalize the archive

Advantages:

• 🚀 High Performance: Memory usage is very low, and it’s fast with large files.
• 🧩 Flexible API: Supports adding data from different sources like files, folders, generators, or text.
• 🌐 Web-Friendly: Automatically calculates HTTP headers like Content-Length and Last-Modified, making integration with frameworks like Flask/Django easy.
• 🛠️ Rich Features: Offers many features like ZIP64 support (for 4GB+ archives), adding comments, and setting the compression level.

---

🧰 Streaming Zip with the Standard Library (stdlib)

It is also possible to stream with Python’s built-in zipfile module, but this requires much more manual coding and is more complex compared to zipstream-ng.

# Only the standard library is used
import os
import io
from zipfile import ZipFile, ZipInfo

# A custom stream class that will act as a temporary buffer
class Stream(io.RawIOBase):
    def __init__(self):
        self._buffer = bytearray()
        self._closed = False
    def close(self):
        self._closed = True
    def write(self, b):
        if self._closed:
            raise ValueError("Cannot write to a closed stream")
        self._buffer += b
        return len(b)
    def readall(self):
        chunk = bytes(self._buffer)
        self._buffer.clear()
        return chunk

# A generator that recursively traverses files and folders
def iter_files(path):
    for dirpath, _, files in os.walk(path, followlinks=True):
        if not files:
            yield dirpath
        for f in files:
            yield os.path.join(dirpath, f)

# A generator that reads a file in chunks
def read_file_chunks(path):
    with open(path, "rb") as fp:
        while True:
            buf = fp.read(1024 * 64)
            if not buf:
                break
            yield buf

# The main zip streaming generator that combines all the parts
def generate_zipstream(path):
    stream = Stream()
    with ZipFile(stream, mode="w") as zf:
        toplevel = os.path.basename(os.path.normpath(path))
        for f in iter_files(path):
            arcname = os.path.join(toplevel, os.path.relpath(f, path))
            zinfo = ZipInfo.from_file(f, arcname)
            with zf.open(zinfo, mode="w") as fp:
                if zinfo.is_dir():
                    continue
                for chunk in read_file_chunks(f):
                    fp.write(chunk)
                    yield stream.readall() # Empty the buffer and send after each write
    yield stream.readall() # Send the last remaining data

# Example usage (assuming a send_stream function):
# send_stream(generate_zipstream("/path/to/files/"))

Disadvantages:

• 🤯 Complex Code: As you can see, you need to write multiple helper functions and classes to manage the process.
• 🔧 Low-Level Management: You need to manually manage data chunks, buffers, and generators, which increases the chance of making mistakes.
• ❌ Missing Features: It is almost impossible to pre-calculate important HTTP headers like Content-Length. This prevents download times from being displayed correctly in the browser.

This comparison clearly shows why zipstream-ng is a much superior solution for modern and scalable applications compared to the standard library.

Conclusion

Real-time compression and streaming is an indispensable technique for modern web applications. While in-memory compression may be sufficient for simple scenarios, using the streaming logic and libraries like zipstream-ng is the right approach for scalable and efficient applications.

See you in another post, take care.