KWAS Academy
Learn. Build. Master.
QUICK START:HTMLCSSJavaScriptTypeScriptPythonSQLReactNext.jsNode.jsDSASystem DesignDevOpsCybersecurityAI / ML
Advanced 18 min readModule: Module 3: Buffers, Streams & Binary Data

Node.js Streams & Memory Optimization (pipeline)

Process massive datasets without memory overflow using Readable, Writable, and Transform streams.

What You Will Learn in This Lesson

  • Why loading a 2GB file into memory with readFile() crashes the process
  • Piping streams with stream.pipeline to prevent memory leaks
  • Transform streams for real-time gzip compression and cryptography

Introduction & Core Concept

Streams are collections of data—just like arrays or strings. The difference is that streams might not be available all at once and don't have to fit in memory.
WHY DOES THIS MATTER IN THE REAL WORLD?

Using streams allows a Node server with 512MB RAM to process 10GB video files smoothly without out-of-memory crashes.

Stream Pipeline with Gzip Compression

javascript
javascript
1
2
3
4
5
6
7
8
9
10
11
12
const fs = require("fs");
const zlib = require("zlib");
const { pipeline } = require("stream/promises");
 
async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    zlib.createGzip(),
    fs.createWriteStream(output)
  );
  console.log("File compressed successfully.");
}

Line-by-Line Technical Breakdown

1Backpressure prevents fast read streams from overwhelming slow network write streams.

Try It Yourself (Interactive Editor)

Modify the code in real-time and click Run to test live browser output and console logs.

Intelligent Code Runner & Live Sandbox[JAVASCRIPT]
JAVASCRIPT SOURCE EDITOR
Interactive Live Code

Industry Best Practices & Professional Standards

  • Always use stream.pipeline instead of .pipe() for robust error cleanup.

Lesson Summary & Core Takeaways

  • Streams enable memory-efficient processing of massive datasets.