magmalake zstd

Tins · zstd

API reference

The zstd package, from zstd 0.1.5, as published on mojoshelf (commit 76190bb31012), by mojo doc from Mojo 1.2.0. Source.

zstd — Mojo bindings to libzstd (one-shot and streaming) via a thin C shim (libzstdmojo.so).

Mirrors zlib.mojo's FFI pattern: a single-call C wrapper (shim/zstd_wrapper.c, built to $CONDA_PREFIX/lib/libzstdmojo.so by the zstd-shim pixi package) is found by _dlopen.open_library (with or without CONDA_PREFIX) and loaded through an OwnedDLHandle. The handle is opened once per process and never closed (_LIB below) — a dlopen/dlclose cycle costs hundreds of microseconds, so opening one per call made the binding's fixed cost dwarf the compression it wrapped. Every FFI-calling worker takes the handle as a BORROWED imm parameter — Mojo destroys a value at its last syntactic use, and if a function both owned the handle and called get_function on it, that call would be the last use, dlclose-ing the library before the obtained function pointer is actually invoked a few lines later. Splitting "hold the handle" (_lib()) from "use the handle" (a _do_* worker that takes it by imm borrow) keeps it alive for the whole worker call, function-pointer invocation included.

zstd is Apache Iceberg's default Parquet codec, a Puffin blob codec, and an Avro codec — this binding targets that headline use as a magmalake leaf tin.

Structs

Compressor

struct Compressor

A streaming zstd compressor over ZSTD_CCtx (the modern API's ZSTD_CStream). Feed chunks to update(); call finish() once at the end to flush the trailing block and frame epilogue.

Implements Movable

__init__

def __init__(out self, level: Int = Int(3))

__deinit__

def __deinit__(deinit self)

update

def update(mut self, chunk: Span[UInt8]) -> List[UInt8]

Compresses chunk, returning whatever compressed bytes libzstd was ready to emit (may be less than a full block; may be empty).

finish

def finish(mut self) -> List[UInt8]

Flushes the trailing block and frame epilogue (checksum, content size if pledged). Call exactly once, after the last update().

Decompressor

struct Decompressor

A streaming zstd decompressor over ZSTD_DCtx (the modern API's ZSTD_DStream). Feed chunks to update() as they arrive; there's no separate flush step, so finish() is a no-op kept for API symmetry with Compressor.

Implements Movable

__init__

def __init__(out self)

__deinit__

def __deinit__(deinit self)

update

def update(mut self, chunk: Span[UInt8]) -> List[UInt8]

Decompresses as much of chunk as libzstd will take in one pass, growing the output buffer until all of chunk is consumed.

finish

def finish(mut self) -> List[UInt8]

No-op — decompression has no trailing flush step. Kept so Compressor/Decompressor share the same update/finish shape.

Functions

compress

def compress(data: Span[UInt8], level: Int = Int(3)) -> List[UInt8]

Compresses data into a single zstd frame at level (1..22).

Zero-length data is a real (if degenerate) case for zstd — it still produces a valid, tiny frame that decompresses back to zero bytes.

decompress

def decompress(data: Span[UInt8]) -> List[UInt8]

Decompresses a single zstd frame.

Uses ZSTD_getFrameContentSize to size the output in one shot when the frame carries it (true for anything compress() produced). Falls back to the streaming API — growing the output buffer as it goes — when the size is unknown (e.g. a frame streamed with an unset pledged size). A genuinely corrupt frame raises either way, with libzstd's error name.

decompress_into

def decompress_into(data: Span[UInt8], dst: Span[UInt8]) -> Int

Decompresses data straight into caller-owned dst — no growth, no extra allocation. For callers that already know the decompressed size (e.g. a Parquet page's declared uncompressed length). Returns the number of bytes written; raises (zstd.decompress_into: Dst_Too_Small, ...) if dst is too small or data is not a valid frame.

is_zstd_frame

def is_zstd_frame(data: Span[UInt8]) -> Bool

Whether data starts with the zstd frame magic number (0xFD2FB528, little-endian) — a quick, allocation-free format sniff.

frame_content_size

def frame_content_size(data: Span[UInt8]) -> Optional[Int]

The decompressed size recorded in a zstd frame header, or None if data isn't a zstd frame or the frame doesn't carry a content size (e.g. streamed with an unset pledged size).