The lz4 package, from lz4 0.1.4, as published on mojoshelf (commit 23cd6e54967d), by mojo doc from Mojo 1.2.0.
Source.
lz4 — Mojo bindings to liblz4 (block format + frame format) via a thin C shim (liblz4mojo.{dylib,so}), loaded through an OwnedDLHandle.
Two independent codecs live in one liblz4:
LZ4_RAW in Parquet): a bare compressed blob with no
size prefix — the uncompressed length must be supplied out of band (that's
exactly what a Parquet page header carries). Legacy Parquet's LZ4
compression carries the same blocks but wrapped in a Hadoop framing of
[BE u32 uncompressed_len][BE u32 compressed_len] pairs.LZ4F): a self-describing container with a magic
number, optional content-size field, and (optionally) checksums. This is
the only codec Apache Iceberg allows for Puffin file footers, which is
why compress_frame always sets the content-size flag by default.Mirrors zlib.mojo's FFI pattern: a single-call C wrapper
(shim/lz4_wrapper.c, built to $CONDA_PREFIX/lib/liblz4mojo.{dylib,so} by the
lz4-shim pixi package) so Mojo never reads back internal library state after
a foreign call. The handle is opened once per process and never closed
(_LIB below) — a dlopen/dlclose cycle costs hundreds of microseconds,
far more than compressing a Parquet page, so opening one per call made the
binding's fixed cost dwarf the work it wrapped. It is passed as a borrowed
(default-convention) parameter to each worker function, so Mojo's ASAP
destruction can't dlclose the library before the C call inside that worker
runs.
compress_blockdef compress_block(data: Span[UInt8], acceleration: Int = Int(1)) -> List[UInt8]Compress to a raw LZ4 block (LZ4_RAW) — no size prefix. The caller (e.g. a Parquet page writer) must record the uncompressed length out of band; decompress_block needs it back to decode.
data: Bytes to compress.acceleration: >=1; higher trades compression ratio for speed
(1 matches LZ4_compress_default).compress_block_hcdef compress_block_hc(data: Span[UInt8], level: Int = Int(9)) -> List[UInt8]Compress to a raw LZ4 block using the high-compression (HC) encoder. Same wire format as compress_block — slower, smaller output.
data: Bytes to compress.level: 1..12 (LZ4HC_CLEVEL_MIN..MAX); higher is slower and smaller.decompress_block_intodef decompress_block_into(data: Span[UInt8], dst: Span[UInt8]) -> IntDecompress a raw LZ4 block into a caller-supplied buffer. dst must be at least as large as the true uncompressed size — LZ4 blocks carry no size of their own, so this can't grow the buffer for you.
Returns the number of bytes written.
decompress_blockdef decompress_block(data: Span[UInt8], uncompressed_size: Int) -> List[UInt8]Decompress a raw LZ4 block. uncompressed_size must be exact — Parquet page headers carry it, since LZ4 blocks carry no size prefix.
compress_framedef compress_frame(data: Span[UInt8], level: Int = Int(0), content_size: Bool = True) -> List[UInt8]Compress to a single LZ4 frame (LZ4F) — the format Apache Iceberg requires for Puffin file footers. content_size=True (the default, and the Puffin requirement) writes the uncompressed length into the frame header so a reader can size its output buffer up front.
data: Bytes to compress.level: 0 = fast mode (LZ4 default); >=1 selects the HC encoder
(1..12, higher is slower and smaller).content_size: Whether to record the uncompressed length in the
frame header.frame_content_sizedef frame_content_size(data: Span[UInt8]) -> Optional[Int]Read the uncompressed content size from an LZ4 frame header without decompressing, if the frame carries one.
Returns None if the frame doesn't carry a content size (LZ4F itself
can't distinguish "unspecified" from a genuine zero-length payload —
both are stored as 0), or raises if data isn't a valid frame header.
is_lz4_framedef is_lz4_frame(data: Span[UInt8]) -> BoolWhether data starts with the LZ4 frame magic number (0x184D2204, little-endian) — the standard way to tell an LZ4F frame apart from a bare LZ4 block, which has no magic at all.
decompress_framedef decompress_frame(data: Span[UInt8]) -> List[UInt8]Decompress a single LZ4 frame. Uses the frame header's content-size field to size the output buffer exactly when present (the case for every frame compress_frame produces, and the Puffin convention); otherwise grows the buffer and retries until the whole frame fits.
decompress_hadoopdef decompress_hadoop(data: Span[UInt8], uncompressed_size: Int) -> List[UInt8]Decompress legacy Parquet LZ4 (Hadoop-framed): a sequence of [BE u32 uncompressed_len][BE u32 compressed_len][block…] chunks, each block an ordinary LZ4_RAW block. This is the format parquet-mr historically wrote for codec LZ4; new writers should use LZ4_RAW (compress_block/decompress_block) instead — see PARQUET-1974.