# iceberg.puffin > A module of iceberg's API. Puffin files: the sidecar format Iceberg keeps deletion vectors and sketches in. Magic Blob1 Blob2 ... BlobN Footer Footer := Magic FooterPayload FooterPayloadSize Flags Magic `Magic` is `PFA1` (0x50 0x46 0x41 0x31). `FooterPayloadSize` and `Flags` are little-endian 4-byte integers; bit 0 of the first flag byte says the payload is a single LZ4 frame rather than plain UTF-8 JSON. The payload is a `FileMetadata` object: a list of `BlobMetadata` (type, fields, snapshot-id, sequence-number, offset, length, compression-codec, properties) plus file-level properties. Two ways in, and the difference matters: * `PuffinFile.open` reads the footer — the trailing 12 bytes first, then just the payload — which is four small ranged reads instead of downloading the file. Use it to *discover* what a Puffin file holds. * `read_dv_blob` skips the footer entirely. A delete manifest entry already records `content_offset` and `content_size_in_bytes` for its deletion vector, and the spec requires them to match the footer exactly, so a scan reads the blob directly and never parses the footer at all. `PuffinWriter` is the other direction: blobs in, a complete file out, with the footer written as plain JSON or as a single LZ4 frame. It is what writes the deletion vectors a merge-on-read delete produces. Only `deletion-vector-v1` is decoded, by roaring.mojo, which verifies the blob's magic and CRC-32. `apache-datasketches-theta-v1` blobs are listed with their metadata (including the `ndv` property) but their sketches are not decoded: that needs the DataSketches Theta format, which nothing in Mojo implements and which no reader needs. ## Structs ### `BlobMetadata` ```mojo struct BlobMetadata ``` One entry of a Puffin footer's `blobs` list. Implements: `Copyable`, `Movable`, `Writable` Fields: - `type: String` - `fields: List[Int]` - `snapshot_id: Int64` - `sequence_number: Int64` - `offset: Int64` - `length: Int64` - `compression_codec: String`: Empty when the blob is uncompressed, as the spec's absent field means. - `property_keys: List[String]` - `property_values: List[String]` #### `property` ```mojo def property(self, key: String) -> String ``` #### `has_property` ```mojo def has_property(self, key: String) -> Bool ``` #### `is_deletion_vector` ```mojo def is_deletion_vector(self) -> Bool ``` #### `referenced_data_file` ```mojo def referenced_data_file(self) -> String ``` A deletion vector's `referenced-data-file` property. #### `cardinality` ```mojo def cardinality(self) -> Int64 ``` A deletion vector's declared number of deleted rows, or -1. #### `write_to` ```mojo def write_to(self, mut writer: T) ``` ### `PuffinFile` ```mojo struct PuffinFile ``` A Puffin file's footer, and the location it came from. Implements: `Copyable`, `Movable` Fields: - `location: String` - `blobs: List[BlobMetadata]` - `property_keys: List[String]` - `property_values: List[String]` - `footer_compressed: Bool` #### `__init__` ```mojo def __init__(out self, var location: String, var blobs: List[BlobMetadata], var property_keys: List[String], var property_values: List[String], footer_compressed: Bool) ``` #### `property` ```mojo def property(self, key: String) -> String ``` #### `created_by` ```mojo def created_by(self) -> String ``` #### `open` ```mojo def open(io: FileIO, location: String) -> Self ``` Read a Puffin footer without reading the blobs. Four ranged reads: the trailing 12 bytes, then the payload, then the two magics that bracket it. On a local file that is one `seek` each; on S3 it is a `HEAD` plus two `GET`s with `Range`. #### `from_payload` ```mojo def from_payload(var location: String, payload: Span[UInt8], compressed: Bool) -> Self ``` Parse a `FileMetadata` JSON payload. ### `PuffinWriter` ```mojo struct PuffinWriter ``` Builds a Puffin file blob by blob, footer last. ```mojo var w = PuffinWriter() _ = w.add_deletion_vector(data_file_path, bitmap) var bytes = w^.finish() ``` The blobs go into the body as they arrive, each recording the `offset` and `length` its `BlobMetadata` will claim; `finish` appends the footer, whose payload is the `FileMetadata` JSON — optionally as a single LZ4 frame, which is the only compression the format allows there. A `deletion-vector-v1` blob is written the way the spec requires and no other way: never compressed, `snapshot-id` and `sequence-number` both -1 (a DV is not owned by a snapshot; the manifest entry that references it is), `fields` empty, and the two properties a reader needs — `referenced-data-file` and `cardinality`. Implements: `Movable` Fields: - `body: List[UInt8]`: Magic, then every blob written so far. - `blobs: List[BlobMetadata]` - `property_keys: List[String]` - `property_values: List[String]` #### `__init__` ```mojo def __init__(out self, var created_by: String = PUFFIN_CREATED_BY) ``` #### `set_property` ```mojo def set_property(mut self, var key: String, var value: String) ``` Set a file-level property, replacing any previous value. #### `add_blob` ```mojo def add_blob(mut self, var type: String, var fields: List[Int], snapshot_id: Int64, sequence_number: Int64, data: Span[UInt8], var compression_codec: String = "", var property_keys: List[String] = List(), var property_values: List[String] = List()) -> Int ``` Append one blob; returns its index in `blobs`. `compression_codec` is applied here: `lz4` means a single LZ4 frame, which is the codec Puffin defines for blob payloads. Anything else is refused rather than written under a name a reader would trust. #### `add_deletion_vector` ```mojo def add_deletion_vector(mut self, referenced_data_file: String, bitmap: Bitmap64) -> Int ``` One `deletion-vector-v1` blob for one data file. The bitmap is framed by roaring.mojo — big-endian length, the `D1 D3 39 64` magic, the portable serialisation and a CRC-32 over both — which is byte-for-byte what a reader verifies. #### `blob` ```mojo def blob(self, i: Int) -> BlobMetadata ``` #### `footer_json` ```mojo def footer_json(self) -> String ``` The `FileMetadata` payload: every blob, then the file properties. #### `finish` ```mojo def finish(self, compress_footer: Bool = False) -> List[UInt8] ``` The complete file: body, then `Magic payload size flags Magic`. `compress_footer` writes the payload as one LZ4 frame and sets bit 0 of the first flag byte, which is the only flag the format defines. LZ4 is also the only footer codec Puffin allows — there is no ZSTD footer — so this is a `Bool` rather than a codec name. ## Functions ### `read_blob_bytes` ```mojo def read_blob_bytes(io: FileIO, location: String, offset: Int64, length: Int64) -> List[UInt8] ``` The raw bytes of one blob, by the offset and length a manifest records. ### `decompress_blob` ```mojo def decompress_blob(var data: List[UInt8], codec: String) -> List[UInt8] ``` Undo a blob's `compression-codec`. Only `lz4` is reachable here: `zstd` is legal in the format but no blob type Iceberg defines uses it, and deletion vectors must omit the codec entirely. ### `read_deletion_vector` ```mojo def read_deletion_vector(io: FileIO, location: String, offset: Int64, length: Int64) -> Bitmap64 ``` Read and decode one `deletion-vector-v1` blob. `decode_iceberg_dv` checks the blob's own framing — the big-endian length, the `D1 D3 39 64` magic and the CRC-32 over both — so a truncated or corrupted vector is an error here rather than silently missing deletes. ### `deleted_positions` ```mojo def deleted_positions(io: FileIO, location: String, offset: Int64, length: Int64) -> List[UInt64] ``` The deleted row positions of one deletion vector, ascending. ## Aliases ### `PUFFIN_MAGIC_0` ```mojo comptime PUFFIN_MAGIC_0 = UInt8(80) ``` ### `PUFFIN_MAGIC_1` ```mojo comptime PUFFIN_MAGIC_1 = UInt8(70) ``` ### `PUFFIN_MAGIC_2` ```mojo comptime PUFFIN_MAGIC_2 = UInt8(65) ``` ### `PUFFIN_MAGIC_3` ```mojo comptime PUFFIN_MAGIC_3 = UInt8(49) ``` ### `PUFFIN_MAGIC_LEN` ```mojo comptime PUFFIN_MAGIC_LEN = 4 ``` ### `PUFFIN_FOOTER_TAIL` ```mojo comptime PUFFIN_FOOTER_TAIL = 12 ``` `FooterPayloadSize` (4) + `Flags` (4) + trailing `Magic` (4). ### `BLOB_DELETION_VECTOR_V1` ```mojo comptime BLOB_DELETION_VECTOR_V1 = String("deletion-vector-v1") ``` ### `BLOB_THETA_V1` ```mojo comptime BLOB_THETA_V1 = String("apache-datasketches-theta-v1") ``` ### `PUFFIN_CREATED_BY` ```mojo comptime PUFFIN_CREATED_BY = String("iceberg.mojo") ``` The `created-by` file property, so a Puffin file says who made it.