magmalake iceberg

iceberg · API

iceberg.manifest_write

Writing manifests and manifest lists.

The two Avro files a commit produces, byte-compatible with what iceberg-rust, PyIceberg and Java write:

  • a manifest (<uuid>-m<n>.avro) — one manifest_entry per file, with the data_file struct the spec's Appendix A fixes for the format version, and the file-level metadata (schema, schema-id, partition-spec, partition-spec-id, format-version, content) that tells a reader how to type the partition column;
  • a manifest list (snap-<id>-<n>-<uuid>.avro) — one manifest_file per manifest, with the counts and the per-partition-field summaries that let a scan skip a manifest without opening it, plus the file-level metadata (snapshot-id, parent-snapshot-id, sequence-number, format-version).

The Avro schema for each is generated per format version rather than hard-coded once, because the field sets genuinely differ: v1 has a required snapshot_id and a required block_size_in_bytes and no content; v2 adds content, sequence_number, file_sequence_number and equality_ids; v3 adds first_row_id, referenced_data_file, content_offset and content_size_in_bytes to data_file and first_row_id to manifest_file. avro.DataFileWriter.set_schema_json writes the generated JSON verbatim, so the header is exactly what was intended rather than a printed approximation.

The partition field is a record typed by the writing spec, which is what makes a partition tuple readable at all: the manifest's own file metadata is the only place that typing exists.

Structs

PartitionTyping

struct PartitionTyping

The result type of every field of a partition spec, flattened.

Implements Copyable, Defaultable, Movable, Sized

Fields
names: List[String]
field_ids: List[Int]
kinds: List[UInt8]
precisions: List[Int]
scales: List[Int]
lengths: List[Int]

__init__

def __init__(out self)

__len__

def __len__(self) -> Int

of

def of(spec: PartitionSpec, schema: Schema) -> Self

WrittenManifest

struct WrittenManifest

A manifest's bytes and everything its manifest-list entry needs.

Implements Copyable, Defaultable, Movable

Fields
bytes: List[UInt8]
added_files_count: Int
existing_files_count: Int
deleted_files_count: Int
added_rows_count: Int64
existing_rows_count: Int64
deleted_rows_count: Int64
min_sequence_number: Int64
partitions: List[FieldSummary]
content: Int

__init__

def __init__(out self)

to_manifest_file

def to_manifest_file(self, path: String, spec_id: Int, snapshot_id: Int64, sequence_number: Int64, first_row_id: Int64, has_first_row_id: Bool) -> ManifestFile

Functions

decimal_byte_width

def decimal_byte_width(precision: Int) -> Int

The fixed width Avro stores a decimal of this precision in.

avro_prim_json

def avro_prim_json(kind: UInt8, precision: Int, scale: Int, length: Int, hint: String) -> String

partition_record_json

def partition_record_json(typing: PartitionTyping) -> String

data_file_record_json

def data_file_record_json(format_version: Int, typing: PartitionTyping) -> String

manifest_entry_schema_json

def manifest_entry_schema_json(format_version: Int, typing: PartitionTyping) -> String

manifest_list_schema_json

def manifest_list_schema_json(format_version: Int) -> String

put

def put(mut names: List[String], mut values: List[Value], n: String, var v: Value)

partition_value

def partition_value(d: Datum, kind: UInt8, precision: Int) -> Value

data_file_value

def data_file_value(df: DataFile, format_version: Int, typing: PartitionTyping) -> Value

manifest_entry_value

def manifest_entry_value(entry: ManifestEntry, format_version: Int, snapshot_id: Int64, typing: PartitionTyping) -> Value

write_manifest

def write_manifest(format_version: Int, schema: Schema, spec: PartitionSpec, content: Int, snapshot_id: Int64, sequence_number: Int64, entries: List[ManifestEntry], codec: String = "deflate") -> WrittenManifest

One manifest, as bytes plus the summary its list entry needs.

manifest_file_value

def manifest_file_value(mf: ManifestFile, format_version: Int) -> Value

write_manifest_list

def write_manifest_list(format_version: Int, snapshot_id: Int64, parent_snapshot_id: Int64, has_parent: Bool, sequence_number: Int64, manifests: List[ManifestFile], codec: String = "deflate") -> List[UInt8]