# iceberg.write

> A module of iceberg's API.

Writing data files: Arrow in, Parquet out, a `DataFile` record back.

`write_data_files` takes Arrow `RecordBatch`es, aligns their columns to the
table's schema **by name** (falling back to the field id when the batch carries
one), partitions each batch by the spec's transforms, writes one Parquet file
per partition per batch through parquet.mojo's writer, and returns the
`data_file` records a manifest needs — record count, file size, split offsets,
and the per-column sizes, value counts, null counts and Appendix-D bounds that
`InclusiveMetricsEvaluator` prunes on.

The statistics come from the Parquet footer the writer just produced rather
than from a second pass over the values: whatever the writer decided the
min and max were is what the manifest reports, which is the only way the two
can agree. Bounds are truncated the way `write.metadata.metrics.default`'s
`truncate(16)` says — a lower bound is cut, an upper bound is cut and then
incremented so it stays an upper bound, and dropped when it cannot be.

## Structs

### `WriteOptions`

```mojo
struct WriteOptions
```

How to write data files.

Implements: `Copyable`, `Defaultable`, `Movable`

Fields:

- `codec: String`
- `row_group_rows: Int`: Rows per row group, which is what parquet.mojo's writer splits on.
- `page_size: Int`: Target *uncompressed* bytes of values per data page.
- `truncate: Int`: `write.metadata.metrics.default = truncate(N)`; 0 disables truncation.

#### `__init__`

```mojo
def __init__(out self)
```

#### `from_properties`

```mojo
def from_properties(properties: Dict[String, String]) -> Self
```

## Functions

### `codec_value`

```mojo
def codec_value(name: String) -> Int32
```

### `truncate_lower`

```mojo
def truncate_lower(raw: List[UInt8], kind: UInt8, n: Int) -> List[UInt8]
```

A lower bound cut to `n` units — code points for a string, bytes otherwise. Cutting a lower bound can only lower it, which is safe.

### `truncate_upper`

```mojo
def truncate_upper(raw: List[UInt8], kind: UInt8, n: Int) -> Tuple[List[UInt8], Bool]
```

An upper bound cut to `n` units and then incremented.

Truncation lowers a value, so an upper bound has to be nudged back up or
it stops bounding. For a string that means incrementing the last code
point that is not the maximum; for binary, the last byte that is not
`0xFF`. When every unit is at its maximum the bound cannot be truncated at
all and the second element of the result is `False` — the spec's answer is
to omit it.

### `bound_bytes`

```mojo
def bound_bytes(v: ScalarValue, kind: UInt8, precision: Int, scale: Int) -> Tuple[List[UInt8], Bool]
```

### `human_partition_value`

```mojo
def human_partition_value(d: Datum, transform_kind: UInt8) -> String
```

The Hive-style rendering of one partition value.

### `escape_path`

```mojo
def escape_path(s: String) -> String
```

Percent-escape everything a Hive path may not contain.

### `partition_path`

```mojo
def partition_path(spec: PartitionSpec, values: List[Datum]) -> String
```

### `align_batch`

```mojo
def align_batch(batch: RecordBatch, schema: Schema) -> List[ColumnTree]
```

The batch's columns in the table's schema order, at the table's types.

A nested column is matched child by child **by field id**, so a batch
whose struct has its fields in another order, under other names, or one
field short still lines up with the table — the same resolution the read
path does, run the other way.

### `write_parquet`

```mojo
def write_parquet(columns: List[ColumnTree], schema: Schema, options: WriteOptions) -> List[UInt8]
```

### `data_file_from_parquet`

```mojo
def data_file_from_parquet(data: List[UInt8], path: String, schema: Schema, spec: PartitionSpec, partition: List[Datum], sort_order_id: Int, options: WriteOptions) -> DataFile
```

Read back the footer we just wrote and turn it into a `data_file`.

### `write_data_files`

```mojo
def write_data_files(io: FileIO, location: String, batches: List[RecordBatch], schema: Schema, spec: PartitionSpec, sort_order_id: Int, options: WriteOptions) -> List[DataFile]
```

Write every batch out, one Parquet file per partition per batch.

## Aliases

### `DEFAULT_METRICS_TRUNCATE`

```mojo
comptime DEFAULT_METRICS_TRUNCATE = 16
```

### `HIVE_NULL`

```mojo
comptime HIVE_NULL = String("__HIVE_DEFAULT_PARTITION__")
```

### `PROP_COMPRESSION`

```mojo
comptime PROP_COMPRESSION = String("write.parquet.compression-codec")
```

### `PROP_PAGE_SIZE`

```mojo
comptime PROP_PAGE_SIZE = String("write.parquet.page-size-bytes")
```

### `PROP_ROW_GROUP_ROWS`

```mojo
comptime PROP_ROW_GROUP_ROWS = String("write.parquet.row-group-size-rows")
```

Not an Iceberg property.

Iceberg spells the row-group target in *bytes*
(`write.parquet.row-group-size-bytes`), and parquet.mojo's writer splits a
batch by a *row count*. Converting between them means guessing a compression
ratio before compressing, which would make the setting mean something
different from what it says, so the byte property is ignored and this one —
under a name no reader will mistake for the spec's — is honoured instead.