# iceberg.delete

> A module of iceberg's API.

Row-level delete and overwrite: which rows go, and how they are recorded.

Two strategies, chosen by the table property `write.delete.mode`:

* **`copy-on-write`** (the default, as in Java, and the only thing PyIceberg
  can do) — every data file with a matching row is rewritten without those
  rows and the original is marked `DELETED`. The result needs no delete files
  at all, so *every* reader can read it, at the cost of rewriting whole files
  to remove one row.
* **`merge-on-read`** — the rows stay where they are and a delete file records
  their positions. Under **v3** that is a **deletion vector**: one Puffin
  `deletion-vector-v1` blob per data file, referenced by a manifest entry with
  `content=1`, `file_format=puffin`, `referenced_data_file` and the
  `content_offset`/`content_size_in_bytes` that must match the blob's footer
  entry exactly. Under **v2** it is a **position delete file**: a Parquet file
  of `(file_path, pos)` pairs — the two reserved field ids 2147483546 and
  2147483545 — sorted by path then position, one per partition.

Whichever strategy is in force, a data file whose rows are *all* deleted is
removed rather than emptied. That is the metadata-delete fast path: when the
scan's residual for a file is `true` — the partition predicate alone already
proves every row matches — the file is not read at all, and the delete costs
one manifest rewrite.

**Merging.** A deletion vector must contain every position deleted from its
data file, not just the new ones: the spec makes a new vector absorb the ones
it replaces, and a reader that finds a vector ignores every position delete
file for that data file. So a second delete against a file that already has a
vector reads the old vector, unions the new positions in, writes a fresh blob,
and marks the old vector's manifest entry `DELETED`. Position delete *files*
are left alone even when their positions are absorbed: one of them can cover
several data files, and removing it would resurrect rows in the others.

**Overwrite** is a delete and an append in one snapshot, always copy-on-write,
with `operation=overwrite`. `dynamic_partition_overwrite` derives the filter
from the data instead of taking one: it replaces exactly the partitions the
new rows land in, which needs no row-level work at all — a partition is
replaced by removing every data file in it.

## Structs

### `RowDelete`

```mojo
struct RowDelete
```

One data file, and which of its rows this delete takes out.

Implements: `Copyable`, `Movable`

Fields:

- `task: FileScanTask`
- `positions: List[UInt64]`: The positions the filter matched, ascending. Live rows only: a row a previous delete already removed is not matched again.
- `existing: List[UInt64]`: Positions already deleted by the delete files that apply to this data file — what a new deletion vector has to absorb.
- `live_rows: Int64`: Rows still alive in the file before this delete.
- `whole_file: Bool`: Every live row matched, so the file goes rather than being marked up.
- `superseded: List[DataFile]`: Delete files this commit replaces — the deletion vectors a new vector absorbs.

#### `__init__`

```mojo
def __init__(out self, var task: FileScanTask, var positions: List[UInt64], var existing: List[UInt64], live_rows: Int64, whole_file: Bool, var superseded: List[DataFile])
```

## Functions

### `delete_mode_of`

```mojo
def delete_mode_of(properties: Dict[String, String]) -> String
```

`write.delete.mode`, defaulting to copy-on-write as Java does.

### `plan_row_deletes`

```mojo
def plan_row_deletes(scan: TableScan, filter_dsl: String, options: ScanOptions = ScanOptions()) -> List[RowDelete]
```

Which rows of which data files a `DELETE ... WHERE` takes out.

### `existing_complement`

```mojo
def existing_complement(record_count: Int64, already: List[Bool]) -> List[UInt64]
```

Every position of a data file that is not already deleted.

### `deleted_row_count`

```mojo
def deleted_row_count(plans: List[RowDelete]) -> Int64
```

### `write_deletion_vectors`

```mojo
def write_deletion_vectors(io: FileIO, location: String, plans: List[RowDelete]) -> List[DataFile]
```

One Puffin file holding a vector per partially-deleted data file.

Every vector carries the file's *whole* delete set — the positions this
commit adds, unioned with the ones already there — because a reader that
finds a vector uses it alone.

### `write_position_deletes`

```mojo
def write_position_deletes(io: FileIO, location: String, plans: List[RowDelete], spec: PartitionSpec, options: WriteOptions) -> List[DataFile]
```

One Parquet file of `(file_path, pos)` per partition, sorted.

The two columns are written under the reserved ids the spec assigns them,
and the rows are sorted by path then position, which is what makes a
reader able to merge them without buffering the whole file.

### `surviving_batch`

```mojo
def surviving_batch(io: FileIO, plan: RowDelete, schema: Schema, spec: PartitionSpec, mapping: NameMapping, options: ScanOptions) -> RecordBatch
```

Everything left of one data file once this delete has taken its rows.

Read with no filter and with `_pos`, then dropped by position, rather than
read with the *negation* of the filter: a three-valued `NOT` does not keep
the rows a two-valued one drops, and "everything the delete did not match"
is the only definition that cannot disagree with the delete itself.

### `rewrite_files`

```mojo
def rewrite_files(io: FileIO, metadata: TableMetadata, plans: List[RowDelete], options: ScanOptions) -> List[DataFile]
```

The replacement data files for every partially-deleted file.

### `prepare_delete`

```mojo
def prepare_delete(io: FileIO, metadata: TableMetadata, filter_dsl: String, mode: String = "", options: ScanOptions = ScanOptions()) -> AppendResult
```

`DELETE FROM t WHERE <filter>`, as one prepared snapshot.

Raises when the filter matches nothing, so a caller cannot commit an empty
snapshot by accident.

### `prepare_delete_from`

```mojo
def prepare_delete_from(io: FileIO, metadata: TableMetadata, plans: List[RowDelete], mode: String, options: ScanOptions = ScanOptions()) -> AppendResult
```

`prepare_delete` with the planning already done — which is what a caller that needs the row count before it commits wants.

### `prepare_overwrite`

```mojo
def prepare_overwrite(io: FileIO, metadata: TableMetadata, batches: List[RecordBatch], filter_dsl: String = "[\22true\22]", options: ScanOptions = ScanOptions()) -> AppendResult
```

Delete every row the filter matches and add `batches`, in one snapshot.

Copy-on-write, always: an overwrite that left merge-on-read deletes behind
would need the added rows to sort after the deletes it wrote in the same
commit, and the spec's scope rules do not let a delete from one commit
skip data added by the same commit.

### `prepare_dynamic_partition_overwrite`

```mojo
def prepare_dynamic_partition_overwrite(io: FileIO, metadata: TableMetadata, batches: List[RecordBatch]) -> AppendResult
```

Replace exactly the partitions the new rows land in.

No row is examined: the new data files say which partitions they belong
to, and every existing data file in one of those partitions is removed
whole. A partition the new data does not touch is left alone, which is the
difference between this and an unfiltered overwrite.

### `write_equality_deletes`

```mojo
def write_equality_deletes(io: FileIO, location: String, rows: RecordBatch, schema: Schema, equality_ids: List[Int], options: WriteOptions) -> DataFile
```

One equality delete file: the values, and the ids they match on.

A row is deleted when every one of the `equality_ids` columns equals the
delete row's value for it, `null` included — the spec's "a null value in a
delete column matches a row if the row's value is null". The file holds
only those columns, under the table's own field ids, and the manifest
entry carries `equality_ids` so a reader knows which they are.

Written with the **unpartitioned** spec, which is what makes the file
apply to the whole table: a partitioned equality delete only reaches its
own partition, and pairing one with the right partition tuple is a job for
a caller that knows the data. Format version 2 only — v3 replaced these
with deletion vectors for positions and v4 is deprecating writing them.

### `prepare_equality_delete`

```mojo
def prepare_equality_delete(io: FileIO, metadata: TableMetadata, rows: RecordBatch, equality_ids: List[Int]) -> AppendResult
```

`DELETE FROM t WHERE (a, b) IN (<rows>)`, as one prepared snapshot.

## Aliases

### `PROP_DELETE_MODE`

```mojo
comptime PROP_DELETE_MODE = String("write.delete.mode")
```

### `MODE_COPY_ON_WRITE`

```mojo
comptime MODE_COPY_ON_WRITE = String("copy-on-write")
```

### `MODE_MERGE_ON_READ`

```mojo
comptime MODE_MERGE_ON_READ = String("merge-on-read")
```

### `POS_DELETE_SCHEMA_JSON`

```mojo
comptime POS_DELETE_SCHEMA_JSON = String("{\22schema-id\22:0,\22type\22:\22struct\22,\22fields\22:[{\22id\22:2147483546,\22name\22:\22file_path\22,\22required\22:true,\22type\22:\22string\22},{\22id\22:2147483545,\22name\22:\22pos\22,\22required\22:true,\22type\22:\22long\22}]}")
```