magmalake iceberg

iceberg · API

iceberg.io

File access: locations in, bytes out — over local files, S3, GCS, Azure or plain HTTP.

Iceberg metadata stores absolute locations — a manifest list names its manifests by full URI, a manifest names its data files by full URI. That is fine in place and awkward everywhere else: a table copied to another path, a warehouse mounted somewhere new, or a test fixture checked into a repo all need those locations redirected without touching the metadata. FileIO therefore resolves every location through an ordered list of prefix rewrites before opening it. FileIO.local() does none; rebase(old, new) adds one.

The backends come from objectstore.mojo: FileIOResolver picks one by URI scheme, configures it from Iceberg property names (s3.endpoint, s3.access-key-id, gcs.oauth2.token, adls.sas-token.<account>, …) and from the storage-credentials a REST catalog vends, and falls back to the AWS_* environment. Everything above this line only ever sees a location and a byte range.

InputFile is objectstore's trait, a superset of the one this module used to declare: location, exists, read_all as before, plus length and read_range, which is what a Parquet reader needs in order to fetch a footer without downloading the whole file.

Structs

FileIO

struct FileIO

Resolves Iceberg locations to bytes.

Rewrites are applied in the order they were added; the first prefix that matches wins. Properties and vended credentials configure the object-store backends, and are ignored for file://.

Implements Copyable, Movable

Fields
resolver: FileIOResolver
base_location: String: The table's own location, used to resolve relative locations.

__init__

def __init__(out self)
def __init__(out self, var resolver: FileIOResolver)

local

def local() -> Self

with_properties

def with_properties(var properties: Dict[String, String]) -> Self

rebase

def rebase(mut self, var old_prefix: String, var new_prefix: String)

Redirect every location under old_prefix to new_prefix.

with_base

def with_base(mut self, var location: String)

Set the table root that relative locations resolve against.

set

def set(mut self, var key: String, var value: String)

Set one storage property, using Iceberg's own property names.

add_storage_credential

def add_storage_credential(mut self, var prefix: String, var config: Dict[String, String])

Absorb one storage-credentials entry from a loadTable response.

properties_for

def properties_for(self, location: String) -> Dict[String, String]

resolve

def resolve(self, location: String) -> String

The location this one should actually be read from.

Local locations come back as bare filesystem paths, which is what the callers that hand a path to open() or listdir() expect; every other scheme comes back as a URI.

absolute

def absolute(self, location: String) -> String

A relative location joined to the table root; anything else as is.

local_path

def local_path(self, location: String) -> String

The filesystem path for a location that must be local.

is_local

def is_local(self, location: String) -> Bool

new_input

def new_input(self, location: String) -> AnyInputFile

read_all

def read_all(self, location: String) -> List[UInt8]

read_range

def read_range(self, location: String, offset: Int, length: Int) -> List[UInt8]

read_ranges

def read_ranges(self, location: String, ranges: List[ByteRange]) -> List[List[UInt8]]

Several ranges of one file, in order. On object storage nearby ranges are coalesced and the rest fetched at once.

read_ranges_many

def read_ranges_many(self, locations: List[String], ranges: List[List[ByteRange]]) -> List[List[List[UInt8]]]

ranges[i] of locations[i] for every i; on S3 every file's ranges are fetched in one batch.

read_all_many

def read_all_many(self, locations: List[String]) -> List[List[UInt8]]

Several whole files, in order; on S3 fetched at once.

length

def length(self, location: String) -> Int

read_text

def read_text(self, location: String) -> String

exists

def exists(self, location: String) -> Bool

new_output

def new_output(self, location: String) -> AnyOutputFile

write_all

def write_all(self, location: String, data: Span[UInt8])

Overwrite location with data, creating parents as needed.

write_new

def write_new(self, location: String, data: Span[UInt8])

Write location, failing if something is already there.

This is the primitive an optimistic commit needs: two writers racing on the same metadata version must not both believe they won. Only a local filesystem really provides it — S3 has no atomic create, which the spec itself calls out — so FilesystemCatalog refuses to commit over an object store without being told to.

write_text

def write_text(self, location: String, text: String)

delete

def delete(self, location: String)

Remove one object. A missing one is not an error.

The location is resolved to a bare filesystem path first when it is local: list() hands back file:// plus whatever path it was given, and file://build/x is not a well-formed URI — a URI parser reads build as the host — so a relative local location has to lose its scheme before an object store's URI parsing gets hold of it.

list

def list(self, prefix: String) -> List[String]

Every location under prefix, as full URIs.

list_names

def list_names(self, directory: String) -> List[String]

The immediate children of a directory, by name.

Object stores have no directories, so the names are derived from the keys: everything under the prefix, cut at the first /, deduplicated. A local directory is listed directly.

Functions

strip_scheme

def strip_scheme(location: String) -> String

file:///a/b -> /a/b. Other schemes are returned unchanged.

is_local_location

def is_local_location(location: String) -> Bool

True when this location names an ordinary filesystem path.

basename

def basename(path: String) -> String

dirname

def dirname(path: String) -> String

join_path

def join_path(a: String, b: String) -> String

Aliases

comptime FILE_SCHEME = String("file://")