magmalake objectstore

objectstore · API

objectstore.gcs

objectstore.gcs — Google Cloud Storage over the S3-compatible XML endpoint.

GCS exposes two APIs. The JSON API (storage.googleapis.com/storage/v1/…) returns JSON, which would mean carrying a JSON parser here purely to read an object listing. The XML API (storage.googleapis.com/<bucket>/<key>) is byte-for-byte the S3 protocol for everything this tin does — GET with Range, PUT, DELETE, and a ListBucketResult document the S3 parser in s3.mojo already reads. So this module uses the XML endpoint, and is correspondingly small.

Authentication is an OAuth2 bearer token supplied by the caller. Minting one from a service-account key requires RS256 JWT signing, and RSA is out of scope — it is a great deal more code than SHA-256 was, for something gcloud auth print-access-token, a metadata server, or the surrounding application already has. Pass the token in as gcs.oauth2.token, which is Iceberg's own property name for exactly this.

GCS also accepts SigV4 with HMAC "interoperability" keys; a caller who has those can point S3Client at https://storage.googleapis.com instead.

Structs

GcsConfig

struct GcsConfig

Implements Copyable, Movable

Fields
endpoint: String
oauth_token: String

__init__

def __init__(out self)
def __init__(out self, var endpoint: String, var oauth_token: String)

from_properties

def from_properties(props: Dict[String, String]) -> Self

Iceberg's GCS property names: gcs.oauth2.token and gcs.service.host.

headers

def headers(self) -> List[Header]

url_for

def url_for(self, bucket: String, key: String) -> String

GcsClient

struct GcsClient

Implements Copyable, Movable

Fields
config: GcsConfig
http: HttpClient

__init__

def __init__(out self, var config: GcsConfig)
def __init__(out self, var config: GcsConfig, var http: HttpClient)

input

def input(self, bucket: String, key: String) -> HttpInputFile

output

def output(self, bucket: String, key: String) -> HttpOutputFile

delete_object

def delete_object(self, bucket: String, key: String)

list_objects

def list_objects(self, bucket: String, prefix: String = "", delimiter: String = "", continuation_token: String = "") -> ListResult

One page of the XML API's ListBucketResult.

GCS implements S3's list-type=2 continuation tokens, so the response parses with s3.parse_list_objects unchanged.

list_all

def list_all(self, bucket: String, prefix: String = "") -> ListResult

Functions

split_gcs_uri

def split_gcs_uri(location: String) -> Tuple[String, String]

gs://bucket/a/b -> ("bucket", "a/b"). gcs:// is accepted too.

Aliases

comptime DEFAULT_GCS_ENDPOINT = String("https://storage.googleapis.com")