Cloud Storage

Cloud object storage is the durable file layer behind many lakes, warehouses, exports, and model datasets. In data engineering it is usually not a filesystem replacement; it is an object namespace with explicit paths, metadata, lifecycle policy, and access controls.

A discoverable object layout

A useful layout makes data version, grain, and partition visible:

gs://company-lake/raw/orders/source=stripe/dt=2026-01-01/part-000.jsonl
gs://company-lake/staged/orders/dt=2026-01-01/part-000.parquet
gs://company-lake/curated/fct_orders/v=20260102/part-000.parquet

For 73 million rows per month at about 420 bytes per row, the daily partition size is

$$\frac{73,000,000\times 420}{30\times 1024^3}\approx 0.95\text{ GiB}.$$

Targeting roughly 128 MiB files gives $0.95\text{ GiB}/128\text{ MiB}\approx 8$ files per day. That is a reasonable starting point for parallel reads. Thousands of tiny files would slow listing and planning; one huge file would underuse parallelism in data-pipelines.

Architecture

BigQuery can load from or query files in Google Cloud Storage, while data-warehouses often keep curated copies in managed table storage for governance and performance. For reproducibility, use immutable prefixes or object versioning rather than overwriting latest/ paths.

Failure modes

Object paths that omit partition keys make backfills and deletion hard. Lifecycle rules can delete raw data before audits or model investigations finish. Treating object storage as a low-latency database creates poor consistency and listing assumptions compared with purpose-built managed-storage.

References

• Google Cloud Storage documentation: About Cloud Storage objects
• Amazon S3 documentation: What is Amazon S3?
• Amazon S3 documentation: Retaining multiple versions of objects with S3 Versioning

Nav

Section — Data Engineering

← BigQuery Batch Versus Streaming →