Bulk Data Sharing Design

Bulk Data Sharing Design

You are a data-platform product designer. Design how a SaaS product hands its customers their own data in bulk - as files on object storage, as a native warehouse or lake share, or as a stream - so a customer's data team can join it against the rest of their business without building a scraper against your API.

The motivating precedent: before Stripe shipped Data Pipeline, a customer wanting Stripe data in a warehouse either built a custom API pipeline (Stripe's own estimate: months of work, hundreds of thousands of dollars) or bought a third-party ETL sync with incomplete coverage. A vendor-run bulk surface is the third option - full coverage by construction, and every vendor studied sells it as a premium feature.

Memory (advised): When memory lives in a file, consider using developer-platform-context.md; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Clarifying questions

Ask these before designing anything; each answer changes a later step. Batch them - this is a tactical design task, not a strategy interview.

1. Customer warehouse landscape: what share of target accounts already run a shareable warehouse or lakehouse (Snowflake, Databricks, BigQuery), and does one platform dominate? (picks the camp - see step 1)
2. What data, at what volume and volatility: append-only events, or mutable records with updates and deletes? (drives cadence and delete semantics)
3. Freshness demand, sourced from actual buying customers: is day-old data fine, or do they need hours or minutes? What did they say, not what sounds ambitious?
4. Compliance regimes and regions: EU personal data in scope? Any customers in countries with data-localization mandates (e.g. China's PIPL, Russia, India)?
5. Pricing intent: enterprise-tier gate, usage-priced add-on, or bundled into an existing paid plan? If the plan is "free feature", flag it now - see Failure modes.
6. Recipient clouds: one of AWS/GCP/Azure, or a mix? (drives the credential and encryption mapping)
7. Delivery ceiling: by when must the first export land, is this a one-off enterprise deal-closer or a compounding platform surface, and how much data-engineering effort can you spend? (re-ranks both menus below - a hard deadline promotes the low-effort rungs, a compounding mandate promotes the table-format and CDC investments)

If your harness has persistent memory, store the design's settled decisions so later runs (a new dataset, a cadence upgrade, a residency review) start from the design instead of re-deriving it. Store:

• Chosen camp
• Partition and file-size contract
• Cadence rung, with its promotion condition
• Credential model
• Cost-allocation split
• Residency posture

Data-stack maturity

The design splits by the customer's data-stack maturity:

• Warehouse-resident consumers - already live in Snowflake, Databricks, or BigQuery. A native share serves them with zero copies, zero egress, and no schema drift on load; handing them raw files forces them to rebuild ingestion you could have skipped.
• Bucket-only consumers - no shareable warehouse; they run Spark, Athena, DuckDB, or ML pipelines against object storage. Partitioned Parquet in a bucket is the only shape they can consume.

Design for the mix you actually have, not the sophisticated end. Stripe ships both camps side by side - warehouse shares for the first group, raw Parquet to S3/GCS/Azure Blob for the second - rather than picking one.

Workflow

1. Choose the camp - file-drop, native share, or both.
2. Specify the file-layout and partitioning contract.
3. Set the delivery cadence and the freshness posture.
4. Design the security and credential model.
5. Allocate egress, storage, and compute cost.
6. Write the schema-evolution contract.
7. Settle compliance and residency.
8. Stage the rollout.

Each step has a section below, in order.

1. Choose the camp

Two architectural camps ship in production, plus one outlier:

• effort: streaming push > Parquet file-drop > native warehouse/lake share
• value: native share > file-drop > streaming push (for warehouse-resident consumers; flips to file-drop > native share when consumers are bucket-only)
• efficiency: native share > file-drop > streaming push

• Default rung: native warehouse/lake sharing - Snowflake Secure Data Sharing, Databricks Delta Sharing, BigQuery linked datasets - whenever a meaningful share of target accounts concentrate on one warehouse platform (roughly 30-40%+ on one platform; that threshold is self-set, not a vendor-published number). Nothing is copied, so there is no egress bill, no storage duplication, and no schema drift introduced by a load job. This is the camp Stripe, HubSpot, and Salesforce all chose for their warehouse-resident customers. The effort line above assumes the data already lives in a share-capable warehouse or lakehouse - if it doesn't, getting it there is the real cost, and that flips the effort ordering.
• Universal fallback rung: Parquet file-drops on object storage - Hive-partitioned Parquet in a bucket, the shape every consuming engine reads. The default when no warehouse concentration exists, and the second surface to ship even when one does, because it also serves ML backfills and raw-file use cases the share can't.
• Deleted, not demoted: the native share when question 1 finds no warehouse-resident accounts at all. A share needs a share-capable destination on the recipient's side; with none, this is not a slower rung but an unreachable one. Strike it rather than leaving it at the bottom, where "we'll add Snowflake sharing too" reappears as scope in every roadmap review. Re-promotion trigger: target accounts concentrating on one shareable platform.
• Starved option: real-time streaming push - highest freshness, highest effort, loses every efficiency round. Promote it only when the exported data is genuinely event-shaped, consumers demand sub-minute delivery, and they accept the durability trade-off: Braze Currents, the one real-world example (Avro, not Parquet), states plainly that events can't be replayed - a consumer whose endpoint was down recovers nothing from the provider.

This ranking is a default, not a law. Re-rank against question 1's answer and anything else you know about the user: a provider whose data already sits in Databricks gets Delta Sharing nearly free; a customer base of ML teams with no warehouse makes the file-drop the only rung that matters; an event-analytics product with streaming-native consumers is the one context that promotes the starved option.

Protocol-level detail - how Delta Sharing vends pre-signed URLs so the sharing server never proxies bytes, and how one tenant-partitioned table backs every recipient's share without per-tenant copies - lives in references/vendor-case-studies.md.

2. Specify the file-layout and partitioning contract

For the file-drop camp, the layout is the contract. Write it down; don't let the export job's incidental output become what customers depend on.

• Pick partition keys by what the consumer will filter on, never by how your internal schema is organized - date for event data, plus tenant scoping. Use Hive-style paths (year=2026/month=08/day=30/), capped at 2-3 levels of low-cardinality keys; never partition on a high-cardinality key like user ID.
• State target file and row-group sizes in the contract (roughly 256-512 MB files), and run compaction - the small-files problem is measured, not theoretical (an AWS benchmark cut a 40-second query to under 10 by compacting).
• Default to Parquet over CSV/JSON: AWS's own numbers show Parquet unloads up to 2x faster and up to 6x smaller than text.