Data Export

Data Export to Google Cloud Storage: permissions step by step

How to prepare a GCS bucket and issue a service-account key scoped to it, so exported partitions land in your own storage.

Google Cloud Storage receives partitioned Parquet or CSV files, copied into a bucket you own. AdShift authenticates with a service-account key you issue and can revoke.

If your data ends up in BigQuery anyway, consider exporting to BigQuery directly instead — that path needs no key from you at all.

This article is the permissions walkthrough. For what gets exported and on what schedule, see Data Export.


What you need

  • A GCP project where you can create a bucket and a service account.
  • About five minutes.

1. Create a dedicated bucket

Use a bucket only for AdShift data. Exported partitions are written and, on a re-delivery, written again under a new version folder.

In the Cloud Console: Cloud Storage → Buckets → Create. Pick the location closest to whatever will read the data, and leave Uniform bucket-level access on — the permissions below are IAM roles, and uniform access is what makes them apply cleanly.

If you want the data under a folder rather than at the bucket root, decide the prefix now, e.g. adshift. You will enter it in the connection.


2. Create a service account for AdShift

IAM & Admin → Service accounts → Create service account. Name it something recognisable, e.g. adshift-data-export. Grant it no project-level roles on the creation screen — the access it needs is on one bucket, granted in the next step.


3. Grant it access to the bucket only

Cloud Storage → your bucket → Permissions → Grant access.

  • Principal: the service account address from step 2.
  • Role: Storage Object Admin (roles/storage.objectAdmin).

Grant this on the bucket, not the project. A project-level grant works too, but it gives AdShift access to every bucket you own — more than the export needs, and the first thing a security review flags.

Object Admin is the simplest fit because a delivery creates, reads and deletes objects: we write the partition, read a blob back during the connection test, and delete the probe object the test leaves behind, as well as clearing a partial partition before a retry. If you prefer to assemble the permissions yourself, the set is storage.objects.create, storage.objects.get, storage.objects.delete and storage.objects.list.


4. Create a key

On the service account: Keys → Add key → Create new key → JSON. The file downloads once.

Paste the whole file into the connection form, { to }. A partial paste fails validation immediately rather than later as an authentication error.

The key is stored encrypted and is never shown back to you; the connection form displays [redacted] and leaving the field blank on an edit keeps the stored value.

If your organisation blocks service-account key creation by policy — the iam.disableServiceAccountKeyCreation constraint — use the BigQuery destination instead, which does not need one.


5. Connect it in AdShift

Go to Data Export → Connections → New connection, pick GCS and fill in:

Field What to enter
Connection name Lowercase letters, digits and hyphens, e.g. gcs-marketing
GCS bucket The bucket name only, without gs://
Path prefix Optional, e.g. adshift. Leave empty to write at the bucket root
Service account JSON The whole key file from step 4

The panel shows a live preview of where data will land as you type.

Click Test connection. Three checks run — write a probe object, read it back, delete it — each reported separately, so a failure names the permission that is missing.

Then click Save, open Reports on the connection, and pick your datasets.


What you get in the bucket

gs://your-bucket/
  adshift/                          <- your path prefix, if you set one
    t=clicks/dt=2026-09-15/h=14/
      data_0_0_0.snappy.parquet
      _SUCCESS
  • t= is the dataset, dt= the date, h= the hour. Daily datasets have no h= element.
  • The layout is Hive-style, so BigQuery external tables, Dataproc and Spark discover the partitions without extra configuration.
  • _SUCCESS is written after every data file. Wait for it before reading a folder; a folder without it is still being written.
  • A re-delivery after late data adds a v=2 element (then v=3, and so on). The first delivery of a partition has no version element. Read the highest version present and treat earlier ones as superseded.
  • If you chose By app folder structure, an app_id= element appears before the version.

If something goes wrong

"Invalid JWT Signature" or "invalid_grant" — the key was truncated on paste, or it has been deleted in GCP. Create a new key and paste the whole file.

403 … does not have storage.objects.create access — the role was granted on the project instead of the bucket, or on a different bucket. Check the Permissions tab of the bucket itself.

The write check passes but delete fails — the service account has Storage Object Creator rather than Object Admin. AdShift needs delete to clear a partial partition before a retry; without it a retried delivery mixes files from two attempts.

Bucket is requester pays — not supported. Deliveries would be billed to AdShift's project with no way to attribute them; turn requester pays off on this bucket.

Objects appear but a downstream job does not see them — it is probably reading before _SUCCESS. A folder without the marker is still being written.


Revoking access

Disable or delete the connection in AdShift first, then delete the key or remove the IAM binding. Doing it the other way round just makes the next delivery fail with a permission error. Data already delivered is in your bucket and is unaffected either way.


See also