Google Cloud Storage destination

Renta writes data from any connected source into your own Google Cloud Storage bucket as Parquet objects, laid out one folder per pipeline and refreshed on every run.

Renta signs every request to the bucket with a service account key that you upload once. Nothing in your bucket has to be publicly reachable, and no person's Google account is involved.

Requirements

The connection needs a service account key, a bucket that already exists, and two IAM roles for that service account.

1. A service account with a JSON key.
   In Google Cloud Console, open IAM & Admin > Service Accounts, create a service account or pick an existing one, and on the Keys tab click Add key > Create new key with the JSON type. The downloaded file is what you upload to Renta.
2. Storage Object Admin on the bucket.
   Renta lists, uploads, and deletes objects inside the folders it manages. Grant roles/storage.objectAdmin on the bucket, or on the project if the bucket is not the only one Renta will write into.
3. Viewer on the project.
   The connection page lists the buckets of the project for you to choose from, and that listing needs storage.buckets.list. The predefined Viewer role (roles/viewer) grants it. Without this role the bucket dropdown stays empty.
4. An existing bucket.
   Renta never creates the bucket. It creates only the folders inside it.

Connect Google Cloud Storage as a destination

A connection stores one service account key, one project, and one bucket.

Go to the CatalogStep 1

1. In the Renta console, open Connections > Catalog in the left sidebar.
2. Expand the Data lake category and click the Google Cloud Storage card.

Upload the service account keyStep 2

Click Upload a Service Account JSON file and select the key you downloaded from Google Cloud.

Renta checks the key with Google right away. A file that is not a service account key, or a key Google rejects, returns an error on this page, and nothing is stored. A valid key opens the connection settings.

Choose the project and the bucketStep 3

1. Destination name.
   A label used only inside the Renta console.
2. Google Cloud Storage project.
   The project of the key is preselected. Other projects appear if the service account can see them.
3. Google Cloud Storage bucket name.
   Pick the bucket Renta writes into. The list shows the buckets of the selected project.

Click Save.

On Save, Renta creates and deletes a marker folder in the bucket with your key. A key that cannot write into the bucket returns an error right away instead of failing on the first pipeline run.

Verify the connectionStep 4

The connection appears in the Destinations list with the Active status and is ready for pipelines. A connection whose bucket has not been chosen yet stays inactive and is not offered in the pipeline wizard.

Replacing the key

Open the connection in Destinations and click Upload a new Service Account JSON file at the top of the settings page. Renta checks the new key against the stored bucket before it replaces the old one, so a key without write access is rejected and the old key stays in place. The project and the bucket do not change.

Create a pipeline into Google Cloud Storage

The steps below use Currency Exchange Rates as the source, and the same flow applies to any other source.

Select the source and its connectionStep 1

In the left sidebar, click New pipeline. Keep the ETL tab selected, pick a source under Already connected, choose the connection to read from, and click Next step.

Select the Google Cloud Storage connectionStep 2

On the Destination step, select Google Cloud Storage under Already connected, then pick the connection. Each card shows the bucket name, the service account email, and the project.

Name the folder and create the pipelineStep 3

Fill in the source settings, then the pipeline settings:

1. Integration name.
   The display name of the pipeline in the Renta console.
2. Table name.
   For a data lake this is the folder name inside the bucket. Keep the generated name or type your own.
3. Date range, Overwrite period, and Schedule work the same way as for any other destination. See pipeline scheduling for the schedule types.

Click Create pipeline. The pipeline appears in the Pipelines section under the ETL tab, together with the folder name, run progress, and the number of extracted rows.

Bucket layout

Everything a pipeline writes lives under its own folder, and the shape inside that folder depends on whether the data carries a date.

The date prefix is what makes the overwrite window cheap: Renta removes the objects of the affected days instead of the whole folder. It is also the layout that BigQuery external tables, Dataproc, and Spark expect, so the partitions are discovered without extra configuration.

Alongside the folder, Renta keeps a zero-byte object named exactly like it. The object marks the folder and holds no data.

Sync strategy

How Renta refreshes the folder depends on the source and on the pipeline settings.

A run that changes the field list is a special case. The folder is deleted and rebuilt from scratch, and the whole period set in Date Range is loaded again.

Troubleshooting

Most problems surface on the connection page, before any pipeline runs.