BigQuery - Revefi Roles

Grant Service Account Access to Project(s) - version

If you have one (or few) BigQuery projects, Revefi recommends to use the simplied version listed here

For fine control roles setup, please follow these steps.

All three roles are required for advanced roles setup.

Role: RevefiBigQueryJobExecutionRole

Revefi requires a context BigQuery project to connect to your instance of BigQuery. The context project is used as the project for billing (cost attribution to Revefi usage).

Create a role named RevefiBigQueryJobExecutionRole and add the following permissions

bigquery.jobs.create
bigquery.jobs.listAll
bigquery.readsessions.create
bigquery.readsessions.getData

Use the following command to create the role. Replace the <ORGANIZATION_ID> with the organization id.

gcloud iam roles create RevefiBigQueryJobExecutionRole --organization=<ORGANIZATION_ID> --title=RevefiBigQueryJobExecutionRole --description="Revefi role for context project" --permissions=bigquery.jobs.create,bigquery.jobs.listAll,bigquery.readsessions.create,bigquery.readsessions.getData

Role: RevefiBigQueryProjectMetadataRole

Create a role named RevefiBigQueryProjectMetadataRole for all the projects you want Revefi to monitor. Add the following permissions to the role

bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.routines.get
bigquery.routines.list
bigquery.tables.get
bigquery.tables.getIamPolicy
bigquery.tables.list
bigquery.jobs.listAll
datalineage.events.get
datalineage.locations.searchLinks
resourcemanager.projects.get

Use the following command to create the role. Replace the <ORGANIZATION_ID> with the organization id.

gcloud iam roles create RevefiBigQueryProjectMetadataRole  --organization=<ORGANIZATION_ID> --title=RevefiBigQueryProjectMetadataRole --description="Revefi role for project metadata access" --permissions=bigquery.datasets.get,bigquery.datasets.getIamPolicy,bigquery.routines.get,bigquery.routines.list,bigquery.tables.get,bigquery.tables.getIamPolicy,bigquery.tables.list,bigquery.jobs.listAll,datalineage.events.get,datalineage.locations.searchLinks,resourcemanager.projects.get

Role: RevefiBigQueryReservationMetadataRole

Create a role named RevefiBigQueryReservationMetadataRole for the project that manages the reservations. Add the following permissions to the role

bigquery.capacityCommitments.list
bigquery.reservationAssignments.list
bigquery.reservations.list

Use the following command to create the role. Replace the <ORGANIZATION_ID> with the organization id.

gcloud iam roles create RevefiBigQueryReservationMetadataRole --organization=<ORGANIZATION_ID> --title=RevefiBigQueryReservationMetadataRole --description="Revefi role for reservation metadata" --permissions=bigquery.capacityCommitments.list,bigquery.reservationAssignments.list,bigquery.reservations.list

Role: RevefiOrgRole

Create a role named RevefiOrgRole at the organization level for cross-project analysis and reservation management. This role enables Revefi to view jobs across the organization and access table storage metadata.

Add the following permissions to the role:

bigquery.jobs.listAll
bigquery.tables.get
bigquery.tables.list

Use the following command to create the role. Replace the <ORGANIZATION_ID> with the organization id.

gcloud iam roles create RevefiOrgRole --organization=<ORGANIZATION_ID> --title="RevefiOrgRole" --description="Custom role for Revefi to list all jobs within an organization and access table storage metadata" --permissions=bigquery.jobs.listAll,bigquery.tables.get,bigquery.tables.list

Note: If you already have a RevefiOrgRole with only the bigquery.jobs.listAll permission, you can update it to include the table storage permissions:

gcloud iam roles update RevefiOrgRole --organization=<ORGANIZATION_ID> --add-permissions=bigquery.tables.get,bigquery.tables.list

Bind Roles to the service account

Bind the roles defined above to your service account. Replace the <ORGANIZATION_ID>, <CONTEXT_PROJECT_ID>, <PROJECT_ID>, <RESERVATION_PROJECT_ID>, <SERVICE_ACCOUNT_EMAIL> placeholders respectively.

Note: Use the service account email created for Revefi.

gcloud projects add-iam-policy-binding <CONTEXT_PROJECT_ID> --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" --role="organizations/<ORGANIZATION_ID>/roles/RevefiBigQueryJobExecutionRole"

Note: Run this command for ALL project ids you want Revefi to monitor.

gcloud projects add-iam-policy-binding <PROJECT_ID> --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" --role="organizations/<ORGANIZATION_ID>/roles/RevefiBigQueryProjectMetadataRole"

Run this command for the reservation project

gcloud projects add-iam-policy-binding <RESERVATION_PROJECT_ID> --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" --role="organizations/<ORGANIZATION_ID>/roles/RevefiBigQueryReservationMetadataRole"

Assign the RevefiOrgRole at the organization level:

This role needs to be assigned at the organization level (not project level) to enable cross-project analysis.

1. Navigate to the Google Cloud Console IAM page
2. Select your organization from the organization drop-down menu at the top of the page
3. Click on "Grant Access"
4. In the "New principals" field, enter the service account email: <SERVICE_ACCOUNT_EMAIL>
5. Assign the "RevefiOrgRole" role
6. Save changes