Cloud Composer

Cloud Composer Setup

The following steps are pre-requisites for creating a connection between Revefi and Google Cloud Composer.

Create a service account for Revefi to access Composer and Airflow APIs

Revefi recommends using an unique service account (revefi-composer-svc) in a project that was created for Revefi app (<REVEFI_APP_PROJECT>). Example service account principal revefi-composer-svc@<REVEFI_APP_PROJECT>.iam.gserviceaccount.com

Follow the steps here to create the service account.

Create a custom role - RevefiComposerMonitoringReadOnlyRole

Create a role named RevefiComposerMonitoringReadOnlyRole and add the following permissions to the role

resourcemanager.projects.get
monitoring.timeSeries.list
logging.logEntries.list

Use the following command to create the role. Replace the <ORGANIZATION_ID> with your organization id.

gcloud iam roles create RevefiCloudMonitoringReadOnlyRole --organization=<ORGANIZATION_ID> --title=RevefiComposerMonitoringReadOnlyRole --description="Revefi role for composer monitoring" --permissions=resourcemanager.projects.get,monitoring.timeSeries.list,logging.logEntries.list

Bind role(s) to the service account

Bind the following roles to the service account you have created in the previous step

roles/composer.user
RevefiCloudMonitoringReadOnlyRole

Command to bind the role

gcloud projects add-iam-policy-binding <REVEFI_APP_PROJECT> --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" --role="organizations/<ORGANIZATION_ID>/roles/RevefiComposerMonitoringReadOnlyRole"
gcloud projects add-iam-policy-binding <REVEFI_APP_PROJECT> --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" --role="roles/composer.user"

Grant access to Airflow APIs

• Perform the steps listed under the following link to grant access to Airflow APIs
  • Specify the Viewer role for the service account
    • Access the Airflow REST API

Revefi Setup

After the service account is created and the roles has been assigned, you can set a connection in Revefi.

1. Within the Revefi app, select Connections tab on the left menu
2. Enter the key JSON that was generated during the service account creation
3. Specify the GCP Regions where your composer environments are located

Click on "Create connection" button to complete the connection setup.