Skip to main content
Version: devel

Google Cloud Composer setup

CI/CD setup​

This setup will allow you to deploy the main branch of your Airflow project from GitHub to Cloud Composer.

  • Create a GitHub repository, for example, by following our how-to guide on deployment for Airflow.

  • In the Google Cloud web interface, go to Source Repositories and create a repository that mirrors your GitHub repository. This will simplify authentication by using this mirroring service.

  • In Cloud Build, add a trigger on commit to the main branch.

  • Point it to your Cloud Build file. In our example, we place our file at build/cloudbuild.yaml.

    trigger-config

  • Go to Cloud Composer, click on the dags folder, and get the bucket name.

    test-composer

  • In your cloudbuild.yaml, set the bucket name.

  • Make sure your repository code is pushed to the main branch.

  • Run the trigger you built (in Cloud Build).

  • Wait a minute, and check if your files have arrived in the bucket. In our case, we added a pipedrive folder, and we can see it appeared.

    bucket-details

Airflow setup​

Adding the libraries needed​

Assuming you have already spun up a Cloud Composer:

  • Make sure the user you added has rights to change the base image (add libraries). I already had these added; you may get away with fewer (not clear in docs):

    • Artifact Registry Administrator;
    • Artifact Registry Repository Administrator;
    • Remote Build Execution Artifact Admin;

  • Navigate to your composer environment and add the needed libraries. In the case of this example pipedrive pipeline, we only need the sdf library, so add the dlt library.

    add-package

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.