Artifact Registry: Set Up for CI/CD
When using the Artifact Registry, you will need to authenticate with the registry. This is done by creating a service account and downloading a JSON key file. This key file is then used to authenticate with the registry.
When using Bitbucket Pipelines, the service account is automatically made available to the pipeline
as an environment variable (JAX_CS_REGISTRY_BITBUCKET). This environment variable is then used to
authenticate with the registry.
Docker
You will need to configure docker in your CI/CD environment so that it's able to authenticate with
Google's Artifact Registry APIs. To do this, you will need to create a file from the environment
variable that contains the service account key file and set the GOOGLE_APPLICATION_CREDENTIALS
to the path of that file.
Authenticating to Jax Artifact Registry
script:
# Make the service account key available
- echo "$JAX_CS_REGISTRY_BITBUCKET" | base64 -d >> /tmp/cred.json
# Set GOOGLE_APPLICATION_CREDENTIALS
- export GOOGLE_APPLICATION_CREDENTIALS=/tmp/cred.json
# Activate the service account
- >
gcloud auth activate-service-account
bitbucket-integration@jax-cs-gedi-build-nc-01.iam.gserviceaccount.com
--key-file=/tmp/cred.json
# Authneticate docker to the relevant registry
# For testing and developement registries
- gcloud auth configure-docker us-east1-docker.pkg.dev
# For production
# - gcloud auth configure-docker us-docker.pkg.dev
Python
Python packages that are hosted in the Artifact Registry also need authentication in order to be
installed. We will outline this process for two dependency managers that you can use to install
packages from the Artifact Registry: pip and poetry.
Poetry
Both poetry and docker are able to use the same credentials to authenticate to the registry. However, we still need to provide these credentials to the poetry install step, wherever it is that we're running the package installation step.
Poetry and Bitbucket Pipelines
Installing the application (and dependencies) in a bitbucket pipelines step can be as simple as
using a Jax CS Base Image and providing the credentials as a file through setting the
GOOGLE_APPLICATION_CREDENTIALS environment variable.
definitions:
steps:
- step: &unittests
name: Run Unittests
image:
name: us-east1-docker.pkg.dev/jax-cs-registry/docker-test/base/python3.9-slim-poetry:latest
username: _json_key_base64
password: '$JAX_CS_REGISTRY_BITBUCKET'
script:
- echo "$JAX_CS_REGISTRY_BITBUCKET" | base64 -d >> /tmp/key-file.json
- export GOOGLE_APPLICATION_CREDENTIALS=/tmp/key-file.json
- poetry install --no-interaction --no-ansi
- poetry run pytest --cov-report xml:coverage-reports/coverage-pytest.xml --cov=metabolomics tests/
- pipe: sonarsource/sonarcloud-scan:1.4.0
- pipe: sonarsource/sonarcloud-quality-gate:0.1.6
Note
The above CICD definition uses the python3.9-slim-poetry base image, available in the Jax
Docker Artifact Registry, which has poetry installed and configured to be able to do this.
If you need to install and configure on a different python image you may need to install and configure poetry in the following ways:
Poetry and Docker in Bitbucket Pipelines
Since both poetry and docker are able to use the same credentials to authenticate to the registry, we just need to provide these credentials to the poetry install step inside the docker image build.
To do this we use a mount with type secret, which can use a file from the build environment in the build step, but which does not bake that file into the resulting image.
In the following Dockerfile we mount the GOOGLE_APPLICATION_CREDENTIALS file as a secret so that
poetry can authenticate with the private PyPi on Artifact Registry.
FROM us-docker.pkg.dev/jax-cs-registry/docker/base/python3.9-slim-poetry:latest
WORKDIR /app
COPY pyproject.toml poetry.lock ./
WORKDIR /app
# --mount provides the credentials we need without baking them into the image
RUN --mount=type=secret,id=google_creds,target=/app/google_creds.json \
export GOOGLE_APPLICATION_CREDENTIALS="/app/google_creds.json" && \
# We don't need a virtual environment in a container
poetry config virtualenvs.create false --no-interaction --no-ansi && \
# Install production dependencies only, do not install the application yet
poetry install --no-dev --no-interaction --no-ansi --no-root
# Copy your source code here
COPY src/ ./src
# Install applicaton package now, this will help dependency caching when building locally.
RUN poetry install --no-dev --no-interaction --no-ansi
USER 1001
ENTRYPOINT ["poetry", "run"]
CMD ["my_application"]
The docker build step for that dockerfile would then look something like this:
The above Dockerfile uses the python3.9-slim-poetry base image, available in the Jax Docker
Artifact Registry, which has poetry installed and configured to be able to do this.
If you want to install and configure poetry yourself, your RUN command would look something like:
# Upgrade Pip
RUN python3 -m pip install --upgrade pip && \
# Download and Install Poetry
curl -sSL https://install.python-poetry.org | python3 - && \
# Add the google artifact registry keyring to poetry
/root/.local/bin/poetry self add "keyrings.google-artifactregistry-auth==1.1.1"
You might also want to put poetry directly on the path:
Poetry and Skaffold
Skaffold is a tool for building and deploying applications to Kubernetes. It can be used to build docker images and deploy them to a Kubernetes cluster. In order to use skaffold with a docker image that depends on a package stored in a private registry, we first need to set up our dockerfile as described above. Then we need to provide the credentials to skaffold so that knows how to provide them to the docker daemon.
apiVersion: skaffold/v2beta29
kind: Config
metadata:
name: metabolomics.cloud
build:
local:
useBuildkit: true
push: false
artifacts:
- image: metabolomics-cloud-app
docker:
dockerfile: Dockerfile
secrets:
- id: google_creds
env: GOOGLE_APPLICATION_CREDENTIALS
deploy:
kubectl:
manifests:
OR
apiVersion: skaffold/v2beta29
kind: Config
metadata:
name: metabolomics.cloud
build:
local:
useBuildkit: true
push: false
artifacts:
- image: metabolomics-cloud-app
docker:
dockerfile: Dockerfile
secrets:
- id: google_creds
src: <path-to-google-creds-json>
deploy:
kubectl:
manifests:
The above skaffold configuration uses the local builder, which builds the docker image locally,
either on your local machine or in a Bitbucket Pipelines step. The secrets section tells skaffold
to mount the GOOGLE_APPLICATION_CREDENTIALS or google credential json file as a secret, which will be available to the
docker daemon when building the image.
For example, the following RUN command uses the google_creds secret to initialize credentials
for the google artifact registry keyring:
RUN --mount=type=secret,id=google_creds,target=/app/google_creds.json \
export GOOGLE_APPLICATION_CREDENTIALS="/app/google_creds.json"
The skaffold configuration above will allow you to use your normal skaffold commands to build and deploy your application. For example, the following command will build and tag the docker image for you.
skaffold build
````
### Pip
It is recommended that you use `poetry` for dependency management as described above,
however, it is also possible to set up and use `pip` without `poetry`.
!!! NOTE: You will still need to mount the credentials as a secret in your docker
build:
```dockerfile
RUN --mount=type=secret,id=google_creds,target=/app/google_creds.json \
export GOOGLE_APPLICATION_CREDENTIALS="/app/google_creds.json" && \
echo "run commands that need acces to GOOGLE_APPLICATION_CREDENTIALS here"
```
There are two methods of setting up authentication for pip:
- [Keyring :octicons-link-external-16:](https://cloud.google.com/artifact-registry/docs/python/authentication){ .md-button target=_blank }
- [Service Account :octicons-link-external-16:](https://cloud.google.com/artifact-registry/docs/python/authentication#sa-key){ .md-button target=_blank }
Once you've set up authentication, you can use pip commands pointed at the Jax CS PyPi registry:
NPM
Attention
Seeking content contributors for this section.
Java Maven
Attention
Seeking content contributors for this section.