Deploy multi-region application with Anthos and Google cloud Spanner

5 min readFeb 21, 2023

Introduction

Some applications have multi-regional coverage and need to use a central database with high data consistency. To address this issue, we will deploy the well-known “onlineboutique” application over two regions and use Google Cloud Spanner multi-region as a database.
We’ll demonstrate the application via Anthos and the implementation of multiclusterservices and multiclusteringress.

Architecture

The above architecture is based on certain Anthos functionalities and describes the operation of the multicluster loadbalancer (ingress).

First, a few explanations of the MCI (Multiclusteringress) and MCS (Multiclusterservice) types.

MCS (Multiclusterservice)

when you want to create a service mesh between several GKE clusters, you can use MCS to expose certain shared services such as vault, or you can have very high availability for your various services and expose them to the other GKE clusters. If the service in question were to go down locally, the traffic would automatically be routed to the other cluster without interruption.

MCI ( Multiclusteringress)

it’s a global ingress ( global http/s load balancer ) that lets you configure a single loadbalancer to expose your applications across different regions and clusters, and route traffic according to geolocation or other strategies to improve the availability of your applications.
The network logic used is called “cold potato”, which means that thanks to the use of an anycast IP and the premium network, the client flow will pass through the PoP closest to the user and continue via the GCP backbone for best performance.

Google Cloud spanner

Google cloud Spanner is a highly distributed, scalable, multi-region and highly consistent relational database supporting SQL and posgresql.

Implementation

We’re going to deploy 2 GKE Autopilot clusters in 2 regions, and then deploy the application there.

gcloud container clusters create-auto gke-london \
    --region=europe-west2 \
    --release-channel=stable \
    --project=vincent-ledan \
    --cluster-secondary-range-name=pods \
    --services-secondary-range-name=services \
    --network=medium-test \
    --subnetwork=subnet-london

gcloud container clusters create-auto gke-belgique \
    --region=europe-west1 \
    --release-channel=stable \
    --project=vincent-ledan \
    --cluster-secondary-range-name=pods \
    --services-secondary-range-name=services \
    --network=medium-test \
    --subnetwork=subnet-belgique

Let’s register the two clusters in our Anthos fleet


gcloud container fleet memberships register gke-london \
    --gke-cluster europe-west2/gke-london \
    --enable-workload-identity \
    --project=vincent-ledan

gcloud container fleet memberships register gke-belgique \
    --gke-cluster europe-west1/gke-belgique \
    --enable-workload-identity \
    --project=vincent-ledan

In the MCI/MCS architecture, we need to assign a GKE cluster as a configuration cluster, which will take care of creating load balancers and managing MCSs. This cluster can also have workloads.

gcloud container fleet ingress enable \
    --config-membership=gke-london \
    --project= vincent-ledan

v_ledan@cloudshell:~ (vincent-ledan-378413)$ gcloud container fleet ingress  describe
createTime: '2023-02-21T10:00:03.204340973Z'
membershipStates:
  projects/240178848426/locations/global/memberships/gke-belgique:
    state:
      code: OK
      updateTime: '2023-02-21T10:43:45.829411927Z'
  projects/240178848426/locations/global/memberships/gke-london:
    state:
      code: OK
      updateTime: '2023-02-21T10:43:45.829410146Z'
name: projects/vincent-ledan-378413/locations/global/features/multiclusteringress
resourceState:
  state: ACTIVE
spec:
  multiclusteringress:
    configMembership: projects/vincent-ledan-378413/locations/global/memberships/gke-london
state:
  state:
    code: OK
    description: Ready to use
    updateTime: '2023-02-21T10:04:43.852760698Z'
updateTime: '2023-02-21T10:43:46.508737172Z'

Now let’s take a look at our Cloud Spanner instance.

For the demo, we’ll create an instance with 2 read/write replicas in the same regions as our GKE clusters. The “eur5” configuration will deploy 2 read/write replicas in europe-west1 and europe-west2.

gcloud spanner instances create \
instance-mondiale --config=eur5 --nodes=1 \
--processing-units=100

The onlineboutique application will use the “carts” database, to create it here’s the command with the diagram.

gcloud spanner databases create carts \
 --instance instance-mondiale \
 --database-dialect GOOGLE_STANDARD_SQL \
 --ddl "CREATE TABLE CartItems (userId STRING(1024), productId STRING(1024), quantity INT64) PRIMARY KEY (userId, productId); CREATE INDEX CartItemsByUserId ON CartItems(userId);"

Now let’s configure the workload identity and cloud spanner.

the kubernetes service account to be used is the “cartservice” service account


gcloud iam service-accounts create spanner-db-user-sa \
    --display-name=spanner-db-user-sa

gcloud spanner databases add-iam-policy-binding carts \
    --instance instance-mondiale \
    --member "serviceAccount:spanner-db-user-sa@vincent-ledan.iam.gserviceaccount.com" \
    --role roles/spanner.databaseUser
gcloud iam service-accounts add-iam-policy-binding spanner-db-user-sa@vincent-ledan.iam.gserviceaccount.com \
    --member "serviceAccount:vincent-ledan.svc.id.goog[onlineboutique/cartservice]" \
    --role roles/iam.workloadIdentityUser

Let’s now deploy the onlineboutique application in both clusters. The following options are used by the Helm chart to use cloud spanner, disable the external front end and process it with the MCI.

helm upgrade onlineboutique oci://us-docker.pkg.dev/online-boutique-ci/charts/onlineboutique \
    --install \
    --create-namespace \
    -n onlineboutique \
    --set cartDatabase.inClusterRedis.create=false \
    --set cartDatabase.type=spanner \
    --set frontend.externalService=false \
    --set cartDatabase.connectionString=projects/vincent-ledan/instances/instance-mondiale/databases/carts \
    --set serviceAccounts.create=true \
    --set serviceAccounts.annotationsOnlyForCartservice=true \
    --set "serviceAccounts.annotations.iam\.gke\.io/gcp-service-account=spanner-db-user-sa@vincent-ledan.iam.gserviceaccount.com"

Now we’re going to configure the MCI/MCS part.

Let’s create and apply the mc.yaml file in the gke-london configuration cluster

apiVersion: networking.gke.io/v1
kind: MultiClusterService
metadata:
  name: boutique-mcs
  namespace: onlineboutique
spec:
  template:
    spec:
      selector:
        app: frontend
      ports:
      - name: web
        protocol: TCP
        port: 8080
        targetPort: 8080
---

apiVersion: networking.gke.io/v1
kind: MultiClusterIngress
metadata:
  name: mci-ingress
  namespace: onlineboutique
spec:
  template:
    spec:
      backend:
        serviceName: boutique-mcs
        servicePort: 8080

The MultiClusterService points to the kubernetes frontend service, which is of type ClusterIP, and exposes this service to other GKE clusters.

MultiClusterIngress references the MCS and takes care of creating the HTTP/S global loadbalancer with 4 NEGs as backend.

Press enter or click to view image in full size

Accessing the application via the loadbalancer IP

You can also check in Google cloud spanner that the DB is correctly populated.

Conclusion

This is just one of many ways to expose your applications.

This architecture is very simple, and it is also possible to integrate the following features:
- Identity aware proxy
- managed certificates
- cloud armor
- custom config for routing between clusters

Services multicluster | Google Kubernetes Engine (GKE) | Google Cloud

Cette page explique comment fonctionnent les services multiclusters (MCS) dans GKE.

cloud.google.com

Multi Cluster Ingress | Google Kubernetes Engine (GKE) | Google Cloud

Cette page décrit le fonctionnement de l'objet Multi Cluster Ingress avec GKE.