Backend

Set up Kubernetes on EKS, Docker on ECR, and Bitbucket Pipelines

The deployment flow for the backend is as follows:

1. Push to master/staging for prod/staging environments respectively

2. Pipeline will format, lint, and test the code

3. If passed, another pipeline will build a Docker image

4. Image is pushed to AWS ECR

5. Pipeline then runs deployment command on EKS to update the image and perform a blue/green switchover

ECR

AWS's Elastic Container Repository is a private repository, used to host our Docker images.

Create two new private repositories to hold the backend Docker images, one for the main image and one to hold intermediary build stages.

Create repository interface

Create a main repo and a stages repo

EKS

Setup

1. Delete the default VPC. This VPC exposes all services in a public subnet. Instead we use a separate VPC for each environment, made up of public and private subnets to restrict access.

2. cd into the Backend Directory (see Backend Template Starter)

3. Edit the cluster details in /kube/eksctl/cluster.yaml to change the cluster name, node group config, etc.

4. Run the following command to create a new VPC, Cluster and Node Group

eksctl create cluster -f kube/eksctl/cluster.yaml --profile profileName

This can take up to 40 minutes

Once the cluster has been created, the connection details should be added to your ~/.kube/config file. Ensure kubectl is connected to the correct cluster by running kubectl config current-context.

Switching to the cluster (before teleport)

The cluster should automatically be added to your kubeconfig, so normally you can skip to the next step. If it hasn't, run this to login to the cluster:

$ eksctl utils write-kubeconfig --cluster=<name> --profile=<profile>

Add cluster to Teleport

To connect to and switch between k8 clusters, we use Teleport. To add the new cluster to the CA Teleport cluster:

1. Ensure you have helm installed on your local machine

2. SSH into the teleport-host machine using tsh ssh root@teleport-host

3. Edit the /etc/teleport.yaml file to add a new static token for a kube node, setting the token to a memorable string (such as the cluster name) under the # Add kube nodes here section

4. Reload the teleport system using systemctl restart teleport

5. Run the following command to create a new namespace for teleport on the cluster, install teleport agent, and connect to the teleport cluster

# Add teleport helm repo
$ helm repo add teleport https://charts.releases.teleport.dev

# Set the variables
$ PROXY_ENDPOINT='dev.chelsea-apps.com:443'
$ JOIN_TOKEN='staticToken'
$ KUBERNETES_CLUSTER_NAME='clusterName'

# Install the teleport agent using helm
$ helm install teleport-agent teleport/teleport-kube-agent \
  --create-namespace \
  --namespace teleport-agent \
  --set roles=kube \
  --set proxyAddr=${PROXY_ENDPOINT?} \
  --set authToken=${JOIN_TOKEN?} \
  --set kubeClusterName=${KUBERNETES_CLUSTER_NAME?}

Make sure you have the correct context set in kubectl before running helm install

If everything went well, you should see the following output from this command. If you see could not get token: NoCredentialProviders: no valid providers in chain, see here. If you see The SSO session associated with this profile has expired or is otherwise invalid, see here.

NAME: teleport-agent
LAST DEPLOYED: Wed Apr 28 20:25:26 2021
NAMESPACE: teleport-agent
STATUS: deployed
REVISION: 1
TEST SUITE: None

Teleport

Install Metrics Server

Metrics Server API allows the cluster to automatically scale pods. Install using

$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Switching clusters

Once the k8 cluster has been added as a node to the Teleport cluster, you should be able to view it:

$ tsh kube ls
Kube Cluster Name Selected 
----------------- -------- 
least-staging     *        
uplevyl-staging            

You can log in to a particular cluster by running:

$ tsh kube login clusterName

Cert Manager

First, set the role ARN for the cert-manager Route 53 Role in kube/helm/cert-manager-values.yaml.

Then install cert-manager:

helm upgrade cert-manager jetstack/cert-manager \
  --install \
  --namespace cert-manager \
  --create-namespace \
  --values "helm/cert-manager-values.yaml" \
  --wait

Error: NoCredentialProviders

If you get the following error when running kubectl commands

could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Unable to connect to the server: getting credentials: exec: executable aws-iam-authenticator failed with exit code 1

You'll need to edit your ~/.kube/config file to remove aws-iam-authenticator command and replace it with aws. This is because aws-iam-authenticator does not work well with SSO.

Open the config file in the editor

$ nano ~/.kube/config

Under the users section, find the user for the context/cluster you're trying to connect to:

...
users:
- name: [email protected]
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - least
      command: aws-iam-authenticator
      env:
      - name: AWS_STS_REGIONAL_ENDPOINTS
        value: regional
      - name: AWS_DEFAULT_REGION
        value: eu-west-2
      - name: AWS_PROFILE
        value: Least
...

Replace the command and args with the following config:

...
users:
- name: [email protected]
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - eks
      - get-token
      - --cluster-name
      - least
      command: aws
      env:
      - name: AWS_STS_REGIONAL_ENDPOINTS
        value: regional
      - name: AWS_DEFAULT_REGION
        value: eu-west-2
      - name: AWS_PROFILE
        value: Leas
...

This will allow you to use SSO to authenticate with kubectl.

Error: The SSO session associated with this profile has expired or is otherwise invalid

This error means the SSO login session has expired (default is 4 hours). You'll need to login again using

$ aws sso login --profile profileName

Pods per instance type

https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txtgithub.com

CI/CD

Enable GitHub Actions to lint, test and deploy the backend.

Create AWS Policies

Three policies are needed:

• ECR push image

• EKS deploy image

• Secrets Manager read secrets

Terraform

These can all be created automatically using the deploy Terraform script in scripts/terraform/deploy/.

First initalise Terraform

terraform init

Then, perform a dry run to confirm the changes

terraform plan

Finally, apply the changes using

terraform apply

This will ask for a set of variables:

Name	Description	Example
aws_region	AWS region to use	eu-west-2
aws_profile	AWS config profile to use	Uplevyl
ecr_repository	Name of the ECR repository to allow access to	com.uplevyl.api
cluster_names	A list of names of EKS clusters to allow access to	["uplevyl-prod", "uplevyl-staging"]

You'll need to add this user to the mapUsers section of aws-auth. This can be done using eksctl. Make sure to replace the arn with the actual arn of the user.

$ eksctl create iamidentitymapping --cluster limelight-staging --profile Limelight --arn arn:aws:iam::034576691895:user/DeployServiceAccount --group system:masters --username DeployServiceAccount

Repository Secrets

Set the following variables under the repository settings in GitHub.

Variable	Description	Example
AWS_ACCESS_KEY_ID	DeployServiceAccount user programmatic access key ID to push and deploy images	N/A
AWS_SECRET_ACCESS_KEY	DeployServiceAccount user programmatic secret access key to push and deploy images	N/A

Workflow Variables

The workflows Main and Deploy require the following environment variables be set:

Name	Description	Example
AWS_REGION	AWS region the secrets and clusters are hosted in	eu-west-2
AWS_SECRETS_STAGING	Name of the staging environment secret in AWS Secrets Manager	staging/uplevyl
AWS_SECRETS_PROD	Name of the production environment secret in AWS Secrets Manager	prod/uplevyl
AWS_CLUSTER_STAGING	Name of the EKS staging cluster	uplevyl-staging
AWS_CLUSTER_PROD	Name of the EKS production cluster	uplevyl-prod
ECR_REPOSITORY	Name of the AWS ECR repository storing the deployment images	com.uplevyl.api

RDS

Production Database

For production databases, we use AWS RDS Serverless v2 Postgres databases. Set up a new database cluster, setting the user and database name to the project name (e.g. limelight). Generate a password using 1Password.

Make sure to select the VPC created when you set up the production EKS cluster

Security Group

The DB will need to be created in a custom VPC security group to allow access to it from the API server.

Select PostgreSQL for the Type, and pick Any IP for the source

Give the SG a name and select the production EKS cluster.

Once created assign it to the DB cluster

Select the group you just created

Teleport - On Hold

RDS is used to host the Postgres database required for the Chelsea Apps backend API. Create a new AWS Aurora Postgres compatible database, and follow these steps to add the database to Teleport to allow developers to login to Postgres via Teleport. Make sure to enable "IAM database authentication".

Create a new policy for Teleport

Create a new policy to allow Teleport to connect to RDS. Replace the Account ID and Resource ID with the correct values for the DB:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
             "rds-db:connect"
         ],
         "Resource": [
             "arn:aws:rds-db:<region>:<account-id>:dbuser:<resource-id>/*"
         ]
      }
   ]
}

The Resource ID can be found on RDS > Configuration

Secrets Manager

Secrets manager is used to store the ES512 public and private key pair, used to sign all JWT tokens generated by the backend.

The keys can be generated using:

# ES512
  # private key
$ openssl ecparam -genkey -name secp521r1 -noout -out ecdsa-p521-private.pem
  # public key
$ openssl ec -in ecdsa-p521-private.pem -pubout -out ecdsa-p521-public.pem

External Secrets

External Secrets Operator replaces Kubernetes Secrets.

Create an IAM account with permissions to access Secrets Manager.

Save the account's Access Credentials to a Kubernetes secret:

$ echo -n 'ACCESS_KEY_ID' > ./access-key
$ echo -n 'SECRET_ACCESS_KEY' > ./secret-access-key
$ kubectl create secret generic awssm-secret --from-file=./access-key  --from-file=./secret-access-key

Then install ESO using Helm:

$ helm repo add external-secrets https://charts.external-secrets.io
$ helm install external-secrets \                                    
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

Kubernetes Secrets (Deprecated)

Environment variables required to run applications can be managed through AWS Secrets Manager, then pulled into Kubernetes and set up as Secrets, which can then be passed to applications.

Ref: https://vinta.ws/code/amazon-eks-setup-kubernetes-external-secrets-with-aws-secret-manager.html

Configure Secrets Backends

Get the cluster's OIDC issuer using:

aws eks describe-cluster --profile=*profile* --name *name* --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///"

Copy this for use in the next step.

Create needed AWS resources

Terraform is used to speed create the necessary resources in AWS. Make sure you have it installed.

cd into the scripts/terraform folder and run

terraform init

This will initalise Terraform in that directory. Then, run the following command:

terraform apply

This will ask for a set of inputs (see below) and will then create a policy, role and attach the two. It will output the role's arn which is needed for the next step.

Inputs:

Name	Description	Example
aws_profile	Name of AWS config profile to use	Uplevyl
aws_region	AWS region to use	eu-west-2
cluster_name	Name of the EKS cluster	uplevyl-prod
oidc_issuer	Cluster OIDC provider. See above or the input description for details	oidc.eks.eu-west-2.amazonaws.com/id/B8CA09CC507EFD3735BA9699D63511F2

Deploy kubernetes-external-secrets Controller

helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/

helm install external-secrets \
external-secrets/kubernetes-external-secrets \
--set env.AWS_REGION=us-east-1 \
--set securityContext.fsGroup=65534 \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"='*YOUR_ROLE_ARN*'

It will automatically create a service account named external-secrets-kubernetes-external-secrets in Kubernetes.

Deploy ExternalSecret

ExternalSecret app-secrets will generate a Secret object with the same name, and the content would look like:

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: uplevyl-environment
spec:
  backendType: secretsManager
  region: us-east-1
  dataFrom:
    - YOUR_SECRET

In the app, add the ExternalSecret under envFrom:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: uplevyl-api
spec:
    replicas: 1
    selector:
        matchLabels:
            app: uplevyl-api
    template:
        metadata:
            labels:
                app: uplevyl-api
        spec:
            terminationGracePeriodSeconds: 60
            containers:
                - name: app
                  image: 031583508418.dkr.ecr.us-east-1.amazonaws.com/com.uplevyl.api:staging-latest
                  imagePullPolicy: Always
                  ports:
                      - containerPort: 4000
                  envFrom:
                      - secretRef:
                            name: uplevyl-environment
                  env:
                      - name: PORT
                        value: '4000'

This will automatically inject the secrets from AWS Secrets Manager, into the Kubernetes ExternalSecret, then into the application itself.

Datadog (Monitoring)

Add the repo to helm

$ helm repo add datadog https://helm.datadoghq.com

Get the Datadog API and App keys from your settings and set into the following environment variables

Key	Value
DD_API_KEY	API key
DD_APP_KEY	App Key

Then install the agent

helm install datadogagent \
--create-namespace \
--namespace datadog-agent \
--set datadog.apiKey=$DD_API_KEY \
--set datadog.appKey=$DD_APP_KEY \
-f kube/datadog/values.yaml datadog/datadog

To change values, update kube/datadog/values.yaml and use the helm upgrade command instead of helm install