219b29f1a6
* add documentation for gcp provider (capg)
* add manifests for gcp provider (capg)
* add cluster templates for control plane and workers
* add site definition to use gcp provider (capg) with
control plane and workers
Zuul Gate Scripts: https://review.opendev.org/#/c/749165/
Change-Id: I3ea07fcafe49fc2698d9d961222550c45b40cd66
787 lines
33 KiB
Markdown
Executable File
787 lines
33 KiB
Markdown
Executable File
# Airshipctl and Cluster API GCP Integration
|
|
|
|
## Overview
|
|
Airshipctl and cluster api gcp integration facilitates usage of `airshipctl` to
|
|
create cluster api management and workload clusters using `gcp as infrastructure
|
|
provider`.
|
|
|
|
|
|
|
|
## Workflow
|
|
A simple workflow that can be tested, involves the following
|
|
operations:
|
|
|
|
**Initialize the management cluster with cluster api and gcp provider
|
|
components**
|
|
|
|
> airshipctl phase run clusterctl-init-ephemeral
|
|
|
|
**Create a workload cluster, with control plane and worker nodes**
|
|
|
|
> airshipctl phase run controlplane-target
|
|
|
|
> airshipctl phase run workers-target
|
|
|
|
## GCP Prerequisites
|
|
|
|
### Create Service Account
|
|
To create and manager clusters, this infrastructure providers uses a service
|
|
account to authenticate with GCP's APIs. From your cloud console, follow [these
|
|
instructions](https://cloud.google.com/iam/docs/creating-managing-service-accounts#creating)
|
|
to create a new service account with Editor permissions. Afterwards, generate a
|
|
JSON Key and store it somewhere safe. Use cloud shell to install ansible,
|
|
packer, and build the CAPI compliant vm image.
|
|
|
|
### Build Cluster API Compliant VM Image
|
|
|
|
#### Install Ansible
|
|
Start by launching cloud shell.
|
|
|
|
$ export GCP_PROJECT_ID=<project-id>
|
|
|
|
$ export GOOGLE_APPLICATION_CREDENTIALS=</path/to/serviceaccount-key.json>
|
|
|
|
$ sudo apt-get update
|
|
|
|
$ sudo apt-get install ansible -y
|
|
|
|
#### Install Packer
|
|
|
|
$ mkdir packer
|
|
|
|
$ cd packer
|
|
|
|
$ wget https://releases.hashicorp.com/packer/1.6.0/packer_1.6.0_linux_amd64.zip
|
|
|
|
$ unzip packer_1.6.0_linux_amd64.zip
|
|
|
|
$ sudo mv packer /usr/local/bin/
|
|
|
|
#### Build GCP Compliant CAPI-Ubuntu Image
|
|
|
|
$ git clone https://sigs.k8s.io/image-builder.git
|
|
|
|
$ cd image-builder/images/capi/
|
|
|
|
$ make build-gce-default
|
|
|
|
$ gcloud compute images list --project ${GCP_PROJECT_ID} --no-standard-images
|
|
|
|
```
|
|
NAME PROJECT FAMILY DEPRECATED STATUS
|
|
cluster-api-ubuntu-1804-v1-16-14-1599066516 virtual-anchor-281401 capi-ubuntu-1804-k8s-v1-16 READY
|
|
```
|
|
|
|
### Create Cloud NAT Router
|
|
|
|
Kubernetes nodes, to communicate with the control plane, pull container images
|
|
from registried (e.g. gcr.io or dockerhub) need to have NAT access or a public
|
|
ip. By default, the provider creates Machines without a public IP.
|
|
|
|
To make sure your cluster can communicate with the outside world, and the load
|
|
balancer, you can create a Cloud NAT in the region you'd like your Kubernetes
|
|
cluster to live in by following [these
|
|
instructions](https://cloud.google.com/nat/docs/using-nat#specify_ip_addresses_for_nat).
|
|
|
|
For reference, use the below images. You can create 2 cloud NAT routers for
|
|
region us-west1 and us-east1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Other Common Pre-requisites
|
|
These prerequistes are required on the VM that will be used to create workload
|
|
cluster on gcp
|
|
|
|
* Install [Docker](https://www.docker.com/)
|
|
* Install [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
|
|
* Install [Kind](https://kind.sigs.k8s.io/)
|
|
* Install
|
|
[Kustomize](https://kubernetes-sigs.github.io/kustomize/installation/binaries/)
|
|
* Install [Airshipctl](https://docs.airshipit.org/airshipctl/developers.html)
|
|
|
|
Also, check [Software Version Information](#Software-Version-Information),
|
|
[Special Instructions](#Special-Instructions) and [Virtual Machine
|
|
Specification](#Virtual-Machine-Specification)
|
|
|
|
## Getting Started
|
|
|
|
Kind will be used to setup a kubernetes cluster, that will be later transformed
|
|
into a management cluster using airshipctl. The kind kubernetes cluster will be
|
|
initialized with cluster API and Cluster API gcp provider components.
|
|
|
|
$ export KIND_EXPERIMENTAL_DOCKER_NETWORK=bridge
|
|
|
|
$ kind create cluster --name capi-gcp
|
|
```
|
|
Creating cluster "capi-gcp" ...
|
|
WARNING: Overriding docker network due to KIND_EXPERIMENTAL_DOCKER_NETWORK
|
|
WARNING: Here be dragons! This is not supported currently.
|
|
✓ Ensuring node image (kindest/node:v1.18.2) 🖼
|
|
✓ Preparing nodes 📦
|
|
✓ Writing configuration 📜
|
|
✓ Starting control-plane 🕹️
|
|
✓ Installing CNI 🔌
|
|
✓ Installing StorageClass 💾
|
|
Set kubectl context to "kind-capi-gcp"
|
|
You can now use your cluster with:
|
|
|
|
kubectl cluster-info --context kind-capi-gcp
|
|
```
|
|
|
|
$ kubectl get pods -A
|
|
|
|
```
|
|
NAMESPACE NAME READY STATUS RESTARTS AGE
|
|
kube-system coredns-66bff467f8-kmg7c 1/1 Running 0 82s
|
|
kube-system coredns-66bff467f8-lg8qc 1/1 Running 0 82s
|
|
kube-system etcd-capi-gcp-control-plane 1/1 Running 0 91s
|
|
kube-system kindnet-dzp8v 1/1 Running 0 82s
|
|
kube-system kube-apiserver-capi-gcp-control-plane 1/1 Running 0 91s
|
|
kube-system kube-controller-manager-capi-gcp-control-plane 1/1 Running 0 90s
|
|
kube-system kube-proxy-zvdh8 1/1 Running 0 82s
|
|
kube-system kube-scheduler-capi-gcp-control-plane 1/1 Running 0 83s
|
|
local-path-storage local-path-provisioner-bd4bb6b75-6drnt 1/1 Running 0 82s
|
|
```
|
|
|
|
## Create airshipctl configuration files
|
|
|
|
$ mkdir ~/.airship
|
|
|
|
$ airshipctl config init
|
|
|
|
Run the below command to configure gcp manifest, and add it to airship config
|
|
|
|
```
|
|
$ airshipctl config set-manifest gcp_manifest --repo primary \
|
|
--url https://opendev.org/airship/airshipctl --branch master \
|
|
--primary --sub-path manifests/site/gcp-test-site --target-path /tmp/airship
|
|
```
|
|
|
|
$ airshipctl config set-context kind-capi-gcp --manifest gcp_manifest
|
|
|
|
```
|
|
Context "kind-capi-gcp" modified.
|
|
```
|
|
$ cp ~/.kube/config ~/.airship/kubeconfig
|
|
|
|
$ airshipctl config get-context
|
|
|
|
```
|
|
Context: kind-capi-gcp
|
|
contextKubeconf: kind-capi-gcp_target
|
|
manifest: gcp_manifest
|
|
|
|
LocationOfOrigin: /home/rishabh/.airship/kubeconfig
|
|
cluster: kind-capi-gcp_target
|
|
user: kind-capi-gcp
|
|
```
|
|
$ airshipctl config use-context kind-capi-gcp
|
|
|
|
```
|
|
Manifest "gcp_manifest" created.
|
|
```
|
|
|
|
$ airshipctl document pull --debug
|
|
|
|
```
|
|
[airshipctl] 2020/08/12 14:07:13 Reading current context manifest information from /home/rishabh/.airship/config
|
|
[airshipctl] 2020/08/12 14:07:13 Downloading primary repository airshipctl from https://review.opendev.org/airship/airshipctl into /tmp/airship
|
|
[airshipctl] 2020/08/12 14:07:13 Attempting to download the repository airshipctl
|
|
[airshipctl] 2020/08/12 14:07:13 Attempting to clone the repository airshipctl from https://review.opendev.org/airship/airshipctl
|
|
[airshipctl] 2020/08/12 14:07:23 Attempting to checkout the repository airshipctl from branch refs/heads/master
|
|
```
|
|
$ airshipctl config set-manifest gcp_manifest --target-path /tmp/airship/airshipctl
|
|
|
|
## Configure gcp site variables
|
|
|
|
`configure project_id`
|
|
|
|
$ cat /tmp/airship/airshipctl/manifests/site/gcp-test-site/target/controlplane/project_name.json
|
|
|
|
```
|
|
[
|
|
{ "op": "replace","path": "/spec/project","value": "<project_id>"}
|
|
]
|
|
```
|
|
|
|
Include gcp variables in clusterctl.yaml
|
|
|
|
The original values for the below variables are as follows:
|
|
```
|
|
GCP_CONTROL_PLANE_MACHINE_TYPE="n1-standard-4"
|
|
GCP_NODE_MACHINE_TYPE="n1-standard-4"
|
|
GCP_REGION="us-west1"
|
|
GCP_NETWORK_NAME="default"
|
|
|
|
GCP_PROJECT="<your_project_id>"
|
|
GCP_CREDENTIALS="$( cat ~/</path/to/serviceaccount-key.json>)"
|
|
```
|
|
|
|
Edit `airshipctl/manifests/site/gcp-test-site/shared/clusterctl/clusterctl.yaml`
|
|
to include gcp variables and their values in base64 encoded format. Use
|
|
https://www.base64decode.org/ if required.
|
|
|
|
To get the GCP_CREDENTIALS in base64 format, use the below command.
|
|
|
|
$ export GCP_B64ENCODED_CREDENTIALS=$( cat ~/</path/to/serviceaccount-key.json> | base64 | tr -d '\n' )
|
|
|
|
$ echo $GCP_B64ENCODED_CREDENTIALS
|
|
|
|
The below shown `clusterctl.yaml`, has encoded the values for all variables except
|
|
GCP_PROJECT and GCP_CREDENTIALS. You can use the base64 encoded values for
|
|
GCP_PROJECT and GCP_CREDENTIALS based on your project.
|
|
|
|
The other remaining variables in the `clusterctl.yaml` are base64 encoded.
|
|
Their original values is as follows:
|
|
|
|
```
|
|
GCP_CONTROL_PLANE_MACHINE_TYPE="n1-standard-4"
|
|
GCP_NODE_MACHINE_TYPE="n1-standard-4"
|
|
GCP_REGION="us-west1"
|
|
GCP_NETWORK_NAME="default"
|
|
```
|
|
|
|
$ cat /tmp/airship/airshipctl/manifests/site/gcp-test-site/shared/clusterctl/clusterctl.yaml
|
|
|
|
```
|
|
apiVersion: airshipit.org/v1alpha1
|
|
kind: Clusterctl
|
|
metadata:
|
|
labels:
|
|
airshipit.org/deploy-k8s: "false"
|
|
name: clusterctl-v1
|
|
init-options:
|
|
core-provider: "cluster-api:v0.3.3"
|
|
bootstrap-providers:
|
|
- "kubeadm:v0.3.3"
|
|
infrastructure-providers:
|
|
- "gcp:v0.3.0"
|
|
control-plane-providers:
|
|
- "kubeadm:v0.3.3"
|
|
providers:
|
|
- name: "gcp"
|
|
type: "InfrastructureProvider"
|
|
variable-substitution: true
|
|
versions:
|
|
v0.3.0: manifests/function/capg/v0.3.0
|
|
- name: "kubeadm"
|
|
type: "BootstrapProvider"
|
|
versions:
|
|
v0.3.3: manifests/function/cabpk/v0.3.3
|
|
- name: "cluster-api"
|
|
type: "CoreProvider"
|
|
versions:
|
|
v0.3.3: manifests/function/capi/v0.3.3
|
|
- name: "kubeadm"
|
|
type: "ControlPlaneProvider"
|
|
versions:
|
|
v0.3.3: manifests/function/cacpk/v0.3.3
|
|
additional-vars:
|
|
GCP_CONTROL_PLANE_MACHINE_TYPE: "bjEtc3RhbmRhcmQtNA=="
|
|
GCP_NODE_MACHINE_TYPE: "bjEtc3RhbmRhcmQtNA=="
|
|
GCP_PROJECT: "<B64ENCODED_GCP_PROJECT_ID>"
|
|
GCP_REGION: "dXMtd2VzdDE="
|
|
GCP_NETWORK_NAME: "ZGVmYXVsdA=="
|
|
GCP_B64ENCODED_CREDENTIALS: "<GCP_B64ENCODED_CREDENTIALS>"
|
|
```
|
|
|
|
## Initialize Management Cluster
|
|
|
|
$ airshipctl phase run clusterctl-init-ephemeral
|
|
|
|
```
|
|
[airshipctl] 2020/09/02 11:14:15 Verifying that variable GCP_REGION is allowed to be appended
|
|
[airshipctl] 2020/09/02 11:14:15 Verifying that variable GCP_B64ENCODED_CREDENTIALS is allowed to be appended
|
|
[airshipctl] 2020/09/02 11:14:15 Verifying that variable GCP_CONTROL_PLANE_MACHINE_TYPE is allowed to be appended
|
|
[airshipctl] 2020/09/02 11:14:15 Verifying that variable GCP_NETWORK_NAME is allowed to be appended
|
|
[airshipctl] 2020/09/02 11:14:15 Verifying that variable GCP_NODE_MACHINE_TYPE is allowed to be appended
|
|
.
|
|
.
|
|
.
|
|
Patching Secret="capg-manager-bootstrap-credentials" Namespace="capg-system"
|
|
Creating Service="capg-controller-manager-metrics-service" Namespace="capg-system"
|
|
Creating Deployment="capg-controller-manager" Namespace="capg-system"
|
|
Creating inventory entry Provider="infrastructure-gcp" Version="v0.3.0" TargetNamespace="capg-system"
|
|
```
|
|
|
|
$ kubectl get pods -A
|
|
```
|
|
NAMESPACE NAME READY STATUS RESTARTS AGE
|
|
capg-system capg-controller-manager-b8655ddb4-swwzk 2/2 Running 0 54s
|
|
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-66c6b6857b-22hg4 2/2 Running 0 73s
|
|
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-688f7ccc56-7g676 2/2 Running 0 65s
|
|
capi-system capi-controller-manager-549c757797-6vscq 2/2 Running 0 84s
|
|
capi-webhook-system capg-controller-manager-d5f85c48d-74gj6 2/2 Running 0 61s
|
|
capi-webhook-system capi-controller-manager-5f8fc485bb-stflj 2/2 Running 0 88s
|
|
capi-webhook-system capi-kubeadm-bootstrap-controller-manager-6b645d9d4c-2crk7 2/2 Running 0 81s
|
|
capi-webhook-system capi-kubeadm-control-plane-controller-manager-65dbd6f999-cghmx 2/2 Running 0 70s
|
|
cert-manager cert-manager-77d8f4d85f-cqp7m 1/1 Running 0 115s
|
|
cert-manager cert-manager-cainjector-75f88c9f56-qh9m8 1/1 Running 0 115s
|
|
cert-manager cert-manager-webhook-56669d7fcb-6zddl 1/1 Running 0 115s
|
|
kube-system coredns-66bff467f8-kmg7c 1/1 Running 0 3m55s
|
|
kube-system coredns-66bff467f8-lg8qc 1/1 Running 0 3m55s
|
|
kube-system etcd-capi-gcp-control-plane 1/1 Running 0 4m4s
|
|
kube-system kindnet-dzp8v 1/1 Running 0 3m55s
|
|
kube-system kube-apiserver-capi-gcp-control-plane 1/1 Running 0 4m4s
|
|
kube-system kube-controller-manager-capi-gcp-control-plane 1/1 Running 0 4m3s
|
|
kube-system kube-proxy-zvdh8 1/1 Running 0 3m55s
|
|
kube-system kube-scheduler-capi-gcp-control-plane 1/1 Running 0 3m56s
|
|
local-path-storage local-path-provisioner-bd4bb6b75-6drnt 1/1 Running 0 3m55s
|
|
```
|
|
|
|
## Create control plane and worker nodes
|
|
|
|
$ airshipctl phase run controlplane-target --debug
|
|
```
|
|
[airshipctl] 2020/09/02 11:21:08 building bundle from kustomize path /tmp/airship/airshipctl/manifests/site/gcp-test-site/target/controlplane
|
|
[airshipctl] 2020/09/02 11:21:08 Applying bundle, inventory id: kind-capi-gcp-target-controlplane
|
|
[airshipctl] 2020/09/02 11:21:08 Inventory Object config Map not found, auto generating Invetory object
|
|
[airshipctl] 2020/09/02 11:21:08 Injecting Invetory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"kind-capi-gcp-target-controlplane"},"name":"airshipit-kind-capi-gcp-target-controlplane","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
|
|
[airshipctl] 2020/09/02 11:21:08 Making sure that inventory object namespace airshipit exists
|
|
configmap/airshipit-kind-capi-gcp-target-controlplane-5ab3466f created
|
|
cluster.cluster.x-k8s.io/gtc created
|
|
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/gtc-control-plane created
|
|
gcpcluster.infrastructure.cluster.x-k8s.io/gtc created
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-control-plane created
|
|
5 resource(s) applied. 5 created, 0 unchanged, 0 configured
|
|
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/gtc-control-plane is NotFound: Resource not found
|
|
gcpcluster.infrastructure.cluster.x-k8s.io/gtc is NotFound: Resource not found
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-control-plane is NotFound: Resource not found
|
|
configmap/airshipit-kind-capi-gcp-target-controlplane-5ab3466f is NotFound: Resource not found
|
|
cluster.cluster.x-k8s.io/gtc is NotFound: Resource not found
|
|
configmap/airshipit-kind-capi-gcp-target-controlplane-5ab3466f is Current: Resource is always ready
|
|
cluster.cluster.x-k8s.io/gtc is Current: Resource is current
|
|
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/gtc-control-plane is Current: Resource is current
|
|
gcpcluster.infrastructure.cluster.x-k8s.io/gtc is Current: Resource is current
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-control-plane is Current: Resource is current
|
|
all resources has reached the Current status
|
|
```
|
|
$ airshipctl phase run workers-target --debug
|
|
```
|
|
[airshipctl] 2020/09/02 11:21:20 building bundle from kustomize path /tmp/airship/airshipctl/manifests/site/gcp-test-site/target/workers
|
|
[airshipctl] 2020/09/02 11:21:20 Applying bundle, inventory id: kind-capi-gcp-target-workers
|
|
[airshipctl] 2020/09/02 11:21:20 Inventory Object config Map not found, auto generating Invetory object
|
|
[airshipctl] 2020/09/02 11:21:20 Injecting Invetory Object: {"apiVersion":"v1","kind":"ConfigMap","metadata":{"creationTimestamp":null,"labels":{"cli-utils.sigs.k8s.io/inventory-id":"kind-capi-gcp-target-workers"},"name":"airshipit-kind-capi-gcp-target-workers","namespace":"airshipit"}}{nsfx:false,beh:unspecified} into bundle
|
|
[airshipctl] 2020/09/02 11:21:20 Making sure that inventory object namespace airshipit exists
|
|
configmap/airshipit-kind-capi-gcp-target-workers-1a36e40a created
|
|
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/gtc-md-0 created
|
|
machinedeployment.cluster.x-k8s.io/gtc-md-0 created
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-md-0 created
|
|
4 resource(s) applied. 4 created, 0 unchanged, 0 configured
|
|
configmap/airshipit-kind-capi-gcp-target-workers-1a36e40a is NotFound: Resource not found
|
|
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/gtc-md-0 is NotFound: Resource not found
|
|
machinedeployment.cluster.x-k8s.io/gtc-md-0 is NotFound: Resource not found
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-md-0 is NotFound: Resource not found
|
|
configmap/airshipit-kind-capi-gcp-target-workers-1a36e40a is Current: Resource is always ready
|
|
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/gtc-md-0 is Current: Resource is current
|
|
machinedeployment.cluster.x-k8s.io/gtc-md-0 is Current: Resource is current
|
|
gcpmachinetemplate.infrastructure.cluster.x-k8s.io/gtc-md-0 is Current: Resource is current
|
|
```
|
|
|
|
$ kubectl get pods -A
|
|
```
|
|
NAMESPACE NAME READY STATUS RESTARTS AGE
|
|
capg-system capg-controller-manager-b8655ddb4-swwzk 2/2 Running 0 6m9s
|
|
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-66c6b6857b-22hg4 2/2 Running 0 6m28s
|
|
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-688f7ccc56-7g676 2/2 Running 0 6m20s
|
|
capi-system capi-controller-manager-549c757797-6vscq 2/2 Running 0 6m39s
|
|
capi-webhook-system capg-controller-manager-d5f85c48d-74gj6 2/2 Running 0 6m16s
|
|
capi-webhook-system capi-controller-manager-5f8fc485bb-stflj 2/2 Running 0 6m43s
|
|
capi-webhook-system capi-kubeadm-bootstrap-controller-manager-6b645d9d4c-2crk7 2/2 Running 0 6m36s
|
|
capi-webhook-system capi-kubeadm-control-plane-controller-manager-65dbd6f999-cghmx 2/2 Running 0 6m25s
|
|
cert-manager cert-manager-77d8f4d85f-cqp7m 1/1 Running 0 7m10s
|
|
cert-manager cert-manager-cainjector-75f88c9f56-qh9m8 1/1 Running 0 7m10s
|
|
cert-manager cert-manager-webhook-56669d7fcb-6zddl 1/1 Running 0 7m10s
|
|
kube-system coredns-66bff467f8-kmg7c 1/1 Running 0 9m10s
|
|
kube-system coredns-66bff467f8-lg8qc 1/1 Running 0 9m10s
|
|
kube-system etcd-capi-gcp-control-plane 1/1 Running 0 9m19s
|
|
kube-system kindnet-dzp8v 1/1 Running 0 9m10s
|
|
kube-system kube-apiserver-capi-gcp-control-plane 1/1 Running 0 9m19s
|
|
kube-system kube-controller-manager-capi-gcp-control-plane 1/1 Running 0 9m18s
|
|
kube-system kube-proxy-zvdh8 1/1 Running 0 9m10s
|
|
kube-system kube-scheduler-capi-gcp-control-plane 1/1 Running 0 9m11s
|
|
local-path-storage local-path-provisioner-bd4bb6b75-6drnt 1/1 Running 0 9m10s
|
|
```
|
|
|
|
To check logs run the below command
|
|
|
|
$ kubectl logs capg-controller-manager-b8655ddb4-swwzk -n capg-system --all-containers=true -f
|
|
|
|
```
|
|
I0902 18:15:30.884391 1 main.go:213] Generating self signed cert as no cert is provided
|
|
I0902 18:15:35.135060 1 main.go:243] Starting TCP socket on 0.0.0.0:8443
|
|
I0902 18:15:35.175185 1 main.go:250] Listening securely on 0.0.0.0:8443
|
|
I0902 18:15:51.111202 1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen" "addr"="127.0.0.1:8080"
|
|
I0902 18:15:51.113054 1 main.go:205] setup "msg"="starting manager"
|
|
I0902 18:15:51.113917 1 leaderelection.go:242] attempting to acquire leader lease capg-system/controller-leader-election-capg...
|
|
I0902 18:15:51.114691 1 internal.go:356] controller-runtime/manager "msg"="starting metrics server" "path"="/metrics"
|
|
I0902 18:15:51.142032 1 leaderelection.go:252] successfully acquired lease capg-system/controller-leader-election-capg
|
|
I0902 18:15:51.145165 1 controller.go:164] controller-runtime/controller "msg"="Starting EventSource" "c
|
|
```
|
|
|
|
$ kubectl get machines
|
|
```
|
|
NAME PROVIDERID PHASE
|
|
gtc-control-plane-cxcd4 gce://virtual-anchor-281401/us-west1-a/gtc-control-plane-vmplz Running
|
|
gtc-md-0-6cf7474cff-zpbxv gce://virtual-anchor-281401/us-west1-a/gtc-md-0-7mccx Running
|
|
```
|
|
|
|
$ kubectl --namespace=default get secret/gtc-kubeconfig -o jsonpath={.data.value} | base64 --decode > ./gtc.kubeconfig
|
|
|
|
$ kubectl get pods -A --kubeconfig ~/gtc.kubeconfig
|
|
|
|
```
|
|
NAMESPACE NAME READY STATUS RESTARTS AGE
|
|
kube-system calico-kube-controllers-6d4fbb6df9-8lf4f 1/1 Running 0 5m18s
|
|
kube-system calico-node-6lmqw 1/1 Running 0 73s
|
|
kube-system calico-node-qtgzj 1/1 Running 1 5m18s
|
|
kube-system coredns-5644d7b6d9-dqd75 1/1 Running 0 5m18s
|
|
kube-system coredns-5644d7b6d9-ls2q9 1/1 Running 0 5m18s
|
|
kube-system etcd-gtc-control-plane-vmplz 1/1 Running 0 4m53s
|
|
kube-system kube-apiserver-gtc-control-plane-vmplz 1/1 Running 0 4m42s
|
|
kube-system kube-controller-manager-gtc-control-plane-vmplz 1/1 Running 0 4m59s
|
|
kube-system kube-proxy-6hk8c 1/1 Running 0 5m18s
|
|
kube-system kube-proxy-b8mqw 1/1 Running 0 73s
|
|
kube-system kube-scheduler-gtc-control-plane-vmplz 1/1 Running 0 4m47s
|
|
```
|
|
|
|
Now, the control plane and worker node are created on google cloud.
|
|
|
|
## Tear Down Clusters
|
|
|
|
If you would like to delete the cluster run the below commands. This will delete
|
|
the control plane, workers, machine health check and all other resources
|
|
associated with the cluster on gcp.
|
|
|
|
$ airshipctl phase render controlplane -k Cluster
|
|
|
|
```
|
|
---
|
|
apiVersion: cluster.x-k8s.io/v1alpha3
|
|
kind: Cluster
|
|
metadata:
|
|
name: gtc
|
|
namespace: default
|
|
spec:
|
|
clusterNetwork:
|
|
pods:
|
|
cidrBlocks:
|
|
- 192.168.0.0/16
|
|
controlPlaneRef:
|
|
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
|
|
kind: KubeadmControlPlane
|
|
name: gtc-control-plane
|
|
infrastructureRef:
|
|
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
|
|
kind: GCPCluster
|
|
name: gtc
|
|
...
|
|
```
|
|
|
|
$ airshipctl phase render controlplane -k Cluster | kubectl delete -f -
|
|
|
|
```
|
|
cluster.cluster.x-k8s.io "gtc" deleted
|
|
```
|
|
|
|
$ kind delete cluster --name capi-gcp
|
|
```
|
|
Deleting cluster "capi-gcp" ...
|
|
```
|
|
|
|
## Reference
|
|
|
|
### Provider Manifests
|
|
|
|
Provider Configuration is referenced from https://github.com/kubernetes-sigs/cluster-api-provider-gcp/tree/master/config
|
|
Cluster API does not support gcp provider out of the box. Therefore, the metadata infromation is added using files in
|
|
airshipctl/manifests/function/capg/data
|
|
|
|
$ tree airshipctl/manifests/function/capg
|
|
|
|
```
|
|
airshipctl/manifests/function/capg
|
|
└── v0.3.0
|
|
├── certmanager
|
|
│ ├── certificate.yaml
|
|
│ ├── kustomization.yaml
|
|
│ └── kustomizeconfig.yaml
|
|
├── crd
|
|
│ ├── bases
|
|
│ │ ├── infrastructure.cluster.x-k8s.io_gcpclusters.yaml
|
|
│ │ ├── infrastructure.cluster.x-k8s.io_gcpmachines.yaml
|
|
│ │ └── infrastructure.cluster.x-k8s.io_gcpmachinetemplates.yaml
|
|
│ ├── kustomization.yaml
|
|
│ ├── kustomizeconfig.yaml
|
|
│ └── patches
|
|
│ ├── cainjection_in_gcpclusters.yaml
|
|
│ ├── cainjection_in_gcpmachines.yaml
|
|
│ ├── cainjection_in_gcpmachinetemplates.yaml
|
|
│ ├── webhook_in_gcpclusters.yaml
|
|
│ ├── webhook_in_gcpmachines.yaml
|
|
│ └── webhook_in_gcpmachinetemplates.yaml
|
|
├── data
|
|
│ ├── capg-resources.yaml
|
|
│ ├── kustomization.yaml
|
|
│ └── metadata.yaml
|
|
├── default
|
|
│ ├── credentials.yaml
|
|
│ ├── kustomization.yaml
|
|
│ ├── manager_credentials_patch.yaml
|
|
│ ├── manager_prometheus_metrics_patch.yaml
|
|
│ ├── manager_role_aggregation_patch.yaml
|
|
│ └── namespace.yaml
|
|
├── kustomization.yaml
|
|
├── manager
|
|
│ ├── kustomization.yaml
|
|
│ ├── manager_auth_proxy_patch.yaml
|
|
│ ├── manager_image_patch.yaml
|
|
│ ├── manager_pull_policy.yaml
|
|
│ └── manager.yaml
|
|
├── patch_crd_webhook_namespace.yaml
|
|
├── rbac
|
|
│ ├── auth_proxy_role_binding.yaml
|
|
│ ├── auth_proxy_role.yaml
|
|
│ ├── auth_proxy_service.yaml
|
|
│ ├── kustomization.yaml
|
|
│ ├── leader_election_role_binding.yaml
|
|
│ ├── leader_election_role.yaml
|
|
│ ├── role_binding.yaml
|
|
│ └── role.yaml
|
|
└── webhook
|
|
├── kustomization.yaml
|
|
├── kustomizeconfig.yaml
|
|
├── manager_webhook_patch.yaml
|
|
├── manifests.yaml
|
|
├── service.yaml
|
|
└── webhookcainjection_patch.yaml
|
|
```
|
|
|
|
#### CAPG Specific Variables
|
|
|
|
capg-resources.yaml consists of `gcp provider specific` variables required to
|
|
initialize the management cluster. The values for these variables can be
|
|
exported before running `airshipctl phase run clusterctl-init-ephemeral` or they can be defined
|
|
explicitly in clusterctl.yaml
|
|
|
|
$ cat airshipctl/manifests/function/capg/v0.3.0/data/capg-resources.yaml
|
|
|
|
```
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: manager-bootstrap-credentials
|
|
namespace: system
|
|
type: Opaque
|
|
data:
|
|
GCP_CONTROL_PLANE_MACHINE_TYPE: ${GCP_CONTROL_PLANE_MACHINE_TYPE}
|
|
GCP_NODE_MACHINE_TYPE: ${GCP_NODE_MACHINE_TYPE}
|
|
GCP_PROJECT: ${GCP_PROJECT}
|
|
GCP_REGION: ${GCP_REGION}
|
|
GCP_NETWORK_NAME: ${GCP_NETWORK_NAME}
|
|
GCP_B64ENCODED_CREDENTIALS: ${GCP_B64ENCODED_CREDENTIALS}
|
|
|
|
```
|
|
|
|
### Cluster Templates
|
|
manifests/function/k8scontrol-capg contains cluster.yaml, controlplane.yaml templates referenced from
|
|
[cluster-template](https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/master/templates/cluster-template.yaml)
|
|
|
|
| Template Name | CRDs |
|
|
| ----------------- | ---- |
|
|
| cluster.yaml | Cluster, GCPCluster |
|
|
| controlplane.yaml | KubeadmControlPlane, GCPMachineTemplate |
|
|
|
|
$ tree airshipctl/manifests/function/k8scontrol-capg
|
|
|
|
```
|
|
airshipctl/manifests/function/k8scontrol-capg
|
|
├── cluster.yaml
|
|
├── controlplane.yaml
|
|
└── kustomization.yaml
|
|
```
|
|
|
|
airshipctl/manifests/function/workers-capg contains workers.yaml referenced from
|
|
[cluster-template](https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/master/templates/cluster-template.yaml)
|
|
|
|
| Template Name | CRDs |
|
|
| ----------------- | ---- |
|
|
| workers.yaml | GCPMachineTemplate, MachineDeployment, KubeadmConfigTemplate |
|
|
|
|
$ tree airshipctl/manifests/function/workers-capg
|
|
```
|
|
airshipctl/manifests/function/workers-capg
|
|
├── kustomization.yaml
|
|
└── workers.yaml
|
|
```
|
|
### Test Site Manifests
|
|
|
|
#### gcp-test-site/shared
|
|
airshipctl phase run clusterctl-init-ephemeral uses
|
|
airshipctl/manifests/site/gcp-test-site/shared/clusterctl to initialize
|
|
management cluster with defined provider components and version.
|
|
|
|
$ tree airshipctl/manifests/site/gcp-test-site/shared
|
|
```
|
|
airshipctl/manifests/site/gcp-test-site/shared
|
|
└── clusterctl
|
|
├── clusterctl.yaml
|
|
└── kustomization.yaml
|
|
```
|
|
|
|
#### gcp-test-site/target
|
|
There are 3 phases currently available in gcp-test-site/target
|
|
|
|
|Phase Name | Purpose |
|
|
|-----------|---------|
|
|
| controlplane | Patches templates in manifests/function/k8scontrol-capg |
|
|
| workers | Patches template in manifests/function/workers-capg | |
|
|
| initinfra | Simply calls `gcp-test-site/shared/clusterctl` |
|
|
|
|
Note: `airshipctl phase run clusterctl-init-ephemeral` initializes all the provider components
|
|
including the gcp infrastructure provider component.
|
|
|
|
#### Patch Merge Strategy
|
|
|
|
Json patches are applied on templates in `manifests/function/k8scontrol-capg`
|
|
from `airshipctl/manifests/site/gcp-test-site/target/controlplane` when
|
|
`airshipctl phase run controlplane-target` is executed
|
|
|
|
Json patches are applied on templates in `manifests/function/workers-capg` from
|
|
`airshipctl/manifests/site/gcp-test-site/target/workers` when `airshipctl phase
|
|
run workers-target` is executed.
|
|
|
|
| Patch Name | Purpose |
|
|
| ------------------------------- | ------------------------------------------------------------------ |
|
|
| controlplane/machine_count.json | patches control plane machine count in template function/k8scontrol-capg |
|
|
| controlplane/machine_type.json | patches control plane machine type in template function/k8scontrol-capg |
|
|
| controlplane/network_name.json | patches control plane network name in template function/k8scontrol-capg |
|
|
| controlplane/project_name.json | patches project id template function/k8scontrol-capg |
|
|
| controlplane/region_name.json | patches region name in template function/k8scontrol-capg |
|
|
| workers/machine_count.json | patches worker machine count in template function/workers-capg |
|
|
| workers/machine_type.json | patches worker machine type in template function/workers-capg |
|
|
| workers/failure_domain.json | patches failure_domain in template function/workers-capg |
|
|
|
|
$ tree airshipctl/manifests/site/gcp-test-site/target/
|
|
|
|
```
|
|
airshipctl/manifests/site/gcp-test-site/target/
|
|
├── controlplane
|
|
│ ├── kustomization.yaml
|
|
│ ├── machine_count.json
|
|
│ ├── machine_type.json
|
|
│ ├── network_name.json
|
|
│ ├── project_name.json
|
|
│ └── region_name.json
|
|
├── initinfra
|
|
│ └── kustomization.yaml
|
|
└── workers
|
|
├── failure_domain.json
|
|
├── kustomization.yaml
|
|
├── machine_count.json
|
|
└── machine_type.json
|
|
|
|
3 directories, 11 files
|
|
|
|
```
|
|
### Software Version Information
|
|
|
|
All the instructions provided in the document have been tested using the
|
|
software and version, provided in this section.
|
|
|
|
#### Virtual Machine Specification
|
|
|
|
All the instructions in the document were perfomed on a Oracle Virtual Box(6.1)
|
|
VM running Ubuntu 18.04.4 LTS (Bionic Beaver) with 16G of memory and 4 VCPUs
|
|
|
|
#### Docker
|
|
|
|
$ docker version
|
|
|
|
```
|
|
Client: Docker Engine - Community
|
|
Version: 19.03.9
|
|
API version: 1.40
|
|
Go version: go1.13.10
|
|
Git commit: 9d988398e7
|
|
Built: Fri May 15 00:25:18 2020
|
|
OS/Arch: linux/amd64
|
|
Experimental: false
|
|
|
|
Server: Docker Engine - Community
|
|
Engine:
|
|
Version: 19.03.9
|
|
API version: 1.40 (minimum version 1.12)
|
|
Go version: go1.13.10
|
|
Git commit: 9d988398e7
|
|
Built: Fri May 15 00:23:50 2020
|
|
OS/Arch: linux/amd64
|
|
Experimental: false
|
|
containerd:
|
|
Version: 1.2.13
|
|
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
|
|
runc:
|
|
Version: 1.0.0-rc10
|
|
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
|
|
docker-init:
|
|
Version: 0.18.0
|
|
GitCommit: fec3683
|
|
```
|
|
|
|
#### Kind
|
|
|
|
$ kind version
|
|
|
|
```
|
|
kind v0.8.1 go1.14.2 linux/amd64
|
|
```
|
|
|
|
#### Kubectl
|
|
|
|
$ kubectl version
|
|
|
|
```
|
|
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
|
|
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
|
|
```
|
|
|
|
#### Go
|
|
|
|
$ go version
|
|
|
|
```
|
|
go version go1.14.1 linux/amd64
|
|
```
|
|
|
|
#### Kustomize
|
|
|
|
$ kustomize version
|
|
|
|
```
|
|
{Version:kustomize/v3.8.0 GitCommit:6a50372dd5686df22750b0c729adaf369fbf193c BuildDate:2020-07-05T14:08:42Z GoOs:linux GoArch:amd64}
|
|
```
|
|
|
|
#### OS
|
|
|
|
$ cat /etc/os-release
|
|
|
|
```
|
|
NAME="Ubuntu"
|
|
VERSION="18.04.4 LTS (Bionic Beaver)"
|
|
ID=ubuntu
|
|
ID_LIKE=debian
|
|
PRETTY_NAME="Ubuntu 18.04.4 LTS"
|
|
VERSION_ID="18.04"
|
|
HOME_URL="https://www.ubuntu.com/"
|
|
SUPPORT_URL="https://help.ubuntu.com/"
|
|
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
|
|
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
|
|
VERSION_CODENAME=bionic
|
|
UBUNTU_CODENAME=bionic
|
|
```
|