How to install AMKO on Tanzu Kubernetes Clusters(TKC) / TKG-Service

How to install AMKO on Tanzu Kubernetes Clusters(TKC) / TKG-Service

***Disclaimer: If you are using TKG-Service with NSX-T network with AMKO, this solution is not supported by VMware at the moment. It would be good for POC / Testing only.

***Update (May 2021): I have tested this on vSphere 7.0U2, NSX-T 3.1 and AMKO 1.4.1, below steps are still valid.

Introduction

Well, let me guess. If you stumble on this blog post, I would assume you have seen the goodness and coolness of AKO (AVI Kubernetes Operator). While AKO provides superb Ingress Services for Kubernetes Cluster, you will start questioning yourself, what happens if you have multiple Kubernetes Cluster? How can I provide High Availablity Ingress Services or even better how can I load balance across my multiple Kubernetes Clsuters? The answer to these questions is AVI Multiple Kubernetes Operator(AMKO). There is a pretty good write-up on AMKO by one of the Solution Architecture from VMware-AVI team titled Deliver Elastic Kubernetes Ingress Controller and Services. I will put the link here for you to read about it.

Something about TKC. To be specific, when I mention about TKC, I’m referring to the TKG-Service. The easiest way to explain what is the TKG-Service is to make a comparison with TKG-Multi-Cloud. By the way, both TKG-S and TKG-M provides you on-demand Kubernetes Clusters. That means, you can create a Kubernetes Clusters using a YAML file with CLI. I tried spinning up two TKG-S in my customer production environment and it tooks less than 20 minutes! In short, basically, TKG-Service needs have vSphere 7.0 with Workload Management whereas TKG-Multi-Cloud, you can run it in vSphere 6.7 and above and in the Public Cloud.

Problem Statement

So, for most of the TKGs, the way you login to the cluster is using Token. However, for AMKO installation, you will need user accounts. Therefore, you will need create service accounts on the TKGs and cluster role binding for it, for the installation of AMKO. In this blog post, I will share the steps on how to do exactly this.

Pre-requisite

I would assume you already have AKO installed on your TKGs. My fellow colleague Ali Al Idrees did a phenomenon job in creating a guide on how to install AKO - NSX ALB in TKC. The additional thing you have to do is to create an additional TKG cluster.

##

Creating Kubernetes Service Accounts

  1. You can use this script to create Kubernetes User Account https://gist.github.com/hustshawn/e6109c6ddb7ed845e7a1298c526588b6

There is an small typo in this script. You have to change base64 -D to base -d.

  1. Create a Cluster Role Binding for the User Account you just created. kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole=cluster-admin --user=tkc—04-admin1

Create gslb-members file.

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://10.30.10.8:6443
  name: tkc-04
- cluster:
    insecure-skip-tls-verify: true
    server: https://10.30.10.9:6443
  name: tkc-05
contexts:
- context:
    cluster: tkc-04
    user: tkc-04-admin1
  name: tkc-04
- context:
    cluster: tkc-05
    user: tkc-05-admin1
  name: tkc-05
current-context: tkc-04
kind: Config
preferences: {}
users:
- name: tkc-04-admin1
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNRENDQWhpZ0F3SUJBZ0lVU2hQZXZnTlZ0TThFeVRQUHpjQ1lqN1pqL2pJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweU1ERXdNVEF3TVRNMk1EQmFGdzB5TVRFdwpNVEF3TVRNMk1EQmFNQzh4RlRBVEJnTlZCQW9UREc5eVoyRnVhWHBoZEdsdmJqRVdNQlFHQTFVRUF4TU5kR3RqCkxUQTBMV0ZrYldsdU1UQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1uQnAzYzYKT1ByRGlBWERhdmQwRklDU2kxdGY3RW5NUkpicEt4NUxZUXllYXVMQktaaWlCcnpFTW1hdTZFRUJmblg3NFd1dgowaTBTY1hRQ0dKTFVCSmtNQ3NMejdIcVdmYTdTTHcyQnQwczV6WW4yQlY4ckpDQ3k2emxLeWVjOUdEM0t4NTRTCjRaSldFV0dIY25hRU5MZytJL29ORlZqc2w2b04rMDNiMVRlUmRDSFY5dHIzL3B6VVd2REJxR3FwTERaeXhiNXIKK1g0bXFXMjNBUTQvNTFjSG1YeUw3Z1NtTGNVNlJQMmpQS2J2Q1FtQlk5eXJNcFN1SWJNdEhqN1RFeHRRbm5jUwpidGQ5TmE5MFg0VXRaTlpVdWcrUVRnZjdzRmh6UzNlakFpMXBKTCsyR3IyQ2ROQjB6ZmNsd3NHb21XWTdPc3k0ClQrOGJwWjJubnRmdFVqMENBd0VBQWFOZU1Gd3dEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUcKQ0NzR0FRVUZCd01DQmdnckJnRUZCUWNEQVRBTUJnTlZIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlJ3Vi9FZAp5YlF1eUJKN0R5MUluZkJOVVI4anBEQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFWcmlmS2VvZWFUZXluVjYvCjc5SzNnVElna3d4V2hVenY4Q1Y0REYvZjF0cVBnVmc0ZkFud01zU3d4ckphNUNqK1A3N1Y5T3U5VGE2WnV6N2QKV2UyRXhqUm83RGJhTFBFT0Z0dFpid2tKYS9MZEUrRUhRK1dlYVRZN2xBTEpPOW9BSzAwMlhtYmpGZUp0RHFBQwpzaGxFZllhZm8yemZMYkErUTk0bWtJUVhlUmhOSjhHRE9oL2F6MjVqT2p4ZzVEK2R5dTcrRXZ2YjVoT1Q0N243CnBHRjcwS09udi9JcW8rd1piU292MW55VDJnS1VyeThvSTNyZ3JmWHVEL2xxRGFwMnp0dXNzY01nbzhvSVNqVkoKNkh5ait6N3UzOUFNN1E4WU1aL0R3MTNQcTdwZE1YT1lqZWpPdW1yd0Z1a0FTa0djd0oxdGlkZzYwdXRPMXNRZgpPeldkS1E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeWNHbmR6bzQrc09JQmNOcTkzUVVnSktMVzEvc1NjeEVsdWtySGt0aERKNXE0c0VwCm1LSUd2TVF5WnE3b1FRRitkZnZoYTYvU0xSSnhkQUlZa3RRRW1Rd0t3dlBzZXBaOXJ0SXZEWUczU3puTmlmWUYKWHlza0lMTHJPVXJKNXowWVBjckhuaExoa2xZUllZZHlkb1EwdUQ0aitnMFZXT3lYcWczN1RkdlZONUYwSWRYMgoydmYrbk5SYThNR29hcWtzTm5MRnZtdjVmaWFwYmJjQkRqL25Wd2VaZkl2dUJLWXR4VHBFL2FNOHB1OEpDWUZqCjNLc3lsSzRoc3kwZVB0TVRHMUNlZHhKdTEzMDFyM1JmaFMxazFsUzZENUJPQi91d1dITkxkNk1DTFdra3Y3WWEKdllKMDBIVE45eVhDd2FpWlpqczZ6TGhQN3h1bG5hZWUxKzFTUFFJREFRQUJBb0lCQUN1WjJTb1FBQTVaUVVVVgpsVUZDNmNkYlpCaHNJM0ZUWHFOS1lwSWFjaSs2OEpodWJDSGJLaCtUT3Qwc0FXOUM5bVJpK2JVKyt1bllxWUlMCmplbjFQY1VoY0JYdmh2UjdIMUF2aGhHaUxjMXBBWDFHR0tVWGVxaENZa2t3VzIvSXFUdDVYaWhRSmtWc3FqYnMKZ1pVUlB4OXQ5bkpzYjJlWFFiTnpjQkxFWmxDcWtKY0piLzlJOGc2R0pzYlN3aVBFQ2ZFS3VLdzlURXUzV2p3eQo4V1hIMnJTb2d1SnhiVkVoNkh6UnpYWkg4R2lzd3h1WmRYNWY0RkduMVFnQ0xvU1VUaFZsZDJaR09hTzIzTjVWCkU1TWdjS002OUh3ekVqWVZrQUFwOGhsOXl3QldtOWhWL3NwY1pmbFhmOXpHOUNwaDRYVHdwMDhRelliN1FUd24KbjJTUW9lRUNnWUVBL2wyb3Q2MUdybm9GVUsxc213REw4dTB3Zmt3cXNBZzAzdDJUMU1uc1NNZXAzd3Y3bk5wTwpScUFpazVMQ21RelEwQ3dwem1YbHcybUJzSStkNTlhNlJLbUFmMmZCQm0zSVFrMmJXNDdETzNoRVRlK3lwSzd1Cm1rVWJYVlFnK25hWHRtaVRJYXA5S1lseDgxTUszWmpyTUp6aURaYnhRai9hQ2tOcEYvcWtlb2tDZ1lFQXl3MTQKc1QwS0lQTjNRdHR0Q2ZoQ0JFbThKYzhYRWxBVlNyNHNBRDJVZTVNNjliRDExUU90SW5rTUF1WTNRdWdFOVBNRgpydUY3SFVLUGJZSTNPdzVwSHZVTUs4b2RWYWJoN2ZURHROZnI1Rnl1QWZjMFhIUHU0WVNEZ1VOeTZRbThnNzRVCkFSeUJ2SGdxMVRhbEFtRjZNTGhrU2RRSzViUzhyV3VkL1krdjNSVUNnWUFabmNVSCttTkwzdTM2R2Z6OS95NksKNEdncHovSGNtSGI2bXNYd2dBVlRyd2ZrZG9oNFcyUFdxUG5TY3MwMkxKdzYycmU3R1hmeUhnWDVpZjBWQkFMdQpOMFBCa09sWGhHYk9ocEpHdmpIYkkxbDluODRWMTZnZjBTWFlXRWlaOEtlR3J6RmlDcEZnUlg4OFNmaW1LR2hICkZRV082TXlUSjJzRjdaUC8zallDUVFLQmdRQzdvZWMxWU1KTFVLK1VDRS9xTm81S2lCaGtsOGtxRXZhWjZzR2UKTjBsT0orOER1bEp3Vk1WWVphTER5eGlscXVGcHVHZzRBZ1JTTlg5MElqQUhVbm14TVovZGx4Y2tYdUhIa2JhSApGai9VOWcrVm9XbXhSVEFwRk5xd1NET1hSSGYvQlZaVzhKQ3lLQmpzaldKTm9PWFl2NHRzN1pSa1QyOUtrWGVkCkozbkxRUUtCZ1FDU080bC8vN0FxcmZzdWFBNDhkS09lbEloRUJwbjRzbHR2alZsVHN1MkVqdDZocnh3RUEydFkKRXNzQkdSSUd2NjJDR1hPZHl2RHRMY2EwODBBT1dSTzRlRjJvcGo3UXp2VjRYbDlHU3NxV1FOV1Yyd0kwSU1lMQpvL3JsQ0NiejhCaXZtSG8vVVZzeE12WS84VlRTMFNGeWlIU0cvQTFpNitQbkxhK2Z2S3c3eVE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
- name: tkc-05-admin1
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNRENDQWhpZ0F3SUJBZ0lVTDlGR3lXa3RGRU1jcE5WTXhEZnpCeU5vNEZnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0ZURVRNQkVHQTFVRUF4TUthM1ZpWlhKdVpYUmxjekFlRncweU1ERXdNVEF3TVRRNU1EQmFGdzB5TVRFdwpNVEF3TVRRNU1EQmFNQzh4RlRBVEJnTlZCQW9UREc5eVoyRnVhWHBoZEdsdmJqRVdNQlFHQTFVRUF4TU5kR3RqCkxUQTFMV0ZrYldsdU1UQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IMTJWN3MKaFRlazZ0ZGxRTTd3aHlmblFXT2hKSXExTnRmb3krRldCOUp3VDRhdFE0V3Z3bDJjY0wxaUFVR2FPMXV2bldRLworUlUvbzNIeEVzVnFSR1BkZEF1VGhZQmp4dmI1WXhEeThVTkFsbW5HS1ppSFVUQzRUZTNhemIxTmZjVUpUNVBlCnJzSHYzYThSaHhFQlBhc3JySmY5U1F1MmNRQ0xmUnBIYWZJTjJtVWVIb1NnYWYvaHZZVlVVckY2ZVgxOUNOVVcKTE1lamMxNWpLbFo5enVYRlp1WjFGbEZDcWZXcnhLUk5LdmlldlhkYnFWNTVtc0MyeFFlUFc1Sk1CNlVOZVp1NwpuRTZ6QjBySXVEYWY0YUQzS1BnRE1BOTZFcGlhZkN1cUFaUnh1TThyaCtpTU92UGpacWFGVEFGQzh1UDZQSjNXCjZtZkVjL2RISzB3WWcwTUNBd0VBQWFOZU1Gd3dEZ1lEVlIwUEFRSC9CQVFEQWdXZ01CMEdBMVVkSlFRV01CUUcKQ0NzR0FRVUZCd01DQmdnckJnRUZCUWNEQVRBTUJnTlZIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlNpRjRiRQpjTmhXZzVBU0R5YVgvR2tCMGZzVXhqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUF6c21QSWVNM1E1emZ4M3QvCk5rRkVFM1M1d0xRaGZCZGpnZTNTaUpLRVhWNnorVHJsV2hqTG04SWQ2ejlkczR2dnBuMWsxVnNISzcrZ09VOCsKWlhrUXpkNjMzb1ZZWGRCSmt2TUNpK1daTWNsRUp1d0NpZWkzRmFKcVNsby9ZNVVPZWxkcHJEeWJxME02UUR1Vgp1QVI4Q3d6QlJHU2VkemYyUnFFY2ZZcTJpMmgyY0ZKMjRPWEpjMW43eTdSd2N3bjNwMXZCSHMrTVpMbTZ2SG5lClViUk9WNjh1RTZhRTh5UURwTEN6MGlPcW1LclFWWEVIL29VWkI2OVlVYktwakN5eXpic3N5NkhMZzhNYWtrbGkKWVE3MUJ2NExIVTVOcHR3VDVvTUpNMFcwWWUzOGRhL1FxMFVFUjNkZGhQeTVUTk9RakwyUlJ5a1lkbXhwTlBwZwp5RVF5M3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBd2ZYWlh1eUZONlRxMTJWQXp2Q0hKK2RCWTZFa2lyVTIxK2pMNFZZSDBuQlBocTFECmhhL0NYWnh3dldJQlFabzdXNitkWkQvNUZUK2pjZkVTeFdwRVk5MTBDNU9GZ0dQRzl2bGpFUEx4UTBDV2FjWXAKbUlkUk1MaE43ZHJOdlUxOXhRbFBrOTZ1d2UvZHJ4R0hFUUU5cXl1c2wvMUpDN1p4QUl0OUdrZHA4ZzNhWlI0ZQpoS0JwLytHOWhWUlNzWHA1ZlgwSTFSWXN4Nk56WG1NcVZuM081Y1ZtNW5VV1VVS3A5YXZFcEUwcStKNjlkMXVwClhubWF3TGJGQjQ5Ymtrd0hwUTE1bTd1Y1RyTUhTc2k0TnAvaG9QY28rQU13RDNvU21KcDhLNm9CbEhHNHp5dUgKNkl3NjgrTm1wb1ZNQVVMeTQvbzhuZGJxWjhSejkwY3JUQmlEUXdJREFRQUJBb0lCQUZRTkpkVnN5WTREZVlFOQo3NGU5MDlRRzh0OHB4WndXUzEzbEUvVGlReUYrYWZnQ3ZYQUJha2tHc2hSZmxBdkp1azlSR3ZPR2hYb2Z5Y1JtCm82dkpFVG4xallRMnVYT1p5VzdkUTREa3hvaE9ST0F4YjNVWnJxVmVPT2NLR240ZVJwSk1KcC85cjkxN0JLWUoKQW03c0NyMVNmOGlkaWpuNUpaOEcwMysvMHJtQlpTZm1wenp2YjlQUmtEa3lqVEdJMVB5UkRmekZWTFhEaDUxSgpkZlFnTEpoWThQbWxpaTh1UzNMN2xsV0FXQ3VrWkp0dGpCMEpvZitMb29sKzNXc1llUitSYjBNN25XZUJDbmwvCkRCWDVEQWk0cGNqQ1dxR0xaWWgwcFoweXZVck1BL2lCWEhUNVJmYjJrYlEraGUxZHlLa3RDVzN1TXVjVEJUeWEKVmtpZG1RRUNnWUVBOXk4N1VhaXp1QUEzOVpWQU1oRVFtWjRmamUvSDJwNnE3N0hza0NOblNPNnAxSk5iak1meQpGSy9QNUl6OElRSzIyTUNhU2VrZUlZVkZ2K1BmVnJXcm9hMVBhZHphTXlac1RDUjN0RHY2dUs2K2RWU3ZaamFMCmpydGhMZ01RcDU2OXNPRVBIdzNhd1pUKzYveGh5cXlyaGVoSFBieVgwQWR3bU1SZHJtT1FHTHNDZ1lFQXlPQ3YKd0pGTXgxQ1BvOGpUQmpKUXVITkpNWUV2QXJ4d1k2WDJwUzYwdTJWajQ0QUtFaE5ERXdhZjhCY2N0TUp5eDdubwpyYTFvd0VDbnFhZC9OV0JBVG1MemIzZ2Nyb0JwZUxZRjk5R2pqaTBxM3BxK29QOFkyb0pGSmpFSWRHQU93b1JZCmdiK0crU2IyM0tQdldTRHM1YU1YN2cwRnNaOERINUNwa2ZCSk94a0NnWUIvMlQwY01sNnFOK3E1cWZXbkw2NWYKVmlySjloQnZvZ3VWcUhoTDRSUGw1Qk9STUpwMGlXSkxrdGU3UmJTT3VtR3FZSXdzMkZkT1RTbFRZK0E3Y0FHUwp3UnBIRGdVSjJjQy83VWpBdnhraDlyZnA1ZUJHeE1XUUVKamwvSTNidEw0MlEySFgxUW1sZ0pRTnFOOUh2ZUdGCmhDdDg0aE8vZEdtbmlrRk9GbzhNU3dLQmdRQ0lvVXJwaitySkp1S01aR3drS3RUOG9HMklYblc3UnJDckZwZE0KZ3hncTNpdXk4Y1BueDNJMzNxbXFoQTNpWk4vdE9XWXB1MmtrV3RXdmIxMzIwQ0R0TUNDa0VPNXhRcVo2bEZRNgoyenlEMS9Ud3RCQTFNVTJXVWtUMlZTT0xCRXNGd3o3d0FYV3N3ZlBtM3hwTlpUZThlZElzK0c2SEFjRCtGMGMvCkhGQ3pBUUtCZ1FEZ3JQMlJOWHlrZXN2VDJFUG1Kd01mZjFrVklrZEl6bTRiR1JtSDFDWVovYTZxTExiSkttbHkKd3NXRDVkSUVIREFhdnJUZ2ZINm9TRUZjVmFQdUVnS2hHQ3R5QTFVeFdLVzZYUWZDSGREWk14NlFTbkZBRWQxUAowV2c5bmkvbG1CdnpCd1NsL0p0eXkxQkw1Mk9sdHIxaFdhd0M5N3RPQW01aHJ3TU50U0NQeXc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

Create gslb_values.yaml file

** This gslb_values.yaml file is based on AMKO 1.2. If you are using AMKO 1.4, please get the lastest values.yaml file using the below command.

helm show values ako/amko --version 1.4.1 > values.yaml

# Default values for amko.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: avinetworks/amko
  pullPolicy: IfNotPresent

configs:
  gslbLeaderController: ""
  controllerVersion: "20.1.1"
  memberClusters:
    - clusterContext: "tkc-04"
    - clusterContext: "tkc-05"
  refreshInterval: 1800
  logLevel: "INFO"

gslbLeaderCredentials:
  username: "admin" #Avi controller username
  password: "VMware1!" #Avi controller password

globalDeploymentPolicy:
  # appSelector takes the form of:
  appSelector:
    label:
      app: gslb
  # Uncomment below and add the required ingress/route/service label
  # appSelector:

  #namespaceSelector takes the form of:
  #namespaceSelector:
  #  label:
  #    ns: gslb   <example label key-value for namespace>
  # Uncomment below and add the reuqired namespace label
  # namespaceSelector:

  # list of all clusters that the GDP object will be applied to, can take any/all values
  # from .configs.memberClusters
  matchClusters:
    - "tkc-04"
    - "tkc-05"

  # list of all clusters and their traffic weights, if unspecified, default weights will be
  # given (optional). Uncomment below to add the required trafficSplit.
  # trafficSplit:
  #   - cluster: "cluster1-admin"
  #     weight: 8
  #   - cluster: "cluster2-admin"
  #     weight: 2

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

service:
  type: ClusterIP
  port: 80

persistentVolumeClaim: ""
mountPath: "/log"
logFile: "amko.log"

Install AMKO

kubectl create secret generic gslb-config-secret --from-file gslb-members -n avi-system

helm install amko/amko --generate-name --version 1.2.1 -f gslb_values.yaml --set configs.gslbLeaderController=10.115.1.41 --namespace=avi-system

Some Screenshots

NSX-T Topology

AVI SE on the same NSX-T Logical Segment as TKG worker nodes

AMKO Pod running successfully

Ingresses created by AKO on NSX ALB Controller

DNS Records with the Ingress FQDN created on NSX ALB GSLB

Testing AMKO - This shows, you will get two A records when you query the same FQDN

dig @10.115.1.3 test4.ako.acepod.com +short

Delete AMKO

kubectl delete secret gslb-config-secret -n avi-system
helm delete $(helm list -n avi-system -q | grep amko) -n avi-systems

Troubleshooting

kubectl get gdp -n avi-system -o yaml kubectl config use-context tkc-05 –kubeconfig gslb-members

kubectl get svc –kubeconfig

Closing Up

Thats about it! Hope you enjoy the greatness provided by AMKO for multi sites, multi cluster DNS Automation and Ingress Services Load Balancing.

Thanks for reading!

Reference

https://avinetworks.com/docs/amko/1.4/install-configure-amko/

https://avinetworks.com/docs/amko/1.2/install-configure-amko/