How to install Openshift Container Platform OCP with NSX-T NCP

The last few blog posts I wrote about the installation steps for Openshift Container Platform (OCP) with NSX-T NCP attracted some good interest from the community as well as VMware internal folks. However, those materials were written quite awhile back and some of software used then were not up to date. My customers were also looking at the later versions of software. Lastly, in OCP 3.11, the ansible playbooks for NSX-T NCP integration comes out of the box and therefore makes the integration much simpler. Therefore, gave the reason to write this blog post.

The high level steps remains unchanged. However, the part 5 in this case has been streamline into the Openshift installation.

Openshift with NSX-T Installation Part 1: Overview

Openshift with NSX-T Installation Part 2: NSX-T

Openshift with NSX-T Installation Part 3: RHEL Preparation

Openshift with NSX-T Installation Part 4: Openshift Installation

Openshift with NSX-T Installation Part 5: NCP and CNI Integration (Combine into Part 4)

Openshift with NSX-T Installation Part 6: Demo App

** For fellow VMware colleagues, to save you time for preparing the RHEL templates and VMs for OCP install, I have exported out the VMs from my Lab. I have uploaded in onedrive. Email me, I will happily share the link to download. Size is about 7GB.

Screen Shot 2019-03-22 at 11.22.47 AM

Components:

  • Compute – vSphere 6.7+ (vCenter Server + ESXi) and Enterprise Plus license
  • Storage – VSAN or other vSphere Datastores
  • Networking & Security – NSX-T 2.3
  • Openshift Container Platform 3.11 Enterprise
  • RHEL 7.6

Software Download:

Here is the complete list of software that needs to be downloaded to deploy Openshift Container Platform and NSX-T.

Software Download URL
NSX-T nsx-unified-appliance-2.3.0.0.0.10085405.ova (From 2.2 onwards, you can deploy NSX-T Controllers and Edges from the NSX-T Manager)
https://my.vmware.com/web/vmware/details?productId=673&downloadGroup=NSX-T-230
nsx-container-2.3.2.11695762.zip https://my.vmware.com/web/vmware/details?productId=673&downloadGroup=NSX-T-230#drivers_tools

Screen Shot 2019-03-22 at 11.01.44 AM

RHEL https://access.redhat.com/downloads/
The version I used: rhel-server-7.6-x86_64-dvd.iso

 

Ansible Hosts File

**Update on 21 June 2019: I notice you will need the hosts file as reference.

https://github.com/vincenthanjs/openshift-ansible-hosts/blob/master/hosts

Openshift Installation & NSX-T NCP Integration

  1. On every node, install docker.

    yum install docker-1.13.1

  2. On the master node or jumphost, run the pre-requisites playbook.

    ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

  3. On every node,

    docker load -i /root/nsx-container-2.3.2.11695762/Kubernetes/nsx-ncp-rhel-2.3.2.11695762.tar

    docker image tag registry.local/2.3.2.11695762/nsx-ncp-rhel nsx-ncp

    Screen Shot 2019-03-22 at 11.14.08 AM

    • On the master node or jumphost, run the deploy-cluster playbook.

      ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

      Screen Shot 2019-03-22 at 11.12.46 AM

Success!!!

Screen Shot 2019-03-22 at 11.26.48 AM

Validation

You can see the NCP and Node Agents being deployed as Pods under the nsx-system namespace.

oc get pod –all-namespaces

Screen Shot 2019-03-22 at 11.28.14 AM

If the NCP integration is successful, you should not see any error logs

oc logs nsx-ncp-279qf -n nsx-system | grep error

Screen Shot 2019-03-22 at 11.42.33 AM

You can also do that for the node agent. However, there are 2 containers in the node-agent pod, you will need to specify the container using -c.

oc logs nsx-node-agent-56f2s -c node-agent -n nsx-system | grep error

On the NSX-T side, if the integration is successful, you will see a bunch of default logical switches, logical routers as well as load balancer being created.

Screen Shot 2019-03-22 at 11.30.21 AM

You can also access the Openshift Container Platform web console using the https://ocp-master:8443. (You might need to add a host DNS entry)

Screen Shot 2019-03-22 at 11.32.39 AM

Demo App Test

Alright. Now lets test whether is the Container Network Interface(CNI) and NCP are working correctly by deploying a demo test application. I normally use the Yelb app for my demo and testing.

  • On the master node,

    oc new-project yelb
    git clone https://github.com/vincenthanjs/yelb-demo.git

    Screen Shot 2019-03-22 at 11.38.25 AM

     

    • <li style="padding-left: 30px;">
  You will need to add policy before you can deploy the Pods. If not you will error in deploying the containers.<br /> <blockquote>
    <p>
      oc adm policy add-scc-to-user anyuid -z default<br /> oc adm policy add-scc-to-user anyuid -z router<br /> oc adm policy add-scc-to-user anyuid -z builder<br /> oc adm policy add-scc-to-user anyuid -z deployer
    </p>
  </blockquote>

  <p>
    <a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.47.41-AM.png"><img class="alignnone size-large wp-image-1145" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.47.41-AM-1024x575.png" alt="Screen Shot 2019-03-22 at 11.47.41 AM" width="665" height="373" /></a></li> 

    <li style="padding-left: 30px;">
      Now, you can deploy the yelb app.<br /> <blockquote>
        <p>
          oc create -f yelb-app.yaml
        </p>
      </blockquote>

      <p>
        <a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.49.33-AM.png"><img class="alignnone size-large wp-image-1146" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.49.33-AM-1024x618.png" alt="Screen Shot 2019-03-22 at 11.49.33 AM" width="665" height="401" /></a></li> 

        <li style="padding-left: 30px;">
          Watch the containers creating.<br /> <blockquote>
            <p>
              watch oc get pod
            </p>
          </blockquote>

          <p>
            <a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.52.22-AM.png"><img class="alignnone size-large wp-image-1147" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.52.22-AM-1024x158.png" alt="Screen Shot 2019-03-22 at 11.52.22 AM" width="665" height="103" /><br /> </a><a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.53.31-AM.png"><img class="alignnone size-large wp-image-1148" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.53.31-AM-1024x146.png" alt="Screen Shot 2019-03-22 at 11.53.31 AM" width="665" height="95" /></a><a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.52.22-AM.png"><br /> </a></li> 

            <li style="padding-left: 30px;">
              NSX-T Load Balancer supports Ingress service type.<br /> <blockquote>
                <p>
                  oc get all
                </p>
              </blockquote>

              <p>
                <a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.54.08-AM.png"><img class="alignnone size-large wp-image-1149" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.54.08-AM-1024x575.png" alt="Screen Shot 2019-03-22 at 11.54.08 AM" width="665" height="373" /></a></li> 

                <li style="padding-left: 30px;">
                  Previously I already had a wildcard domain already pointed to the Openshift Load Balancer virtual IP.<a href="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.58.12-AM.png"><img class="alignnone size-large wp-image-1150" src="http://blog.acepod.com/wp-content/uploads/2019/03/Screen-Shot-2019-03-22-at-11.58.12-AM-1024x576.png" alt="Screen Shot 2019-03-22 at 11.58.12 AM" width="665" height="374" /></a>
                </li></ol> 

                <p style="padding-left: 30px;">
                  You can watch the full Openshift Container Platform installation and integration with NSX-T NCP over here.<br /> <iframe src="https://www.youtube.com/embed/5ZlggXKXwL8" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
                </p>