Open in app

Sign in

Write

Sign in

Kubernetes for Windows How to

Aaron Hsieh
8 min readNov 29, 2020

Reminder: This article contains Kubernetes term and step. Due to I have been in the Kubernetes world for a long time. So, I might miss some information describing. Please leave your message, I could update this article.

Follow by the Kubernetes for Windows experiences. I would like to share Kubernetes for Windows How to. This article will contains the following topics.

  • Structure
  • Installation
  • Verification
  • Experiences of How to

Structure

Kubernetes for Windows contains the follow nodes：

  • At least a Control Plane node.
  • At least a Linux node.
  • At least a Windows node.

Maybe you just need Windows Container only, but we still need Linux OS. There are a lot of Kubernetes components in the Control Plane node. Additionally, the one of magical gMSA service has to be in the Linux node. Therefore, there is no pure Windows infra any more. Even Microsoft, they are keeping embracing Linux (open source). That is why I use Kubernetes for Windows wording, not Windows Kubernetes because of this wording could present more real meaning.

Regarding Kubernetes components introduction, you could refer to the official article (reference 1). In this article, I would like to focus on How the Windows node working.

To ensure Windows node working, Windows node requires the kubelet, kube-proxy components and a network add-on (reference 2).

kubelet. kubelet is running as Windows service in the each Windows node. People could leverage the following PowerShell command to get the status of kubelet service. According to official article, The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy.

get-service kubelet

kube-proxy. In the version 1.14, kube-proxy is running as Windows service in each Windows node such as kubelet. Since 1.16, kube-proxy is runing as Pod. It is more easy to implement it now. That is to say, once kubelet service got problem, kube-proxy won’t work.

In the official article. kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster

Network add-on. kube-proxy maintains the network rules. However, we still need network add-on to implement cross node pod. Currently, I use flannel network add-on for on-premise cluster. For the cloud cluster, I would suggest leverage Cloud provider solution.

Installation

For the Windows node installation, there are two topics.

  • Cloud provider Kubernetes for Windows, e.g. EKS
  • On-premise Kubernetes for Windows

Cloud provider Kubernetes for Windows

Although I have KOps Cluster (a AWS Kubernetes solution), but I only implemented Windows node in EKS. Once I tried KOps for windows. I will publish another How to article.

Back to the EKS for Windows, I would say it is more easy then on-premise cluser. I strongly suggest to leverage eksctl (EKS cli) to create Windows nodegroup. In the past, I tried CloudFormation in version 1.14. I would say that is nightmare. Additionally, if you will use gMSA, that will be more difficult. You will need to implement auto-join AD domain for Windows node AutoScalling. Regarding gMSA, I will introduce it in “gMSA for your Windows Container”. For the Windows node installation, the steps as the following.

  • prepare an bastion EC2 with EKS permission (attched IAM role).
  • use eksctl to create EKS cluster and Linux node.
  • ensure EKS and Linux node ready.
  • use eksctl to create Windows NodeGroup
eksctl create nodegroup \
       --cluster=$CLUST_NAME \
       --name $NODE_NAME \
       --node-ami-family WindowsServer2019CoreContainer \
       --node-type=t3.large \
       --nodes-min=1 \
       --nodes-max=1 \
       --node-private-networking \
       --ssh-public-key=$SSHKEY \
       --node-volume-size=100 \
       --asg-access \
       --external-dns-access \
       --full-ecr-access \
       --alb-ingress-accesseksctl utils install-vpc-controllers --name=$CLUST_NAME --approve
  • update kubeconfig on bastion EC2
aws eks  — region $YOUR_REGION update-kubeconfig  — name $CLUST_NAME
  • After around 10 minutes, you can perform the following command line to check Windows node
kubectl get nodes

On-premise Kubernetes for Windows

On-premise Cluster installation is more difficult then Cloud Cluster. If you need to build multiple Control Plan, that will be more difficult. Futhermore, If your cluster behind of proxy, that will be more and more difficult for implementation. Regarding Kubernetes installation, I will publish it in the future. For now, I list the following check items for Control Plane and Linux node.

Docker service

  • OS: CentsOS 7 or 8
  • install yum-utils, device-mapper-persistent-data, lvm2
  • download and add yum docker repo
  • install docker-ce, docker-ce-cli, containerd.io
  • start docker service

Control Plane (google search the following action, or reference 3)

  • OS: CentOS 7 or 8
  • install nfs-utils, wget
  • disable firewalld
  • disable swap from /etc/fstab
  • download official yum repo for kubctl, kubeadm, kubelet
  • yum install kubectl, kubeadm, kubelet
  • enable kubelet service
  • disable SELinux
  • change docker cgroup to systemd in /etc/docker/daemon.json
  • change k8s cgroup to systemd in /etc/default/kubelet
  • restart docker service
  • change iptables & disable swappiness
  • reload sysctl
  • start/restart kubelet service
  • pre-pull Kubernetes component docker image
  • download kube-flannel.yml, kube-flannel-overlay, kube-proxy-win (reference 4)
  • reboot server
  • kubeadm init. Please follow the below parameters because above YAML files.
kubeadm init --pod-network-cidr=172.31.0.0/16 --service-cidr=172.30.0.0/16 --kubernetes-version=1.18.6
  • install flannel (reference 4)
kubectl apply -f kube-flannel.yml

Linux node. Once Control Plan ready, you could proceed install Linux node.

  • OS should be same as Control Plane
  • install nfs-utils
  • disable firewalld
  • disable swap from /etc/fstab
  • download official yum repo for kubctl, kubeadm, kubelet
  • yum install kubectl, kubeadm, kubelet
  • enable kubelet service
  • disable SELinux
  • change docker cgroup to systemd in /etc/docker/daemon.json
  • change k8s cgroup to systemd in /etc/default/kubelet
  • restart docker service
  • change up tables & disable swappiness
  • reload sysctl
  • start/restart kubelet service
  • reboot server
  • generate kubeadm join command on Control Plane node
kubeadm token create --print-join-command
  • join Kubernetes Cluster. leverage above output of join-command. For example.
kubeadm join {API SERVER} --token XXXXXXXXXXXXXXXX --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxx

Docker for Windows. Once Control Plane and Linux node ready. We could install Windows node now.

  • disable windows firewall by PowerShell
  • install Docker Msft Module
  • install Docker
  • reboot
  • enable docker service and reboot again

Windows node. I strongly recommend to use Windows Service Core.

  • OS: Windows 1809 build or greater
  • disable firewall firewall by PowerShell
netsh firewall set opmode mode=disable
  • pre-download container image
docker pull sigwindowstools/kube-proxy:v1.18.6
docker pull sigwindowstools/flannel:0.12.0
docker pull mcr.microsoft.com/windows/nanoserver:1809
docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest
docker run microsoft/nanoserver:latest
docker pull mcr.microsoft.com/k8s/core/pause:1.2.0
  • download and apply kube-flannel-overlay.yml and kube-proxy-win.yml YAML files (reference 4).
kubectl apply -f kube-flannel-overlay.yml
kubectl apply -f kube-proxy-win.yml
  • install PowerShell activedirectory module (for gMSA. You can skip it, if you don’t need gMSA).
Install-WindowsFeature RSAT-AD-PowerShell
  • Install Powershell CredentialSpec module (for gMSA. You can skip it, if you don’t need gMSA).
Install-Module -Name CredentialSpec
  • change NIC Ethernet0 card name to be Ethernet. Please insert your NIC name instead of Ethernet0.
Rename-NetAdapter "Ethernet0" -NewName Ethernet -ErrorAction SilentlyContinue
  • disable reset of NIC. please don’t disable your connection NIC.
Get-NetAdapter | ? {$_.name -ne "Ethernet"} | Disable-NetAdapter
  • create c:\k directory (practice since 1.14)
mkdir c:\k
  • generate kubeadm join command on Control Plane node
kubeadm token create --print-join-command
  • Please keep the following command line. You might need it when Windows node reboot.
powershell -command "if((docker network ls) -match 'host                nat'){$null}else{docker network create -d nat host}"
  • perform the following command line to check Windows node.
kubectl get nodes

Verification

Before verify your Windows node, you have to install kubectl in your Computer or bastion server. Furthermore, you have to copy content of kubeconfig to your computer ~/.kube/config. I recommend people to use Kubernetes Lenapp. It is really easy to maintain your Clusters or Namespaces. It supports Windows Desktop, MacOS, Linux Desktop.

Once Windows node ready, it is time to deploy a simple Web Pod.

kubectl apply -f aspnet.yaml
  • Verify Pod status. You can get pod to discover pod name, and then descirbe pod detail. Due to Windows Pod size usually is over 3GB. Therefore, you can see that Cluster is pulling image mcr.microsoft.com/dotnet/framework/aspnet:4.8
# get pods
kubectl get pods# output
NAME                     READY   STATUS              RESTARTS   AGE
aspnet-56887957d-4m5xg   0/1     ContainerCreating   0          10m# describe pods
kubectl describe pods aspnet-56887957d-4m5xg# output
Name:           aspnet-56887957d-4m5xg
Namespace:      default
Priority:       0
Node:           <Windows NODE NAME>
Start Time:     Sat, 28 Nov 2020 11:43:56 +0800
Labels:         app=aspnet
                pod-template-hash=56887957d
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/aspnet-56887957d
Containers:
  aspnet:
    Container ID:   
    Image:          mcr.microsoft.com/dotnet/framework/aspnet:4.8
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-72dc8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-72dc8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-72dc8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  beta.kubernetes.io/os=windows
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                     Message
  ----    ------     ----       ----                     -------
  Normal  Scheduled  <unknown>  default-scheduler        Successfully assigned default/aspnet-56887957d-4m5xg to xx-xxx-xxxxxx
  Normal  Pulling    10m        kubelet, xx-xxx-xxxxxx  Pulling image "mcr.microsoft.com/dotnet/framework/aspnet:4.8"
  • After 10 minutes (depends on your network), you can get pods again to check status. Now, you can see aspnet-XXXXX-XXXX pods is running.
# get pods
kubectl get pods# output
NAME                     READY   STATUS    RESTARTS   AGE
aspnet-56887957d-4m5xg   1/1     Running   0          15m
  • perform kubectl to get port number of aspnet service for current verification. As the following result, you get a tcp/31114 number. In the real service, you have to plan network first. Regarding Kubernetes network, I will share it in another article.
# get servicekubectl get service# outputNAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
aspnet       NodePort    172.30.2.14   <none>        80:31114/TCP   19m
  • Open your borwser and type http://Node_IP:31114 to get the following web page. For the real web service, that is another story.

Experience of How to

When version 1.14, it was really difficult to implement Windows node. Now, we can easily add Windows node by kubeadm join such as Linux node. I believe there is a lot of people effort.

Once your Windows node ready. The next challenge will be Windows application deployment and centralize Log. Regarding Windows Application How to, I will share it in Kubernetes for Windows Application.

Maybe someone might ask that Do we really need Windows node? I think this question should be answered by you. For me, I have hybrid mode because of Windows node still have something good, e.g. gMSA. For now, I just shared what I learned and what I experienced.

Reference

Kubernet
Kubernetes For Windows
Windows Containers

--

--

Aaron Hsieh

Written by Aaron Hsieh

25 Followers

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams