Kubernetes for Windows 實做

  • 架構
  • 安裝
  • 驗證
  • 實做心得

架構

  • 至少一台的 Control Plane node。
  • 至少一台的 Linux node。
  • 至少一台的 Windows node。
get-service kubelet

安裝

  • 雲端供應商 Kubernetes for Windows
  • 地端 Kubernetes for Windows

雲端供應商 Kubernetes for Windows

  • 準備一台有 EKS 服務權限的跳板機,這需要 attached role 給一台 EC2。
  • 透過 eksctl 建立 EKS Cluster 與至少一台的 Linux node。
  • 確認 EKS 與 Linux node ready
  • 透過 eksctl 建立 Windows NodeGroup
eksctl create nodegroup \
--cluster=$CLUST_NAME \
--name $NODE_NAME \
--node-ami-family WindowsServer2019CoreContainer \
--node-type=t3.large \
--nodes-min=1 \
--nodes-max=1 \
--node-private-networking \
--ssh-public-key=$SSHKEY \
--node-volume-size=100 \
--asg-access \
--external-dns-access \
--full-ecr-access \
--alb-ingress-access
eksctl utils install-vpc-controllers --name=$CLUST_NAME --approve
  • 更新或新增跳板機上的 kubeconfig。
aws eks  — region $YOUR_REGION update-kubeconfig  — name $CLUST_NAME
  • 大約十分鐘後讀者就可以用下述 command line 檢查你的 Windows node
kubectl get nodes

地端 Kubernetes for Windows

  • OS: CentsOS 7 or 8
  • install yum-utils, device-mapper-persistent-data, lvm2
  • download and add yum docker repo
  • install docker-ce, docker-ce-cli, containerd.io
  • start docker service
  • OS: CentOS 7 or 8
  • install nfs-utils, wget
  • disable firewalld
  • disable swap from /etc/fstab
  • download official yum repo for kubctl, kubeadm, kubelet
  • yum install kubectl, kubeadm, kubelet
  • enable kubelet service
  • disable SELinux
  • change docker cgroup to systemd in /etc/docker/daemon.json
  • change k8s cgroup to systemd in /etc/default/kubelet
  • restart docker service
  • change up tables & disable swappiness
  • reload sysctl
  • start/restart kubelet service
  • pull Kubernetes component docker image
  • download kube-flannel.yml, kube-flannel-overlay, kube-proxy-win (你可以在 github 搜尋筆者公開的)
  • reboot server
  • kubeadm init,請參考下述的參數,因為這跟上述的 flannel 有關,建議採用筆者建議設定。
kubeadm init --pod-network-cidr=172.31.0.0/16 --service-cidr=172.30.0.0/16 --kubernetes-version=1.18.6
  • install flannel (從筆者的公開 github repo 下載的)
kubectl apply -f kube-flannel.yml
  • OS 建議與 Control Plane 相同
  • install nfs-utils
  • disable firewalld
  • disable swap from /etc/fstab
  • download official yum repo for kubctl, kubeadm, kubelet
  • yum install kubectl, kubeadm, kubelet
  • enable kubelet service
  • disable SELinux
  • change docker cgroup to systemd in /etc/docker/daemon.json
  • change k8s cgroup to systemd in /etc/default/kubelet
  • restart docker service
  • change up tables & disable swappiness
  • reload sysctl
  • start/restart kubelet service
  • reboot server
  • generate kubeadm join command on Control Plane node
kubeadm token create --print-join-command
  • join Kubernetes Cluster,用上述 join-command 產生的 command,範例如下:
kubeadm join {API SERVER} --token XXXXXXXXXXXXXXXX --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxx
  • disable windows firewall by PowerShell
  • install Docker Msft Module
  • install Docker
  • reboot
  • enable docker service and reboot again
  • OS: Windows 1809 build or greater
  • disable firewall firewall by PowerShell
netsh firewall set opmode mode=disable
  • 先下載需要的 container image
docker pull sigwindowstools/kube-proxy:v1.18.6
docker pull sigwindowstools/flannel:0.12.0
docker pull mcr.microsoft.com/windows/nanoserver:1809
docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest
docker run microsoft/nanoserver:latest
docker pull mcr.microsoft.com/k8s/core/pause:1.2.0
  • 下載與套用 kube-flannel-overlay.yml and kube-proxy-win.yml(可以從筆者的 GitHub repo 下載)
kubectl apply -f kube-flannel-overlay.yml
kubectl apply -f kube-proxy-win.yml
  • install PowerShell activedirectory modules (for gMSA,如果不需要可以跳過)
Install-WindowsFeature RSAT-AD-PowerShell
  • Install Powershell CredentialSpec module (for gMSA,如果不需要可以跳過)
Install-Module -Name CredentialSpec
  • change NIC Ethernet0 card name to be Ethernet,如果本來網卡是別的名稱可以調整 Ethernet0
Rename-NetAdapter "Ethernet0" -NewName Ethernet -ErrorAction SilentlyContinue
  • disable 第一張網卡的其他網卡,這步要很確定沒有 disable 到正在連線的網卡
Get-NetAdapter | ? {$_.name -ne "Ethernet"} | Disable-NetAdapter
  • create c:\k directory (從 1.14 版以來的習慣)
mkdir c:\k
  • generate kubeadm join command on Control Plane node
kubeadm token create --print-join-command
  • 請保留下面的 command line,在每次 1.18.6 版本 Windows node 重啟後都需要執行。
powershell -command "if((docker network ls) -match 'host                nat'){$null}else{docker network create -d nat host}"
  • 讀者就可以用下述 command line 檢查你的 Windows node
kubectl get nodes

驗證

kubectl apply -f aspnet.yaml
  • 透過 kubectl 檢查 Pod 的狀態,並可以用 get pods 看到的 pod name 檢查 Pod 狀態的細節。因為 Windows Pod 大小常常都是 3GB 以上,所以如下圖所示,Kubernetes 正在拉 mcr.microsoft.com/dotnet/framework/aspnet:4.8 image
# get pods
kubectl get pods
# output
NAME READY STATUS RESTARTS AGE
aspnet-56887957d-4m5xg 0/1 ContainerCreating 0 10m
# describe pods
kubectl describe pods aspnet-56887957d-4m5xg
# output
Name: aspnet-56887957d-4m5xg
Namespace: default
Priority: 0
Node: <Windows NODE NAME>
Start Time: Sat, 28 Nov 2020 11:43:56 +0800
Labels: app=aspnet
pod-template-hash=56887957d
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/aspnet-56887957d
Containers:
aspnet:
Container ID:
Image: mcr.microsoft.com/dotnet/framework/aspnet:4.8
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-72dc8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-72dc8:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-72dc8
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/aspnet-56887957d-4m5xg to xx-xxx-xxxxxx
Normal Pulling 10m kubelet, xx-xxx-xxxxxx Pulling image "mcr.microsoft.com/dotnet/framework/aspnet:4.8"
  • 過了十分鐘後 (取決你的網路下載速度)再檢查一次 Pod status,這次就可以看到 aspnet-56887957d-4m5xg pods 的狀態是 running。
# get pods
kubectl get pods
# output
NAME READY STATUS RESTARTS AGE
aspnet-56887957d-4m5xg 1/1 Running 0 15m
  • 得到 aspnet service 的 port number,如果要正式佈署就需要有完整的網路規劃,這邊先用 node port 的方式連線,如下指令的輸出你可以得到一個 tcp 31114 的 port number。
# get servicekubectl get service# outputNAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
aspnet NodePort 172.30.2.14 <none> 80:31114/TCP 19m
  • 接下來可以透過瀏覽器開啟 http://Node_IP:31114 得到下述熟悉的畫面,至於要怎麼充實 Web service 那又是另外的故事了。

實做心得

參考與註釋

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store