K8S集群搭建

# PART1. 规划

# 1.1 节点规划

IP 地址 主机名 角色 配置
192.168.1.56 longinus-master-1 Master 2C 4G 20G
192.168.1.57 longinus-node-1 Worker 4C 8G 20G
192.168.1.58 longinus-node-2 Worker 4C 8G 20G
192.168.1.59 longinus-node-3 Worker 4C 8G 20G

# 1.2 软件版本规划

软件包 版本 安装性质 获取渠道 贡献者
docker-ce 24.0.7 dub 阿里云 docker
docker-ce-cli 24.0.7 dub 阿里云 docker
containerd.io 1.6.26 dub 阿里云 docker
kubelet 1.28.2 dub 阿里云 kubernetes
kubeadm 1.28.2 dub 阿里云 kubernetes
kubectl 1.28.2 dub 阿里云 kubernetes
kube-apiserver 1.28.2 pod 阿里云 kubernetes
kube-controller-manager 1.28.2 pod 阿里云 kubernetes
kube-scheduler 1.28.2 pod 阿里云 kubernetes
kube-proxy 1.28.2 pod 阿里云 kubernetes
pause 3.9 pod 阿里云 kubernetes
coredns 1.10.1 pod 阿里云 coredns
etcd 3.5.9-0 pod 阿里云 coreos
flannel cniVersion 0.3.1 pod github coreos

# 1.3 安装规划

# 1.3.1 容器选择

各节点都安装 docker-cedocker-ce-clicontainerd.io , 使用 Containerd 作为容器运行时,和 kubelet 交互

Cgroup 管理器,k8s 默认是 systemd, 需要将 Containerd 的 Cgroup 管理器也修改为 systemd (默认是 cgroupfs)

# 1.3.2 CNI 网络插件

使用谷歌嫡系,CoreOS 提供的 flannel 作为网络插件

# 1.3.3 安装步骤

各节点都安装 docker-cedocker-ce-clicontainerd.io 软件包,启动 docker.socketdocker.servicecontainerd.service 服务

各节点都安装 kubeletkubeadmkubectl 软件包,都启动 kubelet.service 服务

longinus-master-1 节点使用 kubeadm 创建集群,且 longinus-master-1 节点作为 master 节点;集群创建后马上安装 flannel 建立私网

集群创建后,将 longinus-node-1longinus-node-2longinus-node-3 节点加入集群作为 worker 节点

运行一个比 helloworld 稍微复杂一些的 demo, 验证应用 pod 可以启动并提供服务

# PART2. 操作系统准备工作

# 2.1 ubuntu 前置检查

# 2.1.1 关闭 ufw

1
2
root@longinus-master-1:~# ufw status
Status: inactive

# 2.1.2 时钟同步

  • 调整时区
1
root@longinus-master-1:~# dpkg-reconfigure tzdata

Asia , 再选 Shanghai 即可

  • 确认时钟同步
1
2
3
4
5
6
7
8
root@longinus-master-1:~# timedatectl status
Local time: Tue 2024-01-16 09:44:38 CST
Universal time: Tue 2024-01-16 01:44:38 UTC
RTC time: Tue 2024-01-16 01:44:37
Time zone: Asia/Shanghai (CST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no

System clock synchronized: yes 即为开启时钟同步服务

# 2.1.3 配置 apt 主源

1
2
3
4
5
6
7
8
9
root@longinus-node-2:~# apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

# 2.1.4 加载 br_netfilter 模块

  • 加载 br_netfilter 模块
1
2
3
root@longinus-master-1:~# modprobe br_netfilter
root@longinus-master-1:~# echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
1
  • 检查
1
2
3
root@longinus-master-1:~# lsmod|grep br_netfilt
br_netfilter 32768 0
bridge 307200 1 br_netfilter

# 2.1.5 启用 IP 转发

  • 启用 IP 转发
1
2
root@longinus-master-1:~# echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
1
  • 检查
1
2
root@longinus-master-1:~# cat /proc/sys/net/ipv4/ip_forward
1

# 2.1.6 启用 overlay 模块

  • 启用
1
root@longinus-master-1:~# modprobe overlay
  • 设置模块在开机启动时加载
1
root@longinus-master-1:~# vim /etc/modules-load.d/modules.conf

在该文件中添加一行,其内容为 overlay

  • 检查
1
2
root@longinus-master-1:~# lsmod | grep overlay
overlay 151552 0

# 2.1.7 安装 ipvsadm 和 ipset

  • 安装
1
root@longinus-master-1:~# apt install -y ipset ipvsadm
  • 检查
1
2
3
4
5
6
7
8
root@longinus-master-1:~# dpkg -l ipset ipvsadm
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============-============-================================>
ii ipset 7.15-1build1 amd64 administration tool for kernel I>
ii ipvsadm 1:1.31-1build2 amd64 Linux Virtual Server support pro>

注意:此处的 ii 表示安装成功

# 2.2 关闭 swap

1
2
3
4
root@longinus-master-1:~# swapoff -a
root@longinus-master-1:~# vim /etc/fstab
root@longinus-master-1:~# swapon --show
root@longinus-master-1:~#

/etc/fstab 中注释掉带有 swap 的行即可

正确结果应为执行 swapon --show 命令后无任何返回结果

# 2.3 配置 docker 和 k8s 的 apt 源

# 2.3.1 配置 docker 的 apt 源

1
root@longinus-master-1:~# curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -

这一步会有个 Warning 级别的错误,正常现象

1
root@longinus-master-1:~# add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"

这一步需要按个回车

# 2.3.2 配置 kubernetes 的 apt 源

1
root@longinus-master-1:~# curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
1
root@longinus-master-1:~# add-apt-repository "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main"

# 2.4 绑定 host 表

1
2
root@longinus-master-1:~# vim /etc/hosts
root@longinus-master-1:~# cat /etc/hosts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
127.0.0.1 localhost
127.0.1.1 template-1

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

# master1节点公网IP
192.168.1.56 k8s-master01.longinus.com k8s-master01.longinus kubeapi.longinus.com

# worker1节点公网IP
192.168.1.57 k8s-node01.longinus.com k8s-node01.longinus

# worker2节点公网IP
192.168.1.58 k8s-nodeginus.com k8s-node02.longinus

# worker3节点公网IP
192.168.1.59 k8s-node03.longinus.com k8s-node03.longinus

# PART3. 安装 docker

# 3.1 安装 docker

  • 安装
1
root@longinus-master-1:~# apt install -y docker-ce docker-ce-cli containerd.io
  • 检查
1
2
3
4
5
6
7
8
9
10
root@longinus-master-1:~# apt list --installed | grep -i -E 'docker|containerd'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

containerd.io/jammy,now 1.6.27-1 amd64 [installed]
docker-buildx-plugin/jammy,now 0.11.2-1~ubuntu.22.04~jammy amd64 [installed,automatic]
docker-ce-cli/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed]
docker-ce-rootless-extras/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed,automatic]
docker-ce/jammy,now 5:24.0.7-1~ubuntu.22.04~jammy amd64 [installed]
docker-compose-plugin/jammy,now 2.21.0-1~ubuntu.22.04~jammy amd64 [installed,automatic]

这里有个警告没关系

# 3.2 启动 docker 相关服务

# 3.2.1 验证是否开机自启

1
2
3
4
root@longinus-master-1:~# systemctl list-unit-files | grep -E 'docker|containerd'
containerd.service enabled enabled
docker.service enabled enabled
docker.socket enabled enabled

# 3.2.2 验证服务是否启动

1
2
3
root@longinus-master-1:~# systemctl status containerd.service
root@longinus-master-1:~# systemctl status docker.service
root@longinus-master-1:~# systemctl status docker.socket

3 个服务应该都是 running 状态

# 3.2.3 查看 docker 版本

1
root@longinus-master-1:~# docker version

# 3.3 配置 containerd

# 3.3.1 备份原配置并使用默认配置替换配置文件

  • 备份原配置
1
root@longinus-master-1:~# cp /etc/containerd/config.toml /etc/containerd/config.toml.ori
  • 使用默认配置替换配置文件
1
root@longinus-master-1:~# containerd config default > /etc/containerd/config.toml

# 3.3.2 设置 Containerd 的 Cgroup 设为 systemd

1
root@longinus-master-1:~# vim /etc/containerd/config.toml

第 125 行:

原文: SystemdCgroup = false

修改为: SystemdCgroup = true

# 3.3.3 pause 镜像路径改为国内源

1
root@longinus-master-1:~# vim /etc/containerd/config.toml

第 61 行:

原文: sandbox_image = "registry.k8s.io/pause:3.6"

修改为: sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

# 3.3.4 重启 containerd 并查看服务状态

1
2
root@longinus-master-1:~# systemctl restart containerd.service
root@longinus-master-1:~# systemctl status containerd.service

# PART4. 安装 Kubernetes

# 4.1 安装 Kubernetes 软件包

1
root@longinus-master-1:~# apt install -y kubelet kubeadm kubectl

注:这一步如果遇到错误

1
2
3
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl

则重新执行 2.3.2 小节,然后执行 apt-get update 即可

# 4.2 确认 kubelet 服务状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
root@longinus-master-1:~# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor prese>
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-01-16>
Docs: https://kubernetes.io/docs/home/
Process: 7108 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_>
Main PID: 7108 (code=exited, status=1/FAILURE)
CPU: 61ms

Jan 16 11:02:01 longinus-master-1 systemd[1]: kubelet.service: Main process exi>
Jan 16 11:02:01 longinus-master-1 systemd[1]: kubelet.service: Failed with resu>
lines 1-12/12 (END)...skipping...
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2024-01-16 11:02:01 CST; 1s ago
Docs: https://kubernetes.io/docs/home/
Process: 7108 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
Main PID: 7108 (code=exited, status=1/FAILURE)
CPU: 61ms

Jan 16 11:02:01 longinus-master-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jan 16 11:02:01 longinus-master-1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

可以看到此时 kubelet 状态为 loaded. 这是正常的。因为 kubelet 服务成功启动的先决条件是需要 kubelet 的配置文件,所在目录 /var/lib/kubelet 还没有建立

查看日志:

1
root@longinus-master-1:~# journalctl -xeu kubelet

可以看到:

1
2
Jan 16 11:05:16 longinus-master-1 kubelet[7227]: E0116 11:05:16.565328    7227 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to>
Jan 16 11:05:16 longinus-master-1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE

# PART5. K8S master 建立单机集群

# 5.1 拉取所需镜像

  • 列出所需镜像
1
2
3
4
5
6
7
8
9
root@longinus-master-1:~# kubeadm config images list
I0116 11:07:18.093407 7309 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28
registry.k8s.io/kube-apiserver:v1.28.5
registry.k8s.io/kube-controller-manager:v1.28.5
registry.k8s.io/kube-scheduler:v1.28.5
registry.k8s.io/kube-proxy:v1.28.5
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.9-0
registry.k8s.io/coredns/coredns:v1.10.1
  • 拉取镜像
1
2
3
4
5
6
7
8
9
root@longinus-master-1:~# kubeadm config images pull
I0116 11:08:28.342502 7361 version.go:256] remote version is much newer: v1.29.0; falling back to: stable-1.28
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.28.5
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.28.5
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.28.5
[config/images] Pulled registry.k8s.io/kube-proxy:v1.28.5
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.9-0
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1

这里注意我们拉取镜像的版本,初始化集群时的版本要和镜像的版本相同

# 5.2 Kubernetes 初始化

# 5.2.1 初始化

1
2
3
4
5
6
7
8
root@longinus-master-1:~# kubeadm init \
--control-plane-endpoint="kubeapi.longinus.com" \
--kubernetes-version=v1.28.5 \
--pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--token-ttl=0 \
--cri-socket unix:///run/containerd/containerd.sock \
--upload-certs

# 5.2.2 选项解释

  • --image-repository : 指定要使用的镜像仓库,默认为 registry.k8s.io
  • --kubernetes-version :kubernetes 程序组件的版本号,它必须要与安装的 kubelet 程序包的版本号相同
  • --control-plane-endpoint : 控制平面的固定访问端点,可以是 IP 地址或 DNS 名称,会被用于集群管理员及集群组件的 kubeconfig 配置文件的 API Server 的访问地址;单控制平面部署时可以不使用该选项
  • --pod-network-cidr :Pod 网络的地址范围,其值为 CIDR 格式的网络地址,通常,Flannel 网络插件的默认为 10.244.0.0/16 ; Project Calico 插件的默认值为 192.168.0.0/16
  • --service-cidr :Service 的网络地址范围,其值为 CIDR 格式的网络地址,默认为 10.96.0.0/12 ; 通常,仅 Flannel 一类的网络插件需要手动指定该地址
  • -apiserver-advertise-address :apiserver 通告给其他组件的 IP 地址,一般应该为 Master 节点的用于集群内部通信的 IP 地址, 0.0.0.0 表示节点上所有可用地址
  • --token-ttl : 共享令牌 (token) 的过期时长,默认为 24 小时,0 表示永不过期;为防止不安全存储等原因导致的令牌泄露危及集群安全,建议为其设定过期时长.
    • 未设定该选项时,在 token 过期后,若期望再向集群中加入其它节点,可以使用如下命令重新创建 token, 并生成节点加入命令
    • kubeadm token create --print-join-command

# 5.2.3 后续操作

成功初始化后会提示后续操作:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

kubeadm join kubeapi.longinus.com:6443 --token yczsfl.oz53z6pdlzg6l8of \
--discovery-token-ca-cert-hash sha256:cad8ad0e1f0553a6989cc9a9a73ffee219b714fd84ed6c1ce68f768ff18b733f \
--control-plane --certificate-key 6351ad49af82922a578621bf4d40dc17abfb2c218d041d2905bc97067771143d

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kubeapi.longinus.com:6443 --token yczsfl.oz53z6pdlzg6l8of \
--discovery-token-ca-cert-hash sha256:cad8ad0e1f0553a6989cc9a9a73ffee219b714fd84ed6c1ce68f768ff18b733f

# 5.2.4 检查

检查 6443 端口 (kube API Server) 是否被监听

1
2
3
4
5
6
7
8
9
10
11
12
13
root@longinus-master-1:~# lsof -i :6443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
kube-cont 8170 root 8u IPv4 155311 0t0 TCP k8s-master01.longinus.com:58406->k8s-master01.longinus.com:6443 (ESTABLISHED)
kube-sche 8179 root 7u IPv4 156409 0t0 TCP k8s-master01.longinus.com:58396->k8s-master01.longinus.com:6443 (ESTABLISHED)
kube-sche 8179 root 8u IPv4 156571 0t0 TCP k8s-master01.longinus.com:37666->k8s-master01.longinus.com:6443 (ESTABLISHED)
kube-apis 8207 root 3u IPv6 155276 0t0 TCP *:6443 (LISTEN)
kube-apis 8207 root 68u IPv6 155385 0t0 TCP k8s-master01.longinus.com:6443->k8s-master01.longinus.com:37666 (ESTABLISHED)
kube-apis 8207 root 70u IPv6 155370 0t0 TCP k8s-master01.longinus.com:6443->k8s-master01.longinus.com:58396 (ESTABLISHED)
kube-apis 8207 root 71u IPv6 155371 0t0 TCP k8s-master01.longinus.com:6443->k8s-master01.longinus.com:58406 (ESTABLISHED)
kube-apis 8207 root 72u IPv6 156908 0t0 TCP k8s-master01.longinus.com:6443->k8s-master01.longinus.com:37680 (ESTABLISHED)
kube-apis 8207 root 85u IPv6 156562 0t0 TCP ip6-localhost:58480->ip6-localhost:6443 (ESTABLISHED)
kube-apis 8207 root 92u IPv6 156569 0t0 TCP ip6-localhost:6443->ip6-localhost:58480 (ESTABLISHED)
kubelet 8296 root 15u IPv4 156907 0t0 TCP k8s-master01.longinus.com:37680->k8s-master01.longinus.com:6443 (ESTABLISHED)

# 5.2.5 配置 kubelet

kubectl 是 kube-apiserver 的命令行客户端程序,实现了除系统部署之外的几乎全部的管理操作,是 kubernetes 管理员使用最多的命令之一.kubectl 需经由 API server 认证及授权后方能执行相应的管理操作,kubeadm 部署的集群为其生成了一个具有管理员权限的认证配置文件 /etc/kubernetes/admin.conf , 它可由 kubectl 通过默认的 $HOME/.kube/config 的路径进行加载。当然,用户也可在 kubectl 命令上使用 --kubeconfig 选项指定一个别的位置.

下面复制认证为 Kubernetes 系统管理员的配置文件至目标用户 (例如当前用户 root) 的家目录下:

1
2
3
4
root@longinus-master-1:~# mkdir ~/.kube
root@longinus-master-1:~# cp /etc/kubernetes/admin.conf ~/.kube/config
root@longinus-master-1:~# ll ~/.kube/config
-rw------- 1 root root 5652 Jan 16 11:22 /root/.kube/config

# 5.3 安装 CNI 网络插件

此处我们使用 Flannel 作为集群的网络插件

# 5.3.1 下载 Pod 的配置文件

1
root@longinus-master-1:~# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# 5.3.2 检查配置文件

配置文件中的 net-conf.json 段需要与 Kubernetes 初始化时的 pod-network-cidr 配置的网段一致:

1
root@longinus-master-1:~# vim kube-flannel.yml
1
2
3
4
5
6
7
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}

# 5.3.3 安装网络插件

1
2
3
4
5
6
7
root@longinus-master-1:~# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

# 5.3.4 查看集群网络状态

# 5.3.4.1 系统级 Pod 状态
1
2
3
4
5
6
7
8
9
10
root@longinus-master-1:~# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-2clkz 1/1 Running 0 44s
kube-system coredns-5dd5756b68-pqfhw 1/1 Running 0 12m
kube-system coredns-5dd5756b68-vjd7k 1/1 Running 0 12m
kube-system etcd-longinus-master-1 1/1 Running 0 13m
kube-system kube-apiserver-longinus-master-1 1/1 Running 0 13m
kube-system kube-controller-manager-longinus-master-1 1/1 Running 0 13m
kube-system kube-proxy-j4bkt 1/1 Running 0 12m
kube-system kube-scheduler-longinus-master-1 1/1 Running 0 13m
# 5.3.4.2 查看指定 namespace 中的 pod 状态
1
2
3
4
5
6
7
8
9
root@longinus-master-1:~# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5dd5756b68-pqfhw 1/1 Running 0 18m
coredns-5dd5756b68-vjd7k 1/1 Running 0 18m
etcd-longinus-master-1 1/1 Running 0 18m
kube-apiserver-longinus-master-1 1/1 Running 0 18m
kube-controller-manager-longinus-master-1 1/1 Running 0 18m
kube-proxy-j4bkt 1/1 Running 0 18m
kube-scheduler-longinus-master-1 1/1 Running 0 18m
# 5.3.4.3 查看主节点状态
1
2
3
4
5
6
root@longinus-master-1:~# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy ok

注意主节点应该不存在任何 ERROR , 组件状态都应为 Healthy

# 5.3.4.4 查看节点列表
1
2
3
root@longinus-master-1:~# kubectl get node
NAME STATUS ROLES AGE VERSION
longinus-master-1 Ready control-plane 19m v1.28.2

# PART6. 其他节点加入集群

# 6.1 创建一个 24 小时内有效的 token

1
2
root@longinus-master-1:~# kubeadm token create --ttl 24h --print-join-command
kubeadm join kubeapi.longinus.com:6443 --token u7xxyg.nxampvy3m22ifuf5 --discovery-token-ca-cert-hash sha256:cad8ad0e1f0553a6989cc9a9a73ffee219b714fd84ed6c1ce68f768ff18b733f

其中:

  • --ttl : 指定 token 的有效期为 24 小时
  • --print-join-command : 打印加入集群的命令

# 6.2 worker 节点加入

longinus-node-1longinus-node-2longinus-node-3 上执行如下命令:

1
root@longinus-node-1:~# kubeadm join kubeapi.longinus.com:6443 --token u7xxyg.nxampvy3m22ifuf5 --discovery-token-ca-cert-hash sha256:cad8ad0e1f0553a6989cc9a9a73ffee219b714fd84ed6c1ce68f768ff18b733f

执行结果如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

注:若需加入 master 节点,执行如下命令即可:

1
kubeadm join kubeapi.longinus.com:6443 --token u7xxyg.nxampvy3m22ifuf5 --discovery-token-ca-cert-hash sha256:cad8ad0e1f0553a6989cc9a9a73ffee219b714fd84ed6c1ce68f768ff18b733f --control-plane

# 6.3 查看节点列表

1
2
3
4
5
6
root@longinus-master-1:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
longinus-master-1 Ready control-plane 38m v1.28.2
longinus-node-1 Ready <none> 11m v1.28.2
longinus-node-2 Ready <none> 10m v1.28.2
longinus-node-3 Ready <none> 10m v1.28.2

# PART7. 应用 Demo

# 7.1 配置容器客户端

crictl 是遵循 CRI 接口规范的一个命令行工具,随 containerd.io 一起安装,常用它来检查和管理 kubelet 节点上的容器运行时和镜像

crictl 下载镜像时使用的默认端点 (endpoint) 是 /var/run/dockershim.sock , 这个已经在 k8s v1.24 起被废弃,需要我们手工指定为新的端点 containerd.sock

重新指定的工作需要通过配置 /etc/crictl.yaml 来完成:

  • 配置 crictl
1
2
root@longinus-master-1:~# vim /etc/crictl.yaml
root@longinus-master-1:~# cat /etc/crictl.yaml
1
2
3
4
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
  • 检查
1
2
3
4
5
6
7
8
9
10
root@longinus-master-1:~# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
cf7dfb1a959a2 ead0a4a53df89 3 hours ago Running coredns 0 aec2d9225b4af coredns-5dd5756b68-pqfhw
7aaaec2646b00 ead0a4a53df89 3 hours ago Running coredns 0 f1c89af71f1b7 coredns-5dd5756b68-vjd7k
dd4e3e673f68c 0dc86fe0f22e6 3 hours ago Running kube-flannel 0 cd03aed7dea12 kube-flannel-ds-2clkz
c7610d72422f0 01cf8d1d322dd 4 hours ago Running kube-proxy 0 e02d52661f316 kube-proxy-j4bkt
143a5a9e78c5a 9ecc4287300e3 4 hours ago Running kube-apiserver 0 cdcaf8463c402 kube-apiserver-longinus-master-1
2cc97f592243e c527ad14e0cd5 4 hours ago Running kube-controller-manager 0 f90326d0098d0 kube-controller-manager-longinus-master-1
79ec5df88b210 73deb9a3f7025 4 hours ago Running etcd 0 d2709cd35d815 etcd-longinus-master-1
092fccb8a6236 babc03668f18a 4 hours ago Running kube-scheduler 0 02159eb6be4d3 kube-scheduler-longinus-master-1

# 7.2 配置简单应用

这里我们尝试使用 crictl 命令拉取一个 nginx 镜像,之后直接使用 kubectlnginx 部署为 pod , 而后改为部署为 3 副本的 pod , 并暴露给集群外

# 7.2.1 拉取镜像

  • 拉取镜像
1
2
root@longinus-master-1:~# crictl pull docker.io/nginx
Image is up to date for sha256:a8758716bb6aa4d90071160d27028fe4eaee7ce8166221a97d30440c8eac2be6
  • 检查
1
2
root@longinus-master-1:~# crictl images|grep nginx
docker.io/library/nginx latest a8758716bb6aa 70.5MB

# 7.2.2 编排 Pod

  • 部署
1
2
root@longinus-master-1:~# kubectl create deployment nginx --image=docker.io/library/nginx:latest
deployment.apps/nginx created
  • 检查
1
2
3
root@longinus-master-1:~# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-75446c786c-9b9nd 1/1 Running 0 50s 10.244.1.2 longinus-node-1 <none> <none>
  • 调整 Pod 的副本数量为 3
1
2
root@longinus-master-1:~# kubectl scale --replicas=3 deployment/nginx
deployment.apps/nginx scaled
  • 检查调整后结果
1
2
3
4
5
root@longinus-master-1:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-75446c786c-6dnxl 1/1 Running 0 86s 10.244.3.2 longinus-node-3 <none> <none>
nginx-75446c786c-9b9nd 1/1 Running 0 3m25s 10.244.1.2 longinus-node-1 <none> <none>
nginx-75446c786c-n5t9k 1/1 Running 0 86s 10.244.2.2 longinus-node-2 <none> <none>
  • 将 nginx 的东西向 80 端口,调整为南北向对外暴露的服务
1
2
root@longinus-master-1:~# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed
  • 检查 service
1
2
3
4
root@longinus-master-1:~# kubectl get service -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h1m <none>
nginx NodePort 10.101.152.202 <none> 80:30788/TCP 25s app=nginx

此时可以访问 3 个 worker 节点的 30788 端口

# 7.2.3 尝试访问应用

# 7.2.3.1 访问 Pod 的 IP (东西向)

在任何一个节点上访问 3 个 pod 的 IP:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@longinus-node-1:~/.kube# curl 10.244.1.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@longinus-node-1:~/.kube# curl 10.244.2.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
root@longinus-node-1:~/.kube# curl 10.244.3.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
# 7.2.3.2 访问 Service (南北向)

访问 http://192.168.1.57:30788/http://192.168.1.58:30788/http://192.168.1.59:30788/ 均可看到 Nginx 的首页

# PART8. 备注

若想在 worker 节点上也使用 kubectl 命令操作集群,需要把控制平面节点上的 ~/.kube/config ,scp 到 worker 节点上,确保该文件的权限为 0600 即可.