博客

kubernetes（k8s）部署

  • Post author:
  • Post category:技术
  • Post comments:5评论

2020年5-6月份的时候部署了2次，留下的操作记录。最近又弄了一次，改动一部分，但记录一下吧。（随着时间迁移可能有一些错误，别干照着干，琢磨琢磨）

0.IP划分

1 LEDE
10 k8s-master01
20 k8s-worker01
21 k8s-worker02
100 Harbor

1.先安装LEDE并安装v2ray插件

2. 3台分别安装centos7，Harbor也需要centos7

3.全部修改host文件

192.168.66.10 k8s-master01
192.168.66.20 k8s-worker01
192.168.66.21 k8s-worker02
192.168.66.100 hub.chongaibao.com

4.安装依赖包（如果想提速，可以改yum地址）

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget net-tools git

5.处理防火墙

systemctl stop firewalld && systemctl disable firewalld
yum install -y iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save

6.关闭虚拟内存

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

7.关闭selinux

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

8.调整内核参数，优化K8S

modprobe br_netfilter
cat > /etc/sysctl.d/kubernetes.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
sysctl -p /etc/sysctl.d/kubernetes.conf

9.调整时区

timedatectl set-timezone Asia/Shanghai
timedatectl set-local-rtc 0
systemctl restart rsyslog
systemctl restart crond

10.关闭postfix

systemctl stop postfix && systemctl disable postfix

11.设置日志方式

mkdir /var/log/journal # 持久化保存日志的目录
mkdir /etc/systemd/journald.conf.d
cat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF
[Journal]
# 持久化保存到磁盘
Storage=persistent

# 压缩历史日志
Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

# 最大占用空间 10
GSystemMaxUse=10G

# 单日志文件最大 200M
SystemMaxFileSize=200M

# 日志保存时间 2 周
MaxRetentionSec=2week

# 不将日志转发到 
syslogForwardToSyslog=no
EOF
systemctl restart systemd-journald

12.升级系统内核版本为4.44

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install -y kernel-lt
grub2-set-default "CentOS Linux (4.4.182-1.el7.elrepo.x86_64) 7 (Core)"
reboot

13.kube-proxy开始ipvs的前置条件

modprobe br_netfilter
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

14.安装Docker

yum install -y yum-utils device-mapper-persistent-data lvm2

导入阿里云源

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

yum update -y && yum install -y docker-ce

重新修改一次引导顺序

grub2-set-default "CentOS Linux (4.4.182-1.el7.elrepo.x86_64) 7 (Core)" && reboot

systemctl start docker
systemctl enable docker

创建 /etc/docker 目录

mkdir /etc/docker

配置 daemon

cat > /etc/docker/daemon.json <<EOF
{
   "exec-opts": ["native.cgroupdriver=systemd"],
   "log-driver": "json-file",
   "log-opts": {
      "max-size": "100m"  
   }
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

重启docker服务

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

15.安装Kubeadmin（主从配置）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3
systemctl enable kubelet.service

拉取镜像（这步需要科学上网，如果失败可以多试几次，当然改源什么的办法也行，手动导入也行）

kubeadm config images pull

16.初始化主节点（主节点！！！！podSubnet需要自己添加（是为flannel做准备），---和后面也需要手动添加）

kubeadm config print init-defaults > kubeadm-config.yaml

编辑kubeadm-config.yaml，照着下面改

localAPIEndpoint:
    advertiseAddress: 192.168.66.10    # 修改内容
kubernetesVersion: v1.18.3    # 修改内容
imageRepository: registry.aliyuncs.com/google_containers    # 20210125 GGan添加，如果不能翻墙需要修改内容
networking:
  podSubnet: "10.244.0.0/16"    # 添加内容
  serviceSubnet: 10.96.0.0/12
---    # 下面全是添加内容
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs
kubeadm init --config=kubeadm-config.yaml --upload-certs | tee kubeadm-init.log

安照提示

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

17.整理安装信息（主节点！！！！)

mkdir install-k8s/core -p
mv kubeadm-config.yaml kubeadm-init.log install-k8s/core

18.安装flannel（主节点！！！！)

cd install-k8s
mkdir plugin
cd plugin
mkdir flannel
cd flannel
wget  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml    # 20210125 GGan添加，如果服务器不能翻墙需要 自己准备kube-flannel.yml上传到服务器
kubectl create -f kube-flannel.yml

19.将worker加进来，命令在kubeadm-init.log里的结尾（worker节点！！！！)

kubeadm join 192.168.66.10:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ae231287834356294e4a88e9eba1e286b9370a8d73dbd04e0c2014513c75659c

这步集群的就算搭建完了

19.5. k8s命令自动补全，20210127添加

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc

20.安装harbor

编辑/etc/docker/daemon.json，添加信息如下：（master+worker+harbor）

"insecure-registries": ["https://hub.chongaibao.com"]
systemctl restart docker

生成证书

mkdir /data/cert -p
cd /data/cert

openssl genrsa -des3 -out server.key 2048 （设置密码）
openssl req -new -key server.key -out server.csr
# 输入密码
# CN
# BJ
# BJ
# chongaiwangguo
# chongaiwangguo
# hub.chongaibao.com
# hub@chongaibao.com
# 回车
# 回车

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
# 输入密码

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
chmod a+x *

安装docker-compose（harbor节点）
上传docker-compose文件

mv docker-compose /usr/local/bin
chmod a+x /usr/local/bin/docker-compose

安装harbor（harbor节点）
上传harbor-offline-installer-v***.tgz文件(我用的2.0.0)

tar zxvf harbor-offline-installer-v***.tgz
cd harbor

编辑harbor.yml.tmpl

hostname: hub.chongaibao.com
https:
  certificate: /data/cert/server.crt
  private_key: /data/cert/server.key
mv harbor.yml.tmpl harbor.yml
./install.sh

# 测试Harbor，随便找台worker
docker login https://hub.chongaibao.com
# admin
# Harbor12345（配置文件里有）
docker pull nginx
docker tag nginx:latest hub.chongaibao.com/library/mytest:v1
docker push hub.chongaibao.com/library/mytest:v1
# 找到worker里这个镜像，删掉
 docker images
 docker rmi -f 【IMAGE ID 】

# 测试集群，到master上
kubectl run nginx-deployment --image=hub.chongaibao.com/library/mytest:v1 --port=80 --replicas=1
kubectl get deployment

21.安装Ingress（主节点）

cd /root/install-k8s/plugin
mkdir ingress
cd ingress
# 在此网址获取部署文件：https://kubernetes.github.io/ingress-nginx/deploy/
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml
kubectl apply -f deploy.yaml

22. Helm（类似k8s的yum）部署（master节点）

cd install-k8s/
mkdir helm
cd helm/
wget https://storage.googleapis.com/kubernetes-helm/helm-v2.15.2-linux-amd64.tar.gz
tar zxvf helm-v2.15.2-linux-amd64.tar.gz
cp -a linux-amd64/helm /usr/local/bin/
chmod a+x /usr/local/bin/helm

创建tiller的yaml文件（rbac-config.yaml）

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

通过yaml文件创建 tiller

kubectl create -f rbac-config.yaml

helm init --service-account tiller

验证，看一下helm和tiller是否正常

helm version

23.部署kubernetes-dashboard

cd /root/install-k8s/plugin
mkdir dashboard
cd dashboard/

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
kubectl apply -f recommended.yaml

编辑dashboard-adminuser.yaml

# Creating a Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
---
# Creating a ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard
kubectl apply -f dashboard-adminuser.yaml

kubectl edit svc kubernetes-dashboard  -n kubernetes-dashboard
将spec.type改为 NodePort
kubectl get svc -n kubernetes-dashboard #获取端口

# 获取token
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

登录web端,chrome可能不行，需要用火狐
ip与端口在svc中获取，token方式登录

24.安装普罗米修斯（监控）（这个后两次我都没装）

cd /root/install-k8s/plugin
mkdir prometheus
cd prometheus/
git clone https://github.com/coreos/kube-prometheus.git
cd kube-prometheus/manifests/

修改grafana-service.yaml，使用nodeport方式访问，内容如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  type: NodePort    # 添加内容
  ports:
  - name: http
    port: 3000
    targetPort: http
    nodePort: 30100    # 添加内容
  selector:
    app: grafana

修改prometheus-service.yaml，改为nodeport方式访问，内容如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  type: NodePort    # 添加内容
  ports:
  - name: web
    port: 9090
    targetPort: web
    nodePort: 30200    # 添加内容
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP

alertmanager-service.yaml，改为nodeport方式访问，内容如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    alertmanager: main
  name: alertmanager-main
  namespace: monitoring
spec:
  type: NodePort    # 添加内容
  ports:
  - name: web
    port: 9093
    targetPort: web
    nodePort: 30300    # 添加内容
  selector:
    alertmanager: main
    app: alertmanager
  sessionAffinity: ClientIP
kubectl apply -f setup/
kubectl apply -f ../manifests/

其它记录

重置命令

kubeadm reset  && systemctl daemon-reload && systemctl restart kubelet

升级命令

kubeadm upgrade apply v1.18.3

helm彻底删除命令

helm delete demo --purge
0 0 votes
文章评分
阅读量： 77
订阅
提醒
guest
5 评论
最旧
最新 得票最多
甘甘
作者
甘甘
2021年2月4日 10:31

补充一下，时间同步策略我用的chrony

0
回复
甘甘
作者
甘甘
2021年4月7日 17:04

今天换了环境装Dashboard，操作略有不同，记录一下

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"
0
回复
甘甘
作者
甘甘
2021年6月18日 11:02

使用containerd

yum list | grep containerd 

yum install containerd.io-1.4.4-3.1.el7 -y

mkdir -p /etc/containerd 
containerd config default > /etc/containerd/config.toml 

# 替换配置文件 
sed -i '/containerd.runtimes.runc.options/a\ \ \ \ \ \ \ \ \ \ \ \ SystemdCgroup = true' /etc/containerd/config.toml 
#sed -i "s#k8s.gcr.io#registry.cn-hangzhou.aliyuncs.com/google_containers#g"  /etc/containerd/config.toml 
#sed -i "s#https://registry-1.docker.io#https://registry.cn-hangzhou.aliyuncs.com#g"  /etc/containerd/config.toml

systemctl enable containerd 
systemctl restart containerd 

vim kubeadm-config.yaml
  nodeRegistration:
    criSocket: /run/containerd/containerd.sock
Last edited 5 年 之前 by 甘甘
0
回复
甘甘
作者
甘甘
2022年3月16日 10:35

封锁节点，让节点变为不可调度

kubectl cordon <node name>

驱逐pod

kubectl drain <node name>

解除封锁

kubectl uncordon <node name>

删除节点

kubectl delete node <node name>

0
回复
甘甘
作者
甘甘
2022年5月17日 15:11

修改默认命名空间：

kubectl config set-context $(kubectl config current-context) –namespace=testns

0
回复
wpdiscuz   wpDiscuz