lihuu's blog
文章 标签 关于 分类
lihuu's blog

ETCD安全高可用集群部署测试

以下是个人搭建etcd集群中的记录，首先使用这篇文章中介绍的脚本创建三个虚拟机，ubuntu-01、

ubuntu-02、ubuntu-03，在三台虚拟机中分别安装 etcd

sudo apt install etcd

安装完成之后打开 /lib/systemd/system/etcd.service 这个service文件

[Unit]
Description=etcd - highly-available key value store
Documentation=https://etcd.io/docs
Documentation=man:etcd
After=network.target
Wants=network-online.target

[Service]
Environment=DAEMON_ARGS=
Environment=ETCD_NAME=%H
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/default/%p
Type=notify
User=etcd
PermissionsStartOnly=true
#ExecStart=/bin/sh -c "GOMAXPROCS=$(nproc) /usr/bin/etcd $DAEMON_ARGS"
ExecStart=/usr/bin/etcd $DAEMON_ARGS
Restart=on-abnormal
#RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd2.service

这个是etcd的service文件，可以看到etcd的配置文件在/etc/defalut/目录中，名称应该是etcd，这个和传统的不一致，我们需要在/etc/etcd/目录中存放配置文件，有两种选择，修改这里的EnvironmentFile，或者把/etc/etcd目录中文件链接到 /etc/default 中，我这里选择第二种方式。

注意如果选择第一种方式，修改完成之后要运行 systemctl daemon-reload etcd让新的配置生效。

修改之后，下面我们来配置etcd集群，我们需要配置一个安全的高可用集群。

下面的内容，应该在**宿主机（HOST）**机器中先生成，然后我们用脚本一键配置，避免出错

  1. 创建ca 根证书

    #创建一个工作目录
mkdir etcd
cd etcd
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=etcd-peer" -days 36500 -out ca.crt

  2. 创建etcd的服务端ca证书

    创建之前首先要有证书的SAN配置文件，如果使用的是kvm下面的虚拟机，可以使用下面的脚本生成

    #!/bin/bash
# 文件保存为generate-ssl-config.sh
vm_list=$(sudo virsh list --state-running | awk 'NR > 2 { print $2 }')

cat >etcd_ssl.cnf <<EOF
[ req ]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[ req_distinguished_name ]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
EOF

IFS=',' read -ra items <<<"$1"

count=1
for item in "${items[@]}"; do
  ip=$(sudo virsh domifaddr "$item" | awk '/ipv4/ {print $4}' | cut -d'/' -f1)
  echo "IP.$count = $ip" >>etcd_ssl.cnf
  ((count++))
done

    生成的类似于如下格式的文件（etcd_ssl.cnf）：

    [ req ]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[ req_distinguished_name ]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
IP.1 = 192.168.122.182
IP.2 = 192.168.122.134
IP.3 = 192.168.122.168

    开始创建证书

    openssl genrsa -out etcd_server.key 2048
openssl req -new -key etcd_server.key -config etcd_ssl.cnf -subj "/CN=etcd-server" -out etcd_server.csr
openssl x509 -req -in etcd_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 36500 -extensions v3_req -extfile etcd_ssl.cnf -out etcd_server.crt

  3. 创建etcd客户端证书

    openssl genrsa -out etcd_client.key 2048
openssl req -new -key etcd_client.key -config etcd_ssl.cnf -subj "/CN=etcd-client" -out etcd_client.csr
openssl x509 -req -in etcd_client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 36500 -extensions v3_req -extfile etcd_ssl.cnf -out etcd_client.crt

  4. 创建etcd配置文件

    这里我们依然使用脚本生成

    #!/bin/bash
# 文件保存为generate-etcd-conf.sh
generate() {
  local name="$1"
  local ip=$(sudo virsh domifaddr "$name" | awk '/ipv4/ {print $4}' | cut -d'/' -f1)
  cat >"$name.etcd.conf" <<EOF
# 节点的名称，每个节点都应该不同
ETCD_NAME=$name
# etcd数据存放目录
ETCD_DATA_DIR=/etc/etcd/data
# 服务端证书和key
ETCD_CERT_FILE=/etc/etcd/pki/etcd_server.crt
ETCD_KEY_FILE=/etc/etcd/pki/etcd_server.key
# 信任ca根证书
ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.crt
ETCD_CLIENT_CERT_AUTH=true
# 客户端连接的地址
ETCD_LISTEN_CLIENT_URLS=https://$ip:2379
ETCD_ADVERTISE_CLIENT_URLS=https://$ip:2379
# 集群节点间相互认证的证书和key
ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd_server.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd_server.key
# 信任ca根证书
ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.crt
# 其他节点连接的地址
ETCD_LISTEN_PEER_URLS=https://$ip:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://$ip:2380

# 集群的名称
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
# 初始化集群的状态，新建集群设置为"new"，加入集群的时候设置为"existing"，
ETCD_INITIAL_CLUSTER_STATE=new
EOF
}

# 拆分参数字符串为数组
IFS=',' read -ra items <<<"$1"

etcdCluesters=""

# 遍历数组并调用函数
for item in "${items[@]}"; do
  generate $item
  ip=$(sudo virsh domifaddr "$item" | awk '/ipv4/ {print $4}' | cut -d'/' -f1)
  if [ -z "$etcdCluesters" ]; then
    etcdCluesters="$item=https://$ip:2380"
  else
    etcdCluesters="$etcdCluesters,$item=https://$ip:2380"
  fi
done

for item in "${items[@]}"; do
  echo "ETCD_INITIAL_CLUSTER=\"$etcdCluesters\"" >>"$item.etcd.conf"
done

    保存为文件之后，直接执行./generate-etcd-conf.sh ubuntu-01,ubuntu-02,ubuntu-03会在当前的目录生成三个配置文件

    生成的配置文件类似于下面的格式：

    ETCD_NAME=ubuntu-01
ETCD_DATA_DIR=/etc/etcd/data
ETCD_CERT_FILE=/etc/etcd/pki/etcd_server.crt

ETCD_KEY_FILE=/etc/etcd/pki/etcd_server.key
ETCD_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.crt
ETCD_CLIENT_CERT_AUTH=true
ETCD_LISTEN_CLIENT_URLS=https://192.168.122.140:2379
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.122.140:2379

ETCD_PEER_CERT_FILE=/etc/etcd/pki/etcd_server.crt
ETCD_PEER_KEY_FILE=/etc/etcd/pki/etcd_server.key
ETCD_PEER_TRUSTED_CA_FILE=/etc/kubernetes/pki/ca.crt
ETCD_LISTEN_PEER_URLS=https://192.168.122.140:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.122.140:2380

ETCD_INITIAL_CLUSTER_TOKEN=etca-cluster
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER="ubuntu-01=https://192.168.122.140:2380,ubuntu-02=https://192.168.122.100:2380,ubuntu-03=https://192.168.122.165:2380"

  5. 最后我们再创建一个脚本，帮助我们，一键配置etcd

    #!/bin/bash
# 文件保存为 config-etcd.sh
# 先放在这里，我们后面会安装kubernetes
sudo mkdir -p /etc/kubernetes/pki
sudo mkdir -p /etc/etcd/pki
sudo mkdir -p /etc/etcd/data
sudo cp ./ca.* /etc/kubernetes/pki
sudo cp ./etcd_server.* /etc/etcd/pki
sudo cp ./etcd_client.* /etc/etcd/pki
sudo chown -R etcd /etc/etcd/data
sudo chown -R etcd /etc/etcd/pki
sudo mv /etc/default/etcd /etc/default/etcd.bak
sudo ln -s /etc/etcd/etcd.conf /etc/default/etcd

  6. 最后一步把我们的生成的文件分别传到我们的机器上

    ca.crt
ca.key
ca.srl
config-etcd.sh
etcd_client.crt
etcd_client.csr
etcd_client.key
etcd_server.crt
etcd_server.csr
etcd_server.key
etcd_ssl.cnf
generate-etcd-conf.sh
generate-ssl-config.sh
ubuntu-01.etcd.conf
ubuntu-02.etcd.conf
ubuntu-03.etcd.conf
    #先打包一下
tar -cvzf etcd.tar.gz etcd/
# 使用scp传输到服务器上
scp etcd.tar.gz ubuntu-01:/home/ubuntu
# 解压运行 config-etcd.sh

  7. 验证集群运行情况

    #使用v3的api
export ETCDCTL_API=3
#查看集群运行情况
etcdctl --cacert=/etc/kubernetes/pki/ca.crt --cert=/etc/etcd/pki/etcd_client.crt --key=/etc/etcd/pki/etcd_client.key --endpoints=https://192.168.122.140:2379,https://192.168.122.100:2379,https://192.168.122.189:2379 endpoint health

    注意这里的cacert、cert 我们都可以放在环境变量中，放在.bashrc文件中，这样在使用etcdctl的时候，就不用每一次都写长长的参数了，类似于下面这种

    export ETCDCTL_API=3 export ETCDCTL_ENDPOINTS=“https://192.168.122.140:2379,https://192.168.122.100:2379,https://192.168.122.189:2379” export ETCDCTL_CACERT="/etc/kubernetes/pki/ca.crt" export ETCDCTL_CERT="/etc/etcd/pki/etcd_client.crt" export ETCDCTL_KEY="/etc/etcd/pki/etcd_client.key"

    如果输出是类似于下面的内容，表示集群搭建成功

    https://192.168.122.140:2379 is healthy: successfully committed proposal: took = 14.744654ms
https://192.168.122.100:2379 is healthy: successfully committed proposal: took = 15.038352ms
https://192.168.122.189:2379 is healthy: successfully committed proposal: took = 17.155474ms

  8. 向已有集群中添加新的节点

    假设我们新创建了一个新的机器 ubuntu-04（这里使用新的机器的方式）

    首先，需要创建新的证书，这里需要修改证书配置文件，把新的机器的ip地址放进去192.168.122.170

    [ req ]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[ req_distinguished_name ]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
IP.1 = 192.168.122.182
IP.2 = 192.168.122.134
IP.3 = 192.168.122.168
IP.4 = 192.168.122.170

    然后用之前的根证书（注意一定要用上一次的根证书），生成新的证书，生成证书的命令和上面一致，和上面节点一样，配置好证书。

    在任意节点的机器上运行

    export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://192.168.122.140:2379,https://192.168.122.100:2379,https://192.168.122.165:2379,https://192.168.122.189:2379"
export ETCDCTL_CACERT="/etc/kubernetes/pki/ca.crt"
export ETCDCTL_CERT="/etc/etcd/pki/etcd_client.crt"
export ETCDCTL_KEY="/etc/etcd/pki/etcd_client.key"

etcdctl member add ubuntu-04 --peer-urls=https://192.168.122.165:2380

    成功之后，会输出如下的内容：

    Member 75bf29390cd64d5e added to cluster ab21a525ae2568ae

ETCD_NAME="ubuntu-04"
ETCD_INITIAL_CLUSTER="ubuntu-02=https://192.168.122.100:2380,ubuntu-04=https://192.168.122.165:2380,ubuntu-03=https://192.168.122.165:2380,ubuntu-01=https://192.168.122.140:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.122.165:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

    然后把这一段内容，放到etcd的配置文件中，然后启动etcd，再次 运行 etcdctl endpoint health，如果出现类似于下面的内容，则表示，新节点加入成功

    https://192.168.122.140:2379 is healthy: successfully committed proposal: took = 14.744654ms
https://192.168.122.100:2379 is healthy: successfully committed proposal: took = 15.038352ms
https://192.168.122.189:2379 is healthy: successfully committed proposal: took = 17.155474ms
https://192.168.122.165:2379 is healthy: successfully committed proposal: took = 17.762056ms

完整的配置脚本见：https://github.com/lihuu/dotfiles/tree/master/etcd