etcd 集群搭建
etcd是go语言开发的,不需要依赖,下载二进制可直接使用。
注意:在使用etcdctl工具时要注意etcd版本,分别设置环境变量 export ETCDCTL_API=3 或着为 2 。二版本和三版本使用参数是不一样的。
下载:
二进制包下载地址:https://github.com/etcd-io/etcd/releases
https://storage.googleapis.com/etcd/v3.2.26/etcd-v3.2.26-linux-amd64.tar.gz
安装:
tar xf etcd-v3.2.26-linux-amd64.tar.gz -C /usr/local/ ln -sv /usr/local/etcd-v3.2.26-linux-amd64 /usr/local/etcd
环境变量:
cat >> /etc/profile.d/etcd.sh <<EOF export ETCDCTL_API=3 export PATH=$PATH:/usr/local/etcd EOF . /etc/profile.d/etcd.sh
DNS解析:
cat >> /etc/hosts <<EOF 192.168.1.163 etcd01 192.168.1.164 etcd02 192.168.1.165 etcd03 EOF
配置:
etcd01:
cat >> /etc/etcd/conf.yml <<EOF name: etcd01 data-dir: /opt/etcd/data listen-client-urls: http://192.168.1.163:2379,http://127.0.0.1:2379 advertise-client-urls: http://192.168.1.163:2379,http://127.0.0.1:2379 listen-peer-urls: http://192.168.1.163:2380 initial-advertise-peer-urls: http://192.168.1.163:2380 initial-cluster: etcd01=http://192.168.1.163:2380,etcd02=http://192.168.1.164:2380,etcd03=http://192.168.1.165:2380 initial-cluster-token: etcd-cluster-token initial-cluster-state: new EOF
etcd02:
cat >> /etc/etcd/conf.yml <<EOF name: etcd02 data-dir: /opt/etcd/data listen-client-urls: http://192.168.1.164:2379,http://127.0.0.1:2379 advertise-client-urls: http://192.168.1.164:2379,http://127.0.0.1:2379 listen-peer-urls: http://192.168.1.164:2380 initial-advertise-peer-urls: http://192.168.1.164:2380 initial-cluster: etcd01=http://192.168.1.163:2380,etcd02=http://192.168.1.164:2380,etcd03=http://192.168.1.165:2380 initial-cluster-token: etcd-cluster-token initial-cluster-state: new EOF
etcd03:
cat >> /etc/etcd/conf.yml <<EOF name: etcd03 data-dir: /opt/etcd/data listen-client-urls: http://192.168.1.165:2379,http://127.0.0.1:2379 advertise-client-urls: http://192.168.1.165:2379,http://127.0.0.1:2379 listen-peer-urls: http://192.168.1.165:2380 initial-advertise-peer-urls: http://192.168.1.165:2380 initial-cluster: etcd01=http://192.168.1.163:2380,etcd02=http://192.168.1.164:2380,etcd03=http://192.168.1.165:2380 initial-cluster-token: etcd-cluster-token initial-cluster-state: new EOF
启动各个节点:
nohub etcd --config-file=/etc/etcd/conf.yml &
查看成员:
# etcdctl member list 20afefb5efd840e5, started, etcd02, http://192.168.1.164:2380, http://127.0.0.1:2379,http://192.168.1.164:2379 2caff3363d0f2466, started, etcd01, http://192.168.1.163:2380, http://127.0.0.1:2379,http://192.168.1.163:2379 7b5545d418633999, started, etcd03, http://192.168.1.165:2380, http://127.0.0.1:2379,http://192.168.1.165:2379
查看leader状态:
curl http://127.0.0.1:2379/v2/stats/leader
查看自身状态:
curl http://127.0.0.1:2379/v2/stats/self
读写数据:
# etcdctl put name Anna OK # etcdctl get name name Anna
API操作:
添加:
# curl -X PUT http://127.0.0.1:2379/v2/keys/message -d value="Hello" {"action":"set","node":{"key":"/message","value":"Hello","modifiedIndex":9,"createdIndex":9}}
查询:
# curl http://127.0.0.1:2379/v2/keys/message {"action":"get","node":{"key":"/message","value":"Hello","modifiedIndex":9,"createdIndex":9}}
删除:
# curl -X DELETE http://127.0.0.1:2379/v2/keys/message {"action":"delete","node":{"key":"/message","modifiedIndex":10,"createdIndex":9},"prevNode":{"key":"/message","value":"Hello","modifiedIndex":9,"createdIndex":9}}
查看所有key-value:
# curl -s http://127.0.0.1:2379/v2/keys/?recursive=true {"action":"get","node":{"dir":true}}
添加启动服务:
cat >> /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify WorkingDirectory=/opt/etcd/ # User=etcd ExecStart=/usr/local/etcd/etcd --config-file=/etc/etcd/conf.yml Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
日志位置:如果启动失败看此日志
tail -f /var/log/messages
使用systemd启动:
systemctl daemon-reload systemctl enable etcd systemctl start etcd systemctl status etcd.service -l
证书配置:
使用TLS来对etcd的通信进行加密,需要事先创建证书。
集群之间通信用的证书:
服务端配置:三个节点可以使用一套证书,也可以各自一套,ca是只有一套的。
--peer-client-cert-auth --peer-trusted-ca-file=ca-peer.crt --peer-cert-file=etcd01-peer.crt --peer-key-file=etcd01-peer.key
服务端认证客户端:
服务端配置:三个节点可以使用一套证书,也可以各自一套,ca是只有一套的。
--client-cert-auth --trusted-ca-file=ca-client.crt --cert-file=etcd01-client.crt --key-file=etcd01-client.key
客户端使用:
curl --cacert ca.crt --cert client.crt --key client.key -L https://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar -v
客户端认证服务端:
服务端配置:客户端需要共认的证书 ca.crt,相当于浏览器验证网站
--cert-file=server.crt --key-file=server.key
客户端的使用:
curl --cacert ca.crt https://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar -v
上面的 ca-peer.crt ca-client.crt 可以是一个ca.crt 也可以是各自的ca。
加密的配置方法
使用到的配置如下:
etcd01:
etcd \ --name=etcd01 \ --initial-advertise-peer-urls=https://192.168.1.163:2380 \ --listen-peer-urls=https://192.168.1.163:2380 \ --listen-client-urls=https://192.168.1.163:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.163:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem
etcd02:
etcd \ --name=etcd02 \ --initial-advertise-peer-urls=https://192.168.1.164:2380 \ --listen-peer-urls=https://192.168.1.164:2380 \ --listen-client-urls=https://192.168.1.164:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.164:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem
etcd03:
etcd \ --name=etcd03 \ --initial-advertise-peer-urls=https://192.168.1.165:2380 \ --listen-peer-urls=https://192.168.1.165:2380 \ --listen-client-urls=https://192.168.1.165:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.165:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem
测试是否成功:
etcdctl member list --endpoints=https://192.168.1.164:2379 --cacert=/etc/etcd/cert/ca.pem --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem # 链接已变成https了 4ca118d6bc9f8f9, started, etcd03, https://192.168.1.165:2380, https://127.0.0.1:2379,https://192.168.1.165:2379 15913609f677881d, started, etcd02, https://192.168.1.164:2380, https://127.0.0.1:2379,https://192.168.1.164:2379 ac2736ca29acf341, started, etcd01, https://192.168.1.163:2380, https://127.0.0.1:2379,https://192.168.1.163:2379
检查健康状态:
etcdctl endpoint health --endpoints=https://192.168.1.164:2379 --cacert=/etc/etcd/cert/ca.pem --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem https://192.168.1.164:2379 is healthy: successfully committed proposal: took = 2.541169ms
添加为系统服务systemd Unit模板:
etcd01:
cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking Type=notify WorkingDirectory=/opt/etcd/ #User=etcd #Group=etcd PIDFile=/run/etcd/etcd.pid ExecStart=/usr/local/etcd/etcd \ --name=etcd01 \ --data-dir=/opt/etcd/data \ --initial-advertise-peer-urls=https://192.168.1.163:2380 \ --listen-peer-urls=https://192.168.1.163:2380 \ --listen-client-urls=https://192.168.1.163:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.163:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
etcd02:
cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking Type=notify WorkingDirectory=/opt/etcd/ #User=etcd #Group=etcd PIDFile=/run/etcd/etcd.pid ExecStart=/usr/local/etcd/etcd \ --name=etcd02 \ --data-dir=/opt/etcd/data \ --initial-advertise-peer-urls=https://192.168.1.164:2380 \ --listen-peer-urls=https://192.168.1.164:2380 \ --listen-client-urls=https://192.168.1.164:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.164:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
etcd03:
cat > /usr/lib/systemd/system/etcd.service <<EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking Type=notify WorkingDirectory=/opt/etcd/ #User=etcd #Group=etcd PIDFile=/run/etcd/etcd.pid ExecStart=/usr/local/etcd/etcd \ --name=etcd03 \ --data-dir=/opt/etcd/data \ --initial-advertise-peer-urls=https://192.168.1.165:2380 \ --listen-peer-urls=https://192.168.1.165:2380 \ --listen-client-urls=https://192.168.1.165:2379,https://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.1.165:2379,https://127.0.0.1:2379 \ --initial-cluster-token=etcd-cluster-token \ --initial-cluster=etcd01=https://192.168.1.163:2380,etcd02=https://192.168.1.164:2380,etcd03=https://192.168.1.165:2380 \ --initial-cluster-state=new \ --client-cert-auth \ --trusted-ca-file=/etc/etcd/cert/ca.pem \ --cert-file=/etc/etcd/cert/etcd.pem \ --key-file=/etc/etcd/cert/etcd-key.pem \ --peer-client-cert-auth \ --peer-trusted-ca-file=/etc/etcd/cert/ca.pem \ --peer-cert-file=/etc/etcd/cert/etcd.pem \ --peer-key-file=/etc/etcd/cert/etcd-key.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF
设置开机启动:
systemctl enable etcd.service systemctl start etcd.service
客户端访问:因为当前配置了服务器端认证客户端,所以客户端要证明自己的身份才能访问服务端。
curl --cacert /etc/etcd/cert/ca.pem \ --cert /etc/etcd/cert/etcd.pem \ --key /etc/etcd/cert/etcd-key.pem \ -L https://192.168.1.163:2379/v2/keys/foo -XPUT -d value=bar -v
单节点运行配置
单节点无认证:
[Unit] Description=Etcd Server After=network.target [Service] Type=forking Type=notify WorkingDirectory=/opt/etcd/ #User=etcd #Group=etcd PIDFile=/run/etcd/etcd.pid ExecStart=/usr/local/etcd/etcd --data-dir=/opt/etcd/data --listen-client-urls=http://0.0.0.0:2379 --advertise-client-urls=http://0.0.0.0:2379 Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target
单节点CA认证:
证书使用的配置文件:openssl.cnf
[req] req_extensions = v3_req distinguished_name = req_distinguished_name [ req_distinguished_name ] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = kubernetes DNS.2 = kubernetes.default DNS.3 = kubernetes.default.svc DNS.4 = kubernetes.default.svc.cluster.local DNS.5 = k8s-master DNS.6 = localhost DNS.7 = etcd01 # 此处的域名就相当于网站的域名,如果是用其他域名或IP访问服务是不被信任的 IP.1 = 10.96.0.1 IP.2 = 192.168.0.72 # 如果也想用IP来访问服务,就需要把IP也添加到此配置文件中
创建证书命令:
# CA openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/CN=etcd" -days 5000 -out ca.crt # etcd服务端使用的证书 openssl genrsa -out etcd_server.key 2048 openssl req -new -key etcd_server.key -subj "/CN=etcd" -out etcd_server.csr -config openssl.cnf openssl x509 -req -in etcd_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd_server.crt -days 5000 -extensions v3_req -extfile openssl.cnf # 客户端访问etcd集群使用的证书 openssl genrsa -out etcd_client.key 2048 openssl req -new -key etcd_client.key -subj "/CN=etcd" -out etcd_client.csr -config openssl.cnf openssl x509 -req -in etcd_client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out etcd_client.crt -days 5000 -extensions v3_req -extfile openssl.cnf
Unit文件:
[Unit] Description=Etcd Server After=network.target [Service] Type=forking Type=notify WorkingDirectory=/var/lib/etcd/ #EnvironmentFile=-/etc/etcd/etcd.conf Environment="ETCDCTL_API=3" ExecStart=/usr/bin/etcd \ --data-dir=/opt/etcd/data \ --listen-client-urls=https://0.0.0.0:2379 \ --advertise-client-urls=https://0.0.0.0:2379 \ --client-cert-auth \ --trusted-ca-file=/etc/kubernetes/cert/ca.crt \ --cert-file=/etc/kubernetes/cert/etcd_server.crt \ --key-file=/etc/kubernetes/cert/etcd_server.key [Install] WantedBy=multi-user.target
如果使用环境变量配置:使用环境变量则不需要在Unit文件里面填写参数了
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.70:2379,https://localhost:2379" ETCD_NAME="etcd01" ETCD_ADVERTISE_CLIENT_URLS="https://etcd01:2379,https://localhost:2379" ETCD_CERT_FILE="/etc/etcd/cert/etcd_server.crt" ETCD_KEY_FILE="/etc/etcd/cert/etcd_server.key" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca.crt"
查看:
etcdctl --ca-file ca.crt --cert-file etcd_client.crt --key-file etcd_client.key --endpoints='https://etcd01:2379' member list
使用yum安装etcd:
安装:
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo yum install etcd -y
openssl:
cat > openssl.cnf <<EOF [req] req_extensions = v3_req distinguished_name = req_distinguished_name [ req_distinguished_name ] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = etcd EOF
创建证书:
# CA openssl genrsa -out ca-client.key 2048 openssl req -x509 -new -nodes -key ca-client.key -subj "/CN=etcd" -days 5000 -out ca-client.crt # etcd服务端使用的证书 openssl genrsa -out etcd_server.key 2048 openssl req -new -key etcd_server.key -subj "/CN=etcd" -out etcd_server.csr -config openssl.cnf openssl x509 -req -in etcd_server.csr -CA ca-client.crt -CAkey ca-client.key -CAcreateserial -out etcd_server.crt -days 5000 -extensions v3_req -extfile openssl.cnf # 客户端访问etcd集群使用的证书 openssl genrsa -out etcd_client.key 2048 openssl req -new -key etcd_client.key -subj "/CN=etcd" -out etcd_client.csr -config openssl.cnf openssl x509 -req -in etcd_client.csr -CA ca-client.crt -CAkey ca-client.key -CAcreateserial -out etcd_client.crt -days 5000 -extensions v3_req -extfile openssl.cnf
配置文件:
#[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_WAL_DIR="" #ETCD_LISTEN_PEER_URLS="http://localhost:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.0.20:2379,https://localhost:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="default" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] #ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380" ETCD_ADVERTISE_CLIENT_URLS="https://etcd:2379,https://localhost:2379" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" #ETCD_INITIAL_CLUSTER="default=http://localhost:2380" #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] #ETCD_PROXY="off" #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" # #[Security] ETCD_CERT_FILE="/etc/etcd/cert/etcd_server.crt" ETCD_KEY_FILE="/etc/etcd/cert/etcd_server.key" ETCD_CLIENT_CERT_AUTH="true" ETCD_TRUSTED_CA_FILE="/etc/etcd/cert/ca-client.crt" #ETCD_AUTO_TLS="false" #ETCD_PEER_CERT_FILE="" #ETCD_PEER_KEY_FILE="" #ETCD_PEER_CLIENT_CERT_AUTH="false" #ETCD_PEER_TRUSTED_CA_FILE="" #ETCD_PEER_AUTO_TLS="false" # #[Logging] #ETCD_DEBUG="false" #ETCD_LOG_PACKAGE_LEVELS="" #ETCD_LOG_OUTPUT="default" # #[Unsafe] #ETCD_FORCE_NEW_CLUSTER="false" # #[Version] #ETCD_VERSION="false" #ETCD_AUTO_COMPACTION_RETENTION="0" # #[Profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" # #[Auth] #ETCD_AUTH_TOKEN="simple"
运行:
systemctl start etcd
查看:
etcdctl --ca-file ca-client.crt --cert-file etcd_client.crt --key-file etcd_client.key --endpoints='https://etcd:2379' member list