RabbitMQ_Cluster+HA(haproxy+keepalived)热备

RabbitMQ_Cluster+HA（haproxy+keepalived）热备

--------------时间：2014-8-15 -------------------四海道达运维组 -----------系统版：Ubutnu Server 14.04

目录

RabbitMQ_Cluster+HA（haproxy+keepalived）热备 ............................................... 1

1. 网络拓扑图 ............................................................................................ 3 2. 修改主机名\dns记录\较对时间 ................................................................ 4

2.1 Rabbit1主机 ................................................................................ 4 2.2 Rabbit2主机 ................................................................................ 5 2.3 Rabbit3主机 ................................................................................ 5 3. 安装配置Rabbit_Cluster 集群 .................................................................. 6

3.1 Rabbit1主机 ................................................................................ 6 3.2 Rabbit2主机 ................................................................................ 8 3.3 Rabbit3主机 .............................................................................. 10 3.4 添加Rabbit_Cluster管理用户 ..................................................... 11 3.5 设置用户tags ............................................................................ 12 3.6 设置用户权限 ............................................................................. 12 4. 设置高可用策略 ................................................................................... 12

4.1 方法一（命令）： ....................................................................... 12 4.2 方法二（web页面）： ................................................................. 13 5. 安装配置 Haproxy+Keeplived ................................................................ 15

5.1 Rabbit2主机 .............................................................................. 15

5.2 Rabbit3主机 .............................................................................. 18 5.3 登录Haproxy的管理端页面 .......................................................... 20 6. 较验Rabbit_Cluster集群服务 ................................................................ 21 7. 较验Haproxy+Keepalived主备服务 ....................................................... 22

7.1 首先查看一下Haproxy和Keepalived的进程 ................................. 22 7.2 模拟故障1：主节点Haproxy进程死掉，进程自动恢复 ................... 22 7.3 模拟故障2：主节点Haproxy进程死掉，进程无法恢复 ................... 23 7.4 模拟故障3：主节点Haproxy恢复正常。 ....................................... 25

1. 网络拓扑图

RabbitMQ_Cluster集群：

Rabbit1：192.168.100.191 （disc） Rabbit2：192.168.100.192 （ram） Rabbit3：192.168.100.193 （ram） Rabbit集群中，必须有一个节点是disc模式。

Haproxy+Keeplived 热备 Master：192.168.100.192 Backup：192.168.100.193 VIP： 192.168.100.195

2. 修改主机名\dns记录\较对时间

注意：所有操作均是root权限，如若不是请sudo -s切换成root

2.1 Rabbit1主机

在/etc/network/interfaces中不要配置dns记录

vim 01_rabbit1_hostname.sh #!/bin/bash

echo "rabbit1" >/etc/hostname

echo "192.168.100.191 rabbit1" >> /etc/hosts echo "192.168.100.192 rabbit2" >> /etc/hosts echo "192.168.100.193 rabbit3" >> /etc/hosts

echo "nameserver 8.8.8.8" >/etc/resolvconf/resolv.conf.d/base

ntpdate -s apt-get -y install ntp service ntp reload reboot exit

2.2 Rabbit2主机

在/etc/network/interfaces中不要配置dns记录

vim 01_rabbit2_hostname.sh #!/bin/bash

echo "rabbit2" >/etc/hostname

echo "192.168.100.191 rabbit1" >> /etc/hosts echo "192.168.100.192 rabbit2" >> /etc/hosts echo "192.168.100.193 rabbit3" >> /etc/hosts

echo "nameserver 8.8.8.8" >/etc/resolvconf/resolv.conf.d/base

ntpdate -s apt-get -y install ntp service ntp reload reboot exit

2.3 Rabbit3主机

在/etc/network/interfaces中不要配置dns记录

vim 01_rabbit3_hostname.sh #!/bin/bash

echo "rabbit3" >/etc/hostname

echo "192.168.100.191 rabbit1" >> /etc/hosts echo "192.168.100.192 rabbit2" >> /etc/hosts echo "192.168.100.193 rabbit3" >> /etc/hosts

echo "nameserver 8.8.8.8" >/etc/resolvconf/resolv.conf.d/base

ntpdate -s apt-get -y install ntp service ntp reload

reboot exit

3. 安装配置Rabbit_Cluster 集群

因为类同重复性工作较多，所以直接采用《shell脚本+人工操作》搭配方式进行搭建，脚本执行过程中需要在rabbit1和rabbit2交互进行。

3.1 Rabbit1主机

vim 02_rabbit1_install.sh #!/bin/bash

apt-get install -y erlang ##安装rabbitmq所依赖的工具erlang

##添加rabbitmq源

echo "deb /debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list

wget /rabbitmq-signing-key-public.asc apt-key add rabbitmq-signing-key-public.asc apt-get update

##安装rabbitmq-server服务

apt-get install -y rabbitmq-server

##方法二

echo “CHDCGETPYWOWREASJUAZ” > /var/lib/rabbitmq/.erlang.cookie

service rabbitmq-server start

# 配置rabbitmq

rabbitmq-plugins enable rabbitmq_management

##停止rabbit服务 rabbitmqctl stop

##查看rabbit状态 rabbitmqctl status

##此处交互，请在rabbit2服务器执行脚本02_rabbit2_install.sh echo "Please run 02_rabbit2_install.sh in the rabbit2 server"

echo "After completion ! ! !"

echo "Press "Enter" to continue .........." read

##因为做rabbitmq_cluster需要确保.erlang.cookie一致性，所以rabbit2和rabbit3上面的.erlang.cookie均需要从rabbit1上面scp复制过去，可以参见如下（方法一），但因为是全新的，其实完全可以写一段相同的内容更省事，参见上文中（方法二见上文），方法一我只做介绍（见下文）在这里，我注释掉，因为脚本中直接使用方法二。

##方法一

#安装expect工具，以实现自动交互上传文件。 #apt-get -y install tcl tk expect #在bash中调用expect命令

#rabbit1第一次向rabbit2机器scp文件 #/usr/bin/expect <<EOF #set timeout 5

#spawn scp /var/lib/rabbitmq/.erlang.cookie #stone@192.168.100.192:/home/stone/cookit/ #set timeout 3 #expect "(yes/no)" #send "yes\r" #expect "password:" #send "lance!@#\r" #expect eof #EOF

# rabbit1第二次或以上向rabbit2机器scp文件 #/usr/bin/expect <<EOF #set timeout 10

#spawn scp /var/lib/rabbitmq/.erlang.cookie #stone@192.168.100.192:/home/stone/cookit/ #set timeout 3 #expect "password:" #send "lance!@#\r" #expect eof #EOF

#echo "Press "Enter" to continue .......... in the rabbit2 server" #echo "After completion ! ! !"

#echo "Press "Enter" to continue .........." #read

##启动rabbitma服务

rabbitmq-server -detached

echo "Press "Enter" to continue .......... in the rabbit2 server" echo "After completion ! ! !"

echo "Press "Enter" to continue .........."

read

##查看Rabbit_Cluster集群的状态 rabbitmqctl cluster_status

echo "Press "Enter" to continue ....Rabbit_Cluster Install is End...." read

exit

3.2 Rabbit2主机

vim 02_rabbit2_install.sh #!/bin/bash

apt-get install -y erlang

useradd -d /var/lib/rabbitmq rabbitmq

echo "deb /debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list

wget /rabbitmq-signing-key-public.asc apt-key add rabbitmq-signing-key-public.asc apt-get update

apt-get install -y rabbitmq-server

##方法二

echo “CHDCGETPYWOWREASJUAZ” > /var/lib/rabbitmq/.erlang.cookie

service rabbitmq-server start

# Configure Setup

rabbitmq-plugins enable rabbitmq_management

rabbitmqctl stop

rabbitmqctl status

##以下是对应rabbit1脚本中的方法一，因为采用的方法二，所以我注释掉它。 #rm -rf /var/lib/rabbitmq/.erlang.cookie

#mkdir -p /home/stone/cookit &&chown stone.stone /home/stone/cookit

#echo "Press "Enter" to continue .......... in the rabbit1 server" #echo "After completion ! ! !"

#echo "Press "Enter" to continue .........." #read

#mv /home/stone/cookit/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie #chown rabbitmq.rabbitmq /var/lib/rabbitmq/.erlang.cookie

##以正常模式启动rabbitmq-server服务 rabbitmq-server -detached

echo "Press "Enter" to continue .......... in the rabbit1 server" echo "After completion ! ! !"

echo "Press "Enter" to continue .........." read

##rabbit2加入集群rabbit

## If you need to join the cluster rabbitmqctl stop_app

#If you use a hard disk (permanent), such as: #rabbitmqctl join_cluster rabbit@rabbit1 #If you use a memory, such as: ##rabit2是ram模式

rabbitmqctl join_cluster --ram rabbit@rabbit1 rabbitmqctl start_app

##如何改变rabbit的模式为ram或disc

#=================================================== #We can Changing node types #rabbit2$ rabbitmqctl stop_app

#rabbit2$ rabbitmqctl change_cluster_node_type disc #or

#rabbit2$ rabbitmqctl change_cluster_node_type ram #rabbit2$ rabbitmqctl start_app

#===================================================

##到此，rabbit2加入集群成功

echo "Please run 03_rabbit3_install.sh in the rabbit3 server" echo "After completion ! ! !"

echo "Press "Enter" to continue .........." read

##查看Rabbit_Cluster集群的状态 rabbitmqctl cluster_status

#=============================== #If you need from the cluster #rabbitmqctl stop_app #rabbitmqctl reset #rabbitmqctl start_app

#===============================

echo "Press "Enter" to continue ....Rabbit_Cluster Install is End...."

read exit

3.3 Rabbit3主机

vim 03_rabbit3_install.sh #!/bin/bash

apt-get install -y erlang

useradd -d /var/lib/rabbitmq rabbitmq

echo "deb /debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list

wget /rabbitmq-signing-key-public.asc apt-key add rabbitmq-signing-key-public.asc apt-get update

apt-get install -y rabbitmq-server

##方法二

echo “CHDCGETPYWOWREASJUAZ” > /var/lib/rabbitmq/.erlang.cookie

service rabbitmq-server start

# Configure Setup

rabbitmq-plugins enable rabbitmq_management

rabbitmqctl stop

rabbitmqctl status

##以正常模式启动rabbitmq-server服务 rabbitmq-server -detached

##rabbit3加入集群rabbit

## If you need to join the cluster rabbitmqctl stop_app

#If you use a hard disk (permanent), such as: #rabbitmqctl join_cluster rabbit@rabbit1 #If you use a memory, such as:

##rabit3是ram模式，直接加入到rabbit2，其实都是一个意思 rabbitmqctl join_cluster --ram rabbit@rabbit2 rabbitmqctl start_app

##如何改变rabbit的模式为ram或disc

#=================================================== #We can Changing node types

#rabbit2$ rabbitmqctl stop_app

#rabbit2$ rabbitmqctl change_cluster_node_type disc #or

#rabbit2$ rabbitmqctl change_cluster_node_type ram #rabbit2$ rabbitmqctl start_app

#===================================================

##到此，rabbit3加入集群成功

echo "Please run 02_rabbit2_install.sh in the rabbit2 server" echo "After completion ! ! !"

echo "Press "Enter" to continue .........." read

##查看Rabbit_Cluster集群的状态 rabbitmqctl cluster_status

#=============================== #If you need from the cluster #rabbitmqctl stop_app #rabbitmqctl reset #rabbitmqctl start_app

#===============================

echo "Press "Enter" to continue ....Rabbit_Cluster Install is End...." read exit

3.4 添加Rabbit_Cluster管理用户

执行完以上的步骤集群中各个节点基本配置完毕了，现在要做的是添加用户和设置权限，请在主节点（rabbit2）上执行： 添加用户，设置密码。在这里需要提醒一下，新版本(3.3.0)的RabbitMQ为了安全性考虑仅允许localhost使用

guest账户，所以最好根据应用需要建立自己的账户。因为要使用web页面管理，所以我建立了两个账户['web_admin','mgmt_admin']: 账号1：web_admin 密码：admin 账号2：mgmt_admin 密码：admin1

/usr/sbin/rabbitmqctl add_user web_admin admin

/usr/sbin/rabbitmqctl add_user mgmt_admin admin1

3.5 设置用户tags

a) 前一步创建的两个用户tag分别设为

['monitoring','administrator']，设置完之后可以使用rabbitmqctl list_users查看 /usr/sbin/rabbitmqctl set_user_tags web_admin monitoring /usr/sbin/rabbitmqctl set_user_tags mgmt_admin administrator

3.6 设置用户权限

下面命令的意义是对mgmt_admin用户赋予所有虚拟主机vhost(/)，配置、写、读的权限(.*，正则表达式)

/usr/sbin/rabbitmqctl set_permissions -p / mgmt_admin ".*" ".*" ".*"

注：web_admin没有配置，默认是只读权限。

4. 设置高可用策略

为了使用HAProxy做负载均衡，必须将整个RabbitMQ集群的状态设置为镜像模式，具体方式是通过以下命令，注意策略的设置也是在3.x版本中添加的功能，2.x版本是没有的。

4.1 方法一（命令）：

/usr/sbin/rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'

注：这里是设置消息均在所有节点进行同步，也即，Rabbit_Cluster任一节点出问题均不影响访问。

4.2 方法二（web页面）：

http://192.168.100.192:15672

用管理员权限（账号：mgmt_admin 密码：admin1）

举例，再添加一个my_ha策略

a) 点击admin菜单-->右侧的Policies选项-->左侧最下下边的Add / update a policy b) 按照图中的内容根据自己的需求填写

c) 点击Add policy添加策略

此时你就会来你的两台rabbitmq服务器的网页管理端amind菜单下看见刚才创建的队列了.

d) 下面我们来添加一个queues队列来看看效果,这里只是测试结果,其它的先不填写

注意红框中的 x-ha-policy = all 这个,网上说没有这个不会进行复制,但是我测试的时候好像可以复制的,至少queues队列是可以的,先添上吧.

在这里边添加的时候你是可以指定Node选项也就是把这个queues放在哪个node节点上,不过做镜像的时候就没有必要了,呵呵

至此，RabbitMQ的安装配置部分结束，可以访问

http://192.168.100.192:15672 （191和193也可访问）用户名web_admin，密码admin。看看一些基本信息

5. 安装配置 Haproxy+Keeplived 5.1 Rabbit2主机

vim 03_rabbit2_ha+keep.sh #!/bin/bash ##安装配置Haproxy

apt-get -y install haproxy ##配置Haproxy

cat <<EOF > haproxy.cfg global

defaults

log mode

global http

log 127.0.0.1 log 127.0.0.1

local0

local1 notice

chroot /var/lib/haproxy user haproxy group haproxy daemon

maxconn 4096

option httplog option dontlognull

retries 3

option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000 timeout check 5s

listen admin_stat

bind *:8888 ##Haproxy管理页面 mode http

stats refresh 30s

stats uri /haproxy_stats

stats realm Haproxy\ Statistics stats auth admin:admin ##登录名和密码 stats hide-version

##监听5670端口，并转发给三个节点的5672端口，采用轮询策略

listen rabbitmq_cluster *:5670

balance roundrobin

server rabbitmq-1 192.168.100.191:5672 check inter 2000 rise 2 fall server rabbitmq-2 192.168.100.192:5672 check inter 2000 rise 2 fall server rabbitmq-3 192.168.100.193:5672 check inter 2000 rise 2 fall EOF

##安装Keepalived

apt-get –y install keepalived ##配置Keepalived

cat <<EOF >/etc/keepalived/keepalived.conf vrrp_script chk_haproxy {

script "/etc/keepalived/scripts/haproxy_check.sh" interval 2 }

vrrp_instance VI_1 {

state MASTER interface eth0

errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http

virtual_router_id 51

priority 101 ##高优先级，主节点 authentication {

auth_type PASS auth_pass rabbitmq }

track_script {

chk_haproxy }

virtual_ipaddress {

192.168.100.195 } EOF

##创建监控Haproxy进程的脚本

mkdir -p /etc/keepalived/scripts/

cat <<EOF > /etc/keepalived/scripts/haproxy_check.sh #!/bin/bash

LOGFILE="/var/log/keepalived-ha-state.log"

pid1=\`ps aux|grep /etc/haproxy/haproxy.cfg | grep -v grep | grep -v bash | wc –l\`

if [ "\$pid1" = "0" ]; then

echo "===========================================" >> \$LOGFILE date +%Y%m%d-%H:%M:%S >> \$LOGFILE

echo "Ready to start haproxy....." >> \$LOGFILE

haproxy -f /etc/haproxy/haproxy.cfg –D ##启动Haproxy

pid2=\`ps aux|grep /etc/haproxy/haproxy.cfg | grep -v grep | grep -v bash |wc –l\`

if [ "\$pid2" = "0" ]; then

echo "Haproxy failed to start !!! So,stop keepalived!!!" >> \$LOGFILE

/etc/init.d/keepalived stop else

echo "Haproxy start success........." >> \$LOGFILE fi fi EOF

##启动keepalived服务，Haproxy服务也会一起启动的。 service keepalived restart

5.2 Rabbit3主机

vim 03_rabbit2_ha+keep.sh #!/bin/bash ##安装配置Haproxy

apt-get -y install haproxy ##配置Haproxy

cat <<EOF > haproxy.cfg global

defaults

log mode

global http

log 127.0.0.1 log 127.0.0.1

local0

local1 notice

chroot /var/lib/haproxy user haproxy group haproxy daemon

maxconn 4096

option httplog option dontlognull

retries 3

option redispatch maxconn 2000 contimeout 5000 clitimeout 50000 srvtimeout 50000 timeout check 5s

listen admin_stat

bind *:8888 ##Haproxy的管理端口 mode http

stats refresh 30s

stats uri /haproxy_stats

errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http

stats realm Haproxy\ Statistics stats auth admin:admin stats hide-version

##监听5670端口，并转发给三个节点的5672端口，采用轮询策略

listen rabbitmq_cluster *:5670

balance roundrobin

server rabbitmq-1 192.168.100.191:5672 check inter 2000 rise 2 fall server rabbitmq-2 192.168.100.192:5672 check inter 2000 rise 2 fall server rabbitmq-3 192.168.100.193:5672 check inter 2000 rise 2 fall EOF

##安装Keepalived

apt-get –y install keepalived ##配置Keepalived

cat <<EOF >/etc/keepalived/keepalived.conf vrrp_script chk_haproxy {

script "/etc/keepalived/scripts/haproxy_check.sh" interval 2 }

vrrp_instance VI_1 {

state MASTER interface eth0 virtual_router_id 51

priority 100 ##低优先级，备节点 authentication {

auth_type PASS auth_pass rabbitmq }

track_script {

chk_haproxy }

virtual_ipaddress {

192.168.100.195 } EOF

##创建监控Haproxy进程的脚本

mkdir -p /etc/keepalived/scripts/

cat <<EOF > /etc/keepalived/scripts/haproxy_check.sh

#!/bin/bash

LOGFILE="/var/log/keepalived-ha-state.log"

pid1=\`ps aux|grep /etc/haproxy/haproxy.cfg | grep -v grep | grep -v bash | wc –l\`

if [ "\$pid1" = "0" ]; then

echo "===========================================" >> \$LOGFILE date +%Y%m%d-%H:%M:%S >> \$LOGFILE

echo "Ready to start haproxy....." >> \$LOGFILE

haproxy -f /etc/haproxy/haproxy.cfg –D ##启动Haproxy

pid2=\`ps aux|grep /etc/haproxy/haproxy.cfg | grep -v grep | grep -v bash |wc –l\`

if [ "\$pid2" = "0" ]; then

echo "Haproxy failed to start !!! So,stop keepalived!!!" >> \$LOGFILE

/etc/init.d/keepalived stop else

echo "Haproxy start success........." >> \$LOGFILE fi fi EOF

##启动keepalived服务，Haproxy服务也会一起启动的。 service keepalived restart

5.3 登录Haproxy的管理端页面

http://192.168.100.192:8888/haproxy_stats http://192.168.100.193:8888/haproxy_stats http://192.168.100.195:8888/haproxy_stats

可以查看到Haproxy的一些基本信息。

6. 较验Rabbit_Cluster集群服务

##前提条件，必须确保有一个Rabbit节点是正常运行的，下面只举一个例。 ##关闭Rabbitmqctl服务节点，基其他节点上查看集群信息 root@rabbit1:~# rabbitmqctl stop

##在Rabbit2节点上查看集群信息

root@rabbit2:~# rabbitmqctl cluster_status

##在Rabbit3节点上查看集群信息

root@rabbit3:~# rabbitmqctl cluster_status

##现在，重新启动Rabbit1节点，并查看集群信息 root@rabbit1:~# rabbitmq-server –detached

##在Rabbit2节点上查看集群信息

root@rabbit2:~# rabbitmqctl cluster_status

##在Rabbit3节点上查看集群信息

root@rabbit3:~# rabbitmqctl cluster_status

本文来源：https://www.bwwdw.com/article/zgxe.html