mysql_mha

admin · 发表于 2020-1-4 21:21:46

1.MHA

1.1.mha介绍

MHA(Master High Availability)是自动的master故障转移和Slave提升的软件包.它是基于标准的MySQL复制(异步/半同步).
  MHA有两部分组成:MHA Manager(管理节点)和MHA Node(数据节点).
  MHA Manager可以单独部署在一台独立机器上管理多个master-slave集群,也可以部署在一台slave上.MHA Manager探测集群的node节点,当发现master出现故障的时候,它可以自动将具有最新数据的slave提升为新的master,然后将所有其它的slave导向新的master上.整个故障转移过程对应用程序是透明的。
  MHA node运行在每台MySQL服务器上(master/slave/manager),它通过监控具备解析和清理logs功能的脚本来加快故障转移的。
u=2647160713,136212046&fm=26&gp=0.jpg.png

u=2647160713,136212046&fm=26&gp=0.jpg.png

1.2.mha原理

-从宕机崩溃的master保存二进制日志事件(binlog events)。
-识别含有最新更新的slave。
-应用差异的中继日志(relay log)到其它slave。
-应用从master保存的二进制日志事件(binlog events)。
-提升一个slave为新master。
-使其它的slave连接新的master进行复制。

1.3.mha工具包

Manager工具:
- masterha_check_ssh : 检查MHA的SSH配置。
- masterha_check_repl : 检查MySQL复制。
- masterha_manager : 启动MHA。
- masterha_check_status : 检测当前MHA运行状态。
- masterha_master_monitor : 监测master是否宕机。
- masterha_master_switch : 控制故障转移(自动或手动)。
- masterha_conf_host : 添加或删除配置的server信息。
Node工具(这些工具通常由MHA Manager的脚本触发,无需人手操作)。
- save_binary_logs : 保存和复制master的二进制日志。
- apply_diff_relay_logs : 识别差异的中继日志事件并应用于其它slave。
- filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)。
- purge_relay_logs : 清除中继日志(不会阻塞SQL线程)。

2.部署mysql2.1.部署环境概览

主机名       vm1       vm2       vm3
ip地址       192.168.56.101       192.168.56.102       192.168.56.103
os版本       CentOS7.7.1908       CentOS7.7.1908       CentOS7.7.1908
mysql用户       mysql:mysql       mysql:mysql       mysql:mysql
mysql版本       5.7.28       5.7.28       5.7.28
mysql家目录       -       -       -
mysql软件目录       /app/mysql/mysql-5.7.28-el7-x86_64       /app/mysql/mysql-5.7.28-el7-x86_64       /app/mysql/mysql-5.7.28-el7-x86_64
mysql数据目录       /app/mysql/data       /app/mysql/data       /app/mysql/data
Server_id       1       2       3
mha版本       0.58       0.58       0.58
mha角色       master/node       slave/node       slave/manager

2.2.软件下载

下载地址
http://mirrors.163.com/centos/7. ... x86_64-DVD-1908.iso
mysql下载地址
https://dev.mysql.com/downloads/mysql/
node下载地址
https://github.com/yoshinorim/mha4mysql-node/releases
manager下载地址
https://github.com/yoshinorim/mha4mysql-manager/releases
mha官方文档
https://github.com/yoshinorim/mha4mysql-manager/wiki/Installation

2.3.安装mysql软件

2.3.1.创建用户

[root@vm1 tmp]# groupadd mysql
[root@vm1 tmp]# useradd -r -g mysql -s /bin/false mysql

2.3.2.创建目录

[root@vm1 tmp]# mkdir -p /app/mysql/data

2.3.3.解压安装包

[root@vm1 tmp]# tar -zxvf mysql-5.7.28-el7-x86_64.tar.gz -C /app/mysql
[root@vm1 tmp]# chown -R mysql:mysql /app/mysql

2.3.4.软件初始化

需要记住屏幕输出的临时密码

[root@vm1]# cd /app/mysql/mysql-5.7.28-el7-x86_64/bin
[root@vm1 bin]# ./mysqld --initialize --user=mysql --basedir=/app/mysql/mysql-5.7.28-el7-x86_64 --datadir=/app/mysql/data
2020-01-04T00:33:43.913280Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-04T00:33:44.362025Z 0 [Warning] InnoDB: New log files created, LSN=45790
2020-01-04T00:33:44.453443Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2020-01-04T00:33:44.517537Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: dd39dbef-2e89-11ea-8b4f-080027e53f17.
2020-01-04T00:33:44.520130Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-01-04T00:33:44.946839Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-04T00:33:45.130378Z 1 [Note] A temporary password is generated for root@localhost

2.3.5.配置参数文件

cat >>/etc/my.cnf<<EOF
[mysqld]
user = mysql
port = 3306
log-error = /app/mysql/data/mysql.err
character-set-server=utf8
log-bin=mysql-bin
server-id=1
gtid_mode=ON
log_slave_updates
enforce_gtid_consistency

2.3.6.配置启动文件

cp mysql.server /etc/init.d/mysql
vi /etc/init.d/mysql

修改如下内容

basedir=/app/mysql/mysql-5.7.28-el7-x86_64
datadir=/app/mysql/data

2.3.7.启动实例，修改密码

/etc/init.d/mysql start
[mysql@vm1 ~]$ mysql -S /app/mysql/data/3306/3306.sock -p
Enter password:

mysql> alter user 'root'@'localhost' identified by 'mysql';
Query OK, 0 rows affected (0.01 sec)</font>

2.4.复制到其他节点

在各节点上启动并测试功能是否正常

2.4.1.主库打包

tar -pzcvf mysql.tar.gz /app/mysql

2.4.2.slave解压

tar -pzxvf mysql.tar.gz -C /

2.4.3.修改uuid
因为是打包复制的原因，从库配置文件的uuid需要修改，文件位置在/app/mysql/data/3306/auto.cnf
[mysql@vm1 3306]$ cat auto.cnf
[auto]
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f17
[mysql@vm2 3306]$ cat auto.cnf
[auto]
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f18
[mysql@vm3 3306]$ cat auto.cnf
[auto]
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f19

2.5.配置GTID一主二从

2.5.1.主库添加rep账户

mysql> grant replication slave on *.* to rep@'192.168.56.%' identified by 'rep';
Query OK, 0 rows affected, 1 warning (0.01 sec)

2.5.2.从库配置主节点

mysql> change master to
master_host='vm1',
master_user='rep',
master_password='rep',
master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.04 sec)

2.5.3.确认配置状态

>mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status \G;
*************************** 1. row ***************************
            Slave_IO_State: Waiting for master to send event
               Master_Host: vm1
               Master_User: rep
               Master_Port: 3306
            Connect_Retry: 60
            Master_Log_File: mysql-bin.000009
      Read_Master_Log_Pos: 194
            Relay_Log_File: vm2-relay-bin.000004
            Relay_Log_Pos: 407
      Relay_Master_Log_File: mysql-bin.000009
         Slave_IO_Running: Yes
         Slave_SQL_Running: Yes
            Replicate_Do_DB:
      Replicate_Ignore_DB:
         Replicate_Do_Table:
   Replicate_Ignore_Table:
   Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
               Last_Errno: 0
               Last_Error:
               Skip_Counter: 0
      Exec_Master_Log_Pos: 194
            Relay_Log_Space: 1281
            Until_Condition: None
            Until_Log_File:
            Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
         Master_SSL_Cipher:
            Master_SSL_Key:
      Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
            Last_IO_Errno: 0
            Last_IO_Error:
            Last_SQL_Errno: 0
            Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
         Master_Server_Id: 1
               Master_UUID: 3439775d-01cb-11ea-8d1d-080027e53f17
         Master_Info_File: /app/mysql/data/3306/master.info
                  SQL_Delay: 0
      SQL_Remaining_Delay: NULL
   Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
         Master_Retry_Count: 86400
               Master_Bind:
   Last_IO_Error_Timestamp:
   Last_SQL_Error_Timestamp:
            Master_SSL_Crl:
         Master_SSL_Crlpath:
         Retrieved_Gtid_Set: 3439775d-01cb-11ea-8d1d-080027e53f17:1-2
         Executed_Gtid_Set: 3439775d-01cb-11ea-8d1d-080027e53f17:1-2
            Auto_Position: 1
      Replicate_Rewrite_DB:
               Channel_Name:
         Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified

2.5.4.配置半同步复制

分别在主从库加载上面两个插件

master：
mysql> install plugin rpl_semi_sync_master soname 'semisync_master.so';
slave：
mysql> install plugin rpl_semi_sync_slave soname 'semisync_slave.so';

开启半同步复制

mater：
mysql> set global rpl_semi_sync_master_enabled = 1;
slave：
mysql> set global rpl_semi_sync_slave_enabled = 1;

#以上的启动方式是在命令行操作，是临时生效的；永久生效需将如下设置写在配置文件中。

master：
plugin-load = rpl_semi_sync_master=semisync_master.so  #此项可以让plugin在任何时候都被mysql加载
rpl_semi_sync_master_enabled = 1

slave：
plugin-load = rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled = 1

#在有的高可用架构下，master和slave需同时启动，以便在切换后能继续使用半同步复制

plugin-load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so"
rpl-semi-sync-master-enabled = 1
rpl-semi-sync-slave-enabled = 1

重启slave上的IO

mysql> stop slave io_thread;
mysql> start slave io_thread;

#如果没有重启，则默认还是异步复制，重启后，slave会在master上注册为半同步复制的slave角色。

3.4. 查看半同步状态

master：
mysql> show status like 'Rpl_semi_sync_master_status';
+-----------------------------+-------+
| Variable_name             | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_status | ON |
+-----------------------------+-------+
slave：
mysql> show status like 'Rpl_semi_sync_slave_status';
+----------------------------+-------+
| Variable_name             | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON |
+----------------------------+--

2.5.5.确认同步正常

主库：

mysql> insert into zhyu values(4);
Query OK, 1 row affected (0.01 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql>  select * from zhyu;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
| 4 |
+------+
4 rows in set (0.00 sec)
从库：

mysql>  select * from zhyu;
ERROR 1046 (3D000): No database selected
mysql> select * from zhyu.zhyu;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
| 4 |
+------+
4 rows in set (0.00 sec)

3.部署MHA

3.1.安装node

在所有节点进行安装

3.1.1.安装依赖包

最好使用ali的镜像源

wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install perl perl-DBI perl-IO-Socket-SSL perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y

3.1.2.rpm安装

[root@vm1 ~]# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing...                         ################################# [100%]
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos ################################# [100%]
[root@vm2 ~]# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing...                         ################################# [100%]
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos ################################# [100%]
[root@vm3 ~]# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing...                         ################################# [100%]
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos ################################# [100%]

3.1.3.确认安装完成

[root@vm1 ~]# ls -lrt /usr/local/bin
total 68
-rwxr-xr-x 1 root root  2445 Sep 17 17:48 dbhome
-rwxr-xr-x 1 root root  6823 Sep 17 17:48 oraenv
-rwxr-xr-x 1 root root  6404 Sep 17 17:48 coraenv
-r-xr-xr-x 1 root root  7525 Jan  2 15:05 save_binary_logs
-r-xr-xr-x 1 root root  8337 Jan  2 15:05 purge_relay_logs
-r-xr-xr-x 1 root root  4807 Jan  2 15:05 filter_mysqlbinlog
-r-xr-xr-x 1 root root 17639 Jan  2 15:05 apply_diff_relay_logs

这些脚本工具通常由MHA Manager的脚本触发，无需人为操作。脚本说明：

apply_diff_relay_logs：识别差异的中继日志事件并将其差异的事件应用于其它slave。
filter_mysqlbinlog：去除不必要的ROLLBACK事件（MHA已不再使用这个工具）。
purge_relay_logs：清除中继日志（不会阻塞SQL线程）。
save_binary_logs：保存和复制master的二进制日志。

3.2.安装manager

3.2.1.安装依赖包

最好使用ali的镜像源

wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install perl perl-DBI perl-IO-Socket-SSL perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y

3.2.2.rpm安装

[root@vm3 ~]# rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
Preparing...                         ################################# [100%]
      package mha4mysql-manager-0.58-0.el7.centos.noarch is already installed

3.2.3.确认安装完成
安装完成后，在/usr/bin/目录下有如下MHA相关文件：

masterha_check_repl
masterha_check_ssh
masterha_check_status
masterha_conf_host
masterha_manager
masterha_master_monitor
masterha_master_switch
masterha_secondary_check
masterha_stop
apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs

3.3.配置ssh互信

三个节点配置ssh互信登录

[root@vm1 .ssh]# ssh vm1 date
Sat Jan  4 10:57:18 CST 2020
[root@vm1 .ssh]# ssh vm2 date
Sat Jan  4 10:57:23 CST 2020
[root@vm1 .ssh]# ssh vm3 date
Sat Jan  4 10:57:26 CST 2020

4.配置mha

4.1.配置manager

4.1.1.编辑配置文件

[root@vm3 ~]# mkdir -p /etc/mha
[root@vm3 masterha]# cat /etc/mha/mha.cnf
[server default]
manager_log=/var/log/mha_mgr.log
manager_workdir=/app/mha
master_binlog_dir=/app/mysql/data
password=mysql
ping_interval=1
remote_workdir=/tmp
repl_password=rep
repl_user=rep
ssh_user=root
user=root
master_ip_failover_script="/app/mha/master_ip_failover"
master_ip_online_change_script="/app/mha/master_ip_online_change"
[server1]
hostname=192.168.56.101
candidate_master=1
master_binlog_dir="/mysql/mysql/data"
[server2]
hostname=192.168.56.102
candidate_master=1
check_repl_delay=0
master_binlog_dir="/mysql/mysql/data"
[server3]
hostname=192.168.56.103
no_master=1
check_repl_delay=0
master_binlog_dir="/mysql/mysql/data"
server default段是manager的一些基本配置参数，server1、server2、server3分别对应复制中的master、第一个slave、第二个slave。该文件的语法要求严格，变量值后不要有多余的空格。主要配置项说明如下。

manager_log：设置manager的日志文件。
manager_workdir：设置manager的工作目录。
master_binlog_dir：设置master保存binlog的位置，以便MHA可以找到master的日志，这里的也就是mysql的数据目录。
master_ip_failover_script：设置自动failover时候的切换脚本。
master_ip_online_change_script：设置手动切换时候的切换脚本。
password：设置mysql中root用户的密码。
ping_interval：设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行railover。
remote_workdir：设置远端mysql在发生切换时binlog的保存位置。
repl_password：设置复制用户的密码。
repl_user：设置复制环境中的复制用户名
secondary_check_script：一旦MHA到hdp4的监控之间出现问题，MHA Manager将会尝试从hdp3登录到hdp4。
shutdown_script：设置故障发生后关闭故障主机脚本。该脚本的主要作用是关闭主机放在发生脑裂，这里没有使用。
ssh_user：设置ssh的登录用户名。
user：设置监控用户为root。
candidate_master：设置为候选master。设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave。
check_repl_delay：默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0，MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master。

4.1.2.建立软连接

[root@vm3 masterha]# cd /app/mysql/bin
[root@vm3 bin]# ln -s mysqlbinlog /usr/bin/mysqlbinlog
[root@vm3 bin]# ln -s mysql /usr/bin/mysql

4.1.3.从库设置参数

set global relay_log_purge=0;
set global read_only=1;

注意，MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF，采用手动清除relay log的方式。默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。

4.1.4.创建管理账号
所有节点添加账户

grant all on *.* to root@'192.168.56.%' identified by 'mysql';

4.1.5.验证ssh

[root@vm3 ~]# masterha_check_ssh  --conf=/etc/masterha_default.cnf
Sat Jan  4 15:16:23 2020 - [info] Reading default configuration from /etc/masterha_default.cnf..
Sat Jan  4 15:16:23 2020 - [info] Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan  4 15:16:23 2020 - [info] Reading server configuration from /etc/masterha_default.cnf..
Sat Jan  4 15:16:23 2020 - [info] Starting SSH connection tests..
Sat Jan  4 15:16:24 2020 - [debug]
Sat Jan  4 15:16:23 2020 - [debug]  Connecting via SSH from root@192.168.56.101(192.168.56.101:22) to root@192.168.56.102(192.168.56.102:22)..
Sat Jan  4 15:16:24 2020 - [debug] ok.
Sat Jan  4 15:16:24 2020 - [debug]  Connecting via SSH from root@192.168.56.101(192.168.56.101:22) to root@192.168.56.103(192.168.56.103:22)..
Sat Jan  4 15:16:24 2020 - [debug] ok.
Sat Jan  4 15:16:25 2020 - [debug]
Sat Jan  4 15:16:24 2020 - [debug]  Connecting via SSH from root@192.168.56.102(192.168.56.102:22) to root@192.168.56.101(192.168.56.101:22)..
Sat Jan  4 15:16:24 2020 - [debug] ok.
Sat Jan  4 15:16:24 2020 - [debug]  Connecting via SSH from root@192.168.56.102(192.168.56.102:22) to root@192.168.56.103(192.168.56.103:22)..
Sat Jan  4 15:16:24 2020 - [debug] ok.
Sat Jan  4 15:16:26 2020 - [debug]
Sat Jan  4 15:16:24 2020 - [debug]  Connecting via SSH from root@192.168.56.103(192.168.56.103:22) to root@192.168.56.101(192.168.56.101:22)..
Sat Jan  4 15:16:25 2020 - [debug] ok.
Sat Jan  4 15:16:25 2020 - [debug]  Connecting via SSH from root@192.168.56.103(192.168.56.103:22) to root@192.168.56.102(192.168.56.102:22)..
Sat Jan  4 15:16:25 2020 - [debug] ok.
Sat Jan  4 15:16:26 2020 - [info] All SSH connection tests passed successfully.

4.1.6.验证repl

[root@vm3 etc]# masterha_check_repl --conf=/etc/masterha_default.cnf
Sat Jan  4 12:16:46 2020 - [info] Reading default configuration from /etc/masterha_default.cnf..
Sat Jan  4 12:16:46 2020 - [info] Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan  4 12:16:46 2020 - [info] Reading server configuration from /etc/masterha_default.cnf..
Sat Jan  4 12:16:46 2020 - [info] MHA::MasterMonitor version 0.58.
Sat Jan  4 12:16:48 2020 - [info] GTID failover mode = 1
Sat Jan  4 12:16:48 2020 - [info] Dead Servers:
Sat Jan  4 12:16:48 2020 - [info] Alive Servers:
Sat Jan  4 12:16:48 2020 - [info] 192.168.56.101(192.168.56.101:3306)
Sat Jan  4 12:16:48 2020 - [info] 192.168.56.102(192.168.56.102:3306)
Sat Jan  4 12:16:48 2020 - [info] 192.168.56.103(192.168.56.103:3306)
Sat Jan  4 12:16:48 2020 - [info] Alive Slaves:
Sat Jan  4 12:16:48 2020 - [info] 192.168.56.102(192.168.56.102:3306)  Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan  4 12:16:48 2020 - [info]    GTID ON
Sat Jan  4 12:16:48 2020 - [info]    Replicating from vm1(192.168.56.101:3306)
Sat Jan  4 12:16:48 2020 - [info]    Primary candidate for the new Master (candidate_master is set)
Sat Jan  4 12:16:48 2020 - [info] 192.168.56.103(192.168.56.103:3306)  Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan  4 12:16:48 2020 - [info]    GTID ON
Sat Jan  4 12:16:48 2020 - [info]    Replicating from vm1(192.168.56.101:3306)
Sat Jan  4 12:16:48 2020 - [info] Current Alive Master: 192.168.56.101(192.168.56.101:3306)
Sat Jan  4 12:16:48 2020 - [info] Checking slave configurations..
Sat Jan  4 12:16:48 2020 - [info]  read_only=1 is not set on slave 192.168.56.102(192.168.56.102:3306).
Sat Jan  4 12:16:48 2020 - [info]  read_only=1 is not set on slave 192.168.56.103(192.168.56.103:3306).
Sat Jan  4 12:16:48 2020 - [info] Checking replication filtering settings..
Sat Jan  4 12:16:48 2020 - [info]  binlog_do_db= , binlog_ignore_db=
Sat Jan  4 12:16:48 2020 - [info]  Replication filtering check ok.
Sat Jan  4 12:16:48 2020 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Sat Jan  4 12:16:48 2020 - [info] Checking SSH publickey authentication settings on the current master..
Sat Jan  4 12:16:48 2020 - [info] HealthCheck: SSH to 192.168.56.101 is reachable.
Sat Jan  4 12:16:48 2020 - [info]
192.168.56.101(192.168.56.101:3306) (current master)
+--192.168.56.102(192.168.56.102:3306)
+--192.168.56.103(192.168.56.103:3306)

Sat Jan  4 12:16:48 2020 - [info] Checking replication health on 192.168.56.102..
Sat Jan  4 12:16:48 2020 - [info]  ok.
Sat Jan  4 12:16:48 2020 - [info] Checking replication health on 192.168.56.103..
Sat Jan  4 12:16:48 2020 - [info]  ok.
Sat Jan  4 12:16:48 2020 - [warning] master_ip_failover_script is not defined.
Sat Jan  4 12:16:48 2020 - [warning] shutdown_script is not defined.
Sat Jan  4 12:16:48 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

5.管理MHA

5.1.启停与查看状态

5.1.1.查看状态

[root@vm3 etc]# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default is stopped(2:NOT_RUNNING).

5.1.2.启动mha

[root@vm3 etc]# nohup  masterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover  &
[1] 11615
[root@vm3 etc]# nohup: ignoring input and appending output to ‘nohup.out’
[root@vm3 etc]# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default (pid:11615) is running(0:PING_OK), master:192.168.56.101

5.1.3.停止mha

[root@vm3 etc]# masterha_stop --conf=/etc/masterha_default.cnf
Stopped masterha_default successfully.
[1]+  Exit 1                nohup masterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover

5.2.自动故障切换

5.2.1.master配置vip

[root@vm1 ~]# ip addr add 192.168.56.104/24 dev enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:43:55:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.101/24 brd 192.168.56.255 scope global noprefixroute enp0s8
   valid_lft forever preferred_lft forever
inet 192.168.56.104/24 scope global secondary enp0s8
   valid_lft forever preferred_lft forever
inet6 fe80::c04a:4f9e:402f:95be/64 scope link noprefixroute
   valid_lft forever preferred_lft forever

5.2.2.关闭master

[root@vm1 ~]# /etc/init.d/mysql stop
Shutting down MySQL............ SUCCESS!
5.2.3.查看candidate master
mysql> show master status \G;
*************************** 1. row ***************************
         File: mysql-bin.000008
      Position: 2707
   Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f17:1-8,
dd39dbef-2e89-11ea-8b4f-080027e53f18:1-4
1 row in set (0.00 sec)

ERROR:
No query specified

5.2.4.vip漂移到vm2

[root@vm2 ~]# ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
      inet6 fe80::9658:182d:b82d:2f95  prefixlen 64  scopeid 0x20<link>
      ether 08:00:27:63:eb:86  txqueuelen 1000  (Ethernet)
      RX packets 914  bytes 77673 (75.8 KiB)
      RX errors 0  dropped 0  overruns 0  frame 0
      TX packets 1030  bytes 86063 (84.0 KiB)
      TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet 192.168.56.102  netmask 255.255.255.0  broadcast 192.168.56.255
      inet6 fe80::c04a:4f9e:402f:95be  prefixlen 64  scopeid 0x20<link>
      inet6 fe80::14e0:ed28:2938:30fc  prefixlen 64  scopeid 0x20<link>
      ether 08:00:27:b2:57:f8  txqueuelen 1000  (Ethernet)
      RX packets 7077  bytes 731716 (714.5 KiB)
      RX errors 0  dropped 0  overruns 0  frame 0
      TX packets 6034  bytes 1458712 (1.3 MiB)
      TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet 192.168.56.104  netmask 255.255.255.0  broadcast 192.168.56.255
      ether 08:00:27:b2:57:f8  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
      inet 127.0.0.1  netmask 255.0.0.0
      inet6 ::1  prefixlen 128  scopeid 0x10<host>
      loop  txqueuelen 1000  (Local Loopback)
      RX packets 2220  bytes 129846 (126.8 KiB)
      RX errors 0  dropped 0  overruns 0  frame 0
      TX packets 2220  bytes 129846 (126.8 KiB)
      TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

5.2.5.查看slave

[root@vm3 data]# mysql -uroot -pmysql
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.28-log MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status \G;
*************************** 1. row ***************************
            Slave_IO_State: Waiting for master to send event
               Master_Host: 192.168.56.102
               Master_User: rep
               Master_Port: 3306
            Connect_Retry: 60
            Master_Log_File: mysql-bin.000008
      Read_Master_Log_Pos: 2707
            Relay_Log_File: vm3-relay-bin.000003
            Relay_Log_Pos: 1491
      Relay_Master_Log_File: mysql-bin.000008
         Slave_IO_Running: Yes
         Slave_SQL_Running: Yes

5.2.6.mha已停止

[root@vm3 mha]# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default is stopped(2:NOT_RUNNING).
[1]+  Done                   nohup masterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover

5.2.7.原master手动转为slave

[root@vm1 ~]# mysql -uroot -pmysql
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.28-log MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> change master to
-> master_host='vm2',
-> master_user='rep',
-> master_password='rep',
-> master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G:
*************************** 1. row ***************************
            Slave_IO_State: Waiting for master to send event
               Master_Host: vm2
               Master_User: rep
               Master_Port: 3306
            Connect_Retry: 60
            Master_Log_File: mysql-bin.000009
      Read_Master_Log_Pos: 234
            Relay_Log_File: vm1-relay-bin.000002
            Relay_Log_Pos: 367
      Relay_Master_Log_File: mysql-bin.000008
         Slave_IO_Running: Yes
         Slave_SQL_Running: Yes
            Replicate_Do_DB:
      Replicate_Ignore_DB:
         Replicate_Do_Table:
   Replicate_Ignore_Table:
   Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
               Last_Errno: 0
               Last_Error:
               Skip_Counter: 0
      Exec_Master_Log_Pos: 194
            Relay_Log_Space: 2154
            Until_Condition: None
            Until_Log_File:
            Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
         Master_SSL_Cipher:
            Master_SSL_Key:
      Seconds_Behind_Master: 20942
Master_SSL_Verify_Server_Cert: No
            Last_IO_Errno: 0
            Last_IO_Error:
            Last_SQL_Errno: 0
            Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
         Master_Server_Id: 2
               Master_UUID: dd39dbef-2e89-11ea-8b4f-080027e53f18
         Master_Info_File: /app/mysql/data/master.info
                  SQL_Delay: 0
      SQL_Remaining_Delay: NULL
   Slave_SQL_Running_State: Waiting for semi-sync ACK from slave
         Master_Retry_Count: 86400
               Master_Bind:
   Last_IO_Error_Timestamp:
   Last_SQL_Error_Timestamp:
            Master_SSL_Crl:
         Master_SSL_Crlpath:
         Retrieved_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f18:1-4
         Executed_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f17:1-8
            Auto_Position: 1
      Replicate_Rewrite_DB:
               Channel_Name:
         Master_TLS_Version:
1 row in set (0.00 sec)

5.3.手动故障切换
mha必须时关闭状态，master出现故障时手动切换。

masterha_master_switch --conf=/etc/masterha_default.cnf--master_state=dead  --dead_master_host=192.168.56.101 --dead_master_port=3306 --new_master_host=192.168.56.102 --new_master_port=3306 --ignore_last_failover

5.4.手动角色切换
手动切换需要关闭mha监控，主机名最好用ip地址。

masterha_master_switch --conf=/etc/masterha_default.cnf  --master_state=alive --new_master_host=192.168.56.102 --orig_master_is_new_slave
[root@vm3 mha]# masterha_master_switch --conf=/etc/masterha_default.cnf  --master_state=alive --new_master_host=192.168.56.102 --orig_master_is_new_slave
Sat Jan  4 18:42:58 2020 - [info] MHA::MasterRotate version 0.58.
Sat Jan  4 18:42:58 2020 - [info] Starting online master switch..
Sat Jan  4 18:42:58 2020 - [info]
Sat Jan  4 18:42:58 2020 - [info] * Phase 1: Configuration Check Phase..
Sat Jan  4 18:42:58 2020 - [info]
Sat Jan  4 18:42:58 2020 - [info] Reading default configuration from /etc/masterha_default.cnf..
Sat Jan  4 18:42:58 2020 - [info] Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan  4 18:42:58 2020 - [info] Reading server configuration from /etc/masterha_default.cnf..
Sat Jan  4 18:42:59 2020 - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.56.101(192.168.56.101:3306)
Sat Jan  4 18:42:59 2020 - [info] Master configurations are as below:
Master 192.168.56.102(192.168.56.102:3306), replicating from vm1(192.168.56.101:3306), read-only
Master 192.168.56.101(192.168.56.101:3306), replicating from vm2(192.168.56.102:3306)

Sat Jan  4 18:42:59 2020 - [info] GTID failover mode = 1
Sat Jan  4 18:42:59 2020 - [info] Current Alive Master: 192.168.56.101(192.168.56.101:3306)
Sat Jan  4 18:42:59 2020 - [info] Alive Slaves:
Sat Jan  4 18:42:59 2020 - [info] 192.168.56.102(192.168.56.102:3306)  Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan  4 18:42:59 2020 - [info]    GTID ON
Sat Jan  4 18:42:59 2020 - [info]    Replicating from vm1(192.168.56.101:3306)
Sat Jan  4 18:42:59 2020 - [info]    Primary candidate for the new Master (candidate_master is set)
Sat Jan  4 18:42:59 2020 - [info] 192.168.56.103(192.168.56.103:3306)  Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan  4 18:42:59 2020 - [info]    GTID ON
Sat Jan  4 18:42:59 2020 - [info]    Replicating from vm1(192.168.56.101:3306)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.56.101(192.168.56.101:3306)? (YES/no): yes
Sat Jan  4 18:43:01 2020 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Sat Jan  4 18:43:01 2020 - [info]  ok.
Sat Jan  4 18:43:01 2020 - [info] Checking MHA is not monitoring or doing failover..
Sat Jan  4 18:43:01 2020 - [info] Checking replication health on 192.168.56.102..
Sat Jan  4 18:43:01 2020 - [info]  ok.
Sat Jan  4 18:43:01 2020 - [info] Checking replication health on 192.168.56.103..
Sat Jan  4 18:43:01 2020 - [info]  ok.
Sat Jan  4 18:43:01 2020 - [info] 192.168.56.102 can be new master.
Sat Jan  4 18:43:01 2020 - [info]
From:
192.168.56.101(192.168.56.101:3306) (current master)
+--192.168.56.102(192.168.56.102:3306)
+--192.168.56.103(192.168.56.103:3306)

To:
192.168.56.102(192.168.56.102:3306) (new master)
+--192.168.56.103(192.168.56.103:3306)
+--192.168.56.101(192.168.56.101:3306)

Starting master switch from 192.168.56.101(192.168.56.101:3306) to 192.168.56.102(192.168.56.102:3306)? (yes/NO): yes
Sat Jan  4 18:43:04 2020 - [info] Checking whether 192.168.56.102(192.168.56.102:3306) is ok for the new master..
Sat Jan  4 18:43:04 2020 - [info]  ok.
Sat Jan  4 18:43:04 2020 - [info] ** Phase 1: Configuration Check Phase completed.
Sat Jan  4 18:43:04 2020 - [info]
Sat Jan  4 18:43:04 2020 - [info] * Phase 2: Rejecting updates Phase..
Sat Jan  4 18:43:04 2020 - [info]
Sat Jan  4 18:43:04 2020 - [info] Executing master ip online change script to disable write on the current master:
Sat Jan  4 18:43:04 2020 - [info] /app/mha/master_ip_online_change --command=stop --orig_master_host=192.168.56.101 --orig_master_ip=192.168.56.101 --orig_master_port=3306 --orig_master_user='root' --new_master_host=192.168.56.102 --new_master_ip=192.168.56.102 --new_master_port=3306 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Sat Jan  4 18:43:04 2020 417539 Set read_only on the new master.. ok.
Sat Jan  4 18:43:04 2020 425674 drop vip 192.168.56.104..
Cannot find device "ens192"
Sat Jan  4 18:43:04 2020 563594 Waiting all running 2 threads are disconnected.. (max 1500 milliseconds)
{'Time' => '1311','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1305','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan  4 18:43:05 2020 075826 Waiting all running 2 threads are disconnected.. (max 1000 milliseconds)
{'Time' => '1312','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1306','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan  4 18:43:05 2020 576437 Waiting all running 2 threads are disconnected.. (max 500 milliseconds)
{'Time' => '1312','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1306','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan  4 18:43:06 2020 089749 Set read_only=1 on the orig master.. ok.
Sat Jan  4 18:43:06 2020 107408 Waiting all running 2 queries are disconnected.. (max 500 milliseconds)
{'Time' => '1313','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1307','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan  4 18:43:06 2020 595752 Killing all application threads..
Sat Jan  4 18:43:06 2020 605704 done.
Sat Jan  4 18:43:06 2020 - [info]  ok.
Sat Jan  4 18:43:06 2020 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Sat Jan  4 18:43:06 2020 - [info] Executing FLUSH TABLES WITH READ LOCK..
Sat Jan  4 18:43:06 2020 - [info]  ok.
Sat Jan  4 18:43:06 2020 - [info] Orig master binlog:pos is mysql-bin.000015:647.
Sat Jan  4 18:43:06 2020 - [info]  Waiting to execute all relay logs on 192.168.56.102(192.168.56.102:3306)..
Sat Jan  4 18:43:06 2020 - [info]  master_pos_wait(mysql-bin.000015:647) completed on 192.168.56.102(192.168.56.102:3306). Executed 0 events.
Sat Jan  4 18:43:06 2020 - [info] done.
Sat Jan  4 18:43:06 2020 - [info] Getting new master's binlog name and position..
Sat Jan  4 18:43:06 2020 - [info]  mysql-bin.000011:638
Sat Jan  4 18:43:06 2020 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Sat Jan  4 18:43:06 2020 - [info] Executing master ip online change script to allow write on the new master:
Sat Jan  4 18:43:06 2020 - [info] /app/mha/master_ip_online_change --command=start --orig_master_host=192.168.56.101 --orig_master_ip=192.168.56.101 --orig_master_port=3306 --orig_master_user='root' --new_master_host=192.168.56.102 --new_master_ip=192.168.56.102 --new_master_port=3306 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Sat Jan  4 18:43:06 2020 751402 Set read_only=0 on the new master.
Sat Jan  4 18:43:06 2020 753434Add vip 192.168.56.104 on ens192..
Cannot find device "ens192"
Sat Jan  4 18:43:06 2020 - [info]  ok.
Sat Jan  4 18:43:06 2020 - [info]
Sat Jan  4 18:43:06 2020 - [info] * Switching slaves in parallel..
Sat Jan  4 18:43:06 2020 - [info]
Sat Jan  4 18:43:06 2020 - [info] -- Slave switch on host 192.168.56.103(192.168.56.103:3306) started, pid: 11512
Sat Jan  4 18:43:06 2020 - [info]
Sat Jan  4 18:43:07 2020 - [info] Log messages from 192.168.56.103 ...
Sat Jan  4 18:43:07 2020 - [info]
Sat Jan  4 18:43:06 2020 - [info]  Waiting to execute all relay logs on 192.168.56.103(192.168.56.103:3306)..
Sat Jan  4 18:43:06 2020 - [info]  master_pos_wait(mysql-bin.000015:647) completed on 192.168.56.103(192.168.56.103:3306). Executed 0 events.
Sat Jan  4 18:43:06 2020 - [info] done.
Sat Jan  4 18:43:06 2020 - [info]  Resetting slave 192.168.56.103(192.168.56.103:3306) and starting replication from the new master 192.168.56.102(192.168.56.102:3306)..
Sat Jan  4 18:43:06 2020 - [info]  Executed CHANGE MASTER.
Sat Jan  4 18:43:06 2020 - [info]  Slave started.
Sat Jan  4 18:43:07 2020 - [info] End of log messages from 192.168.56.103 ...
Sat Jan  4 18:43:07 2020 - [info]
Sat Jan  4 18:43:07 2020 - [info] -- Slave switch on host 192.168.56.103(192.168.56.103:3306) succeeded.
Sat Jan  4 18:43:07 2020 - [info] Unlocking all tables on the orig master:
Sat Jan  4 18:43:07 2020 - [info] Executing UNLOCK TABLES..
Sat Jan  4 18:43:07 2020 - [info]  ok.
Sat Jan  4 18:43:07 2020 - [info] Starting orig master as a new slave..
Sat Jan  4 18:43:07 2020 - [info]  Resetting slave 192.168.56.101(192.168.56.101:3306) and starting replication from the new master 192.168.56.102(192.168.56.102:3306)..
Sat Jan  4 18:43:07 2020 - [info]  Executed CHANGE MASTER.
Sat Jan  4 18:43:07 2020 - [info]  Slave started.
Sat Jan  4 18:43:07 2020 - [info] All new slave servers switched successfully.
Sat Jan  4 18:43:07 2020 - [info]
Sat Jan  4 18:43:07 2020 - [info] * Phase 5: New master cleanup phase..
Sat Jan  4 18:43:07 2020 - [info]
Sat Jan  4 18:43:07 2020 - [info]  192.168.56.102: Resetting slave info succeeded.
Sat Jan  4 18:43:07 2020 - [info] Switching master to 192.168.56.102(192.168.56.102:3306) completed successfully.

6.附件

6.1.master_ip_failover

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command,       $ssh_user,       $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);

my $vip = '192.168.56.104/24';  # Virtual IP
my $key = "1";
my $int = "enp0s8";
my $ssh_start_vip = "/sbin/ifconfig $int:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig $int:$key down";
my $arp_effect = "/sbin/arping -Uq -s192.168.56.104 -I $int 192.168.56.254 -c 3"; # Virtual IP and gatway

#my $test = "echo successfull >/tmp/test.txt";
$ssh_user = "root";
GetOptions(
'command=s'       => \$command,
'ssh_user=s'       => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s'  => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i'  => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

      # $orig_master_host, $orig_master_ip, $orig_master_port are passed.
      # If you manage master ip address at global catalog database,
      # invalidate orig_master_ip here.
      my $exit_code = 1;
      eval {
         print "Disabling the VIP on old master: $orig_master_host \n";
         &stop_vip();
         $exit_code = 0;
      };
      if ($@) {
         warn "Got Error: $@\n";
         exit $exit_code;
      }
      exit $exit_code;
}
elsif ( $command eq "start" ) {

      # all arguments are passed.
      # If you manage master ip address at global catalog database,
      # activate new_master_ip here.
      # You can also grant write access (create user, set read_only=0, etc) here.
      my $exit_code = 10;
      eval {
         print "Enabling the VIP - $vip on the new master - $new_master_host \n";
         &start_vip();
         $exit_code = 0;
      };
      if ($@) {
         warn $@;
         exit $exit_code;
      }
      exit $exit_code;
}
elsif ( $command eq "status" ) {
      print "Checking the Status of the script.. OK \n";
      #`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`;
      &status();
      exit 0;
}
else {
      &usage();
      exit 1;
}
}

# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
`ssh $ssh_user\@$new_master_host \" $arp_effect \"`;
# `ssh $ssh_user\@$new_master_host \" $test \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub status() {
print `ssh $ssh_user\@$orig_master_host \" ip add show $int \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_maste
r_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

6.2.master_ip_online_change

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;

my $_tstart;
my $_running_interval = 0.1;

my $vip = "192.168.56.104";
my $if = "ens192";

my (
  $command,                   $orig_master_is_new_slave,    $orig_master_host,
  $orig_master_ip,             $orig_master_port,             $orig_master_user,
  $orig_master_password,       $orig_master_ssh_user,       $new_master_host,
  $new_master_ip,    $new_master_port,             $new_master_user,
  $new_master_password, $new_master_ssh_user,
);
GetOptions(
  'command=s'                         => \$command,
  'orig_master_is_new_slave' => \$orig_master_is_new_slave,
  'orig_master_host=s'                => \$orig_master_host,
  'orig_master_ip=s'          => \$orig_master_ip,
  'orig_master_port=i'       => \$orig_master_port,
  'orig_master_user=s'       => \$orig_master_user,
  'orig_master_password=s'    => \$orig_master_password,
  'orig_master_ssh_user=s'    => \$orig_master_ssh_user,
  'new_master_host=s'          => \$new_master_host,
  'new_master_ip=s'                   => \$new_master_ip,
  'new_master_port=i'          => \$new_master_port,
  'new_master_user=s'          => \$new_master_user,
  'new_master_password=s'    => \$new_master_password,
  'new_master_ssh_user=s'    => \$new_master_ssh_user,
);

exit &main();
sub drop_vip {
      my $output = `ssh -o ConnectTimeout=15  -o ConnectionAttempts=3 $orig_master_host /sbin/ip addr del $vip/32 dev $if`;

}
sub add_vip {
      my $output = `ssh -o ConnectTimeout=15  -o ConnectionAttempts=3 $new_master_host /sbin/ip addr add $vip/32 dev $if`;

}

sub current_time_us {
  my ( $sec, $microsec ) = gettimeofday();
  my $curdate = localtime($sec);
  return $curdate . " " . sprintf( "%06d", $microsec );
}

sub sleep_until {
  my $elapsed = tv_interval($_tstart);
  if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
  }
}

sub get_threads_util {
  my $dbh                   = shift;
  my $my_connection_id    = shift;
  my $running_time_threshold = shift;
  my $type                = shift;
  $running_time_threshold = 0 unless ($running_time_threshold);
  $type                = 0 unless ($type);
  my @threads;

  my $sth = $dbh->prepare("SHOW PROCESSLIST");
  $sth->execute();

  while ( my $ref = $sth->fetchrow_hashref() ) {
my $id       = $ref->{Id};
my $user    = $ref->{User};
my $host    = $ref->{Host};
my $command = $ref->{Command};
my $state    = $ref->{State};
my $query_time = $ref->{Time};
my $info    = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user)    && $user eq "system user" );
next
   if ( defined($command)
   && $command eq "Sleep"
   && defined($query_time)
   && $query_time >= 1 );

if ( $type >= 1 ) {
   next if ( defined($command) && $command eq "Sleep" );
   next if ( defined($command) && $command eq "Connect" );
}

if ( $type >= 2 ) {
   next if ( defined($info) && $info =~ m/^select/i );
   next if ( defined($info) && $info =~ m/^show/i );
}

push @threads, $ref;
  }
  return @threads;
}

sub main {
  if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
   ## Setting read_only=1 on the new master (to avoid accident)
   my $new_master_handler = new MHA::DBHelper();

   # args: hostname, port, user, password, raise_error(die_on_error)_ or_not
   $new_master_handler->connect( $new_master_ip, $new_master_port,
      $new_master_user, $new_master_password, 1 );
   print current_time_us() . " Set read_only on the new master.. ";
   $new_master_handler->enable_read_only();
   if ( $new_master_handler->is_read_only() ) {
      print "ok.\n";
   }
   else {
      die "Failed!\n";
   }
   $new_master_handler->disconnect();

   # Connecting to the orig master, die if any database error happens
   my $orig_master_handler = new MHA::DBHelper();
   $orig_master_handler->connect( $orig_master_ip, $orig_master_port,
      $orig_master_user, $orig_master_password, 1 );

   ## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
   $orig_master_handler->disable_log_bin_local();
   # print current_time_us() . " Drpping app user on the orig master..\n";
   print current_time_us() . " drop vip $vip..\n";
   #drop_app_user($orig_master_handler);
   &drop_vip();

   ## Waiting for N * 100 milliseconds so that current connections can exit
   my $time_until_read_only = 15;
   $_tstart = [gettimeofday];
   my @threads = get_threads_util( $orig_master_handler->{dbh},
      $orig_master_handler->{connection_id} );
   while ( $time_until_read_only > 0 && $#threads >= 0 ) {
      if ( $time_until_read_only % 5 == 0 ) {
      printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
         current_time_us(), $#threads + 1, $time_until_read_only * 100;
      if ( $#threads < 5 ) {
         print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
            foreach (@threads);
      }
      }
      sleep_until();
      $_tstart = [gettimeofday];
      $time_until_read_only--;
      @threads = get_threads_util( $orig_master_handler->{dbh},
      $orig_master_handler->{connection_id} );
   }

   ## Setting read_only=1 on the current master so that nobody(except SUPER) can write
   print current_time_us() . " Set read_only=1 on the orig master.. ";
   $orig_master_handler->enable_read_only();
   if ( $orig_master_handler->is_read_only() ) {
      print "ok.\n";
   }
   else {
      die "Failed!\n";
   }

   ## Waiting for M * 100 milliseconds so that current update queries can complete
   my $time_until_kill_threads = 5;
   @threads = get_threads_util( $orig_master_handler->{dbh},
      $orig_master_handler->{connection_id} );
   while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
      if ( $time_until_kill_threads % 5 == 0 ) {
      printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
         current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
      if ( $#threads < 5 ) {
         print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
            foreach (@threads);
      }
      }
      sleep_until();
      $_tstart = [gettimeofday];
      $time_until_kill_threads--;
      @threads = get_threads_util( $orig_master_handler->{dbh},
      $orig_master_handler->{connection_id} );
   }

   ## Terminating all threads
   print current_time_us() . " Killing all application threads..\n";
   $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
   print current_time_us() . " done.\n";
   $orig_master_handler->enable_log_bin_local();
   $orig_master_handler->disconnect();

   ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
   $exit_code = 0;
};
if ($@) {
   warn "Got Error: $@\n";
   exit $exit_code;
}
exit $exit_code;
  }
  elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database

# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
   my $new_master_handler = new MHA::DBHelper();

   # args: hostname, port, user, password, raise_error_or_not
   $new_master_handler->connect( $new_master_ip, $new_master_port,
      $new_master_user, $new_master_password, 1 );

   ## Set read_only=0 on the new master
   $new_master_handler->disable_log_bin_local();
   print current_time_us() . " Set read_only=0 on the new master.\n";
   $new_master_handler->disable_read_only();

   ## Creating an app user on the new master
   #print current_time_us() . " Creating app user on the new master..\n";
   print current_time_us() . "Add vip $vip on $if..\n";
   # create_app_user($new_master_handler);
   &add_vip();
   $new_master_handler->enable_log_bin_local();
   $new_master_handler->disconnect();

   ## Update master ip on the catalog database, etc
   $exit_code = 0;
};
if ($@) {
   warn "Got Error: $@\n";
   exit $exit_code;
}
exit $exit_code;
  }
  elsif ( $command eq "status" ) {

# do nothing
exit 0;
  }
  else {
&usage();
exit 1;
  }
}

sub usage {
  print
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
  die;
}

		自动登录	找回密码
密码			立即注册