mysql_mha
1.MHA1.1.mha介绍
MHA(Master High Availability)是自动的master故障转移和Slave提升的软件包.它是基于标准的MySQL复制(异步/半同步).
MHA有两部分组成:MHA Manager(管理节点)和MHA Node(数据节点).
MHA Manager可以单独部署在一台独立机器上管理多个master-slave集群,也可以部署在一台slave上.MHA Manager探测集群的node节点,当发现master出现故障的时候,它可以自动将具有最新数据的slave提升为新的master,然后将所有其 它的slave导向新的master上.整个故障转移过程对应用程序是透明的。
MHA node运行在每台MySQL服务器上(master/slave/manager),它通过监控具备解析和清理logs功能的脚本来加快故障转移的。
1.2.mha原理
-从宕机崩溃的master保存二进制日志事件(binlog events)。
-识别含有最新更新的slave。
-应用差异的中继日志(relay log)到其它slave。
-应用从master保存的二进制日志事件(binlog events)。
-提升一个slave为新master。
-使其它的slave连接新的master进行复制。
1.3.mha工具包
Manager工具:
- masterha_check_ssh : 检查MHA的SSH配置。
- masterha_check_repl : 检查MySQL复制。
- masterha_manager : 启动MHA。
- masterha_check_status : 检测当前MHA运行状态。
- masterha_master_monitor : 监测master是否宕机。
- masterha_master_switch : 控制故障转移(自动或手动)。
- masterha_conf_host : 添加或删除配置的server信息。
Node工具(这些工具通常由MHA Manager的脚本触发,无需人手操作)。
- save_binary_logs : 保存和复制master的二进制日志。
- apply_diff_relay_logs : 识别差异的中继日志事件并应用于其它slave。
- filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)。
- purge_relay_logs : 清除中继日志(不会阻塞SQL线程)。
2.部署mysql2.1.部署环境概览
主机名 vm1 vm2 vm3
ip地址 192.168.56.101 192.168.56.102 192.168.56.103
os版本 CentOS7.7.1908 CentOS7.7.1908 CentOS7.7.1908
mysql用户 mysql:mysql mysql:mysql mysql:mysql
mysql版本 5.7.28 5.7.28 5.7.28
mysql家目录 - - -
mysql软件目录 /app/mysql/mysql-5.7.28-el7-x86_64 /app/mysql/mysql-5.7.28-el7-x86_64 /app/mysql/mysql-5.7.28-el7-x86_64
mysql数据目录 /app/mysql/data /app/mysql/data /app/mysql/data
Server_id 1 2 3
mha版本 0.58 0.58 0.58
mha角色 master/node slave/node slave/manager
2.2.软件下载
下载地址
http://mirrors.163.com/centos/7. ... x86_64-DVD-1908.iso
mysql下载地址
https://dev.mysql.com/downloads/mysql/
node下载地址
https://github.com/yoshinorim/mha4mysql-node/releases
manager下载地址
https://github.com/yoshinorim/mha4mysql-manager/releases
mha官方文档
https://github.com/yoshinorim/mha4mysql-manager/wiki/Installation
2.3.安装mysql软件
2.3.1.创建用户
# groupadd mysql
# useradd -r -g mysql -s /bin/false mysql
2.3.2.创建目录
# mkdir -p /app/mysql/data
2.3.3.解压安装包
# tar -zxvf mysql-5.7.28-el7-x86_64.tar.gz -C /app/mysql
# chown -R mysql:mysql /app/mysql
2.3.4.软件初始化
需要记住屏幕输出的临时密码
# cd /app/mysql/mysql-5.7.28-el7-x86_64/bin
# ./mysqld --initialize --user=mysql --basedir=/app/mysql/mysql-5.7.28-el7-x86_64 --datadir=/app/mysql/data
2020-01-04T00:33:43.913280Z 0 TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-04T00:33:44.362025Z 0 InnoDB: New log files created, LSN=45790
2020-01-04T00:33:44.453443Z 0 InnoDB: Creating foreign key constraint system tables.
2020-01-04T00:33:44.517537Z 0 No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: dd39dbef-2e89-11ea-8b4f-080027e53f17.
2020-01-04T00:33:44.520130Z 0 Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-01-04T00:33:44.946839Z 0 CA certificate ca.pem is self signed.
2020-01-04T00:33:45.130378Z 1 A temporary password is generated for root@localhost
2.3.5.配置参数文件
cat >>/etc/my.cnf<<EOF
user = mysql
port = 3306
log-error = /app/mysql/data/mysql.err
character-set-server=utf8
log-bin=mysql-bin
server-id=1
gtid_mode=ON
log_slave_updates
enforce_gtid_consistency
2.3.6.配置启动文件
cp mysql.server /etc/init.d/mysql
vi /etc/init.d/mysql
修改如下内容
basedir=/app/mysql/mysql-5.7.28-el7-x86_64
datadir=/app/mysql/data
2.3.7.启动实例，修改密码
/etc/init.d/mysql start
$ mysql -S /app/mysql/data/3306/3306.sock -p
Enter password:
mysql> alter user 'root'@'localhost' identified by 'mysql';
Query OK, 0 rows affected (0.01 sec)</font>
2.4.复制到其他节点
在各节点上启动并测试功能是否正常
2.4.1.主库打包
tar -pzcvf mysql.tar.gz /app/mysql
2.4.2.slave解压
tar -pzxvf mysql.tar.gz -C /
2.4.3.修改uuid
因为是打包复制的原因，从库配置文件的uuid需要修改，文件位置在/app/mysql/data/3306/auto.cnf
$ cat auto.cnf
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f17
$ cat auto.cnf
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f18
$ cat auto.cnf
server-uuid=3439775d-01cb-11ea-8d1d-080027e53f19
2.5.配置GTID一主二从
2.5.1.主库添加rep账户
mysql> grant replication slave on *.* to rep@'192.168.56.%' identified by 'rep';
Query OK, 0 rows affected, 1 warning (0.01 sec)
2.5.2.从库配置主节点
mysql> change master to
master_host='vm1',
master_user='rep',
master_password='rep',
master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.04 sec)
2.5.3.确认配置状态
>mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: vm1
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000009
Read_Master_Log_Pos: 194
Relay_Log_File: vm2-relay-bin.000004
Relay_Log_Pos: 407
Relay_Master_Log_File: mysql-bin.000009
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 194
Relay_Log_Space: 1281
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1
Master_UUID: 3439775d-01cb-11ea-8d1d-080027e53f17
Master_Info_File: /app/mysql/data/3306/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 3439775d-01cb-11ea-8d1d-080027e53f17:1-2
Executed_Gtid_Set: 3439775d-01cb-11ea-8d1d-080027e53f17:1-2
Auto_Position: 1
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
2.5.4.配置半同步复制
分别在主从库加载上面两个插件
master：
mysql> install plugin rpl_semi_sync_master soname 'semisync_master.so';
slave：
mysql> install plugin rpl_semi_sync_slave soname 'semisync_slave.so';
开启半同步复制
mater：
mysql> set global rpl_semi_sync_master_enabled = 1;
slave：
mysql> set global rpl_semi_sync_slave_enabled = 1;
#以上的启动方式是在命令行操作，是临时生效的；永久生效需将如下设置写在配置文件中。
master：
plugin-load = rpl_semi_sync_master=semisync_master.so#此项可以让plugin在任何时候都被mysql加载
rpl_semi_sync_master_enabled = 1
slave：
plugin-load = rpl_semi_sync_slave=semisync_slave.so
rpl_semi_sync_slave_enabled = 1
#在有的高可用架构下，master和slave需同时启动，以便在切换后能继续使用半同步复制
plugin-load = "rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so"
rpl-semi-sync-master-enabled = 1
rpl-semi-sync-slave-enabled = 1
重启slave上的IO
mysql> stop slave io_thread;
mysql> start slave io_thread;
#如果没有重启，则默认还是异步复制，重启后，slave会在master上注册为半同步复制的slave角色。
3.4. 查看半同步状态
master：
mysql> show status like 'Rpl_semi_sync_master_status';
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| Rpl_semi_sync_master_status | ON |
+-----------------------------+-------+
slave：
mysql> show status like 'Rpl_semi_sync_slave_status';
+----------------------------+-------+
| Variable_name | Value |
+----------------------------+-------+
| Rpl_semi_sync_slave_status | ON |
+----------------------------+--
2.5.5.确认同步正常
主库：
mysql> insert into zhyu values(4);
Query OK, 1 row affected (0.01 sec)
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
mysql>select * from zhyu;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
| 4 |
+------+
4 rows in set (0.00 sec)
从库：
mysql>select * from zhyu;
ERROR 1046 (3D000): No database selected
mysql> select * from zhyu.zhyu;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
| 4 |
+------+
4 rows in set (0.00 sec)
3.部署MHA
3.1.安装node
在所有节点进行安装
3.1.1.安装依赖包
最好使用ali的镜像源
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install perl perl-DBI perl-IO-Socket-SSL perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y
3.1.2.rpm安装
# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing... #################################
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos #################################
# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing... #################################
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos #################################
# rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
Preparing... #################################
Updating / installing...
1:mha4mysql-node-0.58-0.el7.centos #################################
3.1.3.确认安装完成
# ls -lrt /usr/local/bin
total 68
-rwxr-xr-x 1 root root2445 Sep 17 17:48 dbhome
-rwxr-xr-x 1 root root6823 Sep 17 17:48 oraenv
-rwxr-xr-x 1 root root6404 Sep 17 17:48 coraenv
-r-xr-xr-x 1 root root7525 Jan2 15:05 save_binary_logs
-r-xr-xr-x 1 root root8337 Jan2 15:05 purge_relay_logs
-r-xr-xr-x 1 root root4807 Jan2 15:05 filter_mysqlbinlog
-r-xr-xr-x 1 root root 17639 Jan2 15:05 apply_diff_relay_logs
这些脚本工具通常由MHA Manager的脚本触发，无需人为操作。脚本说明：
apply_diff_relay_logs：识别差异的中继日志事件并将其差异的事件应用于其它slave。
filter_mysqlbinlog：去除不必要的ROLLBACK事件（MHA已不再使用这个工具）。
purge_relay_logs：清除中继日志（不会阻塞SQL线程）。
save_binary_logs：保存和复制master的二进制日志。
3.2.安装manager
3.2.1.安装依赖包
最好使用ali的镜像源
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum install perl perl-DBI perl-IO-Socket-SSL perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes -y
3.2.2.rpm安装
# rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
Preparing... #################################
package mha4mysql-manager-0.58-0.el7.centos.noarch is already installed
3.2.3.确认安装完成
安装完成后，在/usr/bin/目录下有如下MHA相关文件：
masterha_check_repl
masterha_check_ssh
masterha_check_status
masterha_conf_host
masterha_manager
masterha_master_monitor
masterha_master_switch
masterha_secondary_check
masterha_stop
apply_diff_relay_logs
filter_mysqlbinlog
purge_relay_logs
save_binary_logs
3.3.配置ssh互信
三个节点配置ssh互信登录
# ssh vm1 date
Sat Jan4 10:57:18 CST 2020
# ssh vm2 date
Sat Jan4 10:57:23 CST 2020
# ssh vm3 date
Sat Jan4 10:57:26 CST 2020
4.配置mha
4.1.配置manager
4.1.1.编辑配置文件
# mkdir -p /etc/mha
# cat /etc/mha/mha.cnf
manager_log=/var/log/mha_mgr.log
manager_workdir=/app/mha
master_binlog_dir=/app/mysql/data
password=mysql
ping_interval=1
remote_workdir=/tmp
repl_password=rep
repl_user=rep
ssh_user=root
user=root
master_ip_failover_script="/app/mha/master_ip_failover"
master_ip_online_change_script="/app/mha/master_ip_online_change"
hostname=192.168.56.101
candidate_master=1
master_binlog_dir="/mysql/mysql/data"
hostname=192.168.56.102
candidate_master=1
check_repl_delay=0
master_binlog_dir="/mysql/mysql/data"
hostname=192.168.56.103
no_master=1
check_repl_delay=0
master_binlog_dir="/mysql/mysql/data"
server default段是manager的一些基本配置参数，server1、server2、server3分别对应复制中的master、第一个slave、第二个slave。该文件的语法要求严格，变量值后不要有多余的空格。主要配置项说明如下。
manager_log：设置manager的日志文件。
manager_workdir：设置manager的工作目录。
master_binlog_dir：设置master保存binlog的位置，以便MHA可以找到master的日志，这里的也就是mysql的数据目录。
master_ip_failover_script：设置自动failover时候的切换脚本。
master_ip_online_change_script：设置手动切换时候的切换脚本。
password：设置mysql中root用户的密码。
ping_interval：设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行railover。
remote_workdir：设置远端mysql在发生切换时binlog的保存位置。
repl_password：设置复制用户的密码。
repl_user：设置复制环境中的复制用户名
secondary_check_script：一旦MHA到hdp4的监控之间出现问题，MHA Manager将会尝试从hdp3登录到hdp4。
shutdown_script：设置故障发生后关闭故障主机脚本。该脚本的主要作用是关闭主机放在发生脑裂，这里没有使用。
ssh_user：设置ssh的登录用户名。
user：设置监控用户为root。
candidate_master：设置为候选master。设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave。
check_repl_delay：默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0，MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master。
4.1.2.建立软连接
# cd /app/mysql/bin
# ln -s mysqlbinlog /usr/bin/mysqlbinlog
# ln -s mysql /usr/bin/mysql
4.1.3.从库设置参数
set global relay_log_purge=0;
set global read_only=1;
注意，MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF，采用手动清除relay log的方式。默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。
4.1.4.创建管理账号
所有节点添加账户
grant all on *.* to root@'192.168.56.%' identified by 'mysql';
4.1.5.验证ssh
# masterha_check_ssh--conf=/etc/masterha_default.cnf
Sat Jan4 15:16:23 2020 - Reading default configuration from /etc/masterha_default.cnf..
Sat Jan4 15:16:23 2020 - Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan4 15:16:23 2020 - Reading server configuration from /etc/masterha_default.cnf..
Sat Jan4 15:16:23 2020 - Starting SSH connection tests..
Sat Jan4 15:16:24 2020 -
Sat Jan4 15:16:23 2020 - Connecting via SSH from root@192.168.56.101(192.168.56.101:22) to root@192.168.56.102(192.168.56.102:22)..
Sat Jan4 15:16:24 2020 - ok.
Sat Jan4 15:16:24 2020 - Connecting via SSH from root@192.168.56.101(192.168.56.101:22) to root@192.168.56.103(192.168.56.103:22)..
Sat Jan4 15:16:24 2020 - ok.
Sat Jan4 15:16:25 2020 -
Sat Jan4 15:16:24 2020 - Connecting via SSH from root@192.168.56.102(192.168.56.102:22) to root@192.168.56.101(192.168.56.101:22)..
Sat Jan4 15:16:24 2020 - ok.
Sat Jan4 15:16:24 2020 - Connecting via SSH from root@192.168.56.102(192.168.56.102:22) to root@192.168.56.103(192.168.56.103:22)..
Sat Jan4 15:16:24 2020 - ok.
Sat Jan4 15:16:26 2020 -
Sat Jan4 15:16:24 2020 - Connecting via SSH from root@192.168.56.103(192.168.56.103:22) to root@192.168.56.101(192.168.56.101:22)..
Sat Jan4 15:16:25 2020 - ok.
Sat Jan4 15:16:25 2020 - Connecting via SSH from root@192.168.56.103(192.168.56.103:22) to root@192.168.56.102(192.168.56.102:22)..
Sat Jan4 15:16:25 2020 - ok.
Sat Jan4 15:16:26 2020 - All SSH connection tests passed successfully.
4.1.6.验证repl
# masterha_check_repl --conf=/etc/masterha_default.cnf
Sat Jan4 12:16:46 2020 - Reading default configuration from /etc/masterha_default.cnf..
Sat Jan4 12:16:46 2020 - Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan4 12:16:46 2020 - Reading server configuration from /etc/masterha_default.cnf..
Sat Jan4 12:16:46 2020 - MHA::MasterMonitor version 0.58.
Sat Jan4 12:16:48 2020 - GTID failover mode = 1
Sat Jan4 12:16:48 2020 - Dead Servers:
Sat Jan4 12:16:48 2020 - Alive Servers:
Sat Jan4 12:16:48 2020 - 192.168.56.101(192.168.56.101:3306)
Sat Jan4 12:16:48 2020 - 192.168.56.102(192.168.56.102:3306)
Sat Jan4 12:16:48 2020 - 192.168.56.103(192.168.56.103:3306)
Sat Jan4 12:16:48 2020 - Alive Slaves:
Sat Jan4 12:16:48 2020 - 192.168.56.102(192.168.56.102:3306)Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan4 12:16:48 2020 - GTID ON
Sat Jan4 12:16:48 2020 - Replicating from vm1(192.168.56.101:3306)
Sat Jan4 12:16:48 2020 - Primary candidate for the new Master (candidate_master is set)
Sat Jan4 12:16:48 2020 - 192.168.56.103(192.168.56.103:3306)Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan4 12:16:48 2020 - GTID ON
Sat Jan4 12:16:48 2020 - Replicating from vm1(192.168.56.101:3306)
Sat Jan4 12:16:48 2020 - Current Alive Master: 192.168.56.101(192.168.56.101:3306)
Sat Jan4 12:16:48 2020 - Checking slave configurations..
Sat Jan4 12:16:48 2020 - read_only=1 is not set on slave 192.168.56.102(192.168.56.102:3306).
Sat Jan4 12:16:48 2020 - read_only=1 is not set on slave 192.168.56.103(192.168.56.103:3306).
Sat Jan4 12:16:48 2020 - Checking replication filtering settings..
Sat Jan4 12:16:48 2020 - binlog_do_db= , binlog_ignore_db=
Sat Jan4 12:16:48 2020 - Replication filtering check ok.
Sat Jan4 12:16:48 2020 - GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Sat Jan4 12:16:48 2020 - Checking SSH publickey authentication settings on the current master..
Sat Jan4 12:16:48 2020 - HealthCheck: SSH to 192.168.56.101 is reachable.
Sat Jan4 12:16:48 2020 -
192.168.56.101(192.168.56.101:3306) (current master)
+--192.168.56.102(192.168.56.102:3306)
+--192.168.56.103(192.168.56.103:3306)
Sat Jan4 12:16:48 2020 - Checking replication health on 192.168.56.102..
Sat Jan4 12:16:48 2020 - ok.
Sat Jan4 12:16:48 2020 - Checking replication health on 192.168.56.103..
Sat Jan4 12:16:48 2020 - ok.
Sat Jan4 12:16:48 2020 - master_ip_failover_script is not defined.
Sat Jan4 12:16:48 2020 - shutdown_script is not defined.
Sat Jan4 12:16:48 2020 - Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
5.管理MHA
5.1.启停与查看状态
5.1.1.查看状态
# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default is stopped(2:NOT_RUNNING).
5.1.2.启动mha
# nohupmasterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover&
11615
# nohup: ignoring input and appending output to ‘nohup.out’
# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default (pid:11615) is running(0:PING_OK), master:192.168.56.101
5.1.3.停止mha
# masterha_stop --conf=/etc/masterha_default.cnf
Stopped masterha_default successfully.
+Exit 1 nohup masterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover
5.2.自动故障切换
5.2.1.master配置vip
# ip addr add 192.168.56.104/24 dev enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:43:55:92 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.101/24 brd 192.168.56.255 scope global noprefixroute enp0s8
valid_lft forever preferred_lft forever
inet 192.168.56.104/24 scope global secondary enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::c04a:4f9e:402f:95be/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5.2.2.关闭master
# /etc/init.d/mysql stop
Shutting down MySQL............ SUCCESS!
5.2.3.查看candidate master
mysql> show master status \G;
*************************** 1. row ***************************
File: mysql-bin.000008
Position: 2707
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f17:1-8,
dd39dbef-2e89-11ea-8b4f-080027e53f18:1-4
1 row in set (0.00 sec)
ERROR:
No query specified
5.2.4.vip漂移到vm2
# ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>mtu 1500
inet 10.0.2.15netmask 255.255.255.0broadcast 10.0.2.255
inet6 fe80::9658:182d:b82d:2f95prefixlen 64scopeid 0x20<link>
ether 08:00:27:63:eb:86txqueuelen 1000(Ethernet)
RX packets 914bytes 77673 (75.8 KiB)
RX errors 0dropped 0overruns 0frame 0
TX packets 1030bytes 86063 (84.0 KiB)
TX errors 0dropped 0 overruns 0carrier 0collisions 0
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>mtu 1500
inet 192.168.56.102netmask 255.255.255.0broadcast 192.168.56.255
inet6 fe80::c04a:4f9e:402f:95beprefixlen 64scopeid 0x20<link>
inet6 fe80::14e0:ed28:2938:30fcprefixlen 64scopeid 0x20<link>
ether 08:00:27:b2:57:f8txqueuelen 1000(Ethernet)
RX packets 7077bytes 731716 (714.5 KiB)
RX errors 0dropped 0overruns 0frame 0
TX packets 6034bytes 1458712 (1.3 MiB)
TX errors 0dropped 0 overruns 0carrier 0collisions 0
enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>mtu 1500
inet 192.168.56.104netmask 255.255.255.0broadcast 192.168.56.255
ether 08:00:27:b2:57:f8txqueuelen 1000(Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING>mtu 65536
inet 127.0.0.1netmask 255.0.0.0
inet6 ::1prefixlen 128scopeid 0x10<host>
looptxqueuelen 1000(Local Loopback)
RX packets 2220bytes 129846 (126.8 KiB)
RX errors 0dropped 0overruns 0frame 0
TX packets 2220bytes 129846 (126.8 KiB)
TX errors 0dropped 0 overruns 0carrier 0collisions 0
5.2.5.查看slave
# mysql -uroot -pmysql
mysql: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.28-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.56.102
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000008
Read_Master_Log_Pos: 2707
Relay_Log_File: vm3-relay-bin.000003
Relay_Log_Pos: 1491
Relay_Master_Log_File: mysql-bin.000008
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
5.2.6.mha已停止
# masterha_check_status --conf=/etc/masterha_default.cnf
masterha_default is stopped(2:NOT_RUNNING).
+Done nohup masterha_manager --conf=/etc/masterha_default.cnf --remove_dead_master_conf --ignore_last_failover
5.2.7.原master手动转为slave
# mysql -uroot -pmysql
mysql: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.28-log MySQL Community Server (GPL)
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> change master to
-> master_host='vm2',
-> master_user='rep',
-> master_password='rep',
-> master_auto_position=1;
Query OK, 0 rows affected, 2 warnings (0.01 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status \G:
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: vm2
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000009
Read_Master_Log_Pos: 234
Relay_Log_File: vm1-relay-bin.000002
Relay_Log_Pos: 367
Relay_Master_Log_File: mysql-bin.000008
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 194
Relay_Log_Space: 2154
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 20942
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
Master_UUID: dd39dbef-2e89-11ea-8b4f-080027e53f18
Master_Info_File: /app/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Waiting for semi-sync ACK from slave
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f18:1-4
Executed_Gtid_Set: dd39dbef-2e89-11ea-8b4f-080027e53f17:1-8
Auto_Position: 1
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
5.3.手动故障切换
mha必须时关闭状态，master出现故障时手动切换。
masterha_master_switch --conf=/etc/masterha_default.cnf--master_state=dead--dead_master_host=192.168.56.101 --dead_master_port=3306 --new_master_host=192.168.56.102 --new_master_port=3306 --ignore_last_failover
5.4.手动角色切换
手动切换需要关闭mha监控，主机名最好用ip地址。
masterha_master_switch --conf=/etc/masterha_default.cnf--master_state=alive --new_master_host=192.168.56.102 --orig_master_is_new_slave
# masterha_master_switch --conf=/etc/masterha_default.cnf--master_state=alive --new_master_host=192.168.56.102 --orig_master_is_new_slave
Sat Jan4 18:42:58 2020 - MHA::MasterRotate version 0.58.
Sat Jan4 18:42:58 2020 - Starting online master switch..
Sat Jan4 18:42:58 2020 -
Sat Jan4 18:42:58 2020 - * Phase 1: Configuration Check Phase..
Sat Jan4 18:42:58 2020 -
Sat Jan4 18:42:58 2020 - Reading default configuration from /etc/masterha_default.cnf..
Sat Jan4 18:42:58 2020 - Reading application default configuration from /etc/masterha_default.cnf..
Sat Jan4 18:42:58 2020 - Reading server configuration from /etc/masterha_default.cnf..
Sat Jan4 18:42:59 2020 - Multi-master configuration is detected. Current primary(writable) master is 192.168.56.101(192.168.56.101:3306)
Sat Jan4 18:42:59 2020 - Master configurations are as below:
Master 192.168.56.102(192.168.56.102:3306), replicating from vm1(192.168.56.101:3306), read-only
Master 192.168.56.101(192.168.56.101:3306), replicating from vm2(192.168.56.102:3306)
Sat Jan4 18:42:59 2020 - GTID failover mode = 1
Sat Jan4 18:42:59 2020 - Current Alive Master: 192.168.56.101(192.168.56.101:3306)
Sat Jan4 18:42:59 2020 - Alive Slaves:
Sat Jan4 18:42:59 2020 - 192.168.56.102(192.168.56.102:3306)Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan4 18:42:59 2020 - GTID ON
Sat Jan4 18:42:59 2020 - Replicating from vm1(192.168.56.101:3306)
Sat Jan4 18:42:59 2020 - Primary candidate for the new Master (candidate_master is set)
Sat Jan4 18:42:59 2020 - 192.168.56.103(192.168.56.103:3306)Version=5.7.28-log (oldest major version between slaves) log-bin:enabled
Sat Jan4 18:42:59 2020 - GTID ON
Sat Jan4 18:42:59 2020 - Replicating from vm1(192.168.56.101:3306)
It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.56.101(192.168.56.101:3306)? (YES/no): yes
Sat Jan4 18:43:01 2020 - Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Sat Jan4 18:43:01 2020 - ok.
Sat Jan4 18:43:01 2020 - Checking MHA is not monitoring or doing failover..
Sat Jan4 18:43:01 2020 - Checking replication health on 192.168.56.102..
Sat Jan4 18:43:01 2020 - ok.
Sat Jan4 18:43:01 2020 - Checking replication health on 192.168.56.103..
Sat Jan4 18:43:01 2020 - ok.
Sat Jan4 18:43:01 2020 - 192.168.56.102 can be new master.
Sat Jan4 18:43:01 2020 -
From:
192.168.56.101(192.168.56.101:3306) (current master)
+--192.168.56.102(192.168.56.102:3306)
+--192.168.56.103(192.168.56.103:3306)
To:
192.168.56.102(192.168.56.102:3306) (new master)
+--192.168.56.103(192.168.56.103:3306)
+--192.168.56.101(192.168.56.101:3306)
Starting master switch from 192.168.56.101(192.168.56.101:3306) to 192.168.56.102(192.168.56.102:3306)? (yes/NO): yes
Sat Jan4 18:43:04 2020 - Checking whether 192.168.56.102(192.168.56.102:3306) is ok for the new master..
Sat Jan4 18:43:04 2020 - ok.
Sat Jan4 18:43:04 2020 - ** Phase 1: Configuration Check Phase completed.
Sat Jan4 18:43:04 2020 -
Sat Jan4 18:43:04 2020 - * Phase 2: Rejecting updates Phase..
Sat Jan4 18:43:04 2020 -
Sat Jan4 18:43:04 2020 - Executing master ip online change script to disable write on the current master:
Sat Jan4 18:43:04 2020 - /app/mha/master_ip_online_change --command=stop --orig_master_host=192.168.56.101 --orig_master_ip=192.168.56.101 --orig_master_port=3306 --orig_master_user='root' --new_master_host=192.168.56.102 --new_master_ip=192.168.56.102 --new_master_port=3306 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Sat Jan4 18:43:04 2020 417539 Set read_only on the new master.. ok.
Sat Jan4 18:43:04 2020 425674 drop vip 192.168.56.104..
Cannot find device "ens192"
Sat Jan4 18:43:04 2020 563594 Waiting all running 2 threads are disconnected.. (max 1500 milliseconds)
{'Time' => '1311','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1305','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan4 18:43:05 2020 075826 Waiting all running 2 threads are disconnected.. (max 1000 milliseconds)
{'Time' => '1312','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1306','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan4 18:43:05 2020 576437 Waiting all running 2 threads are disconnected.. (max 500 milliseconds)
{'Time' => '1312','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1306','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan4 18:43:06 2020 089749 Set read_only=1 on the orig master.. ok.
Sat Jan4 18:43:06 2020 107408 Waiting all running 2 queries are disconnected.. (max 500 milliseconds)
{'Time' => '1313','db' => undef,'Id' => '4','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm2:12802'}
{'Time' => '1307','db' => undef,'Id' => '5','User' => 'rep','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'vm3:41491'}
Sat Jan4 18:43:06 2020 595752 Killing all application threads..
Sat Jan4 18:43:06 2020 605704 done.
Sat Jan4 18:43:06 2020 - ok.
Sat Jan4 18:43:06 2020 - Locking all tables on the orig master to reject updates from everybody (including root):
Sat Jan4 18:43:06 2020 - Executing FLUSH TABLES WITH READ LOCK..
Sat Jan4 18:43:06 2020 - ok.
Sat Jan4 18:43:06 2020 - Orig master binlog:pos is mysql-bin.000015:647.
Sat Jan4 18:43:06 2020 - Waiting to execute all relay logs on 192.168.56.102(192.168.56.102:3306)..
Sat Jan4 18:43:06 2020 - master_pos_wait(mysql-bin.000015:647) completed on 192.168.56.102(192.168.56.102:3306). Executed 0 events.
Sat Jan4 18:43:06 2020 - done.
Sat Jan4 18:43:06 2020 - Getting new master's binlog name and position..
Sat Jan4 18:43:06 2020 - mysql-bin.000011:638
Sat Jan4 18:43:06 2020 - All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.102', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Sat Jan4 18:43:06 2020 - Executing master ip online change script to allow write on the new master:
Sat Jan4 18:43:06 2020 - /app/mha/master_ip_online_change --command=start --orig_master_host=192.168.56.101 --orig_master_ip=192.168.56.101 --orig_master_port=3306 --orig_master_user='root' --new_master_host=192.168.56.102 --new_master_ip=192.168.56.102 --new_master_port=3306 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Sat Jan4 18:43:06 2020 751402 Set read_only=0 on the new master.
Sat Jan4 18:43:06 2020 753434Add vip 192.168.56.104 on ens192..
Cannot find device "ens192"
Sat Jan4 18:43:06 2020 - ok.
Sat Jan4 18:43:06 2020 -
Sat Jan4 18:43:06 2020 - * Switching slaves in parallel..
Sat Jan4 18:43:06 2020 -
Sat Jan4 18:43:06 2020 - -- Slave switch on host 192.168.56.103(192.168.56.103:3306) started, pid: 11512
Sat Jan4 18:43:06 2020 -
Sat Jan4 18:43:07 2020 - Log messages from 192.168.56.103 ...
Sat Jan4 18:43:07 2020 -
Sat Jan4 18:43:06 2020 - Waiting to execute all relay logs on 192.168.56.103(192.168.56.103:3306)..
Sat Jan4 18:43:06 2020 - master_pos_wait(mysql-bin.000015:647) completed on 192.168.56.103(192.168.56.103:3306). Executed 0 events.
Sat Jan4 18:43:06 2020 - done.
Sat Jan4 18:43:06 2020 - Resetting slave 192.168.56.103(192.168.56.103:3306) and starting replication from the new master 192.168.56.102(192.168.56.102:3306)..
Sat Jan4 18:43:06 2020 - Executed CHANGE MASTER.
Sat Jan4 18:43:06 2020 - Slave started.
Sat Jan4 18:43:07 2020 - End of log messages from 192.168.56.103 ...
Sat Jan4 18:43:07 2020 -
Sat Jan4 18:43:07 2020 - -- Slave switch on host 192.168.56.103(192.168.56.103:3306) succeeded.
Sat Jan4 18:43:07 2020 - Unlocking all tables on the orig master:
Sat Jan4 18:43:07 2020 - Executing UNLOCK TABLES..
Sat Jan4 18:43:07 2020 - ok.
Sat Jan4 18:43:07 2020 - Starting orig master as a new slave..
Sat Jan4 18:43:07 2020 - Resetting slave 192.168.56.101(192.168.56.101:3306) and starting replication from the new master 192.168.56.102(192.168.56.102:3306)..
Sat Jan4 18:43:07 2020 - Executed CHANGE MASTER.
Sat Jan4 18:43:07 2020 - Slave started.
Sat Jan4 18:43:07 2020 - All new slave servers switched successfully.
Sat Jan4 18:43:07 2020 -
Sat Jan4 18:43:07 2020 - * Phase 5: New master cleanup phase..
Sat Jan4 18:43:07 2020 -
Sat Jan4 18:43:07 2020 - 192.168.56.102: Resetting slave info succeeded.
Sat Jan4 18:43:07 2020 - Switching master to 192.168.56.102(192.168.56.102:3306) completed successfully.
6.附件
6.1.master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.56.104/24';# Virtual IP
my $key = "1";
my $int = "enp0s8";
my $ssh_start_vip = "/sbin/ifconfig $int:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig $int:$key down";
my $arp_effect = "/sbin/arping -Uq -s192.168.56.104 -I $int 192.168.56.254 -c 3"; # Virtual IP and gatway
#my $test = "echo successfull >/tmp/test.txt";
$ssh_user = "root";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s'=> \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i'=> \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
#`ssh $ssh_user\@cluster1 \" $ssh_start_vip \"`;
&status();
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
`ssh $ssh_user\@$new_master_host \" $arp_effect \"`;
# `ssh $ssh_user\@$new_master_host \" $test \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub status() {
print `ssh $ssh_user\@$orig_master_host \" ip add show $int \"`;
}
sub usage {
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_maste
r_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
6.2.master_ip_online_change
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;
my $_tstart;
my $_running_interval = 0.1;
my $vip = "192.168.56.104";
my $if = "ens192";
my (
$command, $orig_master_is_new_slave, $orig_master_host,
$orig_master_ip, $orig_master_port, $orig_master_user,
$orig_master_password, $orig_master_ssh_user, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password, $new_master_ssh_user,
);
GetOptions(
'command=s' => \$command,
'orig_master_is_new_slave' => \$orig_master_is_new_slave,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
'orig_master_password=s' => \$orig_master_password,
'orig_master_ssh_user=s' => \$orig_master_ssh_user,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
'new_master_ssh_user=s' => \$new_master_ssh_user,
);
exit &main();
sub drop_vip {
my $output = `ssh -o ConnectTimeout=15-o ConnectionAttempts=3 $orig_master_host /sbin/ip addr del $vip/32 dev $if`;
}
sub add_vip {
my $output = `ssh -o ConnectTimeout=15-o ConnectionAttempts=3 $new_master_host /sbin/ip addr add $vip/32 dev $if`;
}
sub current_time_us {
my ( $sec, $microsec ) = gettimeofday();
my $curdate = localtime($sec);
return $curdate . " " . sprintf( "%06d", $microsec );
}
sub sleep_until {
my $elapsed = tv_interval($_tstart);
if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
}
}
sub get_threads_util {
my $dbh = shift;
my $my_connection_id = shift;
my $running_time_threshold = shift;
my $type = shift;
$running_time_threshold = 0 unless ($running_time_threshold);
$type = 0 unless ($type);
my @threads;
my $sth = $dbh->prepare("SHOW PROCESSLIST");
$sth->execute();
while ( my $ref = $sth->fetchrow_hashref() ) {
my $id = $ref->{Id};
my $user = $ref->{User};
my $host = $ref->{Host};
my $command = $ref->{Command};
my $state = $ref->{State};
my $query_time = $ref->{Time};
my $info = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user) && $user eq "system user" );
next
if ( defined($command)
&& $command eq "Sleep"
&& defined($query_time)
&& $query_time >= 1 );
if ( $type >= 1 ) {
next if ( defined($command) && $command eq "Sleep" );
next if ( defined($command) && $command eq "Connect" );
}
if ( $type >= 2 ) {
next if ( defined($info) && $info =~ m/^select/i );
next if ( defined($info) && $info =~ m/^show/i );
}
push @threads, $ref;
}
return @threads;
}
sub main {
if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
## Setting read_only=1 on the new master (to avoid accident)
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error(die_on_error)_ or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
print current_time_us() . " Set read_only on the new master.. ";
$new_master_handler->enable_read_only();
if ( $new_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
$new_master_handler->disconnect();
# Connecting to the orig master, die if any database error happens
my $orig_master_handler = new MHA::DBHelper();
$orig_master_handler->connect( $orig_master_ip, $orig_master_port,
$orig_master_user, $orig_master_password, 1 );
## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
$orig_master_handler->disable_log_bin_local();
# print current_time_us() . " Drpping app user on the orig master..\n";
print current_time_us() . " drop vip $vip..\n";
#drop_app_user($orig_master_handler);
&drop_vip();
## Waiting for N * 100 milliseconds so that current connections can exit
my $time_until_read_only = 15;
$_tstart = ;
my @threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_read_only > 0 && $#threads >= 0 ) {
if ( $time_until_read_only % 5 == 0 ) {
printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_read_only * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = ;
$time_until_read_only--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Setting read_only=1 on the current master so that nobody(except SUPER) can write
print current_time_us() . " Set read_only=1 on the orig master.. ";
$orig_master_handler->enable_read_only();
if ( $orig_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
## Waiting for M * 100 milliseconds so that current update queries can complete
my $time_until_kill_threads = 5;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
if ( $time_until_kill_threads % 5 == 0 ) {
printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = ;
$time_until_kill_threads--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}
## Terminating all threads
print current_time_us() . " Killing all application threads..\n";
$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
print current_time_us() . " done.\n";
$orig_master_handler->enable_log_bin_local();
$orig_master_handler->disconnect();
## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database
# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
my $new_master_handler = new MHA::DBHelper();
# args: hostname, port, user, password, raise_error_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
## Set read_only=0 on the new master
$new_master_handler->disable_log_bin_local();
print current_time_us() . " Set read_only=0 on the new master.\n";
$new_master_handler->disable_read_only();
## Creating an app user on the new master
#print current_time_us() . " Creating app user on the new master..\n";
print current_time_us() . "Add vip $vip on $if..\n";
# create_app_user($new_master_handler);
&add_vip();
$new_master_handler->enable_log_bin_local();
$new_master_handler->disconnect();
## Update master ip on the catalog database, etc
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
# do nothing
exit 0;
}
else {
&usage();
exit 1;
}
}
sub usage {
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
die;
}
页:
[1]