MySQL: Pacemaker cannot start the failed master as a new slave?
- by quanta
I'm going to setup failover for MySQL replication (1 master and 1 slave)
follow this guide:
https://github.com/jayjanssen/Percona-Pacemaker-Resource-Agents/blob/master/doc/PRM-setup-guide.rst
Here're the output of crm configure show:
node serving-6192 \
attributes p_mysql_mysql_master_IP="192.168.6.192"
node svr184R-638.localdomain \
attributes p_mysql_mysql_master_IP="192.168.6.38"
primitive p_mysql ocf:percona:mysql \
params config="/etc/my.cnf" pid="/var/run/mysqld/mysqld.pid"
socket="/var/lib/mysql/mysql.sock" replication_user="repl"
replication_passwd="x" test_user="test_user" test_passwd="x" \
op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1" \
op monitor interval="2s" role="Slave" timeout="30s"
OCF_CHECK_LEVEL="1" \
op start interval="0" timeout="120s" \
op stop interval="0" timeout="120s"
primitive writer_vip ocf:heartbeat:IPaddr2 \
params ip="192.168.6.8" cidr_netmask="32" \
op monitor interval="10s" \
meta is-managed="true"
ms ms_MySQL p_mysql \
meta master-max="1" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true" globally-unique="false"
target-role="Master" is-managed="true"
colocation writer_vip_on_master inf: writer_vip ms_MySQL:Master
order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start
property $id="cib-bootstrap-options" \
dc-version="1.0.12-unknown" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
no-quorum-policy="ignore" \
stonith-enabled="false" \
last-lrm-refresh="1341801689"
property $id="mysql_replication" \
p_mysql_REPL_INFO="192.168.6.192|mysql-bin.000006|338"
crm_mon:
Last updated: Mon Jul 9 10:30:01 2012
Stack: openais
Current DC: serving-6192 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ serving-6192 svr184R-638.localdomain ]
Master/Slave Set: ms_MySQL
Masters: [ serving-6192 ]
Slaves: [ svr184R-638.localdomain ]
writer_vip (ocf::heartbeat:IPaddr2): Started serving-6192
Editing /etc/my.cnf on the serving-6192 of wrong syntax to test
failover and it's working fine:
svr184R-638.localdomain being promoted to become the master
writer_vip switch to svr184R-638.localdomain
Current state:
Last updated: Mon Jul 9 10:35:57 2012
Stack: openais
Current DC: serving-6192 - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [ serving-6192 svr184R-638.localdomain ]
Master/Slave Set: ms_MySQL
Masters: [ svr184R-638.localdomain ]
Stopped: [ p_mysql:0 ]
writer_vip (ocf::heartbeat:IPaddr2): Started svr184R-638.localdomain
Failed actions:
p_mysql:0_monitor_5000 (node=serving-6192, call=15, rc=7,
status=complete): not running
p_mysql:0_demote_0 (node=serving-6192, call=22, rc=7,
status=complete): not running
p_mysql:0_start_0 (node=serving-6192, call=26, rc=-2, status=Timed
Out): unknown exec error
Remove the wrong syntax from /etc/my.cnf on serving-6192, and restart
corosync, what I would like to see is serving-6192 was started as a new
slave but it doesn't:
Failed actions:
p_mysql:0_start_0 (node=serving-6192, call=4, rc=1,
status=complete): unknown error
Here're snippet of the logs which I'm suspecting:
Jul 09 10:46:32 serving-6192 lrmd: [7321]: info: rsc:p_mysql:0:4: start
Jul 09 10:46:32 serving-6192 lrmd: [7321]: info: RA output:
(p_mysql:0:start:stderr) Error performing operation: The
object/attribute does not exist
Jul 09 10:46:32 serving-6192 crm_attribute: [7420]: info: Invoked:
/usr/sbin/crm_attribute -N serving-6192 -l reboot --name readable -v 0
The full logs: http://fpaste.org/AyOZ/
The strange thing is I can starting it manually:
export OCF_ROOT=/usr/lib/ocf
export OCF_RESKEY_config="/etc/my.cnf"
export OCF_RESKEY_pid="/var/run/mysqld/mysqld.pid"
export OCF_RESKEY_socket="/var/lib/mysql/mysql.sock"
export OCF_RESKEY_replication_user="repl"
export OCF_RESKEY_replication_passwd="x"
export OCF_RESKEY_test_user="test_user"
export OCF_RESKEY_test_passwd="x"
sh -x /usr/lib/ocf/resource.d/percona/mysql start: http://fpaste.org/RVGh/
Did I make something wrong?
