Document Type | Technical Information
Category | Installation
Applicable Product Versions | Tibero7, ProSync 4.6.0
Document Number | PINTI006
Overview
This guide on building a non-stop synchronization environment using ProSync bidirectional CM FailOver` has been described over a total of four parts.
This document covers part 4: CM FailOver operation.
- Part 1: ProSync Bidirectional Installation
- Part 2: ProSync Bidirectional Synchronization Verification
- Part 3: CM FailOver Configuration
- Part 4: CM FailOver Operation
Method
Check A, B Set ProSync CM FailOver Configuration
| Node | Agent Name | ProSync Process |
|---|---|---|
| node1 | agent1_12_34 | ext1, llob |
| node2 | agent2_12_34 | ext2, llob(standby) |
| node3 | agent3_12_34 | apply |
| node4 | agent4_12_34 | apply(standby) |
| Node | Agent Name | ProSync Process |
|---|---|---|
| node3 | agent1_34_12 | ext1, llob |
| node4 | agent2_34_12 | ext2, llob(standby) |
| node1 | agent3_34_12 | apply |
| node2 | agent4_34_12 | apply(standby) |
node1, node2 TAC
- Check CM Group and Agent
- A Set CM ProSync Group: grp_src_12_34
- B Set CM ProSync Group: grp_tar_34_12
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
COMMON network net0 UP (private) 10.10.10.65/29000
COMMON cluster cls0 UP inc: net0, pub: N/A
cls0 file cls0:0 UP /share/tac/CMFILE/CMFILE
cls0 service db12 UP Database, Active Cluster (auto-restart: OFF)
cls0 db db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent1_12_34 UP /share/prosync4_12_34/bin/prs_agent1_12_34.sh, start retry cnt: 0
cls0 agent agent3_34_12 UP /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
COMMON network net1 UP (private) 10.10.10.66/29000
COMMON cluster cls0 UP inc: net1, pub: N/A
cls0 file cls0:0 UP /share/tac/CMFILE/CMFILE
cls0 service db12 UP Database, Active Cluster (auto-restart: OFF)
cls0 db db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent2_12_34 UP /share/prosync4_12_34/bin/prs_agent2_12_34.sh, start retry cnt: 0
cls0 agent agent4_34_12 UP /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================
$ prs12
$ prs_adm
Admin status
prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running
prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running
prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is running
prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running
Instance ID: [prs_12_34]
prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601)
prs_12_34_apply1 (1) is running (prs_agent ID : agent3_12_34, HOST: 10.10.10.67, PORT: 7602)
prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)node3, node4 TAC
- Check CM Group and Agent
- A Set CM ProSync Group: grp_tar_12_34
- B Set CM ProSync Group: grp_src_34_12
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
COMMON network net0 UP (private) 10.10.10.67/29000
COMMON cluster cls0 UP inc: net0, pub: N/A
cls0 file cls0:0 UP /share/tac/CMFILE/CMFILE
cls0 service db34 UP Database, Active Cluster (auto-restart: OFF)
cls0 db db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent3_12_34 UP /share/prosync4_12_34/bin/prs_agent3_12_34.sh, start retry cnt: 0
cls0 agent agent1_34_12 UP /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
COMMON network net1 UP (private) 10.10.10.68/29000
COMMON cluster cls0 UP inc: net1, pub: N/A
cls0 file cls0:0 UP /share/tac/CMFILE/CMFILE
cls0 service db34 UP Database, Active Cluster (auto-restart: OFF)
cls0 db db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent4_12_34 UP /share/prosync4_12_34/bin/prs_agent4_12_34.sh, start retry cnt: 0
cls0 agent agent2_34_12 UP /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================
$ prs34
$ prs_adm
Admin status
prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running
prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running
prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is running
prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running
Instance ID: [prs_34_12]
prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601)
prs_34_12_apply1 (1) is running (prs_agent ID : agent3_34_12, HOST: 10.10.10.65, PORT: 6602)
prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)Operation
ProSync CM FailOver Operation (node1, node2)
Perform node1 CM FailOver
- Forcefully shut down CM on node1.
- After CM shutdown, verify that ProSync Agent and Instance processes are no longer running.
$ cmrctl show
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent1_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
cls0 agent agent3_34_12 UP /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
$ kill -9 ${CM PID}
$ ps -ef |grep prs |grep -v grep
$Verify node2 CM FailOver
- CM on node2 is operating normally.
- Verify ProSync Agent and Instance processes on node2.
- Confirm that the llob process has successfully failed over according to A Set ProSync.
- Confirm that the apply process has successfully failed over according to B Set ProSync.
$ cmrctl show all
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent2_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
cls0 agent agent4_34_12 UP /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================
$ prs12
$ prs_adm
Admin status
prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is stopped
prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running
prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is running
prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running
Instance ID: [prs_12_34]
Agent Process for proc[Extract], num[1] is not running.
prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601)
prs_12_34_apply1 (1) is running (prs_agent ID : agent3_12_34, HOST: 10.10.10.67, PORT: 7602)
prs_12_34_llob (1) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7600)
$ prs34
$ prs_adm
Admin status
prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running
prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running
prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is stopped
prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running
Instance ID: [prs_34_12]
prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601)
prs_34_12_apply1 (1) is running (prs_agent ID : agent4_34_12, HOST: 10.10.10.66, PORT: 6602)
prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
$ ps -ef |grep prs |grep -v grep
tibero 1704500 1 0 03:26 ? 00:00:06 prs_agent agent2_12_34
tibero 1704556 1704500 0 03:26 ? 00:00:19 prs_ext prs_12_34 2 agent2_12_34
tibero 1737778 1 0 04:28 ? 00:00:04 prs_agent agent4_34_12
tibero 1853638 1737778 0 06:59 ? 00:00:00 prs_apply prs_34_12 1 agent4_34_12
tibero 1853643 1704500 0 06:59 ? 00:00:00 prs_llob prs_12_34 1 agent2_12_34Restore node1 CM
- Restore CM on node1.
$ tbcm -b $ tbdown clean $ tbboot
- When checking CM status on node1 and node2, confirm that all agents on node1 are down.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent1_12_34 DOWN /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
cls0 agent agent3_34_12 DOWN /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent2_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
cls0 agent agent4_34_12 UP /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================- When agents have failed over to the opposite node due to CM FailOver, manual FailBack is required.
$ cmrctl start agent --name agent1_12_34 MSG SENDING SUCCESS! $ cmrctl start agent --name agent3_34_12 MSG SENDING SUCCESS!
- After starting the agents, verify that they have started properly.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent1_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
cls0 agent agent3_34_12 UP /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_src_12_34 UP type: prosync (failover: ON)
cls0 group grp_tar_34_12 UP type: prosync (failover: ON)
cls0 agent agent2_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
cls0 agent agent4_34_12 UP /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================- After performing CM FailBack, verify that the processes are properly assigned to the corresponding agents.
$ prs12 $ prs_adm Admin status prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is running prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running Instance ID: [prs_12_34] prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600) prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601) prs_12_34_apply1 (1) is running (prs_agent ID : agent3_12_34, HOST: 10.10.10.67, PORT: 7602) prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600) $ prs34 $ prs_adm Admin status prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is running prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running Instance ID: [prs_34_12] prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600) prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601) prs_34_12_apply1 (1) is running (prs_agent ID : agent3_34_12, HOST: 10.10.10.65, PORT: 6602) prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600) # node1 $ ps -ef |grep prs |grep -v grep tibero 2091029 1 0 07:03 ? 00:00:00 prs_agent agent1_12_34 tibero 2091085 2091029 0 07:03 ? 00:00:00 prs_ext prs_12_34 1 agent1_12_34 tibero 2091089 2091029 0 07:03 ? 00:00:00 prs_llob prs_12_34 1 agent1_12_34 tibero 2091937 1 1 07:04 ? 00:00:00 prs_agent agent3_34_12 tibero 2092004 2091937 0 07:04 ? 00:00:00 prs_apply prs_34_12 1 agent3_34_12 # node2 $ ps -ef |grep prs |grep -v grep tibero 1704500 1 0 03:26 ? 00:00:06 prs_agent agent2_12_34 tibero 1704556 1704500 0 03:26 ? 00:00:20 prs_ext prs_12_34 2 agent2_12_34 tibero 1737778 1 0 04:28 ? 00:00:04 prs_agent agent4_34_12
ProSync CM FailOver Operation (node3, node4)
Perform node3 CM FailOver
$ cmrctl show
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent3_12_34 UP /share/prosync4_12_34/bin/prs_agent3_12_34.sh, start retry cnt: 0
cls0 agent agent1_34_12 UP /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
$ kill -9 ${CM PID}
$ ps -ef |grep prs |grep -v grep
$Verify node4 CM FailOver
- CM on node4 is operating normally.
- Verify ProSync Agent and Instance processes on node4.
- Confirm that the apply process has successfully failed over according to A Set ProSync.
- Confirm that the llob process has successfully failed over according to B Set ProSync.
$ cmrctl show all
Node cm0 is not 'UP' Current status DOWN
If this node has been abnormally terminated, try again after the expire timeout(300 sec)
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent4_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
cls0 agent agent2_34_12 UP /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================
$ prs12
$ prs_adm
Admin status
prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running
prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running
prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is stopped
prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running
Instance ID: [prs_12_34]
prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601)
prs_12_34_apply1 (1) is running (prs_agent ID : agent4_12_34, HOST: 10.10.10.68, PORT: 7602)
prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
$ prs34
$ prs_adm
Admin status
prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is stopped
prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running
prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is running
prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running
Instance ID: [prs_34_12]
Agent Process for proc[Extract], num[1] is not running.
prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601)
prs_34_12_apply1 (1) is running (prs_agent ID : agent3_34_12, HOST: 10.10.10.65, PORT: 6602)
prs_34_12_llob (1) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6600)
$ ps -ef |grep prs |grep -v grep
tibero 3921 661 0 Mar19 pts/0 00:00:00 prs_adm
tibero 1717765 1 0 03:56 ? 00:00:05 prs_agent agent4_12_34
tibero 1733840 1 0 04:26 ? 00:00:05 prs_agent agent2_34_12
tibero 1733891 1733840 0 04:26 ? 00:00:17 prs_ext prs_34_12 2 agent2_34_12
tibero 1856977 1733840 0 07:10 ? 00:00:00 prs_llob prs_34_12 1 agent2_34_12
tibero 1856980 1717765 0 07:10 ? 00:00:00 prs_apply prs_12_34 1 agent4_12_34Restore node3 CM
- Restore CM on node3.
$ tbcm -b $ tbdown clean $ tbboot
- When checking CM status on node3 and node4, confirm that all agents on node3 are down.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent3_12_34 DOWN /share/prosync4_12_34/bin/prs_agent3_12_34.sh, start retry cnt: 0
cls0 agent agent1_34_12 DOWN /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent4_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
cls0 agent agent2_34_12 UP /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================- When agents have failed over to the opposite node due to CM FailOver, manual FailBack is required.
$ cmrctl start agent --name agent3_12_34 MSG SENDING SUCCESS! $ cmrctl start agent --name agent1_34_12 MSG SENDING SUCCESS!
- After starting the agents, verify that they have started properly.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent3_12_34 UP /share/prosync4_12_34/bin/prs_agent3_12_34.sh, start retry cnt: 0
cls0 agent agent1_34_12 UP /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
CLUSTER TYPE NAME STATUS DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
cls0 db db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
cls0 group grp_tar_12_34 UP type: prosync (failover: ON)
cls0 group grp_src_34_12 UP type: prosync (failover: ON)
cls0 agent agent4_12_34 UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
cls0 agent agent2_34_12 UP /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================- After performing CM FailBack, verify that the processes are properly assigned to the corresponding agents.
$ prs12 $ prs_adm Admin status prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is running prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running Instance ID: [prs_12_34] prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600) prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601) prs_12_34_apply1 (1) is running (prs_agent ID : agent3_12_34, HOST: 10.10.10.67, PORT: 7602) prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600) $ prs34 $ prs_adm Admin status prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is running prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running Instance ID: [prs_34_12] prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600) prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601) prs_34_12_apply1 (1) is running (prs_agent ID : agent3_34_12, HOST: 10.10.10.65, PORT: 6602) prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600) # node3 $ ps -ef |grep prs |grep -v grep tibero 2064388 1 1 07:11 ? 00:00:00 prs_agent agent3_12_34 tibero 2064444 2064388 0 07:11 ? 00:00:00 prs_apply prs_12_34 1 agent3_12_34 tibero 2064558 1 1 07:12 ? 00:00:00 prs_agent agent1_34_12 tibero 2064617 2064558 0 07:12 ? 00:00:00 prs_ext prs_34_12 1 agent1_34_12 tibero 2064621 2064558 0 07:12 ? 00:00:00 prs_llob prs_34_12 1 agent1_34_12 # node4 $ ps -ef |grep prs |grep -v grep tibero 3921 661 0 Mar19 pts/0 00:00:00 prs_adm tibero 1717765 1 0 03:56 ? 00:00:05 prs_agent agent4_12_34 tibero 1733840 1 0 04:26 ? 00:00:05 prs_agent agent2_34_12 tibero 1733891 1733840 0 04:26 ? 00:00:17 prs_ext prs_34_12 2 agent2_34_12