Skip to main content

Search

Welcome to Tibero GTS!

Building a Non-Stop Synchronization Environment Using ProSync Bidirectional CM FailOver (3/4)

Document Type | Technical Information

Category | Installation

Applicable Product Version | Tibero7, ProSync 4.6.0

Document Number | PINTI005

 

Overview

This guide on ProSync Bidirectional CM FailOver Non-Stop Synchronization Environment Setup` is described across 4 parts.

This document covers Part 3: CM FailOver Configuration.

  • Part 1: ProSync Bidirectional Installation
  • Part 2: ProSync Bidirectional Synchronization Verification 
  • Part 3: CM FailOver Configuration
  • Part 4: CM FailOver Operation

 

 

Method

A Set: Shut Down All ProSync

  • To configure CM FailOver, all ProSync Instances and Agents must be shut down.
su - tibero
prs12
prs_adm

# Execute on one of node1~4
shutdown prs_12_34

# Execute on node1
shutdown agent agent1_12_34

# Execute on node2
shutdown agent agent2_12_34

# Execute on node3
shutdown agent agent3_12_34

# Execute on node4
shutdown agent agent4_12_34

B Set: Shut Down All ProSync

su -tibero
prs34
prs_adm

# Execute on one of node1~4
shutdown prs_34_12

# Execute on node3
shutdown agent agent1_34_12

# Execute on node4
shutdown agent agent2_34_12

# Execute on node1
shutdown agent agent3_34_12

# Execute on node2
shutdown agent agent4_34_12

A Set ProSync CM FailOver Configuration

Register CM FailOver Scripts

  • CM FailOver scripts are used to switch over ProSync Agents registered in CM.
  • Scripts must be used for each ProSync Agent.

Source TAC (node1, node2)

  • The ProSync engine is on a shared file system and configured separately for each node.
prs12

cd $PRS_HOME/bin
ls -lt |grep prs_[0-9]
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_1.sh
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_0.sh

# Script for node1
mv prs_0.sh agent1_12_34.sh
  
# Script for node2
mv prs_1.sh agent2_12_34.sh
node1
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • Set 3 parameters inside the $PRS_HOME/bin/agent1_12_34.sh file.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_12_34
AGENT_ID=agent1_12_34
... omitted ...
node2
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • Set 3 parameters inside the $PRS_HOME/bin/agent2_12_34.sh file.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_12_34
AGENT_ID=agent2_12_34
... omitted ...

Target TAC (node3, node4)

  • The ProSync engine is on a shared file system and configured separately for each node.
prs12

cd $PRS_HOME/bin
ls -lt |grep prs_[0-9]
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_1.sh
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_0.sh

# Script for node1
mv prs_0.sh agent3_12_34.sh

# Script for node2
mv prs_1.sh agent4_12_34.sh
node3
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • Set 3 parameters inside the $PRS_HOME/bin/agent3_12_34.sh file.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_12_34
AGENT_ID=agent3_12_34
... omitted ...
node4
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • $PRS_HOME/bin/agent4_12_34.sh file with 3 parameters configured.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_12_34
AGENT_ID=agent4_12_34
... omitted ...

Register CM Group

Source TAC (node1, node2)

  • Add a GROUP in CM and register node1 and node2 Agents to the Group.
$ cmrctl show all

Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.65/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.66/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
=====================================================================
  • Register the CM Group from one node.
$ cmrctl add group --name grp_src_12_34 --cname cls0 --grptype prosync --failover true
Resource add success! (group, grp_src_12_34)
  • CM Groups are separated into Source and Target.
    • This applies when extraction processes are Source and application processes are Target.
    • If extraction/application processes both run on Source, no need to configure Target.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
.. omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
=====================================================================

Target TAC (node3, node4)

  • Add a GROUP in CM and register node3 and node4 Agents to the Group.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.67/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.68/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
=====================================================================
  • Register the CM Group from one node.
$ cmrctl add group --name grp_tar_12_34 --cname cls0 --grptype prosync --failover true
Resource add success! (group, grp_tar_12_34)
  • CM Groups are separated into Source and Target.
    • This applies when extraction processes are Source and application processes are Target.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
=====================================================================

Register CM Agents

Source TAC (node1, node2)

  • Agents are registered to each node's CM Group separately.
node1
  • Register the agent on node1.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent1_12_34 --grpname grp_src_12_34 --script /share/prosync4_12_34/bin/agent1_12_34.sh
Resource add success! (agent, agent1_12_34)
  • Agents are managed separately by each node's CM.
$ cmrctl show  all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34     DOWN /share/prosync4_12_34/bin/agent1_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
=====================================================================
node2
  • Register the agent on node2.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent2_12_34 --grpname grp_src_12_34 --script /share/prosync4_12_34/bin/agent2_12_34.sh
Resource add success! (agent, agent2_12_34)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34     DOWN /share/prosync4_12_34/bin/agent1_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34     DOWN /share/prosync4_12_34/bin/agent2_12_34.sh, start retry cnt: 0
=====================================================================

Target TAC (node3, node4)

  • Agents are registered to each node's CM Group separately.
node3
  • Register the agent on node3.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent3_12_34 --grpname grp_tar_12_34 --script /share/prosync4_12_34/bin/agent3_12_34.sh
Resource add success! (agent, agent3_12_34)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34     DOWN /share/prosync4_12_34/bin/agent3_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
=====================================================================
node4
  • Register the agent on node4.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent4_12_34 --grpname grp_tar_12_34 --script /share/prosync4_12_34/bin/agent4_12_34.sh
Resource add success! (agent, agent4_12_34)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34     DOWN /share/prosync4_12_34/bin/agent3_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34     DOWN /share/prosync4_12_34/bin/agent4_12_34.sh, start retry cnt: 0
=====================================================================

ProSync Agent Parameters

  • All Agents 1 to 4 add the following parameters for communication with CM.
  • The parameters below communicate with CM and are controlled by CM.
    • Agents are not started manually via ProSync Admin.
USE_CM=Y
PRS_AGENT_PROC_MAX_FAIL_CNT=10
PRS_AGENT_PROC_STARTUP_TIMEOUT=10
PRS_AGENT_PROC_STATUS_REPLY_TIMEOUT=10
PRS_AGENT_PROC_STATUS_INTERVAL=3
PRS_AGENT_CM_PROBE_TIMEOUT=0
ParameterDescription
USE_CM

Sets whether to use CM. If set to Y, it monitors the status of subprocesses and restarts them upon failure. In case of internal failure, it requests stop via cmrctl. (Default: N)

PRS_AGENT_PROC_MAX_FAIL_CNT

Sets the maximum number of retries to correct status upon subprocess failure. (Default: 10)

PRS_AGENT_PROC_STARTUP_TIMEOUT

Sets the maximum wait time (seconds) after a process startup request until it is properly handled. If startup is not properly handled, it waits the set time and retries.

PRS_AGENT_PROC_STATUS_REPLY_TIMEOUT

Sets the maximum wait time (seconds) for no response from the subprocess. If exceeded, it judges the process as hung and restarts it.

PRS_AGENT_PROC_STATUS_INTERVAL

Sets the interval at which the Agent process sends cmrctl check commands to CM.

PRS_AGENT_CM_PROBE_TIMEOUT

Sets the maximum wait time (seconds) for CM status check. If set to 0, waits indefinitely for cmrctl response. If exceeded, it judges CM as hung and cleans up Agent and subprocesses. (Default: 0) โ€ป Responses are received bypassing standard output, so if response delay is a concern, set to 0.

Source: https://docs.tibero.com/prosync/installation/examples/cm-failover#id-3.-agent-parameter

Cluster $CM_SID.tip

  • Part 1: ProSync Bidirectional Installation sets this.
  • If parameters are not applied, shut down CM, apply parameters, then start CM.
# node1 $CM_SID.tip
CM_ID=0

# node2 $CM_SID.tip
CM_ID=1

# node3 $CM_SID.tip
CM_ID=0

# node4 $CM_SID.tip
CM_ID=1

Start ProSync CM FailOver Mode

  • When registered for CM FailOver operation, normal start via prs_adm is not possible.
  • Start/stop is performed via CM commands.

Source TAC (node1, node2)

  • Initially, both Group and Agents are in DOWN state.
  • If Agents are in DEACT state, check the file registered with --script.

โ€ป Caution
If Agents are DEACT, logs with rc: 127 can be found.

1) File permission errors

2) Check if script file uses #!/bin/sh; if so, change to #!/bin/bash

$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34     DOWN /share/prosync4_12_34/bin/agent1_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34     DOWN /share/prosync4_12_34/bin/agent2_12_34.sh, start retry cnt: 0
=====================================================================
  • Start the Group using CM commands.
$ cmrctl start group --name grp_src_12_34
=================================== SUCCESS! ===================================
 Succeeded to request at each node to boot resources under the group(grp_src_12_34).
 Please use "cmrctl show group --name grp_src_12_34" to verify the result.
================================================================================
  • If the Group starts successfully, Agents on node1 and node2 will show UP status.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.65/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/agent1_12_34.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.66/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/agent2_12_34.sh, start retry cnt: 0
=====================================================================
  • Check ProSync Instance status via ProSync Admin.
$ prs_adm
Admin> status
prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running
prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running
prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is stopped
prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is stopped

Instance ID: [prs_12_34]
prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601)
Agent Process for proc[Apply], num[1] is not running.
prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
  • Check ProSync Agent and Instance processes on node1 and node2.
# node1
$ ps -ef |grep 12_34 |grep -v grep
tibero     1712460       1  0 03:26 ?        00:00:01 prs_agent agent1_12_34
tibero     1712564 1712460  0 03:26 ?        00:00:00 prs_ext prs_12_34 1 agent1_12_34
tibero     1712568 1712460  0 03:26 ?        00:00:00 prs_llob prs_12_34 1 agent1_12_34

# node2
$  ps -ef |grep 12_34 |grep -v grep
tibero     1704500       1  0 03:26 ?        00:00:01 prs_agent agent2_12_34
tibero     1704556 1704500  0 03:26 ?        00:00:00 prs_ext prs_12_34 2 agent2_12_34

Target TAC (node3, node4)

  • Initially, both Group and Agents are in DOWN state.
  • If Agents are in DEACT state, check the file registered with --script.

โ€ป Caution
If Agents are DEACT, logs with rc: 127 can be found.

1) File permission errors

2) Check if script file uses #!/bin/sh; if so, change to #!/bin/bash

$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34     DOWN /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34     DOWN /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
=====================================================================
  • Start the Group using CM commands.
$ cmrctl start group --name grp_tar_12_34
=================================== SUCCESS! ===================================
 Succeeded to request at each node to boot resources under the group(grp_tar_12_34).
 Please use "cmrctl show group --name grp_tar_12_34" to verify the result.
================================================================================
  • If the Group starts successfully, Agents on node3 and node4 will show UP status.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
=====================================================================
  • Check ProSync Instance status via ProSync Admin.
$ prs_adm
Admin> status
prs_agent ID: agent1_12_34, HOST: 10.10.10.65, PORT: 7600, CM_GROUP: grp_src_12_34, CM_ID: 0 is running
prs_agent ID: agent2_12_34, HOST: 10.10.10.66, PORT: 7601, CM_GROUP: grp_src_12_34, CM_ID: 1 is running
prs_agent ID: agent3_12_34, HOST: 10.10.10.67, PORT: 7602, CM_GROUP: grp_tar_12_34, CM_ID: 0 is running
prs_agent ID: agent4_12_34, HOST: 10.10.10.68, PORT: 7603, CM_GROUP: grp_tar_12_34, CM_ID: 1 is running

Instance ID: [prs_12_34]
prs_12_34_ext1 (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
prs_12_34_ext2 (2) is running (prs_agent ID : agent2_12_34, HOST: 10.10.10.66, PORT: 7601)
prs_12_34_apply1 (1) is running (prs_agent ID : agent3_12_34, HOST: 10.10.10.67, PORT: 7602)
prs_12_34_llob (1) is running (prs_agent ID : agent1_12_34, HOST: 10.10.10.65, PORT: 7600)
  • Check ProSync Agent and Instance processes on node3 and node4.
$  ps -ef |grep 12_34 |grep -v grep
tibero     1719532       1  1 03:56 ?        00:00:00 prs_agent agent3_12_34
tibero     1719610 1719532  2 03:56 ?        00:00:00 prs_apply prs_12_34 1 agent3_12_34

$ ps -ef |grep 12_34 |grep -v grep
tibero     1717765       1  2 03:56 ?        00:00:00 prs_agent agent4_12_34

B Set ProSync CM FailOver Configuration

Register CM FailOver Scripts

  • CM FailOver scripts are used to switch over ProSync Agents registered in CM.
  • Scripts must be used for each ProSync Agent.

Source TAC (node3, node4)

  • The ProSync engine is on a shared file system and configured separately for each node.
prs34

cd $PRS_HOME/bin
ls -lt |grep prs_[0-9]
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_1.sh
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_0.sh

# Script for node3
mv prs_0.sh agent1_34_12.sh
  
# Script for node4
mv prs_1.sh agent2_34_12.sh
node3
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • $PRS_HOME/bin/agent1_34_12.sh file with 3 parameters configured.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_34_12
AGENT_ID=agent1_34_12
... omitted ...
node4
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • $PRS_HOME/bin/agent2_34_12.sh file with 3 parameters configured.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_34_12
AGENT_ID=agent2_34_12
... omitted ...

Target TAC (node1, node2)

  • The ProSync engine is on a shared file system and configured separately for each node.
prs34

cd $PRS_HOME/bin
ls -lt |grep prs_[0-9]
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_1.sh
-rwxr-xr-x 1 tibero dba     1148 Mar 17 05:33 prs_0.sh

# Script for node1
mv prs_0.sh agent3_34_12.sh

# Script for node2
mv prs_1.sh agent4_34_12.sh
node1
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • $PRS_HOME/bin/agent3_34_12.sh file with 3 parameters configured.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_34_12
AGENT_ID=agent3_34_12
... omitted ...
node2
  • When CM FailOver occurs, set the script to send commands to the Agent.
    • $PRS_HOME/bin/agent4_34_12.sh file with 3 parameters configured.
#!/bin/bash

export TB_HOME=/tibero/tibero7
export PRS_HOME=/share/prosync4_34_12
AGENT_ID=agent4_34_12
... omitted ...

Register CM Group

Source TAC (node3, node4)

  • Add a GROUP in CM and register node3 and node4 Agents to the Group.
    • Be careful not to confuse with A Set's CM Groups and Agents.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.67/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.68/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
=====================================================================
  • Register the CM Group from one node.
$ cmrctl add group --name grp_src_34_12 --cname cls0 --grptype prosync --failover true
Resource add success! (group, grp_src_34_12)
  • CM Groups are separated into Source and Target.
    • This applies when extraction processes are Source and application processes are Target.
    • If extraction/application processes both run on Source, no need to configure Target.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/agent4_12_34.sh, start retry cnt: 0
=====================================================================

Target TAC (node1, node2)

  • Add a GROUP in CM and register node1 and node2 Agents to the Group.
    • Be careful not to confuse with A Set's CM Groups and Agents.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.65/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.66/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db12       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
=====================================================================
  • Register the CM Group from one node.
$ cmrctl add group --name grp_tar_34_12 --cname cls0 --grptype prosync --failover true
Resource add success! (group, grp_tar_34_12)
  • CM Groups are separated into Source and Target.
    • This applies when extraction processes are Source and application processes are Target.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
=====================================================================

Register CM Agents

Source TAC (node3, node4)

  • Agents are registered to each node's CM Group separately.
node3
  • Register the agent on node3.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent1_34_12 --grpname grp_src_34_12 --script /share/prosync4_34_12/bin/agent1_34_12.sh
Resource add success! (agent, agent1_34_12)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
       cls0    agent   agent1_34_12     DOWN /share/prosync4_34_12/bin/agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
=====================================================================
node4
  • Register the agent on node4.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent2_34_12 --grpname grp_src_34_12 --script /share/prosync4_34_12/bin/agent2_34_12.sh
Resource add success! (agent, agent2_34_12)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
       cls0    agent   agent1_34_12    DOWN  /share/prosync4_34_12/bin/agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
       cls0    agent   agent2_34_12     DOWN /share/prosync4_34_12/bin/agent2_34_12.sh, start retry cnt: 0
=====================================================================

Target TAC (node1, node2)

  • Agents are registered to each node's CM Group separately.
node1
  • Register the agent on node1.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent3_34_12 --grpname grp_tar_34_12 --script /share/prosync4_12_34/bin/agent3_34_12.sh
Resource add success! (agent, agent3_34_12)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
       cls0    agent   agent3_34_12     DOWN /share/prosync4_34_12/bin/agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
=====================================================================
node2
  • Register the agent on node2.
  • Add the following options when registering the Agent:
    • --name Agent name
    • --grpname Group name
    • --script Path to the CM FailOver script set earlier
    • --failover true
$ cmrctl add agent --name agent4_34_12 --grpname grp_tar_34_12 --script /share/prosync4_34_12/bin/agent4_34_12.sh
Resource add success! (agent, agent4_34_12)
  • Agents are managed separately by each node's CM.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
       cls0    agent   agent3_34_12     DOWN /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_34_12       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
       cls0    agent   agent4_34_12     DOWN /share/prosync4_34_12/bin/agent4_34_12.sh, start retry cnt: 0
=====================================================================

ProSync Agent Parameters

  • All Agents 1 to 4 add the following parameters for communication with CM.
  • The parameters below communicate with CM and are controlled by CM.
    • Agents are not started manually via ProSync Admin.
USE_CM=Y
PRS_AGENT_PROC_MAX_FAIL_CNT=10
PRS_AGENT_PROC_STARTUP_TIMEOUT=10
PRS_AGENT_PROC_STATUS_REPLY_TIMEOUT=10
PRS_AGENT_PROC_STATUS_INTERVAL=3
PRS_AGENT_CM_PROBE_TIMEOUT=0
ParameterDescription
USE_CM

Sets whether to use CM. If set to Y, it monitors the status of subprocesses and restarts them upon failure. In case of internal failure, it requests stop via cmrctl. (Default: N)

PRS_AGENT_PROC_MAX_FAIL_CNT

Sets the maximum number of retries to correct status upon subprocess failure. (Default: 10)

PRS_AGENT_PROC_STARTUP_TIMEOUT

Sets the maximum wait time (seconds) after a process startup request until it is properly handled. If startup is not properly handled, it waits the set time and retries.

PRS_AGENT_PROC_STATUS_REPLY_TIMEOUT

Sets the maximum wait time (seconds) for no response from the subprocess. If exceeded, it judges the process as hung and restarts it.

PRS_AGENT_PROC_STATUS_INTERVAL

Sets the interval at which the Agent process sends cmrctl check commands to CM.

PRS_AGENT_CM_PROBE_TIMEOUT

Sets the maximum wait time (seconds) for CM status check. If set to 0, waits indefinitely for cmrctl response. If exceeded, it judges CM as hung and cleans up Agent and subprocesses. (Default: 0) โ€ป Responses are received bypassing standard output, so if response delay is a concern, set to 0.

Source: https://docs.tibero.com/prosync/installation/examples/cm-failover#id-3.-agent-parameter

Cluster $CM_SID.tip

  • Part 1: ProSync Bidirectional Installation sets this.
  • If parameters are not applied, shut down CM, apply parameters, then start CM.
# node1 $CM_SID.tip
CM_ID=0

# node2 $CM_SID.tip
CM_ID=1

# node3 $CM_SID.tip
CM_ID=0

# node4 $CM_SID.tip
CM_ID=1

Start ProSync CM FailOver Mode

  • When registered for CM FailOver operation, normal start via prs_adm is not possible.
  • Start/stop is performed via CM commands.

Source TAC (node3, node4)

  • Initially, both Group and Agents are in DOWN state.
  • If Agents are in DEACT state, check the file registered with --script.

โ€ป Caution
If Agents are DEACT, logs with rc: 127 can be found.

1) File permission errors

2) Check if script file uses #!/bin/sh; if so, change to #!/bin/bash

$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
       cls0    agent   agent1_34_12     DOWN /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
       cls0    agent   agent2_34_12     DOWN /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================
  • Start the Group using CM commands.
$ cmrctl start group --name grp_src_34_12
=================================== SUCCESS! ===================================
 Succeeded to request at each node to boot resources under the group(grp_src_34_12).
 Please use "cmrctl show group --name grp_src_34_12" to verify the result.
================================================================================
  • If the Group starts successfully, Agents on node3 and node4 will show UP status.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net0       UP (private) 10.10.10.67/29000
     COMMON  cluster           cls0       UP inc: net0, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db340 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12       UP type: prosync (failover: ON)
       cls0    agent   agent3_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent3.sh, start retry cnt: 0
       cls0    agent   agent1_34_12       UP /share/prosync4_34_12/bin/prs_agent1_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
     COMMON  network           net1       UP (private) 10.10.10.68/29000
     COMMON  cluster           cls0       UP inc: net1, pub: N/A
       cls0     file         cls0:0       UP /share/tac/CMFILE/CMFILE
       cls0  service           db34       UP Database, Active Cluster (auto-restart: OFF)
       cls0       db          db341 UP(NRML) db34, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_tar_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_src_34_12       UP type: prosync (failover: ON)
       cls0    agent   agent4_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent4.sh, start retry cnt: 0
       cls0    agent   agent2_34_12       UP /share/prosync4_34_12/bin/prs_agent2_34_12.sh, start retry cnt: 0
=====================================================================
  • Check ProSync Instance status via ProSync Admin.
$ prs_adm
Admin> status
prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running
prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running
prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is stopped
prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is stopped

Instance ID: [prs_34_12]
prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601)
Agent Process for proc[Apply], num[1] is not running.
prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
  • Check ProSync Agent and Instance processes on node3 and node4.
# node3
$ ps -ef |grep 34_12 |grep -v grep
tibero     1735739       1  1 04:26 ?        00:00:00 prs_agent agent1_34_12
tibero     1735812 1735739  0 04:26 ?        00:00:00 prs_ext prs_34_12 1 agent1_34_12
tibero     1735816 1735739  0 04:26 ?        00:00:00 prs_llob prs_34_12 1 agent1_34_12

# node4
$  ps -ef |grep 34_12 |grep -v grep
tibero     1733840       1  1 04:26 ?        00:00:00 prs_agent agent2_34_12
tibero     1733891 1733840  0 04:26 ?        00:00:00 prs_ext prs_34_12 2 agent2_34_12

Target TAC (node1, node2)

  • Initially, both Group and Agents are in DOWN state.
  • If Agents are in DEACT state, check the file registered with --script.

โ€ป Caution
If Agents are DEACT, logs with rc: 127 can be found.

1) File permission errors

2) Check if script file uses #!/bin/sh; if so, change to #!/bin/bash

$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
       cls0    agent   agent3_34_12     DOWN /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12     DOWN type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
       cls0    agent   agent4_34_12     DOWN /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================
  • Start the Group using CM commands.
$ cmrctl start group --name grp_tar_34_12
=================================== SUCCESS! ===================================
 Succeeded to request at each node to boot resources under the group(grp_tar_34_12).
 Please use "cmrctl show group --name grp_tar_34_12" to verify the result.
================================================================================
  • If the Group starts successfully, Agents on node3 and node4 will show UP status.
$ cmrctl show all
Resource List of Node cm0
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db120 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12       UP type: prosync (failover: ON)
       cls0    agent   agent1_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent1.sh, start retry cnt: 0
       cls0    agent   agent3_34_12       UP /share/prosync4_34_12/bin/prs_agent3_34_12.sh, start retry cnt: 0
=====================================================================
Resource List of Node cm1
=====================================================================
  CLUSTER     TYPE        NAME       STATUS           DETAIL
----------- -------- -------------- -------- ------------------------
... omitted ...
       cls0       db          db121 UP(NRML) db12, /tibero/tibero7, failed retry cnt: 0
       cls0    group  grp_src_12_34       UP type: prosync (failover: ON)
       cls0    group  grp_tar_34_12       UP type: prosync (failover: ON)
       cls0    agent   agent2_12_34       UP /share/prosync4_12_34/bin/prs_12_34_agent2.sh, start retry cnt: 0
       cls0    agent   agent4_34_12       UP /share/prosync4_34_12/bin/prs_agent4_34_12.sh, start retry cnt: 0
=====================================================================
  • Check ProSync Instance status via ProSync Admin.
$ prs_adm
Admin> status
prs_agent ID: agent1_34_12, HOST: 10.10.10.67, PORT: 6600, CM_GROUP: grp_src_34_12, CM_ID: 0 is running
prs_agent ID: agent2_34_12, HOST: 10.10.10.68, PORT: 6601, CM_GROUP: grp_src_34_12, CM_ID: 1 is running
prs_agent ID: agent3_34_12, HOST: 10.10.10.65, PORT: 6602, CM_GROUP: grp_tar_34_12, CM_ID: 0 is running
prs_agent ID: agent4_34_12, HOST: 10.10.10.66, PORT: 6603, CM_GROUP: grp_tar_34_12, CM_ID: 1 is running

Instance ID: [prs_34_12]
prs_34_12_ext1 (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
prs_34_12_ext2 (2) is running (prs_agent ID : agent2_34_12, HOST: 10.10.10.68, PORT: 6601)
prs_34_12_apply1 (1) is running (prs_agent ID : agent3_34_12, HOST: 10.10.10.65, PORT: 6602)
prs_34_12_llob (1) is running (prs_agent ID : agent1_34_12, HOST: 10.10.10.67, PORT: 6600)
  • Check ProSync Agent and Instance processes on node1 and node2.
# node1
$ ps -ef |grep 34_12 |grep -v grep
tibero     1785507       1  1 04:28 ?        00:00:00 prs_agent agent3_34_12
tibero     1785637 1785507  1 04:28 ?        00:00:00 prs_apply prs_34_12 1 agent3_34_12

# node2
$ ps -ef |grep 34_12 |grep -v grep
tibero     1737778       1  1 04:28 ?        00:00:00 prs_agent agent4_34_12