Skip to main content

Search

Welcome to Tibero GTS!

tpmagent Startup Server Response Delayed.. Restart Process.. Error Occurred

Document Type | Troubleshooting

Category | Administration

Applicable Product Version | 7FS02PS

Document Number | TADTS057

 

Issue

An error "Server Response Delayed.. Restart Process.." occurred when starting the tpmagent of the monitoring DB registered in the SYSMASTER DB.
 
1. tpmagent log
[ERROR] [2024/8/26 15:35:17] Get avaiable message idx fail, maybe SysMaster cannot receive message
[WARNING] [2024/8/26 15:35:17] Fail to find next available idx, next alloc message need to find available idx
[ERROR] [2024/8/26 15:35:30] There's no Server Response... Restart Process

 

2. Check SysMaster DB Status
broker, schema-registry, and zookeeper do not start properly and show status as Restarting.
NAME              IMAGE                                   COMMAND                    SERVICE           CREATED        STATUS                         
analyzer          sysmaster-db-analyzer:8.1               "/bin/sh -c 'if [ ! โ€ฆ"    analyzer          14 hours ago   Up 14 hours                                       
broker            sysmaster-db-kafka-loggable:8.1         "/etc/confluent/dockโ€ฆ"    broker            14 hours ago   Restarting (1) 11 seconds ago                                       
client            sysmaster-db-client:8.1                 "/bin/sh -c 'sh /staโ€ฆ"    client            14 hours ago   Up 14 hours                                                         
collector         sysmaster-db-collector:8.1              "/bin/sh -c 'if [ ! โ€ฆ"    collector         14 hours ago   Up 14 hours                                                         
metadb            tmaxopensql/postgres:14.6               "/tmp/settings/entryโ€ฆ"    metadb            14 hours ago   Up 14 hours                                                         
repodb            tmaxopensql/postgres:14.6               "/tmp/settings/entryโ€ฆ"    repodb            14 hours ago   Up 14 hours                                                         
schema-registry   confluentinc/cp-schema-registry:5.4.0   "/etc/confluent/dockโ€ฆ"    schema-registry   14 hours ago   Restarting (1) 35 seconds ago                                      
sdm               sysmaster-db-sdm:8.1                    "/bin/sh -c 'if [ \"$โ€ฆ"   sdm               14 hours ago   Up 14 hours                                                         
tibero-master     sysmaster-db-tibero-master:8.1          "/bin/sh -c 'if [ ! โ€ฆ"    tibero-master     14 hours ago   Up 14 hours                     
zookeeper         sysmaster-db-zookeeper-loggable:8.1     "/etc/confluent/dockโ€ฆ"    zookeeper         14 hours ago   Restarting (139) 19 seconds ago

 

3. zookeeper Docker log
### zookeeper docker log

[sysmaster@sysmaster]$ docker logs --tail 5 zookeeper > zookeeper_20240829.txt
library initialization failed - unable to allocate file descriptor table - out of memory

 

4. Repeated Occurrence of Same Issue
OOM was confirmed related to this issue and ulimit was changed, but the same issue repeatedly occurred.
open files                          (-n) 65535
max user processes                  (-u) 63353

5.fs.file-max=200000 setting was changed but the same issue repeated

 

Cause

This problem was not caused by actual lack of memory on the DB server, but was identified to be caused by the ulimit setting inside the Docker container.
This is a case where the problem occurred due to ulimit limitations set within the Docker environment.
 
 

Solutions

To resolve the issue, change the ulimit setting inside the Docker container.
Adjust the ulimit values by specifying them in the docker-compose.yml file as follows.
ulimits:
  nofile:
    soft: 65536 
    hard: 65536

 

 

 

Related to