Skip to main content

Search

Welcome to Tibero GTS!

FD Overuse Phenomenon of tbepa Process in AIX Environment

Document Type | Troubleshooting 

Category | Administration

Applicable Product Version | 6 7FS02

Document Number | TADTS076

 

Issue

 In an AIX environment where nofiles is set to unlimited, when using 1024 or more datafiles, the tbepa Process's FD (file descriptor) is used excessively beyond necessity. 

$ ulimit -n 
unlimited

$ ps -ef | grep tbepa 
   tibero 35260120        1   0 14:46:06      -  0:00 tbepa 2457 1 180

$ procfiles -c 35260120 
35260120 : tbepa 2457 1 180
 Current rlimit: 9223372036854775807 file descriptors
---------------------------------------------------------------------------------------------------------------
 FD    TYPE MODE       DEV/RDEV     UID       GID        OPMOD                                          INODE
---------------------------------------------------------------------------------------------------------------
 0      c   ---------  10, 4( 2, 2) tibero    dba        R-W                                            412  
 1      c   ---------  10, 4( 2, 2) tibero    dba        R-W                                            412  
 2      c   ---------  10, 4( 2, 2) tibero    dba        R-W                                            412  
 1024   -   rw-------  10,14        tibero    dba        R-W                                            2182816
 1025   -   rw-------  10,14        tibero    dba        R-W                                            2183174
 1026   -   rw-------  10,14        tibero    dba        R-W  
 ...omitted
 2453   -   rw-------  10,14        tibero    dba        R-W|DS                                         2183529
 2454   -   rw-------  10,14        tibero    dba        R-W|DS                                         2183530
 2455   -   rw-------  10,14        tibero    dba        R-W|DS                                         2183531
 2457   s   ---------  268435455,-1        tibero    dba        R-W     
* tbepa Process uses 2547 FDs

 

 The FD overuse by the tbepa Process causes shortage of FD resources that should be used by other Processes as well, making the operation of the Tibero Instance unstable, and if it continues, the Tibero Instance will go down.

--sys.log
04/30 09:29:04.9300 0-2806 tc_fdpoo:1203 open (stat) failed (errno=23, File table overflow) (flag=020000002) (filename=/tbdata/syssub003.dtf)
04/30 09:29:04.9300 0-2806 tbsvr_er:73 THROW. ec=ERROR_CACHE_FILE_OPEN_FAILED(-1003) [ Unable to open file /tbdata/syssub003.dtf.   ] (csr_id:19241) [tc_fdpool.c:1521:fdpool_get_internal]
04/30 09:29:04.9300 0-2806 tc_fdpoo:638 fdelem alloc 0a00000e1c4454d0
04/30 09:29:04.9300 0-2806 tc_fdpoo:1203 open (stat) failed (errno=23, File table overflow) (flag=020000002) (filename=/tbdata/system001.dtf)
04/30 09:29:04.9300 0-2806 tbsvr_er:73 THROW. ec=ERROR_CACHE_FILE_OPEN_FAILED(-1003) [ Unable to open file /tbdata/system001.dtf.   ] (csr_id:19241) [tc_fdpool.c:1521:fdpool_get_internal]
04/30 09:29:04.9300 0-2806 tc_fdpoo:638 fdelem alloc 0a00000e1c445540
...
04/30 09:31:28.4035 0-0793 tb_asser:47 Internal Error with condition 'tsn_read(low_cache->tsn) >= tsn_read(org->low_cache.tsn)' (2 args) (cf_ckpt.c:108:cf_put_ckpt_progress) (pid=8585652, sessid=793, tid=793)

 

Cause

 The tbepa Process is created by forking from the Working Process (tbsvr_FGWP) of the Session that executed the External Procedure.
 
SQL> select userenv('TID'), pid from v$session ; 

USERENV('TID')                         |       PID
---------------------------------------|----------
180                                    |  64292528

1 row selected.

$ ps -ef | grep 64292528

  tibero 64292528 55642350   0   Oct 29      -  0:29 tbsvr_FGWP000 FGWP000 -t NORMAL -SVR_SID test

$ ps -ef | grep tbepa | grep 180
  tibero 52101678        1   0 15:20:21      -  0:00 tbepa 2456 1 180

* The tbepa Process was called by Session SID 180, which can be known from the last argument,  
  and during the call process, it was forked from the tbsvr_FGWP000 Process with PID 64292528 used by SID 180.
Note
Fork is a system call method where a process duplicates itself to create a new process.

 

During the forking of the Working Process, the FD information used by the Working Process is also duplicated, then unnecessary FD information is deleted and FD resources are released.

At this time, if the Working Process uses 1024 or more FDs, only 1024 FDs are deleted and the remaining FDs are not deleted, causing unnecessary FD occupation. 

The number of FDs in the Working Process is proportional to the number of files used in the Tibero Instance, and generally, this phenomenon occurs when using 1024 or more datafiles.

This phenomenon also occurs in C Gateway Processes (gw4orcl, used in DB Link) that create Processes by forking.

 

Solutions

Apply a patch that modifies the tbepa Process to delete more than 1024 FDs after forking.  (Applied Patch: 342022)

Caution
Apply the patch through technical support provided by Tmax Tibero.
 
Note
If nofiles is set to a specific value other than unlimited and Tibero is restarted, the phenomenon does not occur.
$ ulimit -n 65535
* Temporary change method

# vi /etc/security/limits 
tibero:

        nofiles = 65534

        ...

* Permanent change method

 

 

Related to