Document Type | Troubleshooting
Category | Administration
Applicable Product Version | 6 7FS02
Document Number | TADTS076
Issue
In an AIX environment where nofiles is set to unlimited, when using 1024 or more datafiles, the tbepa Process's FD (file descriptor) is used excessively beyond necessity.
$ ulimit -n unlimited $ ps -ef | grep tbepa tibero 35260120 1 0 14:46:06 - 0:00 tbepa 2457 1 180 $ procfiles -c 35260120 35260120 : tbepa 2457 1 180 Current rlimit: 9223372036854775807 file descriptors --------------------------------------------------------------------------------------------------------------- FD TYPE MODE DEV/RDEV UID GID OPMOD INODE --------------------------------------------------------------------------------------------------------------- 0 c --------- 10, 4( 2, 2) tibero dba R-W 412 1 c --------- 10, 4( 2, 2) tibero dba R-W 412 2 c --------- 10, 4( 2, 2) tibero dba R-W 412 1024 - rw------- 10,14 tibero dba R-W 2182816 1025 - rw------- 10,14 tibero dba R-W 2183174 1026 - rw------- 10,14 tibero dba R-W ...omitted 2453 - rw------- 10,14 tibero dba R-W|DS 2183529 2454 - rw------- 10,14 tibero dba R-W|DS 2183530 2455 - rw------- 10,14 tibero dba R-W|DS 2183531 2457 s --------- 268435455,-1 tibero dba R-W * tbepa Process uses 2547 FDs
The FD overuse by the tbepa Process causes shortage of FD resources that should be used by other Processes as well, making the operation of the Tibero Instance unstable, and if it continues, the Tibero Instance will go down.
--sys.log
04/30 09:29:04.9300 0-2806 tc_fdpoo:1203 open (stat) failed (errno=23, File table overflow) (flag=020000002) (filename=/tbdata/syssub003.dtf)
04/30 09:29:04.9300 0-2806 tbsvr_er:73 THROW. ec=ERROR_CACHE_FILE_OPEN_FAILED(-1003) [ Unable to open file /tbdata/syssub003.dtf. ] (csr_id:19241) [tc_fdpool.c:1521:fdpool_get_internal]
04/30 09:29:04.9300 0-2806 tc_fdpoo:638 fdelem alloc 0a00000e1c4454d0
04/30 09:29:04.9300 0-2806 tc_fdpoo:1203 open (stat) failed (errno=23, File table overflow) (flag=020000002) (filename=/tbdata/system001.dtf)
04/30 09:29:04.9300 0-2806 tbsvr_er:73 THROW. ec=ERROR_CACHE_FILE_OPEN_FAILED(-1003) [ Unable to open file /tbdata/system001.dtf. ] (csr_id:19241) [tc_fdpool.c:1521:fdpool_get_internal]
04/30 09:29:04.9300 0-2806 tc_fdpoo:638 fdelem alloc 0a00000e1c445540
...
04/30 09:31:28.4035 0-0793 tb_asser:47 Internal Error with condition 'tsn_read(low_cache->tsn) >= tsn_read(org->low_cache.tsn)' (2 args) (cf_ckpt.c:108:cf_put_ckpt_progress) (pid=8585652, sessid=793, tid=793)
Cause
SQL> select userenv('TID'), pid from v$session ;
USERENV('TID') | PID
---------------------------------------|----------
180 | 64292528
1 row selected.
$ ps -ef | grep 64292528
tibero 64292528 55642350 0 Oct 29 - 0:29 tbsvr_FGWP000 FGWP000 -t NORMAL -SVR_SID test
$ ps -ef | grep tbepa | grep 180
tibero 52101678 1 0 15:20:21 - 0:00 tbepa 2456 1 180
* The tbepa Process was called by Session SID 180, which can be known from the last argument,
and during the call process, it was forked from the tbsvr_FGWP000 Process with PID 64292528 used by SID 180.NoteFork is a system call method where a process duplicates itself to create a new process.
During the forking of the Working Process, the FD information used by the Working Process is also duplicated, then unnecessary FD information is deleted and FD resources are released.
At this time, if the Working Process uses 1024 or more FDs, only 1024 FDs are deleted and the remaining FDs are not deleted, causing unnecessary FD occupation.
The number of FDs in the Working Process is proportional to the number of files used in the Tibero Instance, and generally, this phenomenon occurs when using 1024 or more datafiles.
This phenomenon also occurs in C Gateway Processes (gw4orcl, used in DB Link) that create Processes by forking.
Solutions
Apply a patch that modifies the tbepa Process to delete more than 1024 FDs after forking. (Applied Patch: 342022)
CautionApply the patch through technical support provided by Tmax Tibero.
NoteIf nofiles is set to a specific value other than unlimited and Tibero is restarted, the phenomenon does not occur.$ ulimit -n 65535 * Temporary change method # vi /etc/security/limits tibero: nofiles = 65534 ... * Permanent change method