Skip to main content

Search

Welcome to Tibero GTS!

DB DOWN Due to Excessive Fd Usage

Document Type | TroubleShooting

Category | Tuning

Applicable Product Version | 6FS06

Error Code | 1003

Document Number | TTUTS001

 

Issue

Due to excessive fd usage, ERROR_CACHE_FILE_OPEN_FAILED(-1003) error occurs, resulting in a DB DOWN situation. 

 

Cause

The cause derivation process is as follows.
 

1. Check fd-related errors in oslog (/var/log/messages)

Fd are created up to the file-max limit set in the /etc/sysctl.conf file, and when files exceeding that cannot be created, logs like the following occur. 

Nov 20 12:10:22 hdsodb1p systemd[54513]: Reached target Exit the Session.
Nov 20 12:10:31 hdsodb1p kernel: VFS: file-max limit 6815744 reached
Nov 20 12:10:38 hdsodb1p kernel: VFS: file-max limit 6815744 reached

 

Set /etc/sysctl.conf as below. 

``
kernel.shmmni = 4096
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 1024  65000
vm.nr_hugepages=102400
``

 

2. Check sid and ilog in slog

(tbsvr.out.25579) THROW occurred in LGWR thread 497, and currently, for this ec, an assert triggers an intentional DB shutdown. Subsequently, DB Down is confirmed. 
[497] 2023/11/20 12:17:37.509 Internal Error with condition '!"error"' (tc.c:343) (pid=27574, sessid=497, tid=497, os_thr_id=27608)

...

[497] callstack dump from tbsvr_DBWR for [LGWR, 497, 27608/27574]

[497] SEE DUMP tbsvr.callstack.27574 (lwpid 27608) BY PSTACK

 

3. Check ilog to identify exact evidence (ilog of session 497)

  • Ilog is a binary file that can only be checked via tbiv, and requires the ilog_fmtmap.bin file at that time.
  • Ilog is created per thread, and the default filename is $TB_SID-threadID.ilog.
11/09 05:26:51.7 0-0497 tc_lgwr.:1643 requested switch 109 -> 110 (next=0001.8ce9dc30
11/09 05:26:51.7 0-0497 tc_fdpoo:1152 open (stat) failed (errno=23, Too many open files in system) (flag=010002) (filename=/dev/vx/rdsk/hdsodb_sysdg01/ctlOHDSODB_01)
11/09 05:26:51.7 0-0497     cf.c:181 control file open failed : '/dev/vx/rdsk/hdsodb_redodg02/ctlOHDSODB_01' (Input/output error)
...

11/09 05:26:51.7 0-0497     tc.c:337 TBR-24003 :  Unable to read control file.
11/09 05:26:51.7 0-0497     tc.c:340 LGWR: shutdown instance due to error 24003
11/09 05:26:51.7 0-0497 tbsvr_tr:229 Internal Error with condition '!"error"' (tc.c:343)

 

After the log switch request (requested switch 109 โ†’ 110), LGWR failed to open the control file due to insufficient fd. As a result, all redo log write operations failed, and the DB was intentionally stopped to maintain data consistency.

 

Solutions

Modify /etc/security/limits.conf file

Modify the Nofile value. Nofile is the number of fds one process can use.
  • Formula: (WTHR_PER_PROC * ((total data files in use) + 15)) + (tbsvr process count + 5) + 100 or 
    roughly 3,000,000
 
 

Modify /etc/sysctl.conf file

Modify the fs.file-max value. fs.file-max is the total number of fds usable by the entire server.
  • Formula: (nofile parameter) x (WTHR_PROC_CNT + PEP_PROC_CNT) or 67108864 
  • How to check current fd usage (Linux: cat /proc/sys/fs/file-nr )
 
Note
 
  • ILOG Creation Method
    ILOG is created per thread, and the default filename format is $TB_SID-threadID.ilog.
    When the file size reaches a certain level, the existing file is renamed to $TB_SID-threadID_date_serialNumber.ilog format 
    and a new ILOG file is created.
     
  • ILOG Checking Method
    Example: To check logs between 2025/04/16 23:00:00 ~ 04/17 00:00:00 for threads 17 and 33,โ†’ first confirm that the logs in that period were generated by threads 17 and 33.
     
  • ILOG Check Command
    tbiv -o 1.txt -r 20250416230000 20250417000000 tbsid-17.ilg, tbsid-33.ilg
     
  • ILOG Parameters
NAME
Description
ILOG_BACKUP_SIZE_LIMIT
Total size limit of backed-up ilog
ILOG_DEST
ilog creation path
ILOG_FILE_BACKUP_ON_BOOT
Whether to create ilog backup and back up existing files on startup
ILOG_FILE_SIZE
Size limit per ilog file. When exceeded, the file is renamed and a new file is created.
ILOG_KEEP_BACKUP_ASIDE
Whether to manage ilog backup size separately. If set to Y, it is managed independently from ILOG_TOTAL_SIZE_LIMIT.
ILOG_MAP
Map file path to specify list in a file instead of command when setting ilog
ILOG_TOTAL_SIZE_LIMIT
Total size of ilog files + ilog backups
USE_ILOG
Whether to use ilog
  • How to Check ILOG
    Run tbiv to convert the binary ilog file into a readable form. 
 
 
 

Related to