Document Type | Technical Information
Category | Monitoring/Inspection
Applicable Product Versions | 6FS07, 6FS07PS, 7FS01, 7FS02, 7FS02PS
Document Number | TMOTI001
Overview
This document explains the method to manually generate dump files using the gdb (GNU Debugger) tool when a Tibero instance is in a Hang state or encounters an error.
Method
1. Create a LOCK state
Session 2 | Session 2 |
|---|---|
| $ tbsql tibero1/tmax SQL> create table t1 (a number); SQL> insert into t1 values(100); SQL> commit; SQL> update t1 set a=101 where a=100; | |
| SQL> update t1 set a=102 where a=100; โ Hang state |
2. Check Process ID
Check the PID (Process ID) of each process included in the Tibero instance.
$ tbdown pid 1779391: MONP 1779393: MGWP 1779394: FGWP0000 1779395: FGWP0001 1779396: FGWP0002 1779397: PEWP0000 1779398: PEWP0001 1779399: PEWP0002 1779400: PEWP0003 1779401: PEWP0004 1779402: PEWP0005 1779403: AGNT 1779404: DBWR 1779405: RCWP
3. Identify the problem process
Check the status of the problematic session.
SQL> col PROG_NAME for a15
SQL> select SID,serial#, serial#,prog_name,client_pid,WLOCK_WAIT from v$session;
SID SERIAL# SERIAL# PROG_NAME CLIENT_PID WLOCK_WAIT
---------- ---------- ---------- --------------- ---------- ------------------
92 278 278 tbsql 1780262
93 286 286 tbsql 1780294 WLOCK_TX
102 4946 4946 tbsql 1791952
3 rows selected.
4. Run and attach gdb debugger
Attach gdb to the server process of the problematic process.
$ gdb tbsvr 1779394 GNU gdb (GDB) Rocky Linux 8.2-20.el8.0.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from tbsvr...done. Attaching to program: /home/viera/tibero7/bin/tbsvr, process 1779394 [New LWP 1779467] [New LWP 1779469] [New LWP 1779471] [New LWP 1779473] [New LWP 1779475] [New LWP 1779479] [New LWP 1779482] [New LWP 1779493] [New LWP 1779506] [New LWP 1779518] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f410dad3307 in epoll_wait () from /lib64/libc.so.6 Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-251.el8_10.11.x86_64 libaio-0.3.112-1.el8.x86_64 libgcc-8.5.0-22.el8_10.x86_64 libstdc++-8.5.0-22.el8_10.x86_64
Set to ignore SIGUSR2
(gdb) handle SIGUSR2 nostop Signal Stop Print Pass to program Description SIGUSR2 No Yes Yes User defined signal 2
5. Check thread status
Check the list of threads.
(gdb) thread apply all bt Thread 11 (Thread 0x7f408980b700 (LWP 1779518)): #0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6 #1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663 #2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953 . . omitted . . Thread 3 (Thread 0x7f408d17b700 (LWP 1779469)): #0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6 #1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663 #2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953 #3 0x0000000000f6de5d in bitq_read_dbginfo (bitmask=..., timeout=timeout@entry=0, event_type=event_type@entry=WE_WLOCK_TX, id1=458778, id2=id2@entry=67, file=file@entry=0x6f4b684 "wlock.c", line=1267, func=0x6f34e80 <func.74998> "wlock_acq_internal_local") at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/bitqueue.c:470 #4 0x0000000000fc0031 in wlock_acq_internal_local (old_mode=0x4, we=WE_WLOCK_TX, timeout=-10, smart=<optimized out>, scope=SO_STMT, mode=WLOCK_PIN, id2=67, id1=458778, type=WLOCK_TX) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:1267 #5 wlock_acq_internal (type=type@entry=WLOCK_TX, id1=458778, id2=67, mode=mode@entry=WLOCK_PIN, --Type <RET> for more, q to quit, c to continue without paging-- scope=scope@entry=SO_STMT, smart=smart@entry=0 '\000', timeout=timeout@entry=-10, we=we@entry=WE_NOEVENT, old_mode=old_mode@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:3824 #6 0x0000000002170107 in wlock_acquire_with_timeout (we=WE_NOEVENT, timeout=-10, scope=SO_STMT, mode=WLOCK_PIN, id2=<optimized out>, id1=<optimized out>, type=WLOCK_TX) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/include/wlock.h:326 #7 tx_wait_internal (xid=..., max_secs=max_secs@entry=-1, we=we@entry=WE_NOEVENT) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/tx/tx.c:990 #8 0x0000000001e8fc2f in td_dsgmt_tx_wait (tsid=3, rowid=..., sgmt_id=<optimized out>, max_secs=-1, xid=...) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/td/td_dsgmt.c:163 #9 td_dsgmt_alloc_itl (snapshot=0x7f410a648c30, databuf=0x0, databuf@entry=0x7f410a72c888, rowid=..., rowid@entry=..., max_secs=max_secs@entry=-1, dblk=dblk@entry=0x7f40be807000, dl=dl@entry=0x7f408d176cc0, rp=rp@entry=0x7f408d176cd0, cleanouted=cleanouted@entry=0 '\000') at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/td/td_dsgmt.c:1097 #10 0x000000000266fa40 in xi_tscan_pick_current (exp_ctx=exp_ctx@entry=0x7f408d176f00, ts_id=<optimized out>, lrowid=..., bufpool=<optimized out>, use=BU_DBX_UPD, wait_secs=wait_secs@entry=-1, has_undo=has_undo@entry=0 '\000', stat=0x7f4088926090, is_tcc=is_tcc@entry=0x7f408d176dc0 "") at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/ex/xi_tscan.c:436 #11 0x000000000253cc3b in op_dml_update_ (exp_ctx=exp_ctx@entry=0x7f408d176f00, exn_dml=exn_dml@entry=0x7f4088925950, idx_ctx=idx_ctx@entry=0x7f408d176ea0) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/ex/op_dml_update.c:295 . . omitted . .
6. Switch to the problematic thread
(gdb) thread 3 [Switching to thread 3 (Thread 0x7f408d17b700 (LWP 1779469))] #0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6
7. Check the call stack of the thread
(gdb) bt #0 0x00007f410dad46be in semtimedop () from /lib64/libc.so.6 #1 0x0000000000f751aa in tb_sem_wait_internal (wait_mode=SEM_WAIT_MODE_BITQ, usec=0, my_tid=<optimized out>) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:663 #2 tb_sem_wait (my_tid=<optimized out>, timeout=timeout@entry=0, wait_mode=wait_mode@entry=SEM_WAIT_MODE_BITQ) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/sem.c:953 #3 0x0000000000f6de5d in bitq_read_dbginfo (bitmask=..., timeout=timeout@entry=0, event_type=event_type@entry=WE_WLOCK_TX, id1=458778, id2=id2@entry=67, file=file@entry=0x6f4b684 "wlock.c", line=1267, func=0x6f34e80 <func.74998> "wlock_acq_internal_local") at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/bitqueue.c:470 #4 0x0000000000fc0031 in wlock_acq_internal_local (old_mode=0x4, we=WE_WLOCK_TX, timeout=-10, smart=<optimized out>, scope=SO_STMT, mode=WLOCK_PIN, id2=67, id1=458778, type=WLOCK_TX) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:1267 #5 wlock_acq_internal (type=type@entry=WLOCK_TX, id1=458778, id2=67, mode=mode@entry=WLOCK_PIN, scope=scope@entry=SO_STMT, smart=smart@entry=0 '\000', timeout=timeout@entry=-10, we=we@entry=WE_NOEVENT, old_mode=old_mode@entry=0x0) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/frame/wlock.c:3824 #6 0x0000000002170107 in wlock_acquire_with_timeout (we=WE_NOEVENT, timeout=-10, scope=SO_STMT, mode=WLOCK_PIN, id2=<optimized out>, id1=<optimized out>, type=WLOCK_TX) at /home/chef/workspace/Build_centos_7/tibero7/src/tbsvr/include/wlock.h:326 #7 tx_wait_internal (xid=..., max_secs=max_secs@entry=-1, we=we@entry=WE_NOEVENT) . . omitted . .
8. Generate dump file
(gdb) call wlock_dump_ddl(&dump_dstream, 0) Thread 4 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 10 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 11 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 9 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 6 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 8 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 2 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 5 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 3 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 7 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 10 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 9 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 2 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 6 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 5 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 4 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 3 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 8 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 7 "tbsvr" received signal SIGUSR2, User defined signal 2. Thread 11 "tbsvr" received signal SIGUSR2, User defined signal 2.
9. Check dump file
PATH : $TB_HOME/instance/$TB_SID/dump/tracedump
$ ls -al | grep dump_test -rw-r--r-- 1 viera viera 18627 Feb 10 14:25 dump_test.trc
Due to lock contention (WLOCK_TX)**, session 93 is waiting for session 92
$ cat dump_test.trc
. . omitted . .
WLOCK = 00007f40918119b0 TYPE = WLOCK_TX ID1 = 458778 ID2 = 67
owner_cnt
X = 1
owners :
ELEM = 00007f40917e8bc8 TID = 92 MODE = X SO_SCOPE = SO_TX
PTRS. SO=00007f40f377d998/00007f40f3d10228, ELEMS=00007f4091811a08/00007f4091811a08
converters :
waiters :
ELEM = 00007f410a6919d0 TID = 93 MODE = PIN SO_SCOPE = SO_STMT
PTRS. SO=00007f410a646a70/00007f410a691a50, ELEMS=00007f4091811a48/00007f4091811a48
bitmask :
. . omitted . .