Skip to main content

Search

Welcome to Tibero GTS!

Memory Swap Out/In, Out of Memory, Server Hang Resolution (Linux)

Document Type | Troubleshooting

Category | Administration

Applicable Product Versions | Tibero 6, Tibero 7

Document Number | TADTS069

Issue

In a Linux environment, when setting a large SGA area (TOTAL_SHM_SIZE) and a high number of SESSIONS (MAX_SESSION_COUNT), the following phenomena may occur. 

 

Memory Swap Out/In

$ sar -W 1 
14:20:00     pswpin/s pswpout/s
14:20:01       120.00   150.00
14:20:02       300.00   220.00
14:20:03       250.00   200.00
14:20:04       180.00   170.00
14:20:05       400.00   350.00
...
  • pswpout/s: Number of pages moved per second to Swap Memory due to lack of free memory space
  • pswpin/s: Number of pages moved per second back to free memory space from Swap Memory as they are reused by processes

 

Out of Memory

$ more /var/log/message
...
Out of memory: Killed process 3760158 (tbsvr) total-vm:74067532kB, anon-rss:8090192kB, file-rss:1456kB, shmem-rss:46178864kB, UID:1501 pgtables:111604kB oom_score_adj:0
...
  • Occurs when Linux can no longer allocate free memory and swap memory is also unavailable

  • In a memory shortage situation, the process that can free the most memory is killed

    In the Tibero Database server, the tbsvr process's SHM area occupies the most memory, so tbsvr is killed

 

Server Hang

$ more /var/log/message
...
Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11283 (tbsvr) of user 1000 killed by SIGABRT - dumping core
Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11280 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash)
Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11284 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash)
Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11278 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash)
Sep 23 10:24:21 DB01 abrt-hook-ccpp: Process 11275 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash)
...
  • When free memory is insufficient and swap memory is used but swap memory has available capacity
  • Performance degradation occurs, and if there is no response for 120 seconds (default: 120 seconds), it is considered a hang and the unresponsive process is killed

 

Cause

The above phenomena can occur if the page table memory usage is not measured even though Tibero memory was estimated relative to the total Linux memory size. 

 

Solution

If the page table memory size is expected to be large when estimating Tibero memory, applying Huge Pages can resolve the issue. 
 

Huge Page Configuration Method (Linux)

There are two main ways to configure Huge Pages in Linux. 

  • Static Huge Page: The system administrator sets only part of the memory area as Huge Page area
  • Transparent Huge Page (THP): The kernel automatically groups 4KB units for handling

 

Static Huge Page

This is the Tibero Huge Page method applied in this document. 

  • The system administrator estimates the memory size to apply Huge Pages

  • Only part of the memory area is configured to use Huge Pages

 

Transparent Huge Page (THP)

The kernel automatically groups page size units for Huge Page handling, not the system administrator

 

Tibero Huge Page

When Tibero instance memory is expected to have a large page table size relative to server memory size, apply Huge Pages using the following method.

 

Tibero Huge Page Formula

PGA Area

  • PGA area is calculated as MEMORY_TARGET - TOTAL_SHM_SIZE.
  • PGA area is non-shared memory per process, so page table size is small.

SGA Area

  • Tibero Huge Page (USE_HUGE_PAGE=Y) parameter applies only to the SGA area.
  • SGA area is accessed by all tbsvr processes, so page table size can be large.
  • TAS Instance and Storage Server Instance are excluded; no need to apply Huge Pages.

 

Non-Huge Page, Page Table Size

Total number of pages = (TOTAL_SHM_SIZE * 1024 * 1024) / 4KB
Assuming TOTAL_SHM_SIZE is in GB, divide the SGA area by page size units (default 4KB).

 

Page table size per process = ("Total number of pages" * 8) / 1024 / 1024 MB
8 is the size of a page table entry in bytes.

 

Total page table size = ("Page table size per process" * "Number of processes") / 1024 GB
The number of processes can be checked with ps -ef |grep $TB_SID |grep -v grep| wc -l. That is, the number of tbsvr processes (+listener) running in the Tibero Database instance.

 

Huge Page, Page Table Size

Total number of pages = (TOTAL_SHM_SIZE * 1024) / 2MB
Assuming TOTAL_SHM_SIZE is in GB, divide the SGA area by Huge Page units (2MB).

 

Page table size per process = ("Total number of pages" * 8) / 1024 KB
8 is the size of a page table entry in bytes.

 

Total page table size = ("Page table size per process" * "Number of processes") / 1024 MB
The number of processes can be checked with ps -ef |grep $TB_SID |grep -v grep| wc -l.
That is, the number of tbsvr processes (+listener) running in the Tibero Database instance.

 

Huge Page Application Procedure

1. Disable THP Feature

  • If Huge Page and THP are both enabled: PGA or other processes may use Huge Pages, so disable the THP feature.
  • Request the system engineer to apply this setting to persist after OS reboot.

 

2. Modify sysctl.conf

  • Estimate the required Huge Page size and activate it in the used memory space.
  • Specify the Linux GROUP ID that will use the allocated Huge Page space.

 

3. Modify limits.conf

  • Set memlock for the Linux Tibero account to prevent the SGA area from being swapped out.

 

4. $TB_SID.tip

  • Activate the USE_HUGE_PAGE=Y parameter in the Tibero Database TIP.
  • Exclude TAS Instance and Storage Server Instance.

 

Kernel Configuration

/etc/sysctl.conf

vm.nr_hugepage=%d
vm.hugetlb_shm_group=%d

vm.nr_hugepage

  • Number of pages to use as hugepages
  • (Number of pages * 2MB) + 1~2% margin is calculated and applied
  • sysctl -p applies the Huge Page memory to used memory

vm.hugetlb_shm_group

  • Specify the group ID of the Tibero engine account
  • Check with cat /etc/passwd

 

/etc/security/limits.conf

tibero soft memlock %d
tibero hard memlock %d
  • Sets memory lock for the Tibero engine account to prevent swap out
  • %d = 2MB * vm.nr_hugepage

 

Transparent Hugepage Disable

 cat /sys/kernel/mm/transparent_hugepage/enabled

The keyword in brackets [] indicates the current mode. 

  • always: always enabled
  • madvise: enabled on application request
  • never: disabled

 

Use either madvise or never.

  • If no solution other than Tibero Database is running, apply never

     

Make sure to set this permanently so it persists after server reboot. 

  • Use a script to change THP to never on server reboot
  • Use grub settings to permanently set THP to never

Note

Set this by one of the two methods above and request the system engineer to apply it. 

 

Tibero Configuration

Tibero Database TIP ($TB_SID.tip)

USE_HUGE_PAGE=Y
  • When USE_HUGE_PAGE is enabled in Tibero, it applies only to the SGA area. 
    • SGA: TOTAL_SHM_SIZE
  • Only the SGA area is applied; the PGA area is still allocated in kernel page size units.

 

 

 

Related to