Document Type | Troubleshooting
Category | Administration
Applicable Product Versions | Tibero 6, Tibero 7
Document Number | TADTS069
Issue
In a Linux environment, when setting a large SGA area (TOTAL_SHM_SIZE) and a high number of SESSIONS (MAX_SESSION_COUNT), the following phenomena may occur.
Memory Swap Out/In
$ sar -W 1 14:20:00 pswpin/s pswpout/s 14:20:01 120.00 150.00 14:20:02 300.00 220.00 14:20:03 250.00 200.00 14:20:04 180.00 170.00 14:20:05 400.00 350.00 ...
- pswpout/s: Number of pages moved per second to Swap Memory due to lack of free memory space
- pswpin/s: Number of pages moved per second back to free memory space from Swap Memory as they are reused by processes
Out of Memory
$ more /var/log/message ... Out of memory: Killed process 3760158 (tbsvr) total-vm:74067532kB, anon-rss:8090192kB, file-rss:1456kB, shmem-rss:46178864kB, UID:1501 pgtables:111604kB oom_score_adj:0 ...
- Occurs when Linux can no longer allocate free memory and swap memory is also unavailable
In a memory shortage situation, the process that can free the most memory is killed
In the Tibero Database server, the tbsvr process's SHM area occupies the most memory, so tbsvr is killed
Server Hang
$ more /var/log/message ... Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11283 (tbsvr) of user 1000 killed by SIGABRT - dumping core Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11280 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash) Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11284 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash) Sep 23 10:24:18 DB01 abrt-hook-ccpp: Process 11278 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash) Sep 23 10:24:21 DB01 abrt-hook-ccpp: Process 11275 (tbsvr) of user 1000 killed by SIGABRT - ignoring (repeated crash) ...
- When free memory is insufficient and swap memory is used but swap memory has available capacity
- Performance degradation occurs, and if there is no response for 120 seconds (default: 120 seconds), it is considered a hang and the unresponsive process is killed
Cause
The above phenomena can occur if the page table memory usage is not measured even though Tibero memory was estimated relative to the total Linux memory size.
Solution
If the page table memory size is expected to be large when estimating Tibero memory, applying Huge Pages can resolve the issue.
Huge Page Configuration Method (Linux)
There are two main ways to configure Huge Pages in Linux.
- Static Huge Page: The system administrator sets only part of the memory area as Huge Page area
- Transparent Huge Page (THP): The kernel automatically groups 4KB units for handling
Static Huge Page
This is the Tibero Huge Page method applied in this document.
The system administrator estimates the memory size to apply Huge Pages
Only part of the memory area is configured to use Huge Pages
Transparent Huge Page (THP)
The kernel automatically groups page size units for Huge Page handling, not the system administrator
Tibero Huge Page
When Tibero instance memory is expected to have a large page table size relative to server memory size, apply Huge Pages using the following method.
Tibero Huge Page Formula
PGA Area
- PGA area is calculated as
MEMORY_TARGET - TOTAL_SHM_SIZE. - PGA area is non-shared memory per process, so page table size is small.
SGA Area
- Tibero Huge Page (
USE_HUGE_PAGE=Y) parameter applies only to the SGA area. - SGA area is accessed by all tbsvr processes, so page table size can be large.
- TAS Instance and Storage Server Instance are excluded; no need to apply Huge Pages.
Non-Huge Page, Page Table Size
Total number of pages = (TOTAL_SHM_SIZE * 1024 * 1024) / 4KB
Assuming TOTAL_SHM_SIZE is in GB, divide the SGA area by page size units (default 4KB).
Page table size per process = ("Total number of pages" * 8) / 1024 / 1024 MB
8 is the size of a page table entry in bytes.
Total page table size = ("Page table size per process" * "Number of processes") / 1024 GB
The number of processes can be checked with ps -ef |grep $TB_SID |grep -v grep| wc -l. That is, the number of tbsvr processes (+listener) running in the Tibero Database instance.
Huge Page, Page Table Size
Total number of pages = (TOTAL_SHM_SIZE * 1024) / 2MB
Assuming TOTAL_SHM_SIZE is in GB, divide the SGA area by Huge Page units (2MB).
Page table size per process = ("Total number of pages" * 8) / 1024 KB
8 is the size of a page table entry in bytes.
Total page table size = ("Page table size per process" * "Number of processes") / 1024 MB
The number of processes can be checked with ps -ef |grep $TB_SID |grep -v grep| wc -l.
That is, the number of tbsvr processes (+listener) running in the Tibero Database instance.
Huge Page Application Procedure
1. Disable THP Feature
- If Huge Page and THP are both enabled: PGA or other processes may use Huge Pages, so disable the THP feature.
- Request the system engineer to apply this setting to persist after OS reboot.
2. Modify sysctl.conf
- Estimate the required Huge Page size and activate it in the used memory space.
- Specify the Linux GROUP ID that will use the allocated Huge Page space.
3. Modify limits.conf
- Set memlock for the Linux Tibero account to prevent the SGA area from being swapped out.
4. $TB_SID.tip
- Activate the
USE_HUGE_PAGE=Yparameter in the Tibero Database TIP. - Exclude TAS Instance and Storage Server Instance.
Kernel Configuration
/etc/sysctl.conf
vm.nr_hugepage=%d vm.hugetlb_shm_group=%d
vm.nr_hugepage
- Number of pages to use as hugepages
(Number of pages * 2MB) + 1~2% marginis calculated and appliedsysctl -papplies the Huge Page memory to used memory
vm.hugetlb_shm_group
- Specify the group ID of the Tibero engine account
- Check with
cat /etc/passwd
/etc/security/limits.conf
tibero soft memlock %d tibero hard memlock %d
- Sets memory lock for the Tibero engine account to prevent swap out
- %d =
2MB * vm.nr_hugepage
Transparent Hugepage Disable
cat /sys/kernel/mm/transparent_hugepage/enabled
The keyword in brackets [] indicates the current mode.
- always: always enabled
- madvise: enabled on application request
- never: disabled
Use either madvise or never.
If no solution other than Tibero Database is running, apply never
Make sure to set this permanently so it persists after server reboot.
- Use a script to change THP to never on server reboot
- Use grub settings to permanently set THP to never
Note
Set this by one of the two methods above and request the system engineer to apply it.
Tibero Configuration
Tibero Database TIP ($TB_SID.tip)
USE_HUGE_PAGE=Y
- When
USE_HUGE_PAGEis enabled in Tibero, it applies only to the SGA area.- SGA:
TOTAL_SHM_SIZE
- SGA:
- Only the SGA area is applied; the PGA area is still allocated in kernel page size units.