Overview

In a Linux environment, when the SGA area (TOTAL_SHM_SIZE) is set large and the number of SESSIONS (MAX_SESSION_COUNT) is set high, phenomena such as Memory Swap Out/In, Out of Memory, and server hang may occur. This document provides guidance on Huge Page, which can be applied as a solution.

Note

Details on the phenomena and Tibero Huge Page configuration methods can be found in the article 'Memory Swap Out/In, Out of Memory, Server Hang Resolution (Linux)'.

Method

Huge Page

• The CPU accesses Physical Memory (hereafter: RSS) through Virtual Memory (hereafter: VSS).
• The CPU uses a Page Table to translate VSS to RSS.
• As a process's memory usage increases, the size of the Page Table also grows, increasing management overhead.
• Linux manages memory by default in 4KB page size units.
• Applications using large amounts of memory must manage many small 4KB pages, which can increase Page Table size and Translation Lookaside Buffer (hereafter: TLB) miss frequency.

getconf -a |grep -i PAGESIZE
PAGESIZE                           4096

MMU and TLB

• The CPU must access RSS through VSS.
• The Memory Management Unit (hereafter: MMU) performs address translation between VSS and RSS.
• The MMU refers to the Page Table during the translation process.
• When the CPU attempts to access RSS via the MMU, it refers to the Page Table each time, which causes performance degradation.
• To speed up translation in the MMU, the TLB is used; the TLB caches recently translated address mappings.

• When the CPU requests an RSS address from the MMU via VSS, it first checks the TLB.

  - If the MMU contains the value, translation to the RSS address occurs immediately without accessing the Page Table.
  - If the MMU does not contain the value, it accesses the Page Table, translates the value, and caches it in the TLB.

Reasons for Using Huge Page

Huge Page is a technology designed to mitigate the limitations of the MMU/TLB structure.

By applying HugePage for memory management, the number of TLB misses can be reduced.
Additionally, it reduces the number of page translations the MMU must process, improving CPU memory access performance.

Page Table Processing

• VSS is translated to RSS through a 4-level Page Table.
• CR3 - PGD - PUD - PMD - PTE - Physical Memory (RSS)

Memory Access with 4KB Page Size (Non-HugePage)

To access 2MB of memory, the CPU must perform 512 accesses through the 4-level PTE process.

• PGD - PUD - PMD - PTE, 4 levels accessed 512 times

Memory Access with 2MB Page Size (HugePage Size 2MB)

To access 2MB of memory, the CPU only needs 3 accesses up to the PMD level.

• PGD - PUD - PMD, 3 levels accessed once

  - PMD unit: 2MB
  - PTE unit: 4KB

Page Table Size Comparison

The difference in page table size between Non-HugePage and Huge Page is explained through the example below.

Case 1. Non-Huge Page: Memory handling when applied in 4KB units

Assuming 50GB of Shared Memory is accessible by 1,000 processes.
The entries required to access 50GB of Shared Memory in RSS are as follows:

• The number of PTE entries created to access 50GB of RSS is 13,107,200
• For 1,000 processes accessing 50GB, the PTE size per process is 100MB
• The total Page Table size for 1,000 processes accessing 50GB is approximately 97GB

In this situation, the Page Table size required by the server is 97GB, which is very large.

Case 2. Huge Page: Memory handling when applied in 2MB units

Assuming 50GB of Shared Memory is accessible by 1,000 processes.
The entries required to access 50GB of Shared Memory in RSS are as follows:

• The number of PDU entries created to access 50GB of RSS is 25,600
• Since PDU is handled in 2MB units, the PTE level is omitted
• For 1,000 processes accessing 50GB, the PDU size per process is 0.19MB
• The total Page Table size for 1,000 processes accessing 50GB is approximately 195MB

As shown in the example above, there is a significant difference in Page Table size between 4KB units (non-hugepage) and 2MB units (hugepage). Using HugePage can greatly reduce the memory used by the Page Table.

Differences Between Non-HugePage and Huge Page

• When estimating memory, the Page Table size is determined by the two values TOTAL_SHM_SIZE and MAX_SESSION_COUNT.
• Consider the Page Table size when deciding whether to apply Huge Page.
• Linux can configure Huge Page sizes as 2MB or 1GB.
  • In Tibero, setting 2MB is sufficient, so 1GB configuration details are omitted.

Reference

Kernel Page Table Memory Usage Before Applying Huge Page

[root@DB01 /]# cat /proc/meminfo |grep -i hugepagesize
Hugepagesize:       62914560 kB

[root@DB02/]# cat /proc/meminfo |grep -i hugepagesize
Hugepagesize:       66914531 kB

• Server Memory TOTAL: 250GB
  • MAX_SESSION_COUNT=5000
  • TOTAL_SHM_SIZE=130G
  • MEMORY_TARGET=160G
  • USE_HUGE_PAGE=N (Default N)
  • Total Page Table size: 126G
• After starting the Tibero Database, it operated normally but the Page Table size gradually increased, causing memory shortage.
• Due to memory shortage, frequent Swap Out/In occurred, leading to performance degradation.
• Due to server performance degradation, process hangs occurred and the Kernel killed processes that were unresponsive for 120 seconds.
• This caused the Tibero Instance to go down.

[root@DB01 /]# cat /proc/meminfo |grep -i hugepagesize
Hugepagesize:       203120 kB

[root@DB02 /]# cat /proc/meminfo |grep -i hugepagesize
Hugepagesize:       212713 kB