For large Oracle Database footprints – say 8GB or above (not really too large any more) configure Linux huge pages and assure Oracle’s SGA is using manual or Automatic Shared Memory Management.
Note Automatic Memory Management (using MEMORY_MAX_TARGET) is not compatible with Linux huge pages. Do not confuse Automatic Shared Memory Management / ASMM (which uses SGA_MAX_MEMORY) with AMM. Automatic Shared Memory Memory IS compatible with allocation of Linux Huge Pages.
Also note: Huge Page configurations with large Oracle Database SGAs should use EXPLICIT huge page configurations – NOT anonymous or transparent huge pages see:
Check system-wide THP usage
Run the following command to check system-wide THP usage:
# grep AnonHugePages /proc/meminfo AnonHugePages: 632832 kB
In the example above – transparent / anonymous huge pages are being used – not good on large RHEL based system that homes a large Oracle database and SGA.
Instead turn off anonymous huge pages and explicitly configure huge pages. See references below.
See Oracle Support Doc ID: 749851.1 (included below)
Oracle Database – Enterprise Edition – Version 184.108.40.206 and later
Linux OS – Version 2.6 and later
IBM: Linux on System z
IBM: Linux on POWER Systems
IBM S/390 Based Linux (31-bit)
This document discusses the interoperability of the Automatic Memory Management (AMM) feature introduced by Oracle Database 11g and the HugePages (HugeTLB) feature of the Linux OS kernel.
This document is to be used by Linux system administrators and Oracle database administrators that work with Oracle Database Server 11g on Linux Operating System.
The 11g AMM feature is enabled by the MEMORY_TARGET / MEMORY_MAX_TARGET instance initialization parameters . That is also the case with a default database instance created using Database Configuration Assistant (DBCA).
With AMM all SGA memory is allocated by creating files under /dev/shm. When Oracle DB does SGA allocations that way HugePages are not reserved/used.
On systems with HugePages in use, attempting to set the MEMORY_TARGET / MEMORY_MAX_TARGET instance initialization parameters may result in the following error message:
AMM should not be confused with Automatic Shared Memory Management (ASMM) where ASMM has no problem with HugePages (See also 1134002.1)
Please also note that ramfs (instead of tmpfs mount over /dev/shm) is not supported for AMM at all. With AMM the Oracle database needs to grow and reduce the size of SGA dynamically. This is not possible with ramfs where it possible and supported with tmpfs (which is the default for the OS installation).
Note that, AMM is setup for ASM instances by default. On the other hand, since the ASM instances do not have a large SGA, using HugePages for ASM instances is not crucial.
If you want to use HugePages make sure that both MEMORY_TARGET / MEMORY_MAX_TARGET initialization parameters are unset (i.e. using “ALTER SYSTEM RESET”) for the database instance.(See also Oracle Database Administrator’s Guide 11g)
NOTE:361323.1 – HugePages on Linux: What It Is… and What It Is Not… (included below)
Linux OS – Version Enterprise Linux 3.0 to Oracle Linux 7.3 with Unbreakable Enterprise Kerne [4.1.12] [Release RHEL3 to OL7U3]
This document describes the HugePages feature in the Linux kernel available for 32-bit and 64-bit architectures. There has been some confusion among the terms and uses related to HugePages. This document should clarify the misconceptions about the feature.
Information in this document is useful for Linux system administrators and Oracle database administrators working with system administrators.
This document covers information about HugePages concept that applies to very large memory (VLM) (>= 4GB) systems for 32-bit and 64-bit architectures including some configuration information and references.
HugePages is a feature integrated into the Linux kernel with release 2.6. This feature basically provides the alternative to the 4K page size (16K for IA64) providing bigger pages.
Regarding the HugePages, there are some other similar terms that are being used like, hugetlb, hugetlbfs. Before proceeding into the details of HugePages, see the definitions below:
Regular Pages and HugePages
This section aims to give a general picture about memory access in virtual memory systems and how pages are referenced.
When a single process works with a piece of memory, the pages that the process uses are reference in a local page table for the specific process. The entries in this table also contain references to the System-Wide Page Table which actually has references to actual physical memory addresses. So theoretically a user mode process (i.e. Oracle processes), follows its local page table to access to the system page table and then can reference the actual physical table virtually. As you can see below, it is also possible (and very common to Oracle RDBMS due to SGA use) that two different O/S processes can point to the same entry in the system-wide page table.
When HugePages are in the play, the usual page tables are employed. The very basic difference is that the entries in both process page table and the system page table has attributes about huge pages. So any page in a page table can be a huge page or a regular page.
HugePages in 2.4 Kernels
The HugePages feature is backported to some 2.4 kernels. Kernel versions 2.4.21-* has this feature (See Note 311504.1 for the distributions with 2.4.21 kernels) but it is implemented in a different way. The feature is completely available. The difference from 2.6 implementation is the organization within the source code and the kernel parameters that are used for configuring HugePages. See Parameters/Setup section below.
Some HugePages Facts/Features
Advantages of HugePages Over Normal Sharing Or AMM (see below)
The Size of a HugePage
The size of a single HugePage varies according to:
The actual size of the HugePage on a specific system can be checked by:
$ grep Hugepagesize /proc/meminfo
The table below shows the sizes of HugePages on different configurations. Note that these are general numbers taken from the most recent versions of the kernels. For a specific kernel source package, you can check for the HPAGE_SIZE macro value (based on HPAGE_SHIFT) for a different (more recent) kernel source tree.
* Some older packaging for the 2.6.5 kernel on SLES8 (like 2.6.5-7.97) can have 2 MB Hugepagesize.
The HugePages reservation feature is fully implemented in 2.6.17 kernel, and thus EL5 (based on 2.6.18) has this feature. The alloc_huge_page() is improved for this. (See kernel source mm/hugetlb.c)
This feature in the Linux kernel enables the Oracle Database to be able to allocate hugepages for the sublevels of the SGA on-demand. The same behaviour is expected for various Oracle Database versions that are certified on EL5.
HugePages and Oracle 11g Automatic Memory Management (AMM)
The AMM and HugePages are not compatible. One needs to disable AMM on 11g to be able to use HugePages. See Document 749851.1 for further information.
What if Not Enough HugePages Configured?
Configuring your Linux OS for HugePages is a delicate process where if you do not configure properly, the system may experience serious problems. If you do not have enough HugePages configured you may encounter:
To avoid / help with such situations Bug 10153816 was filed to introduce a database initialization parameter in 220.127.116.11 (use_large_pages) to help manage which SGAs will use huge pages and potentially give warnings or not start up at all if they cannot get those pages.
What if Too Much HugePages Configured?
It is of course technically possible to configure more than needed. When that is done, the unused part of HugePages allocation will not be available for other purposes on the system and memory shortage can be encountered. Please make sure to configure only for needed amount of hugepages.
The following configurations are a minimal list of documents providing general guidelines to configure HugePages for more than one Oracle RDBMS instance:
For all configurations be sure to have environment variable DISABLE_HUGETLBFS is unset. If it is set (specifically to 1) it will disable the use of HugePages by Oracle database.
The performed configuration is basically based on the RAM installed and combined size of SGA of database instances you are running. Based on that when:
you should revise your HugePages configuration to make it suitable to the new memory framework. If not you may experience one or more problems below on the system:
Kernel Version 2.4
The kernel parameter used for HugePages is vm.hugetlb_pool which is based on MB of memory. RHEL3, Asianux 1.0, SLES8 (Service Pack 3 and over) are examples of distributions with the 2.4 kernels with HugePages support. For the configuration, follow steps below:
1. Start database instance(s)
5. Check available hugepages:
6. Restart database instances
Kernel Version 2.6
The kernel parameter used for HugePages is vm.nr_hugepages which is based on the number of the pages. SLES9, RHEL4 and Asianux 2.0 are examples of distributions with the 2.6 kernel. For the configuration, follow steps below:
1. Start database instance(s)
5. Check available hugepages:
6. Restart database instances
Notes on HugePages in General
For RedHat 6, Oracle Linux 6, SLES 11 and UEK2 kernels please have at “ALERT: Disable Transparent HugePages on SLES11, RHEL6, Oracle Linux 6 and UEK2 Kernels (Doc ID 1557478.1)”
NOTE:341507.1 – Oracle Database Server on Linux on IBM POWER
BUG:10153816 – WHEN USE_LARGE_PAGES=ONLY AND NO HUGEPAGES EXIST STARTUP FAILS NO DIAGNOSTIC
NOTE:465048.1 – ORA-00845 Raised When Starting Instance