Today one customer said the data sync of Oracle Dataguard had issue and our guys checked the alert log then sent below errors to me:
ORA-01034: ORACLE not available ORA-27102: out of memory Linux-x86_64 Error: 12: Cannot allocate memory
Also had some ORA-00313 errors about failure to open data/redo files.
While he also confirmed the instance was opened and was running.
I asked to check the memory usage, detail trace log and system message log file continually.
The system has 200G physical memory and 16G swap while at that time we only got 3M free swap! so we did get memory issue.
Our guys checked the processes using most of the swap and killed some of them while did not fix it as only very small part of swap was freed.
I got more detailed memory info through /proc/meminfo:
[root@erpdg ~]# cat /proc/meminfo MemTotal: 206352512 kB MemFree: 1260976 kB MemAvailable: 520844 kB Buffers: 3160 kB Cached: 41799796 kB SwapCached: 106432 kB Active: 40449436 kB Inactive: 3372480 kB ...... SwapTotal: 16777212 kB SwapFree: 108580 kB ..... HugePages_Total: 72000 HugePages_Free: 71984 HugePages_Rsvd: 113 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 323520 kB DirectMap2M: 24842240 kB DirectMap1G: 186646528 kB
If you read the above output carefully, you would already know the reason.
Then I asked our guys to do below actions:
1. Check the SGA and Memory target settings 2. Stop the database instance 3. sync; echo 3 > /proc/sys/vm/drop_caches 4. start the database instance again 5. cat /proc/meminfo again
The SGA was about 120G, and the new content of /proc/meminfo was as below:
[root@erpdg ~]# cat /proc/meminfo MemTotal: 206352512 kB MemFree: 44060916 kB MemAvailable: 43266384 kB Buffers: 4192 kB Cached: 1122640 kB SwapCached: 39268 kB ...... SwapTotal: 16777212 kB SwapFree: 15826640 kB ...... HugePages_Total: 72000 HugePages_Free: 10672 HugePages_Rsvd: 113 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 323520 kB DirectMap2M: 24842240 kB DirectMap1G: 186646528 kB
So this issue was caused by the HugePage memory.
For some reasons, last time when the database instances was started, it did not make use of HugePage memory as there were about 140G free HugePage memory. After the above actions now we had about 20G HugePage memory. It was correct as the SGA size was about 120G.
