How to fix memory leak from big nfs_inode_cache
I received a report that a Linux server was using over 90% of its memory, but its administrator could not find any process using a large amount. My initial check with free showed nothing unusual.
[root@ezylinux ~]# free -m
total used free shared buffers cached
Mem: 7982 7885 97 0 137 366
-/+ buffers/cache: 7380 601
Swap: 4285 0 4285
I checked the processes again for high memory usage and still found none. I then inspected /proc/meminfo with the following command.
[root@ezylinux ~]# cat /proc/meminfo
MemTotal: 8174308 kB
MemFree: 96112 kB
Buffers: 143332 kB
Cached: 378332 kB
SwapCached: 0 kB
Active: 233340 kB
Inactive: 335120 kB
...
Slab: 7480856 kB
...
VmallocTotal: 34359738367 kB
VmallocUsed: 265956 kB
VmallocChunk: 34359470659 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
Slab was using 7306 MB, about 90% of the system’s 8GB RAM. What is Slab? Slab is the memory pool allocator cache for kernel objects. Based on the information above, run slaptop to find the object using the most memory.
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
8603424 8603424 100% 1.01K 2867808 3 7476347K nfs_inode_cache
3080826 3080737 99% 0.21K 171157 18 684628K dentry_cache
24717 12515 50% 0.52K 3531 7 14124K radix_tree_node
11365 11108 97% 0.74K 2273 5 9092K ext3_inode_cache
The example shows nfs_inode_cache using as much as 7GB of memory. Suggested fix
- Run sync to flush filesystem buffers.
- Run the following command to clear dentries and inodes.
echo 2 > /proc/sys/vm/drop_caches
Or run the following command.
sync && echo 2 > /proc/sys/vm/drop_caches