System

How to fix memory leak from big nfs_inode_cache

I received a report that a Linux server was using over 90% of its memory, but its administrator could not find any process using a large amount. My initial check with free showed nothing unusual.

[root@ezylinux ~]# free -m
total       used       free     shared    buffers     cached
Mem:          7982       7885         97          0        137        366
-/+ buffers/cache:       7380        601
Swap:         4285          0       4285

I checked the processes again for high memory usage and still found none. I then inspected /proc/meminfo with the following command.

[root@ezylinux ~]# cat /proc/meminfo
MemTotal:      8174308 kB
MemFree:         96112 kB
Buffers:        143332 kB
Cached:         378332 kB
SwapCached:          0 kB
Active:         233340 kB
Inactive:       335120 kB
...
Slab:          7480856 kB
...
VmallocTotal: 34359738367 kB
VmallocUsed:    265956 kB
VmallocChunk: 34359470659 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Slab was using 7306 MB, about 90% of the system’s 8GB RAM. What is Slab? Slab is the memory pool allocator cache for kernel objects. Based on the information above, run slaptop to find the object using the most memory.

   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
8603424 8603424 100%    1.01K 2867808        3  7476347K nfs_inode_cache
3080826 3080737  99%    0.21K 171157       18    684628K dentry_cache
24717  12515  50%    0.52K   3531        7     14124K radix_tree_node
11365  11108  97%    0.74K   2273        5      9092K ext3_inode_cache

The example shows nfs_inode_cache using as much as 7GB of memory. Suggested fix

  1. Run sync to flush filesystem buffers.
  2. Run the following command to clear dentries and inodes. echo 2 > /proc/sys/vm/drop_caches

Or run the following command.

 sync && echo 2 > /proc/sys/vm/drop_caches
EOF.← Back to articles