OS hangs caused by excessive use of file descriptors [KES for Linux]

[Egor Erastov]

Symptoms

OS hang, sometimes with open file errors in journals

Customer application degrades with errors "unable to open file", "too many open files"

Hangs and third-party (compatibility) issues often require advanced data collection and are sophisticated to investigate. However, a quick check is possible:

On a system where KESL has worked for some time (not immediately after reboot/restart), validate the output of the following command, ran as root, for numerous records of /usr/bin or /usr/sbin folders

lsof | grep -E 'kesl.+DIR.+\/usr\/s?bin'

 

Root Cause

Under heavy load, KESL may display linear increase in file descriptors usage (sysctl - fs.file-nr) up to system-wide limit (sysctl - fs.file-max) and eventually degradation.

 

Solution

This issue was fixed in KESL 12.1.0.1274, so an update to that or newer version should fix it.