OS hangs caused by excessive use of file descriptors [KES for Linux]

[Egor Erastov]

Symptoms

OS hang, sometimes with open file errors in journals

Customer application degrades with errors "unable to open file", "too many open files"

Hangs and third-party (compatibility) issues often require advanced data collection and are sophisticated to investigate. However, a quick check is possible:

On a system where KESL has worked for some time (not immediately after reboot/restart), validate the output of the following command, ran as root, for numerous records of /usr/bin or /usr/sbin folders

lsof | grep -E 'kesl.+DIR.+\/usr\/s?bin'

 

Root Cause

Under heavy load, KESL may display linear increase in file descriptors usage (sysctl - fs.file-nr) up to system-wide limit (sysctl - fs.file-max) and eventually degradation.

Workaround

Schedule restart of KESL service every week/day, depending on intensity of descriptors growth.

NB: KESL restart will also reset progress of certain tasks like "malware scan" and "database update". Schedule KESL restart outside of tasks timeframes.

Solution

This issue was fixed in KESL 12.1.0.1274, so an update to that or newer version should fix it.