Sunday, January 24, 2016

Hmmm ! Open file count in Linux - What My Understanding Is

It is not that much simple if there is an alert triggered on production or test server having higher open file count. Sometime it happens on busy web infrastructure based on spike traffic levels. The best effort to try figuring out what make that server to utilize more open files. Basically, to clear ground it is better to know the lsof command.

The truth is mostly people go with simple command lsof which displays all open files. To have more clear description better to read through lsof man page completely. Interesting parts are highlighted there. Because lsof command gives much higher value than when you apply filter for particular command, PID or user. Don’t be fall into traps taking misled values for troubleshooting purposes.

Trust me this is actual values taken from one of production server which is yet to go on live roll-out.Actual count is 290133 open file. I believe you don’t have to fool with that value taking granted as a exact file descriptor utilization.

Let’s go with few basic stuff around the corner to see how you could count open file descriptor for particular user, command and PID.

Open File Descriptors for Particular User

Counting file descriptors open by particular user. Here below is the real example where Apache-Tomcat, MySQL Apache HTTPD and standalone internal java modules are running in single server for small client requirement.

[root@RWDAPPSVR tmp]# lsof -u mysql | wc –l
[root@RWDAPPSVR tmp]# lsof -u rewards | wc –l
[root@RWDAPPSVR tmp]# lsof -u root | wc -l

Open File Descriptor for Particular Command

[root@RWDAPPSVR tmp]# lsof -c java | wc -l
[root@RWDAPPSVR tmp]# lsof -c mysqld | wc -l
[root@RWDAPPSVR tmp]# lsof -c httpd | wc –l

Open File Descriptor for Particular PID

[root@RWDAPPSVR tmp]# lsof -p 1812  | wc -l
[root@RWDAPPSVR tmp]# lsof -p 46856 | wc –l

Let’s go with what alternative way to check file descriptor utilization for particular PID.

[root@RWDAPPSVR tmp]# ls /proc/<PID>/fd  | wc -l
[root@RWDAPPSVR tmp]# ls /proc/46856/fd  | wc –l

By looking at figures, you would most probably able to come to conclusion that lsof output is not that much reliable because lsof  open file count and addition of primary process open file count utilization is having huge difference.

It is been heard many times, everything in Linux is a file so for ports sockets are considered as a file in Linux operating system. This is an additional command apart from open files to monitor open network connections using lsof command. 

Advantage here is, it doesn’t require root privilege to see all ports like netstat command. Example of seeing http, https, apache-tomcat, mysql connection are mentioned below.

[root@RWDAPPSVR tmp]# lsof -i :80
[root@RWDAPPSVR tmp]# lsof -i :443
[root@RWDAPPSVR tmp]# lsof -i :8017 | head
[root@RWDAPPSVR tmp]# lsof -i :3306 | head

Output of each command is mentioned here for better understanding.

So to go deeper with this open file descriptor, have you ever thought of file descriptor having limit cap on Linux? If not, yes there is a limit on file descriptor utilization. Now we will see what those limits are and how it could be configured and benefits of having limit on that.

First of all, these are the man pages you would need to go through.

I will mentioned those in point for better understanding.
  • There is a system wide limit for file descriptors which could be configured at Linux kernel level using fs.file-max parameter. To configure that edit /etc/sysctl.conf file and go for a reboot or apply on the fly using sysctl –p /etc/sysctl.conf. 

I believe to keep this nicer and clear. Better to keep your customized kernel parameters in /etc/ sysctl.d/ 99-<server-usage-specific-prefix>-kernel.conf file. E.g. /etc/sysctl.d/ 99-vck-rewards-kernel.conf following recommended way. If so you know what parameters you adjusted as per application and server usage.
  •  And there are soft and hard limits which could configure for open file descriptor as well. To have clear description there is nothing like other than man 5 limits.conf command. Here is the screen snippet taken from limits.conf man page.

  • Normally, if you configure open file descriptor limit for particular user. Configuration looks like below. After configuring exact values which matches for your server capacity. Verify with ulimit command.

  • Login from new SSH session to server or sudo to particular user e.g. rewards, so you will see that above configured values are applied there. 

  •    If configured values are not applied on your new session or login. Make sure is not commented out in /etc/pam.d directory configuration files(e.g. /etc/pam.d/su or /etc/pam.d /system-auth).

Important Note:

   By now, we know system wide file descriptor size determined by fs.file-max kernel parameter. How do you determine that configured value is correct? To know that value you should know current file descriptor utilization.

[root@RWDAPPSVR security]# cat /proc/sys/fs/file-nr

T There are 3 values displayed here. I assume you could go again through screen snippet mentioned above for /proc/sys/fs/file-nr man page. Simply here it is with numbers mapped.
3872    = The number of files presently opened
0       = The number of free file handles
707728  = The maximum number of file handles(value equals to /proc/sys/fs/file-max value.)
Actual File Handlers opened = 3872-0. If this value is near to
file-max, you should be consider increasing file-max value. Hope next time your
sever won’t go out of file descriptor as you know how to configure with great care :) .

 Happy and Enjoy reading. Have a good day for all. Cheers.

No comments:

Post a Comment