Hmmm ! Open file count in Linux - What My Understanding Is

It is not that much simple if there is an alert triggered on production or test server having higher open file count. Sometime it happens on busy web infrastructure based on spike traffic levels. The best effort to try figuring out what make that server to utilize more open files. Basically, to clear ground it is better to know the lsof command.

The truth is mostly people go with simple command lsof which displays all open files. To have more clear description better to read through lsof man page completely. Interesting parts are highlighted there. Because lsof command gives much higher value than when you apply filter for particular command, PID or user. Don’t be fall into traps taking misled values for troubleshooting purposes.

Trust me this is actual values taken from one of production server which is yet to go on live roll-out.Actual count is 290133 open file. I believe you don’t have to fool with that value taking granted as a exact file descriptor utilization.

Let’s go with few basic stuff around the corner to see how you could count open file descriptor for particular user, command and PID.

Open File Descriptors for Particular User

Counting file descriptors open by particular user. Here below is the real example where Apache-Tomcat, MySQL Apache HTTPD and standalone internal java modules are running in single server for small client requirement.

[root@RWDAPPSVR tmp]# lsof -u mysql | wc –l

[root@RWDAPPSVR tmp]# lsof -u rewards | wc –l

[root@RWDAPPSVR tmp]# lsof -u root | wc -l

Open File Descriptor for Particular Command

[root@RWDAPPSVR tmp]# lsof -c java | wc -l

[root@RWDAPPSVR tmp]# lsof -c mysqld | wc -l

[root@RWDAPPSVR tmp]# lsof -c httpd | wc –l

Open File Descriptor for Particular PID

[root@RWDAPPSVR tmp]# lsof -p 1812  | wc -l

[root@RWDAPPSVR tmp]# lsof -p 46856 | wc –l

Let’s go with what alternative way to check file descriptor utilization for particular PID.

[root@RWDAPPSVR tmp]# ls /proc/<PID>/fd  | wc -l
[root@RWDAPPSVR tmp]# ls /proc/46856/fd  | wc –l

By looking at figures, you would most probably able to come to conclusion that lsof output is not that much reliable because lsof  open file count and addition of primary process open file count utilization is having huge difference.

It is been heard many times, everything in Linux is a file so for ports sockets are considered as a file in Linux operating system. This is an additional command apart from open files to monitor open network connections using lsof command. 

Advantage here is, it doesn’t require root privilege to see all ports like netstat command. Example of seeing http, https, apache-tomcat, mysql connection are mentioned below.

[root@RWDAPPSVR tmp]# lsof -i :80
[root@RWDAPPSVR tmp]# lsof -i :443
[root@RWDAPPSVR tmp]# lsof -i :8017 | head
[root@RWDAPPSVR tmp]# lsof -i :3306 | head

Output of each command is mentioned here for better understanding.

So to go deeper with this open file descriptor, have you ever thought of file descriptor having limit cap on Linux? If not, yes there is a limit on file descriptor utilization. Now we will see what those limits are and how it could be configured and benefits of having limit on that.

First of all, these are the man pages you would need to go through.

I will mentioned those in point for better understanding.

• There is a system wide limit for file descriptors which could be configured at Linux kernel level using fs.file-max parameter. To configure that edit /etc/sysctl.conf file and go for a reboot or apply on the fly using sysctl –p /etc/sysctl.conf. 

I believe to keep this nicer and clear. Better to keep your customized kernel parameters in /etc/ sysctl.d/ 99-<server-usage-specific-prefix>-kernel.conf file. E.g. /etc/sysctl.d/ 99-vck-rewards-kernel.conf following recommended way. If so you know what parameters you adjusted as per application and server usage.

•  And there are soft and hard limits which could configure for open file descriptor as well. To have clear description there is nothing like other than man 5 limits.conf command. Here is the screen snippet taken from limits.conf man page.

• Normally, if you configure open file descriptor limit for particular user. Configuration looks like below. After configuring exact values which matches for your server capacity. Verify with ulimit command.

• Login from new SSH session to server or sudo to particular user e.g. rewards, so you will see that above configured values are applied there. 

•    If configured values are not applied on your new session or login. Make sure pam_limits.so is not commented out in /etc/pam.d directory configuration files(e.g. /etc/pam.d/su or /etc/pam.d /system-auth).

Important Note:

   By now, we know system wide file descriptor size determined by fs.file-max kernel parameter. How do you determine that configured value is correct? To know that value you should know current file descriptor utilization.

[root@RWDAPPSVR security]# cat /proc/sys/fs/file-nr

T There are 3 values displayed here. I assume you could go again through screen snippet mentioned above for /proc/sys/fs/file-nr man page. Simply here it is with numbers mapped.

     
3872    = The number of files presently opened
0       = The number of free file handles
707728  = The maximum number of file handles(value equals to /proc/sys/fs/file-max value.)

Actual File Handlers opened = 3872-0. If this value is near to
file-max, you should be consider increasing file-max value. Hope next time your
sever won’t go out of file descriptor as you know how to configure with great care :) .

 Happy and Enjoy reading. Have a good day for all. Cheers.

Special Note on mysqld_safe — MySQL Server Startup Script

When it happens to work on mysql databases, it is common to use mysqld_safe script. However, there are most important points which hidden behind the sense. Sometimes as mysql administrators, implementation engineers, system engineers might be missing that one.

More or less, this page would be familer to people who deal with mysql. Reference is mentioned here in case to have a deeper look and complete reading (http://dev.mysql.com/doc/refman/5.7/en/mysqld-safe.html).

For better understanding, here there are two snippets took from mysqld_safe script comments itself and mysql documentation mysqld_safe section. Still if reader would be wondering what is the point this guy going to make better go through simple wording in that snippets. 

Snippet taken from mysqld_safe script comment

 Snippet taken from mysql documentation mysqld_safe

Does that make sense when reading that? Most probably, it is noticed mysqld_safe script looks for at least one condition to be met to execute. Which is special that other start up script if you happen to work with apache HTTPD, apache Tomcat, WebLogic, and IBM WebSphere servers. In simple terms no matter what when executing mysqld_safe script. Current working directory must be mysql_installation_directory. Things would become worse, when using customized installation location. For instance, most of the time product based companies tend to install mysql as their preferred locations. What if that this fact is not aware to you who is going to implement mysql sever. In normal run, startup script comes with mysql binary installation works perfectly since that mysql.server script keeps that rule not violating.

Trust me, real instance occurred on setting up CentOS 7 cluster with mysql service for my first time. Because mysql resource doesn’t start up throwing below error many times. It doesn’t give clue to start troubleshooting. Oh such a pain again.

Issue on CentOS7 cluster mysql resource agent.

I’m pretty sure that mysql installation perfectly okey. It would be able to start/stop/login using init.d script without CentOS7 cluster.

Having hours sitting in front of my notebook trying to dig deeper resource agent script why this thing is throwing. Finally, it is figured out that simplest condition for mysqld_safe is not catered in mysql resource agent script.Having simple cd mysql_installation_directory works with out issue. 

Caution: If you happened to use mysqld_safe inside any script. Make sure you change directory to mysql installation location. Or follow the guide lines given in mysql documentation if it failed when running script from mysql installation location.

Solution.

Install Missing Manual In CentOS 7

It is nothing so funny that sometimes, even though, you would be searching man pages correctly. Man pages are not available. Same thing happens to me today when using one instance of Azure Could VM. Little bit of googling give me a positive answer and valid reason why it is not available.

This is what I got for the first time when I was looking for a help on proc file system.

Be remember if you came across such situation, sometimes, CentOS minimal/ customized installation (specially for cloud VMs) doesn’t comes with complete man pages.

There is no perfect solution for that but, Internet teaches me that it is possible to install most of the man pages using man-pages package.

[rewards@RWDAPPSVR-Test logs]$ yum info man-pages

After installation, my problem solved. Man pages now showing proc file system help pages. Screenshot for installation and displaying proc man pages shown below.

If you face such situations,hope this workaround helps you.Happy reading.Ensure to enjoy your weekend as you work hard extended hours on week days.Cheers !.

Human way to read your TCP traffic - tcpflow Utility

If you happened to work on low level TCP protocol, there is only better way to troubleshoot. Guess what, people will say different answers. But, Packet capture will do its job. When system running on Linux, most probably, you may take tcpdump in .pcap format using tcpdump command and analyze later on using graphical tool like Wireshark which has much advanced features.

Nonetheless, but having .pcap file and using Wireshark is not real-time and bit cumbersome. You know what I mean when you work on real production environment where people are ready to sit on your head and you are really under pressure most of the working hours.

Tcpflow is a handy tool which will take all your burden and all headaches when it comes to packet monitoring. It is not installed by default in commonly used distribution line CentOS/Fedora/RHEL. .rpm package is freely available to download and install. If you installed tcpflow, no doubt that you wouldn’t look for any other tool. Installation is pretty much easy and straight forward. Just two commands all together.

[root@cluster-node01 installs]#  wget http://pkgs.repoforge.org/tcpflow/tcpflow-0.21-1.2.el6.rf.x86_64.rpm

[root@cluster-node01 installs]# rpm -ivh tcpflow-0.21-1.2.el6.rf.x86_64.rpm

I guess you don’t need a better description here for tcpflow command. Then tcpflow man page give an exact details. I have highlighted what is important. 

Basically, for me tcpflow is helps to monitor traffic goes via different application specific services. Our application calls external Rest API via web Services. Sometimes I used tcpflow to make sure http/https traffic goes via particular Network Interface card. This is one of real example of monitoring request and response traffic using tcpflow command.

[root@RWDAPPSVR ~]# tcpflow -i any port 7000 –c

Above tcpflow command monitors traffic goes via port 7000 on listening on any network interface. And output is written to console itself (-c option). Port and any option is quite understandable for novice users.

Hopefully, I bet you are going to be a fan of tcpflow. Have a good and happy reading Cheers.

Spin It Up Again.....

It was 10th April 2011 last blog was written.Five month has already gone. I know I should write more in my blog in last five months but I feel I wouldn't have better mood to write. Today I will write this blog since I don't want to see my blog dying. So sorry for being silent. Personal things matters to  everyone. Now... I'm proud because still I can breath Still I can see the color of the red-wine sky in the evening. And also I feel the smell of the wind blows around me.

Life is a gift. It is amazing even though it is not fair. And believe me life is a miracle....

© Nuwan Arambage-"transcending verge of life"

Where Linux Shines

We all know the truth that Mr Gates have conquered the Desktop OS market with a large market share. It it the real truth all the time we believe. Most of the computer users think it is really easy to work in Windows rather than other operating systems. In my personal belief , Microsoft have spent bulk of its hardly earned money to make people's mind set dressing with huge marking hype.

We can't blame Microsoft because it is their business. Mr Jobs keeps Apple's promise to his customers while competing with Mr Gates. I'm not a Microsoft nor Apple advocate justifying what they do in their business. I'm not going to compare none of those operating system who makes the big leap in OS race. It is obvious, when it comes to Desktop market Gates and Jobs rule the market even if Other OS like Linux distributions has tiny user community. My question is where does this the Great Linux shows it colors all the way?. Perhaps you can find where Linux shines but probably you will not find precise information regarding this matter. I think why I write this post where you would find more of Linux is being used.

It is no wonder you will use Google search more than three times per day. Believe me,the moment you click the search button you become a Linux user since Google servers are run using Linux distros that are customized by Google engineers. Will you believe me or not. This is the extraction form Wikipedia.”Servers are commodity-class x86 PCs running customized versions of Linux” . If you need more information read Google platform post in Wikipedia.  

Same thing applies to other search engine Yahoo.I don't want repeat my words. However big fish in Yahoo- Sven Dummer, Director of Linux engineering told the story in his own words. “ 75% of Yahoo’s Web sites and services run on Linux. The rest? It runs on FreeBSD.”. As a matter of fact it is the truth. These are confirmed facts but e-commerce web sites are run on Linux server even if not confirmed.

This is the another story that Linux supports to render Oscar award winning film. Any guesses, no one haven't missed a chance to watch Avatar. A movie directed by James Cameron. Special effects contained in Avatar movie had been rendered by Ubuntu Server farm at Weta Digital which is located in Wellington, New Zealand. Most of the blockbuster movies are rendered at Weta Digital creative studio by using Ubuntu server farm. District9, Day the Earth Stood Still, Jumper, King Kong, Lord of the Rings, Fantastic Four, Eragon, X-Men, i-Robot are great products done by Weta Digital. According to system administrator at Weta Digital, he told that  Ubuntu is at the core of almost all their systems. Ubuntu runs on all of their 3D rendering servers, and at least 90% of all their desktop systems. More detailed information about the Ubuntu powered rendering farm that rendered the CGI of Avatar is available from the Information Management and IT World websites.

The system is entirely water-cooled, as traditional air and fan cooling systems are not sufficient during near deadline times, due to constant high load on the servers. The photograph below shows the enormous water cooling system cooling a portion of the server farm.

Other fact you should know that number of stock exchange transactions happens on top Linux based platform. The newcomer who have migrated form Window server to Novell Linux due to overload crash occurred on September 8th 2008 is LSE(London Stock Exchange). Why they migrating is the more important thing to reveal. It is Just 3S's. Security , Speed and obvious Stability. LSE is not the only one which runs Linux. Other than that,you can find more by clicking this link. As a Sri Lankan, I'm really happy because Millennium IT is company located in SL that have provided the stock market solution to Landon Stock Exchange now onwards. Most of the engineers work there are Sri Lankans. So proud to be a Sri Lankan.

Everyone talks about Super computers and its usages. But no one cares what operating system they are rely on. So far , according to the statics it shows 91.8% of super computers run Linux based operating system. How is that, good or bad. You should be really happy, how long this open source Linux goes it own silent way. This is the best ever story that I have heard and I was fully happy about it.

Android is designed on top of Linux kernel. Now mobile devices are moving towards Android platform since it is the new trend in mobile market. Think a what makes Android so special. It is undoubtedly Linux.

Most laptop and notebook manufacturers tend to shipped Linux distributions(more preferably Ubuntu) with their hardware. It is a new tread has already started. Dell computers did it, firstly Asus Inc did it. Now HP keep going with releasing mass market PCs and notebooks with Linux. Electronic manufacturers mostly use Linux in their embedded devices since Linux kernel is highly configurable for specific needs.

So what is the real thing that I need to point out?. Linux is everywhere, its usage is enormous. Everyone might be using it even without knowing it. Could I make a suggestion here. Don't you feel it is worth using Linux in your desktop computer. If you do so, you will be a part of huge community and you will add great value to your skill set.

We are a Windows user from out childhood. Yes it is the truth. But do you want to be a Windows user ever?.Eventually, One thing come to my mind. It is my favorite quote all the time. “Never did the same mistake twice. There are new ones out there try something new so you will learn something new everyday”.

© Nuwan Arambage-"transcending verge of life"

Karmic Koala Ride Part One

Still I remember the day that I have ordered the free Ubuntu CD.I just ordered it because one of my friends told me it is free and they deliver to your address with no cost. I don’t have any idea what Ubuntu is up to and why they give it for free. Why such a big community working on non-profit oriented purpose.I just only heard it is a Linux distribution. I was curious so then I installed it on my desktop. Pretty interesting, different look and feel, but I don’t know how to work with it. Frankly speaking, it is not user friendly like Windows at that time. After two days I decided to remove Ubuntu and then Windows XP becomes my OS in usual way. My first Ubuntu attempt failed and Bill Gates wins. I feel this is like really a confession that states how I finally become an Ubuntu user. Time went up; day by day I was getting stick to Windows.

Suddenly, I was called for an interview. It is a software company that makes ATM solution and banking applications. I have prepared for interview and it is my first interview. While I was interviewed, he told me we are using Sun Solaris in our server so do you have any UNIX knowledge. Ah God, I feel tranquility in my brain. I have no choice, I was thinking whether tell the truth or not. As I said before I don’t have hard experience in UNIX or UNIX like OS.I was trying to handle the situation telling I have studied Operating System in my degree course and we have installed Fedora 11 on VMware. Now, things were getting worse. He keeps asking a lot of smart questions. Eventually, I was failed the interview and they sent me the mail mentioning, at the moment we are unable to recruit you since you are not suit for our requirement. I was so frustrated and keep thinking why the hell this Linux is more important. I don’t want failed twice and try to keep things getting back on track. I have ordered Open Solaris 9. Installed, try to find a way to learn things, unfortunately abandoned again.Window is the only option that I have. I decided to drag with Windows XP.

By the way, things are getting change. Little by little I’m going love open source culture and the way it modeled and works. Seriously I love it now. I strongly believe internal working of software should not be closed from its users and it is also free as free beer.In my personal opinion, Open source community done a great job towards Software industry. At last I become an Ubuntu user not the hard core user as per now, want to be a powered one someday. It is pretty interesting throwing Windows and Gates away from my computer forever and ever. In reality, everything depends on your capacity. How hard you work and how efficiently you work. Converting form Windows to Ubuntu won’t be a hard task at all but your resistance to change will cage you in Windows. In my personal experience, another major reason why I abandoned Linux twice is I don’t know how to work around it at first hand. Sometimes, when you are going to use Ubuntu first time you really become frustrated even if there are forums support, community documentation out there. Finally practicality and experience always matters. Learning in hard way is much better but I would like to share my Ubuntu experience which will at least help you to begin your own Ubuntu journey with less effort and less frustration.

You may have experience installing Windows XP. Installing Ubuntu is same as Windows since it starts along with GUI. Nothing happens behind installing Ubuntu other than boot up the live CD and few mouse clicks. You are able to install Ubuntu without any sweat. After that think what you want to do....Simply I will point down what I need to do with Ubuntu 9.10.

• First you need to install software on top of Ubuntu.
• Removing software and updating installed software.
• If you are a Java developer you need set up development environment such as installing JDK , Eclipse IDE and other required softwares.
• If you want to be a better Linux user, your best friend would be your terminal all the way. I mean you should familiar with frequently used commands.

Up to now, I think most of hard part is over. Meaning that you don't want to shift from Ubuntu to Windows because most of the things that you do is possible in your computer now.

Stay tuned to follow the forth coming post that makes you Ubuntu journey as easy as pie. At first I will cover how to install packages(software) on top of Ubuntu 9.10(Karmic Koala).

© Nuwan Arambage-"transcending verge of life"