java.net.SocketException: Too many open files

Discussion:

Jonty Rhods

2011-10-25 20:03:37 UTC

Hi,

I am using solrj and for connection to server I am using instance of the
solr server:

SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");

I noticed that after few minutes it start throwing exception
java.net.SocketException: Too many open files.
It seems that it related to instance of the HttpClient. How to resolved the
instances to a certain no. Like connection pool in dbcp etc..

I am not experienced on java so please help to resolved this problem.

solr version: 3.4

regards
Jonty

Yonik Seeley

2011-10-25 20:07:35 UTC

Permalink

Post by Jonty Rhods
Hi,
I am using solrj and for connection to server I am using instance of the
SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");

Are you reusing the server object for all of your requests?
By default, Solr and SolrJ use persistent connections, meaning that
sockets are reused and new ones are not opened for every request.

-Yonik
http://www.lucidimagination.com

Post by Jonty Rhods
I noticed that after few minutes it start throwing exception
java.net.SocketException: Too many open files.
It seems that it related to instance of the HttpClient. How to resolved the
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty

Bui Van Quy

2011-10-26 04:28:23 UTC

Permalink

Hi,

I had save problem "Too many open files" but it is logged by Tomcat
server. Please check your index directory if there are too much index
files please execute Solr optimize command. This exception is raised by
OS of server, you can google for researching it.

Post by Yonik Seeley

Post by Jonty Rhods
Hi,
I am using solrj and for connection to server I am using instance of the
SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");

Jonty Rhods

2011-10-26 04:56:19 UTC

Permalink

Hi Yonik,

thanks for reply.

Currently I have more than 50 classes and every class have their own
SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");
Majority of classes connect to core0 however there are many cores which is
connecting from different classes.

My senario is
expecting 400000 to 500000 hit on server every day and the server is
deployed on tomcat 6.20 with 12GB heap size to catalina. My OS is Red hat
Linux (production) and using Ubuntu as development server.
Logically I can make a common class for connecting solr server but now
question is :

1. If I using a common class then I must have to use max connection on
httpclient what will the ideal default setting for my current problem.
2. I am expecting concurrent connection at pick time is minimum 5000 hits to
solr server.
3. If I will use SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0"); using common class across all classes
then will it help to resolve current problem with the current load I dont
want my user experience slow response from solr server to get result.
4. As other users overcome with this issue to increase or to make unlimited
TCP/IP setting on OS level. Is it right approach?

As I am newly shift to Java language so any piece of code will much
appreciated and will much easier for me to understand.

thanks.

regards
Jonty

Post by Yonik Seeley

Post by Jonty Rhods
Hi,
I am using solrj and for connection to server I am using instance of the
SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");

the

Post by Jonty Rhods
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty

Markus Jelsma

2011-10-25 20:02:28 UTC

Permalink

This is on Linux? This should help:

echo fs.file-max = 16384 >> /etc/sysctl.conf

On some distro's like Debian it seems you also have to add these settings to
security.conf, otherwise it may not persist between reboots or even shell
sessions:

echo "systems hard nofile 16384
systems soft nofile 16384" >> /etc/security/limits.conf

Post by Jonty Rhods
Hi,
I am using solrj and for connection to server I am using instance of the
SolrServer server = new CommonsHttpSolrServer("
http://localhost:8080/solr/core0");
I noticed that after few minutes it start throwing exception
java.net.SocketException: Too many open files.
It seems that it related to instance of the HttpClient. How to resolved the
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty

Péter Király

2011-10-25 22:08:30 UTC

Permalink

One note for this. I had a trouble to reset the root's limit in
Ubuntu. Somewhere I read, that Ubuntu doesn't give you even the
correct number of limit. The solution to this problem is to run Solr
under another user.

Péter

Post by Markus Jelsma
echo fs.file-max = 16384 >> /etc/sysctl.conf
On some distro's like Debian it seems you also have to add these settings to
security.conf, otherwise it may not persist between reboots or even shell
echo "systems hard nofile 16384
systems soft nofile 16384" >> /etc/security/limits.conf

--
Péter Király
eXtensible Catalog
http://eXtensibleCatalog.org
http://drupal.org/project/xc

Michael Kuhlmann

2011-10-26 07:46:12 UTC

Permalink

Hi;

we have a similar problem here. We already raised the file ulimit on the
server to 4096, but this only defered the problem. We get a
TooManyOpenFilesException every few months.

The problem has nothing to do with real files. When we had the last
TooManyOpenFilesException, we investigated with netstat -a and saw that
there were about 3900 open sockets in Jetty.

Curiously, we only have one SolrServer instance per Solr client, and we
only have three clients (our running web servers).

We have set defaultMaxConnectionsPerHost to 20 and maxTotalConnections
to 100. There should be room enough.

Sorry that I can't help you, we still have not solved tghe problem on
our own.

Greetings,
Kuli

Jonty Rhods

2012-01-24 06:48:33 UTC

Permalink

Hi Kuli,

Did you get the solution of this problem? I am still facing this problem.
Please help me to overcome this problem.

regards

Post by Michael Kuhlmann
Hi;
we have a similar problem here. We already raised the file ulimit on the
server to 4096, but this only defered the problem. We get a
TooManyOpenFilesException every few months.
The problem has nothing to do with real files. When we had the last
TooManyOpenFilesException, we investigated with netstat -a and saw that
there were about 3900 open sockets in Jetty.
Curiously, we only have one SolrServer instance per Solr client, and we
only have three clients (our running web servers).
We have set defaultMaxConnectionsPerHost to 20 and maxTotalConnections
to 100. There should be room enough.
Sorry that I can't help you, we still have not solved tghe problem on
our own.
Greetings,
Kuli

the

Post by Jonty Rhods
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty

Michael Kuhlmann

2012-01-24 16:40:07 UTC

Permalink

Hi Jonty,

no, not really. When we first had such problems, we really thought that
the number of open files is the problem, so we implemented an algorithm
that performed an optimize from time to time to force a segment merge.
Due to some misconfiguration, this ran too often. With the result that
an optimize was issued before thje previous optimization was finished,
which is a really bad idea.

We removed the optimization calls, and since then we didn't have any
more problems.

However, I never found out the initial reason for the exception. Maybe
there was some bug in Solr's 3.1 version - we're using 3.5 right now -,
but I couldn't find a hint in the changelog.

At least we didn't have this exception for more than two months now, so
I'm hoping that the cause for this has disappeared somehow.

Sorry that I can't help you more.

Greetings,
Kuli

Post by Jonty Rhods
Hi Kuli,
Did you get the solution of this problem? I am still facing this problem.
Please help me to overcome this problem.
regards

the

Post by Jonty Rhods
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty

Sethi, Parampreet

2012-01-24 17:10:35 UTC

Permalink

Hi Jonty,

You can try changing the maximum number of files opened by a process using
command:

ulimit -n XXX

In case, the number of opened files is not increasing with time and just a
constant number which is larger than system default limit, this should fix
it.

-param

Post by Michael Kuhlmann
Hi Jonty,
no, not really. When we first had such problems, we really thought that
the number of open files is the problem, so we implemented an algorithm
that performed an optimize from time to time to force a segment merge.
Due to some misconfiguration, this ran too often. With the result that
an optimize was issued before thje previous optimization was finished,
which is a really bad idea.
We removed the optimization calls, and since then we didn't have any
more problems.
However, I never found out the initial reason for the exception. Maybe
there was some bug in Solr's 3.1 version - we're using 3.5 right now -,
but I couldn't find a hint in the changelog.
At least we didn't have this exception for more than two months now, so
I'm hoping that the cause for this has disappeared somehow.
Sorry that I can't help you more.
Greetings,
Kuli

Post by Jonty Rhods
Hi Kuli,
Did you get the solution of this problem? I am still facing this problem.
Please help me to overcome this problem.
regards

the

Post by Jonty Rhods
instances to a certain no. Like connection pool in dbcp etc..
I am not experienced on java so please help to resolved this problem.
solr version: 3.4
regards
Jonty