TooManyClauses: maxClauseCount is set to 1024

[sawanverma]

Its quite confusing about this error.

I had a situation where i have to turn on the highlighting. In some cases
though the number of docs found for a particular query was for example say
2, the highlighting was coming only for 1. I did some checks and found that
that particular text searched was in the bigger document and was towards the
end of the document.

So i increased the hl.maxAnalyzedChar from default value of 51200 to a
bigger value say 5000000. And then it started working, i mean now the
highlighting was working properly.

Now, i have encountered one more problem with the same error.

There is a document which is returning the maxClauseCount error when i do
search on content:*. The document is quite big big in size and the
hl.maxAnalyzedChars was default i.e. 51200.

I tried decreasing that i found that the error is coming exactly at 31375
char (this is did my setting the hl.maxAnalyzedChars to 31375. it worked
fine till 31374).

Solutions are most welcome as i am in great need of this.

Sample query is as below
http://localhost:8983/solr/test/select/?q=content:* AND
obs_date:[2010-01-01T00:00:00Z%20TO%202011-12-31T23:59:59Z]&fl=content&hl=true&hl.fl=content&hl.snippets=1&hl.fragsize=1500&hl.requireFieldMatch=true&hl.alternateField=content&hl.maxAlternateFieldLength=1500&hl.maxAnalyzedChars=31375&facet.limit=200&facet.mincount=1&start=64&rows=1&sort=obs_date%20desc

Regards,
Sawan



--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965.html
Sent from the Solr - User mailing list archive at Nabble.com.

[pravesh]

Just increase the value of /maxClauseCount/ in your solrconfig.xml. Keep it
large enough.

Best
Pravesh



--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056966.html
Sent from the Solr - User mailing list archive at Nabble.com.

[sawanverma]

Thanks Pravesh.

But won't that hit the query performance? Still what would be the ideal value to increase? Say this error may come even if we increase the value from 1024 to say 5120?
Have tried increasing the value and it had hit the performance.

Regards,
Sawan

From: pravesh [via Lucene] [mailto:ml-node+***@n3.nabble.com]
Sent: Thursday, April 18, 2013 5:06 PM
To: Sawan Verma
Subject: Re: TooManyClauses: maxClauseCount is set to 1024

Just increase the value of maxClauseCount in your solrconfig.xml. Keep it large enough.

Best
Pravesh
________________________________
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056966.html
To unsubscribe from TooManyClauses: maxClauseCount is set to 1024, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056965&code=c2F3YW4udmVybWFAZ2xhc3NiZWFtLmNvbXw0MDU2OTY1fC0xMTI5MDQ2NDY1>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056968.html
Sent from the Solr - User mailing list archive at Nabble.com.

[pravesh]

Update:

Also remove your range queries from the main query and specify it as a
filter query.


Best
Pravesh



--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056969.html
Sent from the Solr - User mailing list archive at Nabble.com.

[Yonik Seeley]

Can you provide a full stack trace of the exception?

There's a maxClauseCount in solrconfig.xml that you can increase to
work around the issue.

-Yonik
http://lucidworks.com

Post by sawanverma
Its quite confusing about this error.
I had a situation where i have to turn on the highlighting. In some cases
though the number of docs found for a particular query was for example say
2, the highlighting was coming only for 1. I did some checks and found that
that particular text searched was in the bigger document and was towards the
end of the document.
So i increased the hl.maxAnalyzedChar from default value of 51200 to a
bigger value say 5000000. And then it started working, i mean now the
highlighting was working properly.
Now, i have encountered one more problem with the same error.
There is a document which is returning the maxClauseCount error when i do
search on content:*. The document is quite big big in size and the
hl.maxAnalyzedChars was default i.e. 51200.
I tried decreasing that i found that the error is coming exactly at 31375
char (this is did my setting the hl.maxAnalyzedChars to 31375. it worked
fine till 31374).
Solutions are most welcome as i am in great need of this.
Sample query is as below
http://localhost:8983/solr/test/select/?q=content:* AND
obs_date:[2010-01-01T00:00:00Z%20TO%202011-12-31T23:59:59Z]&fl=content&hl=true&hl.fl=content&hl.snippets=1&hl.fragsize=1500&hl.requireFieldMatch=true&hl.alternateField=content&hl.maxAlternateFieldLength=1500&hl.maxAnalyzedChars=31375&facet.limit=200&facet.mincount=1&start=64&rows=1&sort=obs_date%20desc
Regards,
Sawan
--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965.html
Sent from the Solr - User mailing list archive at Nabble.com.

[sawanverma]

Hi Yonik,

Thanks for your reply.

I tried increasing the maxClauseCount to a bigger value. But what could be the ideal value and will not that hit the performance? What are the chances that if we increase the value we will not face this issue again?

As you asked pasting below the full trace of the error

Problem accessing /solr/ar/select/. Reason:
maxClauseCount is set to 1024

org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:136)
at org.apache.lucene.search.BooleanQuery.add(BooleanQuery.java:127)
at org.apache.lucene.search.ScoringRewrite$1.addClause(ScoringRewrite.java:51)
at org.apache.lucene.search.ScoringRewrite$1.addClause(ScoringRewrite.java:55)
at org.apache.lucene.search.ScoringRewrite$3.collect(ScoringRewrite.java:95)
at org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:38)
at org.apache.lucene.search.ScoringRewrite.rewrite(ScoringRewrite.java:93)
at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:312)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:158)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.extract(WeightedSpanTermExtractor.java:98)
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor.getWeightedSpanTerms(WeightedSpanTermExtractor.java:391)
at org.apache.lucene.search.highlight.QueryScorer.initExtractor(QueryScorer.java:216)
at org.apache.lucene.search.highlight.QueryScorer.init(QueryScorer.java:185)
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:205)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:490)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:401)
at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:131)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:186)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Regards,
Sawan

From: Yonik Seeley-4 [via Lucene] [mailto:ml-node+***@n3.nabble.com]
Sent: Thursday, April 18, 2013 5:09 PM
To: Sawan Verma
Subject: Re: TooManyClauses: maxClauseCount is set to 1024

Can you provide a full stack trace of the exception?

There's a maxClauseCount in solrconfig.xml that you can increase to
work around the issue.

-Yonik
http://lucidworks.com

Post by sawanverma
Its quite confusing about this error.
I had a situation where i have to turn on the highlighting. In some cases
though the number of docs found for a particular query was for example say
2, the highlighting was coming only for 1. I did some checks and found that
that particular text searched was in the bigger document and was towards the
end of the document.
So i increased the hl.maxAnalyzedChar from default value of 51200 to a
bigger value say 5000000. And then it started working, i mean now the
highlighting was working properly.
Now, i have encountered one more problem with the same error.
There is a document which is returning the maxClauseCount error when i do
search on content:*. The document is quite big big in size and the
hl.maxAnalyzedChars was default i.e. 51200.
I tried decreasing that i found that the error is coming exactly at 31375
char (this is did my setting the hl.maxAnalyzedChars to 31375. it worked
fine till 31374).
Solutions are most welcome as i am in great need of this.
Sample query is as below
http://localhost:8983/solr/test/select/?q=content:* AND
obs_date:[2010-01-01T00:00:00Z%20TO%202011-12-31T23:59:59Z]&fl=content&hl=true&hl.fl=content&hl.snippets=1&hl.fragsize=1500&hl.requireFieldMatch=true&hl.alternateField=content&hl.maxAlternateFieldLength=1500&hl.maxAnalyzedChars=31375&facet.limit=200&facet.mincount=1&start=64&rows=1&sort=obs_date%20desc
Regards,
Sawan
--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965.html
Sent from the Solr - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056967.html
To unsubscribe from TooManyClauses: maxClauseCount is set to 1024, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056965&code=c2F3YW4udmVybWFAZ2xhc3NiZWFtLmNvbXw0MDU2OTY1fC0xMTI5MDQ2NDY1>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056972.html
Sent from the Solr - User mailing list archive at Nabble.com.

[sawanverma]

Yonik,

When i remove the sort part from the query below it works fine. But with
sort it throws the exception

http://localhost:8983/solr/test/select/?q=content:*&fl=content&hl=true&hl.fl=content&hl.maxAnalyzedChars=31375&start=64&rows=1&sort=obs_date%20desc
-- > Throws Exception



http://localhost:8983/solr/test/select/?q=content:*&fl=content&hl=true&hl.fl=content&hl.maxAnalyzedChars=31375&start=64&rows=1
-- Works fine.

From the above its clear that sort is causing the problem. Any idea why is

this happening and how to fix this?

Regards,
sawan





--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4056974.html
Sent from the Solr - User mailing list archive at Nabble.com.

[Shawn Heisey]

Post by sawanverma
Hi Yonik,
Thanks for your reply.
I tried increasing the maxClauseCount to a bigger value. But what could be the ideal value and will not that hit the performance? What are the chances that if we increase the value we will not face this issue again?

Changing the maxBooleanClauses value does not affect performance. It's
just an arbitrary limit on query complexity. You can make it as big as
you want and Solr's performance will not change. For most people, 1024
is plenty. For others, we have no idea how many clauses are needed.

The queries themselves with large numbers of clauses are what affects
performance, and the only way to improve it is to decrease the query
complexity. Chances are good that you are already experiencing the
performance hit associated with large queries. Adding more clauses to a
query will reduce performance. If you find yourself in a situation
where you continually need more boolean clauses, you may need to start
over and create a better design.

The maxBooleanClauses value is just a safety net, created long ago when
Lucene worked differently than it does now. There is a discussion
currently happening among committers about whether that limit even needs
to exist. Very likely the limit in Solr will be removed in the near future.

Thanks,
Shawn

[sawanverma]

Shawn,

Thanks a lot for your reply. But I am confused again if the following query is complex.
http://localhost:8983/solr/test/select/?q=content:*&fl=content&hl=true&hl.fl=content&hl.maxAnalyzedChars=31375&start=64&rows=1&sort=obs_date%20desc

Is that because of content : *? The only unusual thing is the field size of the "content". In this particular case content field has enormously big data. Since this problem comes only when we do a search on * for content field. Is there a way that we can split the doc size?

Regards,
Sawan

From: Shawn Heisey-4 [via Lucene] [mailto:ml-node+***@n3.nabble.com]
Sent: 18 April 2013 PM 09:38
To: Sawan Verma
Subject: Re: TooManyClauses: maxClauseCount is set to 1024

Post by sawanverma
Hi Yonik,
Thanks for your reply.
I tried increasing the maxClauseCount to a bigger value. But what could be the ideal value and will not that hit the performance? What are the chances that if we increase the value we will not face this issue again?

Changing the maxBooleanClauses value does not affect performance. It's
just an arbitrary limit on query complexity. You can make it as big as
you want and Solr's performance will not change. For most people, 1024
is plenty. For others, we have no idea how many clauses are needed.

The queries themselves with large numbers of clauses are what affects
performance, and the only way to improve it is to decrease the query
complexity. Chances are good that you are already experiencing the
performance hit associated with large queries. Adding more clauses to a
query will reduce performance. If you find yourself in a situation
where you continually need more boolean clauses, you may need to start
over and create a better design.

The maxBooleanClauses value is just a safety net, created long ago when
Lucene worked differently than it does now. There is a discussion
currently happening among committers about whether that limit even needs
to exist. Very likely the limit in Solr will be removed in the near future.

Thanks,
Shawn


________________________________
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4057027.html
To unsubscribe from TooManyClauses: maxClauseCount is set to 1024, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056965&code=c2F3YW4udmVybWFAZ2xhc3NiZWFtLmNvbXw0MDU2OTY1fC0xMTI5MDQ2NDY1>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4057060.html
Sent from the Solr - User mailing list archive at Nabble.com.

[Shawn Heisey]

Post by sawanverma
Shawn,
Thanks a lot for your reply. But I am confused again if the following query is complex.
http://localhost:8983/solr/test/select/?q=content:*&fl=content&hl=true&hl.fl=content&hl.maxAnalyzedChars=31375&start=64&rows=1&sort=obs_date%20desc

I hardly know anything about highlighting, so nothing that I say here
may have any relevance to your situtation at all.

A query of content:* strikes me as an invalid query. If you are
shooting for all documents where content exists and excluding those
where it doesn't exist, I would think that 'q=content:[* TO *]' (the TO
must be uppercase) would be a better option.

Exactly how your query gets expanded into a something that exceeds
maxBooleanClauses is a complete mystery to me, and probably does have
something to do with the highlighting.

Thanks,
Shawn

[sawanverma]

Shawn,

Giving content:[* TO *] gives the same error but when I give content:[a TO z] it works fine. Can you please explain what does it mean when I give content:[a TO z]? Can I use this as workaround? The datatype of content field is text_en.

Thanks again for you replies and suggestions.

Regards,
Sawan

From: Shawn Heisey-4 [via Lucene] [mailto:ml-node+***@n3.nabble.com]
Sent: Friday, April 19, 2013 12:33 AM
To: Sawan Verma
Subject: Re: TooManyClauses: maxClauseCount is set to 1024

Post by sawanverma
Shawn,
Thanks a lot for your reply. But I am confused again if the following query is complex.
http://localhost:8983/solr/test/select/?q=content:*&fl=content&hl=true&hl.fl=content&hl.maxAnalyzedChars=31375&start=64&rows=1&sort=obs_date%20desc

I hardly know anything about highlighting, so nothing that I say here
may have any relevance to your situtation at all.

A query of content:* strikes me as an invalid query. If you are
shooting for all documents where content exists and excluding those
where it doesn't exist, I would think that 'q=content:[* TO *]' (the TO
must be uppercase) would be a better option.

Exactly how your query gets expanded into a something that exceeds
maxBooleanClauses is a complete mystery to me, and probably does have
something to do with the highlighting.

Thanks,
Shawn


________________________________
If you reply to this email, your message will be added to the discussion below:
http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4057074.html
To unsubscribe from TooManyClauses: maxClauseCount is set to 1024, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4056965&code=c2F3YW4udmVybWFAZ2xhc3NiZWFtLmNvbXw0MDU2OTY1fC0xMTI5MDQ2NDY1>.
NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4057181.html
Sent from the Solr - User mailing list archive at Nabble.com.

[Shawn Heisey]

Post by sawanverma
Giving content:[* TO *] gives the same error but when I give content:[a TO z] it works fine. Can you please explain what does it mean when I give content:[a TO z]? Can I use this as workaround? The datatype of content field is text_en.

That syntax is a range query. The [* TO *] basically means that you are
requesting all documents where the content field exists (has a value).
It's not very likely that [a TO z] will include all possible documents -
it would not include a value like "zap" for instance, because
alphabetically, that is after z.

I am a little bit confused - why would you want to do highlighting on a
query that matches all documents that contain the content field, or even
all documents? The point of highlighting is to show the parts of the
text that matched your query text, but you don't have any query text.

I think it may be time to back up and tell us what you want to actually
accomplish, rather than trying to deal directly with the error message.
Because it has to do with highlighting, I may not be able to help, but
there are plenty of very smart people here who do understand highlighting.

Thanks,
Shawn

[liubiaoxin1]

set exery core solrconfig.xml: 4096



--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4290157.html
Sent from the Solr - User mailing list archive at Nabble.com.

[liubiaoxin1]

<Loading Image...>



--
View this message in context: http://lucene.472066.n3.nabble.com/TooManyClauses-maxClauseCount-is-set-to-1024-tp4056965p4290158.html
Sent from the Solr - User mailing list archive at Nabble.com.

[Shawn Heisey]

set exery core solrconfig.xml: 4096

Are you absolutely certain that you have set maxBooleanClauses on
*EVERY* core in that Solr instance?

This value is global, across the entire JVM, and the last core that
starts will set the value for all cores.

It is not possible to explicitly control the exact starting order of
your cores. If the config option is missing from the last core that
gets started, then the global setting will be reset back to 1024.

I tried to address this once, but the change was vetoed, and by Apache's
rules, I wasn't allowed to do it.

https://issues.apache.org/jira/browse/SOLR-4586

I hope to try again and make this situation better for Solr.

Thanks,
Shawn

[Erick Erickson]

Another alternative is to use the TermsQueryParser which is intended to
deal with very large lists of values that should be ORed together. It may
be useful if your query pattern matches its intent. see:
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermsQueryParser

Best,
Erick

Post by Shawn Heisey

set exery core solrconfig.xml: 4096

Are you absolutely certain that you have set maxBooleanClauses on
*EVERY* core in that Solr instance?
This value is global, across the entire JVM, and the last core that
starts will set the value for all cores.
It is not possible to explicitly control the exact starting order of
your cores. If the config option is missing from the last core that
gets started, then the global setting will be reset back to 1024.
I tried to address this once, but the change was vetoed, and by Apache's
rules, I wasn't allowed to do it.
https://issues.apache.org/jira/browse/SOLR-4586
I hope to try again and make this situation better for Solr.
Thanks,
Shawn