How to create a core by API?

Discussion:

Mark E. Haase

2015-03-26 16:24:02 UTC

I can't get the Core Admin API to work. I have a brand new installation of
Solr 5.0.0 (in non-cloud mode). I installed using the installation script
(a nice addition!) with default options, so I have Solr in /opt/solr and
its data in /var/solr.

Here's what I'm trying:

curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core
'

But I get this error:

Error CREATEing SolrCore 'new_core': Unable to create core [new_core]
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'/var/solr/data/new_core/conf'

Solr isn't even creating /var/solr/data/new_core, which I guess is the root
of the problem. But /var/solr is owned by the solr user and I can do `sudo
-u solr mkdir /var/solr/data/new_core` just fine. So why isn't Solr making
this directory?

I see that 'instanceDir' is required, but I don't get an error message if I
*don't* use it, so I'm not sure how required it actually is. I'm also not
sure if its supposed to be a full path or a relative path or what, so here
are a couple of other guesses at the correct incantation:

curl '
http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=new_core
'
curl '
http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=/var/solr/data/new_core
'

These both return the same error message as my first try, so no dice...

FWIW, I get the same error message even if I try doing this with the Solr
Admin GUI so I'm really puzzled. Is the GUI supposed to work?

I found a thread on Stack Overflow about this same problem (
http://stackoverflow.com/a/28945428/122763) that suggests using configSet.
Okay, the installer put some configs sets in
/opt/solr/server /opt/solr/server/solr/configsets, and the 'basic_config'
config set has a solrconfig.xml in it, so maybe that would solve my
solrconfig.xml error?

If I compare the HTTP API to the `solr create -c foo` script, it appears
that the script creates the instance directory and copies in conf files *before
*it calls the HTTP API... surely the HTTP API doesn't require the caller to
create a directory and copy files first, does it?

--
Mark E. Haase

Shawn Heisey

2015-03-26 16:52:27 UTC

Permalink

The error message tells you what is wrong.

The CoreAdmin API requires that the instanceDir already exist, with a
conf directory inside it that contains solrconfig.xml, schema.xml, and
any other necessary config files.

If you want completely from-scratch creation without any existing
filesystem layout, you will need to run SolrCloud, which keeps config
files in the zookeeper database. At that point you would be using the
Collections API.

If you go to Core Admin in the admin UI and click the "Add Core" button,
you will see the following note:

|instanceDir| and |dataDir| need to exist before you can create the core

This message is not quite accurate -- the dataDir (defaulting to
${instanceDir}/data) will be created if it does not already exist, and
the user running Solr has the required permissions to create it. The
message also doesn't say anything about the conf directory or the two
required XML files.

Thanks,
Shawn

Erick Erickson

2015-03-26 16:56:26 UTC

Permalink

Ok, you're being confused by cloud, non cloud and all that kinda stuff....

Configsets are SolrCloud only, so forget them since you specified it's
not SolrCloud.

bq: surely the HTTP API doesn't require the caller to create a
directory and copy files first, does it

In fact, yes. The thing to remember here is that you're using a much
older approach that had its roots in the pre-cloud days. The problem
is how do you insure that the configurations are on the node you're
creating the core on? The whole configsets discussion is an attempt to
solve that in SolrCloud by putting the configs in a place any Solr
instance can find them, namely Zookeeper.

But in non-cloud, there's no central repository. You could be firing
the query from node X and creating the core on node Y. So Solr expects
the config files to already be in place; you have to manually copy
them to node Y anyway, why not copy them to the place they'll be
needed?

The scripts make an assumption that you're running on the same node
you're running the scripts for quick-start purposes.

Best,
Erick

Post by Mark E. Haase
I can't get the Core Admin API to work. I have a brand new installation of
Solr 5.0.0 (in non-cloud mode). I installed using the installation script
(a nice addition!) with default options, so I have Solr in /opt/solr and
its data in /var/solr.
curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core
'
Error CREATEing SolrCore 'new_core': Unable to create core [new_core]
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'/var/solr/data/new_core/conf'
Solr isn't even creating /var/solr/data/new_core, which I guess is the root
of the problem. But /var/solr is owned by the solr user and I can do `sudo
-u solr mkdir /var/solr/data/new_core` just fine. So why isn't Solr making
this directory?
I see that 'instanceDir' is required, but I don't get an error message if I
*don't* use it, so I'm not sure how required it actually is. I'm also not
sure if its supposed to be a full path or a relative path or what, so here
curl '
http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=new_core
'
curl '
http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=/var/solr/data/new_core
'
These both return the same error message as my first try, so no dice...
FWIW, I get the same error message even if I try doing this with the Solr
Admin GUI so I'm really puzzled. Is the GUI supposed to work?
I found a thread on Stack Overflow about this same problem (
http://stackoverflow.com/a/28945428/122763) that suggests using configSet.
Okay, the installer put some configs sets in
/opt/solr/server /opt/solr/server/solr/configsets, and the 'basic_config'
config set has a solrconfig.xml in it, so maybe that would solve my
solrconfig.xml error?
If I compare the HTTP API to the `solr create -c foo` script, it appears
that the script creates the instance directory and copies in conf files *before
*it calls the HTTP API... surely the HTTP API doesn't require the caller to
create a directory and copy files first, does it?
--
Mark E. Haase

Mark E. Haase

2015-03-26 17:16:26 UTC

Permalink

Okay, thanks for the feedback. I'll admit that I do find the cloud vs
non-cloud deployment options a constant source of confusion, not the least
of which is due to the name. If I run a single Solr instance on EC2, that's
not "cloud", but if I run a few instances with ZK on my local LAN, that is
"cloud". Mmmkay.

I can't imagine why the API documentation wouldn't mention that the API
can't actually do the thing it's supposed to do (create a core). What's the
purpose of having an HTTP API if I'm expected to already have write access
to the host's file system to use it? Maybe its intended as private API? It
should only be used by Solr itself, e.g. `solr create -c foo` uses the
Cores Admin API to do some (but not all) of its work. But if that's the
case, then the API docs should say that.

From an API consumer's point of view, I'm not really interested in being
forced to learn the history of the project to use the API. The whole point
of creating APIs is to abstract out details that the caller doesn't need to
know, and yet this API requires an understanding of Solr's internal file
structure and history of the project?

Yikes.

Post by Erick Erickson
Ok, you're being confused by cloud, non cloud and all that kinda stuff....
Configsets are SolrCloud only, so forget them since you specified it's
not SolrCloud.
bq: surely the HTTP API doesn't require the caller to create a
directory and copy files first, does it
In fact, yes. The thing to remember here is that you're using a much
older approach that had its roots in the pre-cloud days. The problem
is how do you insure that the configurations are on the node you're
creating the core on? The whole configsets discussion is an attempt to
solve that in SolrCloud by putting the configs in a place any Solr
instance can find them, namely Zookeeper.
But in non-cloud, there's no central repository. You could be firing
the query from node X and creating the core on node Y. So Solr expects
the config files to already be in place; you have to manually copy
them to node Y anyway, why not copy them to the place they'll be
needed?
The scripts make an assumption that you're running on the same node
you're running the scripts for quick-start purposes.
Best,
Erick

Post by Mark E. Haase
I can't get the Core Admin API to work. I have a brand new installation

Post by Mark E. Haase
Solr 5.0.0 (in non-cloud mode). I installed using the installation script
(a nice addition!) with default options, so I have Solr in /opt/solr and
its data in /var/solr.
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core

Post by Mark E. Haase
'
Error CREATEing SolrCore 'new_core': Unable to create core [new_core]
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'/var/solr/data/new_core/conf'
Solr isn't even creating /var/solr/data/new_core, which I guess is the

root

Post by Mark E. Haase
of the problem. But /var/solr is owned by the solr user and I can do

`sudo

Post by Mark E. Haase
-u solr mkdir /var/solr/data/new_core` just fine. So why isn't Solr

making

Post by Mark E. Haase
this directory?
I see that 'instanceDir' is required, but I don't get an error message

if I

Post by Mark E. Haase
*don't* use it, so I'm not sure how required it actually is. I'm also not
sure if its supposed to be a full path or a relative path or what, so

here

Post by Mark E. Haase
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=new_core

Post by Mark E. Haase
'
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=/var/solr/data/new_core

Post by Mark E. Haase
'
These both return the same error message as my first try, so no dice...
FWIW, I get the same error message even if I try doing this with the Solr
Admin GUI so I'm really puzzled. Is the GUI supposed to work?
I found a thread on Stack Overflow about this same problem (
http://stackoverflow.com/a/28945428/122763) that suggests using

configSet.

Post by Mark E. Haase
Okay, the installer put some configs sets in
/opt/solr/server /opt/solr/server/solr/configsets, and the 'basic_config'
config set has a solrconfig.xml in it, so maybe that would solve my
solrconfig.xml error?
If I compare the HTTP API to the `solr create -c foo` script, it appears
that the script creates the instance directory and copies in conf files

*before

Post by Mark E. Haase
*it calls the HTTP API... surely the HTTP API doesn't require the caller

Post by Mark E. Haase
create a directory and copy files first, does it?
--
Mark E. Haase

--
Mark E. Haase
202-815-0201

Mark E. Haase

2015-03-26 17:26:24 UTC

Permalink

Erick, are you sure that configSets don't apply to single-node Solr
instances?

https://cwiki.apache.org/confluence/display/solr/Config+Sets

I don't see anything about Solr cloud there. Also, "configSet" is a
documented argument to the Core Admin API:

https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-CREATE

And one of the few things [I thought] I knew about "cloud" vs "non cloud"
setups was the Collections API is for cloud and Cores API is for non cloud,
right? So why would the non-cloud API take a cloud-only argument?

Post by Mark E. Haase
Okay, thanks for the feedback. I'll admit that I do find the cloud vs
non-cloud deployment options a constant source of confusion, not the least
of which is due to the name. If I run a single Solr instance on EC2, that's
not "cloud", but if I run a few instances with ZK on my local LAN, that is
"cloud". Mmmkay.
I can't imagine why the API documentation wouldn't mention that the API
can't actually do the thing it's supposed to do (create a core). What's the
purpose of having an HTTP API if I'm expected to already have write access
to the host's file system to use it? Maybe its intended as private API? It
should only be used by Solr itself, e.g. `solr create -c foo` uses the
Cores Admin API to do some (but not all) of its work. But if that's the
case, then the API docs should say that.
From an API consumer's point of view, I'm not really interested in being
forced to learn the history of the project to use the API. The whole point
of creating APIs is to abstract out details that the caller doesn't need to
know, and yet this API requires an understanding of Solr's internal file
structure and history of the project?
Yikes.

Post by Mark E. Haase
I can't get the Core Admin API to work. I have a brand new installation

Post by Mark E. Haase
Solr 5.0.0 (in non-cloud mode). I installed using the installation

script

Post by Mark E. Haase
(a nice addition!) with default options, so I have Solr in /opt/solr and
its data in /var/solr.
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core

Post by Mark E. Haase
'
Error CREATEing SolrCore 'new_core': Unable to create core

[new_core]

Post by Mark E. Haase
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'/var/solr/data/new_core/conf'
Solr isn't even creating /var/solr/data/new_core, which I guess is the

root

Post by Mark E. Haase
of the problem. But /var/solr is owned by the solr user and I can do

`sudo

Post by Mark E. Haase
-u solr mkdir /var/solr/data/new_core` just fine. So why isn't Solr

making

Post by Mark E. Haase
this directory?
I see that 'instanceDir' is required, but I don't get an error message

if I

Post by Mark E. Haase
*don't* use it, so I'm not sure how required it actually is. I'm also

not

Post by Mark E. Haase
sure if its supposed to be a full path or a relative path or what, so

here

Post by Mark E. Haase
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=new_core

Post by Mark E. Haase
'
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=/var/solr/data/new_core

Post by Mark E. Haase
'
These both return the same error message as my first try, so no dice...
FWIW, I get the same error message even if I try doing this with the

Solr

Post by Mark E. Haase
Admin GUI so I'm really puzzled. Is the GUI supposed to work?
I found a thread on Stack Overflow about this same problem (
http://stackoverflow.com/a/28945428/122763) that suggests using

configSet.

Post by Mark E. Haase
Okay, the installer put some configs sets in
/opt/solr/server /opt/solr/server/solr/configsets, and the

'basic_config'

Post by Mark E. Haase
config set has a solrconfig.xml in it, so maybe that would solve my
solrconfig.xml error?
If I compare the HTTP API to the `solr create -c foo` script, it appears
that the script creates the instance directory and copies in conf files

*before

Post by Mark E. Haase
*it calls the HTTP API... surely the HTTP API doesn't require the

caller to

Post by Mark E. Haase
create a directory and copy files first, does it?
--
Mark E. Haase

--
Mark E. Haase
202-815-0201

Erick Erickson

2015-03-26 17:31:02 UTC

Permalink

Hmmm, looks like I stand corrected. I haven't kept complete track
there, looks like this one didn't stick in my head.

As far as the docs are concerned, all patches welcome!

Best,
Erick

Post by Mark E. Haase
Erick, are you sure that configSets don't apply to single-node Solr
instances?
https://cwiki.apache.org/confluence/display/solr/Config+Sets
I don't see anything about Solr cloud there. Also, "configSet" is a
https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-CREATE
And one of the few things [I thought] I knew about "cloud" vs "non cloud"
setups was the Collections API is for cloud and Cores API is for non cloud,
right? So why would the non-cloud API take a cloud-only argument?

Post by Mark E. Haase
I can't get the Core Admin API to work. I have a brand new installation

Post by Mark E. Haase
Solr 5.0.0 (in non-cloud mode). I installed using the installation

script

Post by Mark E. Haase
(a nice addition!) with default options, so I have Solr in /opt/solr and
its data in /var/solr.
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core

Post by Mark E. Haase
'
Error CREATEing SolrCore 'new_core': Unable to create core

[new_core]

Post by Mark E. Haase
Caused by: Can't find resource 'solrconfig.xml' in classpath or
'/var/solr/data/new_core/conf'
Solr isn't even creating /var/solr/data/new_core, which I guess is the

root

Post by Mark E. Haase
of the problem. But /var/solr is owned by the solr user and I can do

`sudo

Post by Mark E. Haase
-u solr mkdir /var/solr/data/new_core` just fine. So why isn't Solr

making

Post by Mark E. Haase
this directory?
I see that 'instanceDir' is required, but I don't get an error message

if I

Post by Mark E. Haase
*don't* use it, so I'm not sure how required it actually is. I'm also

not

Post by Mark E. Haase
sure if its supposed to be a full path or a relative path or what, so

here

Post by Mark E. Haase
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=new_core

Post by Mark E. Haase
'
curl '

http://localhost:8983/solr/admin/cores?action=CREATE&name=new_core&instanceDir=/var/solr/data/new_core

Post by Mark E. Haase
'
These both return the same error message as my first try, so no dice...
FWIW, I get the same error message even if I try doing this with the

Solr

configSet.

Post by Mark E. Haase
Okay, the installer put some configs sets in
/opt/solr/server /opt/solr/server/solr/configsets, and the

'basic_config'

*before

Post by Mark E. Haase
*it calls the HTTP API... surely the HTTP API doesn't require the

caller to

Post by Mark E. Haase
create a directory and copy files first, does it?
--
Mark E. Haase

--
Mark E. Haase
202-815-0201

Mark E. Haase

2015-03-26 17:45:22 UTC

Permalink

Post by Erick Erickson
Hmmm, looks like I stand corrected. I haven't kept complete track
there, looks like this one didn't stick in my head.

I'm not saying you're wrong. The configSet parameter doesn't work at all in
my set up, so you might be right... I'm just wondering where that's
documented.

I thought Solr documentation was rough back in the 1.6 days, but wow...
it's gotten shockingly bad in Solr 5.

Post by Erick Erickson
As far as the docs are concerned, all patches welcome!

What kind of patch do you mean? Isn't all the documentation maintained on
confluence?

--
Mark E. Haase
202-815-0201

Erick Erickson

2015-03-26 17:48:08 UTC

Permalink

Got to the comments section and add any corrections you'd like,
that'll get bubbled up.

Best,
Erick

Post by Mark E. Haase

Post by Erick Erickson
Hmmm, looks like I stand corrected. I haven't kept complete track
there, looks like this one didn't stick in my head.

I'm not saying you're wrong. The configSet parameter doesn't work at all in
my set up, so you might be right... I'm just wondering where that's
documented.
I thought Solr documentation was rough back in the 1.6 days, but wow...
it's gotten shockingly bad in Solr 5.

Post by Erick Erickson
As far as the docs are concerned, all patches welcome!

What kind of patch do you mean? Isn't all the documentation maintained on
confluence?
--
Mark E. Haase
202-815-0201

Yonik Seeley

2015-03-26 18:48:04 UTC

Permalink

Post by Mark E. Haase
I'm not saying you're wrong. The configSet parameter doesn't work at all in
my set up, so you might be right... I'm just wondering where that's
documented.

Trying on current trunk, I got it to work:

/opt/code/lusolr_trunk/solr$ curl -XPOST
"http://localhost:8983/solr/admin/cores?action=CREATE&name=demo3&instanceDir=demo3&configSet=basic_configs"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int
name="QTime">769</int></lst><str name="core">demo3</str>
</response>

Although I'm not thrilled with a different parameter name for cloud
vs non-cloud. I come from the camp that believes that overloading is
both natural and easily understood (e.g. I don't find "foo" + "bar"
and 1.5 + 2.5 both using "+" confusing).

-Yonik

Continue reading on narkive:

Search results for 'How to create a core by API?' (Questions and Answers)

replies

Creating Java/flash applications?

started 2009-07-05 07:45:56 UTC

programming & design

replies

What programming language do people use to create games?

started 2010-10-26 17:49:23 UTC

programming & design

replies

How do I make my SQL Server database available online?