Discussion:
Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)
Katherine Mora
2016-02-15 18:23:21 UTC
Permalink
Hello All,

I've configured Solr 5.2.1 to enable SSL by following the instructions listed in the Wiki in Enabling SSL<https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This is working fine. However, if I go to the Solr Admin (Dashboard -> JVM -> Args) or if I list the processes running in the computer, I can see the password that I set in the solr.in.sh script for SOLR_SSL_KEY_STORE_PASSWORD and SOLR_SSL_TRUST_STORE_PASSWORD:

-Dsolr.jetty.truststore.password=XYZ
-Dsolr.jetty.keystore.password=XYZ
-Djavax.net.ssl.trustStorePassword=XYZ
-Djavax.net.ssl.keyStorePassword=XYZ


I have tried securing the passwords using Jetty's Password utility:

java -cp jetty-util-9.2.10.v20150310.jar org.eclipse.jetty.util.security.Password XYZ

And using the "OBF:XYZ" password in solr.in.sh instead but I get an exception java.security.NoSuchAlgorithmException -> java.io.IOException: Keystore was tampered with, or password was incorrect (I'm listing the complete exception below as well)

Additionally, I have tried to remove the lines in the "bin/solr" script that set the passwords in SOLR_SSL_OPTS and eventually in SOLR_OPTS instead, setting the passwords directly in the jetty configuration files located under "server/etc". However, when I do this, I get an exception saying the password cannot be null. It seems like there is a setting that is not listed in the jetty files. I found that "keyManagerPassword" is not listed in the jetty-ssl.xml file and I added it, but I keep getting the same error.

Does anyone know how to prevent the SSL keystore and trust store password from showing up in the Solr Admin by doing the configuration in the jetty files or by securing the passwords?

Thanks in advance for any help you can provide.


Caused by: java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:198)
at javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:205)
at org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
at org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
... 12 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1259)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:214)
at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35)
at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142)
at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118)
at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
at org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:140)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:110)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:461)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:642)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:527)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
... 59 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 68 more


Katherine Mora
Jeff Wu
2016-02-29 12:58:26 UTC
Permalink
Hi Katherine, we had exact the same issue, we need to protect our password.
Anyone who can access to solr server can do "ps -elf|grep java" to grep the
solr commandline, and it has all the password in plain text.

The /bin/solr shell will set 10 related system property:
SOLR_SSL_OPTS=" -Dsolr.jetty.keystore=$SOLR_SSL_KEY_STORE \
-Dsolr.jetty.keystore.password=$SOLR_SSL_KEY_STORE_PASSWORD \
-Dsolr.jetty.truststore=$SOLR_SSL_TRUST_STORE \
-Dsolr.jetty.truststore.password=$SOLR_SSL_TRUST_STORE_PASSWORD \
-Dsolr.jetty.ssl.needClientAuth=$SOLR_SSL_NEED_CLIENT_AUTH \
-Dsolr.jetty.ssl.wantClientAuth=$SOLR_SSL_WANT_CLIENT_AUTH"
SOLR_SSL_OPTS+=" -Djavax.net.ssl.keyStore=$SOLR_SSL_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$SOLR_SSL_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$SOLR_SSL_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$SOLR_SSL_TRUST_STORE_PASSWORD"
and also
SOLR_JETTY_CONFIG+=("--module=https")

The questions we have:
1. We doubt "OBF:XYZ" does not work when you set to solr.in.sh. the
javax.net.ssl can't work with jetty OBF. What we saw is Incorrect password

Caused by: java.io.IOException: Keystore was tampered with, or password was
incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:852)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:732)
at sun.reflect.GeneratedConstructorAccessor280.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)

2. Is there any good sample we can referent to configure in jetty-https.xml
and jetty-ssl.xml to leverage Jetty OBF?
Katherine, can you share your jetty-ssl.xml and jetty-https.xml?
From this link:
http://www.eclipse.org/jetty/documentation/9.2.6.v20141205/configuring-ssl.html,
did you put keystore files under jetty home?
and followed this sample?
<New id="sslContextFactory" class=
"org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="KeyStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
<Set name="KeyManagerPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
<Set name="TrustStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="TrustStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
</New>
Post by Katherine Mora
Hello All,
I've configured Solr 5.2.1 to enable SSL by following the instructions
listed in the Wiki in Enabling SSL<
https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This is
working fine. However, if I go to the Solr Admin (Dashboard -> JVM -> Args)
or if I list the processes running in the computer, I can see the password
that I set in the solr.in.sh script for SOLR_SSL_KEY_STORE_PASSWORD and
-Dsolr.jetty.truststore.password=XYZ
-Dsolr.jetty.keystore.password=XYZ
-Djavax.net.ssl.trustStorePassword=XYZ
-Djavax.net.ssl.keyStorePassword=XYZ
java -cp jetty-util-9.2.10.v20150310.jar
org.eclipse.jetty.util.security.Password XYZ
And using the "OBF:XYZ" password in solr.in.sh instead but I get an
Keystore was tampered with, or password was incorrect (I'm listing the
complete exception below as well)
Additionally, I have tried to remove the lines in the "bin/solr" script
that set the passwords in SOLR_SSL_OPTS and eventually in SOLR_OPTS
instead, setting the passwords directly in the jetty configuration files
located under "server/etc". However, when I do this, I get an exception
saying the password cannot be null. It seems like there is a setting that
is not listed in the jetty files. I found that "keyManagerPassword" is not
listed in the jetty-ssl.xml file and I added it, but I keep getting the
same error.
Does anyone know how to prevent the SSL keystore and trust store password
from showing up in the Solr Admin by doing the configuration in the jetty
files or by securing the passwords?
Thanks in advance for any help you can provide.
java.security.NoSuchAlgorithmException: Error constructing implementation
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at
javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:198)
at
javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:205)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
... 12 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1259)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at
javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at
org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
at
org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
at
org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
at
org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:214)
at
org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118)
at
org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
at
org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
at
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:140)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:110)
at
org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at
org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at
org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
at
org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at
org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:461)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
at
org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
at
org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:642)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:527)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
... 59 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 68 more
Katherine Mora
Katherine Mora
2016-03-03 14:02:04 UTC
Permalink
Hi Jeff,

Are you still having the same issue or did you manage to fix it?

I have the default files that come with the Solr 5.2.1 installation (I’m copying those below).
I’m pretty sure my password is correct, unless the tool is generating one that does not match my version? I’m using the jetty jar that comes with the Solr installation to generate it, so I’m assuming this is not the case.


Jetty-ssl.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">

<!-- ============================================================= -->
<!-- Configure a TLS (SSL) Context Factory -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and either jetty-https.xml or jetty-spdy.xml (but not both) -->
<!-- ============================================================= -->
<Configure id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="solr.jetty.keystore" default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="KeyStorePassword"><Property name="solr.jetty.keystore.password" default="secret"/></Set>
<Set name="TrustStorePath"><Property name="solr.jetty.truststore" default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="TrustStorePassword"><Property name="solr.jetty.truststore.password" default="secret"/></Set>
<Set name="NeedClientAuth"><Property name="solr.jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property name="solr.jetty.ssl.wantClientAuth" default="false"/></Set>
<Set name="excludeProtocols">
<Array type="java.lang.String">
<Item>SSLv3</Item>
</Array>
</Set>
<Set name="ExcludeCipherSuites">
<Array type="String">
<Item>SSL_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_DSS_WITH_DES_CBC_SHA</Item>
<Item>SSL_RSA_EXPORT_WITH_RC4_40_MD5</Item>
<Item>SSL_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA</Item>
</Array>
</Set>

<!-- =========================================================== -->
<!-- Create a TLS specific HttpConfiguration based on the -->
<!-- common HttpConfiguration defined in jetty.xml -->
<!-- Add a SecureRequestCustomizer to extract certificate and -->
<!-- session information -->
<!-- =========================================================== -->
<New id="sslHttpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
<Arg><Ref refid="httpConfig"/></Arg>
<Call name="addCustomizer">
<Arg><New class="org.eclipse.jetty.server.SecureRequestCustomizer"/></Arg>
</Call>
</New>

</Configure>

jetty-https.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">

<!-- ============================================================= -->
<!-- Configure a HTTPS connector. -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and jetty-ssl.xml. -->
<!-- ============================================================= -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">

<!-- =========================================================== -->
<!-- Add a HTTPS Connector. -->
<!-- Configure an o.e.j.server.ServerConnector with connection -->
<!-- factories for TLS (aka SSL) and HTTP to provide HTTPS. -->
<!-- All accepted TLS connections are wired to a HTTP connection.-->
<!-- -->
<!-- Consult the javadoc of o.e.j.server.ServerConnector, -->
<!-- o.e.j.server.SslConnectionFactory and -->
<!-- o.e.j.server.HttpConnectionFactory for all configuration -->
<!-- that may be set here. -->
<!-- =========================================================== -->
<Call id="httpsConnector" name="addConnector">
<Arg>
<New class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server" /></Arg>
<Arg name="acceptors" type="int"><Property name="solr.jetty.ssl.acceptors" default="-1"/></Arg>
<Arg name="selectors" type="int"><Property name="solr.jetty.ssl.selectors" default="-1"/></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<Item>
<New class="org.eclipse.jetty.server.SslConnectionFactory">
<Arg name="next">http/1.1</Arg>
<Arg name="sslContextFactory"><Ref refid="sslContextFactory"/></Arg>
</New>
</Item>
<Item>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config"><Ref refid="sslHttpConfig"/></Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="solr.jetty.host" /></Set>
<Set name="port"><Property name="solr.jetty.https.port" default="8983" /></Set>
<Set name="idleTimeout"><Property name="solr.jetty.https.timeout" default="50000"/></Set>
<Set name="soLingerTime"><Property name="solr.jetty.https.soLingerTime" default="-1"/></Set>
<Set name="acceptorPriorityDelta"><Property name="solr.jetty.ssl.acceptorPriorityDelta" default="0"/></Set>
<Set name="selectorPriorityDelta"><Property name="solr.jetty.ssl.selectorPriorityDelta" default="0"/></Set>
<Set name="acceptQueueSize"><Property name="solr.jetty.https.acceptQueueSize" default="0"/></Set>
</New>
</Arg>
</Call>
</Configure>




-----Original Message-----
From: Jeff Wu [mailto:***@gmail.com]
Sent: Monday, February 29, 2016 12:58 PM
To: solr-***@lucene.apache.org
Subject: Re: Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)

Hi Katherine, we had exact the same issue, we need to protect our password.
Anyone who can access to solr server can do "ps -elf|grep java" to grep the solr commandline, and it has all the password in plain text.

The /bin/solr shell will set 10 related system property:
SOLR_SSL_OPTS=" -Dsolr.jetty.keystore=$SOLR_SSL_KEY_STORE \
-Dsolr.jetty.keystore.password=$SOLR_SSL_KEY_STORE_PASSWORD \
-Dsolr.jetty.truststore=$SOLR_SSL_TRUST_STORE \
-Dsolr.jetty.truststore.password=$SOLR_SSL_TRUST_STORE_PASSWORD \
-Dsolr.jetty.ssl.needClientAuth=$SOLR_SSL_NEED_CLIENT_AUTH \
-Dsolr.jetty.ssl.wantClientAuth=$SOLR_SSL_WANT_CLIENT_AUTH"
SOLR_SSL_OPTS+=" -Djavax.net.ssl.keyStore=$SOLR_SSL_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$SOLR_SSL_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$SOLR_SSL_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$SOLR_SSL_TRUST_STORE_PASSWORD"
and also
SOLR_JETTY_CONFIG+=("--module=https")

The questions we have:
1. We doubt "OBF:XYZ" does not work when you set to solr.in.sh. the javax.net.ssl can't work with jetty OBF. What we saw is Incorrect password

Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:852)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:732)
at sun.reflect.GeneratedConstructorAccessor280.newInstance(Unknown Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)

2. Is there any good sample we can referent to configure in jetty-https.xml and jetty-ssl.xml to leverage Jetty OBF?
Katherine, can you share your jetty-ssl.xml and jetty-https.xml?
From this link:
http://www.eclipse.org/jetty/documentation/9.2.6.v20141205/configuring-ssl.html,
did you put keystore files under jetty home?
and followed this sample?
<New id="sslContextFactory" class=
"org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="KeyStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
<Set name="KeyManagerPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
<Set name="TrustStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="TrustStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
</New>
Post by Katherine Mora
Hello All,
I've configured Solr 5.2.1 to enable SSL by following the instructions
listed in the Wiki in Enabling SSL<
https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This
is working fine. However, if I go to the Solr Admin (Dashboard -> JVM
-> Args) or if I list the processes running in the computer, I can see
the password that I set in the solr.in.sh script for
SOLR_SSL_KEY_STORE_PASSWORD and
-Dsolr.jetty.truststore.password=XYZ
-Dsolr.jetty.keystore.password=XYZ
-Djavax.net.ssl.trustStorePassword=XYZ
-Djavax.net.ssl.keyStorePassword=XYZ
java -cp jetty-util-9.2.10.v20150310.jar
org.eclipse.jetty.util.security.Password XYZ
And using the "OBF:XYZ" password in solr.in.sh instead but I get an
Keystore was tampered with, or password was incorrect (I'm listing the
complete exception below as well)
Additionally, I have tried to remove the lines in the "bin/solr"
script that set the passwords in SOLR_SSL_OPTS and eventually in
SOLR_OPTS instead, setting the passwords directly in the jetty
configuration files located under "server/etc". However, when I do
this, I get an exception saying the password cannot be null. It seems
like there is a setting that is not listed in the jetty files. I found
that "keyManagerPassword" is not listed in the jetty-ssl.xml file and
I added it, but I keep getting the same error.
Does anyone know how to prevent the SSL keystore and trust store
password from showing up in the Solr Admin by doing the configuration
in the jetty files or by securing the passwords?
Thanks in advance for any help you can provide.
java.security.NoSuchAlgorithmException: Error constructing
implementation
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at
javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:198)
at
javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:205)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
... 12 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1259)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at
javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at
org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
at
org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
at
org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
at
org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:214)
at
org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142)
at
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118)
at
org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
at
org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
at
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:140)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:110)
at
org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
at
org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at
org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
at
org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at
org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:461)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
at
org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
at
org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
at org.eclipse.jetty.server.Server.start(Server.java:387)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112)
Caused by: java.io.IOException: Keystore was tampered with, or
password was incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:642)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:527)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.security.Provider$Service.newInstance(Provider.java:1240)
... 59 more
Caused by: java.security.UnrecoverableKeyException: Password
verification failed
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 68 more
Katherine Mora
Zara Parst
2016-03-03 14:41:39 UTC
Permalink
Hello Katherine,

I am sorry to ask this question. But really i need some light on bellow
matter.


I want to run solr in cloud mode . So obliviously I am going to use
zookeeper.


My quorum are distributed on 3 server with static ip , lets say


server.1=xx.xx.x1:2888:3888
server.2=xx.xx.x2:2889:3889
server.3=xx.xx.x3:2890:3890


With solr pointing to this ensemble. Now my concern is how should I protect
it to other unauthorized zkClient to connect above quorum. One way could be
don't open the port for the client but then how will solr connect ?
other problem is how to safeguard quorum interconnection. I observed a
weird behavior that I can point a fourth zookeeper from my local to the
above quorum (i have to know only ip and port which is not tough to find)
and it will be absorbed as a part of quorum and then I can use my local
zkClient to connect my local zoookeeper and have access to quorum which we
don't want. I want to define quorum in a way that foreign zookeeper server
is not able to
become part already configured quorum.

Again one more strange behavior about znode of zookeeper, User A can set
ACL of a znode and user B which can connect to zookeeper but can't see the
content as it will throw ACL error that is fine but strange thing is user B
can still delete the znode of A which he cant see. :(

I think a hell lot of things is not clear about zoookeeper security.

Please can you help me ? And don't forget my thanks in advance.
Post by Katherine Mora
Hi Jeff,
Are you still having the same issue or did you manage to fix it?
I have the default files that come with the Solr 5.2.1 installation (I’m
copying those below).
I’m pretty sure my password is correct, unless the tool is generating one
that does not match my version? I’m using the jetty jar that comes with the
Solr installation to generate it, so I’m assuming this is not the case.
Jetty-ssl.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "
http://www.eclipse.org/jetty/configure_9_0.dtd">
<!-- ============================================================= -->
<!-- Configure a TLS (SSL) Context Factory -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and either jetty-https.xml or jetty-spdy.xml (but not both) -->
<!-- ============================================================= -->
<Configure id="sslContextFactory"
class="org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="solr.jetty.keystore"
default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="KeyStorePassword"><Property
name="solr.jetty.keystore.password" default="secret"/></Set>
<Set name="TrustStorePath"><Property name="solr.jetty.truststore"
default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="TrustStorePassword"><Property
name="solr.jetty.truststore.password" default="secret"/></Set>
<Set name="NeedClientAuth"><Property
name="solr.jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property
name="solr.jetty.ssl.wantClientAuth" default="false"/></Set>
<Set name="excludeProtocols">
<Array type="java.lang.String">
<Item>SSLv3</Item>
</Array>
</Set>
<Set name="ExcludeCipherSuites">
<Array type="String">
<Item>SSL_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_DSS_WITH_DES_CBC_SHA</Item>
<Item>SSL_RSA_EXPORT_WITH_RC4_40_MD5</Item>
<Item>SSL_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA</Item>
</Array>
</Set>
<!-- =========================================================== -->
<!-- Create a TLS specific HttpConfiguration based on the -->
<!-- common HttpConfiguration defined in jetty.xml -->
<!-- Add a SecureRequestCustomizer to extract certificate and -->
<!-- session information -->
<!-- =========================================================== -->
<New id="sslHttpConfig"
class="org.eclipse.jetty.server.HttpConfiguration">
<Arg><Ref refid="httpConfig"/></Arg>
<Call name="addCustomizer">
<Arg><New
class="org.eclipse.jetty.server.SecureRequestCustomizer"/></Arg>
</Call>
</New>
</Configure>
jetty-https.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "
http://www.eclipse.org/jetty/configure_9_0.dtd">
<!-- ============================================================= -->
<!-- Configure a HTTPS connector. -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and jetty-ssl.xml. -->
<!-- ============================================================= -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<!-- =========================================================== -->
<!-- Add a HTTPS Connector. -->
<!-- Configure an o.e.j.server.ServerConnector with connection -->
<!-- factories for TLS (aka SSL) and HTTP to provide HTTPS. -->
<!-- All accepted TLS connections are wired to a HTTP connection.-->
<!-- -->
<!-- Consult the javadoc of o.e.j.server.ServerConnector, -->
<!-- o.e.j.server.SslConnectionFactory and -->
<!-- o.e.j.server.HttpConnectionFactory for all configuration -->
<!-- that may be set here. -->
<!-- =========================================================== -->
<Call id="httpsConnector" name="addConnector">
<Arg>
<New class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server" /></Arg>
<Arg name="acceptors" type="int"><Property
name="solr.jetty.ssl.acceptors" default="-1"/></Arg>
<Arg name="selectors" type="int"><Property
name="solr.jetty.ssl.selectors" default="-1"/></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<Item>
<New class="org.eclipse.jetty.server.SslConnectionFactory">
<Arg name="next">http/1.1</Arg>
<Arg name="sslContextFactory"><Ref
refid="sslContextFactory"/></Arg>
</New>
</Item>
<Item>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config"><Ref refid="sslHttpConfig"/></Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="solr.jetty.host" /></Set>
<Set name="port"><Property name="solr.jetty.https.port" default="8983" /></Set>
<Set name="idleTimeout"><Property name="solr.jetty.https.timeout"
default="50000"/></Set>
<Set name="soLingerTime"><Property
name="solr.jetty.https.soLingerTime" default="-1"/></Set>
<Set name="acceptorPriorityDelta"><Property
name="solr.jetty.ssl.acceptorPriorityDelta" default="0"/></Set>
<Set name="selectorPriorityDelta"><Property
name="solr.jetty.ssl.selectorPriorityDelta" default="0"/></Set>
<Set name="acceptQueueSize"><Property
name="solr.jetty.https.acceptQueueSize" default="0"/></Set>
</New>
</Arg>
</Call>
</Configure>
-----Original Message-----
Sent: Monday, February 29, 2016 12:58 PM
Subject: Re: Prevent the SSL Keystore and Truststore password from showing
up in the Solr Admin and Linux processes (Solr 5.2.1)
Hi Katherine, we had exact the same issue, we need to protect our password.
Anyone who can access to solr server can do "ps -elf|grep java" to grep
the solr commandline, and it has all the password in plain text.
SOLR_SSL_OPTS=" -Dsolr.jetty.keystore=$SOLR_SSL_KEY_STORE \
-Dsolr.jetty.keystore.password=$SOLR_SSL_KEY_STORE_PASSWORD \
-Dsolr.jetty.truststore=$SOLR_SSL_TRUST_STORE \
-Dsolr.jetty.truststore.password=$SOLR_SSL_TRUST_STORE_PASSWORD \
-Dsolr.jetty.ssl.needClientAuth=$SOLR_SSL_NEED_CLIENT_AUTH \
-Dsolr.jetty.ssl.wantClientAuth=$SOLR_SSL_WANT_CLIENT_AUTH"
SOLR_SSL_OPTS+=" -Djavax.net.ssl.keyStore=$SOLR_SSL_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$SOLR_SSL_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$SOLR_SSL_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$SOLR_SSL_TRUST_STORE_PASSWORD"
and also
SOLR_JETTY_CONFIG+=("--module=https")
1. We doubt "OBF:XYZ" does not work when you set to solr.in.sh. the
javax.net.ssl can't work with jetty OBF. What we saw is Incorrect password
Caused by: java.io.IOException: Keystore was tampered with, or password
was incorrect at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:852)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:732)
at sun.reflect.GeneratedConstructorAccessor280.newInstance(Unknown Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)
2. Is there any good sample we can referent to configure in
jetty-https.xml and jetty-ssl.xml to leverage Jetty OBF?
Katherine, can you share your jetty-ssl.xml and jetty-https.xml?
http://www.eclipse.org/jetty/documentation/9.2.6.v20141205/configuring-ssl.html
,
did you put keystore files under jetty home?
and followed this sample?
<New id="sslContextFactory" class=
"org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="KeyStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
<Set name="KeyManagerPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
<Set name="TrustStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="TrustStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
</New>
Post by Katherine Mora
Hello All,
I've configured Solr 5.2.1 to enable SSL by following the instructions
listed in the Wiki in Enabling SSL<
https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This
is working fine. However, if I go to the Solr Admin (Dashboard -> JVM
-> Args) or if I list the processes running in the computer, I can see
the password that I set in the solr.in.sh script for
SOLR_SSL_KEY_STORE_PASSWORD and
-Dsolr.jetty.truststore.password=XYZ
-Dsolr.jetty.keystore.password=XYZ
-Djavax.net.ssl.trustStorePassword=XYZ
-Djavax.net.ssl.keyStorePassword=XYZ
java -cp jetty-util-9.2.10.v20150310.jar
org.eclipse.jetty.util.security.Password XYZ
And using the "OBF:XYZ" password in solr.in.sh instead but I get an
Keystore was tampered with, or password was incorrect (I'm listing the
complete exception below as well)
Additionally, I have tried to remove the lines in the "bin/solr"
script that set the passwords in SOLR_SSL_OPTS and eventually in
SOLR_OPTS instead, setting the passwords directly in the jetty
configuration files located under "server/etc". However, when I do
this, I get an exception saying the password cannot be null. It seems
like there is a setting that is not listed in the jetty files. I found
that "keyManagerPassword" is not listed in the jetty-ssl.xml file and
I added it, but I keep getting the same error.
Does anyone know how to prevent the SSL keystore and trust store
password from showing up in the Solr Admin by doing the configuration
in the jetty files or by securing the passwords?
Thanks in advance for any help you can provide.
java.security.NoSuchAlgorithmException: Error constructing
implementation
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at
javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.java:198)
Post by Katherine Mora
at
javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.java:205)
Post by Katherine Mora
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:513)
Post by Katherine Mora
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactory.java:383)
Post by Katherine Mora
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:165)
Post by Katherine Mora
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
Post by Katherine Mora
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
Post by Katherine Mora
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
Post by Katherine Mora
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
Post by Katherine Mora
... 12 more
Caused by: java.security.NoSuchAlgorithmException: Error constructing
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1259)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at
javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at
org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
Post by Katherine Mora
at
org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
Post by Katherine Mora
at
org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
Post by Katherine Mora
at
org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:214)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:35)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:142)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:118)
Post by Katherine Mora
at
org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:166)
Post by Katherine Mora
at
org.apache.solr.handler.component.ShardHandlerFactory.newInstance(ShardHandlerFactory.java:49)
Post by Katherine Mora
at
org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
Post by Katherine Mora
at
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:140)
Post by Katherine Mora
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:110)
Post by Katherine Mora
at
org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:852)
Post by Katherine Mora
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
Post by Katherine Mora
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
Post by Katherine Mora
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
Post by Katherine Mora
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
Post by Katherine Mora
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at
org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:461)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
Post by Katherine Mora
at
org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
Post by Katherine Mora
at
org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:560)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:235)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
Post by Katherine Mora
at org.eclipse.jetty.server.Server.start(Server.java:387)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
Post by Katherine Mora
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
Post by Katherine Mora
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
Post by Katherine Mora
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Post by Katherine Mora
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager(SSLContextImpl.java:642)
Post by Katherine Mora
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:527)
Post by Katherine Mora
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
Post by Katherine Mora
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Post by Katherine Mora
at
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
Post by Katherine Mora
at java.security.Provider$Service.newInstance(Provider.java:1240)
... 59 more
Caused by: java.security.UnrecoverableKeyException: Password
verification failed
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 68 more
Katherine Mora
Katherine Mora
2016-03-03 15:30:52 UTC
Permalink
Hi Zara,

I think that is done when you generate the self-signed certificate and the key. If you check the documentation: https://cwiki.apache.org/confluence/display/solr/Enabling+SSL#EnablingSSL-Generateaself-signedcertificateandakey it says:

The "-ext SAN=..." keytool option allows you to specify all the DNS names and/or IP addresses that will be allowed during hostname verification



-----Original Message-----
From: Zara Parst [mailto:***@gmail.com]
Sent: Thursday, March 03, 2016 2:42 PM
To: solr-***@lucene.apache.org
Subject: Re: Prevent the SSL Keystore and Truststore password from showing up in the Solr Admin and Linux processes (Solr 5.2.1)

Hello Katherine,

I am sorry to ask this question. But really i need some light on bellow matter.


I want to run solr in cloud mode . So obliviously I am going to use zookeeper.


My quorum are distributed on 3 server with static ip , lets say


server.1=xx.xx.x1:2888:3888
server.2=xx.xx.x2:2889:3889
server.3=xx.xx.x3:2890:3890


With solr pointing to this ensemble. Now my concern is how should I protect it to other unauthorized zkClient to connect above quorum. One way could be don't open the port for the client but then how will solr connect ?
other problem is how to safeguard quorum interconnection. I observed a weird behavior that I can point a fourth zookeeper from my local to the above quorum (i have to know only ip and port which is not tough to find) and it will be absorbed as a part of quorum and then I can use my local zkClient to connect my local zoookeeper and have access to quorum which we don't want. I want to define quorum in a way that foreign zookeeper server is not able to become part already configured quorum.

Again one more strange behavior about znode of zookeeper, User A can set ACL of a znode and user B which can connect to zookeeper but can't see the content as it will throw ACL error that is fine but strange thing is user B can still delete the znode of A which he cant see. :(

I think a hell lot of things is not clear about zoookeeper security.

Please can you help me ? And don't forget my thanks in advance.
Post by Katherine Mora
Hi Jeff,
Are you still having the same issue or did you manage to fix it?
I have the default files that come with the Solr 5.2.1 installation
(I’m copying those below).
I’m pretty sure my password is correct, unless the tool is generating
one that does not match my version? I’m using the jetty jar that comes
with the Solr installation to generate it, so I’m assuming this is not the case.
Jetty-ssl.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "
http://www.eclipse.org/jetty/configure_9_0.dtd">
<!-- ============================================================= -->
<!-- Configure a TLS (SSL) Context Factory -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and either jetty-https.xml or jetty-spdy.xml (but not both) -->
<!-- ============================================================= -->
<Configure id="sslContextFactory"
class="org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="solr.jetty.keystore"
default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="KeyStorePassword"><Property
name="solr.jetty.keystore.password" default="secret"/></Set>
<Set name="TrustStorePath"><Property name="solr.jetty.truststore"
default="./etc/solr-ssl.keystore.jks"/></Set>
<Set name="TrustStorePassword"><Property
name="solr.jetty.truststore.password" default="secret"/></Set>
<Set name="NeedClientAuth"><Property
name="solr.jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property
name="solr.jetty.ssl.wantClientAuth" default="false"/></Set>
<Set name="excludeProtocols">
<Array type="java.lang.String">
<Item>SSLv3</Item>
</Array>
</Set>
<Set name="ExcludeCipherSuites">
<Array type="String">
<Item>SSL_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_DSS_WITH_DES_CBC_SHA</Item>
<Item>SSL_RSA_EXPORT_WITH_RC4_40_MD5</Item>
<Item>SSL_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA</Item>
</Array>
</Set>
<!-- =========================================================== -->
<!-- Create a TLS specific HttpConfiguration based on the -->
<!-- common HttpConfiguration defined in jetty.xml -->
<!-- Add a SecureRequestCustomizer to extract certificate and -->
<!-- session information -->
<!-- =========================================================== -->
<New id="sslHttpConfig"
class="org.eclipse.jetty.server.HttpConfiguration">
<Arg><Ref refid="httpConfig"/></Arg>
<Call name="addCustomizer">
<Arg><New
class="org.eclipse.jetty.server.SecureRequestCustomizer"/></Arg>
</Call>
</New>
</Configure>
jetty-https.xml
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "
http://www.eclipse.org/jetty/configure_9_0.dtd">
<!-- ============================================================= -->
<!-- Configure a HTTPS connector. -->
<!-- This configuration must be used in conjunction with jetty.xml -->
<!-- and jetty-ssl.xml. -->
<!-- ============================================================= -->
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<!-- =========================================================== -->
<!-- Add a HTTPS Connector. -->
<!-- Configure an o.e.j.server.ServerConnector with connection -->
<!-- factories for TLS (aka SSL) and HTTP to provide HTTPS. -->
<!-- All accepted TLS connections are wired to a HTTP connection.-->
<!-- -->
<!-- Consult the javadoc of o.e.j.server.ServerConnector, -->
<!-- o.e.j.server.SslConnectionFactory and -->
<!-- o.e.j.server.HttpConnectionFactory for all configuration -->
<!-- that may be set here. -->
<!-- =========================================================== -->
<Call id="httpsConnector" name="addConnector">
<Arg>
<New class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server" /></Arg>
<Arg name="acceptors" type="int"><Property
name="solr.jetty.ssl.acceptors" default="-1"/></Arg>
<Arg name="selectors" type="int"><Property
name="solr.jetty.ssl.selectors" default="-1"/></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<Item>
<New class="org.eclipse.jetty.server.SslConnectionFactory">
<Arg name="next">http/1.1</Arg>
<Arg name="sslContextFactory"><Ref
refid="sslContextFactory"/></Arg>
</New>
</Item>
<Item>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config"><Ref refid="sslHttpConfig"/></Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="solr.jetty.host" /></Set>
<Set name="port"><Property name="solr.jetty.https.port" default="8983" /></Set>
<Set name="idleTimeout"><Property name="solr.jetty.https.timeout"
default="50000"/></Set>
<Set name="soLingerTime"><Property
name="solr.jetty.https.soLingerTime" default="-1"/></Set>
<Set name="acceptorPriorityDelta"><Property
name="solr.jetty.ssl.acceptorPriorityDelta" default="0"/></Set>
<Set name="selectorPriorityDelta"><Property
name="solr.jetty.ssl.selectorPriorityDelta" default="0"/></Set>
<Set name="acceptQueueSize"><Property
name="solr.jetty.https.acceptQueueSize" default="0"/></Set>
</New>
</Arg>
</Call>
</Configure>
-----Original Message-----
Sent: Monday, February 29, 2016 12:58 PM
Subject: Re: Prevent the SSL Keystore and Truststore password from
showing up in the Solr Admin and Linux processes (Solr 5.2.1)
Hi Katherine, we had exact the same issue, we need to protect our password.
Anyone who can access to solr server can do "ps -elf|grep java" to
grep the solr commandline, and it has all the password in plain text.
SOLR_SSL_OPTS=" -Dsolr.jetty.keystore=$SOLR_SSL_KEY_STORE \
-Dsolr.jetty.keystore.password=$SOLR_SSL_KEY_STORE_PASSWORD \
-Dsolr.jetty.truststore=$SOLR_SSL_TRUST_STORE \
-Dsolr.jetty.truststore.password=$SOLR_SSL_TRUST_STORE_PASSWORD \
-Dsolr.jetty.ssl.needClientAuth=$SOLR_SSL_NEED_CLIENT_AUTH \
-Dsolr.jetty.ssl.wantClientAuth=$SOLR_SSL_WANT_CLIENT_AUTH"
SOLR_SSL_OPTS+=" -Djavax.net.ssl.keyStore=$SOLR_SSL_KEY_STORE \
-Djavax.net.ssl.keyStorePassword=$SOLR_SSL_KEY_STORE_PASSWORD \
-Djavax.net.ssl.trustStore=$SOLR_SSL_TRUST_STORE \
-Djavax.net.ssl.trustStorePassword=$SOLR_SSL_TRUST_STORE_PASSWORD"
and also
SOLR_JETTY_CONFIG+=("--module=https")
1. We doubt "OBF:XYZ" does not work when you set to solr.in.sh. the
javax.net.ssl can't work with jetty OBF. What we saw is Incorrect password
Caused by: java.io.IOException: Keystore was tampered with, or
password was incorrect at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56
)
at
sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.j
ava:225)
at
sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeySto
re.java:70) at java.security.KeyStore.load(KeyStore.java:1445)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager
(SSLContextImpl.java:852)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImp
l.java:732) at
sun.reflect.GeneratedConstructorAccessor280.newInstance(Unknown
Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
nstructorAccessorImpl.java:45) at
java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.security.Provider$Service.newInstance(Provider.java:1595)
2. Is there any good sample we can referent to configure in
jetty-https.xml and jetty-ssl.xml to leverage Jetty OBF?
Katherine, can you share your jetty-ssl.xml and jetty-https.xml?
http://www.eclipse.org/jetty/documentation/9.2.6.v20141205/configuring
-ssl.html
,
did you put keystore files under jetty home?
and followed this sample?
<New id="sslContextFactory" class=
"org.eclipse.jetty.util.ssl.SslContextFactory">
<Set name="KeyStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set name="KeyStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
<Set name="KeyManagerPassword">OBF:1u2u1wml1z7s1z7a1wnl1u2g</Set>
<Set name="TrustStorePath"><Property name="jetty.home" default="."
/>/etc/keystore</Set>
<Set
name="TrustStorePassword">OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4</Set>
</New>
2016-02-15 13:23 GMT-05:00 Katherine Mora
Post by Katherine Mora
Hello All,
I've configured Solr 5.2.1 to enable SSL by following the
instructions listed in the Wiki in Enabling SSL<
https://cwiki.apache.org/confluence/display/solr/Enabling+SSL>. This
is working fine. However, if I go to the Solr Admin (Dashboard -> JVM
-> Args) or if I list the processes running in the computer, I can
-> see
the password that I set in the solr.in.sh script for
SOLR_SSL_KEY_STORE_PASSWORD and
-Dsolr.jetty.truststore.password=XYZ
-Dsolr.jetty.keystore.password=XYZ
-Djavax.net.ssl.trustStorePassword=XYZ
-Djavax.net.ssl.keyStorePassword=XYZ
java -cp jetty-util-9.2.10.v20150310.jar
org.eclipse.jetty.util.security.Password XYZ
And using the "OBF:XYZ" password in solr.in.sh instead but I get an
Keystore was tampered with, or password was incorrect (I'm listing
the complete exception below as well)
Additionally, I have tried to remove the lines in the "bin/solr"
script that set the passwords in SOLR_SSL_OPTS and eventually in
SOLR_OPTS instead, setting the passwords directly in the jetty
configuration files located under "server/etc". However, when I do
this, I get an exception saying the password cannot be null. It
seems like there is a setting that is not listed in the jetty files.
I found that "keyManagerPassword" is not listed in the jetty-ssl.xml
file and I added it, but I keep getting the same error.
Does anyone know how to prevent the SSL keystore and trust store
password from showing up in the Solr Admin by doing the
configuration in the jetty files or by securing the passwords?
Thanks in advance for any help you can provide.
java.security.NoSuchAlgorithmException: Error constructing
implementation
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at
javax.net.ssl.DefaultSSLSocketFactory.throwException(SSLSocketFactory.
java:198)
Post by Katherine Mora
at
javax.net.ssl.DefaultSSLSocketFactory.createSocket(SSLSocketFactory.ja
va:205)
Post by Katherine Mora
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactor
y.java:513)
Post by Katherine Mora
at
org.apache.http.conn.ssl.SSLSocketFactory.createSocket(SSLSocketFactor
y.java:383)
Post by Katherine Mora
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnecti
on(DefaultClientConnectionOperator.java:165)
Post by Katherine Mora
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClie
ntConnectionImpl.java:304)
Post by Katherine Mora
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultR
equestDirector.java:611)
Post by Katherine Mora
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequ
estDirector.java:446)
Post by Katherine Mora
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpC
lient.java:882)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpC
lient.java:82)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpC
lient.java:107)
Post by Katherine Mora
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpC
lient.java:55)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSol
rClient.java:466)
Post by Katherine Mora
... 12 more
Caused by: java.security.NoSuchAlgorithmException: Error
sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1259)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:97)
at
javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:121)
at
org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSo
cketFactory.java:190)
Post by Katherine Mora
at
org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(Sc
hemeRegistryFactory.java:85)
Post by Katherine Mora
at
org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnec
tionManager(SystemDefaultHttpClient.java:121)
Post by Katherine Mora
at
org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(Ab
stractHttpClient.java:484)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(Htt
pClientUtil.java:214)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpC
lientConfigurer.java:35)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpC
lientUtil.java:142)
Post by Katherine Mora
at
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClie
ntUtil.java:118)
Post by Katherine Mora
at
org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpSha
rdHandlerFactory.java:166)
Post by Katherine Mora
at
org.apache.solr.handler.component.ShardHandlerFactory.newInstance(Shar
dHandlerFactory.java:49)
Post by Katherine Mora
at
org.apache.solr.core.CoreContainer.load(CoreContainer.java:328)
Post by Katherine Mora
at
org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDis
patchFilter.java:140)
Post by Katherine Mora
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.jav
a:110)
Post by Katherine Mora
at
org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:138)
at
org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.jav
a:852)
Post by Katherine Mora
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletCo
ntextHandler.java:298)
Post by Katherine Mora
at
1349)
Post by Katherine Mora
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java
:1342)
Post by Katherine Mora
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler
.java:741)
Post by Katherine Mora
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
ycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(Stand
ardStarter.java:41)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at
org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentMa
nager.java:498)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.ja
va:146)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(Scann
ingAppProvider.java:180)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProv
ider.java:461)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(Sca
nningAppProvider.java:64)
Post by Katherine Mora
at
org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:609)
Post by Katherine Mora
at
org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
ycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(Scannin
gAppProvider.java:150)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
ycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(Deployment
Manager.java:560)
Post by Katherine Mora
at
org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.j
ava:235)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
ycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLif
eCycle.java:132)
Post by Katherine Mora
at org.eclipse.jetty.server.Server.start(Server.java:387)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerL
ifeCycle.java:114)
Post by Katherine Mora
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandl
er.java:61)
Post by Katherine Mora
at org.eclipse.jetty.server.Server.doStart(Server.java:354)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeC
ycle.java:68)
Post by Katherine Mora
at
org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1255)
at java.security.AccessController.doPrivileged(Native Method)
at
org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
ava:57)
Post by Katherine Mora
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
orImpl.java:43)
Post by Katherine Mora
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:321)
at org.eclipse.jetty.start.Main.start(Main.java:817)
at org.eclipse.jetty.start.Main.main(Main.java:112)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultKeyManager
(SSLContextImpl.java:642)
Post by Katherine Mora
at
sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImp
l.java:527)
Post by Katherine Mora
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructo
rAccessorImpl.java:57)
Post by Katherine Mora
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
nstructorAccessorImpl.java:45)
Post by Katherine Mora
at
java.lang.reflect.Constructor.newInstance(Constructor.java:526)
Post by Katherine Mora
at java.security.Provider$Service.newInstance(Provider.java:1240)
... 59 more
Caused by: java.security.UnrecoverableKeyException: Password
verification failed
at
sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 68 more
Katherine Mora
Loading...