DIIOP over SSL and SHA-2 on IBM Domino (XWork) Server 9 don’t mesh

IBM has been working hard recently to shore up the SSL vulnerabilities with Domino and XWork server. They have been applying fixes to both Domino 8.5 and 9.0 requiring multiple interim fixes over the past few months. Most recently IBM has made support for SHA-2 SSL certificates available as well so users with Domino websites can finally upgrade their SHA-1 SSL certificates to SHA-2 for better security. However, the new security updates have also had some unsuspecting side effects with other IBM Domino tasks running on the server. In particular we came across a problem accessing the Domino server via DIIOP over SSL.

Normally it is fairly easy to use a Java application to access Domino via DIIOP with or without SSL. With SSL it is necessary to use the TrustedCert.class file generated by the DIIOP task on Domino located in the /domino/java folder located in the Domino data folder. However, most recently accessing DIIOP over SSL was not working on our Domino 9.0.1 Fixpack 3 Interim Fix 2 server using a SHA-2 SSL certificate. Even with the proper TrustedCert.class file access was failing.

Looking at the Domino server console we noticed the following error when the DIIOP task was started:
DIIOP Server: Agent error: keyrng: Could not read certificate

We also noticed the following errors at different intervals when trying to access DIIOP over SSL:
TLS/SSL connection X.X.X.X(63149)-X.X.X.X(64023) failed with rejected protocol version

Also from the Java application trying to connect to DIIOP over SSL we saw the following error:
NotesException: Session closed due to communications failure

To fix: TLS/SSL connection X.X.X.X(63149)-X.X.X.X(64023) failed with rejected protocol version enable SSLv3 on the server.
To do so set “DISABLE_SSLv3=0” in the notes.ini

The same Java code was working fine against a Domino 9.0.1 FP3 IF3 server with a SHA-1 certificate, so, after enabling SSLv3 we switched out the SHA-2 certificate with a SHA-1 SelfCert created by Domino and all was good in the DIIOP world again.

Note that making DIIOP over SSL work opens up your HTTP server to SSL vulnerabilities, so it would be best to either disable HTTP altogether or put the server behind a reverse proxy such as Apache or IHS to protect the HTTP task while allowing access to DIIOP.

We checked the above with IBM and, as of this writing, IBM acknowledges that DIIOP over SSL with a SHA-2 certificate with SSLv3 disabled is not currently supported. They are working on the issue, but this may not be resolved in the near future.

Other Errors you might have seen on the Domino Server Console when investigating:
TLS/SSL connection X.X.X.X(443)-X.X.X.X(11951) failed with server certificate chain signature alogrithms NOT supported by client
TLS/SSL connection X.X.X.X(443)-X.X.X.X(11951) failed with server certificate chain requiring support for SHA384
HTTP Server: SSL handshake failure, IP address [X.X.X.X], Keyring [cert.kyr], [SSL Error: Invalid SSL message], code [4166]
HTTP Server: SSL handshake failure, IP address [X.X.X.X], Keyring [cert.kyr], [SSL Error: Invalid peer], code [4171]