Renewing expired OpenSearch and Mongo certificates in HCL Connections Component Pack

Posted on by

In the past, I’ve written about expiring certificates in your Kubernetes environment. These will hit you if you don’t update your Kubernetes environment within a year. Today, we were hit by other expiring certificates. Those of OpenSearch and Mongo, as part of the HCL Connections Component Pack. Fortunately, we encountered this issue first in our DEV environment. If it wasn’t for our recent migration to Traefik, that would have been the third expiring certificate. During the migration that certificate already got renewed. What we noticed was that after restarting the OpenSearch client pods, they wouldn’t come back up. The OpenSearch client pods entered a CrashLoopBackOff state. The pod logs showed a stack trace containing a rather clear error:

Caused by: java.security.cert.CertificateExpiredException:
NotAfter: Fri Aug 14 06:03:08 GMT 2026

While the issue first surfaced when OpenSearch client pods failed to start, we later discovered that the Mongo client certificates had expired as well. Expired Mongo certificates may go unnoticed for some time because running Mongo pods continue to function until they are restarted. Our Mongo continued to function because the pods hadn’t restarted since the certificates had expired, but the next Mongo restart would have resulted in a second outage.

Component Pack certificates

As these certificates expire after 2 years, it will be a while until you run into this after a migration to a newer Connections version, but it’s important to be aware of them. To see when your certificates will expire, you can use these commands (on your K8s Control Plane) for OpenSearch and Mongo.

OpenSearch:

for cert in $(kubectl -n connections get secret opensearch-secret -o json | \
jq -r '.data|keys[]' | grep '\.pem$')
do
  echo "==== $cert ===="
  kubectl -n connections get secret opensearch-secret \
    -o jsonpath="{.data.$(echo $cert | sed 's/\./\\./g')}" \
    | base64 -d \
    | openssl x509 -noout -dates 2>/dev/null
done

Mongo:

for cert in $(kubectl -n connections get secret mongo-secret -o json | jq -r '.data | keys[]' | grep -E '\.(crt|pem)$')
do
  echo "===== $cert ====="

  kubectl -n connections get secret mongo-secret \
    -o jsonpath="{.data.$(echo $cert | sed 's/\./\\./g')}" \
    | base64 -d \
    | openssl x509 -noout -subject -dates 2>/dev/null
done

This will give you a list like this (OpenSearch):

==== chain-ca.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2036 GMT
==== opensearch-admin.crt.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2028 GMT
==== opensearch-healthcheck.crt.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2028 GMT
==== opensearch-http.crt.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2028 GMT
==== opensearch-metrics.crt.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2028 GMT
==== opensearch-orientme.crt.pem ====
notBefore=Aug 24 09:27:41 2026 GMT
notAfter=Aug 23 09:27:41 2028 GMT
==== opensearch-transport.crt.pem ====
notBefore=Aug 24 09:27:40 2026 GMT
notAfter=Aug 23 09:27:40 2028 GMT

Usually, your certificate expiration date for the Mongo certificates will be the same, as the certificates were created during the bootstrapping of your environment. As you can see the CA certificate is valid for 10 years, while the server certificates are valid for 2 years.

These certificates live in Kubernetes secrets. To be more specific:
OpenSearch:
– opensearch-secret
Mongo:
– mongo-secret
Traefik:
– cnx-tls-secret

Certificate renewal

The supported way to renew these certificates is to re-run the Component Pack bootstrap. This will cause downtime, so the best time to do this is during scheduled maintenance hours, for example while installing a new CR-release.

In your bootstrap.yaml values, you should find the following entries:

  force_regenerate_all: false
  force_regenerate_ingress: false
  force_regenerate_mongo: false
  force_regenerate_opensearch: false

By default, these should be set to false, but this is the time to set at least some of them to true to renew the certificates. Alternatively, you could add env.force_regenerate=true to your helm install command, to overrule the default (false) setting.

We used these commands to redo the bootstrap (with version CR14):

helm uninstall bootstrap -n connections
helm upgrade bootstrap oci://hclcr.io/cnx/bootstrap -i --version 0.1.0-20260420-135024 --namespace connections -f <path-to>/bootstrap.yml --wait

Validation

Use this command to check if your certificates were successfully regenerated:

kubectl -n connections get secret opensearch-secret \
-o jsonpath='{.data.opensearch-http\.crt\.pem}' | base64 -d | openssl x509 -noout -dates

Expected result:

notBefore=<today>
notAfter=<two years later>

You can do the same for Mongo, using this command:

kubectl -n connections get secret mongo-secret -o jsonpath='{.data.user_admin\.pem}' | base64 -d | openssl x509 -noout -dates

Next Steps

After renewing the certificates, restart the affected StatefulSets and Deployments (kubectl -n connections rollout restart sts and kubectl -n connections rollout restart deployment ). Begin with the foundational services such as OpenSearch, MongoDB and, if applicable, Traefik. Once these components are healthy again, restart the remaining Component Pack deployments. Since most Component Pack services depend directly or indirectly on these components, certificate renewal is best performed during a scheduled maintenance window, for example as part of a CR upgrade.

Important: regenerating the certificates does not merely renew the server certificates. The bootstrap process also generates new Root CA and Signing CA certificates. As a result, all applications that trust the previous OpenSearch CA chain must be updated to trust the new CA chain. This means that you will have to perform the steps to import the new certificates in your WebSphere keystore, like you did on the initial installation. Start at step 4.

So concretely,

  1. extract the chain-ca certificate and the OpenSearch metrics keystore on the Kubernetes Control-plane:
kubectl get secret opensearch-secret -n connections -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > "<path-to>/chain-ca.pem"
kubectl get secret opensearch-secret -n connections -o=jsonpath="{.data['opensearch-metrics\.p12']}" | base64 -d > "<path-to>/opensearch-metrics.p12"
  1. Copy these files to the deployment manager and all Connections WebSphere nodes, replacing the existing chain-ca.pem and opensearch-metrics.p12 files (after creating a backup of the current files).
  2. Start wsadmin on your deployment manager
  1. Update the keystore for Metrics
    The password for the newly generated opensearch-metrics.p12 keystore remains the value configured in your bootstrap deployment.
execfile('esSecurityAdmin.py')
enableSslForMetrics('<path-to>/es-keys/elasticsearch-metrics.p12', '<your-opensearch-secret password goes here>', '<path-to>/es-keys/chain-ca.pem', '30099')
  1. Synchronize all WebSphere nodes from the Deployment Manager and restart all WebSphere JVMs hosting Connections applications.

After the node synchronization and restarts have completed, Connections should again be able to establish trusted SSL connections to OpenSearch.

Administrators should consider adding a reminder approximately 21 months after a Component Pack installation or migration to verify the certificate expiration dates.