Removing security-auditlog indexes from your OpenSearch cluster

Posted on by

After my previous article on Querying the OpenSearch index, a colleague and I used the script to take a good look at the OpenSearch index in our environment. That environment, is a large Connections installation (45K users) in the Netherlands. We found two disturbing facts:

  • The cluster status was yellow, due to the QuickResults index not having a replica of all its shards
  • There was a huge amount of secruity-auditlog indexes, one per day. There were as many as the days passed since our migration to Connections 8

It turns out these two issues were closely related.

Why too many small indexes can hurt OpenSearch

Each OpenSearch index is split into shards, the basic storage and search units. Even a tiny index, just a few megabytes, still requires:

  • At least one primary shard,
  • Some metadata stored in the cluster state,
  • JVM heap memory to manage it.

When you create hundreds or thousands of small indexes, like the daily security-auditlog-* indexes, they start to consume a disproportionate amount of resources.

Over time, this leads to:

  • Large cluster state metadata, which slows down the cluster.
  • Higher JVM heap usage, reducing available memory for searches.
  • Replica allocation failures, because there are simply too many shards to place.

In our case, the excessive auditlog shards left no room for the QuickResults index to allocate all its replicas, hence the yellow cluster status.

Do we really need security audit logs?

The security-auditlog-* indexes store:

  • Authentication and authorization events
  • Configuration changes
  • Which user accessed which index, from which IP
  • What actions were allowed or denied

This is very useful if:

  • You have external users or APIs connecting to OpenSearch
  • You need audit trails for compliance
  • You’re troubleshooting access or permission issues

However, in our Connections environment, only the internal Connections applications connect to OpenSearch. We didn’t even realize these audit logs existed, and we had never needed them.

To double-check, I asked two fellow HCL Connections ambassadors, Christoph Stöttner and Urs Meli, both of whom manage large Connections environments. Their verdict was unanimous:

They had completely disabled the audit logs, because they were unnecessary overhead.

Urs kindly shared his script to disable them, which I’ll share below.

Stop the creation of security-audit logs

To disable security audit logging, you simply need to set audit.type to none in the OpenSearch ConfigMaps. Here’s the script (thanks Urs Meli):

#!/bin/sh
NAMESPACE="connections"
echo "patching config maps"
for cm in opensearch-cluster-data-config opensearch-cluster-client-config opensearch-cluster-master-config
do
    kubectl -n $NAMESPACE get cm/$cm -o yaml > $cm.yml
    sed -i 's|audit.type: internal_opensearch|audit.type: none|' $cm.yml
    kubectl -n $NAMESPACE delete cm/$cm
    kubectl apply -f $cm.yml

done

echo "restart opensearch"
for cm in opensearch-cluster-data opensearch-cluster-client opensearch-cluster-master
do
    kubectl -n $NAMESPACE rollout restart statefulset/$cm


done
echo "done updating"

After applying this in our dev environment, it worked perfectly.

Remove old security-audit logs

Above script removes the creation of new logs, but it doesn’t remove the existing ones. So I wrote a script for that. Out of caution, the script allows you to leave some recent logs. Up to you if you want to keep some for now to decide later if you want to use them, or simply remove them all. The script uses the sendRequest script from my previous post.

#!/bin/bash
# Namespace: Cleanup old security-auditlog indexes in OpenSearch
# Author: Martijn de Jong
#
# Deletes daily security-auditlog indexes older than X days
# Requires: sendRequest.sh wrapper for OpenSearch API
# See: https://blog.martdj.nl/2025/07/01/querying-your-opensearch-index/

DAYS_TO_KEEP=30
SENDREQ="/opt/sendRequest/sendRequest.sh"   # Adjust path if needed

cutoff=$(date -d "$DAYS_TO_KEEP days ago" +"%Y.%m.%d")

echo "[INFO] Cutoff date: $cutoff (keeping last $DAYS_TO_KEEP days)"
echo "[INFO] Fetching all security-auditlog indexes..."

# Get all matching indexes
indices=$($SENDREQ GET "/_cat/indices/security-auditlog-*?h=index" | awk '{print $1}')

# Pre-filter for deletion
delete_list=()
for index in $indices; do
  index_date=$(echo "$index" | sed 's/security-auditlog-//')
  if [[ "$index_date" < "$cutoff" ]]; then
    delete_list+=("$index")
  fi
done

if [ ${#delete_list[@]} -eq 0 ]; then
  echo "[INFO] No indexes older than $cutoff found."
  exit 0
fi

echo "[INFO] The following indexes will be deleted:"
printf '%s\n' "${delete_list[@]}"

read -p "Are you sure you want to delete these ${#delete_list[@]} indexes? (y/N) " confirm
if [[ "$confirm" != "y" && "$confirm" != "Y" ]]; then
  echo "[INFO] Aborted."
  exit 0
fi

# Bulk delete in one API call
delete_payload="{\"indices\":[\"$(IFS=\",\"; echo "${delete_list[*]}\")\"]}"
$SENDREQ POST "/_indices/delete" -d "$delete_payload"

echo "[INFO] Deletion request sent. Old auditlog indexes removed."

Tip: If you have hundreds of these indexes, this may take a while. Perfect moment for a coffee break!

When NOT to disable audit logs

Before you blindly turn off audit logs, remember that they can be critical in some environments:

  • If you have external clients connecting to OpenSearch
  • If you need compliance & security audits
  • If you rely on them for troubleshooting access issues

In those cases, you may want to:

  • Keep audit logs but use Index Lifecycle Management (ILM) to auto-delete old ones
  • Or store them on a dedicated cluster

Conclusion

Daily security-auditlog-* indexes are a hidden shard killer in large HCL Connections deployments. Disabling unnecessary audit logs freed up hundreds of shards in our cluster, restored the QuickResults index to green, and reduced OpenSearch memory usage.

If you’re in a similar setup with only internal Connections apps accessing OpenSearch, you can safely disable audit logs and clean up the old ones.

But if you need audit trails for security or compliance, consider retaining and rotating them instead.