Skip to main content

Create OpenSearch

This guide shows how to add an OpenSearch cluster (Apache 2.0) to a Live System running on Kubernetes, and how to give a workload or Jaeger access to it.

The offer is Storage.CaaS.OpenSearch ("OpenSearch on Kubernetes"), under the Storage.CaaS.Search component. It is reconciled by the CaaS Kubernetes agent and works on any managed Kubernetes cluster Fractal Cloud supports (AKS, EKS, GKE, OKE, KaaS). Parameters, output fields and link settings are listed in the Component Reference, for example AWS.

SDK Availability

OpenSearch is offered in the Live System editor of the Fractal Cloud web app wherever the CaaS Kubernetes agent runs in your environment. Support in the Java SDK and the TypeScript SDK is coming in their next releases.

What you get​

  • An OpenSearch cluster of replicas nodes (default 3), each with a storage volume (default 100Gi), installed from the opensearch-project Helm chart into namespace (default opensearch).
  • The security plugin is always on. There are no demo certificates and no default admin password: the agent generates the admin password and a private CA with node and HTTP certificates, and keeps them in Kubernetes Secrets in the component's namespace (adminSecretName, caSecretName output fields).
  • OpenSearch Dashboards, when dashboards is true (dashboardsUrl output field).

Add the component​

The component depends on the Kubernetes cluster it runs on:

{
  "id": "search",
  "type": "Storage.CaaS.OpenSearch",
  "parameters": {
    "replicas": 3,
    "storage": "100Gi",
    "dashboards": true
  },
  "dependencies": ["cluster"]
}

Give a consumer access​

A consumer never receives the admin credential. It links to the OpenSearch component and states what it needs; the OpenSearch component creates a dedicated user for it, with one role limited to the indices under its prefix:

{
  "id": "orders-indexer",
  "type": "CustomWorkloads.CaaS.KubernetesWorkload",
  "links": [
    { "componentId": "search", "settings": { "access": "read-write", "indexPrefix": "orders-" } }
  ]
}
  • access is required: read, write or read-write.
  • indexPrefix defaults to <consumerId>-. Lower-case letters, digits, -, _ and . only.

The user's password and the CA certificate are written into a Kubernetes Secret in the consumer's namespace (keys password and ca.crt). A Kubernetes workload receives SEARCH_URL, SEARCH_USERNAME, SEARCH_PASSWORD_REF, SEARCH_INDEX_PREFIX and SEARCH_CA_REF; the raw password never appears in output fields. Removing the link removes the user, its role and the secret.

The consumer may also live in another Live System of the same environment, by referencing the OpenSearch component.

Store Jaeger traces in OpenSearch​

Observability.CaaS.Jaeger stores its traces in OpenSearch. It needs exactly one link to an OpenSearch component, and a dependency on it so it is deployed once OpenSearch is ready:

{
  "id": "tracing",
  "type": "Observability.CaaS.Jaeger",
  "dependencies": ["cluster", "search"],
  "links": [
    { "componentId": "search", "settings": { "access": "read-write", "indexPrefix": "tracing-jaeger-" } }
  ]
}

Migrate from Elastic​

Fractal Cloud no longer offers Elastic (licensing). Storage.CaaS.Elastic, Observability.CaaS.Elastic and Observability.SaaS.Elastic are retired, and Jaeger now stores its traces in OpenSearch instead of a private Elasticsearch.

What happens to a Live System that already runs Elastic​

  • Nothing is deleted. The agents stop managing the retired types: the component is answered as not supported, and the Elasticsearch, Kibana and APM resources it deployed keep running in the cluster, unmanaged, with their data.
  • No more drift correction or upgrades. Changes to those components are no longer applied, and removing them from the Live System does not delete the cluster resources.
  • Jaeger. A Jaeger deployed before the change is reported as failed until it is linked to an OpenSearch component, with a message saying so. Its pods are left as they are until then. Its old Elasticsearch, the jaeger-secret and the stored traces stay in its namespace and are not migrated.
  • Observability.SaaS.Elastic only published the URLs of an Elastic hosted elsewhere. That service is not touched; remove the component from the Live System.

Moving to OpenSearch​

  1. Add a Storage.CaaS.OpenSearch component to the Live System (see Add the component) and wait until it is Active.
  2. Copy the data you need. Elasticsearch snapshots taken with 7.11 or later cannot be restored into OpenSearch, so copy indices with a reindex tool such as OpenSearch Data Prepper or Logstash, or re-ingest them from their source.
  3. Link each consumer to the OpenSearch component (see Give a consumer access) and point it at the SEARCH_* values instead of the old Elastic endpoint and credentials.
  4. Link Jaeger to OpenSearch (see Store Jaeger traces in OpenSearch). New traces go to OpenSearch; traces stored before the switch are not shown.
  5. Remove the Elastic components from the Live System, then delete what they left in the cluster once nothing uses it: the Elasticsearch, Kibana and ApmServer resources, their secrets and volumes, and the ECK operator if no other Elasticsearch remains.