Optimizing service mesh memory Innovation Release

Hybrid Manager uses Istio as its service mesh to manage network traffic between Postgres clusters and other components. As the number of managed Postgres clusters grows, Istio's memory requirements increase. You can adjust the memory limits for Istio to optimize performance based on your expected workload.

Understanding Istio memory requirements

The memory requirements for Istio depend on several factors:

  • Number of Postgres clusters: More clusters require more memory for service discovery and routing
  • Cluster architecture: Complex architectures (such as PGD clusters with multiple data groups) require more memory than simple standalone clusters
  • Network traffic: Higher network activity increases memory usage

By default, Hybrid Manager configures Istio with conservative memory limits. As your environment grows, you may need to adjust these limits to prevent performance degradation or out-of-memory errors.

The following table shows the recommended memory limits based on the number of Postgres clusters you plan to manage:

Number of Postgres ClustersRecommended Memory Limit
up to 10-20 clusters512Mi
up to 30-40 clusters1Gi
up to 60-70 clusters1.5Gi
scale more if you need> 1.5Gi
Note

These recommendations are guidelines. Your actual requirements may vary depending on your cluster architectures and network traffic patterns. Monitor Istio's memory usage and adjust accordingly.

Configuring Istio memory limits

You can configure the Istio memory limits by setting the memory_limit and cpu_limit values for the upm-istio component.

Tip

Start with the recommended values for your expected cluster count. If you plan to scale beyond 70 clusters, increase the memory limit proportionally (for example, add approximately 512Mi per additional 30 clusters).

Monitoring Istio memory usage

After adjusting the memory limits, monitor Istio's performance to ensure the configuration is appropriate for your workload.

Using Grafana dashboards

Monitor Istio memory usage through the Grafana dashboards available in Hybrid Manager's Launchpad:

  1. From the Hybrid Manager portal, open the user menu and select Launchpad.
  2. On the Grafana card, select View service.
  3. Navigate to Dashboards > Kubernetes.
  4. Select the Kubernetes / Compute Resources / Workload dashboard.
  5. Filter by:
    • namespace: istio-system
    • workload: istiod (for the Istio control plane)

Monitor the Memory Usage panel to see if usage is consistently approaching your configured limits. If it is, consider increasing the memory_limit value.

If Istio is experiencing memory pressure, you might see the following symptoms:

  • Pods restarting frequently
  • OOMKilled (Out of Memory Killed) events in pod descriptions
  • Slow response times for Postgres cluster operations

To check pod status and events:

kubectl get pods -n istio-system
kubectl describe pod <pod-name> -n istio-system

Look for events mentioning OOMKilled or memory-related errors in the output.