I have noSQL database, distributed among many clusters. It has SOLR index and a lot of…
- User
I have noSQL database, distributed among many clusters. It has SOLR index and a lot of data. Once a week administrator is starting reindex process on index on one of the clusters. During this time, this clusters responds very slowly - it takes over 30 seconds. However, other clusters, holding replica of this data, are not reindexing and fully available.
When query is sent to database client and this client tries to query reindexing cluster, it will wait for a response, because cluster - technically - is still online. How to design a system, so that database client will always know which cluster is reindexing, and route queries to other clusters?
Assistant
You need a mechanism to track cluster health/status and implement intelligent routing. Here are several approaches to solve this:
Health Check and Service Discovery Approach
Implement a health check endpoint on each cluster that reports its current status. The endpoint should return different states like "healthy", "reindexing", or "degraded" along with response time metrics. Your database client can periodically poll these endpoints (every 10-30 seconds) and maintain a local cache of cluster states.
When the reindexing process starts, the cluster should update its health status to indicate it's in maintenance mode. The client's load balancer can then exclude this cluster from the rotation until reindexing completes.
Circuit Breaker Pattern
Implement circuit breakers in your database client for each cluster. Set a response time threshold (e.g., 5 seconds) and failure rate. When a cluster consistently responds slowly during reindexing, the circuit breaker opens and routes traffic elsewhere. This provides automatic failover without requiring explicit reindexing notifications.
Administrative Coordination
Create an administrative interface where the reindexing process registers itself before starting. This could be a simple shared state store (Redis, etcd, or even a database table) that tracks which clusters are currently reindexing. Your client checks this state before routing queries.
The reindex script would:
- Register the cluster as "reindexing" in the shared state
- Perform the reindex operation
- Remove the "reindexing" flag when complete
Response Time Monitoring
Implement adaptive routing based on real-time response time monitoring. Your client can track rolling averages of response times for each cluster and automatically prefer faster-responding clusters. When a cluster's response time degrades significantly (like during reindexing), traffic naturally shifts to better-performing replicas.
Hybrid Approach
The most robust solution combines multiple techniques: use administrative coordination for planned maintenance like reindexing, circuit breakers for unexpected issues, and health checks as a backup verification mechanism.
Which of these approaches fits best with your current architecture? Do you have existing service discovery infrastructure or monitoring systems that could support this?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.