I have 3 master eligible nodes and, as per Elasticsearch documentation, to avoid the split brain issue I set minimum_master_nodes to 2.
But that means that if a catastrophe occurs and only one node survives or comes back online, then the cluster will remain inoperative as it won't find enough quorum to elect a master.
Is this solvable with some setting so that the last standing node will form a cluster with itself or do I always need to guarantee at least two nodes are online with my current set up?
If you have two out of your three master eligible nodes that are out of service, then you probably have other issues to worry about than operating your cluster.
The odds that 2/3 master nodes go down should be very low. But if you feel you can't guarantee that condition in your cluster, you might want to increase to 5 master-eligible nodes and set minimum_master_nodes = 3, which would increases the chances of keeping your cluster up.
With three master-eligible nodes, I wouldn't worry too much about that condition, but if you really need to, you could come up with some cron'ed shell script that would check the number of master nodes and if it falls below 2, then you could simply dynamically change the number of minimum master nodes back to 1:
# retrieve the number of master nodes
MASTER_COUNT=$(curl -s 'localhost:9200/_cat/master' | wc -l)
# if we have less than two masters, change the setting
if [ "$MASTER_COUNT" < 2 ]; then
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient": {
"discovery.zen.minimum_master_nodes": 1
}
}'
fi
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments