[elasticsearch] Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded

When trying to post documents to Elasticsearch as normal I'm getting this error:

cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)];

I also see this message on the Elasticsearch logs:

flood stage disk watermark [95%] exceeded ... all indices on this node will marked read-only

Only changing the settings with the following command did not work in my environment:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

I had to also ran the Force Merge API command:

curl -X POST "localhost:9200/my-index-000001/_forcemerge?pretty"

ref: Force Merge API

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

FROM

https://techoverflow.net/2019/04/17/how-to-fix-elasticsearch-forbidden-12-index-read-only-allow-delete-api/

This error is usually observed when your machine is low on disk space. Steps to be followed to avoid this error message

1. Resetting the read-only index block on the index:

   $ curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
   
   Response
   ${"acknowledged":true}
   

2. Updating the low watermark to at least 50 gigabytes free, a high watermark of at least 20 gigabytes free, and a flood stage watermark of 10 gigabytes free, and updating the information about the cluster every minute

    Request
    $curl -X PUT "http://127.0.0.1:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "50gb", "cluster.routing.allocation.disk.watermark.high": "20gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", "cluster.info.update.interval": "1m"}}'
   
     Response
     ${
      "acknowledged" : true,
      "persistent" : { },
      "transient" : {
      "cluster" : {
      "routing" : {
      "allocation" : {
      "disk" : {
        "watermark" : {
          "low" : "50gb",
          "flood_stage" : "10gb",
          "high" : "20gb"
        }
      }
    }
   }, 
   "info" : {"update" : {"interval" : "1m"}}}}}
   

After running these two commands, you must run the first command again so that the index does not go again into read-only mode

By default, Elasticsearch installed goes into read-only mode when you have less than 5% of free disk space. If you see errors similar to this:

Elasticsearch::Transport::Transport::Errors::Forbidden: [403] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}

Or in /usr/local/var/log/elasticsearch.log you can see logs similar to:

flood stage disk watermark [95%] exceeded on [nCxquc7PTxKvs6hLkfonvg][nCxquc7][/usr/local/var/lib/elasticsearch/nodes/0] free: 15.3gb[4.1%], all indices on this node will be marked read-only

Then you can fix it by running the following commands:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'