Elasticsearch error cluster block exception FORBIDDEN 12 index read-only allow delete api flood stage disk watermark exceeded

Question

When trying to post documents to Elasticsearch as normal I m getting this error   cluster block exception  FORBIDDEN 12 index read-only   allow delete  api      I also see this message on the Elasticsearch logs   flood stage disk watermark  95   exceeded     all indices on this node will marked read-only

User · Answer

This error is usually observed when your machine is low on disk space. Steps to be followed to avoid this error message

Resetting the read-only index block on the index:

$ curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Response
${"acknowledged":true}

Updating the low watermark to at least 50 gigabytes free, a high watermark of at least 20 gigabytes free, and a flood stage watermark of 10 gigabytes free, and updating the information about the cluster every minute

 Request
 $curl -X PUT "http://127.0.0.1:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "50gb", "cluster.routing.allocation.disk.watermark.high": "20gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", "cluster.info.update.interval": "1m"}}'

  Response
  ${
   "acknowledged" : true,
   "persistent" : { },
   "transient" : {
   "cluster" : {
   "routing" : {
   "allocation" : {
   "disk" : {
     "watermark" : {
       "low" : "50gb",
       "flood_stage" : "10gb",
       "high" : "20gb"
     }
   }
 }
}, 
"info" : {"update" : {"interval" : "1m"}}}}}

After running these two commands, you must run the first command again so that the index does not go again into read-only mode

User · Answer

By default  Elasticsearch installed goes into read-only mode when you have less than 5  of free disk space  If you see errors similar to this      Elasticsearch  Transport  Transport  Errors  Forbidden     403      error    root cause     type   cluster block exception   reason   blocked   by   FORBIDDEN 12 index read-only   allow delete    api        type   cluster block exception   reason   blocked by     FORBIDDEN 12 index read-only   allow delete  api       status  403    Or in  usr local var log elasticsearch log you can see logs similar to      flood stage disk watermark  95   exceeded on    nCxquc7PTxKvs6hLkfonvg  nCxquc7   usr local var lib elasticsearch nodes 0    free  15 3gb 4 1    all indices on this node will be marked read-only   Then you can fix it by running the following commands   curl -XPUT -H  Content-Type  application json  http   localhost 9200  cluster settings -d     transient      cluster routing allocation disk threshold enabled   false      curl -XPUT -H  Content-Type  application json  http   localhost 9200  all  settings -d    index blocks read only allow delete   null

User · Answer

curl -XPUT -H  Content-Type  application json  http   localhost 9200  all  settings -d    index blocks read only allow delete   null     FROM  https   techoverflow net 2019 04 17 how-to-fix-elasticsearch-forbidden-12-index-read-only-allow-delete-api

User · Answer

Only changing the settings with the following command did not work in my environment  curl -XPUT -H  quot Content-Type  application json quot  http   localhost 9200  all  settings -d    quot index blocks read only allow delete quot   null    I had to also ran the Force Merge API command  curl -X POST  quot localhost 9200 my-index-000001  forcemerge pretty quot   ref  Force Merge API

User · Answer

This happens when Elasticsearch thinks the disk is running low on space so it puts itself into read-only mode   By default Elasticsearch s decision is based on the percentage of disk space that s free  so on big disks this can happen even if you have many gigabytes of free space   The flood stage watermark is 95  by default  so on a 1TB drive you need at least 50GB of free space or Elasticsearch will put itself into read-only mode   For docs about the flood stage watermark see https   www elastic co guide en elasticsearch reference 6 2 disk-allocator html   The right solution depends on the context - for example a production environment vs a development environment   Solution 1  free up disk space  Freeing up enough disk space so that more than 5  of the disk is free will solve this problem  Elasticsearch won t automatically take itself out of read-only mode once enough disk is free though  you ll have to do something like this to unlock the indices     curl -XPUT -H  Content-Type  application json  https    YOUR ELASTICSEARCH ENDPOINT  9200  all  settings -d    index blocks read only allow delete   null     Solution 2  change the flood stage watermark setting  Change the  cluster routing allocation disk watermark flood stage  setting to something else  It can either be set to a lower percentage or to an absolute value  Here s an example of how to change the setting from the docs   PUT  cluster settings      transient          cluster routing allocation disk watermark low    100gb        cluster routing allocation disk watermark high    50gb        cluster routing allocation disk watermark flood stage    10gb        cluster info update interval    1m          Again  after doing this you ll have to use the curl command above to unlock the indices  but after that they should not go into read-only mode again

[elasticsearch] Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded

Examples related to elasticsearch