Dump all documents of Elasticsearch

Question

Is there any way to create a dump file that contains all the data of an index among with its settings and mappings    A Similar way as mongoDB does with mongodump  or as in Solr its data folder is copied to a backup location   Cheers

User · Answer

We can use elasticdump to take the backup and restore it, We can move data from one server/cluster to another server/cluster.

1. Commands to move one index data from one server/cluster to another using elasticdump.

# Copy an index from production to staging with analyzer and mapping:
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=analyzer
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=mapping
elasticdump \
  --input=http://production.es.com:9200/my_index \
  --output=http://staging.es.com:9200/my_index \
  --type=data

2. Commands to move all indices data from one server/cluster to another using multielasticdump.

Backup

multielasticdump \
  --direction=dump \
  --match='^.*$' \
  --limit=10000 \
  --input=http://production.es.com:9200 \
  --output=/tmp

Restore

multielasticdump \
  --direction=load \
  --match='^.*$' \
  --limit=10000 \
  --input=/tmp \
  --output=http://staging.es.com:9200

Note:

If the --direction is dump, which is the default, --input MUST be a URL for the base location of an ElasticSearch server (i.e. http://localhost:9200) and --output MUST be a directory. Each index that does match will have a data, mapping, and analyzer file created.
For loading files that you have dumped from multi-elasticsearch, --direction should be set to load, --input MUST be a directory of a multielasticsearch dump and --output MUST be a Elasticsearch server URL.
The 2nd command will take a backup of settings, mappings, template and data itself as JSON files.
The --limit should not be more than 10000 otherwise, it will give an exception.
Get more details here.

User · Answer

You can also dump elasticsearch data in JSON format by http request  https   www elastic co guide en elasticsearch reference current search-request-scroll html CURL -XPOST  https   ES INDEX  search scroll 10m  CURL -XPOST  https   ES  search scroll  -d    scroll    10m    scroll id    ID

User · Answer

To export all documents from ElasticSearch into JSON  you can use the esbackupexporter tool  It works with index snapshots  It takes the container with snapshots  S3  Azure blob or file directory  as the input and outputs one or several zipped JSON files per index per day  It is quite handy when exporting your historical snapshots  To export your hot index data  you may need to make the snapshot first  see the answers above

User · Answer

For your case Elasticdump is the perfect answer   First  you need to download the mapping and then the index   Install the elasticdump  npm install elasticdump -g    Dump the mapping  elasticdump --input http    lt your es server ip gt  9200 index --output es mapping json --type mapping    Dump the data elasticdump --input http    lt your es server ip gt  9200 index --output es index json --type data       If you want to dump the data on any server I advise you to install esdump through docker  You can get more info from this website Blog Link

User · Answer

The data itself is one or more lucene indices  since you can have multiple shards  What you also need to backup is the cluster state  which contains all sorts of information regarding the cluster  the available indices  their mappings  the shards they are composed of etc   It s all within the data directory though  you can just copy it  Its structure is pretty intuitive  Right before copying it s better to disable automatic flush  in order to backup a consistent view of the index and avoiding writes on it while copying files   issue a manual flush  disable allocation as well  Remember to copy the directory from all nodes   Also  next major version of elasticsearch is going to provide a new snapshot restore api that will allow you to perform incremental snapshots and restore them too via api  Here is the related github issue  https   github com elasticsearch elasticsearch issues 3826

User · Answer

Here s a new tool we ve been working on for exactly this purpose https   github com taskrabbit elasticsearch-dump   You can export indices into out of JSON files  or from one cluster to another

User · Answer

ElasticSearch itself provides a way to create data backup and restoration  The simple command to do it is   CURL -XPUT  localhost 9200  snapshot  lt backup folder name gt   lt backupname gt   -d         indices     lt index name gt         ignore unavailable   true       include global state   false      Now  how to create  this folder  how to include this folder path in ElasticSearch configuration  so that it will be available for ElasticSearch  restoration method  is well explained here  To see its practical demo surf here

User · Answer

Elasticsearch supports this now out of the box   https   www elastic co guide en elasticsearch reference current modules-snapshots html

[elasticsearch] Dump all documents of Elasticsearch

Examples related to elasticsearch