Import Index a JSON file into Elasticsearch

Question

I am new to Elasticsearch and have been entering data manually up until this point  For example I ve done something like this     curl -XPUT  http   localhost 9200 twitter tweet 1  -d         user     kimchy        post date     2009-11-15T14 12 12        message     trying out Elastic Search       I now have a  json file and I want to index this into Elasticsearch   I ve tried something like this too  but no success   curl -XPOST  http   jfblouvmlxecs01 9200 test test 1  -d lane json   How do I import a  json file   Are there steps I need to take first to ensure the mapping is correct

User · Answer

if you are using VirtualBox and UBUNTU in it or you are simply using UBUNTU then it can be useful    wget https   github com andrewvc ee-datasets archive master zip sudo apt-get install unzip  only if unzip module is not installed  unzip master zip cd ee-datasets java -jar elastic-loader jar http   localhost 9200 datasets movie db eloader

User · Answer

I wrote some code to expose the Elasticsearch API via a Filesystem API   It is good idea for clear export import of data for example   I created prototype elasticdriver   It is based on FUSE

User · Answer

As of Elasticsearch 7 7  you have to specify the content type also   curl -s -H  Content-Type  application json  -XPOST localhost 9200  bulk --data-binary   lt absolute path to JSON file gt

User · Answer

I m the author of elasticsearch loader I wrote ESL for this exact problem     You can download it with pip   pip install elasticsearch-loader   And then you will be able to load json files into elasticsearch by issuing   elasticsearch loader --index incidents --type incident json file1 json file2 json

User · Answer

If you want to import a json file into Elasticsearch and create an index  use this Python script  import json from elasticsearch import Elasticsearch  es   Elasticsearch    host    localhost    port   9200    i   0 with open  el dharan json   as raw data      json docs   json load raw data      for json doc in json docs              i   i   1             es index index  ind dharan   doc type  doc dharan   id i  body json dumps json doc

User · Answer

We made a little tool for this type of thing https   github com taskrabbit elasticsearch-dump

User · Answer

just get postman from https   www getpostman com docs environments give it the file location with  test test 1  bulk pretty command

User · Answer

If you are using the elastic search 7 7 or above version then follow below command  curl -H  quot Content-Type  application json quot  -XPOST  quot localhost 9200 bank  bulk  pretty amp refresh quot  --data-binary   quot  Users waseem khan waseem elastic account json quot   On above file path is  Users waseem khan waseem elastic account json   If you are using elastic search 6 x version then you can use the below command    curl -X POST localhost 9200 bank  bulk pretty amp refresh --data-binary   quot  Users waseem khan waseem elastic account json quot  -H  Content-Type  application json  Note  Make sure in your  json file at the end you will add the one empty line otherwise you will be getting below exception   quot error quot       quot root cause quot               quot type quot     quot illegal argument exception quot        quot reason quot     quot The bulk request must be terminated by a newline   n  quot          quot type quot     quot illegal argument exception quot    quot reason quot     quot The bulk request must be terminated by a newline   n  quot      enter code here  quot status quot    400

User · Answer

One thing I ve not seen anyone mention  the JSON file must have one line specifying the index the next line belongs to  for every line of the  pure  JSON file   I E     index     index   shakespeare    type   act    id  0     line id  1  play name   Henry IV   speech number      line number      speaker      text entry   ACT I     Without that  nothing works  and it won t tell you why

User · Answer

The right command if you want to use a file with curl is this   curl -XPOST  http   jfblouvmlxecs01 9200 test  doc 1  -d  lane json   Elasticsearch is schemaless  therefore you don t necessarily need a mapping  If you send the json as it is and you use the default mapping  every field will be indexed and analyzed using the standard analyzer   If you want to interact with Elasticsearch through the command line  you may want to have a look at the elasticshell which should be a little bit handier than curl   2019-07-10  Should be noted that custom mapping types is deprecated and should not be used  I updated the type in the url above to make it easier to see which was the index and which was the type as having both named  test  was confusing

User · Answer

Adding to KenH s answer    curl -s -XPOST localhost 9200  bulk --data-binary  requests   You can replace  requests with  complete path to json file  Note   is important before the file path

User · Answer

I just made sure that I am in the same directory as the json file and then simply ran this  curl -s -H  Content-Type  application json  -XPOST localhost 9200 product default  bulk pretty --data-binary  product json   So if you too make sure you are at the same directory and run it this way  Note  product default  in the command is something specific to my environment  you can omit it or replace it with whatever is relevant to you

User · Answer

You are using     curl -s -XPOST localhost 9200  bulk --data-binary  requests   If  requests  is a json file then you have to change this to     curl -s -XPOST localhost 9200  bulk --data-binary  requests json   Now before this  if your json file is not indexed  you have to insert an index line before each line inside the json file  You can do this with JQ  Refer below link  http   kevinmarsh com 2014 10 23 using-jq-to-import-json-into-elasticsearch html  Go to elasticsearch tutorials  example the shakespeare tutorial  and download the json file sample used and have a look at it  In front of each json object  each individual line  there is an index line  This is what you are looking for after using the jq command  This format is mandatory to use the bulk API  plain json files wont work

User · Answer

Per the current docs  https   www elastic co guide en elasticsearch reference current docs-bulk html       If you   re providing text file input to curl  you must use the   --data-binary flag instead of plain -d  The latter doesn   t preserve newlines    Example      curl -s -XPOST localhost 9200  bulk --data-binary  requests

[json] Import/Index a JSON file into Elasticsearch

Examples related to json

Examples related to elasticsearch