Parsing JSON with Unix tools

Question

I m trying to parse JSON returned from a curl request  like so   curl  http   twitter com users username json        sed -e  s         g         awk -v k  text    n split  0 a       for  i 1  i lt  n  i    print a i      The above splits the JSON into fields  for example          geo enabled  false  friends count  245  profile text color   000000   status   in reply to screen name  null  source   web   truncated  false  text   My status   favorited  false         How do I print a specific field  denoted by the -v k text

User · Answer

i used this to extract video duration from ffprobe json output :

MOVIE_INFO=`ffprobe "path/to/movie.mp4"  -show_streams -show_format -print_format json -v quiet` 
MOVIE_SECONDS=`echo "$MOVIE_INFO"|grep -w \"duration\" |tail -1 | cut -d\" -f4 |cut -d \. -f 1`

it can be used to extract value from any json :

value=`echo "$jsondata"|grep -w \"key_name\" |tail -1 | cut -d\" -f4

User · Answer

To quickly extract the values for a particular key  I personally like to use  grep -o   which only returns the regex s match   For example  to get the  text  field from tweets  something like   grep -Po   text              tweets json   This regex is more robust than you might think  for example  it deals fine with strings having embedded commas and escaped quotes inside them  I think with a little more work you could make one that is actually guaranteed to extract the value  if it s atomic    If it has nesting  then a regex can t do it of course    And to further clean  albeit keeping the string s original escaping  you can use something like    perl -pe  s  text      s       s           I did this for this analysis    To all the haters who insist you should use a real JSON parser -- yes  that is essential for correctness  but   To do a really quick analysis  like counting values to check on data cleaning bugs or get a general feel for the data  banging out something on the command line is faster   Opening an editor to write a script is distracting  grep -o is orders of magnitude faster than the Python standard json library  at least when doing this for tweets  which are  2 KB each    I m not sure if this is just because json is slow  I should compare to yajl sometime   but in principle  a regex should be faster since it s finite state and much more optimizable  instead of a parser that has to support recursion  and in this case  spends lots of CPU building trees for structures you don t care about    If someone wrote a finite state transducer that did proper  depth-limited  JSON parsing  that would be fantastic   In the meantime we have  grep -o      To write maintainable code  I always use a real parsing library   I haven t tried jsawk  but if it works well  that would address point  1   One last  wackier  solution  I wrote a script that uses Python json and extracts the keys you want  into tab-separated columns  then I pipe through a wrapper around awk that allows named access to columns   In here  the json2tsv and tsvawk scripts   So for this example it would be   json2tsv id text  lt  tweets json   tsvawk   print  tweet    id   is     text     This approach doesn t address  2  is more inefficient than a single Python script  and it s a little brittle  it forces normalization of newlines and tabs in string values  to play nice with awk s field record-delimited view of the world   But it does let you stay on the command line  with more correctness than grep -o

User · Answer

One interesting tool that hasn t be covered in the existing answers is using gron written in Go which has a tagline that says Make JSON greppable  which is exactly what it does  So essentially gron breaks down your JSON into discrete assignments see the absolute  path  to it  The primary advantage of it over other tools like jq would be to allow searching for the value without knowing how nested the record to search is present at  without breaking the original JSON structure e g   I want to search for the  twitter username  field from the following link  I just do   gron  https   api github com users lambda    fgrep  twitter username  json twitter username    quot unlambda quot     gron  https   api github com users lambda    fgrep  twitter username    gron -u      quot twitter username quot    quot unlambda quot     As simple as that  Note how the gron -u  short for ungron  reconstructs the JSON back from the search path  The need for fgrep is just to filter your search to the paths needed and not let the search expression be evaluated as a regex  but as a fixed string  which is essentially grep -F  Another example to search for a string to see where in the nested structure the record is under   echo    quot foo quot    quot bar quot    quot zoo quot    quot moo quot   quot fine quot         gron   fgrep  quot fine quot  json foo bar zoo moo    quot fine quot    It also supports streaming JSON with its -s command line flag  where you can continuously gron the input stream for a matching record  Also gron has zero runtime dependencies  You can download a binary for Linux  Mac  Windows or FreeBSD and run it  More usage examples and trips can be found at the official Github page - Advanced Usage As for why you one can use gron over other JSON parsing tools  see from author s note from the project page  Why shouldn t I just use jq   jq is awesome  and a lot more powerful than gron  but with that power comes complexity  gron aims to make it easier to use the tools you already know  like grep and sed

User · Answer

Now that Powershell is cross platform  I thought I d throw its way out there  since I find it to be fairly intuitive and extremely simple     curl -s  https   api github com users lambda    ConvertFrom-Json    ConvertFrom-Json converts the JSON into a Powershell custom object  so you can easily work with the properties from that point forward   If you only wanted the  id  property for example  you d just do this   curl -s  https   api github com users lambda    ConvertFrom-Json   select -ExpandProperty id   If you wanted to invoke the whole thing from within Bash  then you d have to call it like this   powershell  curl -s  https   api github com users lambda    ConvertFrom-Json    Of course there s a pure Powershell way to do it without curl  which would be   Invoke-WebRequest  https   api github com users lambda    select -ExpandProperty Content   ConvertFrom-Json   Finally  there s also  ConvertTo-Json  which converts a custom object to JSON just as easily   Here s an example    New-Object PsObject -Property    Name    Tester   SomeList      one   two   three       ConvertTo-Json   Which would produce nice JSON like this      Name     Tester    SomeList                        one                     two                     three                      Admittedly  using a Windows shell on Unix is somewhat sacrilegious but Powershell is really good at some things  and parsing JSON and XML are a couple of them   This the GitHub page for the cross platform version https   github com PowerShell PowerShell

User · Answer

Following MartinR and Boecko s lead      curl -s  http   twitter com users username json    python -mjson tool   That will give you an extremely grep friendly output  Very convenient     curl -s  http   twitter com users username json    python -mjson tool   grep my key

User · Answer

I ve done this   parsing  a json response for a particular value  as follows   curl  url   grep  var   awk   print  2     sed s     g    Clearly   url here would be the twitter url  and  var would be  text  to get the response for that var   Really  I think the only thing I m doing the OP has left out is grep for the line with the specific variable he seeks  Awk grabs the second item on the line  and with sed I strip the quotes   Someone smarter than I am could probably do the whole think with awk or grep   Now  you could do it all with just sed   curl  url   sed   text  d    sed s   text     g   sed s     g   sed s     g   thus  no awk  no grep   I don t know why I didn t think of that before  Hmmm

User · Answer

This is a good usecase for pythonpy   curl  http   twitter com users username json    py  json load sys stdin   name

User · Answer

Version which uses Ruby and http   flori github com json      lt  file json ruby -e  require  rubygems   require  json   puts JSON pretty generate JSON STDIN read       or more concisely      lt  file json ruby -r rubygems -r json -e  puts JSON pretty generate JSON STDIN read

User · Answer

You could just download jq binary for your platform and run  chmod  x jq      curl  https   twitter com users username json      jq -r   name    It extracts  name  attribute from the json object   jq homepage says it is like sed for JSON data

User · Answer

Parsing JSON is painful in a shell script  With a more appropriate language  create a tool that extracts JSON attributes in a way consistent with shell scripting conventions  You can use your new tool to solve the immediate shell scripting problem and then add it to your kit for future situations   For example  consider a tool jsonlookup such that if I say jsonlookup access token id it will return the attribute id defined within the attribute token defined within the attribute access from stdin  which is presumably JSON data  If the attribute doesn t exist  the tool returns nothing  exit status 1   If the parsing fails  exit status 2 and a message to stderr  If the lookup succeeds  the tool prints the attribute s value   Having created a unix tool for the precise purpose of extracting JSON values you can easily use it in shell scripts   access token   curl  lt some horrible crap gt    jsonlookup access token id    Any language will do for the implementation of jsonlookup  Here is a fairly concise python version      usr bin python                                                                 import sys import json  try  rep   json loads sys stdin read    except      sys stderr write sys argv 0       unable to parse JSON from stdin n       sys exit 2  for key in sys argv 1        if key not in rep          sys exit 1      rep   rep key  print rep

User · Answer

You ve asked how to shoot yourself in the foot and I m here to provide the ammo   curl -s  http   twitter com users username json    sed -e  s         g    awk -v RS      -F     text   print  2     You could use tr -d      instead of sed  But leaving them out completely seems to have the desired effect as well   If you want to strip off the outer quotes  pipe the result of the above through sed  s             g   I think others have sounded sufficient alarm  I ll be standing by with a cell phone to call an ambulance  Fire when ready

User · Answer

There are a number of tools specifically designed for the purpose of manipulating JSON from the command line  and will be a lot easier and more reliable than doing it with Awk  such as jq  curl -s  https   api github com users lambda    jq -r   name   You can also do this with tools that are likely already installed on your system  like Python using the json module  and so avoid any extra dependencies  while still having the benefit of a proper JSON parser  The following assume you want to use UTF-8  which the original JSON should be encoded in and is what most modern terminals use as well  Python 3  curl -s  https   api github com users lambda          python3 -c  quot import sys  json  print json load sys stdin   name    quot   Python 2  export PYTHONIOENCODING utf8 curl -s  https   api github com users lambda          python2 -c  quot import sys  json  print json load sys stdin   name   quot   Frequently Asked Questions Why not a pure shell solution  The standard POSIX Single Unix Specification shell is a very limited language which doesn t contain facilities for representing sequences  list or arrays  or associative arrays  also known as hash tables  maps  dicts  or objects in some other languages   This makes representing the result of parsing JSON somewhat tricky in portable shell scripts  There are somewhat hacky ways to do it  but many of them can break if keys or values contain certain special characters  Bash 4 and later  zsh  and ksh have support for arrays and associative arrays  but these shells are not universally available  macOS stopped updating Bash at Bash 3  due to a change from GPLv2 to GPLv3  while many Linux systems don t have zsh installed out of the box   It s possible that you could write a script that would work in either Bash 4 or zsh  one of which is available on most macOS  Linux  and BSD systems these days  but it would be tough to write a shebang line that worked for such a polyglot script  Finally  writing a full fledged JSON parser in shell would be a significant enough enough dependency that you might as well just use an existing dependency like jq or Python instead  It s not going to be a one-liner  or even small five-line snippet  to do a good implementation  Why not use awk  sed  or grep  It is possible to use these tools to do some quick extraction from JSON with a known shape and formatted in a known way  such as one key per line  There are several examples of suggestions for this in other answers  However  these tools are designed for line based or record based formats  they are not designed for recursive parsing of matched delimiters with possible escape characters  So these quick and dirty solutions using awk sed grep are likely to be fragile  and break if some aspect of the input format changes  such as collapsing whitespace  or adding additional levels of nesting to the JSON objects  or an escaped quote within a string  A solution that is robust enough to handle all JSON input without breaking will also be fairly large and complex  and so not too much different than adding another dependency on jq or Python  I have had to deal with large amounts of customer data being deleted due to poor input parsing in a shell script before  so I never recommend quick and dirty methods that may be fragile in this way  If you re doing some one-off processing  see the other answers for suggestions  but I still highly recommend just using an existing tested JSON parser  Historical notes This answer originally recommended jsawk  which should still work  but is a little more cumbersome to use than jq  and depends on a standalone JavaScript interpreter being installed which is less common than a Python interpreter  so the above answers are probably preferable  curl -s  https   api github com users lambda    jsawk -a  return this name   This answer also originally used the Twitter API from the question  but that API no longer works  making it hard to copy the examples to test out  and the new Twitter API requires API keys  so I ve switched to using the GitHub API which can be used easily without API keys   The first answer for the original question would be  curl  http   twitter com users username json    jq -r   text

User · Answer

If pip is avaiable on the system then     pip install json-query   Examples of usage     curl -s http   0 file json   json-query        key   value           curl -s http   0 file json   json-query my key value    curl -s http   0 file json   json-query my keys  key 1 key 2 key 3    curl -s http   0 file json   json-query my keys 2 value 2

User · Answer

Update  2020  My biggest issue with external tools  e g  Python  was that you have to deal with package managers and dependencies to install them  However  now that we have jq as a standalone  static tool that s easy to install cross-platform via Github Releases and Webi  webinstall dev jq   I d recommend that  Mac  Linux  curl -sS https   webinstall dev jq   bash  Windows 10  curl exe -A MS https   webinstall dev jq   powershell  Original  2011  TickTick is a JSON parser written in bash   lt 250 lines of code  Here s the author s snippit from his article  Imagine a world where Bash supports JSON     bin bash   ticktick sh         people           quot Writers quot            quot Rod Serling quot          quot Charles Beaumont quot          quot Richard Matheson quot                quot Cast quot            quot Rod Serling quot      quot Episodes quot   156           quot Martin Landau quot      quot Episodes quot   2           quot William Shatner quot      quot Episodes quot   2                          function printDirectors       echo  quot   The   people Directors length     Directors are  quot     for director in   people Directors items      do     printf  quot     -  s n quot     director    done          people Directors      quot John Brahm quot    quot Douglas Heyes quot       printDirectors  newDirector  quot Lamont Johnson quot     people Directors push  newDirector     printDirectors  echo  quot Shifted   quot   people Directors shift     printDirectors  echo  quot Popped   quot   people Directors pop     printDirectors

User · Answer

You can try something like this -   curl -s  http   twitter com users jaypalsingh json     awk -F     -v RS       1   text    print

User · Answer

Using standard Unix tools available on most distro s  Also works well with backslashes     and quotes   quot   WARNING  this doesn t come close to the power of jq and will only work with very simple JSON objects  It s an attempt to answer to the original question and in situations where you can t install additional tools  function parse json         echo  1         sed -e  s         g          sed -e  s  quot    quot     quot    quot   g          sed -e  s  quot    quot     quot    quot   g          sed -e  s  quot     quot     quot    quot   g          sed -e  s  quot   quot     quot ---SEPERATOR---  quot   g          awk -F     -v RS  ---SEPERATOR---   quot   1    quot  2  quot    print  quot          sed -e  quot s   quot  2  quot     quot          tr -d  quot  n t quot          sed -e  s    quot   quot  g          sed -e  s         g          sed -e  s     t    g          sed -e  s   quot      -e  s  quot          parse json    quot username quot   quot john  doe quot   quot email quot   quot john doe com quot    username parse json    quot username quot   quot john doe quot   quot email quot   quot john doe com quot    email  --- outputs ---  john  doe johh doe com

User · Answer

I can not use any of the answers here  No available jq  no shell arrays  no declare  no grep -P  no lookbehind and lookahead  no Python  no Perl  no Ruby  no - not even Bash    Remaining answers simply do not work well  JavaScript sounded familiar  but the tin says Nescaffe - so it is a no go  too    Even if available  for my simple need - they would be overkill and slow   Yet  it is extremely important for me to get many variables from the json formatted reply of my modem  I am doing it in a sh with very trimmed down BusyBox at my routers  No problems using awk alone  just set delimiters and read the data  For a single variable  that is all   awk  BEGIN   FS       RS          if   2     login    print  4     test json   Remember I have no arrays  I had to assign within the awk parsed data to the 11 variables which I need in a shell script  Wherever I looked  that was said to be an impossible mission  No problem with that  too   My solution is simple  This code will  1  parse  json file from the question  actually  I have borrowed a working data sample from the most upvoted answer  and pick out the quoted data  plus 2  create shell variables from within the awk assigning free named shell variable names   eval    curl -s  https   api github com users lambda     awk   BEGIN   FS       RS              if   2     login     print  Login     4           if   2     name     print  Name     4           if   2     updated at     print  Updated     4            echo   Login   Name   Updated    No problems with blanks within  In my use  the same command parses a long single line output  As eval is used  this solution is suited for trusted data only  It is simple to adapt it to pickup unquoted data  For huge number of variables  marginal speed gain can be achieved using else if  Lack of array obviously means  no multiple records without extra fiddling  But where arrays are available  adapting this solution is a simple task    maikel sed answer almost works  but I can not comment on it   For my nicely formatted data - it works  Not so much with the example used here  missing quotes throw it off   It is complicated and difficult to modify  Plus  I do not like having to make 11 calls to extract 11 variables  Why  I timed 100 loops extracting 9 variables  the sed function took 48 99 sec and my solution took 0 91 sec  Not fair  Doing just a single extraction of 9 variables  0 51 vs  0 02 sec

User · Answer

A two-liner which uses python  It works particularly well if you re writing a single  sh file and you don t want to depend on another  py file  It also leverages the usage of pipe    echo     field      value     can be replaced by anything printing a json to the stdout    echo     field      value       python -c  import sys  json print json load sys stdin   field

User · Answer

On the basis that some of the recommendations here  esp in the comments  suggested the use of Python  I was disappointed not to find an example   So  here s a one liner to get a single value from some JSON data  It assumes that you are piping the data in  from somewhere  and so should be useful in a scripting context    echo    hostname   test   domainname   example com      python -c  import json sys obj json load sys stdin  print obj  hostname

User · Answer

Here is a good reference  In this case   curl  http   twitter com users username json    sed -e  s         g    awk -v k  text    n split  0 a       for  i 1  i lt  n  i      where   match a i      text      if where   print a i

User · Answer

Niet is a tool that help you to extract data from json or yaml file directly in your shell bash CLI     pip install niet   Consider a json file named project json with the following contents       project        meta          name  project-sample           You can use niet like this     PROJECT NAME   niet project json project meta name    echo   PROJECT NAME  project-sample

User · Answer

Parsing JSON with PHP CLI  Arguably off topic but since precedence reigns this question remains incomplete without a mention of our trusty and faithful PHP  am I right     Using the same example JSON but lets assign it to a variable to reduce obscurity     export JSON    hostname   test   domainname   example com      Now for PHP goodness  using file get contents and the php   stdin stream wrapper      echo  JSON php -r  echo json decode file get contents  php   stdin   - gt hostname     or as pointed out using fgets and the already opened stream at CLI constant STDIN     echo  JSON php -r  echo json decode fgets STDIN  - gt hostname     nJoy

User · Answer

Here is a simple approach for Node js-ready environment  curl -L https   github com trentm json raw master lib json js  gt  json chmod  x json echo    quot hello quot    quot hi quot   quot there quot         json  quot hello hi quot

User · Answer

If you have php   php -r  var export json decode  curl http   twitter com users username json   1       For example  we have resource that provides json with countries iso codes  http   country io iso3 json and we can easily see it in a shell with curl   curl http   country io iso3 json   but it looks not very convenient  and not readable  better parse json and see readable structure   php -r  var export json decode  curl http   country io iso3 json   1       This code will print something like   array      BD    gt   BGD      BE    gt   BEL      BF    gt   BFA      BG    gt   BGR      BA    gt   BIH      BB    gt   BRB      WF    gt   WLF      BL    gt   BLM           if you have nested arrays this output will looks much better     Hope this will helpful

User · Answer

Using Node js  If the system has node installed  it s possible to use the -p print and -e evaulate script flags with JSON parse to pull out any value that is needed   A simple example using the JSON string    foo    bar    and pulling out the value of  foo      node -pe  JSON parse process argv 1   foo      foo    bar     bar   Because we have access to cat and other utilities  we can use this for files     node -pe  JSON parse process argv 1   foo     cat foobar json   bar   Or any other format such as an URL that contains JSON     node -pe  JSON parse process argv 1   name     curl -s https   api github com users trevorsenior   Trevor Senior

User · Answer

here s one way you can do it with awk   curl -sL  http   twitter com users username json    awk -F    -v k  text         gsub               for i 1 i lt  NF i             if    i   k                print  i

User · Answer

This is yet another bash  amp  python hybrid answer  I posted this answer because I wanted to process more complex JSON output  but  reducing the complexity of my bash application  I want to crack open the following JSON object from http   www arcgis com sharing rest info f json in bash       quot owningSystemUrl quot    quot http   www arcgis com quot      quot authInfo quot          quot tokenServicesUrl quot    quot https   www arcgis com sharing rest generateToken quot        quot isTokenBasedSecurity quot   true        In the following example  I created my own implementation of jq and unquote leveraging python  You ll note that once we import the python object from json to a python dictionary we can use python syntax to navigate the dictionary  To navigate the above  the syntax is   data data   quot authInfo quot    data   quot authInfo quot      quot tokenServicesUrl quot     By using magic in bash  we omit data and only supply the python text to the right of data  i e   jq jq     quot authInfo quot     jq     quot authInfo quot      quot tokenServicesUrl quot      Note  with no parameters  jq acts as a JSON prettifier  With parameters  we can use python syntax to extract anything we want from the dictionary including navigating subdictionaries and array elements  Here are the bash python hybrid functions     bin bash -xe  jq py       cat  lt  lt EOF import json  sys data   json load  sys stdin   print  json dumps  data 1  indent   4     EOF    jq       python -c  quot    jq py  quot  1 quot    quot     unquote py       cat  lt  lt EOF import json sys print  json load  sys stdin     EOF    unquote       python -c  quot    unquote py   quot     Here s a sample usage of the bash python functions  curl http   www arcgis com sharing rest info f json   tee arcgis json     quot owningSystemUrl quot   quot https   www arcgis com quot   quot authInfo quot    quot tokenServicesUrl quot   quot https   www arcgis com sharing rest generateToken quot   quot isTokenBasedSecurity quot  true    cat arcgis json   jq            quot owningSystemUrl quot    quot https   www arcgis com quot          quot authInfo quot                quot tokenServicesUrl quot    quot https   www arcgis com sharing rest generateToken quot              quot isTokenBasedSecurity quot   true              cat arcgis json   jq     quot authInfo quot                quot tokenServicesUrl quot    quot https   www arcgis com sharing rest generateToken quot          quot isTokenBasedSecurity quot   true      cat arcgis json   jq     quot authInfo quot      quot tokenServicesUrl quot        quot https   www arcgis com sharing rest generateToken quot   cat arcgis json   jq     quot authInfo quot      quot tokenServicesUrl quot       unquote   https   www arcgis com sharing rest generateToken

User · Answer

For more complex JSON parsing I suggest using python jsonpath module  by Stefan Goessner  -   Install it -   sudo easy install -U jsonpath   Use it -   Example file json  from http   goessner net articles JsonPath  -     store          book               category    reference            author    Nigel Rees            title    Sayings of the Century            price   8 95                   category    fiction            author    Evelyn Waugh            title    Sword of Honour            price   12 99                   category    fiction            author    Herman Melville            title    Moby Dick            isbn    0-553-21311-3            price   8 99                   category    fiction            author    J  R  R  Tolkien            title    The Lord of the Rings            isbn    0-395-19395-8            price   22 99                     bicycle            color    red          price   19 95               Parse it  extract all book titles with price  lt  10  -    cat file json   python -c  import sys  json  jsonpath  print   n  join jsonpath jsonpath json load sys stdin    store book     price  lt  10   title       Will output -  Sayings of the Century Moby Dick   NOTE  The above command line does not include error checking  for full solution with error checking you should create small python script  and wrap the code with try-except

User · Answer

Someone who also has xml files  might want to look at my Xidel  It is a cli  dependency-free JSONiq processor   i e  it also supports XQuery for xml or json processing   The example in the question would be    xidel -e  json  http   twitter com users username json    name      Or with my own  non standard extension syntax    xidel -e  json  http   twitter com users username json   name

User · Answer

Using Bash with Python  Create a bash function in your  bash rc file  function getJsonVal           python -c  import json sys sys stdout write json dumps json load sys stdin  1          Then    curl  http   twitter com users username json    getJsonVal    text    My status      Here is the same function  but with error checking   function getJsonVal        if         -ne 1    -o    -t 0       then        cat  lt  lt EOF Usage  getJsonVal  key   lt   tmp   -- or --   cat  tmp input   getJsonVal  key  EOF        return     fi     python -c  import json sys sys stdout write json dumps json load sys stdin  1         Where    -ne 1 makes sure at least 1 input  and -t 0 make sure you are redirecting from a pipe   The nice thing about this implementation is that you can access nested json values and get json in return      Example     echo    foo     bar    baz    a    1 2 3        getJsonVal    foo    a   1   2   If you want to be really fancy  you could pretty print the data   function getJsonVal           python -c  import json sys sys stdout write json dumps json load sys stdin  1  sort keys True  indent 4           echo    foo     bar    baz    a    1 2 3        getJsonVal    foo           a             1           2           3              bar    baz

User · Answer

Unfortunately the top voted answer that uses grep returns the full match that didn t work in my scenario  but if you know the JSON format will remain constant you can use lookbehind and lookahead to extract just the desired values     echo    TotalPages  33  FooBar   he  llo   anotherValue  100     grep -Po     lt   FooBar                 he  llo   echo    TotalPages  33  FooBar   he  llo   anotherValue  100     grep -Po     lt   TotalPages               33    echo    TotalPages  33  FooBar   he  llo   anotherValue  100     grep -Po     lt   anotherValue               100

User · Answer

Here is the answer for shell nerds using POSIX shell  with local  and egrep  JSON sh  4 7 KB    This thing has plenty of test cases  so it should be correct  It is also pipeable  It is used in the package manager for bash  bpkg

User · Answer

If someone just wants to extract values from simple JSON objects without the need for nested structures  it is possible to use regular expressions without even leaving the bash   Here is a function I defined using bash regular expressions based on the JSON standard   function json extract       local key  1   local json  2    local string regex                    local number regex  -  0  1-9  0-9       0-9      eE   -   0-9        local value regex    string regex    number regex  true false null    local pair regex      key      space        space       value regex       if      json       pair regex      then     echo   sed  s         g   lt  lt  lt     BASH REMATCH 1       else     return 1   fi     Caveats  objects and arrays are not supported as value  but all other value types defined in the standard are supported  Also  a pair will be matched no matter how deep in the JSON document it is as long as it has exactly the same key name   Using OP s example     json extract text    curl  http   twitter com users username json    My status    json extract friends count    curl  http   twitter com users username json    245

User · Answer

I needed something in BASH that s short and would run without dependencies beyond vanilla Linux LSB and Mac OS for both python 2 7  amp  3 and handle errors  e g  would report json parse errors and missing property errors without spewing python exceptions   json-extract        if      1             1      -h       1      -        1      --help       then     echo  Extract top level property value from json document      echo    Usage  json-extract  lt property gt     lt file-path gt         echo    Example 1  json-extract status  tmp response json      echo    Example 2  echo  JSON STRING   json-extract-file status      echo    Status codes  0 - success  1 - json parse error  2 - property missing    else     python -c   import sys  json  ntry  obj   json load open sys argv 2      nexcept  sys exit 1  ntry  print obj sys argv 1    nexcept  sys exit 2     1     2 - dev stdin     fi

User · Answer

You can use jshon   curl  http   twitter com users username json    jshon -e text

User · Answer

Use Python s JSON support instead of using awk  Something like this  curl -s http   twitter com users username json         python -c  quot import json sys obj json load sys stdin  print obj  name     quot

User · Answer

There is an easier way to get a property from a json string  Using a package json file as an example  try this      usr bin env bash my val    json    lt package json  node -pe  JSON parse process env json   version        We re using process env because this gets the file s contents into node js as a string without any risk of malicious contents escaping their quoting and being parsed as code

User · Answer

You can use bashJson  It   s a wrapper for the Python s JSON module and can handle complex JSON data   Let s consider this exmaple JSON data from the file test json         name   Test tool        author   hack4mer        supported os             osx                 foo   bar                min version    10 12               tested on     10 1 10 13                      ubuntu                 min version  14 04               tested on    16 04                     Following commands read data from this example JSON file    bashjson sh test json name   Prints  Test Tool    bashjson sh test json supported os osx foo   Prints  bar    bashjson sh test json supported os osx tested on   Prints   10 1 10 13

User · Answer

There is also a very simple but powerful JSON CLI processing tool fx     https   github com antonmedv fx    Examples  Use anonymous function     echo    key    value      fx  x   gt  x key  value   If you don t pass anonymous function param         code will be automatically transformed into anonymous function  And you can get access to JSON by this keyword     echo   1 2 3     fx  this map x   gt  x   2    2  4  6    Or just use dot syntax too     echo    items     one   1      fx  items one 1   You can pass any number of anonymous functions for reducing JSON     echo    items     one    two       fx  this items   this 1   two   You can update existing JSON using spread operator     echo    count   0     fx      this  count  1     count   1    Just plain JavaScript  Don t need to learn new syntax     UPDATE 2018-11-06  fx now has interactive mode        https   github com antonmedv fx

[json] Parsing JSON with Unix tools

Examples related to json

Examples related to bash

Examples related to parsing