[json] Read JSON data in a shell script

In shell I have a requirement wherein I have to read the JSON response which is in the following format:

 { "Messages": [ { "Body": "172.16.1.42|/home/480/1234/5-12-2013/1234.toSort", "ReceiptHandle": "uUk89DYFzt1VAHtMW2iz0VSiDcGHY+H6WtTgcTSgBiFbpFUg5lythf+wQdWluzCoBziie8BiS2GFQVoRjQQfOx3R5jUASxDz7SmoCI5bNPJkWqU8ola+OYBIYNuCP1fYweKl1BOFUF+o2g7xLSIEkrdvLDAhYvHzfPb4QNgOSuN1JGG1GcZehvW3Q/9jq3vjYVIFz3Ho7blCUuWYhGFrpsBn5HWoRYE5VF5Bxc/zO6dPT0n4wRAd3hUEqF3WWeTMlWyTJp1KoMyX7Z8IXH4hKURGjdBQ0PwlSDF2cBYkBUA=", "MD5OfBody": "53e90dc3fa8afa3452c671080569642e", "MessageId": "e93e9238-f9f8-4bf4-bf5b-9a0cae8a0ebc" } ] }

Here I am only concerned with the "Body" property value. I made some unsuccessful attempts like:

 jsawk -a 'return this.Body' 

or

 awk -v k="Body" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]} 

But that did not suffice. Can anyone help me with this?

This question is related to json bash shell

The answer is


Here is a crude way to do it: Transform JSON into bash variables to eval them.

This only works for:

  • JSON which does not contain nested arrays, and
  • JSON from trustworthy sources (else it may confuse your shell script, perhaps it may even be able to harm your system, You have been warned)

Well, yes, it uses PERL to do this job, thanks to CPAN, but is small enough for inclusion directly into a script and hence is quick and easy to debug:

json2bash() {
perl -MJSON -0777 -n -E 'sub J {
my ($p,$v) = @_; my $r = ref $v;
if ($r eq "HASH") { J("${p}_$_", $v->{$_}) for keys %$v; }
elsif ($r eq "ARRAY") { $n = 0; J("$p"."[".$n++."]", $_) foreach @$v; }
else { $v =~ '"s/'/'\\\\''/g"'; $p =~ s/^([^[]*)\[([0-9]*)\](.+)$/$1$3\[$2\]/;
$p =~ tr/-/_/; $p =~ tr/A-Za-z0-9_[]//cd; say "$p='\''$v'\'';"; }
}; J("json", decode_json($_));'
}

use it like eval "$(json2bash <<<'{"a":["b","c"]}')"

Not heavily tested, though. Updates, warnings and more examples see my GIST.

Update

(Unfortunately, following is a link-only-solution, as the C code is far too long to duplicate here.)

For all those, who do not like the above solution, there now is a C program json2sh which (hopefully safely) converts JSON into shell variables. In contrast to the perl snippet, it is able to process any JSON, as long as it is well formed.

Caveats:

  • json2sh was not tested much.
  • json2sh may create variables, which start with the shellshock pattern () {

I wrote json2sh to be able to post-process .bson with Shell:

bson2json()
{
printf '[';
{ bsondump "$1"; echo "\"END$?\""; } | sed '/^{/s/$/,/';
echo ']';
};

bsons2json()
{
printf '{';
c='';
for a;
do
  printf '%s"%q":' "$c" "$a";
  c=',';
  bson2json "$a";
done;
echo '}';
};

bsons2json */*.bson | json2sh | ..

Explained:

  • bson2json dumps a .bson file such, that the records become a JSON array
    • If everything works OK, an END0-Marker is applied, else you will see something like END1.
    • The END-Marker is needed, else empty .bson files would not show up.
  • bsons2json dumps a bunch of .bson files as an object, where the output of bson2json is indexed by the filename.

This then is postprocessed by json2sh, such that you can use grep/source/eval/etc. what you need, to bring the values into the shell.

This way you can quickly process the contents of a MongoDB dump on shell level, without need to import it into MongoDB first.


tl;dr

$ cat /tmp/so.json | underscore select '.Messages .Body' 
["172.16.1.42|/home/480/1234/5-12-2013/1234.toSort"]

Javascript CLI tools

You can use Javascript CLI tools like

Example

Select all name children of a addons:

underscore select ".addons > .name"

The underscore-cli provide others real world examples as well as the json:select() doc.


Similarly using Bash regexp. Shall be able to snatch any key/value pair.

key="Body"
re="\"($key)\": \"([^\"]*)\""

while read -r l; do
    if [[ $l =~ $re ]]; then
        name="${BASH_REMATCH[1]}"
        value="${BASH_REMATCH[2]}"
        echo "$name=$value"
    else
        echo "No match"
    fi
done

Regular expression can be tuned to match multiple spaces/tabs or newline(s). Wouldn't work if value has embedded ". This is an illustration. Better to use some "industrial" parser :)


Examples related to json

Use NSInteger as array index Uncaught SyntaxError: Unexpected end of JSON input at JSON.parse (<anonymous>) HTTP POST with Json on Body - Flutter/Dart Importing json file in TypeScript json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 190) Angular 5 Service to read local .json file How to import JSON File into a TypeScript file? Use Async/Await with Axios in React.js Uncaught SyntaxError: Unexpected token u in JSON at position 0 how to remove json object key and value.?

Examples related to bash

Comparing a variable with a string python not working when redirecting from bash script Zipping a file in bash fails How do I prevent Conda from activating the base environment by default? Get first line of a shell command's output Fixing a systemd service 203/EXEC failure (no such file or directory) /bin/sh: apt-get: not found VSCode Change Default Terminal Run bash command on jenkins pipeline How to check if the docker engine and a docker container are running? How to switch Python versions in Terminal?

Examples related to shell

Comparing a variable with a string python not working when redirecting from bash script Get first line of a shell command's output How to run shell script file using nodejs? Run bash command on jenkins pipeline Way to create multiline comments in Bash? How to do multiline shell script in Ansible How to check if a file exists in a shell script How to check if an environment variable exists and get its value? Curl to return http status code along with the response docker entrypoint running bash script gets "permission denied"