By auser


2009-12-23 21:46:58 8 Comments

I'm trying to parse JSON returned from a curl request, like so:

curl 'http://twitter.com/users/username.json' |
    sed -e 's/[{}]/''/g' | 
    awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}'

The above splits the JSON into fields, for example:

% ...
"geo_enabled":false
"friends_count":245
"profile_text_color":"000000"
"status":"in_reply_to_screen_name":null
"source":"web"
"truncated":false
"text":"My status"
"favorited":false
% ...

How do I print a specific field (denoted by the -v k=text)?

30 comments

@martinr 2009-12-23 22:28:45

Use Python's JSON support instead of using awk!

Something like this:

curl -s http://twitter.com/users/username.json | \
    python -c "import json,sys;obj=json.load(sys.stdin);print(obj['name']);"

@martinr 2009-12-23 22:45:43

Pardon me for trying to come up with a good response...: I shall try harder. Partisanship requires more than writing an awk script to shake it off!

@m3nda 2016-02-15 07:23:51

Why do you use the obj variable in that oneliner solution?. It's useless and is not stored anyway at all? You write less using json.load(sys.stdin)['"key']" as example like: curl -sL httpbin.org/ip | python -c "import json,sys; print json.load(sys.stdin)['origin']".

@Liu Hao 2020-07-10 13:21:12

Here is a simple approach for Node.js-ready environment:

curl -L https://github.com/trentm/json/raw/master/lib/json.js > json
chmod +x json
echo '{"hello":{"hi":"there"}}' | ./json "hello.hi"

@Mingye Wang 2020-05-04 18:07:28

Here is the answer for shell nerds using POSIX shell (with local) and egrep: JSON.sh, 4.7 KB.

This thing has plenty of test cases, so it should be correct. It is also pipeable. It is used in the package manager for bash, bpkg.

@Vanquished Wombat 2020-05-04 18:13:25

+1 for returning to an old question with new info. You might want to add your answer in as a comment against the accepted answer otherwise being down here no-one will notice it maybe.

@Brian Campbell 2009-12-23 21:59:30

There are a number of tools specifically designed for the purpose of manipulating JSON from the command line, and will be a lot easier and more reliable than doing it with Awk, such as jq:

curl -s 'https://api.github.com/users/lambda' | jq -r '.name'

You can also do this with tools that are likely already installed on your system, like Python using the json module, and so avoid any extra dependencies, while still having the benefit of a proper JSON parser. The following assume you want to use UTF-8, which the original JSON should be encoded in and is what most modern terminals use as well:

Python 3:

curl -s 'https://api.github.com/users/lambda' | \
    python3 -c "import sys, json; print(json.load(sys.stdin)['name'])"

Python 2:

export PYTHONIOENCODING=utf8
curl -s 'https://api.github.com/users/lambda' | \
    python2 -c "import sys, json; print json.load(sys.stdin)['name']"

Historical notes

This answer originally recommended jsawk, which should still work, but is a little more cumbersome to use than jq, and depends on a standalone JavaScript interpreter being installed which is less common than a Python interpreter, so the above answers are probably preferable:

curl -s 'https://api.github.com/users/lambda' | jsawk -a 'return this.name'

This answer also originally used the Twitter API from the question, but that API no longer works, making it hard to copy the examples to test out, and the new Twitter API requires API keys, so I've switched to using the GitHub API which can be used easily without API keys. The first answer for the original question would be:

curl 'http://twitter.com/users/username.json' | jq -r '.text'

@Szymon Sadło 2016-06-17 09:51:43

@thrau +1. jq it is available in the repository and super easy to use so it's much better than jsawk. I tested both for a few minutes, jq won this battle

@Martijn Pieters 2016-09-09 11:28:35

Note that in Python 2, if you are piping the output to another command then the print statement will always encode to ASCII because you are using Python in a pipe. Insert PYTHONIOENCODING=<desired codec> into the command to set a different output encoding, suitable for your terminal. In Python 3, the default is UTF-8 in this case (using the print() function).

@Andy Fraley 2018-04-20 14:56:35

Install jq on OSX with brew install jq

@Serge Stroobandt 2018-10-26 21:52:45

curl -s is equivalent to curl --silent, whereas jq -r means jq --raw-output i.e. without string quotes.

@NotTooTechy 2020-05-15 14:28:57

python -c "import requests;r=requests.get('api.github.com/users/lambda');print r.json()['name'];" . The simpliest!

@Bernie Reiter 2020-07-07 15:34:48

@The simpliest! I have just tried your suggestion: $ python -c "import requests;r=requests.get('api.github.com/users/lambda');print r.json()['name'];" and received this error message: Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named requests

@Anand Singh 2018-08-13 12:00:28

You can use bashJson

It’s a wrapper for the Python's JSON module and can handle complex JSON data.

Let's consider this exmaple JSON data from the file test.json

{
    "name":"Test tool",
    "author":"hack4mer",
    "supported_os":{
        "osx":{
            "foo":"bar",
            "min_version" : 10.12,
            "tested_on" : [10.1,10.13]
        },
        "ubuntu":{
            "min_version":14.04,
            "tested_on" : 16.04
        }
    }
}

Following commands read data from this example JSON file

./bashjson.sh test.json name

Prints: Test Tool

./bashjson.sh test.json supported_os osx foo

Prints: bar

./bashjson.sh test.json supported_os osx tested_on

Prints: [10.1,10.13]

@Stephen Quan 2017-04-26 07:30:23

This is yet another bash & python hybrid answer. I posted this answer because I wanted to process more complex JSON output, but, reducing the complexity of my bash application. I want to crack open the following JSON object from http://www.arcgis.com/sharing/rest/info?f=json in bash:

{
  "owningSystemUrl": "http://www.arcgis.com",
  "authInfo": {
    "tokenServicesUrl": "https://www.arcgis.com/sharing/rest/generateToken",
    "isTokenBasedSecurity": true
  }
}

In the following example, I created my own implementation of jq and unquote leveraging python. You'll note that once we import the python object from json to a python dictionary we can use python syntax to navigate the dictionary. To navigate the above, the syntax is:

  • data
  • data[ "authInfo" ]
  • data[ "authInfo" ][ "tokenServicesUrl" ]

By using magic in bash, we omit data and only supply the python text to the right of data, i.e.

  • jq
  • jq '[ "authInfo" ]'
  • jq '[ "authInfo" ][ "tokenServicesUrl" ]'

Note, with no parameters, jq acts as a JSON prettifier. With parameters we can use python syntax to extract anything we want from the dictionary including navigating subdictionaries and array elements.

Here's a working example that demonstrates the above:

jq_py() {
cat <<EOF
import json, sys
data = json.load( sys.stdin )
print( json.dumps( data$1, indent = 4 ) )
EOF
}

jq() {
  python -c "$( jq_py "$1" )"
}

unquote_py() {
cat <<EOF
import json,sys
print( json.load( sys.stdin ) )
EOF
}

unquote() {
  python -c "$( unquote_py )"
}

curl http://www.arcgis.com/sharing/rest/info?f=json | tee arcgis.json
# {"owningSystemUrl":"https://www.arcgis.com","authInfo":{"tokenServicesUrl":"https://www.arcgis.com/sharing/rest/generateToken","isTokenBasedSecurity":true}}

cat arcgis.json | jq
# {
#     "owningSystemUrl": "https://www.arcgis.com",
#     "authInfo": {
#         "tokenServicesUrl": "https://www.arcgis.com/sharing/rest/generateToken",
#         "isTokenBasedSecurity": true
#     }
# }

cat arcgis.json | jq '[ "authInfo" ]'
# {
#     "tokenServicesUrl": "https://www.arcgis.com/sharing/rest/generateToken",
#     "isTokenBasedSecurity": true
# }

cat arcgis.json | jq '[ "authInfo" ][ "tokenServicesUrl" ]'
# "https://www.arcgis.com/sharing/rest/generateToken"

cat arcgis.json | jq '[ "authInfo" ][ "tokenServicesUrl" ]' | unquote
# https://www.arcgis.com/sharing/rest/generateToken

@Alexander Mills 2017-11-11 06:09:57

There is an easier way to get a property from a json string. Using a package.json file as an example, try this:

#!/usr/bin/env bash
my_val="$(json=$(<package.json) node -pe "JSON.parse(process.env.json)['version']")"

We're using process.env because this gets the file's contents into node.js as a string without any risk of malicious contents escaping their quoting and being parsed as code.

@Charles Duffy 2018-07-12 19:18:48

Using string concatenation to substitute values into into a string parsed as code allows arbitrary node.js code to be run, meaning it's exceedingly unsafe to use with random content you got off the Internet. There's a reason safe/best-practice ways to parse JSON in JavaScript don't just evaluate it.

@Alexander Mills 2019-04-19 17:57:13

@CharlesDuffy not sure I follow but the JSON.parse call should be safer, as require() can actually run foreign code, JSON.parse can't.

@Charles Duffy 2019-04-19 18:32:02

That's true if-and-only-if your string is actually injected into the JSON runtime in such a way as to bypass the parser. I don't see the code here doing that reliably. Pull it from an environment variable and pass it to JSON.parse() and yes, you're unambiguously safe... but here, the JSON runtime is receiving the (untrusted) content in-band with the (trusted) code.

@Charles Duffy 2019-04-19 18:34:37

...similarly, if you have your code read the JSON from file as a string and pass that string to JSON.parse(), you're safe then too, but that's not happening here either.

@Charles Duffy 2019-04-19 18:40:42

To give you a concrete example, run printf '`+require("child_process").exec("touch owned")+`' >package.json before your first example. Sure, the code throws an error (because I didn't take the time to make it not), but it also runs touch owned.

@Alexander Mills 2019-04-19 20:22:06

I still don't follow, JSON.parse will never execute code, it can only return a string. If you have control over whether the result of the JSON.parse gets run, then it's safe. Otoh, require(x) can run foreign code.

@Charles Duffy 2019-04-19 20:25:03

Did you actually run the example I gave and look at whether a file named owned exists afterwards? Once it's proved that the exploit works, then we can go into how.

@Charles Duffy 2019-04-19 20:27:55

...ahh, heck, might as well go into the "how" immediately. The problem is that you're substituting the shell variable, which you intend to be passed to JSON.parse(), into the code. You're assuming that putting literal backticks will keep the contents literal, but that's a completely unsafe assumption, because literal backticks can exist in the file content (and thus the variable), and thus can terminate the quoting and enter an unquoted context where the values are executed as code.

@Alexander Mills 2019-04-19 20:32:15

Ok so you're saying because of the backticks it might execute some bash code, so maybe what you're saying is if you want to pass a dynamic string to JSON.parse use something other than backticks? If you have an improvement for the first example definitely lmk b/c it's not that pretty as it is.

@Charles Duffy 2019-04-19 20:32:28

@Ugur Kazdal 2019-10-28 17:18:40

i used it like node -pe "var config = require('./output.json'); console.log(config.response.docs)" #so thank you.

@Mike 2019-03-25 14:37:07

I needed something in BASH that's short and would run without dependencies beyond vanilla Linux LSB and Mac OS for both python 2.7 & 3 and handle errors, e.g. would report json parse errors and missing property errors without spewing python exceptions:

json-extract () {
  if [[ "$1" == "" || "$1" == "-h" || "$1" == "-?" || "$1" == "--help" ]] ; then
    echo 'Extract top level property value from json document'
    echo '  Usage: json-extract <property> [ <file-path> ]'
    echo '  Example 1: json-extract status /tmp/response.json'
    echo '  Example 2: echo $JSON_STRING | json-extract-file status'
    echo '  Status codes: 0 - success, 1 - json parse error, 2 - property missing'
  else
    python -c $'import sys, json;\ntry: obj = json.load(open(sys.argv[2])); \nexcept: sys.exit(1)\ntry: print(obj[sys.argv[1]])\nexcept: sys.exit(2)' "$1" "${2:-/dev/stdin}"
  fi
}

@paulkmoore 2011-12-06 13:05:53

On the basis that some of the recommendations here (esp in the comments) suggested the use of Python, I was disappointed not to find an example.

So, here's a one liner to get a single value from some JSON data. It assumes that you are piping the data in (from somewhere) and so should be useful in a scripting context.

echo '{"hostname":"test","domainname":"example.com"}' | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["hostname"]'

@Joe Heyming 2014-04-11 22:06:53

I enhanced this answer below to use a bash function: curl 'some_api' | getJsonVal 'key'

@RussellStewart 2014-09-14 20:18:51

pythonpy (github.com/russell91/pythonpy is almost always a better alternative to python -c, although it does have to be installed with pip. just pipe the json to py --ji -x 'x[0]["hostname"]'. If you didn't want to use the built in json_input support, you could still get those import automatically as py 'json.loads(sys.stdin)[0]["hostname"]'

@akavel 2015-03-23 13:05:17

Thanks! For more quick&dirty JSON parsing I've wrapped it into a bash function: jsonq() { python -c "import sys,json; obj=json.load(sys.stdin); print($1)"; } so that I could write: curl ...... | jsonq 'json.dumps([key["token"] for key in obj], indent=2)' & more of similar scary stuff... Btw, obj[0] seems unnecessary, it looks like just obj works OK in default cases (?).

@Adam K Dean 2016-03-01 17:58:36

Thanks. I've made this respect JSON a bit better than print: jsonq() { python -c "import sys,json; obj=json.load(sys.stdin); sys.stdout.write(json.dumps($1))"; }

@CyberEd 2016-08-16 20:49:26

obj[0] causes an error when parsing { "port":5555 }. Works fine after removing [0].

@Anton Medvedev 2018-01-25 17:51:04

There is also a very simple but powerful JSON CLI processing tool fxhttps://github.com/antonmedv/fx

Example of JSON formatting in Bash terminal

Examples

Use anonymous function:

$ echo '{"key": "value"}' | fx "x => x.key"
value

If you don't pass anonymous function param => ..., code will be automatically transformed into anonymous function. And you can get access to JSON by this keyword:

$ echo '[1,2,3]' | fx "this.map(x => x * 2)"
[2, 4, 6]

Or just use dot syntax too:

$ echo '{"items": {"one": 1}}' | fx .items.one
1

You can pass any number of anonymous functions for reducing JSON:

$ echo '{"items": ["one", "two"]}' | fx "this.items" "this[1]"
two

You can update existing JSON using spread operator:

$ echo '{"count": 0}' | fx "{...this, count: 1}"
{"count": 1}

Just plain JavaScript. Don't need to learn new syntax.


UPDATE 2018-11-06

fx now has interactive mode (!)

https://github.com/antonmedv/fx

@tripleee 2018-10-29 08:25:30

If you are promoting your own creation, you need to be explicit about it. See How not to be a spammer.

@Pila 2018-10-18 13:21:30

I can not use any of the answers here. No available jq, no shell arrays, no declare, no grep -P, no lookbehind and lookahead, no Python, no Perl, no Ruby, no - not even Bash... Remaining answers simply do not work well. JavaScript sounded familiar, but the tin says Nescaffe - so it is a no go, too :) Even if available, for my simple need - they would be overkill and slow.

Yet, it is extremely important for me to get many variables from the json formatted reply of my modem. I am doing it in a sh with very trimmed down BusyBox at my routers! No problems using awk alone: just set delimiters and read the data. For a single variable, that is all!

awk 'BEGIN { FS="\""; RS="," }; { if ($2 == "login") {print $4} }' test.json

Remember I have no arrays? I had to assign within the awk parsed data to the 11 variables which I need in a shell script. Wherever I looked, that was said to be an impossible mission. No problem with that, too.

My solution is simple. This code will: 1) parse .json file from the question (actually, I have borrowed a working data sample from the most upvoted answer) and pick out the quoted data, plus 2) create shell variables from within the awk assigning free named shell variable names.

eval $( curl -s 'https://api.github.com/users/lambda' | 
awk ' BEGIN { FS="\""; RS="," };
{
    if ($2 == "login") { print "Login=\""$4"\"" }
    if ($2 == "name") { print "Name=\""$4"\"" }
    if ($2 == "updated_at") { print "Updated=\""$4"\"" }
}' )
echo "$Login, $Name, $Updated"

No problems with blanks within. In my use, the same command parses a long single line output. As eval is used, this solution is suited for trusted data only. It is simple to adapt it to pickup unquoted data. For huge number of variables, marginal speed gain can be achieved using else if. Lack of array obviously means: no multiple records without extra fiddling. But where arrays are available, adapting this solution is a simple task.

@maikel sed answer almost works (but I can not comment on it). For my nicely formatted data - it works. Not so much with the example used here (missing quotes throw it off). It is complicated and difficult to modify. Plus, I do not like having to make 11 calls to extract 11 variables. Why? I timed 100 loops extracting 9 variables: the sed function took 48.99 sec and my solution took 0.91 sec! Not fair? Doing just a single extraction of 9 variables: 0.51 vs. 0.02 sec.

@Ehsan Chavoshi 2016-06-22 08:50:41

i used this to extract video duration from ffprobe json output :

MOVIE_INFO=`ffprobe "path/to/movie.mp4"  -show_streams -show_format -print_format json -v quiet` 
MOVIE_SECONDS=`echo "$MOVIE_INFO"|grep -w \"duration\" |tail -1 | cut -d\" -f4 |cut -d \. -f 1`

it can be used to extract value from any json :

value=`echo "$jsondata"|grep -w \"key_name\" |tail -1 | cut -d\" -f4

@tripleee 2018-06-18 12:55:48

If this is supposed to be valid shell script, the spaces around the equals sign in the last fragment are a syntax error.

@Ehsan Chavoshi 2018-06-19 13:22:01

@tripleee Corrected. Tnx.

@Herve 2018-02-12 15:37:32

Niet is a tool that help you to extract data from json or yaml file directly in your shell/bash CLI.

$ pip install niet

Consider a json file named project.json with the following contents:

{
  project: {
    meta: {
      name: project-sample
    }
}

You can use niet like this:

$ PROJECT_NAME=$(niet project.json project.meta.name)
$ echo ${PROJECT_NAME}
project-sample

@Helder Pereira 2017-09-20 14:33:32

If someone just wants to extract values from simple JSON objects without the need for nested structures, it is possible to use regular expressions without even leaving the bash.

Here is a function I defined using bash regular expressions based on the JSON standard:

function json_extract() {
  local key=$1
  local json=$2

  local string_regex='"([^"\]|\\.)*"'
  local number_regex='-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?'
  local value_regex="${string_regex}|${number_regex}|true|false|null"
  local pair_regex="\"${key}\"[[:space:]]*:[[:space:]]*(${value_regex})"

  if [[ ${json} =~ ${pair_regex} ]]; then
    echo $(sed 's/^"\|"$//g' <<< "${BASH_REMATCH[1]}")
  else
    return 1
  fi
}

Caveats: objects and arrays are not supported as value, but all other value types defined in the standard are supported. Also, a pair will be matched no matter how deep in the JSON document it is as long as it has exactly the same key name.

Using OP's example:

$ json_extract text "$(curl 'http://twitter.com/users/username.json')"
My status

$ json_extract friends_count "$(curl 'http://twitter.com/users/username.json')"
245

@vsbehere 2020-01-08 13:10:46

Helder Pereira can we extract nested property values with this function?

@user2233949 2017-03-20 12:31:52

Now that Powershell is cross platform, I thought I'd throw its way out there, since I find it to be fairly intuitive and extremely simple.

curl -s 'https://api.github.com/users/lambda' | ConvertFrom-Json 

ConvertFrom-Json converts the JSON into a Powershell custom object, so you can easily work with the properties from that point forward. If you only wanted the 'id' property for example, you'd just do this:

curl -s 'https://api.github.com/users/lambda' | ConvertFrom-Json | select -ExpandProperty id

If you wanted to invoke the whole thing from within Bash, then you'd have to call it like this:

powershell 'curl -s "https://api.github.com/users/lambda" | ConvertFrom-Json'

Of course there's a pure Powershell way to do it without curl, which would be:

Invoke-WebRequest 'https://api.github.com/users/lambda' | select -ExpandProperty Content | ConvertFrom-Json

Finally, there's also 'ConvertTo-Json' which converts a custom object to JSON just as easily. Here's an example:

(New-Object PsObject -Property @{ Name = "Tester"; SomeList = @('one','two','three')}) | ConvertTo-Json

Which would produce nice JSON like this:

{
"Name":  "Tester",
"SomeList":  [
                 "one",
                 "two",
                 "three"
             ]

}

Admittedly, using a Windows shell on Unix is somewhat sacrilegious but Powershell is really good at some things, and parsing JSON and XML are a couple of them. This the GitHub page for the cross platform version https://github.com/PowerShell/PowerShell

@MartinThé 2018-11-19 15:05:33

I used to dislike PowerShell, but I must admit the handling of JSON as objects is pretty nice.

@Daniel Sokolowski 2016-11-20 20:03:45

Unfortunately the top voted answer that uses grep returns the full match that didn't work in my scenario, but if you know the JSON format will remain constant you can use lookbehind and lookahead to extract just the desired values.

# echo '{"TotalPages":33,"FooBar":"he\"llo","anotherValue":100}' | grep -Po '(?<="FooBar":")(.*?)(?=",)'
he\"llo
# echo '{"TotalPages":33,"FooBar":"he\"llo","anotherValue":100}' | grep -Po '(?<="TotalPages":)(.*?)(?=,)'
33
#  echo '{"TotalPages":33,"FooBar":"he\"llo","anotherValue":100}' | grep -Po '(?<="anotherValue":)(.*?)(?=})'
100

@tripleee 2018-06-18 12:54:45

You never actually know the order of elements in a JSON dictionary. They are, by definition, unordered. This is precisely one of the fundamental reasons why rolling your own JSON parser is a doomed approach.

@Alexey Dubkov 2016-08-27 04:37:44

If pip is avaiable on the system then:

$ pip install json-query

Examples of usage:

$ curl -s http://0/file.json | json-query
{
    "key":"value"    
}

$ curl -s http://0/file.json | json-query my.key
value

$ curl -s http://0/file.json | json-query my.keys.
key_1
key_2
key_3

$ curl -s http://0/file.json | json-query my.keys.2
value_2

@Pruthvi Chitrala 2017-08-25 18:39:50

How to find json-array length usage json-query package?

@jnrg 2011-06-12 08:04:15

Following MartinR and Boecko's lead:

$ curl -s 'http://twitter.com/users/username.json' | python -mjson.tool

That will give you an extremely grep friendly output. Very convenient:

$ curl -s 'http://twitter.com/users/username.json' | python -mjson.tool | grep my_key

@juan 2013-03-28 13:58:28

How would you extract a specific key, as OP is asking?

@Andrea Richiardi 2013-05-12 04:31:55

Best answer so far imho, no need to install anything else on most distros and you can | grep field. Thanks!

@Cheeso 2014-06-04 00:42:37

All this does is format the JSON, if I'm not mistaken. It does not allow the caller to select a particular field from the output, as would an xpath solution, or something based on "JSON Pointer".

@RussellStewart 2014-09-14 20:14:54

or you could use pythonpy: curl 'http://twitter.com/users/username.json' | py --json_input -x 'x.name'

@christopher 2017-12-28 10:15:11

I just end up with a key value pair, but not the value in and of itself.

@tripleee 2018-06-18 12:57:21

The grep without anchors will pick up any line which contains the field name. Guess what happens if the field is called something very common and short like id. (You can fix that with grep "\"id\":" but that still gives you the entire line, not just the value, or just the name of the field if its value is another complex structure.)

@jnrg 2018-08-01 23:27:42

Guys it's 2018, ignore my comment , use jq for sure ;-)

@CpILL 2018-09-03 13:10:08

jq is not typically installed while python is. Also, once your in Python you might as well go the whole way and parse it with import json...

@Joe Heyming 2014-02-06 05:53:32

Using Bash with Python

Create a bash function in your .bash_rc file

function getJsonVal () { 
    python -c "import json,sys;sys.stdout.write(json.dumps(json.load(sys.stdin)$1))"; 
}

Then

$ curl 'http://twitter.com/users/username.json' | getJsonVal "['text']"
My status
$ 

Here is the same function, but with error checking.

function getJsonVal() {
   if [ \( $# -ne 1 \) -o \( -t 0 \) ]; then
       cat <<EOF
Usage: getJsonVal 'key' < /tmp/
 -- or -- 
 cat /tmp/input | getJsonVal 'key'
EOF
       return;
   fi;
   python -c "import json,sys;sys.stdout.write(json.dumps(json.load(sys.stdin)$1))";
}

Where $# -ne 1 makes sure at least 1 input, and -t 0 make sure you are redirecting from a pipe.

The nice thing about this implementation is that you can access nested json values and get json in return! =)

Example:

$ echo '{"foo": {"bar": "baz", "a": [1,2,3]}}' |  getJsonVal "['foo']['a'][1]"
2

If you want to be really fancy, you could pretty print the data:

function getJsonVal () { 
    python -c "import json,sys;sys.stdout.write(json.dumps(json.load(sys.stdin)$1, sort_keys=True, indent=4))"; 
}

$ echo '{"foo": {"bar": "baz", "a": [1,2,3]}}' |  getJsonVal "['foo']"
{
    "a": [
        1, 
        2, 
        3
    ], 
    "bar": "baz"
}

@Cheeso 2014-06-04 00:53:10

One-liner without the bash function: curl http://foo | python -c 'import json,sys;obj=json.load(sys.stdin);print obj["environment"][0]["name"]'

@Per Johansson 2014-06-27 09:17:55

sys.stdout.write() if you want it to work with both python 2 and 3.

@Joe Heyming 2014-07-07 20:14:04

I'm thinking that it should change to system.stdout.write(obj$1). That way you can say: getJsonVal "['environment']['name']", like @Cheeso 's example

@Joe Heyming 2016-09-22 22:57:27

@Narek In that case, it would look like this: function getJsonVal() { py -x "json.dumps(json.loads(x)$1, sort_keys=True, indent=4)"; }

@boecko 2011-04-27 09:46:51

Version which uses Ruby and http://flori.github.com/json/

$ < file.json ruby -e "require 'rubygems'; require 'json'; puts JSON.pretty_generate(JSON[STDIN.read]);"

or more concisely:

$ < file.json ruby -r rubygems -r json -e "puts JSON.pretty_generate(JSON[STDIN.read]);"

@lucapette 2011-05-04 10:57:29

this is my favourite ;) BTW you can short it with ruby -rjson to require the library

@Zack Morris 2018-08-15 20:50:02

Note that the final ; is not required in Ruby (it's only used for concatenating statements that would normally be on separate lines into a single line).

@V. K. 2015-11-18 14:24:07

If you have php:

php -r 'var_export(json_decode(`curl http://twitter.com/users/username.json`, 1));'

For example:
we have resource that provides json with countries iso codes: http://country.io/iso3.json and we can easily see it in a shell with curl:

curl http://country.io/iso3.json

but it looks not very convenient, and not readable, better parse json and see readable structure:

php -r 'var_export(json_decode(`curl http://country.io/iso3.json`, 1));'

This code will print something like:

array (
  'BD' => 'BGD',
  'BE' => 'BEL',
  'BF' => 'BFA',
  'BG' => 'BGR',
  'BA' => 'BIH',
  'BB' => 'BRB',
  'WF' => 'WLF',
  'BL' => 'BLM',
  ...

if you have nested arrays this output will looks much better...

Hope this will helpful...

@Max Barrass 2015-05-25 03:45:50

Here is a good reference. In this case:

curl 'http://twitter.com/users/username.json' | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) { where = match(a[i], /\"text\"/); if(where) {print a[i]} }  }'

@Viktova 2018-03-19 12:18:46

this answer should get the highest vote, most if not all of the other answers are package(php, python, etc..) dependent.

@tripleee 2018-06-18 13:00:58

No, on the contrary, anything with a useless use of sed should not receive any more upvotes.

@Max Barrass 2018-06-21 04:47:45

SecKarma, Exactly! topic said UNIX tools right? tripleee, got any ON TOPIC sample code for us to review?

@RussellStewart 2014-09-14 20:12:29

This is a good usecase for pythonpy:

curl 'http://twitter.com/users/username.json' | py 'json.load(sys.stdin)["name"]'

@m3nda 2016-02-15 07:25:59

Even shorter, python -c module here :) nice.

@maikel 2014-10-30 14:38:52

Native Bash version: Also works well with backslashes (\) and quotes (")

function parse_json()
{
    echo $1 | \
    sed -e 's/[{}]/''/g' | \
    sed -e 's/", "/'\",\"'/g' | \
    sed -e 's/" ,"/'\",\"'/g' | \
    sed -e 's/" , "/'\",\"'/g' | \
    sed -e 's/","/'\"---SEPERATOR---\"'/g' | \
    awk -F=':' -v RS='---SEPERATOR---' "\$1~/\"$2\"/ {print}" | \
    sed -e "s/\"$2\"://" | \
    tr -d "\n\t" | \
    sed -e 's/\\"/"/g' | \
    sed -e 's/\\\\/\\/g' | \
    sed -e 's/^[ \t]*//g' | \
    sed -e 's/^"//'  -e 's/"$//'
}


parse_json '{"username":"john, doe","email":"[email protected]"}' username
parse_json '{"username":"john doe","email":"[email protected]"}' email

--- outputs ---

john, doe
[email protected]

@rtc11 2016-04-06 12:03:38

This is awesome. But if the JSON string contains more than one email key, the parser will output [email protected]""[email protected]

@alexmngn 2019-03-15 14:53:40

Doesn't work if there's a dash in the email like [email protected]

@Adam Kurkiewicz 2014-12-04 17:47:39

A two-liner which uses python. It works particularly well if you're writing a single .sh file and you don't want to depend on another .py file. It also leverages the usage of pipe |. echo "{\"field\": \"value\"}" can be replaced by anything printing a json to the stdout.

echo "{\"field\": \"value\"}" | python -c 'import sys, json
print(json.load(sys.stdin)["field"])'

@Andrew Barber 2014-12-04 19:38:46

The question was not looking for a Python solution. See the comments, too.

@coolaj86 2011-12-10 03:49:32

TickTick is a JSON parser written in bash (<250 lines of code)

Here's the author's snippit from his article, Imagine a world where Bash supports JSON:

#!/bin/bash
. ticktick.sh

``  
  people = { 
    "Writers": [
      "Rod Serling",
      "Charles Beaumont",
      "Richard Matheson"
    ],  
    "Cast": {
      "Rod Serling": { "Episodes": 156 },
      "Martin Landau": { "Episodes": 2 },
      "William Shatner": { "Episodes": 2 } 
    }   
  }   
``  

function printDirectors() {
  echo "  The ``people.Directors.length()`` Directors are:"

  for director in ``people.Directors.items()``; do
    printf "    - %s\n" ${!director}
  done
}   

`` people.Directors = [ "John Brahm", "Douglas Heyes" ] ``
printDirectors

newDirector="Lamont Johnson"
`` people.Directors.push($newDirector) ``
printDirectors

echo "Shifted: "``people.Directors.shift()``
printDirectors

echo "Popped: "``people.Directors.pop()``
printDirectors

@Ed Randall 2018-09-13 12:59:56

As the only robust pure-bash answer on here, this deserves more upvotes.

@Thomas Fournet 2019-09-26 13:34:38

Is there any way to print this people variable into a json string again ? That would be extremely useful

@Akito 2020-04-10 15:18:56

Finally an answer not recommending Python or other atrocious methods... Thank you!

@shlomosh 2014-04-01 08:57:52

For more complex JSON parsing I suggest using python jsonpath module (by Stefan Goessner) -

  1. Install it -

sudo easy_install -U jsonpath

  1. Use it -

Example file.json (from http://goessner.net/articles/JsonPath) -

{ "store": {
    "book": [ 
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      { "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      { "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

Parse it (extract all book titles with price < 10) -

$ cat file.json | python -c "import sys, json, jsonpath; print '\n'.join(jsonpath.jsonpath(json.load(sys.stdin), 'store.book[?(@.price < 10)].title'))"

Will output -

Sayings of the Century
Moby Dick

NOTE: The above command line does not include error checking. for full solution with error checking you should create small python script, and wrap the code with try-except.

@Sridhar Sarnobat 2016-08-20 05:27:31

I was having a little trouble installing jsonpath so installed jsonpath_rw instead, so here is something similar you can try if the above doesn't work: 1) /usr/bin/python -m pip install jsonpath-rw 2) cat ~/trash/file.json | /usr/bin/python -c "from jsonpath_rw import jsonpath, parse; import sys,json; jsonpath_expr = parse('store.book[0]'); out = [match.value for match in jsonpath_expr.find(json.load(sys.stdin))]; print out;" (I used the full path to the python binary because I was having some issues with multiple pythons installed).

@Brendan OConnor 2011-07-27 23:24:46

To quickly extract the values for a particular key, I personally like to use "grep -o", which only returns the regex's match. For example, to get the "text" field from tweets, something like:

grep -Po '"text":.*?[^\\]",' tweets.json

This regex is more robust than you might think; for example, it deals fine with strings having embedded commas and escaped quotes inside them. I think with a little more work you could make one that is actually guaranteed to extract the value, if it's atomic. (If it has nesting, then a regex can't do it of course.)

And to further clean (albeit keeping the string's original escaping) you can use something like: | perl -pe 's/"text"://; s/^"//; s/",$//'. (I did this for this analysis.)

To all the haters who insist you should use a real JSON parser -- yes, that is essential for correctness, but

  1. To do a really quick analysis, like counting values to check on data cleaning bugs or get a general feel for the data, banging out something on the command line is faster. Opening an editor to write a script is distracting.
  2. grep -o is orders of magnitude faster than the Python standard json library, at least when doing this for tweets (which are ~2 KB each). I'm not sure if this is just because json is slow (I should compare to yajl sometime); but in principle, a regex should be faster since it's finite state and much more optimizable, instead of a parser that has to support recursion, and in this case, spends lots of CPU building trees for structures you don't care about. (If someone wrote a finite state transducer that did proper (depth-limited) JSON parsing, that would be fantastic! In the meantime we have "grep -o".)

To write maintainable code, I always use a real parsing library. I haven't tried jsawk, but if it works well, that would address point #1.

One last, wackier, solution: I wrote a script that uses Python json and extracts the keys you want, into tab-separated columns; then I pipe through a wrapper around awk that allows named access to columns. In here: the json2tsv and tsvawk scripts. So for this example it would be:

json2tsv id text < tweets.json | tsvawk '{print "tweet " $id " is: " $text}'

This approach doesn't address #2, is more inefficient than a single Python script, and it's a little brittle: it forces normalization of newlines and tabs in string values, to play nice with awk's field/record-delimited view of the world. But it does let you stay on the command line, with more correctness than grep -o.

@Yola 2011-11-12 19:11:20

what if parameter last in tuple thwn at the end no comma but right brace. And +1 for sure.

@Brendan OConnor 2011-11-12 22:33:12

Hi Yola, right, it depends on the input. You have to look at it first.

@Robert 2013-12-04 01:52:51

You forgot about integer values. grep -Po '"text":(\d*?,|.*?[^\\]",)'

@Brendan OConnor 2013-12-05 02:02:39

Robert: Right, my regex was written only for string values for that field. Integers could be added as you say. If you want all types, you have to do more and more: booleans, null. And arrays and objects require more work; only depth-limited is possible, under standard regexes.

@jfs 2014-08-24 20:50:10

1. jq .name works on the command-line and it doesn't require "opening an editor to write a script". 2. It doesn't matter how fast your regex can produce wrong results

@JeffCharter 2015-09-06 19:37:03

and if you only want the values you can just throw awk at it. | grep -Po '"text":.*?[^\\]",'|awk -F':' '{print $2}'

@Jens 2016-06-08 13:14:15

It seems that on OSX the -P option is missing. I tested on OSX 10.11.5 and grep --version was grep (BSD grep) 2.5.1-FreeBSD. I got it working with the "extended regex" option on OSX. The command from above would be grep -Eo '"text":.*?[^\\]",' tweets.json.

@Brendan OConnor 2016-06-09 14:18:11

Yeah, jq is much better. When I wrote the original post it didn't exist yet or wasn't widespread.

@Marcus Wolschon 2016-06-30 06:18:55

it does not deal with the value appearing multiple times in the JSON as you can't specify the path that leads to the one you want inside the tree.

@user1 2016-11-16 06:37:54

it worked with me but how can i receive results in a file for this command grep -Po '"text":.*?[^\]",' tweets.json

@mgutt 2019-08-02 19:20:52

@JeffCharter You don't need awk. Instead use zero-length assertion like grep -oP '(?<="id": )[0-9]+' to get only the integer value

@mgutt 2019-08-02 19:27:43

@JeffCharter or grep -oP '(?<="text": ").*?[^\\](?=",)' for string values.

@aexl 2020-06-29 12:20:28

Important: I had to use grep --color=never on Debian 10

@Shane 2020-08-11 17:37:51

On OSX, to get just the value itself I ended up with this: grep -Eo '"text":.*?[^\\]",' tweets.json | sed -e 's/[\"\,\: ]*//g' | sed -e 's/text//g')

@mcnabicus 2014-02-02 21:28:01

Parsing JSON is painful in a shell script. With a more appropriate language, create a tool that extracts JSON attributes in a way consistent with shell scripting conventions. You can use your new tool to solve the immediate shell scripting problem and then add it to your kit for future situations.

For example, consider a tool jsonlookup such that if I say jsonlookup access token id it will return the attribute id defined within the attribute token defined within the attribute access from stdin, which is presumably JSON data. If the attribute doesn't exist, the tool returns nothing (exit status 1). If the parsing fails, exit status 2 and a message to stderr. If the lookup succeeds, the tool prints the attribute's value.

Having created a unix tool for the precise purpose of extracting JSON values you can easily use it in shell scripts:

access_token=$(curl <some horrible crap> | jsonlookup access token id)

Any language will do for the implementation of jsonlookup. Here is a fairly concise python version:

#!/usr/bin/python                                                               

import sys
import json

try: rep = json.loads(sys.stdin.read())
except:
    sys.stderr.write(sys.argv[0] + ": unable to parse JSON from stdin\n")
    sys.exit(2)
for key in sys.argv[1:]:
    if key not in rep:
        sys.exit(1)
    rep = rep[key]
print rep

@JayQuerie.com 2013-08-27 15:11:22

Using Node.js

If the system has installed, it's possible to use the -p print and -e evaulate script flags with JSON.parse to pull out any value that is needed.

A simple example using the JSON string { "foo": "bar" } and pulling out the value of "foo":

$ node -pe 'JSON.parse(process.argv[1]).foo' '{ "foo": "bar" }'
bar

Because we have access to cat and other utilities, we can use this for files:

$ node -pe 'JSON.parse(process.argv[1]).foo' "$(cat foobar.json)"
bar

Or any other format such as an URL that contains JSON:

$ node -pe 'JSON.parse(process.argv[1]).name' "$(curl -s https://api.github.com/users/trevorsenior)"
Trevor Senior

@Rnd_d 2013-11-25 23:02:08

thanks! but in my case it's working only with -e flag node -p -e 'JSON.parse(process.argv[1]).foo' '{ "foo": "bar" }'

@nicerobot 2014-05-07 19:19:18

Pipes! curl -s https://api.github.com/users/trevorsenior | node -pe "JSON.parse(require('fs').readFileSync('/dev/stdin').toStrin‌​g()).name"

@Eliran Malka 2017-03-01 15:05:48

this is my favourite solution; use a language (javascript) to parse a data-structure that is natural to it (JSON). seems the most correct. also - node is probably already available on the system, and you won't have to mangle with jq's binaries (which looks like another correct choice).

@Youness 2017-07-04 23:35:47

This is the bash script function: # jsonv get the json object value for a specific attribute # first parameter is the json document # second parameter is the attribute which value should be returned get_json_attribute_value() { node -pe 'JSON.parse(process.argv[1])[process.argv[2]]' "$1" "$2" }

@Ilya Boyandin 2018-10-15 10:21:03

The following works with Node.js 10: cat package.json | node -pe 'JSON.parse(fs.readFileSync(0)).version'

Related Questions

Sponsored Content

7 Answered Questions

[SOLVED] Why does Google prepend while(1); to their JSON responses?

53 Answered Questions

[SOLVED] Can comments be used in JSON?

  • 2008-10-28 20:39:03
  • Michael Gundlach
  • 2429910 View
  • 7851 Score
  • 53 Answer
  • Tags:   json comments

28 Answered Questions

[SOLVED] How do I parse a string to a float or int?

36 Answered Questions

[SOLVED] What is the correct JSON content type?

34 Answered Questions

[SOLVED] How to parse JSON in Java

  • 2010-04-07 09:00:21
  • Muhammad Maqsoodur Rehman
  • 1714524 View
  • 1083 Score
  • 34 Answer
  • Tags:   java json parsing

55 Answered Questions

[SOLVED] How can I pretty-print JSON in a shell script?

24 Answered Questions

[SOLVED] How do I POST JSON data with cURL?

9 Answered Questions

[SOLVED] Why can't Python parse this JSON data?

  • 2010-05-14 15:54:20
  • michele
  • 2646319 View
  • 1454 Score
  • 9 Answer
  • Tags:   python json parsing

24 Answered Questions

[SOLVED] pretty-print JSON using JavaScript

Sponsored Content