Top > Linux > Counting JSON Records with jq Command

Counting JSON Records with jq Command

Linux

This article summarizes how to count the number of records in JSON using the jq command, depending on the JSON format.

Introduction

Recently, I needed to count the number of records in a JSON file on the Linux command line. For CSV files, counting the number of lines is sufficient, but for JSON files, the counting method must be adjusted based on the format.

This article provides a guide on how to count the number of records for different JSON formats using the jq command.

Note: This article was translated from my original post.

Environment

OS: The commands were tested on Amazon Linux 2.

Counting JSON Records

We will look at how to count records in three different JSON formats:

  • JSON Documents with Array Wrapping
  • JSON Documents with Object Wrapping
  • JSON Lines

JSON Documents with Array Wrapping

# array.json
[
  {"id":1,"name":"aaa"},
  {"id":2,"name":"bbb"},
  {"id":3,"name":"ccc"},
  {"id":4,"name":"ddd"},
  {"id":5,"name":"eee"}
]

To count the number of records in a JSON file structured as an array (array.json), execute the following command:

cat array.json | jq 'length'
# Output: 5

This command first uses cat to read the JSON file and pipes (|) its content into the jq command. The jq command then applies the length filter to retrieve the number of records.

Note: The built-in length function in jq returns the number of elements based on the input type. If the input is an array, it returns the number of elements in the array.

From the jq manual:

length
  The builtin function length gets the length of various different types of value:
   ·  The length of a string is the number of Unicode codepoints it contains (which will be the same as its JSON-encoded length in bytes if it´s pure ASCII).
   ·  The length of an array is the number of elements.
   ·  The length of an object is the number of key-value pairs.
   ·  The length of null is zero.

JSON Documents with Object Wrapping

# object.json
{
  "records": [
    {"id":1,"name":"aaa"},
    {"id":2,"name":"bbb"},
    {"id":3,"name":"ccc"},
    {"id":4,"name":"ddd"},
    {"id":5,"name":"eee"}
  ]
}

To count the number of records stored in the "records" array inside an object-based JSON file (object.json), execute the following command:

cat object.json | jq '.records | length'
# Output: 5

This command applies the jq filter '.records | length'. First, .records extracts the array stored under "records", then the length function counts the number of elements in the array.

JSON Lines

# lines.json
{"id":1,"name":"aaa"}
{"id":2,"name":"bbb"}
{"id":3,"name":"ccc"}
{"id":4,"name":"ddd"}
{"id":5,"name":"eee"}

To count the number of records in a JSON Lines formatted file (lines.json), execute the following command:

cat lines.json | jq -s 'length'
# Output: 5

Using the -s (or --slurp) option in jq, the entire input is treated as a single array, resulting in an output like this:

$ cat lines.json | jq -s
[
  {
    "id": 1,
    "name": "aaa"
  },
  {
    "id": 2,
    "name": "bbb"
  },
  {
    "id": 3,
    "name": "ccc"
  },
  {
    "id": 4,
    "name": "ddd"
  },
  {
    "id": 5,
    "name": "eee"
  }
]

Applying the length filter to this array allows us to count the number of records.

Note: Even if the input file contains arbitrary line breaks, the command will still work correctly:

# lines.json
{
  "id":1,
  "name":"aaa"}
{"id":2,"name":"bbb"}
{"id":3,
"name":"ccc"}
{
  "id":4,
  "name":"ddd"
}
{"id":5,"name":"eee"
}

Record count:

cat lines.json | jq -s 'length'
# Output: 5

Conclusion

This article summarized how to count the number of records in JSON files using the jq command.

Since JSON files can be structured in multiple ways, it is important to check the input format before applying any processing.

The jq command provides a flexible way to handle various JSON formats, making it a useful tool for such tasks.

References