jq transformation

I was working on a terraform module that creates cloudwatch alarms. AWS uniquely identifies metric with dimensions. All dimensions must be supplied to select the metric. In this case, the dimension comes from CWAgent and terraform external data source is used to obtain the dimension. A bit of jq work is needed to transform the json output.

First, let’s see what the json returned by awscli looks like

CWAgent metrics falls under the CWAgent namespace. Here I search for the metric for a particular instance and its / filesystem.

❯ aws cloudwatch list-metrics --namespace CWAgent --metric-name disk_inodes_free --dimensions Name=InstanceId,Value=i-0236c44a8b655735f Name=path,Value=/
{
    "Metrics": [
        {
            "Namespace": "CWAgent",
            "MetricName": "disk_inodes_free",
            "Dimensions": [
                {
                    "Name": "path",
                    "Value": "/"
                },
                {
                    "Name": "InstanceId",
                    "Value": "i-0236c44a8b655735f"
                },
                {
                    "Name": "AutoScalingGroupName",
                    "Value": "eks-whk1-bea-icc-mbk-dev-eks-mbk-01-ng01-88c0d68a-7582-23aa-e140-aea3e6c026a1"
                },
                {
                    "Name": "ImageId",
                    "Value": "ami-0a5637e1e2c6310a7"
                },
                {
                    "Name": "InstanceType",
                    "Value": "c5.4xlarge"
                },
                {
                    "Name": "device",
                    "Value": "nvme0n1p1"
                },
                {
                    "Name": "fstype",
                    "Value": "xfs"
                }
            ]
        }
    ]
}

jq journey begins

I am only interested in the device and fstype dimensions. Let me show you the complete command chains and I’ll break them down later. With these series of commands, I am able to produce a map that terraform is happy to take in. With the external data source, I can reference the results using data.external.disk-device.result.device

❯ aws cloudwatch list-metrics --namespace CWAgent --metric-name disk_inodes_free --dimensions Name=InstanceId,Value=i-0236c44a8b655735f Name=path,Value=/ | \
jq '.Metrics[] | .Dimensions[] | select ((.Name=="device") or (.Name=="fstype")) | { (.Name): (.Value)}' | \
jq -s 'add // {"device":"unknown", "fstype":"unknown"}'
{
  "device": "nvme0n1p1",
  "fstype": "xfs"
}

To break these down, first use select to obtain the device and fstype dimensions.

❯ cat json | jq '.Metrics[] | .Dimensions[] | select ((.Name=="device") or (.Name=="path"))'
{
  "Name": "path",
  "Value": "/"
}
{
  "Name": "device",
  "Value": "nvme0n1p1"
}

Next, transform the key-value objects into maps

❯ cat json2 | jq '{ (.Name): (.Value)}'
{
  "path": "/"
}
{
  "device": "nvme0n1p1"
}

Next, add these maps together using jq slurp and add

❯ cat json3 | jq -s 'add'
{
  "path": "/",
  "device": "nvme0n1p1"
}

Finally, create hard-coded output if null was returned. That happens when cloudwatch agent is not installed on the instance, or metric was not published. Terraform external data source expects a valid json. It will error out if null is returned.

❯ cat json3 | jq -s 'add //  {"device":"unknown", "fstype":"unknown"}'
{
  "path": "/",
  "device": "nvme0n1p1"
}

Wrapping up

With this shell script, I can then use terraform external data source, provide the instance id as input, and provide device and fstype as input to other modules. Here is the terraform code and the complete get-cwagent-device.sh script.

data "external" "disk-device" {
  program = ["bash", "${path.module}/get-cwagent-device.sh"]
  query = {
    input = var.ec2-instance-id
  }
}

#!/bin/bash
eval "$(jq -r '@sh "id=\(.input)"')"

aws cloudwatch list-metrics --namespace CWAgent --metric-name disk_inodes_free \
--dimensions Name=InstanceId,Value=$id Name=path,Value=/ | \
jq '.Metrics[] | .Dimensions[] | select ((.Name=="device") or (.Name=="fstype")) | { (.Name): (.Value)}' | \
jq -s 'add // {"device":"unknown", "fstype":"unknown"}'

Honestly, I do not really know what I did. It’s hours of trial and error and I thought I may share it to save others some time.