Creating a test harness for validating Renovate Custom Datasource configuration

As I've written about before, I'm a big fan of Renovate.

This last week, I've properly got stuck into Renovate's new Custom Datasources functionality, which have been great. Sebastian has written about them in the past and I'd recommend reading it for the full background on what Custom Datasources are and why they're great.

Similar to my post Creating a test harness for validating Renovate regex manager rules, one concern for maintainability is that you need to write JSONata rules to transform from one form of JSON to another, which many may not be familiar with.

JSONata is an interesting "JSON query and transformation language" and not one I'd dug into before. To get me started, I asked qwen2.5-coder:32b in Ollama to help get me going, as I generally recommend not using random online tools for testing things (in particular with proprietary data or APIs) and wanted to avoid using their handy playground for testing.

JSONata isn't something that many folks will be familiar with, and similar to a regex, a human needs to be able to read the expression, understand what it's doing, and maintain it over time.

To make matters slightly worse, because we're needing to embed this into our Renovate JSON config, it means we convert from pretty-printed JSONata queries like:

{
    "releases": $[license_class="oss"].{
        "version": version,
        "releaseTimestamp": timestamp_created,
        "changelogUrl": url_changelog,
        "sourceUrl": url_source_repository
    },
    "homepage": $[license_class="oss"][0].url_project_website
}

To a squashed JSON string:

{
  "customDatasources": {
    "...": {
      "transformTemplates": [
        "{ \"releases\": $.{\"version\": version,\"releaseTimestamp\": timestamp_created,\"changelogUrl\": url_changelog,\"sourceUrl\": url_source_repository},\"homepage\": $[license_class=\"oss\"][0].url_project_website}"
      ]
    }
  }
}

So how do we try and make this more maintainable over time? My usual first step is seeing if we can introduce a test harness.

Example

This will build on top of the examples from Sebastian's post, and the completed code can be found in the repository on GitLab.com, which builds upon the regex manager testing.

First, we'll create a new file, __testdata/hashicorp-consul-oss.json, from which we'll feed in a subset of the API response from the Consul releases API.

This allows us to avoid needing to do outbound HTTP to fetch this data each time we run the tests, and allows us to provide a smaller surface for our tests to validate, without having to be as exhaustive for all the possible versions that may be returned from the API.

Remember to look at updating it over time to make sure that it still works based on the new data coming back from the API!

To get us started, the minimum test we'll want to write is:

At a minimum:

describe('custom datasources', () => {
  describe('hashicorp consul', () => {
    const datasource = baseConfig.customDatasources!['hashicorp-consul']

    it('transforms correctly', () => {
      const contents = readFileSync('./__testdata/hashicorp-consul-oss.json')
      const apiResponse = JSON.parse(contents.toString('utf8'))

      const template = datasource.transformTemplates![0]
      const expression = jsonata(template)

      const expected = {
        "releases": [
          {
            "version": "1.20.1",
          },
          {
            "version": "1.18.2",
          },
        ],
      }

      // as this returns a promise, and we need to resolve it, but then modify the actual response to handle an outstanding JSONata issue
      expression.evaluate(apiResponse).then(function(rawActual: any) {
        // For some reason, we're being hit with https://github.com/jsonata-js/jsonata/issues/296
        const actual = JSON.parse(JSON.stringify(rawActual))

        expect(actual).toStrictEqual(expected)
      })
    })
  })
})

This makes sure that we get the bare minimum required for Custom Datasources, and ensures that we can then start writing our implementation to match this.

This allows us to write the following Renovate config:

{
  "customDatasources": {
    "hashicorp-consul": {
      "transformTemplates": [
        "{ \"releases\": $.{\"version\": version}}"
      ]
    }
  }
}

However, in Sebastian's blog, you'll notice there's a lot more rich data we can feed into Custom Datasources, allowing Renovate to leverage this rich data for presenting to the user.

You could see that this is now something that's straightforward to incrementally add, in a Test Driven Development (TDD) fashion, as we can now add a new entry in the expected JSON response.

This results in the following final test:

it('transforms correctly', () => {
  const contents = readFileSync('./__testdata/hashicorp-consul-oss.json')
  const apiResponse = JSON.parse(contents.toString('utf8'))

  const template = datasource.transformTemplates![0]
  const expression = jsonata(template)

  const expected = {
    "releases": [
      {
        "version": "1.20.1",
        // these are new!
        "releaseTimestamp": "2024-10-30T16:59:30.860Z",
        "changelogUrl": "https://github.com/hashicorp/consul/blob/release/1.20.1/CHANGELOG.md",
        "sourceUrl": "https://github.com/hashicorp/consul"
      },
      {
        "version": "1.18.2",
        // these are new!
        "releaseTimestamp": "2024-05-17T14:41:12.287Z",
        "changelogUrl": "https://github.com/hashicorp/consul/blob/release/1.18.2/CHANGELOG.md",
        "sourceUrl": "https://github.com/hashicorp/consul"
      },
    ],
    // this is new!
    "homepage": "https://www.consul.io"
  }

  // as this returns a promise, and we need to resolve it, but then modify the actual response
  expression.evaluate(apiResponse).then(function(rawActual: any) {
    // For some reason, we're being hit with https://github.com/jsonata-js/jsonata/issues/296
    const actual = JSON.parse(JSON.stringify(rawActual))

    expect(actual).toStrictEqual(expected)
  })
})

This would then be implemented with the following JSONata query:

{
    "releases": $.{
        "version": version,
        "releaseTimestamp": timestamp_created,
        "changelogUrl": url_changelog,
        "sourceUrl": url_source_repository
    },
    "homepage": $[0].url_project_website
}

Which would then be seen with the following Renovate config:

{
  "customDatasources": {
    "hashicorp-consul": {
      "transformTemplates": [
        "{ \"releases\": $.{\"version\": version}}"
      ]
    }
  }
}

Et voila! We've now got a test harness for our JSONata transformations, using the same version of the jsonata library that Renovate uses, and allows us to maintain these much more straightforwardly over time!