Creating a test harness for validating Renovate regex manager rules

As I've written about before, I'm a big fan of Renovate.

One of my favourite Renovate features is the Custom Manager Support using Regex, which allows you to capture dependencies in files that Renovate doesn't understand out-of-the-box.

However, this brings to mind an xkdc:

Having to maintain the regexes can get rather tricky, especially as some of them can be rather complex, or work across multiple lines.

Over the years I've written a fair few of these, and in the recent month I've been writing quite a few more, and have really been feeling the pain of trying to ensure they work.

I recently added some additional logging to Renovate for this, alongside running Renovate one-offs, but it wasn't quite the experience I wanted - I really wanted a test harness. So that's what I built!

Fixing a regex

I noticed recently that my article about Buildkite agent image management with Renovate had a bug, so let's use this as an example of how to test-drive a fix.

Setting up the test harness

The completed code can be found in the repository on GitLab.com. Below you'll find a step-by-step guide for setting it up.

Let's start by creating a new Typescript project:

npm i --save-dev typescript
npm i --save-dev @tsconfig/node18
# in the case you want to pin to the version of Renovate you're using
npm i --save-dev renovate@37.363.5

Then we'll create the tsconfig.json:

{
  "extends": "@tsconfig/node18/tsconfig.json"
}

Next, we want to set up our actual test framework, which in this case we'll use Jest:

npm i --save-dev jest @types/jest
npm i --save-dev ts-jest
npx ts-jest config:init

(You don't have to use Jest, but I wanted to take advantage of its snapshotting functionality for better visibility and control over assertions, and it happens to be what Renovate itself uses)

Now we've got our dependencies, we want to set up the renovate.json that we want to test:

{
  "regexManagers": [
    {
      "fileMatch": [
        "buildkite\\.ya?ml",
        "\\.buildkite/.+\\.ya?ml$",
        "\\.buildkite/.+/.+\\.ya?ml$"
      ],
      "matchStrings": [
        "image:\\s*\"?(?<depName>[^\\s]+):(?<currentValue>[^\\s\"]+)\"?"
      ],
      "datasourceTemplate": "docker"
    }
  ]
}

With that setup complete, we can now start writing tests:

import { extractPackageFile } from 'renovate/dist/modules/manager/custom/regex'
import { ExtractConfig } from 'renovate/dist/modules/manager/types'

describe('renovate.json', () => {
  const baseConfig = require('./renovate.json')
  const packageFile = 'UNUSED'

  describe('buildkite images', () => {
    // unfortunately we have to index into this right now, until https://github.com/renovatebot/renovate/issues/21760 is complete
    const config: ExtractConfig = baseConfig.customManagers[0]

    const fileContents = {
      'image with tag and quotes': `
      image: "golang:1.19"
      `,
      'image with tag but no quotes': `
      image: golang:1.19
      `,
    }

    it('matches an image with tag and quotes', () => {
      const content = fileContents['image with tag and quotes']

      const res = extractPackageFile(content, packageFile, config)

      expect(res).toMatchSnapshot({
        deps: [
          {
            depName: 'golang',
            currentValue: '1.19'
          }
        ]
      })
    })

    it('matches an image with tag but no quotes', () => {
      const content = fileContents['image with tag but no quotes']

      const res = extractPackageFile(content, packageFile, config)

      expect(res).toMatchSnapshot({
        deps: [
          {
            depName: 'golang',
            currentValue: '1.19'
          }
        ]
      })
    })
  })
});

This isn't exhaustive testing, but gives us a good starting point.

Test-driving new functionality

So now we've got the harness set up, let's try and change the regex, rather than just work based on what we already have in place.

Let's say that we also want to update Docker images referenced in Buildkite pipelines that are using digest pinning, such as:

image: "golang:1.22@sha256:0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b"

Well, we can start by writing a new test case:

    it('matches an image with tag and digest and quotes', () => {
      const content = fileContents['image with tag and digest and quotes']

      const res = extractPackageFile(content, packageFile, config)

      expect(res).toMatchSnapshot({
        deps: [
          {
            depName: 'golang',
            currentValue: '1.22',
            currentDigest: 'sha256:0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b'
          }
        ]
      })
    })

Now when we run this we can see a failure:

  ● renovate.json › buildkite images › matches an image with tag and digest and quotes

    expect(received).toMatchSnapshot(properties)

    Snapshot name: `renovate.json buildkite images matches an image with tag and digest and quotes 1`

    - Expected properties  - 3
    + Received value       + 2

      Object {
        "deps": Array [
          Object {
    -       "currentDigest": "sha256:0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b",
    -       "currentValue": "1.22",
    -       "depName": "golang",
    +       "currentValue": "0b55ab82ac2a54a6f8f85ec8b943b9e470c39e32c109b766bbc1b801f3fa8d3b",
    +       "depName": "golang:1.22@sha256",
          },
        ],
      }

      57 |                      const res = extractPackageFile(content, packageFile, config)
      58 |
    > 59 |                      expect(res).toMatchSnapshot({
         |                                  ^
      60 |                              deps: [
      61 |                                      {
      62 |                                              depName: 'golang',

      at Object.<anonymous> (renovate.spec.ts:59:16)

 › 1 snapshot failed.
Snapshot Summary
 › 1 snapshot failed from 1 test suite. Inspect your code changes or run `npm run npx -- -u` to update them.

We can see here that our regex is currently capturing the digest as part of the name of the dependency and the digest as the actual tag, which isn't right 😅

From here, we can then either perform some trial-and-error, or do some more calculated work to modify our regex in our renovate.json, in this case to:

-        "image:\\s*\"?(?<depName>[^\\s]+):(?<currentValue>[^\\s\"]+)\"?"
+        "image:\\s*\"?(?<depName>[^\\s:@\"]+)(?::(?<currentValue>[-a-zA-Z0-9.]+))?(?:@(?<currentDigest>sha256:[a-zA-Z0-9]+))?\"?"

Now when we run this, our tests pass 🙌🏼

Even if it helps no one else, this is going to save me a tonne of time.