Celebrating dependency-management-data's second birthday

I had originally intended to write this on Sunday, to coincide with the two year anniversary since the first commit to dependency-management-data (DMD), but haven't had a chance to, with being this week at the excellent State of Open Con - more on that in a separate post I'm still writing.

I wanted to make sure I followed last year's post and do another reflection on a year in the project.

In the last year, there's been slightly less work directly committed on the project, but that doesn't mean there's been less work in/around it - in some of my "off hours", outside of working hours, there have been a number of times that I've instead worked on additional work on the dependency-management-data work at Elastic, in particular feeding into a presentation to my VP around potential "risks" in our Open Source usage, all powered through insights dependency-management-data gave us, which wouldn't be possible with other tools!

Similar to what I described in Does the tech industry thrive on free work?, this is something I've intentionally done with that time outside-of-work, knowing that I could've instead worked on the Open Source project. Although it's not been directly working on the OSS project, it has absolutely improved some views of the data that we may want to make easier.

That being said, over the last year there have still been a number of great additions to the project, and the wider ecosystem, including a number of key improvements to renovate-graph and renovate-to-sbom.

To follow the structure of last year's post, we have:

You'll notice that there's been some up-and-down for some of these metrics, which is absolutely fine, and I'm still very happy with - in particular the fact that we've had some new external contributors (and therefore users) as well as the wonderful Ashley Wolf, Director of GitHub's Open Source Program Office, who spoke at the Linux Foundation Member Summit about Dependency Risk Management: A Guide for OSPOs, which very heavily + prominently featured insights + data that was derived from DMD!

For some other numbers, of the 68 releases:

(Aside: this was much easier to calculate given the project follows conventional commits!)

Of those breaking changes:

• renovate-to-sbom: ignore dependencies without a CurrentVersion (c3c412f8)
• renovate: prioritise packageName over depName (20a77d8b)

• dependency-health: reduce concurrent calls to Ecosystems (b45f692c)

• sbom: remove the requirement for a Repo Key (71b685f9)
  • This was a big one, and fed into release v0.100.0, which worked out quite nice time-wise
• graphql: add pagination for advisories data (30471442)
• scorecards: correctly override Renovate-driven results on import (06075fa0)
• sbom: add package_url on imports (ccb812f4)
• remove database anonymisation (858bcfec)
• policies: remove policy_violations table (d0d62380)
• require to Go 1.22 (1dafa890)
• web: require query parameter to load all data from reports (5cd68030)
• move infrastructure advisories to their own report (377cccf0)
• remove Dependabot tables (23d8fb1f)

Speaking of breaking changes, in 0.91.0 (2024-04-07), we added the compatible_since functionality as a means to provide an external insight into whether tooling would support the version of the database it was interacting with. As far as I know, no one (other than Elastic) are directly using this, but it gives a great opportunity to evolve tools and the database, and was a great opportunity to consider "what changes should be deemed as breaking" and was documented under the Compatible Since docs.

I did actually think I'd made more breaking changes over the year, so that's interesting that there wasn't that much in there.

Not included here is that there were a number of releases to renovate-graph that make it much easier to work in projects that already have Renovate set up - but that we don't want to fully follow config, in case i.e. some packages are disabled for lookups, as well as resolving local> presets when running in --platform local.

I - somewhat unfairly - capitalised on the xz-utils backdoor + social engineering attack, and wrote What can we learn about the backdooring of xz/liblzma, using OpenSSF Security Scorecards and dependency-management-data?.

I also got around to starting to write my own SQL query interface, with an embedded web database browser, so you don't have to run datasette if you're resource constrained, and also fed into making it possible to run sql-studio as an alternative SQLite browser.

To make it possible to provide some custom End Of Life data for internal dependencies, I released changes that allowed querying endoflife.date in Policies and a couple of weeks later, wrote about how I made some significant performance improvements for Policies.

In December, the release of renovate-packagedata-diff was quite a monumental and long-running piece of work and - although I can't show you - was used at an internal hackathon to provide a really interesting diff view of changes in a given PR, and feeding into other insights we can give on changing dependencies.

In more local development, I wrote about my local SQLite workflow in Neovim which has been a really huge improvement for my own productivity, and later in the year set up a keybinding, Super+Shift+d, which would open a window with a pre-filled query, for my local DMD database.

I've recently built on top of this for more DMD-specific autocomplete - although it's not yet Open Source (due to shame at the code, more than anything) - and it's already made huge improvement to my personal workflow, in particular when at work, and trying to perform more complex queries, or querying i.e. "show me the owner for this repo" and giving me a more contextual autocomplete around which repository names there are over the org, etc.

Going back to the project's roots, I wrote How to use Dependency Management Data to discover which dependencies are participating in Hacktoberfest.

You'll notice that the number of issues on the backlog has grown - but not necessarily shrunk - and this is largely because I'm using it as a "while I think of this, it'd be good to go on the backlog", to feed into future design thinking, or see "oh, I've got some time, let me find an interesting thing to build".

I've also made a number of improvements to renovate-graph and there've been some community contributions (👏🏼) to renovate-to-sbom, which has also fed improvements with the underlying Renovate support in dependency-management-data. Last year I ended the post with:

And finally, I've been considering what it could look like if it were to be something I would make a paid offering around this.

Literally on New Year's Eve, I soft-launched the landing page for a to-be-built (enough caveats for you?) deps.fyi, a product being built on top of dependency-management-data, which I'm going to be doing some thinking about over the next year or so.

I'm happy with what I've got done this last year, seeing increased internal adoption at companies I know are using it, and excited to see how much gets delivered this coming year!