Go 1.24's go tool is one of the best additions to the ecosystem in years

For those that aren't aware, one of the big changes in February's upcoming Go 1.24 release is the new go tool command, and tool directive in the go.mod to manage any tools your project uses. I'm incredibly excited about this, and in my opinion, this is one of the best changes we've had in recent years in the ecosystem as a whole.

I've been meaning to write this post since the first release candidate for Go 1.24 landed, but after reading John Howard's Exploring the new "go tool" support in Go 1.24 this morning, I thought I should write my thoughts up.

What is it?

Within your Go codebases, there's often some additional tools that you need to have installed to be able to build/test/deploy the project.

Sometimes this will a dependency that's needed for go generateing, or it may be that you want to pipe your go test output into a JUnit-compatible format, so your CI platform can provide more useful metadata.

For each of these, you have two choices:

• require that the user knows how to install them, i.e. by knowing to run make deps or just setup before building anything on the project (which will then i.e. go install the commands)
• use the tools.go pattern to make it so you can just run go generate, and that'll call the right dependency via go run

My preference is tools.go pattern, but there are two key problems with this approach.

Firstly, there's a performance hit of using a tools.go. It's something that is slightly noticeable, moreso if your project relies upon a lot of go run i.e. with lots of go generates, because prior to Go 1.24, the go run invocations were not cached.

Secondly, it also leads to dependency tree bloat, because you have to record your dependency on i.e. github.com/sqlc-dev/sqlc/cmd/sqlc which then gets recorded in your go.mod, and then anyone using your module will then see that as an indirect (transitive) dependency.

This was something we worked on for oapi-codegen's v2 release to further reduce unnecessary dependencies, and make things a bit cleaner for our consumers. This is somewhat mitigated by Go's module graph pruning which won't download dependencies that aren't used, but consumers may still see the dependencies coming in as an indirect dependency, which may not be ideal (especially as it can then bloat their indirect dependencies, which then gets passed on to their consumers and so on .

Dependency tree bloat can also be further mitigated by splitting your tools.go into a separate module, which makes it more awkward to invoke dependencies but makes sure that none of your consumers will be seeing any tool-related dependencies.

For those who know me as co-maintainer of oapi-codegen, you'll know that the tools.go pattern is our explicit recommendation and we believe is better than installing it as a binary, so it's probably unsurprising that I'm very excited about this as an option to manage dependencies.

How does it work?

I've started playing around with this on a branch of the dependency-management-data project, where I've got a mix of different tools that need to be installed and used.

Let's take a worked example of how we'd move over calls to oapi-codegen to the new go tool pattern.

Existing state

For instance let's say that we have the following tools.go in its own module:

# tools/go.mod
module dmd.tanna.dev/tools

go 1.22.0

require (
	github.com/99designs/gqlgen v0.17.49
	github.com/oapi-codegen/oapi-codegen/v2 v2.4.1
	github.com/sqlc-dev/sqlc v1.26.0
)

We can then see that we invoke this via go run:

// internal/ecosystems/generate.go
//go:generate go run -modfile=../../tools/go.mod github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --config=config.yaml openapi.yaml

Migrating

To start migrating over to go tool, we need to make sure that we've first pulled in the new version of Go in our top-level Go module:

 module dmd.tanna.dev

-go 1.22.7
+go 1.24

-toolchain go1.23.2
+toolchain go1.24rc2

Next, we need to pull in a tool dependency on oapi-codegen's CLI tool - notice that you need the full path to the command that's being invoked:

# NOTE the full import path
% go get -tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen@v2.4.1

We could also do this by hand, but doing it via go get simplifies this a little.

From here, we'll notice that our go.mod has a few other changes:

@@ -57,12 +57,16 @@ require (
        github.com/cenkalti/backoff/v4 v4.3.0 // indirect
        github.com/cespare/xxhash/v2 v2.3.0 // indirect
        github.com/charmbracelet/lipgloss v0.10.0 // indirect
+       github.com/dprotaso/go-yit v0.0.0-20220510233725-9ba8df137936 // indirect
        github.com/dustin/go-humanize v1.0.1 // indirect
        github.com/felixge/httpsnoop v1.0.4 // indirect
+       github.com/getkin/kin-openapi v0.127.0 // indirect
        github.com/go-ini/ini v1.67.0 // indirect
        github.com/go-logfmt/logfmt v0.6.0 // indirect
        github.com/go-logr/logr v1.4.2 // indirect
        github.com/go-logr/stdr v1.2.2 // indirect
+       github.com/go-openapi/jsonpointer v0.21.0 // indirect
+       github.com/go-openapi/swag v0.23.0 // indirect
        github.com/gobwas/glob v0.2.3 // indirect
        github.com/google/go-querystring v1.1.0 // indirect
        github.com/gorilla/mux v1.8.1 // indirect
@@ -72,16 +76,22 @@ require (
        github.com/hashicorp/go-retryablehttp v0.7.5 // indirect
        github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
        github.com/inconshreveable/mousetrap v1.1.0 // indirect
+       github.com/invopop/yaml v0.3.1 // indirect
+       github.com/josharian/intern v1.0.0 // indirect
        github.com/klauspost/compress v1.17.11 // indirect
        github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
+       github.com/mailru/easyjson v0.7.7 // indirect
        github.com/mattn/go-isatty v0.0.20 // indirect
        github.com/mattn/go-runewidth v0.0.15 // indirect
        github.com/mitchellh/mapstructure v1.5.0 // indirect
+       github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826 // indirect
        github.com/muesli/reflow v0.3.0 // indirect
        github.com/muesli/termenv v0.15.2 // indirect
        github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
        github.com/ncruces/go-strftime v0.1.9 // indirect
+       github.com/oapi-codegen/oapi-codegen/v2 v2.4.1 // indirect
        github.com/olekukonko/tablewriter v0.0.5 // indirect
+       github.com/perimeterx/marshmallow v1.1.5 // indirect
        github.com/prometheus/client_golang v1.20.5 // indirect
        github.com/prometheus/client_model v0.6.1 // indirect
        github.com/prometheus/common v0.60.1 // indirect
@@ -91,8 +101,10 @@ require (
        github.com/rivo/uniseg v0.4.7 // indirect
        github.com/sirupsen/logrus v1.9.4-0.20230606125235-dd1b4c2e81af // indirect
        github.com/sosodev/duration v1.3.1 // indirect
+       github.com/speakeasy-api/openapi-overlay v0.9.0 // indirect
        github.com/spf13/pflag v1.0.5 // indirect
        github.com/tchap/go-patricia/v2 v2.3.1 // indirect
+       github.com/vmware-labs/yaml-jsonpath v0.3.2 // indirect
        github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
        github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
        github.com/yashtewari/glob-intersection v0.2.0 // indirect
@@ -110,11 +122,13 @@ require (
        go.opentelemetry.io/otel/metric v1.32.0 // indirect
        go.opentelemetry.io/proto/otlp v1.3.1 // indirect
        golang.org/x/exp v0.0.0-20231108232855-2478ac86f678 // indirect
+       golang.org/x/mod v0.18.0 // indirect
        golang.org/x/net v0.30.0 // indirect
        golang.org/x/oauth2 v0.23.0 // indirect
        golang.org/x/sys v0.27.0 // indirect
        golang.org/x/term v0.25.0 // indirect
        golang.org/x/time v0.5.0 // indirect
+       golang.org/x/tools v0.22.0 // indirect
        google.golang.org/genproto/googleapis/api v0.0.0-20241104194629-dd2ea8efbc28 // indirect
        google.golang.org/genproto/googleapis/rpc v0.0.0-20241104194629-dd2ea8efbc28 // indirect
        google.golang.org/grpc v1.68.0 // indirect
@@ -128,3 +142,5 @@ require (
        modernc.org/token v1.1.0 // indirect
        sigs.k8s.io/yaml v1.4.0 // indirect
 )
+
+tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen

From here, we can see:

• there is a tool directive for github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen
• the containing Go module for the CLI, github.com/oapi-codegen/oapi-codegen/v2, is now an indirect dependency
• any other required dependencies of github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen are now indirect dependencies

Now we've done this, we could run:

% go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --help
Usage of /home/jamie/.cache/go-build/0e/0e04736601c8bbef785d372de02859bf8f39405aae9ccbf371477b0f2d8df755-d/oapi-codegen:
# ...

With this tool set up, we can now modify i.e. internal/ecosystems/generate.go like so to use the new go tool:

 package ecosystems

-//go:generate go run -modfile=../../tools/go.mod github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --config=config.yaml openapi.yaml
+//go:generate go tool github.com/oapi-codegen/oapi-codegen/v2/cmd/oapi-codegen --config=config.yaml openapi.yaml

Then running go generate ./internal/ecosystems works as it did before 🚀

Performance implications

A less scientific view than Howard John's article above, but we can see a slight improvement in performance:

# first time using `go tool`, from a fresh cache directory
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  55.05s user 4.57s system 531% cpu 11.220 total
# a subsequent call
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  0.59s user 0.18s system 424% cpu 0.181 total
# another just to see
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  0.57s user 0.25s system 404% cpu 0.202 total

Compare this to the previous implementation:

# first time using `go run`, from a fresh cache directory
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  50.29s user 3.67s system 536% cpu 10.063 total
# a subsequent call
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  1.04s user 0.21s system 185% cpu 0.677 total
# another just to see
% time go generate ./internal/ecosystems
go generate ./internal/ecosystems  1.02s user 0.26s system 191% cpu 0.669 total

Notice that the first call is similar in speed, but the use of go tool's subsequent calls are still faster.

I'm a big fan of the fact that as of Go 1.24+ the go runs will be cached, so even if you don't move over to go tool, you'll get a performance boost!

Concerns

Now, there are still a few things I've noticed while doing the migration that aren't necessarily what I expected.

go.mod implications

Something interesting is that the usage of the tool dependencies being treated as an indirect dependency is that they're present in the dependency tree, and treated like any other indirect dependency.

I'd also have preferred that we had just used // tool instead of // indirect, but I can see why this is likely the choice that's made - so they're treated like any other dependency - but making them less clear as only being required for tools could lead to issues with clashing dependencies, or where you upgrade an indirect dependency and then that breaks other things.

This means that tools such as Renovate need to be a little more involved in how to do the updates, but that's all in hand.

gqlgen fails to run with Go 1.24rc2

Something I've noticed while playing around with this is that gqlgen struggles to run with Go 1.24rc2, which feels like an upstream Go issue, but it looks like that may be related to the use of /x/tools 🤔

It may be interesting to find out what else gets affected by this - please give the RC a test!

Closing

Overall, I'm feeling very positive about it, and improving the way that dependencies get installed if they should be built from source, but there are dependencies such as golangci-lint which don't recommend building from source and instead using their pre-built binaries, which is fair, and is unlikely to change here.