How do you represent a JSON field in Go that could be absent, null or have a value?

If you're a follower of my blog you'll know that just one of the Open Source projects I maintain is the oapi-codegen OpenAPI-to-Go code generator.

Last year, we received a feature request to handle the case where a JSON field may be one of three states - unspecified, set to null, or a given value - and it turns out it's a rather hard problem to solve.

For instance, let's say that we've got the following cases:

The field isn't specified:

{
}

The field is explicitly set to null:

{
  "field": null
}

The field is explicitly set to a value:

{
  "field": "this is a nullable field"
}

In OpenAPI 3.0.x, this is controlled via the nullable field, and the solution right now in oapi-codegen is to produce the type:

type S struct {
	Field *string `json:"field,omitempty"`
}

However, for a consumer of this struct, it's unclear whether the field was unspecified, or if it was set to null as they both result in Field == nil. This can be a little frustrating, and can be a significant hurdle if these values have a semantic difference in your API.

Some internal work at Elastic has meant that we've needed to add support for this, so my colleagues Ashutosh Kumar and Sebastien Guilloux and I have been working on this on-and-off for the last month or so, and have discovered that this is a really awkward problem 🫣

Over the years, there have been several other attempts at this, such as:

• https://stackoverflow.com/questions/36601367/json-field-set-to-null-vs-field-not-there
• https://www.calhoun.io/how-to-determine-if-a-json-key-has-been-set-to-null-or-not-provided/
• https://stackoverflow.com/questions/36601367/json-field-set-to-null-vs-field-not-there
• https://github.com/guregu/null/issues/39
• https://github.com/99designs/gqlgen/issues/1416 (which also links to lots of other libraries and attempts to do so)

But one problem we found was that no one seems to have solved it when you want to both marshal and unmarshal (serialise and deserialise) the data 😅 Which seemed very odd, and surprising that there's no built-in way to do this.

An initial version of our solution looked like Jon Calhoun's, with an updated signature now generics are available in Go:

type Nullable[T any] struct {
	// Value is the underlying value
	Value T
	// Set indicates whether the field was sent
	Set bool
	// Null indicates that the field was set explicitly as Null. Only true if `Set` is also true.
	Null bool
}

Although we could get this to work with marshalling and unmarshalling a required field, such as:

type S struct {
	Field Nullable[string] `json:"field"`
}

Trying to do the same with an optional field wasn't so lucky:

type S struct {
	OptionalField Nullable[string] `json:"field,omitempty"`
}

// or

type S struct {
	OptionalField *Nullable[string] `json:"field,omitempty"`
}

In both cases, the optional field wouldn't fulfill all of the cases, and having burned a fair bit of time on the problem we felt like it wouldn't be solvable, especially after trawling through the prior art in solving this, several pages of search results and issues on the Go tracker (of which many referred to closed proposals to make this possible).

However, we did eventually come to this excellent solution by KumanekoSakura:

// Code taken from https://github.com/oapi-codegen/nullable/blob/v1.0.0/nullable.go

// Nullable is a generic type, which implements a field that can be one of three states:
//
// - field is not set in the request
// - field is explicitly set to `null` in the request
// - field is explicitly set to a valid value in the request
//
// Nullable is intended to be used with JSON marshalling and unmarshalling.
//
// Internal implementation details:
//
// - map[true]T means a value was provided
// - map[false]T means an explicit null was provided
// - nil or zero map means the field was not provided
//
// If the field is expected to be optional, add the `omitempty` JSON tags. Do NOT use `*Nullable`!
//
// Adapted from https://github.com/golang/go/issues/64515#issuecomment-1841057182
type Nullable[T any] map[bool]T

// other fields and methods omitted

func (t Nullable[T]) MarshalJSON() ([]byte, error) {
	// if field was specified, and `null`, marshal it
	if t.IsNull() {
		return []byte("null"), nil
	}

	// if field was unspecified, and `omitempty` is set on the field's tags, `json.Marshal` will omit this field

	// otherwise: we have a value, so marshal it
	return json.Marshal(t[true])
}

func (t *Nullable[T]) UnmarshalJSON(data []byte) error {
	// if field is unspecified, UnmarshalJSON won't be called

	// if field is specified, and `null`
	if bytes.Equal(data, []byte("null")) {
		t.SetNull()
		return nil
	}
	// otherwise, we have an actual value, so parse it
	var v T
	if err := json.Unmarshal(data, &v); err != nil {
		return err
	}
	t.Set(v)
	return nil
}

The ingenious approach here to use a map, which due to how encoding/json handles empty values alongside the isEmptyValue method allows us to use our Nullable type, without making it *Nullable. If we had made it *Nullable, we'd lose some the ability to understand whether a field was unspecified or null, when unmarshalling.

So following on from this, we've adapted KumanekoSakura's code, and released this as its own library, github.com/oapi-codegen/nullable, which aims to be a standalone, zero dependency, library for the purpose of knowing whether a JSON field is nullable or not.

I realise in the Go community a lot of folks prefer to avoid dependencies for dependencies' sake, instead copying code around between projects, but I thought I would still release it as its own project, so it can be consumed where necessary, if at least to make it much clearer exactly what is required for it.

There's more detail of example usage + output in the testable examples on pkg.go.dev.

We'll also be working to add the ability for oapi-codegen to generate you Nullable types - if you've requested it via configuration - so you can reap the benefits 👏🏽

Got any thoughts, or prior art we've missed? Let me know via the means in the footer 👇🏽