This post is about a library and command I created called optional with the help of go generate. The code is here: https://github.com/markphelps/optional if you want to follow along.

I’ve been coding in Go for some time now, and one thing I miss from my Java days is the ability to express an option type. An option type (from the Java docs) is:

A container object which may or may not contain a non-null value. If a value is present, isPresent() will return true and get() will return the value.

This was useful in Java because it allowed you to avoid the dreaded NPE (Null Pointer Exception) by enabling you to return Optional types that wrapped the value that you were interested in.

Now I get it, Go is not Java. But, as an experiment, I wanted to see what option types in Go would look like and how they could be implemented in a language without generics.

I started out with some example code on how I would like the API to look for an option string type:

package optional

type String struct {
    string  string
    present bool
}

func EmptyString() String {
    return String{}
}

func OfString(s string) String {
    return String{string: s, present: true}
}

func (o *String) Set(s string) {
    o.string = s
    o.present = true
}

func (o *String) Get() string {
    return o.string
}

func (o *String) Present() bool {
    return o.present
}

This implementation is a little different from the Java version since I’m working with string not *string, so there is no nil/null, but you get the idea.

Once I got the API how I liked it, I started to think about how I would implement this for the other built-in types. Since Go does not have generics like Java, I couldn’t just write a single option implementation that would work for all types. Instead, I would have to write an option implementation for every single type! This is obviously not ideal, and would lead to much duplication and most likely bugs.

There had to be a better way..

Go Generate to the Rescue

After some googling, I came across this post on the Golang blog describing go generate, a tool that can be used to generate code. This was exactly what I was looking for.

Basically you pass any command to go generate and it will execute it. You can not only invoke generate with arguments from the command line, but also put directives in your code such as: //go:generate foo which will execute the foo command when you run go generate with no args.

The stringer tool is a great example of how powerful this can be, by allowing you to generate String() methods for your iota defined constants.

I decided to make heavy use of go generate to generate option implementations for the built-in Go primitives using examples I found in the stringer package to guide me.

To start, I looked at the example code I had written for the option string type above, and extracted that into a template. This came out looking something like this:

package {{ .PackageName }}

// {{ .OutputName }} is an optional {{ .TypeName }}
type {{ .OutputName }} struct {
    {{ .TypeName | unexport }} {{ .TypeName }}
    present bool
}

// Empty{{ .OutputName | title }} returns an empty {{ .PackageName }}.{{ .OutputName }}
func Empty{{ .OutputName | title }}() {{ .OutputName }} {
    return {{ .OutputName }}{}
}

// Of{{ .TypeName | title }} creates a {{ .PackageName }}.{{ .OutputName }} from a {{ .TypeName }}
func Of{{ .TypeName | title }}({{ .TypeName | first }} {{ .TypeName }}) {{ .OutputName }} {
    return {{ .OutputName }}{ {{ .TypeName | unexport }}: {{ .TypeName | first }}, present: true}
}

// Set sets the {{ .TypeName }} value
func (o *{{ .OutputName }}) Set({{ .TypeName | first }} {{ .TypeName }}) {
    o.{{ .TypeName | unexport }} = {{ .TypeName | first }}
    o.present = true
}

// Get returns the {{ .TypeName }} value
func (o *{{ .OutputName }}) Get() {{ .TypeName }} {
    return o.{{ .TypeName | unexport }}
}

// Present returns whether or not the value is present
func (o *{{ .OutputName }}) Present() bool {
    return o.present
}

This can be a little hard to read at first, but templates are just plain text with some special syntax added on top. Templates allow you to output variable values from your Go code using the .VariableName syntax, as well as pipe | those values to your own defined functions using a FuncMap.

In my code I defined a struct to hold the data that I needed to pass to my template, as well as some helper functions to deal with capitalization and things like that:

type data struct {
  Timestamp   time.Time
  PackageName string
  TypeName    string
  OutputName  string
}

var (
    funcMap = template.FuncMap{
        "title": strings.Title,
        "first": func(s string) string {
            return strings.ToLower(string(s[0]))
        },
        "unexport": func(s string) string {
            return strings.ToLower(string(s[0])) + string(s[1:])
        },
    }
)

Then its as simple as parsing the template giving it the FuncMap defined, executing the template giving it a buffer, and then writing that buffer out to a file:

t := template.Must(template.New("").Funcs(funcMap).Parse(tmpl))

var buf bytes.Buffer

err := t.Execute(&buf, data)
if err != nil {
  return nil, err
}

err = ioutil.WriteFile(filename, buf.Bytes(), 0644)
if err != nil {
  log.Fatalf("writing output: %s", err)
}

Using this method I was able to generate over 20 implementations of option types almost instantly! Better yet, this could even be extended to allow others to generate option types for their own code.

Extracting and Testing

In order to allow others to create option types of their own, I needed to extract my code into a command. This would allow others to run my code from go generate or the command line if they wished. To do this I moved my main code into cmd/optional/main.go, added some flag parsing and ability to read from STDIN, and abstracted it a bit to be more general.

After this, users could now invoke the optional tool on the command line with $ optional -type=Foo or using go generate directives in their source: //go:generate optional -type=Foo. This would create an option wrapper type for the user defined Foo type.

Now that the command existed, I wanted to figure out a way to write tests for it. Not only did I want to make sure it functioned as intended, but I wanted to be able to ensure that I didn’t break functionality down the road if I added new features/refactored. The question then became, how do you best write tests for code generators? I could obviously write unit level tests for the code generator itself, but I wanted to go higher level and verify that it was in fact generating valid go code as whole files.

Again after some searching, I came across a pattern of testing using something called ‘Golden Files’. Basically the idea is that you compare your test results to a known good file, the golden file, byte by byte. This works especially well for code generation since the output is itself a file. This technique is discussed in more depth in Mitchell Hashimoto’s - Advanced Testing with Go talk from Gophercon 2017.

I implemented this pattern by first generating the golden files for a couple types, one primitive and two user defined, and placing them in the testdata folder. Then, I wrote a test golden_test.go that reads in each golden file and compares it to it’s generated in-memory counterpart line by line:

g := generator{
    packageName: test.packageName,
    outputName:  test.outputName,
    typeName:    test.typeName,
}

output, err := g.generate()
if err != nil {
    t.Error(err)
}

input, err := ioutil.ReadFile("./testdata/" + test.filename)
if err != nil {
    t.Error(err)
}

want := bufio.NewScanner(bytes.NewReader(input))
got := bufio.NewScanner(bytes.NewReader(output))

count := 0

for want.Scan() {
    got.Scan()

    // skip comment lines at top
    if count < 2 {
        count++
        continue
    }

    assert.Equal(t, want.Bytes(), got.Bytes())
    count++
}

If any of the bytes do not match, the test will fail and I’ll know I introduced a bug or broke some backwards compatibility.

Try it Out

Hopefully this post helped show you the power of go generate, as well as some new techniques in working with templates and golden files.

Try using the optional library in some of your code to perhaps reduce some boilerplate, or generate your own types using the optional command and let me know what you think!