I’ve decided, for the sake of insanity, to work my way through the O’Reilly Programming Rust book, since I feel like my Rust has become, shall we say, rusty. And I’m doing it with my usual sense of attitude, that "There has to be a better way!"

First, I took this mess from the very first example (another greatest common denominator function… you’d think tutorial writers would figure out most of us would rather write something useful, ne):

let mut numbers == Vec::new();
for arg in std::env::args().skip(1) {
        numbers.push(u64::from_str(&arg)
                     .expect('error parsing argument'));
}

And said, "I know there are way to initialize and map values automatically in other languages. Rust should too." And I dug through the documentation and discovered, why yes, Rust does:

let numbers = Vec::from_iter(
    std::env::args().skip(1)
        .map(|x| u64::from_str(&x).expect("Error parsing argument")));

That’s pretty dense, but it does the job, and it uses the exact same allocations, too.

The next step was the actual reduction (I’d already written the GCD function in a library):

let mut d = numbers[0];
for m in &numbers[1..] {
    d = gcd(d, *m);
}

Could I one-liner that? Why yes:

    let d = &numbers[1..].iter().fold(numbers[0], |acc, x| gcd(acc, *x));

I know there are people who hate this kind of programming, but I dunno, I find it much more readable. It says that d is a reduction via GCD of all the numbers in a container. It’s actually less confusing than the loop.

To me, at least.

I spent much of the first day of my sabbatical at Go Northwest, a conference sponsored by my previous employer, and one for which I already had tickets. It was somewhat informative, although I learned more about JSON Web Tokens and Macaroons (thank you, Tess Rinearson) than I did anything at all about Go, the programming language. Most of the Go-related stuff seemed to be fairly high level and well within my skillset. I did learn a bit about blockchain applications and still remain skeptical that there are any use cases for them that don’t involve trying to hide stuff from government agencies.

One thing that I did pay strong attention to was David Crenshaw, a guy who had done what I did: spent a year not working for anyone, only to start up a whole bunch of passive income opportunities. And one thing he said stuck with me: "Almost all of the solutions being sold today are distributed systems but, except for a small class of business problems, even smaller than you think, nobody needs them. Very few problems exceed the size of a single large instance." (An Amazon "large" is a two-core CPU with 8GB of RAM.) This has often been my experience, too; there’s nothing in the world that needs a fifty-instance fleet.

The only reason to really worry that much about this sort of thing is when you need geographic distribution for either performance or reliability reasons. But geographic distribution and contend delivery networking is a different problem from what most distributed systems are trying to solve. You don’t even need that much observability if your system is fairly small.

I firmly believe Crenshaw’s right, and that the state-of-the-art in medium-sized deployments is being ignored while people chase the Kubernetes money. Google and Amazon have every right and need to do that; they have money and engineering time to burn. The rest of us don’t.

So, today I did I thing I’ve never done before. I quit.

In all my career as a software developer, I’ve never quit from a position like this. In college I quit a few jobs that weren’t software development, such as the warehouse job, the data entry position, and the pizza delivery service. I’ve quit a few small start-ups that weren’t paying me what I was worth, but then at the time nobody was getting paid what they were theoretically worth, and every single one of those start-ups was still incredibly valuable: they let me keep my resume alive, and they let me learn useful skills. But all the big job endings, from CompuServe, F5, and Isilon, had been either shutdowns or layoffs. CompuServe was just axed by some holding company owned by AOL. F5 and Isilon let me go during the massive layoffs of the Big Tech Bubble and the Great Recession, respectively.

I’ve never just… left. Certainly not when the opportunities at the company were pretty good. At Isilon I was definitely going stale, but the same can’t be said for F5 or CompuServe. At those, I’d been learning great things up until the very end. At F5 we’d had a great ANTLR project underway, and at CompuServe I’d been knee-deep into making contributions to the Python standard library and maintaining the original mod-logging library documentation for Apache.

Today, I quit because I’d lost my niche.

Today I left my job a Splunk for no other reason than that I wanted to. I’d been learning a fantastic array of new stuff. I had Kubernetes deployments working reliably, was deploying them on AWS, had transitioned away from Ansible onto something using Terraform and Ksonnet, was writing a Kubernetes-based microservice in Golang to gather billing information for tenanted systems, doing internal logging and using the Prometheus client libraries for observability (but actually scraping the data into Splunk instead, which was hella fun), and had a Grafana-to-Splunk dashboard transformer kinda sorta working. Codeship, Gitlab, Jenkins, Swagger, Docker, Make, Go, you name it, I’m doing it. And a lot of Python, too, as it remains the official language of Splunk Enterprise Engineering. Oh, and the front-end of our proxy is written in React, too, which they gave to me because “You’re a UI guy, right?”

I was a UI guy. In fact, that was why Splunk hired me in the first place. I’d literally written the book on writing SPAs (Single Page Applications) using Backbone, JQuery, and REST, after learning how to do that at a previous start-up. I had a niche as an expert single-page-application developer using Python middle tiers and SQL back-ends. I could, in a pinch, even do it in Ruby on Rails, but it would take a week, whereas if you gave me Django and Postgresql I would have it up and running in an hour.

I had an elevator pitch. You can still read it. I was someone who could do the whole stack, even on AWS, well enough to get your start-up off the ground. I did it five times: two failures, two lifestyle successes, and one runaway success that allowed me to survive the 2008 recession, paying for Omaha’s medicines and the kids’ schooling, without having to tighten my belt too much.

Looking at the list of skills I’ve got from Splunk, I can’t say that I have an elevator pitch anymore. Everything I’ve been doing for the past six months has been too fast, too new, too much all at once for me to develop a sense of expertise in anything— well, except for writing a Prometheus-instrumented microservice in Go. Maybe that’s not a bad niche, since the bulk of software development is just plumbing anyway these days, a paint-by-numbers process of input, map, filter, output. The problem is that anyone can do plumbing. I like to think I’m better at the really hard problem in that area of development than most people– but then I’m reminded that most people don’t care about readable code anymore. It’s all replaceable anyway.

Part of this coming year will be to figure out what my next elevator pitch will be, and if there’s a place in the industry where I want to pitch it. I was good at industrial websites, and I was good at server design, and I was good at UIs for embedded systems. The question is: what will I be good at in 2020? What will I need to be good at in 2020?

I guess I’m about to find out.

29Jun

Simple At The Bottom

Posted by Elf Sternberg as Uncategorized

There’s a quote by Rich Hickey, the creator of the Clojure Programming Language, floating around the Internet that goes like this:

Simplicity is hard work. But, there’s a huge payoff. The person who has a genuinely simpler system – a system made out of genuinely simple parts, is going to be able to effect the greatest change with the least work. He’s going to kick your ass. He’s gonna spend more time simplifying things up front and in the long haul he’s gonna wipe the plate with you because he’ll have that ability to change things when you’re struggling to push elephants around.

I’ve been thinking about this in the context of Elon Musk’s venture and adventures, especially the gushing press coverage from more recent books that talk about Musk’s dedication to "first principles," and the idea, which so far seems to be paying off for Musk, that people are doing things not because they’re correct, but because they’re familiar.

SpaceX is the obvious example. Musk ran the numbers, realized that the material cost of a launch vehicle is 4% of its cost, asked why launches are so damned expensive, and set out to prove that they shouldn’t be. He’s trying to do the same thing with tunneling machines, subways, electric cars, solar roofs, and batteries.

I’m a software developer who’s long felt more of a kinship for computer science than for computer engineering. And the longer I work in this business, the more I feel like there’s a first principles issue that is missing somewhere.

For example, in the early 2000’s there was a lot of buzz about "object oriented databases." They were all the rage until someone tried to implement them at scale, and the processing cost of updating all those rows every time a change happened was enormous; the relationship between objects in an OODB constituted an incredibly expensive forest of directed acyclic graphs to maintain. And yet, at the same time, everyone was working with spreadsheets, and underneath the grid, a spreadsheet is just a forest of directed acyclic graphs. The secret, it turned out, was to only update the parts you could see; calculated cell values that were out of sight and didn’t affect the view didn’t matter.

The technical term for updating only what output the user currently wants is laziness. OODBs work just fine so long as the results are lazy.

I have this nagging notion that at the user layer, almost everything is over-engineered to be easy rather than simple. That there’s a missing idea. That many of the features we see in applications: the DAGs of spreadsheets and garbage collectors, the page catalogs of databases, the piece tables and gap buffers of word processors, and so forth, would be significantly easier to understand if all the weirdness of it, the humanness of it, were boiled down to a couple of declarative tables that explained to the machine what the human thought these terms meant.

Because underneath it all, every programming language in the world is semantic sugar around memory allocation, assignment, loops, and conditions. And if your language has first-class functions, tail calls and pattern matching, you’ve replaced your loops and conditions with something smarter.

And Clojure isn’t it. Because Clojure isn’t simple at the bottom. Clojure is Java at the bottom. The Lisp Reader in Clojure is LispReader.java, and to me, that screams that there’s more work to be done.

I have a problem with the shiny. It’s the whole ADHD/Interictal thing interacting. There are so many things I want to learn and I haven’t got the time to learn all of them. Right now, I’m going back to a well I’ve gone to a number of times and dived deep into interpreters and… other things.

Current Learning Spree:

SICP (Again!)

Last week I made my way up to the end of Chapter 2 of The Structure and Interpretation of Computer Languages. My impression, after finishing Chapter 2, is I now get why Haskell and Lisp are lumped together as "functional languages," but, as a writer, I can say that the theme and premise of both languages is very different. I can start to see just how easy it would be to implement both an Object Oriented language in Lisp, and how easy it would be to implement Hindley-Milner in classical McCarthy Lisp or its derivatives like Racket and CL, and also why it would be a mistake to do so.

The generic interfaces of classical Scheme seem like a lot of typing. The amount of typing that one has to do, as well as the mystery types of classic Lisp’s unlabled tuples, are both ergonomic hitches that a postmodern Lisp has to overcome, and I’m not sure how.

Build Systems à la Carte

I also read the first ten pages of Mokhov, Mitchell, and Peyton-Jones’ paper Build Systems à la Carte, a lovely little paper about 30 pages long in which the authors try to prove (and do a pretty good job, all things considered) of trying to find the abstraction in build systems. They do a pretty good job, creating a common vocabulary for build systems that not only encompasses classical systems like xmkmf and make and even ninja, but somehow manages to encompass Microsoft Excel as well!

It occurred to me as I was reading it that if "the store" is the unification of the (local) repository and the filesystem, then version control systems are also build systems with narrow task capability: the tasks function’s job is to, for a given hash, drive the filesystem to match that hatch. There’s an abstraction layer here that’s backward looking, rather than MMP-J’s forward looking, but I can feel there’s a commonality here.

Parsing With First Class Derivatives

I’ve been ooh-shinied a lot this week. My Rust skills are getting remarkably rusty as I neglect them, but I want to get back into them. The paper Parsing With First Class Derivatives might be just the hook I need. The examples look Ocaml-ish, but I think I can parse them enough to get a Rusty version working, if I’m crazy enough. What attracted me most to the paper was section 3.3, which seems to imply a principled way to tackle Landin’s "Offside Rule," which is important for whitespace-delimited languages like Python, YAML, and Coffeescript.

Don’t ask why I care. You won’t like the answer.

The first two parts of my swagger tutorial [Part 1, Part 2] were dedicated to the straightforward art of getting swagger up and running. While I hope they’re helpful, the whole point of those was to get you to the point where you had the Timezone project, so I could show you how to add Command Line Arguments to a Swagger microservice.

One thing that I emphasized in Go Swagger Part 2 was that configure_timeofday.go is the only file you should be touching, it’s the interface between the server and your business logic. Every example of adding new flags to the command line, even the one provided by the GoSwagger authors, starts by modifying the file cmd/<project>-server/main.go, one of those files clearly marked // DO NOT EDIT.

We’re not going to edit files marked // DO NOT EDIT.

To understand the issue, though, we have to understand the tool swagger uses for handling command line arguments, go-flags.

Go-Flags

go-flags is the tool Swagger uses by default for handling command line arguments. It’s a clever tool that uses Go’s tags and reflection features to encode the details of the CLI directly into a structure that will hold the options passed in on the command line.

The implementation

We’re going to add a single feature: the default timezone. In Part 2, we hard-coded the default timezone into the handler, but what if we want to change the default timezone more readily than recompiling the binary every time? The Go, Docker, and Kubernetes crowd argue that that’s acceptable, but I still want more flexibility.

To start, we’re going to open a new file it the folder with our handlers and add a new file, timezone.go. We’re going to put a single tagged structure with a single field to hold our timezone CLI argument, and we’re going to use go-flags protocol to describe our new command line flag.

Here’s the whole file:

<<timezone.go>>=
package timeofday
type Timezone struct {
    Timezone string  `long:"timezone" short:"t" description:"The default time zone" env:"GOTIME_DEFAULT_TIMEZONE" default:"UTC"`
}
@

If you want to know what you can do with go-flags, open the file ./restapi/server.go and examine the Server struct there, and compare its contents to what you see when you type timeofday-server --help. You can learn a lot by reading even the generated source code. As always, // DO NOT EDIT THIS FILE.

Next, go into configure_timeofday.go, and find the function configureFlags. This, unsurprisingly, is where this feature is supposed to go.

We’ve already imported the timeofday package, so we have access to our new Timezone type. Right above configureFlags, let’s create an instance of this struct and populate it with defaults:

<<add timezone to configure_timeofday.go>>=
var Timezone = timeofday.Timezone{
    Timezone: "UTC",
}
@

See the comment in configureFlags? Note that swag package? You’ll have to add it to the imports. It should be already present, as it came with the rest of the swagger installation. Just add:

<<new import for swag>>=
    swag "github.com/go-openapi/swag"
@

And now modify configureFlags():

<<rewrite configureFlags>>=
func configureFlags(api *operations.TimeofdayAPI) {
    api.CommandLineOptionsGroups = []swag.CommandLineOptionsGroup{
        swag.CommandLineOptionsGroup{
            ShortDescription: "Time Of Day Service Options",
            Options: &Timezone,
        },
    }
}
@

See that ShortDescription there? When you run the --help option for the server, you’ll see a section labeled "Application Options", and another labeled "Help Options". We’re adding a new section, "Service Options", which will include our customizations. This conceptually allows us to distinguish between routine options of a microservice, and the specific options of this microservice.

Always distinguish between your framework and your business logic. (I’ve often seen this written as "always distinguish between execution exceptions and business exceptions," and it’s great advice is similarly here.)

You can now build your server (go build ./cmd/timeofday-server/), and run it (./timeofday-server --help), and you’ll see your new options. Of course, they don’t do anything, we haven’t modified your business logic!

The Context Problem

This is where most people have a problem. How do the values that now populate the Timezone struct make their way down to the handlers? There are a number of ways to do this. The "edit main.go" people just make it a global variable available to the whole server, but I’m here to tell you doing so is sad and you should feel sad if you do it. What we have here, in our structure that holds our CLI options, is a context. How do we set the context?

The correct way is to modify the handlers so they have the context when they’re called upon. The way we do that is via the oldest object-oriented technique of all time, one that dates all the way back to 1964 and the invention of Lisp: closures. A closure wraps one or more functions in an environment (a collection of variables outside those functions), and preserves handles to those variables even when those functions are passed out of the environment as references. A garbage-collected language like Go makes this an especially powerful technique because it means that anything in the environment for which you don’t keep handles will get collected, leaving only what matters.

So, let’s do it. Remember these lines in configure_timeofday.go, from way back?

    api.TestGetHandler = operations.TestGetHandlerFunc(func(params operations.TestGetParams) middleware.Responder {
        return middleware.NotImplemented("operation .TestGet has not yet been implemented")
    })

See that function that actually gets passed to TestHandlerGetFunc()? It’s anonymous. We broke it out and gave it a name and stuff and filled it out with business logic and made it work. We’re going to go back and replace those lines, again, so they look like this:

    api.TimeGetHandler = operations.TimeGetHandlerFunc(timeofday.GetTime(&Timezone))
    api.TimePostHandler = operations.TimePostHandlerFunc(timeofday.PostTime(&Timezone))

Those are no longer references to functions. They’re function calls! What do those functions return? Well, we know TimeGetHandlerFunc() is expecting a reference to a function, so that function call had better return a reference to a function.

And indeed it does:

func GetTime(timezone *Timezone) func(operations.TimeGetParams) middleware.Responder{
    defaultTZ := timezone.Timezone

    // Here's the function we return:
    return func(params operations.TimeGetParams) middleware.Responder {
        // Everything else is the same... except we need *two* levels of
        // closing } at the end!

Now, instead of returning a function defined at compile time, we returned a function reference that is finalized when GetTime() is called, and it now holds a permanent reference to our Timezone object. Do the same thing for PostTime.

There’s one more thing we have to do. We’ve moved our default timezone to the configure_timeofday.go file, so we don’t need it here anymore:

func getTimeOfDay(tz *string) (*string, error) {
    t := time.Now()
    utc, err := time.LoadLocation(*tz)
    if err != nil {
        return nil, errors.New(fmt.Sprintf("Time zone not found: %s", *tz))
    }

    thetime := t.In(utc).String()
    return &thetime, nil
}

And that’s it. That’s everything. You can add all the command line arguments you want, and only preserve the fields that are relevant to the particular handler you’re going to invoke.

You can now build and run the server, but with a command line:

$ go build ./cmd/timeofday-server/
$ ./timeofday-server --port=8080 --timezone="America/Los_Angeles"

And test it with curl:

$ curl 'http://localhost:8020/timeofday/v1/time?timezone=America/New_York'
{"timeofday":"2018-03-30 23:44:47.701895604 -0400 EDT"}
$ curl 'http://localhost:8020/timeofday/v1/time'
{"timeofday":"2018-03-30 20:44:54.525313806 -0700 PDT"}

Note that the default timezone is now PDT, or Pacific Daily Time, which corresponds to the America/Los_Angeles entry in the database in late March.

And that’s how you add command line arguments to Swagger servers correctly without exposing your CLI settings to every other function in your server. If you want to see the entirety of the source code, the advanced version on the repository has it all.

Review of Part One

In Part One of Go-Swagger, we generated a on OpenAPI 2.0 server with REST endpoints. The server builds and responds to queries, but every valid query ends with “This feature has not yet been implemented.”

It’s time to implement the feature.

I want to emphasize that with Go Swagger there is only one generated file you need to touch. Since our project is named timezone, the file will be named restapi/configure_timezone.go. Our first step will be to break those “not implemented” functions out into their own Go package. That package will be our business logic. The configure file and the business logic package will be the only things we change.

A reminder: The final source code for this project is available on Github, however Parts One & Two deal with the most common implementation, a server with hard-coded default values. For these chapters, please consult that specific version of the code.

Break out the business logic

Create a new folder in your project root and call it timeofday.

Open up your editor and find the file restapi/configure_timeofday.go. In your swagger.yml file you created two endpoints and gave them each an operationId: TimekPost and TimeGet. Inside configure_timeofday.go, you should find two corresponding assignments in the function configureAPI(): TimeGetHandlerFunc and ClockPostHandlerFunc. Inside those function calls, you’ll find anonymous functions.

I want you to take those anonymous functions, cut them out, and paste them into a new file inside the timeofday/ folder. You will also have to create a package name and import any packages being used. Now your file, which I’ve called timeofday/handlers.go, looks like this (note that you’ll have to change your import paths as you’re probably not elfsternberg. Heck, I’m probably not elfsternberg):

<<handlers.go before implementation>>=
package timeofday

import(
  "github.com/go-openapi/runtime/middleware"
  "github.com/elfsternberg/timeofday/restapi/operations"
)

func GetTime(params operations.TimeGetParams) middleware.Responder {
  return middleware.NotImplemented("operation .TimeGet has not yet been implemented")
}

func PostTime(params operations.TimePostParams) middleware.Responder {
  return middleware.NotImplemented("operation .TimePost has not yet been implemented")
}
@

And now go back to restapi/configure_timeofday.go, add github.com/elfsternberg/timeofday/clock to the imports, and change the handler lines to look like this:

<<configuration lines before implementation>>=
    api.TimeGetHandler = operations.TimeGetHandlerFunc(timeofday.GetTime)
    api.TimePostHandler = operations.TimePostHandlerFunc(timeofday.PostTime)
@

Implementation

Believe it or not, you’ve now done everything you need to do except the business logic. We’re going to honor the point of OpenAPI and the `// DO NOT EDIT“ comments, and not modify anything exceept the contents of our handler.

To understand our code, though, we’re going to have to read some of those files. Let’s go look at /models. In here, you’ll find the schemas you outlined in the swagger.yml file turned into source code. If you open one, like many files generated by Swagger, you’ll see it reads // DO NOT EDIT. But then there’s that function there, Validate(). What if you want to do advanced validation for custom patterns or inter-field relations not covered by Swagger’s validators?

Well, you’ll have to edit this file. And figure out how to live with it. We’re not going to do that here. This exercise is about not editing those files. But we can see, for example, that the Timezone object has a field, Timezone.Timezone, which is a string, and which has to be at least three bytes long.

The other thing you’ll have to look at is the restapi/operations folder. In here you’ll find GET and POST operations, the parameters they accept, the responses they deliver, and lots of functions only Swagger cares about. But there are a few we care about.

Here’s how we craft the GET response. Inside handlers.go, you’re going to need to extract the requested timezone, get the time of day, and then return either a success message or an error message. Looking in the operations files, there are a methods for good and bad returns, as we described in the swagger file.

<<gettime implementation>>=
func GetTime(params operations.TimeGetParams) middleware.Responder {
    var tz *string = nil

    if (params.Timezone != nil) {
        tz = params.Timezone
    }

    thetime, err := getTimeOfDay(params.Timezone)

@

The first thing to notice here is the params field: we’re getting a customized, tightly bound object from the server. There’s no hope of abstraction here. The next is that we made the Timezone input optional, so here we have to check if it’s nil or not. if it isn’t, we need to set it. We do this here because we need to cast params.Timezone into a pointer to a string, because Go is weird about types.

We then call a (thus far undefined) function called getTimeOfDay.

Let’s deal with the error case:

<<gettime implementation>>=
    if err != nil {
        return operations.NewTimeGetNotFound().WithPayload(
            &models.ErrorResponse {
                int32(operations.TimeGetNotFoundCode),
                swag.String(fmt.Sprintf("%s", err)),
            })
    }
@

That’s a lot of references. We have a model, an operation, and what’s that “swag” thing? In order to satisfy Swagger’s strictness, we use only what Swagger offers: for our 404 case, we didn’t find the timezone requested, so we’re returning the ErrorResponse model populated with a numeric code and a string, extracted via fmt, from the err returned from our time function. The 404 case for get is called, yes, NewClockGetNotFound, and then WithPayload() decorates the body of the response with content.

The good path is similar:

<<gettime implementation>>=
    return operations.NewClockGetOK().WithPayload(
        &models.Timeofday{
            Timeofday: *thetime,
        })
}
@

Now might be a good time to go look in models/ and /restapi/options, to see what’s available to you. You’ll need to do so anyway, because unless you go to the git repository and cheat, I’m going to leave it up to you to implement the PostTime().

There’s still one thing missing, though: the actual time of day. We’ll need a default, and we’ll need to test to see if the default is needed. The implementation is straightforward:

<<timeofday function>>=
func getTimeOfDay(tz *string) (*string, error) {
        defaultTZ := "UTC"

        t := time.Now()
        if tz == nil {
                tz = &defaultTZ
        }

        utc, err := time.LoadLocation(*tz)
        if err != nil {
                return nil, errors.New(fmt.Sprintf("Time zone not found: %s", *tz))
        }

        thetime := t.In(utc).String()
        return &thetime, nil
}
@

Now, if you’ve written everything correctly, and the compiler admits that you have (or you can cheat and download the 0.2.0-tagged version from the the repo), you’ll be able to build, compile, and run the server, and see it working:

$ go build ./cmd/timeofday-server/
$ ./timeofday-server --port=8080

And then test it with curl:

$ curl 'http://localhost:8020/timeofday/v1/time'
{"timeofday":"2018-03-31 02:57:48.814683375 +0000 UTC"}
$ curl 'http://localhost:8020/timeofday/v1/time?timezone=UTC'
{"timeofday":"2018-03-31 02:57:50.443200906 +0000 UTC"}
$ curl 'http://localhost:8020/timeofday/v1/time?timezone=America/Los_Angeles'
{"timeofday":"2018-03-30 19:57:59.886650128 -0700 PDT"}

And that’s the end of Part 2. If you’ve gotten this far, congratulations! Just a reminder, a working version of this server is available under the “0.2.0” tag at the repo.

On to Part 3

Introduction

${WORK} has me writing microservices in Go, using OpenAPI 2.0 / Swagger. While I’m not a fan of Go (that’s a bit of an understatement) I get why Go is popular with enterprise managers, it does exactly what it says it does. It’s syntactically hideous. I’m perfectly happy taking a paycheck to write in it, and I’m pretty good at it already. I just wouldn’t choose it for a personal project.

But if you’re writing microservices for enterprise customers, yes, you should use Go, and yes, you should use OpenAPI and Swagger. So here’s how it’s done.

All of the files for this tutorial are available from the elfsternberg/go-swagger-tutorial repo at github. There are two phases to this tutorial, and the first phase is the base Go Swagger implementation. I strongly recommend that if you’re going to check out the source code in its entirety, that you start with the Basic Version, and only check out the Advanced version when you get to Part 3.

Just be aware that if you see stuff that looks like <<this>>, or a single @ alone on a line, that’s just part of my code layout; do not include those in your source code, they’re not part of Go or Swagger. Sorry about that.

Go Swagger!

Swagger is a specification that describes the ndpoints for a webserver’s API, usually a REST-based API. HTTP uses verbs (GET, PUT, POST, DELETE) and endpoints (/like/this) to describe things your service handles and the operations that can be performed against it.

Swagger starts with a file, written in JSON or YAML, that names each and every endpoint, the verbs that endpoint responds to, the parameters that endpoint requires and takes optionally, and the possible responses, with type information for every field in the inputs and outputs.

Swagger tooling then takes that file and generates a server ready to handle all those transactions. The parameters specified in the specification file are turned into function calls and populated with "Not implemented" as the only thing they return.

Your job

In short, for a basic microservice, it’s your job to replace those functions with your business logic.

There are three things that are your responsibility:

  1. Write the specification that describes exactly what the server accepts as requests and returns as responses, and generate a server from this specification.
  2. Write the business logic.
  3. Glue the business logic into the generated server.

In Go-Swagger, there is exactly one file in the generated code that you need to change. Every other file is labeled "DO NOT EDIT." This one file, called configure_project.go, has a top line that says "This file is safe to edit, and will not be replaced if you re-run swagger." That exactly one file should be the only one you ever need to change.

The setup

You’ll need Go. I’m not going to go into setting up Go on your system; there are perfectly adequate guides elsewhere. You will need to install swagger and dep.

Once you’ve set up your Go environment (set up $GOPATH and $PATH), you can just:

$ go get -u github.com/golang/dep/cmd/dep
$ go get -u github.com/go-swagger/go-swagger/cmd/swagger

Initialization

Now you’re going to create a new project. Do it in your src directory somewhere, under your $GOPATH.

$ mkdir project timeofday
$ cd timeofday
$ git init && git commit --allow-empty -m "Init commit: Swagger Time of Day."
$ swagger init spec --format=yaml --title="Timeofday" --description="A silly time-of-day microservice"

You will now find a new swagger file in your project directory. If you open it up, you’ll see short header describing the basic features Swagger needs to understand your project.

Operations

Swagger works with operations, which is a combination of a verb and an endpoint. We’re going to have two operations which do the same thing: return the time of day. The two operations use the same endpoint, but different verbs: GET and POST. The GET argument takes an optional timezone as a search option; the POST argument takes an optional timezone as a JSON argument in the body of the POST.

First, let’s version our API. You do that with Basepaths:

<<version the API>>=
basePath: /timeofday/v1
@

Now that we have a base path that versions our API, we want to define our endpoint. The URL will ultimately be /timeofday/v1/time, and we want to handle both GET and POST requests, and our responses are going to be Success: Time of day or Timezone Not Found.

<<define the paths>>=
paths:
  /time:
    get:
      operationId: "GetTime"
      parameters:
        - in: path
          name: Timezone
          schema:
            type: string
            minLength: 3
      responses:
        200:
          schema:
            $ref: "#/definitions/TimeOfDay"
        404:
          schema:
            $ref: "#/definitions/NotFound"
    post:
      operationId: "PostTime"
      parameters:
        - in: body
          name: Timezone
          schema:
            $ref: "#/definitions/Timezone"
      responses:
        200:
          schema:
            $ref: "#/definitions/TimeOfDay"
        404:
          schema:
            $ref: "#/definitions/NotFound"
@

The $ref entries are a YAML thing for referring to something else. The octothorpe symbol (#) indicates "look in the current file." So now we have to create those paths:

<<schemas>>=
definitions:
  NotFound:
    type: object
    properties:
      code:
        type: integer
      message:
        type: string
  Timezone:
    type: object
    properties:
      Timezone:
        type: string
        minLength: 3
  TimeOfDay:
    type: object
    properties:
      TimeOfDay: string
@

This is really verbose, but on the other hand it is undeniably complete: these are the things we take in, and the things we respond with.

So now your file looks like this:

<<swagger.yml>>=
swagger: "2.0"
info:
  version: 0.1.0
  title: timeofday
produces:
  - application/json
consumes:
  - application/json
schemes:
  - http

# Everything above this line was generated by the swagger command.
# Everything below this line you have to add:

<<version the API>>

<<schemas>>

<<define the paths>>
@

Now that you have that, it’s time to generate the server!

$ swagger generate server -f swagger.yml

It will spill out the actions it takes as it generates your new REST server. Do not follow the advice at the end of the output. There’s a better way. Use dep, which will automagically find all your dependencies for you, download them to a project-specific vendor/ folder, and lock the specific commit in the record so version creep won’t break your project in the future. dep has become even Google’s recommended dependency control mechanism. Just run:

$ dep init

Dependency management in Go is a bit of a mess, but the accepted solution now is to use dep rather than go get. This creates a pair of files, one describing the Go packages that your file uses, and one describing the exact versions of those packages that you last downloaded and used in the ./vendor/ directory under your project root.

Now you can build the server:

$ go build ./cmd/timeofday-server/

And then you can run it. Feel free to change the port number:

$ ./timeofday-server --port=8082

You can now tickle the server:

$ curl http://localhost:8082/
{"code":404,"message":"path / was not found"}
$ curl http://localhost:8082/timeofday/v1/time
" function .GetTime is not implemented"

Congratulations! You have a working REST server that does, well, nothing.

For Part 2, we’ll make our server actually do things.

27Mar

Engineering Notebook: More Rust Basics

Posted by Elf Sternberg as Uncategorized

Continuing on with my engineering notebook, here’s what I’ve been learning about Rust recently. Mostly still following Beingessner’s book, which is about memory management primitives in Rust.

The first thing, as I covered last time is that the book teaches Rust’s peculiar (but excellent) flavor of memory management. Along the way, it teaches Rust’s sum types (using enum as its keyword), matching with completion, and Rust’s maybe types (using Option<>).

One thing it doesn’t teach is about is tuple structs. I had to look them up; I keep encountering them in other people’s code, but they weren’t obvious to me and I couldn’t remember seeing them in The Rust Programming Language, so I kept calling them "tuple types," but no, they’re called tuple structs, and you access their internals via a dotted index:

struct Color(i32, i32, i32); // Red, Green, Blue
let lightskyblue = Color(135, 206, 250);
let amount_of_blue = lightskyblue.2;

Box<T>: The thing with Rust is that you still have to care, a lot, about whether something is on the stack or the heap. Using the Box<T> type automatically puts something on the heap, and is tracked in such a way that Rust deletes it automatically when the handle to it goes out of scope.

Rc<T> is a wrapper around Box that allows for multiple handles to point to T, counting the number of handles referring to T, and automatically deleting T when the count goes to zero.

Arc<T> is just like Rc<T> except that it also wraps the counter in a mutex handler and allows for multiple threads to reference the object safely.

RefCell<T> allows one to borrow a referenced object from inside an Rc or Arc or other immutable container. This allows the user to mutate the contents of what would other be an immutable. When using RefCell<T>, you must use .borrow() or .borrow_mut() to "dynamically borrow" and object that normally would not be borrow-able.

Rust continues to be… weird. This one in particular gets me:

pub struct List<T> { head: Link<T>, }
type Link<T> = Option<Rc<Node<T>>>;
struct Node<T> { elem: T, next: Link<T>, }
pub struct Iter<'a, T:'a> { next: Option<&'a Node<T>>, }

// In body of impl<T> List<T>:
pub fn iter(&self) -> Iter<T> {
        Iter { next: self.head.as_ref().map(|node| &**node) }
}

I mean, I get that Rc is a pointer, and as_ref turns the head object into a reference to the head object, and map only works if node is not None, so we have to dereference it twice, and then to make Rustc happy that we’re saving a reference mark it as a reference to this twice-dereferenced thing, but… oy. That just looks ugly.

And yes, I get that I have to internalize all this before I can even begin to write Rust meaningfully.

24Mar

Programmer Competency Matrix

Posted by Elf Sternberg as Uncategorized

So I’ve been looking at Sijin Joseph’s “Programmer competency matrix,” and I’m going to go through it because I kinda want to know where I stand. But I have to tell Sijin once thing: information architecture is something he needs to improve on. By using the first person in his “about me” page, he somehow managed to avoid telling people his name throughout most of the blog! As a recovering web designer, I’m shocked whenever I see someone’s blog or website work really really hard to avoid putting its identity in the header.

So let me go through the matrix.

Computer Science

Data Structures: Level 2, almost 3

I’m a little weak on some data structures, especially those that are hybrids, such as an efficient Least-Recently-Used hashtabe, as well as those that lead to persistent storage.

Algorithms: Level 2, almost 3

More or less the same thing.

Systems Programming: Level 2, almost 3

Again, I feel like I’m weak with some of the features of systems production, such as dynamic linking, JIT compilation, and garbage collection. I know what those things are, and I’ve implemented primitive versions of them, but things like generational garbage collection and peephole optimization aren’t yet in my vocabulary.

Software Engineering

Version Control: Level 3

I use Git for everything. I started with CVS, then SVN, now Git. I use Git for tracking my documentation. Hell, I use Git Flow to track my fiction writing!

Build Automation: Level 3

I’ve written build automation systems. I understand how build systems can be partial or complete; can be topological, reordering, or recursive; can have static or dynamic depedencies; can perform a cutoff if a dependency is identical to the previous iteration and therefore its dependents don’t need rebuilding.

Automated Testing: Level 2

I love getting into TDD when a problem becomes complex enough that functional products need to be defined, described, and generated. I don’t have much strength with UI or performance tests.

Problem Decomposition: Level 3

Also: knows damn well when not to implement the super-clever hyper-abstract generic solution when the next guy who has to look at and maintain my code isn’t a level 3!

Systems Decomposition: Level 3

But… I have a tendency to work only at level 2. I don’t enjoy doing heavy integration at the process or container level; it’s abstract and distant, and feels a bit like abdication to entropy.

Communication: Level 2

It depends on the kind of communication. I have no trouble describing what I want to buid, or am going to build; I have more than once saved a project by being the only one on the team who knows how to drive a UML rendering tool. But sprint planning, negotiating for who builds what when, that’s always rough.

Code organization: Level 3

I work really hard to make my code readable and beautiful. I know the next person to read it will hopefully appreciate it. I use a thesaurus constantly to make sure I’ve used exactly the right name for a variable or other symbol.

Source tree organization: Level 3

I tend to work simply, with the best practices of my industry in mind. Kinda a reason I don’t like Go: there are two “best practices,” and they’re completely at odds with each other.

Code readability: Level 3

Cyclomatic complexity checks for the win! This one is actually a bit hard to hit in my favorite language, Lisp, because it encourages that kind of complexity with anonymous functions and efficient local dispatch, but I manage.

Defensive coding: Level 2

I’m okay at it, and I know what to look for. I use linters and other semantic checkers a lot. But I’m definitely falling back on common tools, rather than custom tools, for my efforts.

Error handling: Level 2

I’d love to be level 3, but level 3 is only possible in languages the industry doesn’t use.

IDE: Level 3

I work in Emacs. What do you think?

API: Level 2

I’m not sure what he means here by “API.” Does he mean the ones we use in common practice, like HTTP and REST and GRPC? Sure, I know those. But what about the ones built on top, like OpenAPI? Those are as varied as the use cases. Does he mean LibC? I don’t work much in LibC these days.

Frameworks: Level 2

I consider that a badge of honor. There are too many frameworks out there. The only reason to write a framework these days is to fill in a gap in a language that doesn’t already have one. Frameworks are like Patterns: holes in the language.

Requirements: Level 3

This actually goes back to “communication.” Because I know Rational Rose, I know how to extract requirements and scope them out better than most people.

Scripting: Level 3

Just go look at my Github!

Database: Level 2, almost 3

My SQL is pretty good; I’ve even published solid and testable extensions to Postgres. I do know some optimizations and can talk about storage engines, but the industry is getting pretty fuzzy these days.

Experience

Languages with professional expertise: Level 2, almost 3

And yes, I hit the bonus round. I can talk about monads.

Platforms with professional experience: Level 2

I’ve been pretty monogamous about my platforms. That gives me a lot of depth when talking about Unix-based services on AWS, but yeah, I can’t talk too much about Azure or GKE at the moment.

Years of professional experience: Level 3

Yep. More than ten years of experiences.

Knowledge:

Tools: Level 3

Yep, written tools other people use. Knows better than to use an ORM.

Languages: Level 3

Try me.

Codebase: Level 2

Here’s the thing about Sijin’s “level 3:” I don’t believe this is a valuable skill. If your codebase has this kind of headaround problem, you have a codebase with poor decomposition.

Upcoming technologies: Level 3

I tend to be a neophile and love playing with new things. Unfortunately, I often love playing with new obscure things like Typed Assembly language and Python Lisp, which means that my neophilia often has me working in spaces where there’s not much legacy life. Ah, well.

Platform internals: Level 2

Sijin’s “level 3” seems to be all over the place. Sometimes it’s industrial, and sometimes, as in this case, it’s hyper-academic stuff only ever seen by, like, 20 people worldwide.

Book: Level 3

Yes, I’ve actually read *Structure and Interpretation of Computer Programs” and “Lisp in Small Pieces” all the way through. I like how “Mythical Man Month” is a required reading for Level 2.

Blogs: Level 3

Whatareyoulookin’at?

Subscribe to Feed

Categories

Calendar

November 2018
M T W T F S S
« Oct    
 1234
567891011
12131415161718
19202122232425
2627282930