Hardscrabble 🍫

By Maxwell Jacobson

See also: the archives and an RSS feed

The Broom

April 19, 2020

Note: this was originally published on Thomas Countz’s newsletter Pseudocode, but I’ve copied it here after that website seems to be no longer up. I copied this from the internet archive in September of 2023.

Hi there, I’m Max Jacobson. I’m a senior software engineer at Vimeo, working on Vimeo OTT, a platform where anyone can launch their own paid video subscription service. Thomas asked me to share a bit about my role. In my opinion, being a “senior software engineer” means something different everywhere you go, and often means something different within the same company. Here’s a bit about what it means to me.

In February, the OTT engineering department went to the LeFrak Center in Prospect Park for a team outing (this was back when people went outside). We went curling. You know, like this thing: 🥌. We split off into teams, learned the rules, learned the basic technique, learned the terminology (it’s good to have “the hammer”), and played a few “ends”. Everyone got to try out the various roles: lead, second, vice-skip, and skip.

In the moment, my top priority was to not fall on my ass. In the weeks since, as my team has worked tightly together on a big project, I’m struck by what a good choice of activity it was to illustrate the various ways individuals can contribute to a team. As a senior engineer, sometimes you’re the skip: taking a step back to assess the state of things and offering direction for your teammates. Other times you’ve got the broom, and you’re sweeping furiously at the ice to smooth the glide path for the stone your teammate is throwing. Once in a while, you’ll throw the stone yourself.

Making software is hard, especially when you’re new. As a senior engineer, you do what you can to make it easier. That can mean all kinds of things: research, plan-making, pairing, answering questions, code review, documenting, white boarding, whatever it takes. Often, the way to have the biggest impact is to sweep the ice and get out of the way.

Some nerds on ice trying to curl Me preparing to throw a stone

Being Austin Powers in the workplace

March 26, 2020

I struggle with pop culture references. When I don’t get them, I feel uncultured. When I do get them, sometimes I pretend I didn’t get them, so I don’t need to acknowledge that, for example, I’ve seen all of the movies in the Marvel Cinematic Universe. Other times they’re just not funny and if you say you didn’t get the reference, that’s a polite reason that you’re not laughing.

So anyway, I’d like to make a pop culture reference. I really never do this. There’s that saying that “sarcasm is the lowest form of wit”. I agree with the sentiment: sarcasm is also bad, but for me it places at least above:

  1. pop culture references
  2. photos of signs

But one must sometimes stoop.

There’s that scene in Austin Powers: The Spy Who Shagged Me (1999) when the title character (Austin Powers) participates in a photo shoot. He breezes in, barks orders at the models and crew, snaps a bunch of photos, and the very moment he’s bored, he says, “And I’m spent”, and hands off the camera. He doesn’t even look. He just holds it over his shoulder, and an assistant hurries forward to take it.

I think about this thirty second scene all the time. I hadn’t actually seen it in twenty years, but it’s never really left me. Like all great art, it shows us something true: a confident attitude lets you get away with a lot.

That attitude! In my memory, he tosses the camera over his shoulder, knowing someone would catch it.

Sometimes in the workplace, I feel like Austin. It’s not entirely intentional. Whenever groups of people come together, power dynamics emerge. I’ve been on teams where everyone is so nice. We all glance around to one another’s eyes, saying things like, “No, you go first” and “If that even makes sense” and “I could be totally wrong” and “I don’t want to step on anyone’s toes”. I respect the impulse here, and it’s not that I love to step on toes, but when collective politeness gels into collective inaction, that’s a problem. When that happens, someone is going to do something about it. And I’ve often decided that it should be me, because I’m nice, and I’ll do a good job. It’s probably the case that most dictators also think they’re helping.

I’m definitely the guy in the meeting who says things like, “I think we all agree that we want to do X, for reasons A, B, and C. Some of us think we should approach it this way, but there are some reservations that haven’t been fully addressed. Is that right? Can we explore those some more?” Just a little recap, a little bridge building, a little supporting of the points being made, and a nudge toward a resolution. I’m 100% doing my best effort to channel Brian Lehrer, the fantastic radio host who speaks every weekday with journalists and experts in various fields, asking questions that get exactly to the heart of what matters. He also takes calls from New Yorkers, often scatter-minded and emotional, and he helps them tell their story. He rarely offers his own opinions (this morning he offered a gentle defense of the lima bean), but he drives the conversation exactly where he wants it to go.

Social psychologists talk about the fundamental attribution error. This is the one that Carlin was talking about here:

We have privileged access to our own minds, so it’s often possible to feel like we know why we’re doing the things we’re doing, and it’s in our self-interest to justify our own actions, so that’s what we do.

Sometimes in a workplace, someone will toss the camera over their shoulder to someone else. They did the fun part, and they got bored, and they want someone else to finish it up. That way, they can go do something else that’s fun. Sometimes we call this delegating, and it’s a good thing, because it’s giving an opportunity to someone that wants it. Sometimes it kind of sucks for that person. I think it’s not always easy to tell which one it is when you’re the one tossing the camera. But you do your best.

One last pop culture reference for today. This is related.

Over the last few years, I’ve gotten really into The Mountain Goats, a wonderful and prolific band that has been putting out wonderful records every one to three years for the last twenty-five years. Their most recent album, In League with Dragons (2019) is characteristically wonderful. I particularly like Clemency for the Wizard King, Waylon Jennings Live!, and the final song on the album: Sicilian Crest. That last one has stuck with me. I hate to talk about Trump on my blog, but it’s about this moment when right wing “strong man” leaders have emerged here in the states, and in Brazil, and in Hungary, and in Russia. Perversely, it tries to explore and explain their appeal by celebrating it. It suggests that, when things are bad, and the people are scared, we’re particularly vulnerable and that vulnerability can be very easily exploited by someone who promises to protect us. Like with Austin Powers, we’ll let them get away with anything.

I didn’t really get that from the text, I got it mostly from this podcast interview about the song, in which he describes the song as “quasi-fascist”.

I recommend that podcast very highly, even if you don’t like The Mountain Goats, although I suspect you already do or you’re about to.

Is it perhaps also the case that we’re drawn to Strong Man, Austin Powers types in the workplace? I think it’s true that most CEOs aren’t just men, but tall men. Even though, you know, that’s very stupid.

(Of course, Austin Powers is hanging a lantern on the absurdity of these men by placing a gnomish weirdo with a bad accent in that position, and that’s why it was great political art as well.)

Look to the West

Look to the man

Bearing the Sicilian crest

Despite my not wanting to celebrate such men, when I saw them play this song live, and I was singing along, Look to the man, I started to believe it. And when they drew the outro out practically indefinitely, letting the piano notes cascade over me in wave after wave, I never wanted it to end.

And I’m spent.

Are my blog posts getting longer?

March 25, 2020

Earlier, I was chatting with a coworker about blogging and speculated that my blog posts have gotten longer over time. Tonight, I thought I’d check if that was true, so I wrote a little script:

$ ruby app.rb
Avg word count by year
2011    1133.0
2012    554.57
2013    639.44
2014    676.81
2015    491.5
2016    1155.29
2017    1573.86
2018    816.0
2019    1125.0
2020    3757.0

▂▁▁▁▁▂▃▁▂█

Well, not as clean a trend as I thought. Interesting.

Here’s the quick-and-dirty script which should work for any Jekyll blog:

Dir.glob("./_posts/*.md").each_with_object({}) do |path, obj|
  path.match(%r{^./_posts/(\d{4})})[1].to_i.tap do |year|
    obj[year] ||= []
    obj[year] << File.read(path).split(/\s+/).count
  end
end.
  sort_by(&:first).tap do |word_counts_by_year|
    puts "Avg word count by year"

    word_counts_by_year.map do |year, counts|
      [year, (counts.sum / counts.length.to_f).round(2)]
    end.map do |year, avg|
      puts "#{year}\t#{avg}"

      avg
    end.tap do |avgs|
      puts
      # https://github.com/holman/spark
      system *avgs.map(&:to_s).unshift("spark")
    end
  end

As a fun little exercise, I tried writing without using any local variables. Not to sublog a former coworker, but I did work with someone who I never saw use a local variable. He never mentioned it, and I never asked. Sound off in the comments if you think this is a fun style.

(I don’t have comments but do take care).

The chef and food writer J. Kenji López-Alt makes fantastic cooking videos with names like “Late night dan dan noodles” in which he quietly whips himself up a midnight snack without overthinking it too much.

I’ll call this: late night code.

Get a shredder

March 25, 2020

the byproduct of my shredder

Times are weird with the coronavirus, but there’s something you can do that will help you feel more control of your life: get a shredder, and shred your junk mail.

Here’s the one I got: https://www.amazon.com/gp/product/B00YFTHJ9C. It’s completely fine.

Here’s how it works: credit card companies insist on asking you to get a new credit card. You’re asking me this now? Discover Card, you’ve asked me this every few months for my entire adult life. Discover this: my shredder.

Hi, The Greater Boston Food Bank. I completely support your cause, and what you’re doing is more important now than probably ever, and it’s true that I donated to you once, because I went to a wedding in Cambridge where the couple asked us to donate to you rather than get a gift, but I kind of think you’ve spent a good chunk of the money I gave you on postage and pamphlets at this point, and now I’ve let you sit on my desk for two months out of some kind of guilt? I’m so sorry, but you know what else is hungry is my shredder. No, that’s bad. I’m leaving the awful joke in and making one more donation. Your long game worked.

Receipts? Why not give them a quick shred?

Flyer from the local dentist? It’s gone now.

Amy Klobuchar postcard from early in the campaign when I was keeping my options open? It’s time for you to come off the fridge and go into the shredder.

When I made the purchase, I was thinking it would be mainly for shredding sensitive documents, and I very occasionally use it for that. But I didn’t know I would find it so therapeutic to just make things go away. This is perhaps gruesome, but every time I use it, I think of that one scene in Fargo where they’re putting a person in a woodchipper. I don’t find that therapeutic (don’t worry, this isn’t my confession that I’m a psychopath) but it’s very nice to just… make … problems … disappear.

Not even disappear, exactly, but make problems into confetti.

Get a shredder.

proof I'm not a monster

Coconut Estate

January 22, 2020

As a software engineer who has spent the last 6-7ish years working full-time jobs and listening to podcasts, I’ve spent a meaningful amount of time day-dreaming about starting my own company. Here’s how I’d do it:

First I’d make a side project, as a nights and weekends thing. I’d charge for it, and people would buy it, and then I’d quit my day job. Maybe I’d do some freelance work in the early days to make ends meet. Maybe I’d do a podcast about the whole process and sell ads on it, to share some behind the scenes stories and get people interested. Eventually I’d rent an office in Brooklyn with a few desks and start hiring people.

In 2018 this day-dreaming bubbled over the pot. It was spring and I was in Rome, walking and looking at buildings and thinking. At some point, on a bench outside Vatican City, I had an idea. When I got back to New York, I spent the next seven months building a prototype. I rented a desk at a co-working space. I paid GitHub for private repos (this was before they were free). I bought a domain name. I made a twitter account. I chose a tech stack. I got a website online.

Eventually I gave up. I turned my focus back to my day job. I got into tennis and running. I found a new day job.

And I never wrote about any of it here until now.

So, while I still remember all this, here’s a bunch of information about what I did and how I did it and why I stopped.

the name

I never had a proper name for it, but the whole time I worked on it, I called it Coconut Estate. It’s a reference to my favorite song, which is about an obsessive zealot who destroys himself in his search for knowledge, and has no regrets. As a product name, it’s probably very bad, but there were times I thought about sticking with it. Maybe people will find it intriguing, I would think. I spent $5.94 on the domain coconutestate.top. I would joke that the other name I was considering, which comes from another mewithoutYou song, was Unbearably Sad. That would make Coconut Estate sound better by comparison. I liked that Coconut Estate sounded like a place that you enter and enjoy spending time in.

the product idea

Coconut Estate would be a website for roadmaps. You could sign up, and then share a roadmap, such as “roadmap to get good at tennis”. Others could follow that roadmap, and learn to get good at tennis. It would be kind of like a curriculum that you could annotate with whatever links, text, resources you want, for each step. I imagined it as a kind of recursive thing, so perhaps one of the steps would be “learn how to serve”, and you could drill down and that would be a whole roadmap unto itself.

This was inspired in part by a blog post, Roadmap for Learning Rails1, published about ten years ago by one Wyatt Greene, who I do not know. When I was starting to re-learn how to program in 2012 or so, I must have googled “ruby on rails roadmap”, and found it. It was so helpful. Web development was very complicated2. I didn’t know where to start. I felt overwhelmed. The blog post included a flow chart, which helped structure my learning. It told me: forget all that, start here. And that helped me relax.

I thought: I’m sure programming isn’t the only thing that’s complicated. I imagined a whole community flourishing, of people writing similar roadmaps out of the goodness of their hearts, about all kinds of topics. I imagined people’s lives changing as they self-improved by following roadmaps. Because of the thing I made.

I also imagined that having a really nice interface that made it super easy to build and explore these roadmaps would be irresistible, and more useful than a JPEG embedded in a blog post.

the product idea: phase two

The plan was to focus on getting that off the ground, and then to build out this second phase, which is geared to business. I imagined a model kind of like GitHub: free to use for your personal, public stuff, but you pay to use it at work. And maybe the fun, positive public side would make people feel good about the tool and want to bring it into their workplace.

Some of the business use-cases I was imagining:

  • a roadmap for how to on-board a new team member
  • a roadmap for how to onboard a new customer
  • a roadmap for how to offboard a team member, including all of the things that you need to revoke their access to
  • a roadmap for how to perform your team’s monthly security audit

Et cetera.3

If the public-facing option was to be basically “luxe wikihow”, the private-facing part was basically a checklist-oriented knowledge base. In fact, the other primary inspiration for this was The Checklist Manifesto, a book that I never actually read past the first chapter. As I understood it, it details how hospitals use checklists to ensure that they follow their processes every time, because the alternative inevitably leads to mistakes.

how I organized myself

I created a git repository called coconut-estate internal. I still have it. It looks like this:

.
├── PLAN.txt
├── README.md
├── TODOs
│   └── max.txt
├── TOLEARNS
│   └── max.txt
├── architecture
│   ├── dns.txt
│   ├── infrastructure.txt
│   ├── toolbox-design.txt
│   └── web-app-stack.txt
├── brainstorms
│   └── max
│       ├── 2018-04\ handbook\ product\ notes.txt
│       ├── 2018-04\ misc\ going\ indie\ thoughts.txt
│       ├── 2018-04\ security\ audit\ product\ notes.txt
│       ├── 2018-04-05\ roadmap\ product\ notes.txt
│       ├── 2018-05\ deploying\ to\ digital\ ocean\ thoughts.txt
│       ├── 2018-05-21\ private\ roadmaps\ I\ make\ for\ myself.txt
│       ├── 2018-06-03\ random\ thought.txt
│       ├── 2018-06-17\ women\ and\ email.txt
│       ├── 2018-07-29\ docker.txt
│       ├── 2018-08-10\ art.txt
│       ├── 2018-11-05\ perks.txt
│       ├── 2018-12-08\ dunbar.txt
│       ├── 2018-12-09\ more\ dunbar\ notes.txt
│       ├── 2018-12-11\ more\ dunbar\ thoughts.txt
│       └── 2019-04-07\ dunbar\ thought.txt
├── finance
│   └── expenses.rb
├── marketing
│   ├── competitive\ analysis.txt
│   ├── feedback
│   │   ├── 2018-04-05\ sarah.txt
│   │   ├── 2018-04-09\ nick.txt
│   │   ├── 2018-04-22\ gavin.txt
│   │   ├── 2018-05-01\ gordon.txt
│   │   ├── 2018-05-02\ dan.txt
│   │   └── 2018-07-19\ harsh.txt
│   ├── name-ideas.txt
│   ├── slogans.txt
│   └── strategy.txt
├── todo-list-to-revamp-terraform-and-digital-ocean.txt
└── work-logs
    └── max.txt

9 directories, 36 files

It was basically a junk drawer. A place to put thoughts related to the project. I’m very much a plaintext kind of person, and this suited me very well. Some of the files were meant to be living documents that I could keep up-to-date over time, while others were snapshots of specific conversations or brainstorms. I namespaced everything with my name in case anyone else joined in the future and also wanted to record their brainstorms or work logs there.

Here’s an example excerpt from brainstorms/max/2018-06-17\ women\ and\ email.txt, since that file name jumped out at me as I was just reviewing this:

Just yesterday I was chatting with three software engineers from Ellevest, a financial investing startup that caters primarily to women. I don’t think I’d ever thought to cater Coconut Estate primarily to women, but why not? It’s kinda like the “mobile first” of product design; it’s harder to make a website mobile-friendly if you start by making it desktop friendly, and maybe it’s true that it’s harder to make a product women-friendly if you start by catering to “everyone”.

I shared that thought with Sarah and she was like … I don’t think that’s a good analogy.

Huh! I’d forgotten all of that. I’m not sure how valuable that thought was, but if thoughts are like lightning bugs around you, the natural thing is for them to flicker off and disappear into the night. Having a hole-punched jar nearby encourages capturing those thoughts, some of which might be valuable.

The file I updated most often was work-logs/max.txt. This one was directly inspired by https://brson.github.io/worklog.html, the “work log” of a Rust programmer that I had stumbled on at some point. There were times that I kept one at work, on a notebook at my desk, and found it helpful to remember how I had spent my time, and keep me somewhat accountable as I continued to spend my time.

Here’s an excerpt:

2018-09-03

  • 11:39
  • At home, thinking about rewriting the front-end app in Elm. I’ve been following the progress of Level.app, an indie slack competitor that Derrick Reimer is working on. He’s doing it as an OSS thing, which is neat. He’s using Elixir/Phoenix and Elm. My coworker Scott is also interested in Elm, and my former colleague Todd was also enthusiastic about it. The latest episode of Reimer’s podcast talks about upgrading to Elm 0.19 and how it improved some performance and asset size stuff. For me, the big appeal is the type safety and the “no runtime exceptions” promise. I started working my way through the guide yesterday, and this morning I watched the “Let’s be mainstream!” talk from Evan Czaplicki. The guide is mostly clicking with me. The other precipiating incident here is that at the day job, we’re currently considering adopting a more modern front-end tool like React or Vue, and I’m wondering if we should be considering elm. I haven’t made much of a commitment to Ember.js at this point, and I haven’t found it to be extraordinarily intuitive, so I’m tempted to give Elm a shot.
  • 14:23
  • At Konditori on Washington.
  • Plannning to keep working through the guide, with the goal of standing up a dockerized hello world web app today. Stretch goal 1: deploy it to prod. Stretch goal 2: have it load the roadmaps and display them on the page.
  • A few things that are appealing about elm so far:
    • a static type system
    • no nil, runtime exceptions
    • fast
    • an opinionated linter (third-party but still)
    • A kind/entertaining leader (per the 1.5 talks I watched)
    • At least attempts to be beginner-friendly with its emphasis on guides, docs, and error messages
    • no npm
    • Tests feel less necessary
  • A few things that are unappealing about elm so far
    • learning curve on syntax
    • tooling is a little minimalist and it’s not super clear how I’m supposed to be using these
    • Releases happen but maybe not that often?
  • 21:01
  • Took a bunch of breaks but more recently got to the good stuff in the guide and feel like I have the general idea of how this thing works and I kind of want to try to push thru to prod.
  • 22:37
  • Shipped a basic sketch of a site layout to prod and opened a PR (https://github.com/maxjacobson/coconut-estate/pull/4) so it’ll be easy to pick up where I left off… Big things to figure out next:

    1. how to hook up graphql
    2. how to do asset fingerprinting
    3. how to use elm reactor with a spa (maybe won’t?)
  • Overall quite pleased.

Others were more spartan:

2018-09-10

  • 22:57
  • refactoring some elm
  • 23:53
  • I give up! Too complicated.

I’m really grateful that I took notes as I was working. For me, putting thoughts to words helps me know what my thoughts are. It also made me very aware of how much time I was putting into the project. This reminded me that I was taking the project seriously, but also helped keep me very aware of how slowly it was going, which grew discouraging over time.

I did eventually open source the code, but the internal repo is just for me.

the tech stack

Here’s an overview of some of the most important technologies I used:

  • Rust
  • Elm
  • Docker (for development)
  • Digital Ocean
  • Terraform
  • Let’s Encrypt
  • PostgreSQL
  • GraphQL
  • systemd

I was motivated by a few things:

  1. wanting to use things that I found interesting
  2. wanting to use things that seemed like they’d help me build something very sturdy and reliable
  3. wanting to use things that would keep costs down

Some things I wasn’t motivated by:

  1. Building quickly
  2. Keeping things simple

In retrospect, these were the wrong motivations, if my goal was to actually finish something. Which, nominally, it was.

how I organized the code

I made a single repo that had all the code in it. At work, we were having success using such a monorepo, and it felt right. I made the top-level of the repo into a Cargo workspace, which is Rust’s built-in monorepo-like concept. The idea is that you can have several sibling Rust projects that you can think of as a family. By the end, I had four members in my workspace:

  1. api
  2. secrets_keeper
  3. toolbox
  4. authorized_keys_generator

Additionally, alongside the rust projects, I had two non-Rust codebases, each in their own subdirectory:

  1. website
  2. terraform

I’ll go through each of those in a bit of detail, providing some highlights of how and why they were built.

api

The primary back-end service was called api. I chose Rust because I found Rust interesting. I didn’t really know it, beyond the basics. I got much better at Rust while building this, although I still consider myself probably an advanced beginner at best. Most of what I’ve learned in my career I’ve learned from colleagues, either by copying what they did or soliciting their feedback. I’ve never had that with Rust, and so I really just have no idea if I’m doing anything right. But I got to a point where I felt somewhat productive, which was very gratifying, and several years in the making.

I used clap to define a CLI interface for the api program. This was mainly helpful because it gave me an easy way to define a few required command line flags that the api needs to boot. I particularly fell in love with clap while building this project, as it made it extremely easy to build a nice CLI program with subcommands and flags.

I used diesel as an ORM for interacting with the database. Diesel is created by Sean Griffin, who I had listened to talking about building Diesel on The Bike Shed for ages and was curious to try. It’s excellent. I didn’t really stress test it, but everything I wanted to do, it had thoughtfully modeled within Rust’s type system.

I decided to build a GraphQL API rather than a RESTful one. It felt trendy at the time. Moreover, at work we were integrating with GitHub’s GraphQL API, and I didn’t understand how any of that integration code worked. Learning about GraphQL APIs by building a simple one was very helpful for me to learn the important concepts, and that made it so much more clear what our client code was doing at work. I used the juniper crate to define the schema, and used actix-web4 to serve the requests. GraphQL is really cool. I admire the web dev community for opening its heart to a perspective that entirely rejects REST. I’m not anti-REST, but I like for established ideas to be challenged. And I like that there is an option available for projects where it’s important to give the maximum flexibility to front-end developers.

The one other interesting thing about the api was how I went about doing authentication and authorization. I decided to do claims-based authorization using JSON Web Tokens (JWT). To be totally honest, I’m not sure that I totally figured out a nice, ergonomic way of working with JWT, although the jsonwebtokens crate was very easy to work with. The basic flow is:

  • User signs up or signs in via an API request
  • In response, the API renders a token, which encodes their claims, which look like:
    {
      "user_id": 3,
      "site_admin": false
    }
    
  • The front-end can then do two things:
    1. store the token in local storage to use to authenticate future requests
    2. actually deserialize the claims from the token, and use that to make decisions about what to try and render (for example, whether to show an admin-only widget on a page)
  • When the API receives subsequent requests to access data, it will inspect the provided token to determine:
    1. what claims are the requestor making about who they are?
    2. are the claims signed, meaning they were issued by the API itself?
    3. have the claims expired, meaning they were once issued by the API itself, but we want to reject the request and force them to re-authenticate? (I didn’t actually implement this part)

Overall this felt good. Building something using JWT was a great way for me to learn about JWT. For example, I learned that if you’re using a JWT token for an API you’re integrating with, you can Base64-decode the three segments of the token and actually inspect the claims that have been serialized into the token. You could even attempt to change the claims, re-encode them, and make a new token, but it shouldn’t work, because the signature will reveal that you’ve tampered with the claims.

Perhaps I would have felt more comfortable with JWT over time.

secrets_keeper

I decided that I wanted to roll my own service for managing secrets: secrets_keeper. For example, api needs to know a PostgreSQL username and password to establish a database connection. It reads those credentials from the environment, but how do they get to the environment? I think I probably over-engineered my answer to that question.

secrets_keeper is a second API web service, designed to run behind a firewall. It has exactly two routes:

  1. GET /secrets
  2. POST /secrets

I built this one as a RESTful API using warp rather than GraphQL and actix-web. I was really just having fun poking around and taking my self-directed whistle-stop tour through the community. Of course, like api, it also uses clap to define its CLI.

Here’s how you use it:

  1. Create a new secret by making a POST /secrets request, with a body that looks like:
    {
      "group": "api",
      "name": "POSTGRES_USERNAME",
      "value": "coconutpg"
    }
    
  2. Before starting up the api service, fetch the secrets for the api group by requesting GET /secrets?group=api, and then export all of secrets you get back into your environment

Internally, it just stores the secrets in plaintext files on the filesystem. There’s absolutely no authorization built into secrets_keeper, so it’s very important that it run behind a firewall which will be responsible for making sure that only authorized personnel can read and write secrets.

toolbox

At the beginning, I wanted to build all of the ops-related scripting in Rust, so I created toolbox. I thought this would help me keep my Rust skills sharp. I gradually let that go and shifted to doing lots of scripting in plain-old shell scripting. But this exists.

Of course, it uses clap to define its CLI interface. Here’s what it looks like to use it:

$ bin/toolbox secrets write --help
toolbox-secrets-write
Write secrets to the secrets keeper service

USAGE:
    toolbox secrets write [OPTIONS] <VARIABLE> <VALUE> --group <GROUP>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -e, --env <ENVIRONMENT>    Specifies which environment to target
    -g, --group <GROUP>        Which group of secrets does this secret belong to?

ARGS:
    <VARIABLE>    The environment variable name for the new secret
    <VALUE>       The value of the new secret

clap takes care of generating that help text based on the registered subcommands, which is pretty neat.

The main responsibility that lingered in toolbox was this command for registering a new secret with the production secrets_keeper. It would have been much simpler and lighter-weight to make a shell script that uses curl to make the POST request.

Because the secrets keeper is not publicly available, there’s some extra work that you need to set up in order to make this request from your local environment. You know what, let’s talk about it.

We’re going to illustrate the bastion host pattern.

The idea is that there are three computers involved:

*************
* my laptop *
*************
    ||
    \/
***********
* bastion *
***********
    ||
    \/
***********
* secrets *
* keeper  *
***********

In words: The networking is configured so that the only computer that is allowed to make requests to the secrets keeper is the bastion server. Additionally, the only computer that is allowed to make SSH connections to the secrets keeper server is the bastion server. The bastion and the secrets server have their authorized keys defined to only allow known administrators to SSH into them.

That means a few things:

  1. if you were to SSH onto the bastion server, you could then SSH into the secrets keeper server, but if you were to try to SSH directly into the secrets keeper server, it would be like it didn’t even exist.
  2. if you were to SSH onto the bastion server, you would be able to use curl to read and write secrets, but if you try from your laptop it will be like the site doesn’t even exist.

My laptop has an SSH key on it that identifies me as a known adminstrator. So it would be cool if I could make a request from my laptop that tunnels all the way through the bastion server to the secrets server, and allows the response to tunnel all the way back.

Well, we can.

There’s an ssh command you can run which creates an “ssh tunnel”. While the tunnel remains open, you can make network requests to a specified port on localhost, and ssh will take care of “forwarding” the request through the bastion server.

The bastion pattern is something I had learned at work, but building my own implementation of it helped reinforce why it is valuable and how it works.

authorized_keys_generator

The final Rust service in my workspace was called authorized_keys_generator. In the last section, we talked a bit about how we use an authorized keys file to govern who can SSH into the production infrastructure. We could manually generate that file, but I felt the need to automate it. This was inspired by codeclimate/popeye, a tool that generates an authorized keys file based on keys registered with AWS.

At one point, at Code Climate, we talked about instead pulling this from GitHub, which exposes each users’s public keys at, e.g., https://github.com/maxjacobson.keys. I thought I’d try building a simple version of that concept for my project, so I made a simple CLI (again, using clap), that lets you do:

$ authorized_keys_generator --usernames maxjacobson dhh

And it would print out:

# authorized keys generated from authorized_keys_generator

# @maxjacobson
<first key here>

# @maxjacobson
<second key here>

# @dhh
<another key here>

You could take that text and use it as an authorized keys file on a server. I imagined later on I might build in support for you to provide a GitHub org and team, and it would then take care of looking up the users in that team, but it didn’t come to that.

website

The other significant directory in my monorepo with application code was website, which represented the front-end of the website. I used create-elm-app to scaffold a “hello world” Elm app, and boogied from there.

I had really never used Elm before, and it was kind of a lark that very quickly started to feel very right. At the time, all of my front-end experience was with JavaScript and jQuery. This was my first exposure to:

  1. a functional programming language
  2. a front-end web app framework
  3. the declarative UI pattern

I had been planning to use Ember.js, which I’d been meaning to try for years. I still do want to try it some day, I think. Elm was just something I was curious to read about, and I got ensnared pretty quickly. The Elm guide is just very good: Very friendly and persuasive and not very long. It feels like you can learn it in an afternoon, and you kind of can, if you’re in the right mood.

Over time, I did sour a little bit on Elm, but I mostly blame myself. I don’t think I designed my system very well to scale to support many pages and many actions. I would have appreciated more guidance from the guide on how I’m supposed to do that, I suppose.

I found that it was very easy to build something that was:

  1. very fast
  2. very reliable (they promise no runtime errors and the hype is real)
  3. very easy to refactor, knowing the compiler will help you along

I definitely wrote a lot of awkward Elm code. I never fully mastered how to use the various functional programming combinators that keep your code natural and streamlined.

The relationship between Elm and JavaScript frequently made my brain melt a little bit. Part of the idea with the Elm code is that the Elm code has no side effects. Your program has an entry point, and it receives some parameters, and depending on what the parameters, are, it resolves to some value, and that’s it. You don’t use it to actually do anything, you just take in some parameters, and use those to deterministically produce a single value.

Here are some things you can’t “do”:

  • You can’t make a network request.
  • You can’t look up the current time
  • You can’t write to local storage

Here’s what you can do:

You can define these 3-ish entrypoints, each of which answers a single question:

  1. init – what is the initial state of the application?
  2. update – if, hypothetically, someone were to interact with the application, how might that change the state of the application?
  3. view – what does the application look like, based on the current state of the application?

Each of these is totally pure and has no side effects. But like, of course, we… do want to have side effects. So while you can’t “do” those things in Elm, you can use Elm to do those things. Here’s what I mean: The Elm code compiles to JavaScript code, and gets run as JavaScript code in a browser. And JavaScript can do whatever it wants. So when you want to write some Elm code that has a side effect, it’s this sort of coy two step process:

  1. whisper a “command” into the air, for example: would someone please make this network request?
  2. describe what should happen if someone did perform that command

This much is all outlined in the Elm guide, and grows to feel natural over time. Kind of.

One major asset of the Elm community is its very active Slack channel. When I was getting stuck, I found myself spending time there lurking and occasionally asking questions.

One thing about Elm that strikes me as a liability is the paucity of releases. I think that Rust has the right idea with the release train model, which helps them achieve the goal of “stability without stagnation”. Elm sometimes goes over a year without any release at all in the core compiler. That doesn’t tell the whole story: there is activity, in that time, in the package ecosystem, including in the core packages (in theory). But it seems to take a deliberately slow, thoughtful pace that strikes me as a bit discomfiting.

I genuinely love to see the areas of focus that the Elm compiler developers choose. In their 2019 release, 0.19.1, they focused mostly on improving their already-good compiler error messages. That’s excellent. Given that they’re taking the tack of keeping the language small and slow-moving, I admire that they’re polishing that bauble as much as they can, which should help new people get into it even easier.

After I got my head over my skis with Elm, we started using React at work, and I found that the concept of a declarative UI framework with an internal state that you can update based on user interaction was oddly familiar. React was all new to me, but it went down fairly easy, and I credit my experience with Elm. And I did find myself missing the types.

terraform

The final significant codebase in the monorepo was terraform. This directory contained the definitions of all of the components of my infrastructure. I wanted to follow the “infrastructure as code” trend here, using HashiCorp terraform because we were using it at work and I barely understood any of its core concepts. Actually being responsible for setting it up from scratch was hugely valuable for me to learn terraform fundamentals.

Here’s the idea for the not-yet-inducted who may be reading this:

Let’s say you want to deploy your API service to the cloud, and you choose Digital Ocean as your vendor. You’ll need to create a number of resources within Digital Ocean:

And, actually a number of additional resources. One way that you can create those is to log in to the website and click some buttons to create them for you. That works well, but it’s up to you to remember which buttons you pressed, which is not easy.

Another option is to write some scripts to interact with the superb API, creating all of the resources that you need. That’s also pretty good, but what if you want to just change one thing about a resource. For example, Digital Ocean droplets can have tags. Let’s say you decide later on that you want to add a new tag to a droplet. If your script is set up to only create the resource, it won’t be easily changed to actually update a resource. Or, what if you want to delete some resources? That’s a whole new set of scripts.

This is all where terraform comes in. You don’t write any scripts to do anything, at all. Instead you describe the resources that you want to exist, with the qualities that you want them to have. Then you use the terraform CLI to apply those descriptions. The CLI will figure out what it needs to create, update, or delete. It just sort of figures it out. It’s excellent.

I really enjoyed using Digital Ocean with Terraform. I was using AWS at work at the time, which was my first experience with a big cloud vendor. It was very helpful for me to see what was in common and what was different between AWS and Digital Ocean. It gave me an appreciation for the staggeringly vast number of services that AWS offers. Digital Ocean is pretty quickly launching new things to close the gap, like Kubernetes support, managed databases, generating SSL certificates for your load balancers… I like the idea of using smaller players sometimes, and I’d be happy to try Digital Ocean again for something else.

deploys

All right, so that’s my tour through the code. The last section, about terraform, is a great segue into another topic I’d like to talk about: deploys. Sigh. This part is hard. I did not do anything particularly elegant here.

It’s possible to use terraform to help you deploy changes to your application code, but it requires a lot of cleverness. I didn’t do that.

Instead, I had terraform be responsible for:

  • creating servers
  • creating SSL certificates
  • registering systemd units that define a service that ought to run, such as: the api service
  • install a dummy script that stands in for the application code which doesn’t actually do anything.

Then, I wrote some manual deploy scripts which take care of:

  1. building the new version of the code
  2. uploading the new build to the production service
  3. stopping the service in production
  4. moving the new build into place
  5. starting the service in production

Here’s an example:

#!/bin/sh

set -ex

bin/prod/build api

bin/scp deploy-artifacts/api \
  www.coconutestate.top:/mnt/website/binary/api-new

bin/ssh www.coconutestate.top sudo systemctl stop api.service

bin/ssh www.coconutestate.top mv /mnt/website/binary/api-new /mnt/website/binary/api

bin/ssh www.coconutestate.top sudo systemctl restart api.service

As you can see, all of my helper scripts tend to reference other helper scripts. It’s a kind of rat’s nest of helper scripts.

The downside of this approach is that it incurs a moment of downtime. I tried to minimize it by uploading the assets before swapping it into place. It didn’t really matter, because no one was using the website at any point, so I was happy to compromise on uptime.

types

This project was definitely the zenith of my interest in types:

  • all of the data in the database hews to the types defined in the database’s schema
  • as data is loaded into memory in the Rust code, it is always deserialized into Rust types
  • when requests are made to the API, the request must adhere to the types defined in the GraphQL schema
  • when the API responds to requests, must adhere to the types defined in the GraphQL schema
  • when the Elm front-end wants to talk to the API, it uses Elm types that represent the GraphQL schema’s queries
  • then of course the Elm front-end deserializes the responses into memory in the browser, using Elm types that represent the expected structure of the responses

Frankly, it was a lot.

It led to quite a lot of boilerplate, modeling all of those types. And, of course, the more boilerplate the more opportunity for human error. And, more pressingly, human boredom.

cheapskate ci

So, did I write any tests? Not really. I felt like the types were keeping me in check enough.

I did use some code formatters:

I thought it would be cool to add some CI that made sure all of my code was well-formatted before I could merge anything into my default branch. One challenge was that I had kept my repo private, and the various CI providers all required you to pay to run builds for your private repo. As if to demonstrate the fact that I was more motivated by dorking around and learning things than actually launching a product, I took a detour to build something new:

cheapskate-ci.

Here’s the idea: instead of having a .circleci/config.yml in your codebase, add a cheapskate-ci.toml. Mine looked like:

[ci]
steps = [
  "docker-compose run --rm build cargo fmt --all -- --check",
  "terraform fmt -write=false -check=true -list=true -diff=true terraform",
  "docker-compose run --rm build cargo check --quiet --all",
  "elm-format --validate website",
  "docker-compose run --rm website elm-app test",
]

[github]
repo = "maxjacobson/coconut-estate"

Then, feel free to run: cheapskate-ci run --status to run those steps, and report a pass/fail status to GitHub regarding the latest commit. Then you can make that status a required status on GitHub. All taken care of, for free. You know, for cheapskates, like me.

It reminded me of my first programming job when we didn’t have CI, we just had an honor system. You open a pull request. You get an approval You run the tests locally, and if they pass, you merge the pull request. Looking back that feels so hard to believe.

(We got CI eventually)

Oh, and also…

It will prompt you for a GitHub token so that it can create a commit status on your behalf. I actually made another library, called psst, which cheapskate-ci uses to prompt you for info and persist the provided value for later use. I was really into that kind of thing at the time.

what was rewarding about all this

I’ve tried to sprinkle in some things that I found rewarding about this process. I’ll summarize and sneak in a couple more:

  1. Working independently meant that I was exposed to a lot of technical work that others were taking care of at work, or which were set up long before I joined and no one needed to think about anymore. This gave me opportunities to learn things that I otherwise wouldn’t have, and which put me in a better position to debug and operate production issues, particularly networking issues
  2. Working independently meant that I could make every technical decision, and make a lot of mistakes, in a low stakes environment. Sometimes following my curiosity worked out great and gave me some confidence in my technical instincts. Other times they were a disaster, which was genuinely humbling and helped clarify where I have room to grow.
  3. Playing product manager was so fun. Thinking about what to build is so fun. Talking to people about your idea and getting their feedback is so fun. Brainstorming how you might market your thing is actually so fun.
    • Oh: thank you very much to everyone who gave me feedback on this at any point, or listened to me talk about it. It meant a lot and I thought it was fun.

what was discouraging

  1. Actually finding time to work on it and feeling like progress is so slow is kind of a bummer
  2. The valleys you go through when you doubt your idea and worry that all the effort you’ve put in is a bummer
  3. As you do more market research and start finding that there are other things out there that are similar to your idea, and you start feeling like a bit of a fraud, it’s kind of a bummer
  4. I thought it would be more fun to join a coworking space, but working there on nights and evenings, it was always empty and a little lonesome

what I learned about myself

When you work on a team, you can learn about how you work on a team. On a team, hopefully, your team supports you, and provides you feedback to keep you on track. On your own, if you start to drift out of your lane, you’re going to just keep drifting. On a team, if your energy flags, your teammates can pick up the slack and the project keeps moving forard. On your own, it just … kind of … stops.

I think that everyone will drift in a different direction and probably stop somewhere interesting. For myself, I learned that when my natural instincts are unchecked, I’m inclined to fuss over the code and try to get things just right, and I’m completely unmotivated by actually delivering completed products to other human beings at any point.

Good to know!

why I stopped working on this

Oh, good question.

My enthusiasm petered out. It was very, very slow going. The technical choices I made were optimized more for my own curiosity and intellectual pleasure and less for actually being scrappy and shipping something. Ultimately, for all the work I did, I built very, very few features, bordering on none at all.

What I actually built:

  1. users can sign up, sign out, sign in
  2. signed in users can create a roadmap, providing a name
  3. signed in users can see the list of roadmap names

That’s it!

Lol.

I believe that the backend architecture was fairly extensible and could have fairly easily grown to have more functionality.

The front-end, however, kind of stalled out. I hit the limits of my Elm knowledge and wasn’t able to keep extending it easily. I needed to refactor it somehow but didn’t know how. Probably would have been surmountable but I ran out of steam.

The ops was a pain. I made it too complicated, and doing things like renewing SSL certificates was tedious and required these very precise sequencings. I should’ve just put it on Heroku.

Thinking about the possibility of launching it, I imagined it flopping. I imagined no one actually signing up for it, or people coming by and finding it a ghost town with no roadmaps on it. I thought about how much grit I’d need to keep promoting it, and I felt very tired. At some point I found myself asking myself, how is this better than WikiHow again?

Around that time, two other things were going on:

  1. I had a different idea that was new and shiny which I got more excited about, spent a while brainstorming and day-dreaming about that idea, and then realized that a bunch of apps were already doing it, and none of them seemed that cool
  2. I got a mortgage and bought an apartment (another thing I haven’t written about here) and in some ways I felt like I could do one or the other: buy an apartment or go independent. I’m not sure if that’s actually true, but it’s how I felt.

And so, I basically just made peace with moving on. I felt like I got a ton out of the project. Maybe one day I’ll try something else. I’m still listening to the podcasts.

  1. Since, apparently, deleted. Thank goodness for the Internet Archive. 

  2. I can’t imagine how much more overwhelming it must seem now. 

  3. Et cetera. Et cetera! 

  4. Although I haven’t worked on this project in ages, I was sad to recently learn that the maintainer of actix-web got burnt out on negative feedback and walked away from the project. I found actix-web very easy to work with in large part because of the effort he put in to providing examples. Hopefully he knows his work was appreciated by a lot of people. As I’m writing this, I’m seeing that the project is actually going to carry on under a new maintainer, who somehow saw that happen and thought “sign me up”. 

tree, but respecting your gitignore

October 30, 2019

Whenever I’m setting up a new computer, there’s a bunch of CLI programs I tend to install. Things like:

  • git
  • jq
  • tree

I actually have a list of them in my dotfiles. Honestly, there aren’t that many.

One that I like is: tree.

It’s good. It prints out the files in a visual way that is pretty easy to look at.

The only problem with it is that it doesn’t respect the .gitignore file. That means that it will list all of the files, even the ones I don’t actually care about, like logs or temporary files.

Honestly, that’s fair. The tree program predates git by at least a decade.

After git became the thing that everyone uses there came a new generation of tools that are git-aware. I’m thinking of things like rg, the grep alternative. It came of age in a git world, and by default it respects your .gitignore. That’s nice!

Should I wait for someone to make a second generation tree, written in Rust? Should I make a second generation tree, written in Rust?

Well, I could. But I don’t really have to. As of 1.8.0, tree has this kind of magic, unintuitively named option --fromfile. Let’s see how it works:

Let’s say you have this file, animals.txt:

animals/dogs/chihuahua.txt
animals/dogs/terrier.txt
animals/amphibians/frogs.txt

And then you run:

$ tree --fromfile animals.txt

It will print:

animals.txt
└── animals
    ├── amphibians
    │   └── frogs.txt
    └── dogs
        ├── chihuahua.txt
        └── terrier.txt

3 directories, 3 files

Look at that! It’s printing out a lovely tree based on some arbitrary input. Those files don’t even exist, I just made them up and wrote them in a list.

In this context, the flag name “from file” kind of makes sense. Instead of loading the list of files from the actual file system, it loads the list of files from a file. But you don’t actually need to write your list to a file at all. Instead, you can just pipe the list into the tree command, like so:

$ echo "animals/dogs/chihuahua.txt
animals/dogs/terrier.txt
animals/amphibians/frogs.txt" | tree --fromfile

And that works just the same. Hmm. Even though there’s no file. Hmm.

Now that we can print a tree from some arbitrary input, we can think of it as a building block. If we can gather a list of the files that we want to print, we can now ask tree to print them as a tree, and it will.

So now the problem we need to solve is: how do we gather the list of files in a directory, while respecting the .gitignore?

The simplest thing to do is:

$ git ls-files

This will ask git to print out all of the files which it is tracking all in a big long list. That’s not bad. Let’s take it for a spin:

$ git ls-files | tree --fromfile

It works pretty well!

Here’s the one problem: what about new files, which will be tracked by git but just haven’t been committed yet?

Here’s another problem: what if you’re not in a git repository? I mean, I usually am, but not always.

So here’s what I’m doing: using yet another tool. We’ve been talking about tree, which seems to lack a second generation alternative. Let’s turn our attention to find, which appears to date back to the 70s. find is very good. It finds and lists files. It has options for doing things like filtering them. Not bad. Let’s try combining that with tree --fromfile:

$ find . -type f -name "*ruby*" | tree --fromfile

In the source code for this blog, that prints out:

.
└── .
    ├── _posts
    │   ├── 2014-06-23-whoa-rubys-alias-is-weirder-than-i-realized.md
    │   ├── 2014-10-19-ruby-build-chruby-and-yosemite.md
    │   ├── 2015-03-29-ruby-keyword-arguments-arent-obvious.md
    │   ├── 2015-06-02-gemfiles-are-ruby-files.md
    │   ├── 2015-11-09-how-to-tell-ruby-how-to-compare-numbers-to-your-object.md
    │   ├── 2015-12-14-how-to-make-a-progress-bar-in-ruby.md
    │   └── 2017-07-02-there-are-no-rules-in-ruby.md
    └── talks
        └── there-are-no-rules-in-ruby
            └── slides
                └── assets
                    └── ruby.svg

6 directories, 8 files

So now we can use tree to find the structure of our repositories, filtered down to just certain files.

That filtered down to files with “ruby” in the filename.

What if we want to filter down to files where “ruby” is in the text of the file? Perhaps like so:

$ rg --files-with-matches ruby | tree --fromfile

This is actually super useful. You might wonder something like “what are all of the places in my large code base that reference this class/module that I want to change?” You could find that out, and see it visually, like so:

$ rg --files-with-matches "Url2png" --type ruby | tree --fromfile

And see:

.
└── app
    ├── facades
    │   └── template.rb
    ├── models
    │   ├── layout.rb
    │   └── site.rb
    └── services
        └── url2png.rb

4 directories, 4 files

Now I feel more confident that it won’t be too painful to touch that.

But, let’s get back to find. Given that find is so old, it has no idea about git. But, similar to rg, there’s a git-aware modern version with a two letter name: fd.

Coincidentally, but not surprisingly, both rg and fd are written in Rust.

With fd, we can find the list of files, and it will:

  1. respect the .gitignore
  2. include files that are not yet tracked by git

You just run it like this:

$ fd --type f --hidden --exclude .git

Some notes:

  • We use --type f so that it will print out files (by default it will also print out a line for each directory, which I personally do not care about)
  • We use --hidden so that it will include dotfiles
  • We use --exclude .git so that it will not print out the contents of the hidden directory that git uses to keep track of all of its data.

So, bringing it all together, a tree that respects your .gitignore:

$ fd --type f --hidden --exclude .git | tree --fromfile

Done!

Well, almost done. I am very happy with this behavior, but it’s obviously a chore to invoke. I’m not going to type all that out all the time. I already forgot the various flags. So what to do?

To be honest, I’m not sure what’s best. What I’ve done is this: in my login shell configuration, I’ve defined a function

treeeee() {
  fd --type f --hidden --exclude .git | tree --fromfile
}

And so now, sometimes, I’ll type treeeee (I type treee and then hit tab because I can’t remember how many es there are). Not my best-named thing, but hey, what can you do.

I've started initializing git repositories in the weirdest places

November 26, 2018

Earlier tonight I caught myself doing this:

cd ~/Downloads
git init
git commit --allow-empty --message ':sunrise:'

Then adding this to a README.md file:

This is my downloads folder

Then adding this to a .gitignore file:

*
!.gitignore
!README.md

And then running:

git add -A
git commit -m 'Add a repository for my downloads folder'

And while it felt extraordinarily natural, I laughed imagining a stranger observing this little ritual and wondering what the hell I was doing.

So what the hell was I doing?

I’ll start with the particulars of what it was and then dig into the why.

What?

  1. I initialized a new git repository in my downloads folder
  2. I added a sunrise commit
  3. I added a README file explaining the purpose of the repository
  4. I added a gitignore file which tells git to ignore everything, except for itself and the README
  5. I committed those two files

Why?

I don’t plan to have any code in this repository. I don’t plan to push this repository anywhere. I might not ever change that README to say anything else, or commit any other files. What’s the point?

I’m going to say some things which are willfully naive and I want you to come with me on this little journey: what does it mean for a folder to be the downloads folder? I suppose it means that one downloads files there. I suppose most browsers will put files there when you use the browser to download them. I suppose other applications, like Slack, might do the same. I suppose as the owner of the downloads folder, it’s my responsibility to tend the folder, perhaps to periodically delete the files there? Or maybe I don’t care and I’ll just let them pile up?

You know this and I know this, but we weren’t born with this knowledge. We scraped together this knowledge through years of trial and error. We’re unlikely to forget these things. We know how to use computers, especially our own ones.

But our computers have so many directories on them and it can be challenging to organize them all in a way that makes sense. And it can be hard to remember the decisions we made about how to organize them when we made those decisions a few years in the past.

For a few years, I’ve been making little repositories like this one all over my computers. The READMEs will include little notes to self about:

  • what is the purpose of this folder?
  • how do I use it?
  • where should I put things?
  • what’s the thinking behind those decisions?

The git log helps me remember how long I’ve been following whatever system I’m currently following (or neglecting to follow), which I’m not very proud to admit is something I find interesting.

I’ve been doing this specifically for most of the top-level directories in my Dropbox (things like src and writing and Documents) and frankly I enjoy very much stumbling on things like this:

cd ~/Dropbox\ \(Personal\)/Documents
git show

And seeing:

commit eeb9f6b2e24179a7995b4e6b52995270e8cec759
Author: Max Jacobson <max@hardscrabble.net>
Date:   Sun Aug 28 23:38:13 2016 -0400

    init with readme

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..edb6eed
--- /dev/null
+++ b/README.md
@@ -0,0 +1,7 @@
+# Documents
+
+I copied this stuff over from my ~/Documents on my macbook pro
+
+wtf is all this shit?
+
+I'd like to find a proper home for it in my dropbox...

And, uh: I haven’t!

Feel free to be like me.

Commit your skeletons right away

November 26, 2018

I was just writing a post about habits around starting new git repositories and there was one additional thought that isn’t quite related but which I also want to say and so now I’m really blogging tonight and coming back to you with a second post.

Please commit your skeletons right away.

Imagine you’re making a new rails app, and you use the rails command to generate a skeleton for your new application:

rails new the_facebook
cd the_facebook
$ git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .gitignore
        .ruby-version
        Gemfile
        Gemfile.lock
        README.md
        Rakefile
        app/
        bin/
        config.ru
        config/
        db/
        lib/
        log/
        package.json
        public/
        storage/
        test/
        tmp/
        vendor/

nothing added to commit but untracked files present (use "git add" to track)

This command will generate a whole bunch of files. It will also initialize a new git repository. But it doesn’t commit those new files for you.

I urge you: please commit them right away (after your sunrise commit).

Why? Because these files were all generated for you by a computer, and the computer deserves credit. Kind of. Really, it’s because you’re about to make a bunch of changes to those files and you’re also about to forget which of those lines you wrote and which of those lines the computer wrote. You’re going to be maintaining this code base for the rest of your life. I mean, maybe. It’s really helpful to look at git blame and figure out who wrote which lines and why and in my opinion it can actually be helpful to have that context all the way down to the very beginning.

The same is true of cargo new for rust people and jekyll new for blogger people. The ember new command is a welcome exception: it commits its skeleton and throws in some cute ASCII art for free.

If you already didn’t do this with your repository, rest assured that it doesn’t really matter, it’s just kind of nice.

Sunrise commits

November 26, 2018

I’ve picked up the habit from some people I’ve worked with that whenever I create a new repository, I make an initial empty commit that has the commit message :sunrise: and no changes in it. I thought it might be helpful to jot down some context on why I do that, or at least why I think I do that.

Starting new repositories

When you initialize a new git repository, it doesn’t yet have any commits in it. Let’s say you create a new repository:

mkdir my-great-repository
cd my-great-repository
git init

And then ask git to tell you about it:

git status

It will print out:

On branch master

No commits yet

nothing to commit (create/copy files and use "git add" to track)

And if you ask git to tell you about the commits:

git log

It won’t be able to:

fatal: your current branch 'master' does not have any commits yet

Let’s try to make a commit and see what happens:

git commit --message "some commit"

It didn’t let us:

On branch master

Initial commit

nothing to commit

We tried to make a commit, but we hadn’t staged any changes to be included in the commit, and so git was like … no. Which is kind of fair, since ordinarily the point of making a commit is to introduce a change to the code base. But the first commit is kind of special: what does it even mean to make a change to nothing?

I encourage you not to spend too long pondering that question.

There are basically two ways out of this:

  1. Actually add some files and commit them
  2. Tell git that you don’t mind having an empty commit, and make an empty commit

The first way: having the first commit include some files

The first way is probably what most people do, since it’s pretty straight-forward:

echo "Hello" > README.md
git add README.md
git commit --message "some commit"

The last command will output:

[master (root-commit) e52641c] some commit

Notice the part that says (root-commit), which is how you know that it’s the first commit.

This is basically fine.

Running git log works how you might suspect: it shows that there’s one commit.

Running git show works just fine: it prints out the details of the latest commit, including the changes.

It gets a little more complicated if you want to use git diff. Let’s say you want to construct a git diff command which will display the changes you introduced in your first commit. What would that look like?

git diff ??? e52641c

Bizarrely, there is a way to do this, and it looks like this:

git diff 4b825dc642cb6eb9a060e54bf8d69288fbee4904 e52641c

What is that first sha? Basically don’t worry about it. It’s a constant in libgit2, although apparently it might change if git changes the algorithm it uses to generate hashes.

The second way: making a sunrise commit

The other thing that you could’ve done was make a sunrise commit:

git commit --allow-empty --message ":sunrise:"

What’s going on here?

This time, git lets us make a commit even though we haven’t staged any change, because we specifically passed the --allow-empty flag to the commit command.

The commit message is short and sweet and paints a picture that fills your heart with hope.

That’s it.

Some advantages to making a sunrise commit:

  1. You can make a new repository on GitHub and push your sunrise commit to your default branch, and then immediately check out a feature branch and start working on sketching out the initial project structure, and open a PR to introduce that.
  2. If you want to make a new branch that has a totally empty tree, you can checkout your sunrise commit and then branch off from there. There are other ways to do that but they melt my brain a little more.
  3. All of the meaningful commits in your repository will have a parent, making them easily diffed.
  4. You feel the simple pleasure of following the recommendation from a blog post.
  5. Probably some other reason that I’m forgetting (feel free to tell me).

A handy alias

If you find yourself following this pattern, you may want to add this handy alias:

git config --global alias.sunrise "commit --allow-empty --message ':sunrise:'"

That way, you can run simply git sunrise after you initialize a new repository.

Note: this will render as the sunrise emoji on GitHub. You can feel free to use the actual emoji. I don’t because emoji don’t render properly in terminal emulators on Linux, at least in my experience.

Shout out

I’m pretty sure I picked up this habit from Devon Blandin.

git cleanup-branches

March 1, 2018

Do you clean up your git branches as you go or are you, like I am, a lazy hoarder?

$ git branch | wc -l
150

Look at all those things I did that I don’t care about anymore.

Yesterday I googled a little to try and find some magic incantation that would just clean up my branches for me. There are some, but I find that they’re either too conservative or too liberal for me. By “too conservative” I mean that they try to only delete branches that have been merged, except that they’re not actually very accurate, because they aren’t aware of GitHub’s “Squash and Merge” or “Rebase and Merge”, which I use pretty much exclusively. By “too liberal” I mean that some people recommend just literally deleting all of your branches.

I want to have control over the situation.

I can just run git branch -D branch-name-goes-here over and over, one-by-one, for all of my branches, but that would take several minutes, which I definitely technically have, but don’t want to spend that way, even while curled up with a podcast.

What I really kind of want is some kind of interactive process that gives me total control but doesn’t take that long to do. So I made a little shell script, which looks like this to use:

gif demonstrating git cleanup-branches which lets you interactively delete branches

As you may notice, it takes some loose inspiration from git’s interactive rebase.

It does something like this:

  1. get your list of branches
  2. open your default editor (whatever you have the $EDITOR global variable set to) (vim for me)
  3. wait for you to mark which branches should be deleted
  4. delete the ones you marked

git lets you plug in little scripts by just naming them git-whatever-you-want and putting that script on your $PATH and I think it’s fun to take advantage of that.

Here’s the latest version of the script as of this writing:

#!/usr/bin/env bash

set -euo pipefail

file="/tmp/git-cleanup-branches-$(uuidgen)"

function removeCurrentBranch {
  sed -E '/\*/d'
}

function leftTrim {
  sed -E 's/\*?[[:space:]]+//'
}


all_branches=$(git branch | removeCurrentBranch | leftTrim)

# write branches to file
for branch in $all_branches; do
  echo "keep $branch" >> $file
done

# write instructions to file
echo "

# All of your branches are listed above
# (except for the current branch, which you can't delete)
# change keep to d to delete the branch
# all other lines are ignored" >> $file

# prompt user to edit file
$EDITOR "$file"

# check each line of the file
cat $file | while read -r line; do

  # if the line starts with "d "
  if echo $line | grep --extended-regexp "^d " > /dev/null; then
    # delete the branch
    branch=$(echo $line | sed -E 's/^d //')

    git branch -D $branch
  fi
done

# clean up
rm $file

It follows the “chainable shell function” pattern I’ve written about before.

It uses set -o pipefail, my favorite recent discovery in shell scripting, which makes sure that each command succeeds, not just each expression. I should probably do a separate blog post about that with more detail.

Yeah, I guess that’s pretty much it. Have fun shell scripting out there.