My day job is at the IT department of University of Oslo.

I’m also a co-organiser of Devopsdays Oslo and Devopsdays Stockholm. Sometimes I go to other conferences.

I live in Norway.

Setting up gopass with a YubiKey on macOS

I have set up gopass as a work password store, backed by age encryption with a YubiKey holding the private key, and a git remote for sync. This is my notes about how I set the whole thing up

The goal

• Password store managed by gopass
• Secrets encrypted with age, private key on a YubiKey (via age-plugin-yubikey)
• Store committed to git, synced to a remote repo
• PIN + touch required to decrypt, nothing else

Architecture, briefly

There are two layers of encryption in this stack, and conflating them is the source of every confusion in this thread:

1. Your secrets (foo.age files in the store) are encrypted with age to a YubiKey recipient (age1yubikey1...). The matching private key lives on the YubiKey hardware. Decryption requires PIN + touch.
2. The gopass keyring file (~/.config/gopass/age/identities) holds the YubiKey identity stub (AGE-PLUGIN-YUBIKEY-...), which is just a pointer to a slot on a specific YubiKey serial. gopass encrypts this file with a passphrase (age scrypt mode), not with the YubiKey. The passphrase is unavoidable; macOS Keychain + Touch ID can make it invisible day-to-day.

The identity stub isn’t sensitive — useless without the physical YubiKey. The keyring encryption exists because gopass uses the same code path for plugin identities and native AGE-SECRET-KEY-... identities, and for the latter it genuinely matters.

Install

1

brew install gopass age age-plugin-yubikey

Generate the YubiKey identity

Plug in the YubiKey and run:

1

age-plugin-yubikey --generate --slot 1 --touch-policy cached --pin-policy once

Policy choices for a password store:

• PIN policy once — PIN required once per session
• Touch policy — I use cached — touch once, cached 15 seconds (convenient)

The command produces two strings:

• An identity (AGE-PLUGIN-YUBIKEY-1...) — pointer to the YubiKey slot
• A recipient (age1yubikey1...) — public key, used to encrypt secrets

Get them back any time with:

1
2

age-plugin-yubikey --identity   # shows the AGE-PLUGIN-YUBIKEY- line and the recipient
age-plugin-yubikey --list       # just the recipient

Backup matters

If this YubiKey breaks or is lost and it’s the only recipient, the store is unrecoverable. Before going further, decide on a backup recipient: a second physical YubiKey, or a passphrase-protected age key stored offline. Add it as an additional recipient in the next step.

Set up the gopass store

Do not redirect age-plugin-yubikey --identity into ~/.config/gopass/age/identities directly. gopass expects an encrypted keyring file there and will choke trying to “decrypt” the plaintext you wrote. Several guides online suggest this; it’s wrong for current gopass.

Instead, run:

1

gopass setup --crypto age --storage gitfs

When asked, choose to enter a passphrase (recommend y over the auto-generated one). This passphrase protects the local keyring file. Remember it — there’s no recovery.

gopass setup will also generate its own internal age keypair and add it to the keyring. We’ll deal with that in the next step.

Add the YubiKey identity to the keyring

1

gopass age identities add

It prompts for:

1. Passphrase — the keyring passphrase you just set
2. Age identity — paste the AGE-PLUGIN-YUBIKEY-1... string (the single line, not the # comment lines around it)
3. Recipient — paste the matching age1yubikey1...

Verify:

1

gopass age identities

You’ll see two identities — the auto-generated one and the YubiKey one. This is the first footgun: both are now in the keyring, and gopass silently uses every identity as a recipient when encrypting new secrets. Your “YubiKey-only” store is anything but.

The critical part: remove the auto-generated identity

Check what your store is actually encrypting to:

1

gopass recipients

You’ll see both keys. Remove the non-YubiKey one:

1
2
3

gopass age identities remove age1.....     # the non-yubikey identity
gopass recipients remove age1......         # the non-yubikey recipient
gopass recipients update                       # re-encrypt existing secrets

To verify it actually worked, look at a secret file directly:

1
2

gopass insert test/check
cat ~/.local/share/gopass/stores/root/test/check.age | head -5

You want to see only a -> piv-p256 stanza in the age header. If you also see -> X25519, the auto-generated key is still encrypting alongside your YubiKey — your secrets can be decrypted by a soft key sitting in your home directory, which is exactly the threat model the YubiKey is meant to defeat.

This is the step the official docs do not emphasize, and it bit me hard. gopass insert and gopass show will happily succeed even when nothing is actually using the YubiKey, because the soft key works without prompting. You can go a long time thinking your YubiKey setup is live when it isn’t.

Set up the git remote

The store is already a git repo (gitfs backend). Add the remote and push:

1
2

gopass git remote add --store root origin git@github.example:you/secrets.git
gopass sync

gopass sync does pull → commit → push. With core.autosync = true (default), inserts and edits push automatically.

Verify the round-trip

1
2

gopass insert test/yk
gopass show test/yk

Expected:

• YubiKey LED blinks (touch needed, since policy is cached and >15s since last touch)
• PIN prompt on first decrypt of the session
• Touch ID prompt (or passphrase) for the keyring — once per session if Keychain is enabled

If gopass show returns the value without the YubiKey blinking at all, go back to the verification step above and check the age header for a stray -> X25519 stanza.

Make the keyring passphrase invisible

Three options for handling the keyring passphrase:

1
2
3
4
5
6
7
8

# Option 1: macOS Keychain (Touch ID releases the passphrase)
gopass config age.usekeychain true

# Option 2: in-memory agent
gopass config age.agent-enabled true
gopass config age.agent-timeout 28800   # 8 hours

# Option 3: both

Option 1 is the most convenient on macOS — Touch ID replaces typing. Option 2 is the cross-platform answer. Since I’m only using a mac I went for option one.

Daily UX summary

After all this, day-to-day looks like:

1. First gopass command in a session: Touch ID prompt (keychain releases the keyring passphrase)
2. First decrypt of the session: YubiKey PIN prompt
3. Each decrypt after 15s of inactivity: YubiKey LED blinks, touch it
4. Rapid successive decrypts within 15s: no interaction needed
5. Insert/edit: git push happens automatically; if SSH key is in Secretive or similar, expect one more Touch ID prompt for that

The SSH Touch ID prompt — if it appears — is from your SSH agent (Secretive in my case), not from gopass. It says “SecretAgent” and mentions ssh-needs-auth.

Lessons

1. Don’t write the identities file by hand. Use gopass age identities add. The file is meant to be encrypted; pre-populating it with plaintext is the most-recommended-online and most-broken approach.
2. Always inspect a real .age file after setup to confirm only -> piv-p256 stanzas are present. The CLI’s gopass recipients output is necessary but not sufficient — gopass silently includes its internal identity as a recipient if it’s in the keyring.
3. The keyring passphrase is structural, not optional. Accept it and offload to Keychain or the agent.

Now I have a work password store I trust, with the actual security boundary on the YubiKey where I wanted it. Next: a separate private store mounted alongside, using a different YubiKey slot — same drill, but at least now I know where the trapdoors are.

How to manage dotfiles in a bare git-repo

How to set up:

Initialize an empty git repo in a folder in $HOME:

1

git init --bare $HOME/.cfg

Now we want to set up an alias for git to work on this repo:

1

alias config='/usr/bin/git --git-dir=$HOME/.cfg/ --work-tree=$HOME' 

We want to hide untraced files, so we explicitly choose which files to track:

1

config config --local status.showUntrackedFiles no

Make config persistent, I use zsh, so I add the alias to .zshrc

1

echo "alias config='/usr/bin/git --git-dir=$HOME/.cfg/ --work-tree=$HOME'" >> $HOME/.zshrc

now its time to add a few existing config files to the repo:

1
2
3

config add .zshrc
config add .vimrc
config commit

I created a github repo and pushed my config to github:

1
2

config remote add origin <github-repo-url>
config push

Now, time to check out my config on another machine. First, checkout the repo using the --bare option

1

git clone --bare <git-repo-url> $HOME/.cfg

Define the config-alias

1

alias config='/usr/bin/git --git-dir=$HOME/.cfg/ --work-tree=$HOME'

And last: Check out the repo

1

config checkout

Getting ipv6 to work with rpi, docker and pihole

This is how I made pihole run in a docker container with ipv6 support.

Step one: Configure docker for ipv›6:

1
2
3
4
5

$ cat /etc/docker/daemon.json
{
        "ipv6": true,
        "fixed-cidr-v6": <your-ipvc-cidr>
}

then we need to create a ipv6 network for docker:

1

docker network create --ipv6 --driver bridge --subnet "fd01::/64" ipv6

restart docker

1

systemctl restart docker

I use docker-compose to manage the container, so I added the following to docker-compose.yml

1
2

      ServerIPv6: <this is the ip of your rpi host>
      IPv6: "true"

and choose the ipv6 network:

1
2
3
4

networks:
  default:
    external:
      name: ipv6

rebuild the docker container:

1

$ docker-compose up -d --no-deps --build pihole

And voila! The pihole instance should now come up with an valid ipv6 address - and should respond to DNS queries on the ipv6 interface.

CS3 conference 2020

I recently attended the CS3 conference in Copenhagen. I did a talk about our experiences from running ceph in production for almost 3 years, and here is a link to the slides from my talk

Some useful gpg commands

I recently ran into a problem with some gpg encrypted file, where the files were encrypted with another key than they were supposed to. Therefore I had to reencrypt them with another key. Every time I run into an unusual task with OpenGPG I find myself banging my head into a wall of useless documentation. Here’s some of the useful commands I found which let me resolve the issue:

First I had to find out what keys the file was actually encrypted with.

gpg --list-packets file.gpg - lists contents of a gpg encrypted package - including the key IDs used for encrypting the file. (The man page is of course less useful, since it only says “List only the sequence of packets. This command is only useful for debugging.” - but I haven’t found another way to do what I want)

% gpg --pinentry-mode cancel --list-packets <file>.gpg 2>&1 - if you only want to see which keys a file is encrypted with. (note, it will list the keys to STDERR, so therefor I added the redirect to STDOUT so I could grep for the key I was looking for)

% gpg -k --keyid-format short and % gpg -k --keyid-format long - There is a long and short format for GPG keys. If you want to see the short or long format for the keyid when listing keys, add the —-keyid-format option.

Note: if you always want to print for instance long format, you could add keyid-format long to ~/.gnupg/gpg.conf

Want to know the fingerprint of the subkeys? Use gpg -K --with-subkey-fingerprint --keyid-format none

A few useful links:

• security - How to check if a GPG encrypted file is encrypted using a specific public key? - Super User
• pgp - How to change the gpg key of the pass password store - Ask Ubuntu

Summing up the CS3 conference 2019

This year started with a visit to Rome and the 5th edition of the CS3 (cloud services for synchronisation and sharing) - a yearly European conference that started as a workshop between Universities, National Research and Education networks and Research Centres in an attempt to exchange experiences in extending traditional storage technologies to the cloud.

This was my second visit to the conference, last time was two years ago, and I worked for the University of Oslo and visited primarily because of my interest in sync and sharing solutions. This time, I visited on behalf of my current company, and I have moved a bit down the stack, and I held a short presentation on how we are building an open source load balancer for our next generation storage clusters. I’m not going to say a lot about my talk, I hope it went OK.

The conference itself is packed with content. It is a single track conference spanning three days, and a total of almost 60 talks - which is a LOT! - I’m just linking to a couple of my favourite talks and a couple of observations.

Opening keynote by Davide Salomoni

• Davide Salomoni from INFN did a great job explaining some of the challenges with the current state of machine learning. Great to see a talk about machine learning that is genuinely interesting and not “hyped” in any ways.

The Magic Pocket - How dropbox is handling large data sets

by Said Babayev and Martin Meusburger This was an interesting talk, because it is always fun to hear about how large companies do operations. It is interesting to hear how they have managed to use SMR-drives in production. We have tried, but gave up, but then dropbox is on an completely different scale than us…

Scaling Tightly Coupled Algorithms on AWS

by Scott Eberhardt While I’m not convinced yet that it is a good idea to run your HPC workloads on Amazon, the talk was mostly about scaling, and that’s what made the talk interesting.

One thing I would love to see at a future conference is more attendees from the nordic countries. There is a lot of exiting things going on here, but I feel we are missing out on a bit of the cooperation that happens between the

Rome is a nice city, but January can be cold, rainy and windy. However, I got a few hours of sightseeing in nice weather on Thursday afternoon before I had to return home. The highlight was definitely seeing the Jackson Pollock exhibition at Complesso Del Vittoriano.

Summary from devopsdays Copenhagen

Last week I visited Devopsdays Copenhagen for the second time (which also happened to be the second time the conference was arranged).

Being part of the organising team for two other nordic devopsdays, it is fun to visit the third one! And one particular thing I like about the current state of the nordic devopsdays is the great overlap between the different organiser teams. It really gives the feeling of being a part of a larger community.

The venue

Last year, they had the conference at a concert hall (which used to be an old stable) - which was a very nice venue, but just a bit too small. This year, they had moved it to a movie theatre, which maybe lacked a bit of charm, but it made that up with lots of space, good seats, good audio/video and good catering. And of course there was popcorn!

The talks

All talks were excellent. Period. It was great to finally see Matty Strattons talk, I have missed a few opportunities to see it before. I also enjoyed Heidi Waterhouse’s talk, and Ken Mugrage did a new talk which was very good – with some real world experience thrown in. Nataliya Remez did a talk about safety which was really interesting. All the Ignites were great this year (probably due to the fact that I didn’t do an ignite).

Multi track conferences

Devopsdays Copenhagen was a multi track conference this time, and I must admin I’m not a big fan of multi-track devopsdays. I can see that it made it possible to put in a couple narrower, more technical talks (I saw a couple of them and they were good) - but it also changes the openspace sessions and the hallway track in some way, when everyone have seen the same talks, we are in some ways on the same level when we come to the open space session, but now it was always the question about “did you see this or that talk?”. I think the openspace sessions are the most important part of a devopsdays conference, and the talks just lead up to that, and give everyone a common ground to stand on.

If I can give some advice to devopsdays organisers, it would be to make the conference a single track conference, even if that means you have to make some very hard choices when it comes to selecting talks.

Ignite karaoke

One of the highlights of the programs were a session with ignite karaokes. Quite a few people had the guts to get up in front of the large audience and do an improvised talk with slides they hadn’t seen before. Great fun and a great success!

Mob programming

Emily Bache did a talk about techical leadership in devops, where she discussed the use of mob programming – and she also hosted two introduction sessions to mob programming, during the open space part of the programs. Attendees had the option to either participate or observe, I chose to observe one of the sessions, and it was very interesting to see how the “mob” became more confident and started to cooperate on solving the problem during the course of the exercise. This is definitely something I want to try at some point.

So long, and thanks for all the #tacops!

Since I have done this two times, I guess it is now a tradition: Me and a few others gathered at the best taco place in Copenhagen: Hija de sanchez after the conference for a round of delicious #tacops. And with that - time to go home! Thanks to the danish organisers for making such an awesome event!

My old blog wasn’t updated in a long while, and I have planned to move from Jekyll to Hugo for a long while, so I finally set up everything.. I’m actually not sure where the source for my old blog is, but if I find it, I might republish some older stuff. And I have planned to publish more content soon.

I’m currently using the purehugo theme, because I like the simplicity and the look. But there is a few things I miss, so I might change that in the future.

Of course I couldn’t stick with my first theme for more than one day, so I’m experimenting with others. I’m currently using a modified Jane theme. I Also like the Story theme, I might try that out soon as well. While it seems to be a lot of good themes for Hugo, a lot of them are not maintained, so that limits the options a bit. Anyways, I’ll make sure to keep the About section updated with the current info about what I use.

The site is hosted on github pages, with cloudflare in front for SSL certs. A nice thing about the setup is that everything is very easy to set up, and free for personal blogs. How I set everything up might be a topic for a future blog post!

Books mentioned at devopsdays Oslo 2017

There was a lot of books mentioned during the talks at Devopsdays Oslo 2017. I compiled a list of all the books mentioned. If you want to see the talks, you can find links to the recordings on the Devopsdays website (the books are listed by the speaker)

Mike Long

• The Structure of Scientific Revolutions by Thomas S. Kuhn
• The Phoenix Project By Gene kim

Stein Inge Morisbak

• Driving Technical Change by Terrence Ryan

Trond Arve Wasskog

• Test Driven Development: By Example by Kent Beck
• Continuous Integration: Improving Software Quality and Reducing Risk by Paul M. Duvall
• Release It!: Design and Deploy Production-Ready Software by Michael T. Nygard
• Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble
• The Phoenix Project
• Lean Enterprise: How High Performance Organizations Innovate at Scale by Jez Humble,Joanne Molesky and Barry O’Reilly
• Site Reliability Engineering: How Google Runs Production Systems by Niall Richard Murphy, Betsy Beyer, Chris Jones and Jennifer Petoff

Mark Burgess

• In Search of Certainty: The Science of Our Information Infrastructure by Mark Burgess