zfs / zvol / partition does not show up
On our Proxmox virtual machine I had to go into a volume to quickly fix an IP address. The volume exists on the VM host, so surely mounting is easy. Right? I checked in /dev/zvol/pve2-pool/ where I found the disk: # ls /dev/zvol/pve2-pool/vm-125-virtio0* total 0 lrwxrwxrwx 1 root root 10 Dec 29 15:55 vm-125-virtio0 -> ../../zd48 Good, there's a disk: # fdisk -l /dev/zd48 Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes Disklabel type: dos Disk identifier: 0x000aec27 Device Boot Start End Sectors Size Id Type /dev/zd48p1 * 2048 97656831 97654784 46.
(I’m sorry, this time the recap is in Dutch instead of English. If you’re reading this, you’re probably Dutch anyway, so it won’t be a problem.) COVID-jaar 2020 is een vreemd jaar geweest, maar voor OSSO was het ook een goed jaar. Waar 2020 nog in mineur begon, omdat we eind 2019 nog een dierbare collega verloren aan een emigratiewens naar het land van de kangoeroes, hebben we in 2020 twee nieuwe collega’s mogen verwelkomen.
zfs destroy / dataset is busy
Just now, I tried to remove a ZFS dataset, and it reported dataset is busy for no apparent reason. # zfs list -r data NAME USED AVAIL REFER MOUNTPOINT data 3.12T 405G 251M /data data/kubernetes-logging 2.08T 405G 2.08T /data/kubernetes/logging data/rook-config 36.5M 405G 36.5M /data/rook-config data/rook-data 1.03T 708G 753G - # zfs destroy data/kubernetes-logging cannot destroy 'data/kubernetes-logging': dataset is busy The usual suspects were checked: The dataset was not mounted (cat /proc/mounts | grep kubernetes).
cumulus / postfix in the right vrf
Cumulus Linux is a network operating system. It is a switch, but it also runs Linux OS, allowing us to run our automation tools on it. We use it to automate the configuration of our network. A network where we use VRF (virtual routing and forwarding) to separate customer traffic. The presence of VRFs in the OS however means that we have to tell the daemons in which VRF to run.
offsite / on-the-fly encrypted backups / gocryptfs
Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data. That scheme works like this: there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server; you grant ssh+rsync access to that system; but only to a specific path; on that path, you mount an encrypted view of your filesystem — a.
pgp on yubikey / refresh expiry
Generally, I try to follow security best practices. This means that I have my PGP signing, authentication and encryption keys on my YubiKey, and I have configured the keys to expire after a year. Unfortunately, refreshing the expiry every year is not quite enough to store how to do that into muscle memory. Here are the steps relevant to my use case. Putting the keys on the YubiKey in the first place is worth a post of its own.
tls / testing certificate chains / easycert
The openssl client is a very versatile tool, but also a bit cryptic. The easycert utility from the ossobv/vcutil scripts makes validating/managing certificates easier. easycert from ossobv/vcutil has a few modes of operation: CLI, CGI, generating certificates and testing certificates. Nowadays we mostly use the testing mode: -T The utility is a convenient wrapper around openssl s_client and x509 calls. Get it from github.com/ossobv/vcutil easycert. Usage Run it like this:
excel / generate sheet password collision
Yesterday, I demonstrated how to brute force the Excel protected sheet/cells password. (Write protection! Not read protection a.k.a. encryption!) Today, I figured there must be a faster way, as the hash is not at all complicated. After fiddling around a little, I hacked together this bit of Python: def reverse_f(wanted): "Calculate Excel protected sheet password" # https://wjd.nu/notes/2020#excel-generate-sheet-password-collision # https://wjd.nu/notes/2020#libreoffice-asking-for-cell-password-brute-force def reverse_rotate(v): "Right shift by one, rotating the right most bit to bit 15" if v & 0x1: return (v >> 1) | 0x4000 return v >> 1 chars =  valid_tokens = tuple([ord(i) for i in ( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' '0123456789')]) # Length 9 should be enough to go down from 16 to 7 bits: # we skip some shorter solutions, but there are only a few hashes # that benefit from that.
libreoffice / asking for cell password / brute force
While we were editing a provider-supplied Excel document using LibreOffice, at seemingly random times, it would show a popup asking us for a password to a cell. This popup would only go away if we set a new (non-blank) password on it. Annoying! Apparently, it has to do with Sheet and Cell protection whereby an editing user is disallowed to edit certain cells/rows/sheets in a document. Having certain cells marked read-only, sure.
docker unprivileged user / becoming root
My colleague was rightly annoyed that our USER www-data docker images greatly hindered effective debugging. Can we become root again, while still keeping the additional secure-by-default non-root images? If we have enough permissions on the filesystem, then: yes, we can. Take the following example, where we’ll be looking at a myproject pod. (You can skip the Kubernetes steps if you already know where the Docker instance resides.) $ kubectl get pods -o wide myproject-66dd6b4dd-jskgf NAME READY STATUS AGE IP NODE myproject-66dd6b4dd-jskgf 1/1 Running 64d 10.
nss-dns4only / libc / disable AAAA lookups
Have you ever noticed how some applications can do AAAA DNS record lookups even though the host has no IPv6 connectivity? That means double DNS lookups for zero profit. Why is that? And how can you disable it? Problem To make a long story short, a common combination of circumstances can cause useless gratuitous AAAA lookups: Applications that are IPv6 ready (or applications that don’t care); on hosts using libc; where IPv6 is enabled (net.
gitlab / securing public repositories
In the past, GitLab repositories were created with Public Visibility by default. Now they have a more sensible security setting. Still, it can be nice to assert that public repositories are not Public-by-Accident. How? Well, one fix is to check that Public repositories are in a whitelisted public namespace (e.g. /public/). That way it’s immediately obvious that the repositories herein are visible to everyone. Use a Private browser and go to: https://YOUR_GITLAB_INSTANCE/explore/projects
more or less useless tips and tricks 3
More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. gsettings set org.gnome.desktop.calendar show-weekdate true — to enable week numbers in the gnome-shell datetime calendar popup. (You may need to set LC_TIME to en_GB so the week starts on a Monday instead of, American style, on a Sunday. You’ll probably have set LC_PAPER too already, to get A4 paper size printing defaults.
encryption decryption speed / gnupg / openssl
We were looking at encryption ingredients the other day. Because, if we want to compare encryption methods, we shouldn’t compare apples and oranges. With that newfound knowledge, we can run a few speed tests. The aggregated data (raw data sources can be found below): EncryptionDecryption user (ms)sys (ms)total (ms)mem (10K) user (ms)sys (ms)total (ms)mem (10K) gpg 1.4 76453547999354 1573832216060359 gpg 2.2 28633403203507 62123406552515 gpg 2.3* 27522813033527 44943184812536 gpg 2.3* nohw 45082814789 52575473187865537 openssl 17543392093 506421391812506 openssl nohw 35643443908504 27653963161507 customcrypt 333842137592458 356543540002461 First a few notes about the graph:
encryption / vocabulary / long term storage
While investigating the most appropriate encryption cipher and format, I realised I didn’t have enough vocabulary on the subject. This post aims to close that knowledge gap somewhat. I’m looking at symmetric ciphers here, as they are used when storing lots of data. (In fact, when encrypting larger amounts of data, public/private key encryption (an asymmetric cipher) is only used to encrypt a separate key for the symmetric cipher, which is then used for bulk of the data.
saltstack / printf IO-error / debianutils / which
After some upgrades, I suddenly noticed unexpected sh: printf: I/O error output. Some debugging later, it turns out that it’s the Dash way of informing us of a PIPE error. Apparently salt’s cmd.run can cause so little output buffering, that the debianutils which command can be aborted mid-output. Not-broken example: $ which python3 python false | head -n1 /usr/bin/python3 Broken example, through a salt cmd.run call: $ salt 'example.com' cmd.run 'which python3 python false | head -n1' example.
supermicro / java / console redirection / kvm
Connecting to the new SuperMicro iKVM management interfaces requires a working Java in your browser (IcedTea browser plugin). It will launch a Java Web Start (javaws) application. And java plugins (and needless web forms) are a pain in the behind. Is there a better way? Obviously, running from the web interface just means downloading a Java application, and running it locally. So why can’t we do that directly, and skip the IcedTea Java browser plugins?
asterisk pbx / generated doxygen docs
On the Asterisk PBX Development page you’ll find a reference to Doxygen generated documentation: Most of the documentation related to the source code is embedded in the source files and is processed with Doxygen. The latest version of the Doxygen generated source documentation can be found on http://doxygen.asterisk.org. However, it cannot be found there, as that URL has been returning a 503 Service Unavailable for quite some time now.
pgloader import / mysql to postgresql
When loading old (Django) projects in K8S, we’ve decided to give PostgreSQL a go as default database. Here are some notes that aid in importing. After looking around, pgloader seems to be the right tool for the job. Feature and stability wise it beats any other solution. And it’s available on recent Debian/Ubuntu. We need access to the remote PostgreSQL db; because pgloader will not provide an SQL dump (for reasons).
setting up powerdns slave / untrusted host
When migrating our nameserver setup to start using DNSSEC, a second requirement was to offload a resolver to somewhere off-network. You want your authoritative nameservers to be distributed both accross different geographical regions, networks and top level domains. That means, don’t do this: ns1.thedomain.com - datacenter X in Groningen ns2.thedomain.com - datacenter X in Groningen Do do this: ns1.thedomain.com - datacenter X in Groningen ns2.thedomain.org - datacenter Y in Amsterdam In our case, we could use a third nameserver in a separate location: a virtual machine hosted by someone other than us.
openssl / sign / subject alternative names
Recently, an automated job of mine failed because the latest and greatest Python refused to validate an SSL certificate with 127.0.0.1 in the common name (CN). Apparently CN=127.0.0.1 will not be accepted anymore, as using the common name for hostname validation has been deprecated for ages now. The fix? Use subject alternative names (SANs). Generally, you’ll already have these when you have your certificates signed by somebody else. But if you’re signing certificates yourself, you’ll need to know how to pass them to openssl:
3cx voip / letsencrypt tls
Can you get your 3CX Phone System to connect to your SIP provider Trunk over TLS, when the server uses a Let’s Encrypt certificate? 3CX documentation on this topic is scarce. There are posts like 3CX Forum: SIP TLS: It’s fairly straightforward: Your provider must give you a TLS Root Certificate (.pem) so you can encrypt the traffic. If they have SRV records the system will automatically know where to connect for TLS mode (Auto Discovery option on General trunk tab), or the provider might just tell you to change your port to the one they are listening to for TLS connections.
(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2019 highlights We are looking for new colleagues! The new website is finally up! Built a Cumulus network with automated deployment using Ansible and Netbox as the source of truth. Managed kubernetes has become a staple. Extended our ISO27001:2017 and NEN7510:2017 certifications. Visited Legoland as a team building exercise. Many improvements! Ronald may actually leave for australia, like for real, not kidding, probably, if they finally let him in, in the not too distant future.
(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2018 highlights Edgar joined our team Managed kubernetes moving forward Many awesome upstream kubernetes releases Keeping up integrating improvements in our default kubernetes “distribution” Further expanding our managed kubernetes install base Updated NEN7510:2011 to NEN7510:2017 ISO27001 ISMS improvements and further integrating customer scope New website coming up - bijna klaar TM.
gbp buildpackage / gpg2
If you prefer gpg2 over gpg, building a debian package with debuild or gbp buildpackage may require some fiddling. In my case, I’m using gpg2 instead of gpg for signing because unlike version 1, version 2 does PGP key forwarding. That way I can sign on a remote machine, using a local PGP key card. However gbp buildpackage, dpkg-buildpackage and debuild are hardwired to call gpg. And — it turns out — using a simple /usr/local/bin/gpg to /usr/bin/gpg2 symlink was not sufficient to convince gbp (and debuild) to use the gpg2 binary, while for dpkg-buildpackage that is sufficient.