proxmox / virtio-blk / disk by-id
Why does the virtio-blk /dev/vda block device not show up in /dev/disk/by-id? Yesterday, I wrote about how Proxmox VE attaches scsi0 and virtio0 block devices differently. That is the starting point for todays question: how come do I get /dev/sda in /dev/disk/by-id while /dev/vda is nowhere to be found? This question is relevant if you're used to referencing disks through /dev/disk/by-id (for example when setting up ZFS, using the device identifiers).
proxmox / alter default create vm parameters
The Proxmox Virtual Environment has defaults when creating a new VM, but it has no option to change those defaults. Here's a quick example of hacking in some defaults. Why? (Changing SCSI controller does not change existing disks) In the next post I wanted to talk about /dev/disk/by-id and why disks that use the VirtIO SCSI controller do not show up there. A confusing matter in this situation was that creating a VM disk using a different SCSI controller and then switching does not change the storage driver for the existing disks completely!
openvpn / hardened fox-it openvpn-nl
Today, we will be evaluating OpenVPN-NL — “[a] hardened version of OpenVPN that includes as many of the security measures required to operate in a classified environment as possible — and whether we can use it as a drop-in replacement for regular OpenVPN. While OpenVPN allows many insecure configurations, such as turning off encryption, or the use of outdated cryptographic functions in security critical places, the goal of OpenVPN-NL — a fork created and maintained by Fox-IT — is to strip insecure configuration and verify that the distributed version is uncompromised.
postgresql inside kubernetes / no space left on device
Running PostgreSQL inside Kubernetes? Getting occasional "No space left on device" errors? Know that 64MB is not enough for everyone. With the advent of more services running inside Kubernetes, we're now running into new issues and complexities specific to the containerization. For instance, to solve the problem of regular file backups of distributed filesystems, we've resorted to using rsync wrapped inside a pod (or sidecar). And now for containerized PostgreSQL, we're running into an artificial memory limit that needs fixing.
chromium snap / wrong fonts
So, since a couple of weeks my snap-installed Chromium browser on Ubuntu Focal started acting up: suddenly it chooses the wrong fonts on some web pages. The chosen fonts are from the ~/.local/share/fonts/ directory. Look! That's not the correct font. And it's even more apparent that the font is off when seeing the source view. Bah. That's not even a monospaced font. A fix that appeared to work — but unfortunately only temporarily — involves temporarily moving the custom local fonts out of the way and then flushing the font cache:
stale apparmor config / mysql refuses to start
So, recently we had an issue with a MariaDB server that refused to start. Or, actually, it would start, but before long, SystemD would kill it. But why? # systemctl start mariadb.service Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details. After 90 seconds, it would be killed. systemctl status mariadb.service shows the immediate cause: # systemctl status mariadb.service ... systemd: mariadb.
zfs / zvol / partition does not show up
On our Proxmox virtual machine I had to go into a volume to quickly fix an IP address. The volume exists on the VM host, so surely mounting is easy. Right? I checked in /dev/zvol/pve2-pool/ where I found the disk: # ls /dev/zvol/pve2-pool/vm-125-virtio0* total 0 lrwxrwxrwx 1 root root 10 Dec 29 15:55 vm-125-virtio0 -> ../../zd48 Good, there's a disk: # fdisk -l /dev/zd48 Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes Disklabel type: dos Disk identifier: 0x000aec27 Device Boot Start End Sectors Size Id Type /dev/zd48p1 * 2048 97656831 97654784 46.
(I’m sorry, this time the recap is in Dutch instead of English. If you’re reading this, you’re probably Dutch anyway, so it won’t be a problem.) COVID-jaar 2020 is een vreemd jaar geweest, maar voor OSSO was het ook een goed jaar. Waar 2020 nog in mineur begon, omdat we eind 2019 nog een dierbare collega verloren aan een emigratiewens naar het land van de kangoeroes, hebben we in 2020 twee nieuwe collega’s mogen verwelkomen.
zfs destroy / dataset is busy
Just now, I tried to remove a ZFS dataset, and it reported dataset is busy for no apparent reason. # zfs list -r data NAME USED AVAIL REFER MOUNTPOINT data 3.12T 405G 251M /data data/kubernetes-logging 2.08T 405G 2.08T /data/kubernetes/logging data/rook-config 36.5M 405G 36.5M /data/rook-config data/rook-data 1.03T 708G 753G - # zfs destroy data/kubernetes-logging cannot destroy 'data/kubernetes-logging': dataset is busy The usual suspects were checked: The dataset was not mounted (cat /proc/mounts | grep kubernetes).
cumulus / postfix in the right vrf
Cumulus Linux is a network operating system. It is a switch, but it also runs Linux OS, allowing us to run our automation tools on it. We use it to automate the configuration of our network. A network where we use VRF (virtual routing and forwarding) to separate customer traffic. The presence of VRFs in the OS however means that we have to tell the daemons in which VRF to run.
offsite / on-the-fly encrypted backups / gocryptfs
Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data. That scheme works like this: there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server; you grant ssh+rsync access to that system; but only to a specific path; on that path, you mount an encrypted view of your filesystem — a.
pgp on yubikey / refresh expiry
Generally, I try to follow security best practices. This means that I have my PGP signing, authentication and encryption keys on my YubiKey, and I have configured the keys to expire after a year. Unfortunately, refreshing the expiry every year is not quite enough to store how to do that into muscle memory. Here are the steps relevant to my use case. Putting the keys on the YubiKey in the first place is worth a post of its own.
tls / testing certificate chains / easycert
The openssl client is a very versatile tool, but also a bit cryptic. The easycert utility from the ossobv/vcutil scripts makes validating/managing certificates easier. easycert from ossobv/vcutil has a few modes of operation: CLI, CGI, generating certificates and testing certificates. Nowadays we mostly use the testing mode: -T The utility is a convenient wrapper around openssl s_client and x509 calls. Get it from github.com/ossobv/vcutil easycert. Usage Run it like this:
excel / generate sheet password collision
Yesterday, I demonstrated how to brute force the Excel protected sheet/cells password. (Write protection! Not read protection a.k.a. encryption!) Today, I figured there must be a faster way, as the hash is not at all complicated. After fiddling around a little, I hacked together this bit of Python: def reverse_f(wanted): "Calculate Excel protected sheet password" # https://wjd.nu/notes/2020#excel-generate-sheet-password-collision # https://wjd.nu/notes/2020#libreoffice-asking-for-cell-password-brute-force def reverse_rotate(v): "Right shift by one, rotating the right most bit to bit 15" if v & 0x1: return (v >> 1) | 0x4000 return v >> 1 chars =  valid_tokens = tuple([ord(i) for i in ( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' '0123456789')]) # Length 9 should be enough to go down from 16 to 7 bits: # we skip some shorter solutions, but there are only a few hashes # that benefit from that.
libreoffice / asking for cell password / brute force
While we were editing a provider-supplied Excel document using LibreOffice, at seemingly random times, it would show a popup asking us for a password to a cell. This popup would only go away if we set a new (non-blank) password on it. Annoying! Apparently, it has to do with Sheet and Cell protection whereby an editing user is disallowed to edit certain cells/rows/sheets in a document. Having certain cells marked read-only, sure.
docker unprivileged user / becoming root
My colleague was rightly annoyed that our USER www-data docker images greatly hindered effective debugging. Can we become root again, while still keeping the additional secure-by-default non-root images? If we have enough permissions on the filesystem, then: yes, we can. Take the following example, where we’ll be looking at a myproject pod. (You can skip the Kubernetes steps if you already know where the Docker instance resides.) $ kubectl get pods -o wide myproject-66dd6b4dd-jskgf NAME READY STATUS AGE IP NODE myproject-66dd6b4dd-jskgf 1/1 Running 64d 10.
nss-dns4only / libc / disable AAAA lookups
Have you ever noticed how some applications can do AAAA DNS record lookups even though the host has no IPv6 connectivity? That means double DNS lookups for zero profit. Why is that? And how can you disable it? Problem To make a long story short, a common combination of circumstances can cause useless gratuitous AAAA lookups: Applications that are IPv6 ready (or applications that don’t care); on hosts using libc; where IPv6 is enabled (net.
gitlab / securing public repositories
In the past, GitLab repositories were created with Public Visibility by default. Now they have a more sensible security setting. Still, it can be nice to assert that public repositories are not Public-by-Accident. How? Well, one fix is to check that Public repositories are in a whitelisted public namespace (e.g. /public/). That way it’s immediately obvious that the repositories herein are visible to everyone. Use a Private browser and go to: https://YOUR_GITLAB_INSTANCE/explore/projects
more or less useless tips and tricks 3
More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. gsettings set org.gnome.desktop.calendar show-weekdate true — to enable week numbers in the gnome-shell datetime calendar popup. (You may need to set LC_TIME to en_GB so the week starts on a Monday instead of, American style, on a Sunday. You’ll probably have set LC_PAPER too already, to get A4 paper size printing defaults.
encryption decryption speed / gnupg / openssl
We were looking at encryption ingredients the other day. Because, if we want to compare encryption methods, we shouldn’t compare apples and oranges. With that newfound knowledge, we can run a few speed tests. The aggregated data (raw data sources can be found below): EncryptionDecryption user (ms)sys (ms)total (ms)mem (10K) user (ms)sys (ms)total (ms)mem (10K) gpg 1.4 76453547999354 1573832216060359 gpg 2.2 28633403203507 62123406552515 gpg 2.3* 27522813033527 44943184812536 gpg 2.3* nohw 45082814789 52575473187865537 openssl 17543392093 506421391812506 openssl nohw 35643443908504 27653963161507 customcrypt 333842137592458 356543540002461 First a few notes about the graph:
encryption / vocabulary / long term storage
While investigating the most appropriate encryption cipher and format, I realised I didn’t have enough vocabulary on the subject. This post aims to close that knowledge gap somewhat. I’m looking at symmetric ciphers here, as they are used when storing lots of data. (In fact, when encrypting larger amounts of data, public/private key encryption (an asymmetric cipher) is only used to encrypt a separate key for the symmetric cipher, which is then used for bulk of the data.
saltstack / printf IO-error / debianutils / which
After some upgrades, I suddenly noticed unexpected sh: printf: I/O error output. Some debugging later, it turns out that it’s the Dash way of informing us of a PIPE error. Apparently salt’s cmd.run can cause so little output buffering, that the debianutils which command can be aborted mid-output. Not-broken example: $ which python3 python false | head -n1 /usr/bin/python3 Broken example, through a salt cmd.run call: $ salt 'example.com' cmd.run 'which python3 python false | head -n1' example.
supermicro / java / console redirection / kvm
Connecting to the new SuperMicro iKVM management interfaces requires a working Java in your browser (IcedTea browser plugin). It will launch a Java Web Start (javaws) application. And java plugins (and needless web forms) are a pain in the behind. Is there a better way? Obviously, running from the web interface just means downloading a Java application, and running it locally. So why can’t we do that directly, and skip the IcedTea Java browser plugins?
asterisk pbx / generated doxygen docs
On the Asterisk PBX Development page you’ll find a reference to Doxygen generated documentation: Most of the documentation related to the source code is embedded in the source files and is processed with Doxygen. The latest version of the Doxygen generated source documentation can be found on http://doxygen.asterisk.org. However, it cannot be found there, as that URL has been returning a 503 Service Unavailable for quite some time now.
pgloader import / mysql to postgresql
When loading old (Django) projects in K8S, we’ve decided to give PostgreSQL a go as default database. Here are some notes that aid in importing. After looking around, pgloader seems to be the right tool for the job. Feature and stability wise it beats any other solution. And it’s available on recent Debian/Ubuntu. We need access to the remote PostgreSQL db; because pgloader will not provide an SQL dump (for reasons).