Blog

Blog

partially removed pve node / proxmox cluster

The case of the stale (removed but not removed) PVE node in our Proxmox cluster. On one of our virtual machine clusters, a node — pve3 — had been removed on purpose, yet is was still visible in the GUI with a big red cross (because it was unavailable). This was not only ugly, but also caused problems for the node enumeration done by proxmove. The node had been properly removed, according to the removing a cluster node documentation.

Read more

enable noisy build / opensips

How do you enable the noisy build when building OpenSIPS? The one where the actual gcc invocations are not hidden. In various projects the compilation and linking steps called by make are cleaned up, so you only see things like: Compiling db/db_query.c Compiling db/db_id.c ... This looks cleaner. But sometimes you want to see (or temporarily change) the compilation/linking call: gcc -g -O9 -funroll-loops -Wcast-align -Wall [...] -c db/db_query.c -o db/db_query.

Read more

missing serial / scsi / disk by-id

When you have a lot of storage devices, it's best practice to assign them to raid arrays or ZFS pools by something identifiable. And preferably something that's also readable when outside a computer. Commonly: the disk manufacturer and the serial number. Usually, both the disk manufacturer and the disk serial number are printed on a small label on the disk. So, if you're in the data center replacing a disk, one glance is sufficient to know you got the correct disk.

Read more

smtp_domain / gitlab configuration

What is the smtp_domain in the GitLab configuration? There is also a smtp_address and smtp_user_name; so what would you put in the “domain” field? Contrary to what the examples on GitLab Omnibus SMTP lead you to believe: smtp_domain is the HELO/EHLO domain; i.e. your hostname. RFC 5321 has this to say about the HELO/EHLO parameter: o The domain name given in the EHLO command MUST be either a primary host name (a domain name that resolves to an address RR) or, if the host has no name, an address literal, as described in Section 4.

Read more

yubico otp / pam / openvpn

Quick notes on setting up pam_yubico.so with OpenVPN. Add to OpenVPN server config: plugin /usr/lib/x86_64-linux-gnu/openvpn/plugins/openvpn-plugin-auth-pam.so openvpn # Use a generated token instead of user/password for up # to 16 hours, so you'll need to re-enter your otp daily. auth-gen-token 57600 Sign up at https://upgrade.yubico.com/getapikey/. It's really quick. Store client_id and secret (or id and key respectively). You'll need them in the config below. Get PAM module: # apt-get install --no-install-recommends libpam-yubico Create /etc/pam.

Read more

proxmox / virtio-blk / disk by-id

Why does the virtio-blk /dev/vda block device not show up in /dev/disk/by-id? Yesterday, I wrote about how Proxmox VE attaches scsi0 and virtio0 block devices differently. That is the starting point for todays question: how come do I get /dev/sda in /dev/disk/by-id while /dev/vda is nowhere to be found? This question is relevant if you're used to referencing disks through /dev/disk/by-id (for example when setting up ZFS, using the device identifiers).

Read more

proxmox / alter default create vm parameters

The Proxmox Virtual Environment has defaults when creating a new VM, but it has no option to change those defaults. Here's a quick example of hacking in some defaults. Why? (Changing SCSI controller does not change existing disks) In the next post I wanted to talk about /dev/disk/by-id and why disks that use the VirtIO SCSI controller do not show up there. A confusing matter in this situation was that creating a VM disk using a different SCSI controller and then switching does not change the storage driver for the existing disks completely!

Read more

openvpn / hardened fox-it openvpn-nl

Today, we will be evaluating OpenVPN-NL — “[a] hardened version of OpenVPN that includes as many of the security measures required to operate in a classified environment as possible — and whether we can use it as a drop-in replacement for regular OpenVPN. While OpenVPN allows many insecure configurations, such as turning off encryption, or the use of outdated cryptographic functions in security critical places, the goal of OpenVPN-NL — a fork created and maintained by Fox-IT — is to strip insecure configuration and verify that the distributed version is uncompromised.

Read more

postgresql inside kubernetes / no space left on device

Running PostgreSQL inside Kubernetes? Getting occasional "No space left on device" errors? Know that 64MB is not enough for everyone. With the advent of more services running inside Kubernetes, we're now running into new issues and complexities specific to the containerization. For instance, to solve the problem of regular file backups of distributed filesystems, we've resorted to using rsync wrapped inside a pod (or sidecar). And now for containerized PostgreSQL, we're running into an artificial memory limit that needs fixing.

Read more

chromium snap / wrong fonts

So, since a couple of weeks my snap-installed Chromium browser on Ubuntu Focal started acting up: suddenly it chooses the wrong fonts on some web pages. The chosen fonts are from the ~/.local/share/fonts/ directory. Look! That's not the correct font. And it's even more apparent that the font is off when seeing the source view. Bah. That's not even a monospaced font. A fix that appeared to work — but unfortunately only temporarily — involves temporarily moving the custom local fonts out of the way and then flushing the font cache:

Read more

stale apparmor config / mysql refuses to start

So, recently we had an issue with a MariaDB server that refused to start. Or, actually, it would start, but before long, SystemD would kill it. But why? # systemctl start mariadb.service Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details. After 90 seconds, it would be killed. systemctl status mariadb.service shows the immediate cause: # systemctl status mariadb.service ... systemd[1]: mariadb.

Read more

zfs / zvol / partition does not show up

On our Proxmox virtual machine I had to go into a volume to quickly fix an IP address. The volume exists on the VM host, so surely mounting is easy. Right? I checked in /dev/zvol/pve2-pool/ where I found the disk: # ls /dev/zvol/pve2-pool/vm-125-virtio0* total 0 lrwxrwxrwx 1 root root 10 Dec 29 15:55 vm-125-virtio0 -> ../../zd48 Good, there's a disk: # fdisk -l /dev/zd48 Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes Disklabel type: dos Disk identifier: 0x000aec27 Device Boot Start End Sectors Size Id Type /dev/zd48p1 * 2048 97656831 97654784 46.

Read more

recap 2020

(I’m sorry, this time the recap is in Dutch instead of English. If you’re reading this, you’re probably Dutch anyway, so it won’t be a problem.) COVID-jaar 2020 is een vreemd jaar geweest, maar voor OSSO was het ook een goed jaar. Waar 2020 nog in mineur begon, omdat we eind 2019 nog een dierbare collega verloren aan een emigratiewens naar het land van de kangoeroes, hebben we in 2020 twee nieuwe collega’s mogen verwelkomen.

Read more

zfs destroy / dataset is busy

Just now, I tried to remove a ZFS dataset, and it reported dataset is busy for no apparent reason. # zfs list -r data NAME USED AVAIL REFER MOUNTPOINT data 3.12T 405G 251M /data data/kubernetes-logging 2.08T 405G 2.08T /data/kubernetes/logging data/rook-config 36.5M 405G 36.5M /data/rook-config data/rook-data 1.03T 708G 753G - # zfs destroy data/kubernetes-logging cannot destroy 'data/kubernetes-logging': dataset is busy The usual suspects were checked: The dataset was not mounted (cat /proc/mounts | grep kubernetes).

Read more

cumulus / postfix in the right vrf

Cumulus Linux is a network operating system. It is a switch, but it also runs Linux OS, allowing us to run our automation tools on it. We use it to automate the configuration of our network. A network where we use VRF (virtual routing and forwarding) to separate customer traffic. The presence of VRFs in the OS however means that we have to tell the daemons in which VRF to run.

Read more

offsite / on-the-fly encrypted backups / gocryptfs

Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data. That scheme works like this: there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server; you grant ssh+rsync access to that system; but only to a specific path; on that path, you mount an encrypted view of your filesystem — a.

Read more

pgp on yubikey / refresh expiry

Generally, I try to follow security best practices. This means that I have my PGP signing, authentication and encryption keys on my YubiKey, and I have configured the keys to expire after a year. Unfortunately, refreshing the expiry every year is not quite enough to store how to do that into muscle memory. Here are the steps relevant to my use case. Putting the keys on the YubiKey in the first place is worth a post of its own.

Read more

tls / testing certificate chains / easycert

The openssl client is a very versatile tool, but also a bit cryptic. The easycert utility from the ossobv/vcutil scripts makes validating/managing certificates easier. easycert from ossobv/vcutil has a few modes of operation: CLI, CGI, generating certificates and testing certificates. Nowadays we mostly use the testing mode: -T The utility is a convenient wrapper around openssl s_client and x509 calls. Get it from github.com/ossobv/vcutil easycert. Usage Run it like this:

Read more

excel / generate sheet password collision

Yesterday, I demonstrated how to brute force the Excel protected sheet/cells password. (Write protection! Not read protection a.k.a. encryption!) Today, I figured there must be a faster way, as the hash is not at all complicated. After fiddling around a little, I hacked together this bit of Python: def reverse_f(wanted): "Calculate Excel protected sheet password" # https://wjd.nu/notes/2020#excel-generate-sheet-password-collision # https://wjd.nu/notes/2020#libreoffice-asking-for-cell-password-brute-force def reverse_rotate(v): "Right shift by one, rotating the right most bit to bit 15" if v & 0x1: return (v >> 1) | 0x4000 return v >> 1 chars = [] valid_tokens = tuple([ord(i) for i in ( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' '0123456789')]) # Length 9 should be enough to go down from 16 to 7 bits: # we skip some shorter solutions, but there are only a few hashes # that benefit from that.

Read more

libreoffice / asking for cell password / brute force

While we were editing a provider-supplied Excel document using LibreOffice, at seemingly random times, it would show a popup asking us for a password to a cell. This popup would only go away if we set a new (non-blank) password on it. Annoying! Apparently, it has to do with Sheet and Cell protection whereby an editing user is disallowed to edit certain cells/rows/sheets in a document. Having certain cells marked read-only, sure.

Read more

docker unprivileged user / becoming root

My colleague was rightly annoyed that our USER www-data docker images greatly hindered effective debugging. Can we become root again, while still keeping the additional secure-by-default non-root images? If we have enough permissions on the filesystem, then: yes, we can. Take the following example, where we’ll be looking at a myproject pod. (You can skip the Kubernetes steps if you already know where the Docker instance resides.) $ kubectl get pods -o wide myproject-66dd6b4dd-jskgf NAME READY STATUS AGE IP NODE myproject-66dd6b4dd-jskgf 1/1 Running 64d 10.

Read more

nss-dns4only / libc / disable AAAA lookups

Have you ever noticed how some applications can do AAAA DNS record lookups even though the host has no IPv6 connectivity? That means double DNS lookups for zero profit. Why is that? And how can you disable it? Problem To make a long story short, a common combination of circumstances can cause useless gratuitous AAAA lookups: Applications that are IPv6 ready (or applications that don’t care); on hosts using libc; where IPv6 is enabled (net.

Read more

gitlab / securing public repositories

In the past, GitLab repositories were created with Public Visibility by default. Now they have a more sensible security setting. Still, it can be nice to assert that public repositories are not Public-by-Accident. How? Well, one fix is to check that Public repositories are in a whitelisted public namespace (e.g. /public/). That way it’s immediately obvious that the repositories herein are visible to everyone. Use a Private browser and go to: https://YOUR_GITLAB_INSTANCE/explore/projects

Read more

more or less useless tips and tricks 3

More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. gsettings set org.gnome.desktop.calendar show-weekdate true — to enable week numbers in the gnome-shell datetime calendar popup. (You may need to set LC_TIME to en_GB so the week starts on a Monday instead of, American style, on a Sunday. You’ll probably have set LC_PAPER too already, to get A4 paper size printing defaults.

Read more

encryption decryption speed / gnupg / openssl

We were looking at encryption ingredients the other day. Because, if we want to compare encryption methods, we shouldn’t compare apples and oranges. With that newfound knowledge, we can run a few speed tests. The aggregated data (raw data sources can be found below): EncryptionDecryption user (ms)sys (ms)total (ms)mem (10K) user (ms)sys (ms)total (ms)mem (10K) gpg 1.4 76453547999354 1573832216060359 gpg 2.2 28633403203507 62123406552515 gpg 2.3* 27522813033527 44943184812536 gpg 2.3* nohw 45082814789 52575473187865537 openssl 17543392093 506421391812506 openssl nohw 35643443908504 27653963161507 customcrypt 333842137592458 356543540002461 First a few notes about the graph:

Read more