Blog

So, these new Kioxia NVMe drives were incrementing the num_err_log_entries as soon as they were inserted into the machine. But the error said INVALID_FIELD. What gives? In contrast to the other (mostly Intel) drives, these drives started incrementing the num_err_log_entries as soon as they were plugged in: # nvme smart-log /dev/nvme21n1 Smart Log for NVME device:nvme21n1 namespace-id:ffffffff ... num_err_log_entries : 932 The relevant errors should be readable in the error-log. All 64 errors in the log looked the same:

In yesterday's post about not being able to connect to the SuperMicro iKVM IPMI, I wondered “why stunnel/openssl did not send error 45 (certificate_expired) for a not-yet-valid certificate.” Here's a closer examination. Quick recap: yesterday, I got SSL alert/error 42 as response to a client certificate that was not yet valid. The server was living in 2015 and refused to accept a client certificate that would be valid first in 2016.

Today I was asked to look at a machine that disallowed iKVM IPMI console access. It allowed access through the “iKVM/HTML5”, but when connecting using the “Console Redirection” (Java client, see also ipmikvm) it would quit after 10 failed attempts. TL;DR: The clock of the machine had been reset to a timestamp earlier than the first validity of the supplied client certificate. After changing the BMC time from 2015 to 2021, everything worked fine again.

The case of the stale (removed but not removed) PVE node in our Proxmox cluster. On one of our virtual machine clusters, a node — pve3 — had been removed on purpose, yet is was still visible in the GUI with a big red cross (because it was unavailable). This was not only ugly, but also caused problems for the node enumeration done by proxmove. The node had been properly removed, according to the removing a cluster node documentation.

How do you enable the noisy build when building OpenSIPS? The one where the actual gcc invocations are not hidden. In various projects the compilation and linking steps called by make are cleaned up, so you only see things like: Compiling db/db_query.c Compiling db/db_id.c ... This looks cleaner. But sometimes you want to see (or temporarily change) the compilation/linking call: gcc -g -O9 -funroll-loops -Wcast-align -Wall [...] -c db/db_query.c -o db/db_query.

When you have a lot of storage devices, it's best practice to assign them to raid arrays or ZFS pools by something identifiable. And preferably something that's also readable when outside a computer. Commonly: the disk manufacturer and the serial number. Usually, both the disk manufacturer and the disk serial number are printed on a small label on the disk. So, if you're in the data center replacing a disk, one glance is sufficient to know you got the correct disk.

What is the smtp_domain in the GitLab configuration? There is also a smtp_address and smtp_user_name; so what would you put in the “domain” field? Contrary to what the examples on GitLab Omnibus SMTP lead you to believe: smtp_domain is the HELO/EHLO domain; i.e. your hostname. RFC 5321 has this to say about the HELO/EHLO parameter: o The domain name given in the EHLO command MUST be either a primary host name (a domain name that resolves to an address RR) or, if the host has no name, an address literal, as described in Section 4.

Quick notes on setting up pam_yubico.so with OpenVPN. Add to OpenVPN server config: plugin /usr/lib/x86_64-linux-gnu/openvpn/plugins/openvpn-plugin-auth-pam.so openvpn # Use a generated token instead of user/password for up # to 16 hours, so you'll need to re-enter your otp daily. auth-gen-token 57600 Sign up at https://upgrade.yubico.com/getapikey/. It's really quick. Store client_id and secret (or id and key respectively). You'll need them in the config below. Get PAM module: # apt-get install --no-install-recommends libpam-yubico Create /etc/pam.

Why does the virtio-blk /dev/vda block device not show up in /dev/disk/by-id? Yesterday, I wrote about how Proxmox VE attaches scsi0 and virtio0 block devices differently. That is the starting point for todays question: how come do I get /dev/sda in /dev/disk/by-id while /dev/vda is nowhere to be found? This question is relevant if you're used to referencing disks through /dev/disk/by-id (for example when setting up ZFS, using the device identifiers).

The Proxmox Virtual Environment has defaults when creating a new VM, but it has no option to change those defaults. Here's a quick example of hacking in some defaults. Why? (Changing SCSI controller does not change existing disks) In the next post I wanted to talk about /dev/disk/by-id and why disks that use the VirtIO SCSI controller do not show up there. A confusing matter in this situation was that creating a VM disk using a different SCSI controller and then switching does not change the storage driver for the existing disks completely!

Today, we will be evaluating OpenVPN-NL — “[a] hardened version of OpenVPN that includes as many of the security measures required to operate in a classified environment as possible — and whether we can use it as a drop-in replacement for regular OpenVPN. While OpenVPN allows many insecure configurations, such as turning off encryption, or the use of outdated cryptographic functions in security critical places, the goal of OpenVPN-NL — a fork created and maintained by Fox-IT — is to strip insecure configuration and verify that the distributed version is uncompromised.

Running PostgreSQL inside Kubernetes? Getting occasional "No space left on device" errors? Know that 64MB is not enough for everyone. With the advent of more services running inside Kubernetes, we're now running into new issues and complexities specific to the containerization. For instance, to solve the problem of regular file backups of distributed filesystems, we've resorted to using rsync wrapped inside a pod (or sidecar). And now for containerized PostgreSQL, we're running into an artificial memory limit that needs fixing.

So, since a couple of weeks my snap-installed Chromium browser on Ubuntu Focal started acting up: suddenly it chooses the wrong fonts on some web pages. The chosen fonts are from the ~/.local/share/fonts/ directory. Look! That's not the correct font. And it's even more apparent that the font is off when seeing the source view. Bah. That's not even a monospaced font. A fix that appeared to work — but unfortunately only temporarily — involves temporarily moving the custom local fonts out of the way and then flushing the font cache:

So, recently we had an issue with a MariaDB server that refused to start. Or, actually, it would start, but before long, SystemD would kill it. But why? # systemctl start mariadb.service Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details. After 90 seconds, it would be killed. systemctl status mariadb.service shows the immediate cause: # systemctl status mariadb.service ... systemd[1]: mariadb.

On our Proxmox virtual machine I had to go into a volume to quickly fix an IP address. The volume exists on the VM host, so surely mounting is easy. Right? I checked in /dev/zvol/pve2-pool/ where I found the disk: # ls /dev/zvol/pve2-pool/vm-125-virtio0* total 0 lrwxrwxrwx 1 root root 10 Dec 29 15:55 vm-125-virtio0 -> ../../zd48 Good, there's a disk: # fdisk -l /dev/zd48 Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes Disklabel type: dos Disk identifier: 0x000aec27 Device Boot Start End Sectors Size Id Type /dev/zd48p1 * 2048 97656831 97654784 46.

(I’m sorry, this time the recap is in Dutch instead of English. If you’re reading this, you’re probably Dutch anyway, so it won’t be a problem.) COVID-jaar 2020 is een vreemd jaar geweest, maar voor OSSO was het ook een goed jaar. Waar 2020 nog in mineur begon, omdat we eind 2019 nog een dierbare collega verloren aan een emigratiewens naar het land van de kangoeroes, hebben we in 2020 twee nieuwe collega’s mogen verwelkomen.

Just now, I tried to remove a ZFS dataset, and it reported dataset is busy for no apparent reason. # zfs list -r data NAME USED AVAIL REFER MOUNTPOINT data 3.12T 405G 251M /data data/kubernetes-logging 2.08T 405G 2.08T /data/kubernetes/logging data/rook-config 36.5M 405G 36.5M /data/rook-config data/rook-data 1.03T 708G 753G - # zfs destroy data/kubernetes-logging cannot destroy 'data/kubernetes-logging': dataset is busy The usual suspects were checked: The dataset was not mounted (cat /proc/mounts | grep kubernetes).

Cumulus Linux is a network operating system. It is a switch, but it also runs Linux OS, allowing us to run our automation tools on it. We use it to automate the configuration of our network. A network where we use VRF (virtual routing and forwarding) to separate customer traffic. The presence of VRFs in the OS however means that we have to tell the daemons in which VRF to run.

Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data. That scheme works like this: there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server; you grant ssh+rsync access to that system; but only to a specific path; on that path, you mount an encrypted view of your filesystem — a.

Generally, I try to follow security best practices. This means that I have my PGP signing, authentication and encryption keys on my YubiKey, and I have configured the keys to expire after a year. Unfortunately, refreshing the expiry every year is not quite enough to store how to do that into muscle memory. Here are the steps relevant to my use case. Putting the keys on the YubiKey in the first place is worth a post of its own.

The openssl client is a very versatile tool, but also a bit cryptic. The easycert utility from the ossobv/vcutil scripts makes validating/managing certificates easier. easycert from ossobv/vcutil has a few modes of operation: CLI, CGI, generating certificates and testing certificates. Nowadays we mostly use the testing mode: -T The utility is a convenient wrapper around openssl s_client and x509 calls. Get it from github.com/ossobv/vcutil easycert. Usage Run it like this:

Yesterday, I demonstrated how to brute force the Excel protected sheet/cells password. (Write protection! Not read protection a.k.a. encryption!) Today, I figured there must be a faster way, as the hash is not at all complicated. After fiddling around a little, I hacked together this bit of Python: def reverse_f(wanted): "Calculate Excel protected sheet password" # https://wjd.nu/notes/2020#excel-generate-sheet-password-collision # https://wjd.nu/notes/2020#libreoffice-asking-for-cell-password-brute-force def reverse_rotate(v): "Right shift by one, rotating the right most bit to bit 15" if v & 0x1: return (v >> 1) | 0x4000 return v >> 1 chars = [] valid_tokens = tuple([ord(i) for i in ( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' '0123456789')]) # Length 9 should be enough to go down from 16 to 7 bits: # we skip some shorter solutions, but there are only a few hashes # that benefit from that.

While we were editing a provider-supplied Excel document using LibreOffice, at seemingly random times, it would show a popup asking us for a password to a cell. This popup would only go away if we set a new (non-blank) password on it. Annoying! Apparently, it has to do with Sheet and Cell protection whereby an editing user is disallowed to edit certain cells/rows/sheets in a document. Having certain cells marked read-only, sure.

My colleague was rightly annoyed that our USER www-data docker images greatly hindered effective debugging. Can we become root again, while still keeping the additional secure-by-default non-root images? If we have enough permissions on the filesystem, then: yes, we can. Take the following example, where we’ll be looking at a myproject pod. (You can skip the Kubernetes steps if you already know where the Docker instance resides.) $ kubectl get pods -o wide myproject-66dd6b4dd-jskgf NAME READY STATUS AGE IP NODE myproject-66dd6b4dd-jskgf 1/1 Running 64d 10.

Have you ever noticed how some applications can do AAAA DNS record lookups even though the host has no IPv6 connectivity? That means double DNS lookups for zero profit. Why is that? And how can you disable it? Problem To make a long story short, a common combination of circumstances can cause useless gratuitous AAAA lookups: Applications that are IPv6 ready (or applications that don't care); on hosts using libc; where IPv6 is enabled (net.

In the past, GitLab repositories were created with Public Visibility by default. Now they have a more sensible security setting. Still, it can be nice to assert that public repositories are not Public-by-Accident. How? Well, one fix is to check that Public repositories are in a whitelisted public namespace (e.g. /public/). That way it’s immediately obvious that the repositories herein are visible to everyone. Use a Private browser and go to: https://YOUR_GITLAB_INSTANCE/explore/projects

More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. gsettings set org.gnome.desktop.calendar show-weekdate true — to enable week numbers in the gnome-shell datetime calendar popup. (You may need to set LC_TIME to en_GB so the week starts on a Monday instead of, American style, on a Sunday. You’ll probably have set LC_PAPER too already, to get A4 paper size printing defaults.

We were looking at encryption ingredients the other day. Because, if we want to compare encryption methods, we shouldn’t compare apples and oranges. With that newfound knowledge, we can run a few speed tests. The aggregated data (raw data sources can be found below): EncryptionDecryption user (ms)sys (ms)total (ms)mem (10K) user (ms)sys (ms)total (ms)mem (10K) gpg 1.4 76453547999354 1573832216060359 gpg 2.2 28633403203507 62123406552515 gpg 2.3* 27522813033527 44943184812536 gpg 2.3* nohw 45082814789 52575473187865537 openssl 17543392093 506421391812506 openssl nohw 35643443908504 27653963161507 customcrypt 333842137592458 356543540002461 First a few notes about the graph:

While investigating the most appropriate encryption cipher and format, I realised I didn’t have enough vocabulary on the subject. This post aims to close that knowledge gap somewhat. I’m looking at symmetric ciphers here, as they are used when storing lots of data. (In fact, when encrypting larger amounts of data, public/private key encryption (an asymmetric cipher) is only used to encrypt a separate key for the symmetric cipher, which is then used for bulk of the data.

After some upgrades, I suddenly noticed unexpected sh: printf: I/O error output. Some debugging later, it turns out that it's the Dash way of informing us of a PIPE error. Apparently salt's cmd.run can cause so little output buffering, that the debianutils which command can be aborted mid-output. Not-broken example: $ which python3 python false | head -n1 /usr/bin/python3 Broken example, through a salt cmd.run call: $ salt 'example.com' cmd.run 'which python3 python false | head -n1' example.

Connecting to the new SuperMicro iKVM management interfaces requires a working Java in your browser (IcedTea browser plugin). It will launch a Java Web Start (javaws) application. And java plugins (and needless web forms) are a pain in the behind. Is there a better way? Obviously, running from the web interface just means downloading a Java application, and running it locally. So why can’t we do that directly, and skip the IcedTea Java browser plugins?

Recently, an automated job of mine failed because the latest and greatest Python refused to validate an SSL certificate with 127.0.0.1 in the common name (CN). Apparently CN=127.0.0.1 will not be accepted anymore, as using the common name for hostname validation has been deprecated for ages now. The fix? Use subject alternative names (SANs). Generally, you’ll already have these when you have your certificates signed by somebody else. But if you’re signing certificates yourself, you’ll need to know how to pass them to openssl:

Can you get your 3CX Phone System to connect to your SIP provider Trunk over TLS, when the server uses a Let’s Encrypt certificate? 3CX documentation on this topic is scarce. There are posts like 3CX Forum: SIP TLS: It’s fairly straightforward: Your provider must give you a TLS Root Certificate (.pem) so you can encrypt the traffic. If they have SRV records the system will automatically know where to connect for TLS mode (Auto Discovery option on General trunk tab), or the provider might just tell you to change your port to the one they are listening to for TLS connections.

(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2019 highlights We are looking for new colleagues! The new website is finally up! Built a Cumulus network with automated deployment using Ansible and Netbox as the source of truth. Managed kubernetes has become a staple. Extended our ISO27001:2017 and NEN7510:2017 certifications. Visited Legoland as a team building exercise. Many improvements! Ronald may actually leave for australia, like for real, not kidding, probably, if they finally let him in, in the not too distant future.

When loading old (Django) projects in K8S, we've decided to give PostgreSQL a go as default database. Here are some notes that aid in importing. After looking around, pgloader seems to be the right tool for the job. Feature and stability wise it beats any other solution. And it's available on recent Debian/Ubuntu. We need access to the remote PostgreSQL db; because pgloader will not provide an SQL dump (for reasons).

On the Asterisk PBX Development page you'll find a reference to Doxygen generated documentation: Most of the documentation related to the source code is embedded in the source files and is processed with Doxygen. The latest version of the Doxygen generated source documentation can be found on http://doxygen.asterisk.org. However, it cannot be found there, as that URL has been returning a 503 Service Unavailable for quite some time now.

(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2018 highlights Edgar joined our team Managed kubernetes moving forward Many awesome upstream kubernetes releases Keeping up integrating improvements in our default kubernetes “distribution” Further expanding our managed kubernetes install base Updated NEN7510:2011 to NEN7510:2017 ISO27001 ISMS improvements and further integrating customer scope New website coming up - bijna klaar TM.

If you prefer gpg2 over gpg, building a debian package with debuild or gbp buildpackage may require some fiddling. In my case, I’m using gpg2 instead of gpg for signing because unlike version 1, version 2 does PGP key forwarding. That way I can sign on a remote machine, using a local PGP key card. However gbp buildpackage, dpkg-buildpackage and debuild are hardwired to call gpg. And — it turns out — using a simple /usr/local/bin/gpg to /usr/bin/gpg2 symlink was not sufficient to convince gbp (and debuild) to use the gpg2 binary, while for dpkg-buildpackage that is sufficient.

Just now I ran into an IPython interpreter inside a Docker container inside Kubernetes misbehaving: After starting ipython inside a kubectl for a second time, IPython wouldn’t show the input prompt. It only showed the output prompt. Turns out it was due to the terminal settings. For some reasons, after logging out of kubectl exec, the next exec would get 0 rows and 0 columns; as if someone had run stty rows 0 on the terminal.

In Debian/Stretch, the default ViM settings have been changed — for the worse, in my opinion. However, undoing the bad settings is not a matter of fixing them in your ~/.vimrc, because when that file is detected no defaults at all are set. The quick fix is to create a custom /etc/vim/vimrc.local file with the following settings: " Instead of auto-sourcing this afterwards, source it now. source $VIMRUNTIME/defaults.vim let g:skip_defaults_vim = 1 " Now we undo the "wrong" settings.

A while, I’ve been looking at a stray /core file in some of our daily Xenial Docker images. Time to find out where it comes from. Tracing with a few well placed RUN ls -l /core || true, tells us that the dump appeared after a large RUN statement and not during one. Running gdb on the core revealed that it was a dump of auplink, a part of Docker. Opening the core on a Xenial machine with docker installed, showed the following backtrace:

After upgrading from Ubuntu 17.10 to Ubuntu 18.04, and rebooting, the GNOME Display Manager (gdm) went into a restart loop. No promised speed gains. Instead, I got an unusable desktop. Being quick with CTRL+ALT+F3, I could enter my username and password in the text console after a couple attempts — the gdm restart would continuously steal console/tty focus — after which a sudo systemctl stop gdm was possible. This left me with a shell and plenty of time to examine the situation.

A quick howto on checking SSL/TLS client certificates from Django/Python. Generally, when you want to use client certificates, you’ll let the HTTPS server (e.g. NGINX) do the certificate validation. For NGINX you’d add this config, and be done with it. # TLS server certificate config: ... # TLS client certificate config: ssl_verify_client on; # or 'optional' ssl_client_certificate /PATH/TO/my-ca.crt; ... location ... { ... # $ssl_client_s_dn contains: "/C=.../O=.../CN=...", where you're # generally interested in the CN-part (commonName) to identify the # "who".

Where do you place the application inside the Docker image? In the past, when deploying Python/Django projects, we’d put them in /srv/django-projects/APPNAME on a (possibly shared) machine. The python-virtualenv that came with it, went into /srv/virtualenvs/APPNAME. Now that we’re dockerizing many projects, we don’t need the virtualenv (there is only one environment) and we don’t need the APPNAME either (there is only one application). So, where should we place the project?

After having gotten used to Unity on the Ubuntu desktop, with Ubuntu Artful it is time to say goodbye. When Ubuntu first added the Unity shell with just the sidebar with big buttons, in favor of the more traditional GNOME with its Windows 95 style interface, many were skeptical, me included. But removing the clutter was good, and I've happily worked with it for years. And you really don't want to waste time tweaking your desktop away from the OS provided defaults.

A mismash of bugs and workarounds causes the copy buffer (X selection) to get wiped some of the time in my recent desktop environment. And that in a seemingly unpredictable manner. The following bug is mostly in play: GNOME VTE soft reset wipes selection That bug causes: reset(1) to wipe the middle-mouse (primary) buffer (although this differs per system — could not put my finger on this); reset(1) to wipe the clipboard buffer, but only if the reset was called from window that originated the current clipboard buffer contents; GNU screen(1) initialization to misbehave as reset does, as described above — even through an ssh session — by wiping the buffer, if TERM=xterm-256color.

Today we ran into a dovecot/imap crash on a Xenial box. The Dovecot in question was the patched dovecot-2.2.22. Due to an as of yet unexplained cause, reading mail through Thunderbird mail client worked fine, but when opening a message with Roundcube (webmail), most messages would give an odd error about a “message that could not be opened”. An IMAP trace of Roundcube revealed that the IMAP server stopped responding after the client A0004 UID FETCH command.

Information regarding Meltdown and Spectre attacks. Current state Waiting for software patch availability. Patched ubuntu kernels are available for testing. Updates: 20180104: Created blogpost 20180105: Added new information/links 20180105: Status update 20180108: Added information from Redhat about performance impact from patches. 20180108: Updated links list. 20180108: Status update Links https://spectreattack.com / https://meltdownattack.com (same site) https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-every-modern-processor-has-unfixable-security-flaws/ https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-heres-what-intel-apple-microsoft-others-are-doing-about-it/ https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/ High level description CVE’s Spectre - CVE-2017-5715 Spectre - CVE-2017-5753 Meltdown - CVE-2017-5754 As described on: https://spectreattack.

2017 ISO27001 certified + NEN7510 Ewout joined our team as an SRE More projects opensourced at github: https://github.com/ossobv Uradesign designed logo’s for several of our open source projects Started providing Kubernetes as a Service / Managed Kubernetes Lots of interesting stuff

In 2015, I wrote a quick recipe to use Vim F7-key flake8 checking for both python2 and python3 using the nvie/vim-flake8 Vim plugin. Here's a quick update that works today. Tested on Ubuntu/Zesty. $ sudo apt-get install python3-flake8 # py3 version, no cli $ sudo pip -H install python-flake8 # py2 version, with cli $ sudo cp /usr/local/bin/flake8{,.2} # copy to flake8.2 $ sudo sed -i -e 's/python$/python3/' /usr/local/bin/flake8 # update shebang $ sudo mv /usr/local/bin/flake8{,.