/ Recent content in Home on OSSO B.V. Hugo -- gohugo.io Wed, 20 Nov 2019 14:14:49 +0100 /blog/2026/linux-local-root-exploit-module-vetting/ Fri, 08 May 2026 00:00:00 +0000 /blog/2026/linux-local-root-exploit-module-vetting/ Recently, we were greeted with the Copy Fail Linux kernel vulnerability. Mitigating this was a matter of denylisting a module. But, only eight days later, there was another exploit, also (ab)using AF_ALG and kernel module autoloading. I'm betting this is not the last, now that the kernel is scrutinized using AI models that keep getting more advanced. Luckily, we had our machine inventory up to date. So when CVE-2026-31431 ("Copy Fail") came along, deploying a mitigation was a matter of: /blog/2026/macos-tahoe-ecn-slow-downloads/ Wed, 04 Feb 2026 00:00:00 +0000 /blog/2026/macos-tahoe-ecn-slow-downloads/ Recently, a customer reported an intermittent but frustrating issue: since upgrading to macOS 26 (Tahoe), downloads from our servers were occasionally crawling. Not always, and not for everyone. The culprit turned out to be a combination of Explicit Congestion Notification (ECN), NIC offloading limitations, and the way classic congestion control algorithms react to "well-intentioned" signals. The macOS Tahoe ECN lottery The first mystery was the intermittency. We noticed that some connections were fast, while others to the same server appeared throttled. /blog/2025/recap2025/ Fri, 19 Dec 2025 08:30:00 +0100 /blog/2025/recap2025/ 2025 – Finding the Turbo Button A year of momentum. After the “certification heavy” focus of 2024, 2025 felt like shifting into a higher gear. We’ve seen our infrastructure move to 100G, our hardware embrace a new generation of speed, and our team grow into a tighter, faster unit. Looking back at the last twelve months, the theme hasn’t just been “maintenance” — it’s been about refinement. We’ve been busy cleaning out the old (goodbye spinning rust and legacy loadbalancers) to make room for the new (hello EPYC processors and Nokia core routers). /blog/2025/k8s-gateway-api/ Tue, 16 Dec 2025 00:00:00 +0000 /blog/2025/k8s-gateway-api/ Transitioning K8S towards the Gateway API We’ve heard the chatter about the retirement of the ingress-nginx ingress-controller. It seems to be a hot topic! We’ve had quite a few people ask: “Hey OSSO, have you seen this already?" Yes! We have! Actually, we’re right on top of it. We plan to smoothly transition and adapt, and today we want to clarify OSSO’s approach going forward. This post outlines the transition from the established Kubernetes Ingress resource to its successor Gateway API, detailing the reasons for the transition, our proposed migration path, and the key architectural concepts. /blog/2025/ubuntu-gnome-config-issues-2025/ Tue, 02 Dec 2025 00:00:00 +0000 /blog/2025/ubuntu-gnome-config-issues-2025/ I had flushed my desktop and was ready to start fresh. Still using Ubuntu for now. It does take a few hacks and tricks to get my system right. Disable and purge snap For one, I'm removing snap. The Ubuntu Snap takeover has cost me so many man hours. As mentioned in the linked post, removing it and going with the Linux Mint packages was a great decision. Not once have I regretted not having Snap installed. /blog/2025/proxmox-vm-no-arp-replies-mellanox-nic/ Tue, 30 Sep 2025 00:00:00 +0000 /blog/2025/proxmox-vm-no-arp-replies-mellanox-nic/ Recently, when upgrading a Proxmox host from 7.2 to 8.4, we ran into a strange issue. The hypervisor host had network connectivity, but the VM guests did not. The upgrade was continued to Proxmox 9.0, but that did not fix the problem. The host was fine. We could ssh in, ping the gateway, do updates, whatever — but the guests had trouble communicating. In particular: the guests appeared unable to find the gateway at layer 2. /blog/2025/zabbix-wrong-nvme-temperature-readings/ Thu, 18 Sep 2025 00:00:00 +0000 /blog/2025/zabbix-wrong-nvme-temperature-readings/ After installing some new nodes Zabbix began reporting NVMe drive temperatures that were... alarming. Think "above 60 °C" alarming. But when I checked the same drives manually, the values were sane. So, either the drives were gaslighting me, or Zabbix was. The search Decomposing what our monitoring scripts do was really just calling nvme-cli: # nvme smart-log /dev/nvme0n1 | grep temper temperature : 33 °C (306 K) Nothing wrong there. /security/ Tue, 16 Sep 2025 13:40:31 +0200 /security/ Information Security at OSSO At OSSO, information security is a core component of our operations, not an afterthought. We maintain a robust security posture to protect our infrastructure and customer data, and we believe in being transparent about our practices. This page outlines our formal certifications and our implementation of key industry standards. /blog/2025/loopback-through-leaf-single-nic-iperf-test/ Sun, 06 Jul 2025 00:00:00 +0000 /blog/2025/loopback-through-leaf-single-nic-iperf-test/ When bringing up new servers, especially with new or untested network interface cards (NICs) or cables, you might want to test their raw network throughput. While standard iperf tests between two different machines are common, sometimes you want to isolate the test to a single NIC, pushing traffic through it and back, effectively looping it at a nearby switch. This verifies the physical path, the SFP, and the NIC's transmit/receive capabilities without involving another server's NIC or network stack. /blog/2025/invalid-elf-header-magic-rook-ceph-k8s/ Sat, 31 May 2025 00:00:00 +0000 /blog/2025/invalid-elf-header-magic-rook-ceph-k8s/ Yesterday, we noticed some previously unseen messages in the logs: Invalid ELF header magic: != \x7fELF. Of course an ELF binary should start with a valid header. Who/what is trying to run something else? These messages appeared on Kubernetes nodes that were recently installed and not yet fully in use for production workloads. Coincidentally, these particular nodes were running Ubuntu 24.04 (Noble Numbat), while the rest of this cluster was still on older versions. /blog/2025/nginx-no-realip-in-logs-bug/ Fri, 30 May 2025 00:00:00 +0000 /blog/2025/nginx-no-realip-in-logs-bug/ Recently we were doing a vulnerability scan on an external endpoint of a website. During this time, we noticed simultaneous suspicious activity coming from inside our internal network. What's the deal? To make a long story short: we use a proxy (like HAProxy) in front of our webservers, and it speaks the Proxy Protocol to pass along the original client's IP address. That way, when a request hits our internal nginx server, the logs show the real client's IP, not the IP of the reverse proxy that actually made the connection. /blog/2025/supermicro-ikvm-expired-certificate/ Thu, 29 May 2025 00:00:00 +0000 /blog/2025/supermicro-ikvm-expired-certificate/ Supermicro computers generally work very well. At our company we've been using them for ages. One thing that does keep giving us the occasional trouble however is the BMC. Today, we'll be looking at expired certificates of the iKVM interface. Introduction First, some definitions, which we generally use interchangeably: The Baseboard Management Controller (BMC) is the chipset core of the Intelligent Platform Management Interface (IPMI) implementation, which we mostly use to remotely attach a Keyboard, Video and Mouse (KVM, over IP). /blog/2025/pveproxy-systemd-journald-perl-buffering/ Tue, 15 Apr 2025 00:00:00 +0000 /blog/2025/pveproxy-systemd-journald-perl-buffering/ Because I was debugging — what later turned out to be — an application firewall that was messing with my connections to Proxmox pveproxy on port 8006, I wanted more debug logs. Telling pveproxy to give me more info gave me a bit of a hard time. Connections from a remote peer would get disconnected, and it initially looked like pveproxy was to blame. Step one: get more info from pveproxy. /blog/2025/sed-regular-expressions-optimizing/ Mon, 17 Mar 2025 00:00:00 +0000 /blog/2025/sed-regular-expressions-optimizing/ Last week, we were looking at using sed(1) to remove line feeds from CSV files. The regular expressions in that post could use some further inspection. I'm sorry that this is a rather dry post. I wanted to get the numbers out there because the differences are significant. But without the example runs you're just left with numbers. Feel free to skip right to the conclusions. To recap: we were working on multiline CSV files. /blog/2025/sed-remove-line-feeds-csv/ Sat, 15 Mar 2025 00:00:00 +0000 /blog/2025/sed-remove-line-feeds-csv/ Can you use sed(1) to remove line feeds? Yes you can. Even though sed, the stream editor, is line based, you can use it to remove line feeds from output. In this post we'll be exploring how that is done and look into how a slight refactoring step can provide a big speed gain. The task at hand is removing line feeds from CSV file with multiline strings. Why? Because I want to use grep and awk on it, and multiline strings would complicate that a lot. /blog/2024/recap2024/ Fri, 20 Dec 2024 09:30:00 +0100 /blog/2024/recap2024/ 2024 – a story in four acts The end of the year. A good time to reflect on what happened the past twelve months. The ups, the downs. What did we achieve? And what can we look forward to? Some of you have asked to be updated on what we’re working on at OSSO. We’ll gladly share the gist here in this recap, which is in English for once (after four Dutch editions in 2020, 2021, 2022 and 2023). /blog/2024/mysql-binlog-replay-max-allowed-packet/ Fri, 03 May 2024 00:00:00 +0000 /blog/2024/mysql-binlog-replay-max-allowed-packet/ Trying to replay MySQL binlogs? And running into max_allowed_packet errors? Fear not. Likely this is not corruption, but packets that really are that big. In 2021 I wrote about mariabackup and selective table restore. That blog entry shows how one might restore a mariabackup-saved snapshot of a MariaDB (MySQL) database. Specifically, it focuses on recovering only one or two tables selectively and it details optional GPG decryption and decompression. Here's a recap of the mandatory steps for full recovery and additionally how to replay binlogs to a specific point in time. /blog/2024/mariadb-check-table-galera-locking/ Wed, 17 Apr 2024 00:00:00 +0000 /blog/2024/mariadb-check-table-galera-locking/ After upgrading some database nodes from MariaDB 10.3 to 10.6 we encountered some issues with tables not being fully correct. We'd like to CHECK (and maybe REPAIR) TABLE the entire database. But the database must not block queries on other nodes of the cluster. The corruption we saw might have been a small corruption that had crept in during even earlier upgrades, we don't know. But we do know that we'd like to get this sorted before the corruption gets worse after further upgrades. /blog/2024/systemd-networkd-wait-online-stalling-and-failing/ Fri, 29 Mar 2024 00:00:00 +0000 /blog/2024/systemd-networkd-wait-online-stalling-and-failing/ systemd-networkd-wait-online.service failing? Maybe it's IPv6. Do you have a list of failed systemd units that looks like this? # systemctl list-units --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured Then that could be due to IPv6 networking not being set up properly. Check the interfaces list: # ip -br a lo UNKNOWN 127.0.0.1/8 ::1/128 ens18 UP 10.20.30.41/31 fe80::a124:092f:fa18:109e/64 ens19 UP 192.168.1.3/31 fe80::a124:092f:fa15:01d5/64 Proper IPv4 IPs, and only link-local IPs for IPv6? /blog/2024/nmap-ssl-enum-ciphers-haproxy-tls-no-results/ Fri, 22 Mar 2024 00:00:00 +0000 /blog/2024/nmap-ssl-enum-ciphers-haproxy-tls-no-results/ When doing an nmap ssl-enum-ciphers scan on a Haproxy machine, I got zero working ciphers. But connecting with TLS worked just fine. What is up? While updating TLS ciphers to disable certain unsafe ciphers, we wanted to test the current offers. nmap with the ssl-enum-ciphers script should work fine for this, and it is reasonably fast: $ nmap -Pn --script=ssl-enum-ciphers -p 443 google.com ... 443/tcp open https | ssl-enum-ciphers: | TLSv1. /blog/2024/gpg-agent-ssh-ed25519-agent-refused/ Tue, 23 Jan 2024 00:00:00 +0000 /blog/2024/gpg-agent-ssh-ed25519-agent-refused/ After putting in all the work to get ED25519 OpenPGP keys in my Yubikey smart card, I was slightly frustrated that the SSH support "sometimes" didn't work. I thought I had tested that it worked, but today it didn't. $ ssh my_server sign_and_send_pubkey: signing failed for ED25519 "cardno:000612345678" from agent: agent refused operation walter@my_server: Permission denied (publickey). That's odd. Adding some debug to the gpg-agent — debug 1024 in gpg-agent.conf — got me this: /blog/2023/recap2023/ Fri, 22 Dec 2023 18:13:00 +0100 /blog/2023/recap2023/ 2023 – betere omgevingen, betere security Waar we vorig jaar druk bezig zijn geweest legacy op te ruimen, was dit jaar meer een jaar van de uitbreidingen: ArgoCD, NetworkPolicies, Cilium, Loki/Mimir… ContainerDay Security Met het gehele OSSO team zijn we afgelopen maart naar Hamburg geweest, naar ContainerDay Security 2023. Daar hebben we bijgepraat over de huidige stand van zaken op security gebied binnen Kubernetes. Hier kwam het laatste zetje wat we nodig hadden om Cilium te gaan uitproberen voor Kubernetes networking. /blog/2023/mplayer-screen-saver-wayland/ Sun, 15 Oct 2023 00:00:00 +0000 /blog/2023/mplayer-screen-saver-wayland/ I don't usually watch movies from my laptop, but when I do, I don't want the screen saver to kick in. Occasionally, I notice that my old time movie player mplayer does not inhibit the screen saver or screen blanking. That means that 5 minutes into a movie, the screen turns black. I don't think this used to be the case. Maybe it's because I'm running the Wayland display server in GNOME now. /blog/2023/bash-postfix-health-check-dev-tcp/ Wed, 20 Sep 2023 00:00:00 +0000 /blog/2023/bash-postfix-health-check-dev-tcp/ Using /dev/tcp in bash for a health check? Here's an example. I had a script that used netcat to connect to a Postfix email daemon to check its health status. To avoid pipelining errors I had it sleep between each write. The core looked somewhat like this: messages=$(for x in \ 'EHLO localhost' \ 'MAIL FROM:<healthz@localhost>' \ 'RCPT TO:<postmaster@example.com>' \ RSET \ QUIT do sleep 0. /blog/2023/gpgv-can-t-allocate-lock-for/ Mon, 18 Sep 2023 00:00:00 +0000 /blog/2023/gpgv-can-t-allocate-lock-for/ gpgv prints out a warning that it cannot allocate a lock. This looks like something should be fixable, but it isn't. Observed with gpg version 2.2.27-3ubuntu2.1: $ gpgv </dev/null gpgv: can't allocate lock for '/home/walter/.gnupg/trustedkeys.gpg' gpgv: verify signatures failed: Unknown system error The issue at hand here is the “gpgv: can't allocate lock for '/home/walter/.gnupg/trustedkeys.gpg'”. Can we fix something to suppress that? TL;DR: no Investigation Fire up the debugger gdb and break at keybox_lock. /blog/2023/etckeeper-git-pack-objects-died-of-signal-9/ Fri, 15 Sep 2023 00:00:00 +0000 /blog/2023/etckeeper-git-pack-objects-died-of-signal-9/ Is your etckeeper dying on the git gc? On several machines I have now seen git run out of memory when trying to do repo optimization and garbage collection. Usually this happened in /etc where we like to have etckeeper. It might look like this: ... warning: The last gc run reported the following. Please correct the root cause and remove .git/gc.log Automatic cleanup will not be performed until the file is removed. /blog/2023/segfault-in-library-addr2line-objdump/ Thu, 14 Sep 2023 00:00:00 +0000 /blog/2023/segfault-in-library-addr2line-objdump/ Yesterday, we spotted some SEGFAULTs on an Ubuntu/Focal server. We did not have core dumps, but the kernel message in dmesg was sufficient to find a culprit. The observed messages were these: nginx[854]: segfault at 6d702e746379 ip 00007ff40dc2f5a3 sp 00007fffd51c8420 error 4 in libperl.so.5.30.0[7ff40dbc7000+166000] Code: 48 89 43 10 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 0f b6 7f 30 48 c1 e8 03 48 29 f8 48 89 c3 74 89 48 8b 02 <4c> 8b 68 10 4d 85 ed 0f 84 28 01 00 00 0f b6 40 30 49 c1 ed 03 49 nginx[951947]: segfault at 10 ip 00007fba4a1645a3 sp 00007ffe57b0f8a0 error 4 in libperl. /blog/2023/qpress-qz1-extension/ Tue, 12 Sep 2023 00:00:00 +0000 /blog/2023/qpress-qz1-extension/ This is a quick note to my future self. As we're using qpress less in favor of lz4 that has been available on Ubuntu Focal and above, we're inclined to forget what the .qz1 extension means. Wondering what files with the .qz1 extension do? This concerns single stream qpress compressed files. Historically qpress uses the .qp extension, but that concerns multifile archives. The qpress binary can write compressed streams to stdout, but it will not decompress them to stdout. /blog/2023/dirmngr-keyserver-disable-ipv6-bionic/ Fri, 01 Sep 2023 00:00:00 +0000 /blog/2023/dirmngr-keyserver-disable-ipv6-bionic/ This morning a build pipeline failed. dirmngr called by apt-key tried to use IPv6, even though it was disabled. The build logs had this to say: 21.40 + apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8 21.50 Warning: apt-key output should not be parsed (stdout is not a terminal) 21.57 Executing: /tmp/apt-key-gpghome.KzTTOZjgZP/gpg.1.sh --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8 21.65 gpg: keyserver receive failed: Cannot assign requested address This was strange for a number of reasons: /blog/2023/mariadb-gdb-debugging-shutdown-deadlock-part-2/ Thu, 31 Aug 2023 00:00:00 +0000 /blog/2023/mariadb-gdb-debugging-shutdown-deadlock-part-2/ We were looking at a core dump of MariaDB instance that was in a deadlocked state. It was killed with SIGABRT. We now like to get MySQL thread info from it. Python support in gdb is here to help. To recap: we were dissecting the core dump with gdb (gdb /usr/sbin/mariadbd core.208161), found out that it was waiting on one of the threads to stop before stopping completely. We now want to know what exactly it was waiting for. /blog/2023/mariadb-gdb-debugging-shutdown-deadlock-part-1/ Wed, 30 Aug 2023 00:00:00 +0000 /blog/2023/mariadb-gdb-debugging-shutdown-deadlock-part-1/ I was asked to look into a MariaDB deadlock during shutdown. We had a core dump (luckily). Now it's time to dissect it. Ensure you can get core dumps For starters, you want a nice core dump whenever you hit an issue. We do. Here are some tips to ensure you do too. You need to have a couple of parameters set correctly: the equivalent of the systemd LimitCORE=infinity and the assurance that the working directory is writable (maybe WorkingDirectory=/var/lib/mysql). /blog/2023/laptop-battery-discharge-logging/ Sat, 08 Jul 2023 00:00:00 +0000 /blog/2023/laptop-battery-discharge-logging/ I recently got a new Framework Laptop. It is generally nice. Apart from the reflective screen, and the excessive battery consumption. In suspend mode, it draws too much battery: more than a Watt. This is a known issue. The worst offenders are the expansion cards. For instance the USB-A cards consume about 350mW each, just by being plugged in. To do some testing, I whipped up a script allowing easy access to battery usage logs: discharge-log /blog/2023/viewing-unencrypted-traffic-ltrace-bpftrace/ Mon, 03 Jul 2023 00:00:00 +0000 /blog/2023/viewing-unencrypted-traffic-ltrace-bpftrace/ Can we view TLS-encrypted traffic on the originating or terminating host, without having to decode the data from the wire? This is a question that comes up every now and then when trying to debug a service by looking at how it communicates. For the most insight, we should capture the encrypted traffic and use the (logged!) pre-master secret keys. See example details in make-master-secret-log discussing how to have HAProxy log them. /blog/2023/removing-auditd-disabling-logging/ Fri, 30 Jun 2023 00:00:00 +0000 /blog/2023/removing-auditd-disabling-logging/ After installing auditd for testing purposes and removing it again, my kernel logs got flooded with messages. How do I disable them? If you happened to have installed auditd, it is likely that the kernel audit subsystem was enabled. Even when there are no rules left (auditctl -l) you can still get more messages in your kernel logs than before. For instance, after uninstalling auditd, I still get the following ones: /blog/2023/netplan-docker0-bind-on-172-17-0-1/ Thu, 15 Jun 2023 00:00:00 +0000 /blog/2023/netplan-docker0-bind-on-172-17-0-1/ If you want to bind your host-service to a the docker IP, exposing it to docker instances, means that that IP needs to exist first. If it doesn't, your log might look like this: LOG: listening on IPv4 address "127.0.0.1", port 5432 LOG: could not bind IPv4 address "172.17.0.1": Cannot assign requested address WARNING: could not create listen socket for "172.17.0.1" LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" As you probaby know, you cannot bind to an IP that is not configured on an interface anywhere — barring the net. /blog/2023/ansible-ipv6-addresses-without-link-local/ Thu, 13 Apr 2023 00:00:00 +0000 /blog/2023/ansible-ipv6-addresses-without-link-local/ Trying to get the IPv6 addresses as ansible fact, but getting unwanted link_local scope addresses? Maybe you're landing here because you did not see a solution at “all_ipv6_addresses includes link local addresses”. (I find it most odd that they locked the GitHub ticket, eliminating the possibility for anyone to reply with a fix.) The ansible_all_ipv6_addresses | reject("ansible.utils.in_network", "fe80::/10") construct works for me. For example: local-address=127.0.0.1, ::1, {{ ( ansible_all_ipv4_addresses + ( ansible_all_ipv6_addresses|reject("ansible. /blog/2023/zabbix-server-jammy-upgrade-missing-font/ Fri, 31 Mar 2023 00:00:00 +0000 /blog/2023/zabbix-server-jammy-upgrade-missing-font/ The other day, we upgraded the host OS for our Zabbix Server from Ubuntu/Focal to Ubuntu/Jammy. This caused all text to go missing from the rendered graphs. The php (uwsgi) logs had the following to say: PHP Warning: imagettfbbox(): Could not find/open font in /usr/share/zabbix/include/graphs.inc.php on line 600 At least that's a pretty clear message. Through a quick php live hack we learned that it tried to open /usr/share/zabbix/assets/fonts/graphfont.ttf. This file was a symlink to /etc/alternatives/zabbix-frontend-font and that was a symlink to /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans. /blog/2023/postfix-no-system-resources-proxy-protocol/ Thu, 02 Mar 2023 00:00:00 +0000 /blog/2023/postfix-no-system-resources-proxy-protocol/ Connecting to Postfix and getting a "421 4.3.2 No system resources"? Maybe you forgot you're using the (HAProxy) Proxy Protocol... If you're trying to connect to your Postfix mail daemon, and it looks like this: $ nc localhost 25 ... wait for 5 seconds ... 421 4.3.2 No system resources Then I bet you're using HAProxy as reverse proxy to your mailserver and you have the following configured: $ postconf | grep ^postscreen_upstream postscreen_upstream_proxy_protocol = haproxy postscreen_upstream_proxy_timeout = 5s To test a direct connection, you'll need to prefix your traffic with the proxy protocol v1 handshake. /blog/2023/oneliner-finding-fixed-kernel-bugs/ Fri, 17 Feb 2023 00:00:00 +0000 /blog/2023/oneliner-finding-fixed-kernel-bugs/ Recently we were bitten by an old kernel bug on Ubuntu that only rarely triggers. Finding out where the problem was is easier if you know where to look. We had no kernel logs to go on. Only a hanging machine with no output. And a hunch that the running kernel version linux-image-5.4.0-122 was the likely culprit. Was there another way than meticulously reading all changelogs and changes to find out which bug we're dealing with? /blog/2023/cephfs-einval-specified-for-ceph-dir-subvolume/ Mon, 06 Feb 2023 00:00:00 +0000 /blog/2023/cephfs-einval-specified-for-ceph-dir-subvolume/ What could be the case when creating a subvolume in CephFS throws an EINVAL VolumeException? A client of ours recently got errors when creating a 24GiB subvolume: ceph fs subvolume create cephfs-filesystem \ example1 --group_name=thegroup \ --size=25769803776 But, instead of silence and a new subvolume, they got this in their face: a big Python backtrace. Error EINVAL: Traceback (most recent call last): File "ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 271, in discover self.fs.stat(self.base_path) File "cephfs. /blog/2023/windows-openvpn-unexpected-default-route/ Fri, 27 Jan 2023 00:00:00 +0000 /blog/2023/windows-openvpn-unexpected-default-route/ The other day, I was looking into a VPN client issue. The user could connect, they would get their routes pushed, but they would then proceed to use the VPN for all traffic instead of just the routes we provided them. We did not push a default route, because this VPN server exposed a small internal network only. Any regular internet surfing should be done directly. So, when I looked at a tcpdump I was baffled when I saw that DNS lookups were attempted through the OpenVPN tunnel: /blog/2023/kubernetes-crl-support-with-the-front-proxy-client-and-haproxy/ Mon, 23 Jan 2023 00:00:00 +0000 /blog/2023/kubernetes-crl-support-with-the-front-proxy-client-and-haproxy/ “Why does kube-apiserver not take a CRL file?” Kubernetes clusters at OSSO are usually setup using PKI infrastructure from which we create client certificates for users as well. Unfortunately, users can sometimes be a little careless [citation needed] and sometimes they manage to share their keys with the world. PKI caters for lost certificates through the issuance of periodic (or ad-hoc) Certificate Revocation Lists (CRLs), in which the PKI admins can place certificates that have been compromised: if a certificate is listed in the CRL, it is revoked, and will not be accepted by the TLS agent. /blog/2023/django-1-8-python-3-10/ Tue, 17 Jan 2023 00:00:00 +0000 /blog/2023/django-1-8-python-3-10/ After upgrading a machine to Ubuntu/Jammy there was an old Django 1.8 project that refused to run with the newer Python 3.10. ... File "django/db/models/sql/query.py", line 11, in <module> from collections import Iterator, Mapping, OrderedDict ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py) This was relatively straight forward to fix, by using the following patch. Some parts were stolen from a stackoverflow response by Elias Prado. --- a/django/core/paginator.py 2023-01-11 14:09:04. /blog/2023/sysctl-modules-load-order-nf-conntrack/ Tue, 10 Jan 2023 00:00:00 +0000 /blog/2023/sysctl-modules-load-order-nf-conntrack/ Recently we ran into an issue where connections were unexpectedly aborted. Connections from a NAT-ed client (a K8S pod) to a server would suddently get an old packet (according to the sequence number) in the middle of the data. This triggered the Linux NAT-box to issue a reset packet (RST). Setting the kernel flag to mitigate this behaviour required some knowledge of module load order during boot. Spurious retransmits causing connection teardown To start off: we observed that traffic from a pod to a server got disconnected. /blog/2022/recap2022-part2/ Fri, 23 Dec 2022 10:30:00 +0100 /blog/2022/recap2022-part2/ 2022 – genereren van plaatjes door de computer Als je een beetje tech-nieuws hebt gevolgd, kan het je niet ontgaan zijn dat er allerhande sprongen in de kunstmatige intelligentie (AI) gebeurd zijn het afgelopen jaar. Machine learning (ML) bestaat natuurlijk al langer. Maar in 2022 ging de bal voor computer generated images echt rollen. En niet alleen images. GPT-3 en ChatGPT laten zien dat computergegeneerde teksten ook al verbluffend goed kunnen zijn. /blog/2022/recap2022-part1/ Tue, 20 Dec 2022 10:13:00 +0100 /blog/2022/recap2022-part1/ 2022 – afgesloten met gezelligheid Bij OSSO hebben we het jaar afgesloten met een lekker kerstdiner. Sfeerlampjes aan 💡 Goed eten 🥩 Familie erbij 👪 Dat gezegd hebbende, beseffen we ons maar al te goed dat de Russische agressie in Oekraïne enorm leed veroorzaakt. We hopen dat de politiek genoeg wil kan tonen om David te blijven steunen in z’n strijd om de indringers weer buiten de landsgrenzen te zetten. /blog/2022/avoiding-255-31-bit-prefixes/ Wed, 19 Oct 2022 00:00:00 +0000 /blog/2022/avoiding-255-31-bit-prefixes/ At OSSO, we've been using a spine-leaf architecture in the datacenter, using BGP and Layer 3 to the host. This means that we can have any IP address of ours just pop up anywhere in our network, simply by adding a prefix on a leaf switch. We sacrifice half of our IP space for this. But we gain simplicity by avoiding all Layer 2 tricks. TL;DR: Avoid IP addresses ending in . /blog/2022/supermicro-x9drw-quest-for-kvm/ Fri, 09 Sep 2022 00:00:00 +0000 /blog/2022/supermicro-x9drw-quest-for-kvm/ I'm connected to an “ancient” Supermicro machine — according to today's standards — that saw the light somewhere around 2013. I'm looking for a way to access the KVM module (Keyboard, Video, Mouse) so I can update it safely. You know, to be able to fix boot issues if they arise. Unfortunately, the firmware is rather old and I cannot get the iKVM application to run, like I'm used to. /blog/2022/chromium-browser-without-ubuntu-snap-linux-mint/ Wed, 31 Aug 2022 00:00:00 +0000 /blog/2022/chromium-browser-without-ubuntu-snap-linux-mint/ In 2019, Clement "Clem" Lefebvre of Linux Mint, wrote these profetic words: “As long as snap is a solution to a problem, it’s great. Just like Flatpak, it can solve some of the real issues we have with frozen package bases. It can provide us with software we couldn’t otherwise run as packages. When it starts replacing packages for no good reason though, when it starts harming our interaction with upstream projects and software vendors and reducing our choice, it becomes a threat. /blog/2022/falco-helm-upgrade-labelselector-field-immutable/ Tue, 30 Aug 2022 00:00:00 +0000 /blog/2022/falco-helm-upgrade-labelselector-field-immutable/ Today I got this unusual error when upgrading the Falco helm chart from 1.19.4 to 2.0+. Error: UPGRADE FAILED: cannot patch "falco" with kind DaemonSet: DaemonSet.apps "falco" is invalid: spec.selector: Invalid value: v1.LabelSelector{ MatchLabels:map[string]string{"app.kubernetes.io/instance":"falco", "app.kubernetes.io/name":"falco"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil) }: field is immutable The explanation is here as given by Stackoverflow user misha2048: You cannot update selectors for [...] ReplicasSets, Deployments, DaemonSets [...] from my-app: ABC to my-app: XYZ and then simply [apply the changes]. /blog/2022/flipper-zero-multi-tool-developing/ Thu, 11 Aug 2022 00:00:00 +0000 /blog/2022/flipper-zero-multi-tool-developing/ Here are some pointers on how to get started editing/developing plugins for the Flipper Zero multi-tool. (When writing this, the stable version was at 0.63.3. Things are moving fast, so some of the next bits may be outdated when you read them.) Starting Starting the Flipper Zero and adding an SD-card is documented in Flipper Zero first-start. Now you can use all the nice pentest features already included. The SD-card is necessary to unlock some features. /blog/2022/ubuntu-jammy-ssh-rsa-keys/ Tue, 12 Jul 2022 00:00:00 +0000 /blog/2022/ubuntu-jammy-ssh-rsa-keys/ With the new Ubuntu/Jammy we also get tighter security settings. Here are some aliases that will let you connect to older ssh servers. For access to old Cisco routers, we already had the first two options in this alias; we now add two more: # Alias on Ubuntu/Jammy with ssh 8.9p1-3+ to access old routers/switches: alias ssholdhw="ssh \ -oKexAlgorithms=+diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1 \ -oCiphers=+aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc \ -oHostkeyAlgorithms=+ssh-rsa \ -oPubkeyAcceptedKeyTypes=+ssh-rsa" That fixes so we can connect to old Cisco and old HP equipment. /blog/2022/thunderbird-opening-links-ubuntu/ Sun, 08 May 2022 00:00:00 +0000 /blog/2022/thunderbird-opening-links-ubuntu/ For some reason, opening links from Thunderbird stopped working. When clicking a URL, I expected Chromium to open the website, but nothing happened. After visiting a few bug reports and the Thunderbird advanced configuration, I turned my attention to xdg-open: $ xdg-open 'https://wjd.nu' ERROR: not connected to the gnome-3-38-2004 content interface. Okay. So it wasn't a Thunderbird problem at all. The culprit was that I had been doing some housekeeping in snap. /blog/2022/dnssec-validation-authoritative-server/ Thu, 24 Mar 2022 00:00:00 +0000 /blog/2022/dnssec-validation-authoritative-server/ The delv(1) tool is the standard way to validate DNSSEC signatures. By default it will validate up to the DNS root zone, for which it knows and trusts the DNSKEY. If you want to validate only a part of a chain, you'll need to know a few things. Regular DNSSEC validation Using delv is normally as simple as this: $ delv -t A @1.1.1.1 dnssec.works. ; fully validated dnssec.works. 3600 IN A 5. /blog/2022/nvme-drive-refusing-efi-boot/ Thu, 10 Mar 2022 00:00:00 +0000 /blog/2022/nvme-drive-refusing-efi-boot/ UEFI is the current boot standard. Instead of fighting it, we've adopted it as the default for all hardware machines we install. We've had some issues in the past, but they could all be attributed to a lack of knowledge by the operator, not by a problem with EFI itself. But, this time we couldn't figure out why the SuperMicro machine refused to boot from these newly installed EFI partitions: no bootable UEFI device found. /blog/2022/fat16-filesystem-layout/ Wed, 09 Mar 2022 00:00:00 +0000 /blog/2022/fat16-filesystem-layout/ First there was FAT, then FAT12, FAT16 and finally FAT32. Inferior filesystems nowadays, but nevertheless both ubiquitous and mandatory for some uses. And sometimes you need to be aware of the differences. A short breakdown of FAT16 follows — we'll skip the older FAT as well as various uncommon settings, because those are not in active use. Sector size The storage device defines (logical) sector sizes. This used to be 512 bytes per sector for a long time (we're skipping pre-hard disk tech), but this is now rapidly moving to 4096 bytes per sector on newer SSD and NVMe drives. /blog/2022/reading-matryoshka-elf-dirtypipez/ Tue, 08 Mar 2022 00:00:00 +0000 /blog/2022/reading-matryoshka-elf-dirtypipez/ While looking at the clever dirtypipez.c exploit, I became curious how this elfcode was constructed. On March 7 2022, Max Kellerman disclosed a vulnerability he found in Linux kernel 5.8 and above called The Dirty Pipe Vulnerability. Peter (blasty) at haxx.in quickly created a SUID binary exploit for it, called dirtypipez.c. This code contains a tiny ELF binary which writes another binary to /tmp/sh — the ELF Matryoshka doll. I was wondering how one parses this code — to ensure it does what it says it does, and just because. /blog/2022/rst-tables-with-htmldjango-emoji-two-columns-wide/ Mon, 28 Feb 2022 00:00:00 +0000 /blog/2022/rst-tables-with-htmldjango-emoji-two-columns-wide/ For a project, we're using Django to generate a textual report. For readability, it is in monospace text. And we've done it in reStructuredText (RST) so we can generate an HTML document from it as well. A table in RST might look like this: +-----------+-------+ | car brand | users | +===========+=======+ | Peugeot | 2 | +-----------+-------+ | Saab | 1 | +-----------+-------+ | Volvo | 4 | +-----------+-------+ Transforming this to HTML with a rst2html(1) generates a table similar to this: /blog/2022/curious-termios-error-switching-to-asyncio-serial/ Sat, 22 Jan 2022 00:00:00 +0000 /blog/2022/curious-termios-error-switching-to-asyncio-serial/ My Python code that interfaces with a serial port stopped working when refactoring the code to use asyncio. It started raising Invalid argument exceptions from tcsetattr(3). Why would asynchronous Python misbehave? Was there a bug in serial_asyncio? TL;DR: When interfacing with an openpty(3) pseudoterminal — which I used to emulate a real serial port — setting parity and bytesize is not supported. But an error would only show up when tcsetattr(3) was called twice, which happened only in the asyncio case. /blog/2021/recap2021/ Fri, 24 Dec 2021 18:10:00 +0100 /blog/2021/recap2021/ 2021 – het jaar waarin alles ingehaald wordt. Zo eindigde de recap van vorig jaar. Iets té optimistisch naar nu blijkt. De pandemie duurt voort en heeft zijn weerslag gehad op allerlei processen. Contact met onze klanten speelde nu noodgedwongen nog vaker digitaal af. Maar de OSSO-workflow kon grotendeels gehandhaafd blijven. Hier volgen wat downs en (gelukkig vooral) ups van het afgelopen jaar. Challenges 😢Onze gewaardeerde collega Edgar besloot werk aan te nemen dat dichter bij huis was. /blog/2021/systemd-zpool-import-zfs-mount-dependencies/ Fri, 19 Nov 2021 00:00:00 +0000 /blog/2021/systemd-zpool-import-zfs-mount-dependencies/ On getting regular ZFS mount points to work with systemd dependency ordering. ZFS on Ubuntu is nice. And so is systemd. But getting them to play nice together sometimes requires a little extra effort. A problem we were facing was that services would get started before their respective mount points had all been made available. For example, for some setups, we have a local-storage ZFS zpool that holds the /var/lib/docker directory. /blog/2021/zpool-import-no-pools-stale-zdb-labels/ Fri, 22 Oct 2021 00:00:00 +0000 /blog/2021/zpool-import-no-pools-stale-zdb-labels/ Today, when trying to import a newly created ZFS pool, we had to supply the -d DEV argument to find the pool. # zpool import no pools available to import But I know it's there. # zpool import local-storage cannot import 'local-storage': no such pool available And by specifying -d with a device search path, it can be found: # zpool import local-storage -d /dev/disk/by-id Success! # zpool list -oname NAME bpool local-storage rpool Manually specifying a search path is not real convenient. /blog/2021/letsencrypt-root-certificate-validation-on-jessie/ Mon, 04 Oct 2021 00:00:00 +0000 /blog/2021/letsencrypt-root-certificate-validation-on-jessie/ On getting LetsEncrypt certificates to work on Debian/Jessie or Cumulus Linux 3 again. Since last Thursday the 30th, the old LetsEncrypt certificate root stopped working at 14:01 UTC. This was a known and anticipated issue. All certificates had long been double signed by a new root that doubled as intermediate. Unfortunately, this does not mean that everything worked on older platforms with OpenSSL 1.0.1 or 1.0.2. See this Debian/Jessie box — we see similar behaviour on Cumulux Linux 3. /blog/2021/umount-l-needs-make-slave/ Sun, 03 Oct 2021 00:00:00 +0000 /blog/2021/umount-l-needs-make-slave/ The other day I learned — the hard way — that umount -l can be dangerous. Using the --make-slave mount option makes it safer. The scenario went like this: A virtual machine on our Proxmox VE cluster wouldn't boot. No biggie, I thought. Just mount the filesystem on the host and do a proper grub-install from a chroot: # fdisk -l /dev/zvol/zl-pve2-ssd1/vm-215-disk-3 /dev/zvol/zl-pve2-ssd1/vm-215-disk-3p1 * 2048 124999679 124997632 59.6G 83 Linux /dev/zvol/zl-pve2-ssd1/vm-215-disk-3p2 124999680 125827071 827392 404M 82 Linux swap / Solaris # mount /dev/zvol/zl-pve2-ssd1/vm-215-disk-3p1 /mnt/root # cd /mnt/root # for x in dev proc sys; do mount --rbind /$x $x; done # chroot /mnt/root There I could run the necessary commands to fix the boot procedure. /blog/2021/a-singal-17-is-raised/ Tue, 28 Sep 2021 00:00:00 +0000 /blog/2021/a-singal-17-is-raised/ When running the iKVM software on the BMC of SuperMicro machines, we regularly see an interesting "singal" typo. (For the interested, we use a helper script to access the KVM console: ipmikvm. Without it, you need Java support enabled in your browser, and that has always given us trouble. The ipmikvm script logs on to the web interface, downloads the required Java bytecode and runs it locally.) Connect to somewhere, wait for the KVM console to open, close it, and you might see something like this: /blog/2021/mariabackup-selective-table-restore/ Sun, 12 Sep 2021 00:00:00 +0000 /blog/2021/mariabackup-selective-table-restore/ When using mariabackup (xtrabackup/innobackupex) for your MySQL/MariaDB backups, you get a snapshot of the mysql lib dir. This is faster than doing an old-style mysqldump, but it is slightly more complicated to restore. Especially if you just want access to data from a single table. Assume you have a big database, and you're backing it up like this, using the mariadb-backup package: # ulimit -n 16384 # mariabackup \ --defaults-file=/etc/mysql/debian.cnf \ --backup \ --compress --compress-threads=2 \ --target-dir=/var/backups/mysql \ [--parallel=8] [--galera-info] . /blog/2021/apt-downgrading-back-to-current-release/ Tue, 31 Aug 2021 00:00:00 +0000 /blog/2021/apt-downgrading-back-to-current-release/ If you're running an older Debian or Ubuntu, you may sometimes want to check out a newer version of a package, to see if a particular bug has been fixed. I know, this is not supported, but this scheme Generally Works (*): replace the current release name in /etc/apt/sources.list, with the next release — e.g. from bionic to focal do an apt-get update and an apt-get install SOME-PACKAGE You can test the package while replacing the sources. /blog/2021/k8s-lightweight-redirect/ Sun, 15 Aug 2021 00:00:00 +0000 /blog/2021/k8s-lightweight-redirect/ Spinning up pods just to for parked/redirect sites? I think not. Recently, I had to HTTP(S)-redirect a handful of hostnames to elsewhere. Pointing them into our well maintained K8S cluster was the easy thing to do. It would manage LetsEncrypt certificates automatically using cert-manager.io. From the cluster, I could spin up a service and an nginx deployment with a bunch of redirect/302 rules. However, spinning up one or more nginx instances just to have it do simple redirects sounds like overkill. /blog/2021/traverse-path-permissions-namei/ Thu, 12 Aug 2021 00:00:00 +0000 /blog/2021/traverse-path-permissions-namei/ How does one traverse a long path to quickly find out where you lack permissions? So, I wanted to test some stuff in Debian/Buster. I already had an LXC container through LXD. I just needed to get some source files to the right place. lxd$ sudo zfs list | grep buster data/containers/buster-builder 692M 117G 862M /var/snap/lxd/common/lxd/storage-pools/data/containers/buster-builder lxd$ sudo zfs mount data/containers/buster-builder Make sure there's somewhere where I can write: lxd$ sudo mkdir \ /var/snap/lxd/common/lxd/storage-pools/data/containers/buster-builder/rootfs/home/osso/walter lxd$ sudo chown walter \ /var/snap/lxd/common/lxd/storage-pools/data/containers/buster-builder/rootfs/home/osso/walter Awesome. /blog/2021/migrating-vm-interfaces-eth0-to-ens18/ Fri, 06 Aug 2021 00:00:00 +0000 /blog/2021/migrating-vm-interfaces-eth0-to-ens18/ How about finally getting rid of eth0 and eth1 in those ancient Ubuntu VMs that you keep upgrading? Debian and Ubuntu have been doing a good job at keeping the old names during upgrades. But it's time to move past that. We expect ens18 and ens19 now. There's no need to hang on to the past. (And you have moved on to Netplan already, yes?) Steps: rm /etc/udev/rules.d/80-net-setup-link.rules update-initramfs -u rm /etc/systemd/network/50-virtio-kernel-names. /blog/2021/kioxia-nvme-num-err-log-entries-0xc004-smartctl/ Thu, 05 Aug 2021 00:00:00 +0000 /blog/2021/kioxia-nvme-num-err-log-entries-0xc004-smartctl/ So, these new Kioxia NVMe drives were incrementing the num_err_log_entries as soon as they were inserted into the machine. But the error said INVALID_FIELD. What gives? In contrast to the other (mostly Intel) drives, these drives started incrementing the num_err_log_entries as soon as they were plugged in: # nvme smart-log /dev/nvme21n1 Smart Log for NVME device:nvme21n1 namespace-id:ffffffff ... num_err_log_entries : 932 The relevant errors should be readable in the error-log. All 64 errors in the log looked the same: /blog/2021/openssl-error-42-certificate-not-yet-valid/ Fri, 18 Jun 2021 00:00:00 +0000 /blog/2021/openssl-error-42-certificate-not-yet-valid/ In yesterday's post about not being able to connect to the SuperMicro iKVM IPMI, I wondered “why stunnel/openssl did not send error 45 (certificate_expired) for a not-yet-valid certificate.” Here's a closer examination. Quick recap: yesterday, I got SSL alert/error 42 as response to a client certificate that was not yet valid. The server was living in 2015 and refused to accept a client certificate that would be valid first in 2016. /blog/2021/supermicro-ikvm-sslv3-alert-bad-certificate/ Thu, 17 Jun 2021 00:00:00 +0000 /blog/2021/supermicro-ikvm-sslv3-alert-bad-certificate/ Today I was asked to look at a machine that disallowed iKVM IPMI console access. It allowed access through the “iKVM/HTML5”, but when connecting using the “Console Redirection” (Java client, see also ipmikvm) it would quit after 10 failed attempts. TL;DR: The clock of the machine had been reset to a timestamp earlier than the first validity of the supplied client certificate. After changing the BMC time from 2015 to 2021, everything worked fine again. /blog/2021/partially-removed-pve-node-proxmox-cluster/ Thu, 20 May 2021 00:00:00 +0000 /blog/2021/partially-removed-pve-node-proxmox-cluster/ The case of the stale (removed but not removed) PVE node in our Proxmox cluster. On one of our virtual machine clusters, a node — pve3 — had been removed on purpose, yet is was still visible in the GUI with a big red cross (because it was unavailable). This was not only ugly, but also caused problems for the node enumeration done by proxmove. The node had been properly removed, according to the removing a cluster node documentation. /blog/2021/enable-noisy-build-opensips/ Tue, 11 May 2021 00:00:00 +0000 /blog/2021/enable-noisy-build-opensips/ How do you enable the noisy build when building OpenSIPS? The one where the actual gcc invocations are not hidden. In various projects the compilation and linking steps called by make are cleaned up, so you only see things like: Compiling db/db_query.c Compiling db/db_id.c ... This looks cleaner. But sometimes you want to see (or temporarily change) the compilation/linking call: gcc -g -O9 -funroll-loops -Wcast-align -Wall [...] -c db/db_query.c -o db/db_query. /blog/2021/missing-serial-scsi-disk-by-id/ Wed, 05 May 2021 00:00:00 +0000 /blog/2021/missing-serial-scsi-disk-by-id/ When you have a lot of storage devices, it's best practice to assign them to raid arrays or ZFS pools by something identifiable. And preferably something that's also readable when outside a computer. Commonly: the disk manufacturer and the serial number. Usually, both the disk manufacturer and the disk serial number are printed on a small label on the disk. So, if you're in the data center replacing a disk, one glance is sufficient to know you got the correct disk. /blog/2021/smtp-domain-gitlab-configuration/ Thu, 29 Apr 2021 00:00:00 +0000 /blog/2021/smtp-domain-gitlab-configuration/ What is the smtp_domain in the GitLab configuration? There is also a smtp_address and smtp_user_name; so what would you put in the “domain” field? Contrary to what the examples on GitLab Omnibus SMTP lead you to believe: smtp_domain is the HELO/EHLO domain; i.e. your hostname. RFC 5321 has this to say about the HELO/EHLO parameter: o The domain name given in the EHLO command MUST be either a primary host name (a domain name that resolves to an address RR) or, if the host has no name, an address literal, as described in Section 4. /blog/2021/yubico-otp-pam-openvpn/ Mon, 26 Apr 2021 00:00:00 +0000 /blog/2021/yubico-otp-pam-openvpn/ Quick notes on setting up pam_yubico.so with OpenVPN. Add to OpenVPN server config: plugin /usr/lib/x86_64-linux-gnu/openvpn/plugins/openvpn-plugin-auth-pam.so openvpn # Use a generated token instead of user/password for up # to 16 hours, so you'll need to re-enter your otp daily. auth-gen-token 57600 Sign up at https://upgrade.yubico.com/getapikey/. It's really quick. Store client_id and secret (or id and key respectively). You'll need them in the config below. Get PAM module: # apt-get install --no-install-recommends libpam-yubico Create /etc/pam. /blog/2021/proxmox-virtio-blk-disk-by-id/ Thu, 08 Apr 2021 00:00:00 +0000 /blog/2021/proxmox-virtio-blk-disk-by-id/ Why does the virtio-blk /dev/vda block device not show up in /dev/disk/by-id? Yesterday, I wrote about how Proxmox VE attaches scsi0 and virtio0 block devices differently. That is the starting point for todays question: how come do I get /dev/sda in /dev/disk/by-id while /dev/vda is nowhere to be found? This question is relevant if you're used to referencing disks through /dev/disk/by-id (for example when setting up ZFS, using the device identifiers). /blog/2021/proxmox-alter-default-create-vm-parameters/ Wed, 07 Apr 2021 00:00:00 +0000 /blog/2021/proxmox-alter-default-create-vm-parameters/ The Proxmox Virtual Environment has defaults when creating a new VM, but it has no option to change those defaults. Here's a quick example of hacking in some defaults. Why? (Changing SCSI controller does not change existing disks) In the next post I wanted to talk about /dev/disk/by-id and why disks that use the VirtIO SCSI controller do not show up there. A confusing matter in this situation was that creating a VM disk using a different SCSI controller and then switching does not change the storage driver for the existing disks completely! /blog/2021/openvpn-hardened-fox-it-openvpn-nl/ Thu, 21 Jan 2021 00:00:00 +0000 /blog/2021/openvpn-hardened-fox-it-openvpn-nl/ Today, we will be evaluating OpenVPN-NL — “[a] hardened version of OpenVPN that includes as many of the security measures required to operate in a classified environment as possible — and whether we can use it as a drop-in replacement for regular OpenVPN. While OpenVPN allows many insecure configurations, such as turning off encryption, or the use of outdated cryptographic functions in security critical places, the goal of OpenVPN-NL — a fork created and maintained by Fox-IT — is to strip insecure configuration and verify that the distributed version is uncompromised. /blog/2021/postgresql-inside-kubernetes-no-space-left-on-device/ Fri, 15 Jan 2021 00:00:00 +0000 /blog/2021/postgresql-inside-kubernetes-no-space-left-on-device/ Running PostgreSQL inside Kubernetes? Getting occasional "No space left on device" errors? Know that 64MB is not enough for everyone. With the advent of more services running inside Kubernetes, we're now running into new issues and complexities specific to the containerization. For instance, to solve the problem of regular file backups of distributed filesystems, we've resorted to using rsync wrapped inside a pod (or sidecar). And now for containerized PostgreSQL, we're running into an artificial memory limit that needs fixing. /blog/2021/chromium-snap-wrong-fonts/ Tue, 05 Jan 2021 00:00:00 +0000 /blog/2021/chromium-snap-wrong-fonts/ So, since a couple of weeks my snap-installed Chromium browser on Ubuntu Focal started acting up: suddenly it chooses the wrong fonts on some web pages. The chosen fonts are from the ~/.local/share/fonts/ directory. Look! That's not the correct font. And it's even more apparent that the font is off when seeing the source view. Bah. That's not even a monospaced font. A fix that appeared to work — but unfortunately only temporarily — involves temporarily moving the custom local fonts out of the way and then flushing the font cache: /blog/2021/stale-apparmor-config-mysql-refuses-to-start/ Sat, 02 Jan 2021 00:00:00 +0000 /blog/2021/stale-apparmor-config-mysql-refuses-to-start/ So, recently we had an issue with a MariaDB server that refused to start. Or, actually, it would start, but before long, SystemD would kill it. But why? # systemctl start mariadb.service Job for mariadb.service failed because a timeout was exceeded. See "systemctl status mariadb.service" and "journalctl -xe" for details. After 90 seconds, it would be killed. systemctl status mariadb.service shows the immediate cause: # systemctl status mariadb.service ... systemd[1]: mariadb. /blog/2021/zfs-zvol-partition-does-not-show-up/ Fri, 01 Jan 2021 00:00:00 +0000 /blog/2021/zfs-zvol-partition-does-not-show-up/ On our Proxmox virtual machine I had to go into a volume to quickly fix an IP address. The volume exists on the VM host, so surely mounting is easy. Right? I checked in /dev/zvol/pve2-pool/ where I found the disk: # ls /dev/zvol/pve2-pool/vm-125-virtio0* total 0 lrwxrwxrwx 1 root root 10 Dec 29 15:55 vm-125-virtio0 -> ../../zd48 Good, there's a disk: # fdisk -l /dev/zd48 Disk /dev/zd48: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes Disklabel type: dos Disk identifier: 0x000aec27 Device Boot Start End Sectors Size Id Type /dev/zd48p1 * 2048 97656831 97654784 46. /blog/2020/recap2020/ Wed, 16 Dec 2020 12:50:01 +0100 /blog/2020/recap2020/ (I’m sorry, this time the recap is in Dutch instead of English. If you’re reading this, you’re probably Dutch anyway, so it won’t be a problem.) COVID-jaar 2020 is een vreemd jaar geweest, maar voor OSSO was het ook een goed jaar. Waar 2020 nog in mineur begon, omdat we eind 2019 nog een dierbare collega verloren aan een emigratiewens naar het land van de kangoeroes, hebben we in 2020 twee nieuwe collega’s mogen verwelkomen. /blog/2020/zfs-destroy-dataset-is-busy/ Fri, 11 Dec 2020 00:00:00 +0000 /blog/2020/zfs-destroy-dataset-is-busy/ Just now, I tried to remove a ZFS dataset, and it reported dataset is busy for no apparent reason. # zfs list -r data NAME USED AVAIL REFER MOUNTPOINT data 3.12T 405G 251M /data data/kubernetes-logging 2.08T 405G 2.08T /data/kubernetes/logging data/rook-config 36.5M 405G 36.5M /data/rook-config data/rook-data 1.03T 708G 753G - # zfs destroy data/kubernetes-logging cannot destroy 'data/kubernetes-logging': dataset is busy The usual suspects were checked: The dataset was not mounted (cat /proc/mounts | grep kubernetes). /blog/2020/cumulus-postfix-in-the-right-vrf/ Tue, 27 Oct 2020 00:00:00 +0000 /blog/2020/cumulus-postfix-in-the-right-vrf/ Cumulus Linux is a network operating system. It is a switch, but it also runs Linux OS, allowing us to run our automation tools on it. We use it to automate the configuration of our network. A network where we use VRF (virtual routing and forwarding) to separate customer traffic. The presence of VRFs in the OS however means that we have to tell the daemons in which VRF to run. /blog/2020/offsite-on-the-fly-encrypted-backups-gocryptfs/ Fri, 23 Oct 2020 00:00:00 +0000 /blog/2020/offsite-on-the-fly-encrypted-backups-gocryptfs/ Earlier, I wrote about using encfs to do on-the-fly encrypted backups (using encfs). The idea was that you grant ssh+rsync access to a backup system, but that that system does not know what it is backing up. This provides a layer of security between your backup provider and your private data. That scheme works like this: there is a remote system doing periodic incremental rsync backups, like a PlanB Backup server; you grant ssh+rsync access to that system; but only to a specific path; on that path, you mount an encrypted view of your filesystem — a. /blog/2020/pgp-on-yubikey-refresh-expiry/ Tue, 13 Oct 2020 00:00:00 +0000 /blog/2020/pgp-on-yubikey-refresh-expiry/ Generally, I try to follow security best practices. This means that I have my PGP signing, authentication and encryption keys on my YubiKey, and I have configured the keys to expire after a year. Unfortunately, refreshing the expiry every year is not quite enough to store how to do that into muscle memory. Here are the steps relevant to my use case. Putting the keys on the YubiKey in the first place is worth a post of its own. /blog/2020/tls-testing-certificate-chains-easycert/ Thu, 10 Sep 2020 00:00:00 +0000 /blog/2020/tls-testing-certificate-chains-easycert/ The openssl client is a very versatile tool, but also a bit cryptic. The easycert utility from the ossobv/vcutil scripts makes validating/managing certificates easier. easycert from ossobv/vcutil has a few modes of operation: CLI, CGI, generating certificates and testing certificates. Nowadays we mostly use the testing mode: -T The utility is a convenient wrapper around openssl s_client and x509 calls. Get it from github.com/ossobv/vcutil easycert. Usage Run it like this: /blog/2020/excel-generate-sheet-password-collision/ Wed, 09 Sep 2020 00:00:00 +0000 /blog/2020/excel-generate-sheet-password-collision/ Yesterday, I demonstrated how to brute force the Excel protected sheet/cells password. (Write protection! Not read protection a.k.a. encryption!) Today, I figured there must be a faster way, as the hash is not at all complicated. After fiddling around a little, I hacked together this bit of Python: def reverse_f(wanted): "Calculate Excel protected sheet password" # https://wjd.nu/notes/2020#excel-generate-sheet-password-collision # https://wjd.nu/notes/2020#libreoffice-asking-for-cell-password-brute-force def reverse_rotate(v): "Right shift by one, rotating the right most bit to bit 15" if v & 0x1: return (v >> 1) | 0x4000 return v >> 1 chars = [] valid_tokens = tuple([ord(i) for i in ( 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' 'abcdefghijklmnopqrstuvwxyz' '0123456789')]) # Length 9 should be enough to go down from 16 to 7 bits: # we skip some shorter solutions, but there are only a few hashes # that benefit from that. /blog/2020/libreoffice-asking-for-cell-password-brute-force/ Tue, 08 Sep 2020 00:00:00 +0000 /blog/2020/libreoffice-asking-for-cell-password-brute-force/ While we were editing a provider-supplied Excel document using LibreOffice, at seemingly random times, it would show a popup asking us for a password to a cell. This popup would only go away if we set a new (non-blank) password on it. Annoying! Apparently, it has to do with Sheet and Cell protection whereby an editing user is disallowed to edit certain cells/rows/sheets in a document. Having certain cells marked read-only, sure. /blog/2020/docker-unprivileged-user-becoming-root/ Fri, 03 Jul 2020 00:00:00 +0000 /blog/2020/docker-unprivileged-user-becoming-root/ My colleague was rightly annoyed that our USER www-data docker images greatly hindered effective debugging. Can we become root again, while still keeping the additional secure-by-default non-root images? If we have enough permissions on the filesystem, then: yes, we can. Take the following example, where we’ll be looking at a myproject pod. (You can skip the Kubernetes steps if you already know where the Docker instance resides.) $ kubectl get pods -o wide myproject-66dd6b4dd-jskgf NAME READY STATUS AGE IP NODE myproject-66dd6b4dd-jskgf 1/1 Running 64d 10. /blog/2020/nss-dns4only-libc-disable-aaaa-lookups/ Mon, 25 May 2020 00:00:00 +0000 /blog/2020/nss-dns4only-libc-disable-aaaa-lookups/ Have you ever noticed how some applications can do AAAA DNS record lookups even though the host has no IPv6 connectivity? That means double DNS lookups for zero profit. Why is that? And how can you disable it? Problem To make a long story short, a common combination of circumstances can cause useless gratuitous AAAA lookups: Applications that are IPv6 ready (or applications that don't care); on hosts using libc; where IPv6 is enabled (net. /blog/2020/gitlab-securing-public-repositories/ Sun, 24 May 2020 00:00:00 +0000 /blog/2020/gitlab-securing-public-repositories/ In the past, GitLab repositories were created with Public Visibility by default. Now they have a more sensible security setting. Still, it can be nice to assert that public repositories are not Public-by-Accident. How? Well, one fix is to check that Public repositories are in a whitelisted public namespace (e.g. /public/). That way it’s immediately obvious that the repositories herein are visible to everyone. Use a Private browser and go to: https://YOUR_GITLAB_INSTANCE/explore/projects /blog/2020/more-or-less-useless-tips-and-tricks-3/ Sat, 23 May 2020 00:00:00 +0000 /blog/2020/more-or-less-useless-tips-and-tricks-3/ More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. gsettings set org.gnome.desktop.calendar show-weekdate true — to enable week numbers in the gnome-shell datetime calendar popup. (You may need to set LC_TIME to en_GB so the week starts on a Monday instead of, American style, on a Sunday. You’ll probably have set LC_PAPER too already, to get A4 paper size printing defaults. /blog/2020/encryption-decryption-speed-gnupg-openssl/ Wed, 13 May 2020 00:00:00 +0000 /blog/2020/encryption-decryption-speed-gnupg-openssl/ We were looking at encryption ingredients the other day. Because, if we want to compare encryption methods, we shouldn’t compare apples and oranges. With that newfound knowledge, we can run a few speed tests. The aggregated data (raw data sources can be found below): EncryptionDecryption user (ms)sys (ms)total (ms)mem (10K) user (ms)sys (ms)total (ms)mem (10K) gpg 1.4 76453547999354 1573832216060359 gpg 2.2 28633403203507 62123406552515 gpg 2.3* 27522813033527 44943184812536 gpg 2.3* nohw 45082814789 52575473187865537 openssl 17543392093 506421391812506 openssl nohw 35643443908504 27653963161507 customcrypt 333842137592458 356543540002461 First a few notes about the graph: /blog/2020/encryption-vocabulary-long-term-storage/ Fri, 01 May 2020 00:00:00 +0000 /blog/2020/encryption-vocabulary-long-term-storage/ While investigating the most appropriate encryption cipher and format, I realised I didn’t have enough vocabulary on the subject. This post aims to close that knowledge gap somewhat. I’m looking at symmetric ciphers here, as they are used when storing lots of data. (In fact, when encrypting larger amounts of data, public/private key encryption (an asymmetric cipher) is only used to encrypt a separate key for the symmetric cipher, which is then used for bulk of the data. /blog/2020/saltstack-printf-io-error-debianutils-which/ Wed, 29 Apr 2020 00:00:00 +0000 /blog/2020/saltstack-printf-io-error-debianutils-which/ After some upgrades, I suddenly noticed unexpected sh: printf: I/O error output. Some debugging later, it turns out that it's the Dash way of informing us of a PIPE error. Apparently salt's cmd.run can cause so little output buffering, that the debianutils which command can be aborted mid-output. Not-broken example: $ which python3 python false | head -n1 /usr/bin/python3 Broken example, through a salt cmd.run call: $ salt 'example.com' cmd.run 'which python3 python false | head -n1' example. /blog/2020/supermicro-java-console-redirection-kvm/ Wed, 12 Feb 2020 00:00:00 +0000 /blog/2020/supermicro-java-console-redirection-kvm/ Connecting to the new SuperMicro iKVM management interfaces requires a working Java in your browser (IcedTea browser plugin). It will launch a Java Web Start (javaws) application. And java plugins (and needless web forms) are a pain in the behind. Is there a better way? Obviously, running from the web interface just means downloading a Java application, and running it locally. So why can’t we do that directly, and skip the IcedTea Java browser plugins? /blog/2020/openssl-sign-subject-alternative-names/ Fri, 31 Jan 2020 00:00:00 +0000 /blog/2020/openssl-sign-subject-alternative-names/ Recently, an automated job of mine failed because the latest and greatest Python refused to validate an SSL certificate with 127.0.0.1 in the common name (CN). Apparently CN=127.0.0.1 will not be accepted anymore, as using the common name for hostname validation has been deprecated for ages now. The fix? Use subject alternative names (SANs). Generally, you’ll already have these when you have your certificates signed by somebody else. But if you’re signing certificates yourself, you’ll need to know how to pass them to openssl: /blog/2020/3cx-voip-letsencrypt-tls/ Thu, 30 Jan 2020 00:00:00 +0000 /blog/2020/3cx-voip-letsencrypt-tls/ Can you get your 3CX Phone System to connect to your SIP provider Trunk over TLS, when the server uses a Let’s Encrypt certificate? 3CX documentation on this topic is scarce. There are posts like 3CX Forum: SIP TLS: It’s fairly straightforward: Your provider must give you a TLS Root Certificate (.pem) so you can encrypt the traffic. If they have SRV records the system will automatically know where to connect for TLS mode (Auto Discovery option on General trunk tab), or the provider might just tell you to change your port to the one they are listening to for TLS connections. /blog/2019/recap2019/ Thu, 19 Dec 2019 13:28:57 +0000 /blog/2019/recap2019/ (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2019 highlights We are looking for new colleagues! The new website is finally up! Built a Cumulus network with automated deployment using Ansible and Netbox as the source of truth. Managed kubernetes has become a staple. Extended our ISO27001:2017 and NEN7510:2017 certifications. Visited Legoland as a team building exercise. Many improvements! Ronald may actually leave for australia, like for real, not kidding, probably, if they finally let him in, in the not too distant future. /blog/2019/pgloader-import-mysql-to-postgresql/ Fri, 20 Sep 2019 00:00:00 +0000 /blog/2019/pgloader-import-mysql-to-postgresql/ When loading old (Django) projects in K8S, we've decided to give PostgreSQL a go as default database. Here are some notes that aid in importing. After looking around, pgloader seems to be the right tool for the job. Feature and stability wise it beats any other solution. And it's available on recent Debian/Ubuntu. We need access to the remote PostgreSQL db; because pgloader will not provide an SQL dump (for reasons). /blog/2019/asterisk-pbx-generated-doxygen-docs/ Wed, 04 Sep 2019 00:00:00 +0000 /blog/2019/asterisk-pbx-generated-doxygen-docs/ On the Asterisk PBX Development page you'll find a reference to Doxygen generated documentation: Most of the documentation related to the source code is embedded in the source files and is processed with Doxygen. The latest version of the Doxygen generated source documentation can be found on http://doxygen.asterisk.org. However, it cannot be found there, as that URL has been returning a 503 Service Unavailable for quite some time now. /blog/2018/recap2018/ Mon, 10 Dec 2018 10:08:57 +0000 /blog/2018/recap2018/ (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ ✧ﾟ･: *ヽ(◕ヮ◕ヽ) 2018 highlights Edgar joined our team Managed kubernetes moving forward Many awesome upstream kubernetes releases Keeping up integrating improvements in our default kubernetes “distribution” Further expanding our managed kubernetes install base Updated NEN7510:2011 to NEN7510:2017 ISO27001 ISMS improvements and further integrating customer scope New website coming up - bijna klaar TM. /blog/2018/gbp-buildpackage-gpg2/ Thu, 08 Nov 2018 00:00:00 +0000 /blog/2018/gbp-buildpackage-gpg2/ If you prefer gpg2 over gpg, building a debian package with debuild or gbp buildpackage may require some fiddling. In my case, I’m using gpg2 instead of gpg for signing because unlike version 1, version 2 does PGP key forwarding. That way I can sign on a remote machine, using a local PGP key card. However gbp buildpackage, dpkg-buildpackage and debuild are hardwired to call gpg. And — it turns out — using a simple /usr/local/bin/gpg to /usr/bin/gpg2 symlink was not sufficient to convince gbp (and debuild) to use the gpg2 binary, while for dpkg-buildpackage that is sufficient. /blog/2018/kubectl-broken-terminal-ipython/ Thu, 20 Sep 2018 00:00:00 +0000 /blog/2018/kubectl-broken-terminal-ipython/ Just now I ran into an IPython interpreter inside a Docker container inside Kubernetes misbehaving: After starting ipython inside a kubectl for a second time, IPython wouldn’t show the input prompt. It only showed the output prompt. Turns out it was due to the terminal settings. For some reasons, after logging out of kubectl exec, the next exec would get 0 rows and 0 columns; as if someone had run stty rows 0 on the terminal. /blog/2018/vimrc-debian-stretch/ Wed, 19 Sep 2018 00:00:00 +0000 /blog/2018/vimrc-debian-stretch/ In Debian/Stretch, the default ViM settings have been changed — for the worse, in my opinion. However, undoing the bad settings is not a matter of fixing them in your ~/.vimrc, because when that file is detected no defaults at all are set. The quick fix is to create a custom /etc/vim/vimrc.local file with the following settings: " Instead of auto-sourcing this afterwards, source it now. source $VIMRUNTIME/defaults.vim let g:skip_defaults_vim = 1 " Now we undo the "wrong" settings. /blog/2018/core-file-docker-image-auplink/ Tue, 18 Sep 2018 00:00:00 +0000 /blog/2018/core-file-docker-image-auplink/ A while, I’ve been looking at a stray /core file in some of our daily Xenial Docker images. Time to find out where it comes from. Tracing with a few well placed RUN ls -l /core || true, tells us that the dump appeared after a large RUN statement and not during one. Running gdb on the core revealed that it was a dump of auplink, a part of Docker. Opening the core on a Xenial machine with docker installed, showed the following backtrace: /blog/2018/ubuntu-bionic-crashing-gdm-eglgetdisplay/ Wed, 11 Apr 2018 00:00:00 +0000 /blog/2018/ubuntu-bionic-crashing-gdm-eglgetdisplay/ After upgrading from Ubuntu 17.10 to Ubuntu 18.04, and rebooting, the GNOME Display Manager (gdm) went into a restart loop. No promised speed gains. Instead, I got an unusable desktop. Being quick with CTRL+ALT+F3, I could enter my username and password in the text console after a couple attempts — the gdm restart would continuously steal console/tty focus — after which a sudo systemctl stop gdm was possible. This left me with a shell and plenty of time to examine the situation. /blog/2018/checking-client-ssl-certificate-from-python/ Mon, 26 Mar 2018 00:00:00 +0000 /blog/2018/checking-client-ssl-certificate-from-python/ A quick howto on checking SSL/TLS client certificates from Django/Python. Generally, when you want to use client certificates, you’ll let the HTTPS server (e.g. NGINX) do the certificate validation. For NGINX you’d add this config, and be done with it. # TLS server certificate config: ... # TLS client certificate config: ssl_verify_client on; # or 'optional' ssl_client_certificate /PATH/TO/my-ca.crt; ... location ... { ... # $ssl_client_s_dn contains: "/C=.../O=.../CN=...", where you're # generally interested in the CN-part (commonName) to identify the # "who". /blog/2018/docker-application-placement-paths/ Wed, 14 Feb 2018 00:00:00 +0000 /blog/2018/docker-application-placement-paths/ Where do you place the application inside the Docker image? In the past, when deploying Python/Django projects, we’d put them in /srv/django-projects/APPNAME on a (possibly shared) machine. The python-virtualenv that came with it, went into /srv/virtualenvs/APPNAME. Now that we’re dockerizing many projects, we don’t need the virtualenv (there is only one environment) and we don’t need the APPNAME either (there is only one application). So, where should we place the project? /blog/2018/ubuntu-goodbye-unity-welcome-gnome-shell/ Thu, 01 Feb 2018 00:00:00 +0000 /blog/2018/ubuntu-goodbye-unity-welcome-gnome-shell/ After having gotten used to Unity on the Ubuntu desktop, with Ubuntu Artful it is time to say goodbye. When Ubuntu first added the Unity shell with just the sidebar with big buttons, in favor of the more traditional GNOME with its Windows 95 style interface, many were skeptical, me included. But removing the clutter was good, and I've happily worked with it for years. And you really don't want to waste time tweaking your desktop away from the OS provided defaults. /blog/2018/screen-wipes-copy-buffer/ Wed, 10 Jan 2018 00:00:00 +0000 /blog/2018/screen-wipes-copy-buffer/ A mismash of bugs and workarounds causes the copy buffer (X selection) to get wiped some of the time in my recent desktop environment. And that in a seemingly unpredictable manner. The following bug is mostly in play: GNOME VTE soft reset wipes selection That bug causes: reset(1) to wipe the middle-mouse (primary) buffer (although this differs per system — could not put my finger on this); reset(1) to wipe the clipboard buffer, but only if the reset was called from window that originated the current clipboard buffer contents; GNU screen(1) initialization to misbehave as reset does, as described above — even through an ssh session — by wiping the buffer, if TERM=xterm-256color. /blog/2018/dovecot-roundcube-mail-read-error/ Mon, 08 Jan 2018 00:00:00 +0000 /blog/2018/dovecot-roundcube-mail-read-error/ Today we ran into a dovecot/imap crash on a Xenial box. The Dovecot in question was the patched dovecot-2.2.22. Due to an as of yet unexplained cause, reading mail through Thunderbird mail client worked fine, but when opening a message with Roundcube (webmail), most messages would give an odd error about a “message that could not be opened”. An IMAP trace of Roundcube revealed that the IMAP server stopped responding after the client A0004 UID FETCH command. /blog/2018/meltdown-spectre-attacks/ Thu, 04 Jan 2018 11:06:33 +0000 /blog/2018/meltdown-spectre-attacks/ Information regarding Meltdown and Spectre attacks. Current state Waiting for software patch availability. Patched ubuntu kernels are available for testing. Updates: 20180104: Created blogpost 20180105: Added new information/links 20180105: Status update 20180108: Added information from Redhat about performance impact from patches. 20180108: Updated links list. 20180108: Status update Links https://spectreattack.com / https://meltdownattack.com (same site) https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-every-modern-processor-has-unfixable-security-flaws/ https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-heres-what-intel-apple-microsoft-others-are-doing-about-it/ https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/ High level description CVE’s Spectre - CVE-2017-5715 Spectre - CVE-2017-5753 Meltdown - CVE-2017-5754 As described on: https://spectreattack. /blog/2017/recap-2017/ Thu, 21 Dec 2017 15:25:12 +0000 /blog/2017/recap-2017/ 2017 ISO27001 certified + NEN7510 Ewout joined our team as an SRE More projects opensourced at github: https://github.com/ossobv Uradesign designed logo’s for several of our open source projects Started providing Kubernetes as a Service / Managed Kubernetes Lots of interesting stuff /hosting/ Tue, 12 Dec 2017 11:22:22 +0100 /hosting/ /kubernetes/ Tue, 12 Dec 2017 11:22:22 +0100 /kubernetes/ OSSO provides managed kubernetes clusters including full operational support. Compared to public cloud providers You should run your workload where it makes sense. Be it from a cost perspective, technical criteria, security requirements, or level of available support. At OSSO, we focus on our strong points and where we can provide the most value. Where we differ from the public cloud providers: High reliability OSSO manages the full infrastructure stack. It provides us with full insight and control. /open-source/ Tue, 12 Dec 2017 11:22:22 +0100 /open-source/ /operations/ Tue, 12 Dec 2017 11:10:20 +0100 /operations/ /about/ Tue, 12 Dec 2017 11:07:30 +0100 /about/ /blog/2017/flake8-vim-python2-python3/ Sun, 03 Dec 2017 00:00:00 +0000 /blog/2017/flake8-vim-python2-python3/ In 2015, I wrote a quick recipe to use Vim F7-key flake8 checking for both python2 and python3 using the nvie/vim-flake8 Vim plugin. Here's a quick update that works today. Tested on Ubuntu/Zesty. $ sudo apt-get install python3-flake8 # py3 version, no cli $ sudo pip -H install python-flake8 # py2 version, with cli $ sudo cp /usr/local/bin/flake8{,.2} # copy to flake8.2 $ sudo sed -i -e 's/python$/python3/' /usr/local/bin/flake8 # update shebang $ sudo mv /usr/local/bin/flake8{,. /blog/2017/availability-during-holiday-december-2017/ Fri, 01 Dec 2017 08:30:37 +0000 /blog/2017/availability-during-holiday-december-2017/ Starting the 16th of December we are on leave. We return to the office on the 2nd of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability. /blog/2017/reprepro-multiversion-build-recipe/ Thu, 30 Nov 2017 00:00:00 +0000 /blog/2017/reprepro-multiversion-build-recipe/ We used to use reprepro (4.17) to manage our package repository. However, it did not support serving multiple versions of the same package. The Benjamin Drung version from GitHub/profitbricks/reprepro does. Here’s our recipe to build it. $ git clone -b 5.1.1-multiple-versions https://github.com/profitbricks/reprepro.git $ cd reprepro It lacks a couple of tags, so we’ll add some lightweight ones. $ git tag 4.17.1 2d93fa35dd917077e9248c7e564648da3a5f1fe3 && git tag 4.17.1-1 0c9f0f44a84f67ee5f14bccf6507540d4f7f8e39 && git tag 5. /blog/2017/maintenance-network-mediacentrale-nov-1st-2017-2200/ Wed, 01 Nov 2017 17:01:59 +0000 /blog/2017/maintenance-network-mediacentrale-nov-1st-2017-2200/ Maintenance network Mediacentrale On November 1st 2017 after 22:00 we will upgrade our network in the Mediacentrale. Due to roadworks around Julianaplein in Groningen that will impact our current connections, we will move our network traffic to an upgraded router and fiber path, thus minimizing downtime related to these roadworks. Furthermore, this maintenance also results in an upgrade in our capacity to the Mediacentrale, as we will upgrade from a 1G to 10G infrastructure. /blog/2017/linux-process-uptime-exact/ Mon, 11 Sep 2017 00:00:00 +0000 /blog/2017/linux-process-uptime-exact/ How to get (semi)exact uptime values for processes? If you look at the ps faux listing, you’ll see a bunch of values: walter 27311 0.8 1.8 5904852 621728 ? SLl sep06 61:05 \_ /usr/lib/chromium-browser/... walter 27314 0.0 0.2 815508 80852 ? S sep06 0:00 | \_ /usr/lib/chromium-brow... walter 27316 0.0 0.0 815508 14132 ? S sep06 0:01 | | \_ /usr/lib/chromium-... That second value (27311) is the PID, the tenth (61:05) how much CPU time has been spent. /blog/2017/sudo-cron-silence-logging-authlog/ Wed, 30 Aug 2017 00:00:00 +0000 /blog/2017/sudo-cron-silence-logging-authlog/ Do you use sudo for automated tasks? For instance to let the Zabbix agent access privileged information? Then your auth.log may look a bit flooded, like this: Aug 30 10:51:44 sudo: zabbix : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/iptables -S INPUT Aug 30 10:51:44 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 30 10:51:44 sudo: pam_unix(sudo:session): session closed for user root Or, if you run periodic jobs by root from cron, you get this: /blog/2017/powerdns-pdnsutil-remove-record/ Wed, 23 Aug 2017 00:00:00 +0000 /blog/2017/powerdns-pdnsutil-remove-record/ The PowerDNS nameserver pdnsutil utility has an add-record, but no remove-record. How can we remove records programmatically for many domains at once? Step one: make sure we can list all domains. For our PowerDNS 4 setup, we could do the following: $ list_all() { ( for type in master native; do pdnsutil list-all-zones $type; done ) | grep -vE '^.$|:' | sort -V; } $ list_all domain1.tld domain2.tld ... Step two: filter the domains where we want to remove anything. /blog/2017/gdb-debugging-asterisk-ao2-containers/ Fri, 23 Jun 2017 00:00:00 +0000 /blog/2017/gdb-debugging-asterisk-ao2-containers/ One of our Asterisk telephony machines appeared to “leak” queue member agents. That is, refuse to ring them because they were supposedly busy. When trying to find the cause, there weren’t any data dumping functions for the container I wanted to inspect in the CLI. In this case the pending_members which is of type struct ao2_container. So, we had to resort to using gdb to inspect the data. The struct ao2_container container data type itself looks like this: /blog/2017/letsencrypt-expiry-mails-unsubscribe/ Sat, 03 Jun 2017 00:00:00 +0000 /blog/2017/letsencrypt-expiry-mails-unsubscribe/ Today I got one of these Letsencrypt Expiry mails again. It looks like this: Your certificate (or certificates) for the names listed below will expire in 19 days (on 21 Jun 17 19:38 +0000). Please make sure to renew your certificate before then, or visitors to your website will encounter errors. [domain here] ... If you want to stop receiving all email from this address, click [link here] (Warning: this is a one-click action that cannot be undone) I don’t need this particular domain anymore. /blog/2017/puppet-pip-version-facter/ Tue, 30 May 2017 00:00:00 +0000 /blog/2017/puppet-pip-version-facter/ Every once in a while I have to deal with machines provisioned by puppet. I can’t seem to get used to the fact that --test not only tests, but actually does. It displays what it does though output, which is nice. To test without applying, you need the --noop flag. But, today I wanted to bring up the quick fix to this old warning/error: Error: Facter: error while resolving custom fact "pip_version": undefined method `[]' for nil:NilClass The cause of the issue is an old version of pip(1) which has no --version parameter. /blog/2017/ubuntu-zesty-apt-dns-timeout-srv-records/ Thu, 18 May 2017 00:00:00 +0000 /blog/2017/ubuntu-zesty-apt-dns-timeout-srv-records/ Ever since I updated from Ubuntu/Yakkety to Zesty, my apt-get(1) would sit and wait a while before doing actual work: $ sudo apt-get update 0% [Working] Madness. Let’s see what it’s doing… $ sudo strace -f -s 512 apt-get update ... [pid 5603] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 ... [pid 5603] sendto(3, "\1\271\1\0\0\1\0\0\0\0\0\0\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1", 46, MSG_NOSIGNAL, NULL, 0) = 46 [pid 5603] poll([{fd=3, events=POLLIN}], 1, 5000 <unfinished ...> ... [pid 5600] select(8, [5 6 7], [], NULL, {0, 500000}) = 0 (Timeout) . /blog/2017/squashing-old-git-history/ Sat, 13 May 2017 00:00:00 +0000 /blog/2017/squashing-old-git-history/ You may have an internal project that you wish to open source. When starting the project, you didn’t take that into account, so it’s likely to contain references to private data that you do not wish to share. Step one would be to clean things up. If this is a slow process, this can take time, while in the mean time the project gets updates. Now, at one point you’re confident that at commit X1000, the project contains only non-private data. /blog/2017/loadbalancer-maintenance-22-february-2017/ Mon, 13 Feb 2017 11:59:05 +0000 /blog/2017/loadbalancer-maintenance-22-february-2017/ In the night of Wednesday (22nd of Febr. 2017) to Thursday (23rd of Febr. 2017) between 23:45 and 03:00 we will perform maintenance on our loadbalancers. ####Maintenance window 22-02-2017 23:45 - 23-02-2017 03:00 ####Description Loadbalancers will be upgraded to increase throughput and enable new capabilities. ####Impact When maintenance starts, we’ll reroute all traffic to the secondary loadbalancer. Customers that have a multi-site setup should therefore not have any service interruption. /blog/2017/detect-invisible-selection-copy-buffer-chrome/ Thu, 26 Jan 2017 00:00:00 +0000 /blog/2017/detect-invisible-selection-copy-buffer-chrome/ In Look before you paste from a website to terminal the author rightly warns us about carelessly pasting any input from a web page into the terminal. This LookBeforePaste Chrome Extension is a quick attempt at trying to warn the user. Example output when pressing CTRL-C on the malicious code: Heuristics are defined as follows. They could certainly be improved, but it’s a start. function isSuspicious(node) { if (node.nodeType == node. /blog/2016/convert-dehydrated-certbot-letsencrypt-config/ Wed, 21 Dec 2016 00:00:00 +0000 /blog/2016/convert-dehydrated-certbot-letsencrypt-config/ If you find yourself in the situation that you have to reuse your Letsencrypt credentials/account generated by Dehydrated (a bash Letsencrypt interface) with the official Certbot client, like me, you’ll want to convert your config files. In my case, I wanted to change my e-mail address, and the Dehydrated client offered no such command. With Certbot you can do this: $ certbot register --update-registration --account f65c... But you’ll need your credentials in a format that Certbot groks. /blog/2016/mysql-deterministic-reads-sql-data/ Fri, 16 Dec 2016 00:00:00 +0000 /blog/2016/mysql-deterministic-reads-sql-data/ Can I use the MySQL function characteristic DETERMINISTIC in combination with READS SQL DATA and do I want to? TL;DR If the following two groups of statements are the same to you, you want the DETERMINISTIC characteristic on your FUNCTION, even if you have READS SQL DATA. SET @id = (SELECT my_func()); SELECT * FROM my_large_table WHERE id = @id; -- versus SELECT * FROM my_large_table WHERE id = my_func(); (All of this is tested with MySQL 5. /blog/2016/availability-during-holiday-december-2016/ Wed, 14 Dec 2016 11:16:10 +0000 /blog/2016/availability-during-holiday-december-2016/ Starting the 17th of December we are on leave. We return to the office on the 2nd of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability. /blog/2016/patch-a-day-pdns-recursor-broken-edns-lookups/ Thu, 01 Dec 2016 00:00:00 +0000 /blog/2016/patch-a-day-pdns-recursor-broken-edns-lookups/ Last month, our e-mail exchange (Postfix) started having trouble delivering mail to certain destinations. These destinations all appeared to be using Microsoft Office 365 for their e-mail. What was wrong? Who was to blame? And how to fix it? The problem appeared like this: Nov 16 17:04:08 mail postfix/smtp[13330]: warning: no MX host for umcg.nl has a valid address record Nov 16 17:04:08 mail postfix/smtp[13330]: 1D1D21422C2: to=<-EMAIL-@umcg.nl>, relay=none, delay=2257, delays=2256/0.02/0.52/0, dsn=4. /blog/2016/patch-a-day-dovecot-broken-mime-parts-xenial/ Wed, 30 Nov 2016 00:00:00 +0000 /blog/2016/patch-a-day-dovecot-broken-mime-parts-xenial/ At times, Dovecot started spewing messages into dovecot.log about a corrupted index cache file because of “Broken MIME parts”. This happened on Ubuntu/Xenial with dovecot_2.2.22-1ubuntu2.2: imap: Error: Corrupted index cache file dovecot.index.cache: Broken MIME parts for mail UID 33928 in mailbox INBOX: Cached MIME parts don't match message during parsing: Cached header size mismatch (parts=4100...) imap: Error: unlink(dovecot.index.cache) failed: No such file or directory (in mail-cache.c:28) imap: Error: Corrupted index cache file dovecot. /blog/2016/tmpfs-files-not-found-systemd/ Tue, 15 Nov 2016 00:00:00 +0000 /blog/2016/tmpfs-files-not-found-systemd/ While debugging a problem with EDNS records, I wanted to get some cache info from the PowerDNS pdns-recursor. rec_control dump-cache should supply it, but I did not see it. # rec_control dump-cache out.txt Error opening dump file for writing: Permission denied Doh, it’s running as the pdns user. Let’s write in /tmp. # rec_control dump-cache /tmp/out.txt dumped 42053 records # less /tmp/out.txt /tmp/out.txt: No such file or directory Wait what? No files? /blog/2016/setting-up-powerdns-slave-untrusted-host/ Wed, 02 Nov 2016 00:00:00 +0000 /blog/2016/setting-up-powerdns-slave-untrusted-host/ When migrating our nameserver setup to start using DNSSEC, a second requirement was to offload a resolver to somewhere off-network. You want your authoritative nameservers to be distributed both accross different geographical regions, networks and top level domains. That means, don't do this: ns1.thedomain.com - datacenter X in Groningen ns2.thedomain.com - datacenter X in Groningen Do do this: ns1.thedomain.com - datacenter X in Groningen ns2.thedomain.org - datacenter Y in Amsterdam In our case, we could use a third nameserver in a separate location: a virtual machine hosted by someone other than us. /blog/2016/mysql-sys-schema-mysqldump-failure/ Fri, 28 Oct 2016 00:00:00 +0000 /blog/2016/mysql-sys-schema-mysqldump-failure/ After upgrading the mysql-server to 5.7 and enabling GTIDs, the mysql-backup script started spewing errors. Warning: A partial dump from a server that has GTIDs will by default include the GTIDs of all transactions, even those that changed suppressed parts of the database. If you don't want to restore GTIDs, pass --set-gtid-purged=OFF. To make a complete dump, pass --all-databases --triggers --routines --events. (...repeated for every database schema...) mysqldump: Couldn't execute 'SHOW FIELDS FROM `host_summary`': View 'sys. /blog/2016/copy-pasting-into-java-applications-x11/ Thu, 27 Oct 2016 00:00:00 +0000 /blog/2016/copy-pasting-into-java-applications-x11/ The other day I was rebooting our development server. It has full disk encryption, and the password for it has to be specified at boot time, long before it has network access. Even though the machine is in the same building, walking over there is obviously not an option. The machine has IPMI, like all modern machines do, so we can connect a virtual console over the local network. For that, we use the SuperMicro ipmiview tool. /blog/2016/packaging-supermicro-ipmiview-debian/ Tue, 25 Oct 2016 00:00:00 +0000 /blog/2016/packaging-supermicro-ipmiview-debian/ Do you want to quickly deploy SuperMicro ipmiview on your desktop? IPMI is a specification for monitoring and management of computer hardware. Usually this is used for accessing servers in a data center when the regular remote login is not available. Think: hard rebooting a stuck machine, specifying the full disk encryption password at boot time, logging onto a machine where the remote login (ssh daemon) has disappeared. The SuperMicro IPMI devices have an embedded webserver, but it requires Java to access the console. /blog/2016/golang-statically-linked/ Tue, 18 Oct 2016 00:00:00 +0000 /blog/2016/golang-statically-linked/ So, Go binaries are supposed to be statically linked. That’s nice if you run inside cut-down environments where not even libc is available. But sometimes they use shared libraries anyway? TL;DR: Use CGO_ENABLED=0 or -tags netgo to create a static executable. Take this example: $ go version go version go1.6.2 linux/amd64 $ go build gocollect.go $ file gocollect gocollect: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, \ interpreter /lib64/ld-linux-x86-64. /blog/2016/sipp-travis-osx-build-openssl/ Wed, 05 Oct 2016 00:00:00 +0000 /blog/2016/sipp-travis-osx-build-openssl/ A couple of days ago our SIPp Travis CI builds started failing due to missing OpenSSL include files. Between SIPp build 196 and SIPp build 215 the OSX builds on Travis started failing with the following configure error: checking openssl/bio.h usability... no checking openssl/bio.h presence... no checking for openssl/bio.h... no configure: error: <openssl/bio.h> header missing It turns out that something had changed in the build environment and OpenSSL headers and libraries were no longer reachable. /blog/2016/lxc-create-image-debian-squeeze/ Wed, 14 Sep 2016 00:00:00 +0000 /blog/2016/lxc-create-image-debian-squeeze/ I’m quite happy with our LXC environment on which I’ve got various Debian and Ubuntu build VMs so I can package backports and other fixes into nice .deb packages. Today I needed an old Debian/Squeeze machine to build backports on. Step one: check the lists. $ lxc remote list +-----------------+--------------------------+---------------+-----+-----+ | NAME | URL | PROTOCOL | PUB | STC | +-----------------+--------------------------+---------------+-----+-----+ | images | https://images.linuxco...| simplestreams | YES | NO | +-----------------+--------------------------+---------------+-----+-----+ | local (default) | unix:// | lxd | NO | YES | +-----------------+--------------------------+---------------+-----+-----+ | ubuntu | https://cloud-images. /blog/2016/planned-maintenance-22-august-2016/ Wed, 10 Aug 2016 13:04:16 +0000 /blog/2016/planned-maintenance-22-august-2016/ In the night of Monday (August. 22nd) to Tuesday (August. 23rd) between 23:45 and 06:00 we will perform network maintenance on our core network. ####Maintenance window Monday (August. 22nd) to Tuesday (August. 23rd) between 23:45 and 06:00 (CEST) ####Description One of the core routers (CR2) at TCN will be relocated to a new rack to allow further expansion on this site and make some desired improvements in the meantime. Impact During the maintenance, routing will be adjusted to free CR2 of traffic. /blog/2016/http-proxy-httpoxy-vulnerability-dutch/ Tue, 19 Jul 2016 08:03:33 +0000 /blog/2016/http-proxy-httpoxy-vulnerability-dutch/ Een stukje uitleg over de HTTP_PROXY vulnerability die op https://httpoxy.org/ gemeld is. Je site is vulnerable indien aan de volgende criteria wordt voldaan: Je hebt een webapplicatie die zelf webaanroepen doet; bijvoorbeeld communicatie met een payment provider of interne backend (d.m.v. python-requests, curl, http_get, etc…). De webserver laat de Proxy header door als HTTP_PROXY. Dit is standaard zo. De HTTP_PROXY header komt tussen de environment variabelen. Dit is niet zo onder bijvoorbeeld python met uwsgi, maar wel bij CGI applicaties en ook bij php-fpm en apache-mod_php! /blog/2016/planned-maintenance-21-july-2016/ Tue, 12 Jul 2016 12:31:31 +0000 /blog/2016/planned-maintenance-21-july-2016/ In the night of Thursday (Jul. 21st) to Friday (Jul. 22nd) between 23:00 and 04:00 we will perform network maintenance on our core network. ####Maintenance window Thursday (Jul. 21st) to Friday (Jul. 22nd) between 23:00 and 04:00 (CEST) ####Description Extra network capacity on our core network will be deployed and tested. During the maintenance, routing will be adjusted to free the links affected of any traffic. Impact No impact expected. Increased chance for short service disruptions due to changes in network. /blog/2016/letsencrypt-license-update-show-differences/ Sat, 09 Jul 2016 00:00:00 +0000 /blog/2016/letsencrypt-license-update-show-differences/ This morning, Let’s Encrypt e-mailed me that the Subscriber Agreement was updated; but it had no diff. Let’s Encrypt Subscriber, We’re writing to let you know that we are updating the Let’s Encrypt Subscriber Agreement, effective August 1, 2016. You can find the updated agreement (v1.1.1) as well as the current agreement (v1.0.1) in the “Let’s Encrypt Subscriber Agreement” section of the following page: https://letsencrypt.org/repository/ Thank you for helping to secure the Web by using Let’s Encrypt. /blog/2016/planned-maintenance-17-jun-2016/ Fri, 10 Jun 2016 12:14:48 +0000 /blog/2016/planned-maintenance-17-jun-2016/ In the night of Friday (Jun. 17th) to Saturday (Jun. 18th) between 23:00 and 4:00 we will perform network maintenance on our core network. ####Maintenance window Friday (Jun. 17th 2016) to Saturday (Jun. 18th 2016) between 23:00 and 4:00. ####Description Extra network capacity on our core network will be deployed and tested. During the maintenance routing will be adjusted to allow active maintenance on certain fiber paths. Impact No impact expected. /blog/2016/apt-insufficiently-signed-weak-digest/ Thu, 24 Mar 2016 00:00:00 +0000 /blog/2016/apt-insufficiently-signed-weak-digest/ When adding our own apt repository to a new Ubuntu/Xenial machine, I got a “insufficiently signed (weak digest)” error. # apt-get update ... W: gpgv:/var/lib/apt/lists/partial/ppa.osso.nl_ubuntu_dists_xenial_InRelease: The repository is insufficiently signed by key 4D1...0F5 (weak digest) Confirmed it with gpgv. # gpgv --keyring /etc/apt/trusted.gpg \ /var/lib/apt/lists/ppa.osso.nl_ubuntu_dists_xenial_InRelease gpgv: Signature made Wed 23 Mar 2016 10:14:48 AM UTC using RSA key ID B36530F5 gpgv: Good signature from "PPA-OSSO-NL <support+ppa@osso.nl>" # gpgv --weak-digest sha1 --verbose --keyring /etc/apt/trusted. /blog/2016/lxcfs-proc-uptime/ Wed, 23 Mar 2016 00:00:00 +0000 /blog/2016/lxcfs-proc-uptime/ When removing the excess LXC and LXD package from the LXC guest and working around Ubuntu/Xenial reboot issues I noticed the lxcfs mounts on my LXC guest. (No, you don’t need the lxcfs package on the guest.) guest# mount | grep lxc lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) lxcfs on /proc/swaps type fuse. /blog/2016/lxc-ubuntu-xenial-reboot/ Tue, 22 Mar 2016 00:00:00 +0000 /blog/2016/lxc-ubuntu-xenial-reboot/ The current Ubuntu/Xenial guest image on our new LXD container host contained too many packages. It held the lxd package and a bunch of lxc packages. They are not needed on the container guest. At some point before or after removing them, for some reason the ZFS container got unmounted. This went unnoticed until I tried a reboot: guest# reboot lxd# lxc exec guest /bin/bash error: Container is not running. lxd# lxc start guest error: Error calling 'lxd forkstart guest /var/lib/lxd/containers /var/log/lxd/guest/lxc. /blog/2016/renaming-lxd-managed-lxc-container/ Mon, 21 Mar 2016 00:00:00 +0000 /blog/2016/renaming-lxd-managed-lxc-container/ Renaming an LXD managed LXC container is not straight forward. But if you want to rename the host from inside the container, you should do so on the outside as well. If you don’t, you may notice that for instance the DHCP manual IP address assignment doesn’t work as expected. Creating a new LXC container For example, we’ll create a new container called walter-old with a fresh Debian/Jessie on it. /blog/2016/missing-sofiles-linker-asterisk-pjsip/ Sun, 21 Feb 2016 00:00:00 +0000 /blog/2016/missing-sofiles-linker-asterisk-pjsip/ When compiling Asterisk with a PJProject debianized using the debian/ directory to Ubuntu/Trusty, I got the following compile error: $ gcc -o chan_pjsip.so -pthread -shared -Wl,--version-script,chan_pjsip.exports,--warn-common \ chan_pjsip.o pjsip/dialplan_functions.o -lpjsua2 -lstdc++ -lpjsua -lpjsip-ua \ -lpjsip-simple -lpjsip -lpjmedia-codec -lpjmedia-videodev -lpjmedia-audiodev \ -lpjmedia -lpjnath -lpjlib-util -lsrtp -lpj -lm -lrt -lpthread \ -lSDL2 -lavformat -lavcodec -lswscale -lavutil -lv4l2 -lopencore-amrnb \ -lopencore-amrwb /usr/bin/ld: cannot find -lSDL2 /usr/bin/ld: cannot find -lavformat /usr/bin/ld: cannot find -lavcodec /usr/bin/ld: cannot find -lswscale /usr/bin/ld: cannot find -lavutil /usr/bin/ld: cannot find -lv4l2 /usr/bin/ld: cannot find -lopencore-amrnb /usr/bin/ld: cannot find -lopencore-amrwb collect2: error: ld returned 1 exit status That’s odd. /blog/2016/cve-2015-7547-glibc-getaddrinfo-stack-based-buffer-overflow/ Tue, 16 Feb 2016 17:01:38 +0000 /blog/2016/cve-2015-7547-glibc-getaddrinfo-stack-based-buffer-overflow/ On February 16, 2016 details on a vulnerability in glibc were released (CVE-2015-7547). The vulnerability is remotely exploitable and affects a lot of systems. More info will be added later when more information is available. We started emergency patch procedures for our environments and managed customer environments. Summary Classification: Critical. Remote exploitation possible. Impact: Wide impact, all services that use glibc and perform dns resolving are vulnerable. upstream description The glibc DNS client side resolver is vulnerable to a stack-based buffer overflow when the getaddrinfo() library function is used. /blog/2016/python-xinetd-virtualenv/ Fri, 12 Feb 2016 00:00:00 +0000 /blog/2016/python-xinetd-virtualenv/ So, while developing a server application for a client, my colleague Harm decided it would be a waste of our programming time to add TCP server code. Inetd and friends can do that really well. The amount of new connects to the server would be minimal, so the overhead of spawning a new Python process for every connect was negligible. Using xinetd as an inetd server wrapper is simple. The config would look basically like this: /blog/2016/planned-maintenance-tcn-feb-2016/ Fri, 05 Feb 2016 17:24:25 +0000 /blog/2016/planned-maintenance-tcn-feb-2016/ In the night of Friday (Feb. 12th) to Saturday (Feb. 13th) between 1:00 and 2:00 our co-location provider will perform maintenance on the PDU’s in one of our racks. Customers we consider to be directly affected (bare metal servers) will receive an additional notification. Outside of that it’s mostly OSSO infrastructure services that are affected. ####Maintenance window 01:00-02:00 on 13th of Februari 2016. ####Description The PDU’s of one of the racks will be taken into service one by one (A and B feed). /blog/2016/salt-master-losing-children/ Fri, 15 Jan 2016 00:00:00 +0000 /blog/2016/salt-master-losing-children/ I recently set up psdiff on a few of my servers as a basic means to monitor process activity. It disclosed that my SaltStack master daemon — which I’m running as a non-privileged user — was losing a single child, exactly 24 hours after I had ran salt commands. This seemed to be a recurring phenomenon. The salt server — version 0.17.5+ds-1 on Ubuntu Trusty — was running these processes: /blog/2016/polyglot-xhtml/ Wed, 13 Jan 2016 00:00:00 +0000 /blog/2016/polyglot-xhtml/ Polyglot XHTML: Serving pages that are valid HTML and valid XML at the same time. A number of documents have been written on the subject, which I shall not repeat here. My summary: HTML5 is not going away. XHTML pages validate in the browser. If you can get better validation during the development of your website, then you’ll save yourself time and headaches. Thus, for your development environment, you’ll set the equivalent of this: /blog/2015/availability-december-2015/ Mon, 21 Dec 2015 14:59:35 +0000 /blog/2015/availability-december-2015/ From 24th of December we are on leave and return to the office on the 4th of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability. /blog/2015/router-upgrade-rug-rh-pop-dec-2015/ Mon, 30 Nov 2015 15:42:07 +0000 /blog/2015/router-upgrade-rug-rh-pop-dec-2015/ In the night of Monday (Dec. 7th) to Tuesday (Dec. 8th) between 1:00 and 6:00 we will upgrade the router at our RUG Rekenhal POP. Impact is limited to IP Access locations and IP Transit customers on this POP. ####Maintenance window 01:00-04:00 on 8th of December 2015. ####Description We will upgrade the router to allow planned network upgrades. Impact The router at the RUG Rekenhal POP will be unavailable for 30min-60min. /blog/2015/asterisk-editline-key-bindings/ Sat, 21 Nov 2015 00:00:00 +0000 /blog/2015/asterisk-editline-key-bindings/ Getting the Asterisk PBX CLI to work more like you’re used to from the (readline) bash shell can, be a time-saver. For example, you may want reverse-i-search (^R), backward word deletion (^W) and word skipping (^<arrow-left> and ^<arrow-right>). It can be done, but you must configure the editline library in a similar manner as you would configure .inputrc. Support for the .editrc configuration file was added in May 2011 (git commit d508a921). /blog/2015/planned-maintenance-virtual-servers-tcn-nov-2015/ Thu, 19 Nov 2015 15:21:27 +0000 /blog/2015/planned-maintenance-virtual-servers-tcn-nov-2015/ In the night of Thursday (Nov. 26th) to Friday (Nov. 27th) between 1:00 and 6:00 we will perform maintenance to the virtual server infrastructure at location TCN. ####Maintenance window 01:00-06:00 on 27th of November 2015 ####Description Virtual server infrastructure will be upgraded to a new major software release. Due to the major version upgrade and incompatibility between versions servers will experience downtime for 15-60 minutes Impact Virtual servers will be offline for ~15 minutes (up to 60 minutes worst case). /blog/2015/encfs-recursion-into-itself/ Thu, 12 Nov 2015 00:00:00 +0000 /blog/2015/encfs-recursion-into-itself/ We wanted to use EncFS to be able to store encrypted backups. The requirements for that are: The backup server initiates the backup. That’s where we configure which hours are safe (resource wise) and which files need backing up (etc, home, root, srv, …). And it means the backup server can safely be placed behind a gateway disallowing all incoming connections. The backup server cannot know the passwords of files. /blog/2015/encfs-configure-libboost/ Tue, 10 Nov 2015 00:00:00 +0000 /blog/2015/encfs-configure-libboost/ I ran into an obscure Could not link against ! error when configuring EncFS: ~/src$ apt-get source encfs ... ~/src$ cd encfs-1.7.4/ ~/src/encfs-1.7.4$ ./configure ... configure: WARNING: BOOST_CPPFLAGS -I/usr/include checking whether the Boost::Serialization library is available... yes configure: error: Could not link against ! That’s odd. And not immediately obvious how to fix. For starters we need all the dependencies that Debian defines: ~/src/encfs-1.7.4$ sed -e '/^Build-Depends: /!d;s/^[^:]*: //;s/([^)]*)//g;s/,//g' \ debian/control debhelper librlog-dev librlog5 libfuse-dev libssl-dev pkg-config libboost-serialization-dev libboost-filesystem-dev quilt dh-autoreconf ~/src/encfs-1. /blog/2015/core-router-issue-oct-nov-2015/ Wed, 04 Nov 2015 12:11:12 +0000 /blog/2015/core-router-issue-oct-nov-2015/ Service disruptions on one of the core routers location (CR1) in TCN. 5 Nov 2015 - 01:00 CR1 Supervisor placement Following up the maintenance finished on Monday, we will add an extra Router Supervisor in CR1 for extended redundancy in case of main Supervisor failure. This maintenance will be carried out tonight, at 01:00 on the 5th of November. There is no expected impact. This maintenance is a followup on the hardware replacement of the Supervisor in CR1. /blog/2015/planned-network-maintenance-0100-0300-6-nov-2015/ Wed, 28 Oct 2015 13:54:19 +0000 /blog/2015/planned-network-maintenance-0100-0300-6-nov-2015/ In the night of Thursday (Nov. 5th) to Friday (Nov. 6th) between 1:00 and 3:00 we will perform network maintenance at our TCN co-location. ####Maintenance window 01:00-03:00 on 6th of November 2015 ####Description We will perform changes in the network configuration which will cause a change of mac address of the gateway for each subnet. Impact Depending on the device it may take some time to pick up this change. Generally, busy servers will pick up the changes almost instantly and servers which are mostly idle may take a while. /blog/2015/scapy-dns-server-snippet/ Mon, 05 Oct 2015 00:00:00 +0000 /blog/2015/scapy-dns-server-snippet/ A few days ago, the Scapy project was brought to my attention. Scapy is an internet packet manipulation library for Python2. It can be used to sniff and decode packets, or to generate your own custom packets. In the most basic form, it runs on raw sockets, sniffing and decoding traffic like tcpdump. See the sniff() examples and the send(IP(dst="1.2.3.4") / ICMP()) example for sending a simple packet. But just as easily, it works on regular datagram sockets — those that you don’t need CAP_NET_RAW powers for. /blog/2015/flake8-vim-python2-python3/ Tue, 15 Sep 2015 00:00:00 +0000 /blog/2015/flake8-vim-python2-python3/ To syntax check Python code before executing, I use flake8. And when coding in the Vim editor, I use the vim-flake8 plugin that allows me to hit <F7> to quickly check for errors in the file I’m currently working in. But, there are currently two common flavors of Python: python2 and python3. And therefore flake8 comes in two flavors as well — you guessed it — a python2 and a python3 flavor. /blog/2015/python-subprocess-winch/ Tue, 01 Sep 2015 00:00:00 +0000 /blog/2015/python-subprocess-winch/ While I was writing a Python tool to wrap C Gdb so I could fetch some info out of it automatically, I ran into the issue that it reads the terminal size (lines x columns) to adjust its output. I wanted consistent machine readable output, so I enlarged the terminal size programmatically: now row based output would not get wrapped by Gdb. Later I noticed that it would cease to use the terminal size — in fact, use the default 80 columns — if I also redirected stderr to a non-tty. /blog/2015/debian-packaging-asterisk-13/ Tue, 18 Aug 2015 00:00:00 +0000 /blog/2015/debian-packaging-asterisk-13/ As of this writing, Debian testing (stretch) contains Asterisk version 13.1.0. The Debian source as GIT repository is here: https://anonscm.debian.org/git/pkg-voip/asterisk.git (browse) Packaging a newer version is not that hard, if we start out with the debian/ directory kindly supplied by the Debian maintainers. Hints to get things running: Use a local git repository By using a local git repository in your unpacked Asterisk dir, you can quickly restart from scratch any time you mess anything up. /blog/2015/on-the-fly-encrypted-backups/ Wed, 24 Jun 2015 00:00:00 +0000 /blog/2015/on-the-fly-encrypted-backups/ I was wondering how easy it was to encrypt files before rsyncing them away to the backup machine. A quick search turned up the suggestion to use encfs by the user Thor on ServerFault. That looks like a decent solution. Let’s figure out if it meets our needs. The idea is that we do this: # mount read-only encrypted virtual copy of unencrypted local data: encfs --reverse -o ro ~/data/ ~/. /blog/2015/monitoring-process-open-files-limit/ Thu, 23 Apr 2015 00:00:00 +0000 /blog/2015/monitoring-process-open-files-limit/ Here, an awesome shell one-liner to find which process uses the most files, relative to its max-open-files soft limit. $ for x in /proc/[0-9]* do fds=0 max=`awk '/^Max open files/ {print $4}' $x/limits 2>/dev/null` && for t in $x/fd/*; do fds=$((fds+1)); done && test "${max:-0}" -gt 0 && echo $((fds*100/max)) ${x##*/} done | sort -rn | while read l do pid=${l##* }; echo "$l`readlink /proc/$pid/exe`"; break; done 57 16674 /usr/lib/dovecot/imap-login So, my imap-login (pid 16674) apparently uses 57% percent of its allowed max open files. /blog/2015/converting-unprintable-pdf-imagemagick/ Wed, 11 Mar 2015 00:00:00 +0000 /blog/2015/converting-unprintable-pdf-imagemagick/ Okay, so we all know that printers are sent from hell, but we still need to use them from time to time. Today, we were trying to print a PDF document with bar codes on it. Amazingly enough, the text on the PDF looked fine, but the bar codes (images) appeared as if they were wrapped at the wrong place. Luckily, convert(1) from ImageMagick came to the rescue: $ convert -density 300 -define pdf:fit-page=A4 input. /blog/2015/proxmox-resource-usage/ Wed, 04 Mar 2015 00:00:00 +0000 /blog/2015/proxmox-resource-usage/ As I mentioned the other day, my VM was slow, so I needed a way to figure out which VM guests were causing the heavy load on our Proxmox platform. I hacked up proxtop to enumerate the top resource users: $ ./proxtop -t day proxmox.example.com monitor@pve Password:<enter password> SORTED BY: cpu, avg ... SORTED BY: diskread, avg ------------------ #0: 3.1 MiB/s pve10 (acme-bugs-bunny) #1: 1.3 MiB/s pve07 (customerX-private) #2: 992.3 KiB/s pve10 (acme-road-runner) . /blog/2015/proxmox-api-python-module/ Mon, 02 Mar 2015 00:00:00 +0000 /blog/2015/proxmox-api-python-module/ So, my VM was slow, and I needed to know which VM guest was eating all the resources. These VM containers are all managed by Proxmox; which is great, but it doesn’t show which VM guest is eating all the resources. Luckily, Proxmox provides an API to get that info. The docs pointed to two API modules for Python, my language of choice for these kinds of jobs: proxmoxer and pyproxmox. /blog/2015/zabbix-api-python-module/ Fri, 27 Feb 2015 00:00:00 +0000 /blog/2015/zabbix-api-python-module/ Today, my choice of Python modules to Interface with Zabbix. They are all pretty similar, so that made it harder to choose. Here the six modules, as mentioned on the Zabbix wiki are, in the order of my preference. Note that second and third came close, but I favor clean documented code and fewer dependencies. The last ones didn’t get tested because of my Python3 requirement. zabbix-client # pip: zabbix-client # pep: 99% # last-update: Aug. /blog/2015/asterisk-dialplan-variable-expansion-security/ Thu, 12 Feb 2015 00:00:00 +0000 /blog/2015/asterisk-dialplan-variable-expansion-security/ Even after writing plenty of Asterisk PBX dialplan, I occasionally get bitten by the unintuitiveness of the parser. A few rules to avoid mistakes, are: Always use double quotes on no side of the expression, or better yet, on both if there is a chance that the value is empty: $[${HANGUPCAUSE}=17] or $["${value_which_may_be_empty}"="somevalue"] Otherwise try to avoid double quotes (and semi-colons, and backslashes) whenever possible. If you need to escape them, it’s too easy to get it wrong. /blog/2015/ghost-glibc-gethostbyname-buffer-overflow/ Wed, 28 Jan 2015 10:25:50 +0000 /blog/2015/ghost-glibc-gethostbyname-buffer-overflow/ A high risk security issue in glibc was disclosed last night. Because of the potential high impact we started our emergency patch procedures for osso managed environments and notify customers with self managed environments. Ghost vulnerability details Qualys discovered a buffer overflow in dns resolve functions in the GNU C library (glibc). They created a proof of concept exploit for exim and dubbed the vulnerability "GHOST". All processes that might do dns lookups are susceptible to attacks when using a vulnerable glibc version. /blog/2015/gitlab-upgrade-ruby-bundle/ Wed, 28 Jan 2015 00:00:00 +0000 /blog/2015/gitlab-upgrade-ruby-bundle/ While we do Python VirtualEnv stuff every day, we rarely do Ruby environments. And after the Ubuntu dist-upgrade, the Ruby dependencies for our GitLab were broken — as was expected. This happens for Python pip installed packages as well. They’re linked against older system libraries, which have been removed by the upgrade. How to fix the Gitlab dependencies? Browse through the upgrade docs to find a bundle install command. # cd /home/git/gitlab # sudo -u git -H bundle install \ --without development test postgres --deployment # for MySQL That did… absolutely nothing — again, as was expected. /blog/2015/fail2ban-started-e-mail-disable/ Sat, 10 Jan 2015 00:00:00 +0000 /blog/2015/fail2ban-started-e-mail-disable/ Tired of the Fail2ban start and stop e-mails? Especially after a manual fail2ban restart, the [Fail2Ban] vsftpd: stopped on HOSTNAME and [Fail2Ban] vsftpd: started on HOSTNAME mail tuple is too spammy. Quick fix to disable them: Create a new file, named /etc/fail2ban/actions.d/sendmail-no-start-stop.local: diff --git /etc/fail2ban/action.d/sendmail-no-start-stop.local /etc/fail2ban/action.d/sendmail-no-start-stop.local new file mode 100644 index 0000000..cb7ecb9 --- /dev/null +++ /etc/fail2ban/action.d/sendmail-no-start-stop.local @@ -0,0 +1,3 @@ +[Definition] +actionstart = +actionstop = And — you’re using mta = sendmail right? /blog/2015/git-gnutls-handshake-failed-nginx-ciphers/ Fri, 09 Jan 2015 00:00:00 +0000 /blog/2015/git-gnutls-handshake-failed-nginx-ciphers/ When trying to keep up with all the TLS/SSL security changes, you need to modify your nginx config every now and then. The good TLS config may look like this: # nginx.conf: http { ssl_certificate /etc/ssl/MY_DOMAIN.pem; ssl_certificate_key /etc/ssl/MY_DOMAIN.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA; ssl_session_cache shared:SSL:5m; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security "max-age=15768000; includeSubDomains"; And the above config is accompanied by a fairly good A grade from the Qualys SSL Labs Analyzer. /blog/2015/uuid-storage-mysql/ Tue, 06 Jan 2015 00:00:00 +0000 /blog/2015/uuid-storage-mysql/ Storing an UUID in MySQL efficiently: DROP FUNCTION IF EXISTS uuidbin; CREATE FUNCTION uuidbin(uuid_val varchar(36)) RETURNS varbinary(16) DETERMINISTIC SQL SECURITY INVOKER RETURN CONCAT(UNHEX(LEFT(uuid_val,8)),UNHEX(MID(uuid_val,10,4)), UNHEX(MID(uuid_val,15,4)),UNHEX(MID(uuid_val,20,4)), UNHEX(RIGHT(uuid_val,12))); DROP FUNCTION IF EXISTS uuidstr; CREATE FUNCTION uuidstr(uuid_val varbinary(16)) RETURNS varchar(36) DETERMINISTIC SQL SECURITY INVOKER RETURN LOWER(CONCAT(HEX(LEFT(uuid_val,4)),'-',HEX(MID(uuid_val,5,2)), '-',HEX(MID(uuid_val,7,2)),'-',HEX(MID(uuid_val,9,2)), '-',HEX(RIGHT(uuid_val,6)))); Now you can create your uuid columns with type binary(16). And conversion is easy: mysql> select uuidstr(uuidbin(uuidstr(uuidbin(uuidstr(uuidbin( 'a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f')))))) as uuid_back_and_forth; +--------------------------------------+ | uuid_back_and_forth | +--------------------------------------+ | a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f | +--------------------------------------+ /blog/2015/django-makemessages-slow/ Mon, 05 Jan 2015 00:00:00 +0000 /blog/2015/django-makemessages-slow/ Django makemessages can be quite slow on larger projects. $ time python ../manage.py makemessages -lnl -ddjango processing language nl real 0m8.203s user 0m2.670s sys 0m5.763s Why does it take so long? Well, it’s system call heaven: $ strace -f python ../manage.py makemessages -lnl -ddjango \ >tmp.log 2>&1 $ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log | sort | uniq -c | sort -n | tail -n10 10893 rt_sigaction 16179 stat 16819 fcntl 22875 access 27833 read 32469 open 33650 fstat 40891 mprotect 69181 mmap 1267039 close For every file, a call to xgettext(1) is made. /blog/2014/photo-exif-timestamp-filesystem-mtime/ Mon, 17 Nov 2014 00:00:00 +0000 /blog/2014/photo-exif-timestamp-filesystem-mtime/ Sometimes, after a stray copy operation, your filesystem times may reflect the time the files were copied instead of when the file was actually last altered. For example this image folder here: $ ls -l phone2013 total 320856 -rw-rw-r-- 1 walter walter 1524591 nov 17 21:52 2012-10-28 08.54.58.jpg -rw-rw-r-- 1 walter walter 1534840 nov 17 21:52 2012-10-28 08.55.04.jpg -rw-rw-r-- 1 walter walter 1635908 nov 17 21:52 2012-10-28 08.55.09.jpg ... -rw-rw-r-- 1 walter walter 1600504 nov 17 21:52 2013-10-22 11. /blog/2014/python-ctypes-socket-datagram/ Fri, 17 Oct 2014 00:00:00 +0000 /blog/2014/python-ctypes-socket-datagram/ So, I was really simply trying to figure out why talking to my OpenSIPS instance over a datagram unix socket failed. If I had bothered to check the server logs, I would immediately have seen that it was a simple stupid permission issue. Instead, I ended up reimplementing recvfrom and sendto in Python using the ctypes library. Which was completely useless, since Python socket.recvfrom and socket.sendto already work properly. To let the time spent on that not go to a complete waste, I give you (and myself) an example of ctypes usage. /blog/2014/rsyslog-cron-deleting-rules/ Wed, 10 Sep 2014 00:00:00 +0000 /blog/2014/rsyslog-cron-deleting-rules/ Syslog generally works fine as it is, so I don’t need to poke around in it often. That also means that I forget how to tweak it. How did you move those every-5-minutes cron jobs out of /var/log/syslog? The rules (selection + action) look like this in the Debian default config: *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log The manual has this to say about it: You can specify multiple facilities with the same priority pattern in one statement using the comma (,) operator. /blog/2014/maintenance-power-datacenter-tcn-13-20-27-sept/ Tue, 09 Sep 2014 09:36:33 +0000 /blog/2014/maintenance-power-datacenter-tcn-13-20-27-sept/ One of our datacenter locations (TCN Telehouse) will have major maintenance on its power infrastructure this month. They scheduled 4 maintenance windows of 1,5 hours each during which either the A or B feed will be powerless. In the last weeks we’ve double checked our infrastructure and this week we will finish our last preparations. All equipment that is not equipped with dual power supplies is connected to an ATS (Automatic Transfer Switch) to achieve power redundancy. /blog/2014/daemon-reparented-init-user/ Sat, 06 Sep 2014 00:00:00 +0000 /blog/2014/daemon-reparented-init-user/ While I was battling an obscure Ubuntu shutdown issue — more about that later — I noticed that daemonized jobs started from my X session were not reparented to PID 1 init, but to a custom init --user, owned by me. What? I cannot start daemon that outlives my X session? That’s right, I cannot. Check this out: $ sh -c 'sleep 61 &' $ ps faxu | egrep 'init|sleep 61' root 1 . /blog/2014/git-resetting-merges/ Fri, 05 Sep 2014 00:00:00 +0000 /blog/2014/git-resetting-merges/ Today’s git question: does git reset undo a merge or only parts of it? TL;DR: It undoes the entire merge. If you think about it logically, it must, since an object describes the entire state of the repository. But it can feel awkward and unexpected that older items than the object that we’re resetting to, are removed as well. Let’s just try it. Set up a repository with two branches: /blog/2014/apt-hold-upgrades-dependencies/ Fri, 15 Aug 2014 00:00:00 +0000 /blog/2014/apt-hold-upgrades-dependencies/ Recently I wrote about cherry picking upgrades. Sometimes you’ll want to do the inverse. For that purpose there exists apt-mark hold (and its counterpart apt-mark unhold). For example, you may to delay the mysql upgrade I mentioned, for now. In that case you do: # apt-mark hold mysql-client-5.5 mysql-common mysql-server-5.5 mysql-server-core-5.5 Now you can apt-get upgrade all the other packages while the mysql packages stay on hold. Note that these are shown in the held list every time you run upgrade, so you won’t forget about them. /blog/2014/squirrelmail-clicking-on-empty-subject/ Thu, 14 Aug 2014 00:00:00 +0000 /blog/2014/squirrelmail-clicking-on-empty-subject/ SquirrelMail on Debian/Wheezy (2:1.4.23~svn20120406-2) stopped showing (none) for e-mails that lack a subject. Now I cannot open any subject-less mail because there is nothing to click on. The quick fix: --- /usr/share/squirrelmail/functions/mailbox_display.php.orig 2014-08-15 10:37:37.000000000 +0200 +++ /usr/share/squirrelmail/functions/mailbox_display.php 2014-08-15 10:38:27.000000000 +0200 @@ -268,6 +268,9 @@ function printMessageInfo($imapConnectio $title = str_replace('"', "''", $title); $td_str .= " title=\"$title\""; } + if (!$subject) { + $subject = '(none)'; + } $td_str .= ">$flag$subject$flag_end</a>$bold_end"; echo html_tag( 'td', $td_str, 'left', $hlt_color ); break; /blog/2014/import-one-database-from-sql-dump/ Thu, 17 Jul 2014 14:05:19 +0000 /blog/2014/import-one-database-from-sql-dump/ Ever needed to restore only one database on a MySQL server and found out you only had one SQL dump containing all databases? Its quite common to dump all databases in one SQL file (mysqldump –all-databases or -A). But when using multiple databases on one MySQL instance you often need to restore just one of them. The minimal effort solution: mysql --one-database desired_db_name < alldatabases.sql fix! /blog/2014/compose-key-irony-punctuation-x11/ Thu, 26 Jun 2014 00:00:00 +0000 /blog/2014/compose-key-irony-punctuation-x11/ Transcript follows: [him] did I mention I'll be off from work earlier today because I'm having dinner with friends. I'll be off earlier today because I'm having dinner with friends. [me] where did you say you were going? [him] I'll be having dinner at the Grand Cafe Apparently the irony was lost on him. I should’ve used emoticons. But! Instead of emoticons, one may also use the irony punctuation: ⸮ /blog/2014/apt-cherry-pick-upgrades-dependencies/ Mon, 09 Jun 2014 00:00:00 +0000 /blog/2014/apt-cherry-pick-upgrades-dependencies/ So, doing an apt-get upgrade on a Debian or Ubuntu machine sometimes does more than you want at once. See this upgrade example I encountered just now: # apt-get upgrade ... The following packages will be upgraded: curl dpkg ifupdown iproute libcurl3 libcurl3-gnutls libgnutls26 libmysqlclient18 libsnmp-base libsnmp15 libssl1.0.0 libxml2 linux-firmware linux-generic-lts-quantal mysql-client-5.5 mysql-client-core-5.5 mysql-common mysql-server mysql-server-5.5 mysql-server-core-5.5 openssh-client openssh-server openssl tzdata update-manager-core whoopsie 26 upgraded, 0 newly installed, 0 to remove and 2 not upgraded. /blog/2014/vim-position-markers/ Sun, 08 Jun 2014 00:00:00 +0000 /blog/2014/vim-position-markers/ Did you ever wonder what the '<,'> characters mean when you CTRL-V visual block select text in vim? For example: you press CTRL-V and select a bit of text. Then type : (colon). Instead of just the colon, you see: :'<,'>. You append s/^/#/ hit enter. As requested, the selected block is now “commented out”. That’s a nice feature, but why the funny characters? In order to understand that, we remind you of the % (percent sign) that we use to select the entire file. /blog/2014/vim-reformat-textwidth-72/ Sat, 07 Jun 2014 00:00:00 +0000 /blog/2014/vim-reformat-textwidth-72/ My .vimrc usually starts out with this. Syntax highlighting is super, and my terminals always have a black background. The modeline option enables me and others to set certain options for certain files only. Like: {# vim: syntax=htmldjango: #} to mark a .html file as using the django html syntax instead of regular html syntax. See also my Inserting vim modelines tip. syn onset bg=darkset modelineSecond, since I develop a lot in Python, I enable the vim-flake8 python source code checker plugin: /blog/2014/postgresql-upgrade-ubuntu/ Thu, 08 May 2014 00:00:00 +0000 /blog/2014/postgresql-upgrade-ubuntu/ I always forget how easy it is to upgrade postgresql on Ubuntu (from 9.1 to 9.3 this time). It seems like a pain to have to manually upgrade the cluster, but when it comes down to it, it’s self-documenting and quick. My shell session basically went like this: $ sudo apt-get install postgresql-9.3 ... The following extra packages will be installed: postgresql-client-9.3 ... $ sudo /etc/init.d/postgresql stop * Stopping PostgreSQL 9. /blog/2014/openssh-nagle-too-much-buffering/ Wed, 07 May 2014 00:00:00 +0000 /blog/2014/openssh-nagle-too-much-buffering/ Recently I tried to open a connection to a remote server over SSH at a new location. The connection opened just fine, but it seemed that a few bytes kept getting buffered. It looked like this first animated gif you see. After a long wait, you realise that the data you’re wating just won’t come. First after pressing a key, you get the data. This isn’t workable… Enumerating the possible culprits, there could really only be the wifi-nat-modem — a Thomson TG789vn, Telia device — doing extra buffering, possibly conflicting with the Nagle algorithm (TCP_NODELAY). /blog/2014/ubuntu-trusty-git-diff-color/ Sat, 26 Apr 2014 00:00:00 +0000 /blog/2014/ubuntu-trusty-git-diff-color/ On my recently upgraded Ubuntu Trusty (14.04) machine, git diff started producing colorized output. That’s nice, but it’d be even nicer if it recognised that I’m using a dark background. Put this in your ~/.gitconfig. This colorscheme is the one you’re used to from vim. [color "diff"] meta = green bold frag = yellow bold old = red bold new = cyan bold /blog/2014/heart-bleed-openssl-security-issue/ Tue, 08 Apr 2014 20:31:01 +0000 /blog/2014/heart-bleed-openssl-security-issue/ Last night an important security vulnerability was made public with corresponding security updates. It risks exposing private keys when vulnerable. OpenSSL was vulnerable starting from their OpenSSL 1.0.1 release on 14th of March 2012 till OpenSSL 1.0.1g released on 7th of April 2014. Two security teams independently reported this issue and it’s safe to assume others did as well. On top of that it’s not possible to trace whether you were successfully exploited. /blog/2014/internet-connectivity-issue-3132014/ Mon, 31 Mar 2014 10:55:43 +0000 /blog/2014/internet-connectivity-issue-3132014/ This morning we experienced packet loss on one of our links. After rerouting the traffic the problem was resolved. Only routes over the GN-IX were affected. timeline 12:15 first notification of the packetloss. 12:36 - 12:40 Hickups in the internet connectivity due to rerouting of traffic. after 12:40 No more issues for our customers. Offhour followup re-enable temporary disabled links and rebalance traffic. Background information The issue occurred because GN-IX hit its capacity limit on their links between Groningen and Amsterdam. /blog/2014/zabbix-counting-security-updates/ Sat, 22 Mar 2014 00:00:00 +0000 /blog/2014/zabbix-counting-security-updates/ When you’re monitoring security update availability using Zabbix or some other monitoring tool, you’ll need a method to discern regular updates from security updates. I’ve seen my collegues do this: $ /usr/lib/update-notifier/apt-check --human-readable | grep security | awk '{print $1}' But that requires an install of the update-notifier-common package. (Note the -common. The main package has tons of requirements you don’t need.) In the quest for less dependencies — less installed packages — I used aptitude to get the info. /blog/2014/mysql-innodb-process-locks-in-limbo/ Wed, 12 Mar 2014 13:18:50 +0000 /blog/2014/mysql-innodb-process-locks-in-limbo/ After switching a virtual IP around with keepalived we experienced a locking issue. Some client process on the server had some locks which did not get released but we could not see which query caused it. This broke our MySQL replication in this case, it was waiting on the locks to be released while executing the binlogs. We located the locks with mysql> show engine innodb status\G; The specific transaction did not show any query details but did show us a whole list of locks. /blog/2014/freebsd-fix-cheat-sheet/ Wed, 12 Mar 2014 12:05:43 +0000 /blog/2014/freebsd-fix-cheat-sheet/ For me, using FreeBSD is still a bit like eating soup with a fork. Everything seems to make perfect sense but when I get to work I feel crippled and get annoyed by small differences with a GNU/Linux environment. A post to reduce the horror. The examples below work for a clean FreeBSD 10 install. This post is mostly useful for Linux users. I welcome additional tips. software management You can now use pkg which is quite friendly. /blog/2014/ddos-mitigated-ntp-amplification/ Thu, 20 Feb 2014 15:13:07 +0000 /blog/2014/ddos-mitigated-ntp-amplification/ Today we received a DDoS on our network which caused a service interruption for our customers for about 20 minutes. This blogpost is a short report on the impact and nature of the attack. Impact Impact was network wide and caused degraded service for our customers between 14:45 and 15:08, a little over 20 minutes. The graph shows the impact as seen from our UK monitoring node (off net). Incoming traffic All our uplinks were saturated. /blog/2014/python-parsestring-silently-skips-entities/ Wed, 15 Jan 2014 00:00:00 +0000 /blog/2014/python-parsestring-silently-skips-entities/ The Python xml.dom.minidom parseString silently skips over unknown entities. The only entities it does know, are <, >, &, ' and " and of course the numeric entities &#nn; and &#xhh;. That’s obvious, because those are the only ones defined in the XML 1.0 spec. However, if you’re parsing XHTML documents, it’s not nice that the entity references to special characters silently get dropped. Other people have stubled on the same issue, like in parsing xml containing &entities; with minidom and Problem with minidom and special chars in HTML. /blog/2014/bson-json-converter/ Mon, 13 Jan 2014 00:00:00 +0000 /blog/2014/bson-json-converter/ A simple script to convert BSON data to JSON data: bson2json.py (view) Example usage: $ bson2json.py /var/backups/mongodb/all-dbs.mon/graylog2/streams.bson --pretty [ { "_id": "506ed227dc1d710c0700000e", "additional_columns": [], "alarm_active": true, "alarm_callbacks": [ "org.graylog2.emailalarmcallback.callback.EmailAlarmCallback", "org.graylog2.execalarmcallback.callback.ExecAlarmCallback" ], "alarm_limit": 80, "alarm_period": 5, "alarm_timespan": 5, "created_at": "2012-10-05T12:27:19Z", ... /blog/2014/thunderbird-reply-only-selected-text/ Wed, 08 Jan 2014 00:00:00 +0000 /blog/2014/thunderbird-reply-only-selected-text/ Apparently I’m not the only one who randomly selects text as they read. My colleagues complained about this issue too. If you click Reply in Thunderbird Mail, only the text you recently selected is included in the new message. That’s not what I wanted! Luckily the Mozilla developers realised this too. Go to about:config and flip the switch. mailnews.reply_quoting_selection = false /blog/2013/amavis-tag-subject-virus/ Thu, 26 Sep 2013 00:00:00 +0000 /blog/2013/amavis-tag-subject-virus/ Today we got a suspiciously good looking e-mail in the inbox. Someone who supposedly got a reminder about an unpaid invoice from us. The mail contained a zip-file with two scans. The first was a PDF, the second was an executable (a virus obviously). So.. where was the Amavis virus/spam scanner in all this? Show headers revealed that something was detected: X-Amavis-Alert: BANNED, message contains .exe,scan2/HP scan scan =?iso-8859-1?Q?HYJKIOPH5600002.=E2=80=AEfdp.exe?= Then why weren’t we informed? /blog/2013/gnome-calculator-missing-menu/ Tue, 24 Sep 2013 00:00:00 +0000 /blog/2013/gnome-calculator-missing-menu/ After the upgrade of my desktop to Ubuntu Raring (13.04) my gnome-calculator’s menu bar had become unreachable. I don’t need the menu, except that it went into default BASIC mode. And I need the PROGRAMMING mode. The configuration seemed to be okay (accessible through gconf-editor): $ gconftool /apps/gcalctool --dump | grep -B1 -A4 mode <entry> <key>mode</key> <value> <string>PROGRAMMING</string> </value> </entry> <entry> <key>modetype</key> <value> <string>PROGRAMMING</string> </value> </entry> But that was apparently the old config. /blog/2013/teamviewer-without-all-ia32-libs/ Mon, 16 Sep 2013 00:00:00 +0000 /blog/2013/teamviewer-without-all-ia32-libs/ A quick rundown on installing TeamViewer without a gazillion ia32-libs. The problem: if you attempt to install teamviewer_linux_x64.deb on your 64-bit machine, the ia32-libs dependency tries to install more than 200 packages. That not only feels like overkill, it takes a hell of a long time too. The solution: alter the dependency list in the .deb and create a small metapackage that references only the required libs. What follows, is the steps how. /blog/2013/mysql-count-occurrences/ Wed, 11 Sep 2013 00:00:00 +0000 /blog/2013/mysql-count-occurrences/ Voilà, a MySQL function to count occurrences of a character (or a string of characters). DROP FUNCTION IF EXISTS OCCURRENCES; delimiter // CREATE FUNCTION OCCURRENCES (`needle` VARCHAR(255), `hackstack` TEXT) RETURNS INT NOT DETERMINISTIC READS SQL DATA SQL SECURITY INVOKER BEGIN DECLARE `result` INT DEFAULT -1; DECLARE `pos` INT DEFAULT 0; DECLARE `skip` INT DEFAULT LENGTH(`needle`); REPEAT SET `pos` = (SELECT LOCATE(`needle`, `hackstack`, `pos` + `skip`)); SET `result` = `result` + 1; UNTIL `pos` = 0 END REPEAT; RETURN `result`; END; // delimiter ; Now you can do things like this: /blog/2013/mysql-datetime-indexes/ Fri, 06 Sep 2013 00:00:00 +0000 /blog/2013/mysql-datetime-indexes/ MySQL has many odd quirks. One that bit us recently was this: regression: >=mysql-5.4 utf8 collations are marked as not ascii compatible When using the utf8_unicode_ci collation, datetime column comparisons against strings would ignore any indexes. The lack of working indexes obviously caused huge performance degradation. Our bug report was ignored in Februari. Apparently a new bug was opened in March: Datetime field comparisons do not work properly with utf8_unicode_ci collation /blog/2013/thunderbird-postfix-dkim-invalid-body-hash/ Wed, 04 Sep 2013 00:00:00 +0000 /blog/2013/thunderbird-postfix-dkim-invalid-body-hash/ Mozilla Thunderbird uses an odd max line length of 999 + CRLF: 1001 characters. When using DKIM preprocessing, this can result in DKIM validation failure. To reproduce, we would send a mail that didn’t wrap well with line lengths in excess of 999 characters. Like this mail with 1000 'x' characters: Thunderbird splits that into 999 times 'x', and CRLF and another line with a single 'x'. However, that first line gets split again. /blog/2013/virtualenv-pil-pillow-mess/ Fri, 30 Aug 2013 00:00:00 +0000 /blog/2013/virtualenv-pil-pillow-mess/ Numerous articles have been written about why you want to install Pillow instead of PIL to get the Python Imaging tools. Like Problems with PIL? Use Pillow instead! (Find more by searching for “IOError: decoder zip not available”.) This note concerns something more insidious: a seemingly broken Pillow installation after the removal of PIL. ~$ mkvirtualenv piltest (piltest)~$ pip install PIL (piltest)~$ pip freeze | grep -i pil PIL==1.1.7 Now this should work: /blog/2013/two-node-mariadb-galera-cluster-with-arbiter/ Thu, 18 Jul 2013 14:01:09 +0000 /blog/2013/two-node-mariadb-galera-cluster-with-arbiter/ Caveats arbiter - 10.10.10.1 node1 - 10.10.10.10 node1 - 10.10.10.11 Installing the nodes update /etc/hosts to make sure all the machines are resolvable, perform this on all 3 hosts (2 active mysql nodes and the machine the arbiter resides on). Update the hostnames and addresses accordingly. $ cat << EOF >> /etc/hosts 10.10.10.1 arbiter 10.10.10.10 node1 10.10.10.11 node2 EOF Setup APT to use the MariaDB repositories /blog/2013/webserver-ssl-configuration-cheatsheet/ Mon, 15 Apr 2013 19:49:14 +0000 /blog/2013/webserver-ssl-configuration-cheatsheet/ Just a quick post with the most straight forward way to configure http webservers/proxies in regard to ssl certificate chains. I’m open for improvements if it improves readability or reduces linecount. :-) pound In /etc/pound/pound.cfg you only need one line in the HTTPS listener: ListenHTTPS Cert "/etc/pound/file-with-key-and-cert-chain.pem" apache2 SSLCertificateFile /path/to/file-with-key-and-cert-chain.pem SSLCertificateChainFile /path/to/file-with-key-and-cert-chain.pem /blog/2013/fixing-ssl-certificate-chains-and-verify-they-work-properly/ Mon, 15 Apr 2013 15:02:51 +0000 /blog/2013/fixing-ssl-certificate-chains-and-verify-they-work-properly/ Most browser trusted SSL certificates work with intermediate certificates nowadays. The CA only ships the root certificates for inclusion with the browsers and because they do this your certificate can’t be verified if you don’t include the intermediate certificates in your webserver configuration. chain order So you’re in the bottom in a chain of trust. Your certificate is signed by a certificate which is signed by another certificate and so on. /blog/2013/brief-dbase-backup-bonanza/ Mon, 15 Apr 2013 00:00:00 +0000 /blog/2013/brief-dbase-backup-bonanza/ Of course you do daily backups of your data. For your databases, you generally need a dump of the data for it to be useful. For your and my reference, here are a few database dump scripts. Make sure /var/backups is backed up daily. Observe that we keep extra backups around. I’ve found that the need to access an old database is far more common than accessing old files (from the backup storage). /blog/2013/probook-4510s-high-fan-speed/ Sun, 14 Apr 2013 00:00:00 +0000 /blog/2013/probook-4510s-high-fan-speed/ So, there have been numerous reports around the internet that the HP ProBook 4510s has fan speed issues. In my case, the following happens: Turn on the laptop: no problem. Using the laptop while on AC: no problem. Switch on the laptop from suspend mode without AC plugged in: problem! The fan goes into overdrive and stays there. It's related to the temperature measurements, somehow. According to the sensors(1) tool, it's the temp6 value that's off. /blog/2013/scan-for-new-hotplug-added-disks/ Thu, 11 Apr 2013 11:53:51 +0000 /blog/2013/scan-for-new-hotplug-added-disks/ If you add new disks to a virtual machine you don’t have to reboot to be able to use them. I assume this works the same for normal hotswappable disks as well but that just doesn’t happen too often these days. Didn’t test for all drivers/use cases but the example below is in a VMWARE environment. It has to be handled as scsi by the kernel obviously. check for current disks: /blog/2013/mysql-show-procedure-grant/ Fri, 05 Apr 2013 00:00:00 +0000 /blog/2013/mysql-show-procedure-grant/ How to I grant myself permissions to show MySQL FUNCTION and PROCEDURE bodies? It isn’t this: mysql> grant all privileges on procedure *.* to walter; ERROR 1144 (42000): Illegal GRANT/REVOKE command; please consult the manual to see which privileges can be used But it is this: mysql> grant select on mysql.proc to walter; Query OK, 0 rows affected (0.00 sec) /blog/2013/python-twisted-exec-environment/ Thu, 21 Mar 2013 00:00:00 +0000 /blog/2013/python-twisted-exec-environment/ Does Python Twisted pass the parent environment to child processes? By default no, but if you pass env=None then it does. Ergo, default is env={}. Let’s build a quick example. For those unfamiliar with twisted this may provide a quick intro. import os from twisted.internet import protocol, reactor, utils This is what we’re going to “run”: proc = ['/bin/sh', '-c', 'export'] #kwargs = {'env': None} kwargs = {} A quick way to show output and end after the 2 runs. /blog/2013/python-temporarily-blocking-signals/ Wed, 20 Mar 2013 00:00:00 +0000 /blog/2013/python-temporarily-blocking-signals/ There is no way to “block” signals temporarily from critical sections (since this is not supported by all Unix flavors). Says the python signal module manual. But I’m using Linux, where it is possible to block signals, so I don’t think that limitation applies. And it doesn’t. pysigset takes the burden off calling sigprocmask(2) through ctypes and provides a “pythonic” interface to temporarily blocking signals in python. from signal import SIGINT, SIGTERM from pysigset import suspended_signals with suspended_signals(SIGINT, SIGTERM): # Signals are blocked here. /blog/2013/darwin-sed-limited-regular-expressions/ Sat, 16 Mar 2013 00:00:00 +0000 /blog/2013/darwin-sed-limited-regular-expressions/ For someone who is used to using only GNU sed(1) it may come as a surprise that some of the metacharacters don’t work with sed on other OS’es. Specifically: sed on Darwin (BSD) does not grok \+, \| and \? when in basic regular expression mode (the default). Switching to extended (modern) regular expression isn’t a good idea if you’re aiming for compatibility, because the option -E differs from the GNU sed option -r. /blog/2013/callerid-in-rpid-opensips-kamailio/ Tue, 12 Mar 2013 00:00:00 +0000 /blog/2013/callerid-in-rpid-opensips-kamailio/ For reuse, an OpenSIPS/Kamailio snippet to translate commonly used SIP (VoIP) phone Caller-ID (CLI) headers into a single one (Remote-Party-ID). It tries these headers in order, to do a best guess of what the caller wants: P-Asserted-Identity (with Privacy) P-Preferred-Identity (with Privacy) Remote-Party-ID From Of course you’ll have to run the found CLI against an allow list, but this code expects that to be done on the next hop. route[sub_cli_as_rpid] { $var(tmp_name) = ""; # (nothing) $var(tmp_num) = "Anonymous"; # (unknown) $var(tmp_clir) = 0; # PAI/PPI-privacy if (is_present_hf("Privacy")) { if (! /blog/2013/more-or-less-useless-tips-and-tricks-2/ Fri, 18 Jan 2013 00:00:00 +0000 /blog/2013/more-or-less-useless-tips-and-tricks-2/ More or less useless/useful tips and tricks, bundled together. They weren’t worthy of a box div on their own. I gave them only a li each. kill -WINCH $$ — when your terminal is messed up where the row moves up one line before you’ve reached the line-length ($COLUMNS): a SIGWINCH signal to the current shell will make everything alright again. hash -r — you moved applications around in your $PATH and bash claims that some applications don’t exist in your $PATH even though you (and ls) know that they do: /blog/2013/upgrading-ubuntu-command-line/ Tue, 15 Jan 2013 13:03:27 +0000 /blog/2013/upgrading-ubuntu-command-line/ When upgrading Ubuntu command line, many people probably still change the release name in the sources.list and update && dist-upgrade. Although is still works fine, Ubuntu also provides extras on this front. Upgrade the new way: sudo do-release-upgrade Whats the benefit? It provides things like a temporary SSH server and probably does other sanity checks to increase the chances of an successful upgrade and eases the troubleshooting if things go wrong. /blog/2013/etckeeper/ Tue, 15 Jan 2013 13:01:35 +0000 /blog/2013/etckeeper/ One useful tool we recently started using is etckeeper. It provides version control over your /etc/ directory, which may prove quite useful when you maintain system in co-op way with your customers. Etckeeper also comes with hooks for apt, so even if you use it terribly it does give you an insightful history of when which package was installed. Installation and setup in Ubuntu: apt-get install git etckeeper # change VCS in /etc/etckeeper/etckeeper. /blog/2012/canon-mf8350-driver-hell/ Wed, 12 Dec 2012 00:00:00 +0000 /blog/2012/canon-mf8350-driver-hell/ Building Canon MF8350Cdn (and other) CUPS drivers for Linux Debian and/or Ubuntu on amd64 is still a pain in the behind. Problems encountered during the installation, include: Regular make installed stuff in different places but forgot many parts. Solution: use dpkg-buildpackage The libtool copied wasn’t able to build for shared libs. Solution: remove --enable-shared/--disable-shared command line options. amd64 had to be added to the architecture targets. A shell script had to lose a bashism. /blog/2012/easy-certificate-generation-testing/ Thu, 25 Oct 2012 00:00:00 +0000 /blog/2012/easy-certificate-generation-testing/ If I’m going to be requesting SSL certificates more often, I’d better automate the process a bit. The result: easycert.sh (view) Possible invocation styles: $ easycert.sh -h Usage: easycert.sh -c NL -l Groningen -o OSSO\ B.V. -e info@osso.nl osso.nl Usage: easycert.sh osso.nl "/C=NL/L=Groningen/O=OSSO B.V./CN=osso.nl/" Usage: easycert.sh -T www.osso.nl 443 Generating a key and certificate: $ easycert.sh -o "My Company" mycompany.com Subject: /C=NL/L=Groningen/O=My Company/CN=mycompany.com/emailAddress=info@osso.nl Enter to proceed... Generating RSA private key, 4096 bit long modulus . /blog/2012/setuid-seteuid-uid-euid/ Fri, 17 Aug 2012 00:00:00 +0000 /blog/2012/setuid-seteuid-uid-euid/ So, what is the difference between uid and euid and the setuid and seteuid calls? Hao Chen, David Wagner and Drew Dean wrote an excellent paper called Setuid Demystified. It explains all the ins and outs. To answer the question, we need only parts of the article. Let the quoting begin. Each process has three user IDs: the real user ID (real uid, or ruid), the effective user ID (effective uid, or euid), and the saved user ID (saved uid, or suid). /blog/2012/postfix-submission-smtpd-client-restrictions-sleep/ Mon, 16 Jul 2012 00:00:00 +0000 /blog/2012/postfix-submission-smtpd-client-restrictions-sleep/ After tweaking my postfix configuration, I apparently broke submission on port 587. Every time I connected, I immediately got: 554 5.7.1 <my.host.name[1.2.3.4]>: Client host rejected: Access denied That’s strange. Postfix is supposed to reject unauthenticated clients only in master.cf: submission inet n - n - - smtpd -o smtpd_tls_security_level=encrypt -o smtpd_tls_auth_only=yes -o smtpd_client_restrictions=permit_sasl_authenticated,reject But if it rejects me at connect time, I don’t have a chance to identify myself. /blog/2012/new-ipython-old-django/ Mon, 09 Jul 2012 00:00:00 +0000 /blog/2012/new-ipython-old-django/ IPython after version 0.10 is not friends with older Django (e.g. 1.1.x) versions anymore. shell = IPython.Shell.IPShell(argv=[]) AttributeError: 'module' object has no attribute 'Shell' This is fixed in newer Django’s, but this isn’t backported. Here, a patch. (Nothing more than a diff between the old and the new Django version.) --- django/core/management/commands/shell.py 2012-03-28 16:10:28.000000000 +0200 +++ django/core/management/commands/shell.py 2012-01-24 10:27:50.405338739 +0100 @@ -8,9 +8,38 @@ help='Tells Django to use plain Python, not IPython. /blog/2012/serialize-json-date-microsoft-extension/ Thu, 21 Jun 2012 00:00:00 +0000 /blog/2012/serialize-json-date-microsoft-extension/ Bertrand Le Roy describes how Microsoft added a Date object extension to JSON in a compatible fashion to implement serialization and serialization of timezone agnostic datetimes. Our current approach is using a small loophole in the JSON specs. In a JSON string literal, you may (or may not) escape some characters. Among those characters, weirdly enough, there is the slash character ('/'). This is weird because there actually is no reason that I can think of why you’d want to do that. /blog/2012/gigaset-n300a-respect-srv/ Thu, 14 Jun 2012 00:00:00 +0000 /blog/2012/gigaset-n300a-respect-srv/ Does the Siemens Gigaset N300A handle SRV records? Yes it does.. but.. Let’s look at a bit of Gigaset DNS traffic: 09:24:26.059850 IP gigaset.local.32978 > nameserver.local.53: 50512+ NAPTR? gigaset.voip.example.com. (28) 09:24:26.061552 IP nameserver.local.53 > gigaset.local.32978: 50512 0/1/0 (87) 09:24:26.063894 IP gigaset.local.32978 > nameserver.local.53: 25738+ SRV? _sip._udp.gigaset.voip.example.com. (38) 09:24:26.064445 IP nameserver.local.53 > gigaset.local.32978: 25738 2/3/2 SRV proxy1.voip.example.com.:5060 10 0, SRV proxy2.voip.example.com.:5060 20 0 (231) 09:24:26.066939 IP gigaset.local.32978 > nameserver.local.53: 22676+ SRV? _sip. /blog/2012/thunderbird-mailing-list-reply/ Wed, 13 Jun 2012 00:00:00 +0000 /blog/2012/thunderbird-mailing-list-reply/ How do you reply to a mailing list post when you do not have the mail in your INBOX? With Thunderbird it is easy enough, as long as you know how. How it works Mail threads are matched by comparing the In-Reply-To header with the Message-ID. Here’s an example from an Asterisk project reviewboard mailing: Date: Fri, 08 Jun 2012 08:08:42 -0000 Message-ID: <20120608080842.8103.32910@hotblack.digium.com> In-Reply-To: <20120607143847.27705.11556@hotblack.digium.com> References: <20120607143847.27705.11556@hotblack.digium.com> Subject: Re: [asterisk-dev] [Code Review] Fix issue of unrecognized inbound ACK when Asterisk responds to an INVITE with a 481 Every e-mail message has a globally unique message identifier. /blog/2012/python-base85-ascii85/ Wed, 06 Jun 2012 00:00:00 +0000 /blog/2012/python-base85-ascii85/ So python’s base64 does not have a b85decode function? Adobe uses it the ASCII-85 encoding in PDF and PostScript files. Here is a quick and dirty one hacked together. See the wikipedia article for the ASCII-85 (base85) specs. Prologue; we only need sys to print a warning. # vim: set ts=8 sw=4 sts=4 et ai: # Example base85 decoder, Walter Doekes 2012 import sys Split the data up into 5-character chunks; 5 characters encode 4 octets. /blog/2012/libreoffice-spreadsheet-l10n-date-format/ Mon, 04 Jun 2012 00:00:00 +0000 /blog/2012/libreoffice-spreadsheet-l10n-date-format/ For LibreOffice’s oocalc on latest Ubuntu (libreoffice-base 1:3.5.2-2ubuntu1 to take the locale settings into account for the date types, the LC_CTYPE needs to be set. $ LC_CTYPE=en_US.UTF-8 oocalc This causes a date input of 31-01-2012 to not get parsed as a date. $ LC_CTYPE=nl_NL.UTF-8 oocalc This causes the same input of 31-01-2012 to get properly understood as the DD-MM-YYYY format. That does not make sense. LC_CTYPE should be used for character classification, collation and case conversion. /blog/2012/ubuntu-sip-video-softphone/ Mon, 14 May 2012 00:00:00 +0000 /blog/2012/ubuntu-sip-video-softphone/ So, I wanted to test video support with Asterisk. That was easier said than done, because the SIP softphones that ship with Ubuntu don’t all do what they promise. This was done on a setup that works for numerous hardphones and PBXs out there. Looking through the registration list at any given time reveals at least 40+ different user agents and a large multiple of that if you take the different versions into account. /blog/2012/django-mark-safe-translatables/ Fri, 04 May 2012 00:00:00 +0000 /blog/2012/django-mark-safe-translatables/ Look at this snippet of Django code in models.py, and in particular the help_text bit: from django.db import models from django.utils.translation import ugettext_lazy as _ from django.utils.safestring import mark_safe class MyModel(models.Model): my_field = models.CharField(max_length=123, help_text=mark_safe(_('Some <b>help</b> text.'))) For those unfamiliar with Django. A quick run-down: The definition of MyModel creates a mapping between the MyModel class and a underlying app_mymodel table in a database. That table will consist of two columns: id, an automatic integer as primary key (created by default), and my_field, a varchar/text field of at most 123 characters. /blog/2012/ipython-classic-mode-precise-pangolin/ Mon, 30 Apr 2012 00:00:00 +0000 /blog/2012/ipython-classic-mode-precise-pangolin/ The Ubuntu do-release-upgrade broke my ipython classic mode. The ipython package was upgraded, and apparently the configuration parser was changed. In bash, I want colors to help me find the beginning and end of output — see this bug report for others agreeing with me that the derogatory comment about “focus should be on the output, not on the prompt” in the skeleton .bashrc is is retared, but I diverge — in ipython, I just want to see the nice >>> blocks that I’m used to and no extra spaces. /blog/2012/safe-asterisk-init-d/ Thu, 12 Apr 2012 00:00:00 +0000 /blog/2012/safe-asterisk-init-d/ An init.d script to stop and start safe_asterisk started asterisk. If asterisk is not stopped in 5 seconds, it is forcibly killed. safe_asterisk-init.d (view) # wget http://wjd.nu/files/2012/04/safe_asterisk-init.d -O/etc/init.d/asterisk ; chmod 755 /etc/init.d/asterisk Also possibly useful, the changes I made to safe_asterisk on a machine where: there wasn’t a tty left to spam output on, root is configured in /etc/aliases to a sane destination, /var/spool/asterisk is the asterisk user homedir anyway, and, attempting to set maxfiles to the highest value possible, wasn’t allowed. /blog/2012/sip-digest-calculation/ Thu, 29 Mar 2012 00:00:00 +0000 /blog/2012/sip-digest-calculation/ Every one in a while, I see an unexpected 403 response to a SIP client’s REGISTER request. Thusfar the digest response calculation has never been wrong, but it feels good to get that check out of the way and move on to other possible causes. For your enjoyment and mine, a Bourne-shell compatible shell script that calculates (qop-less) Digest authentication responses. Download: hahacalc.sh (view) $ hahacalc Usage: hahacalc.sh USERNAME REALM METHOD DIGESTURI NONCE [PASSWORD] [COMPARE] Or: hahacalc. /blog/2012/python-virtualenv-global-site-packages/ Tue, 13 Mar 2012 00:00:00 +0000 /blog/2012/python-virtualenv-global-site-packages/ If you’re switching from Ubuntu Oneiric to Ubuntu Precise and you’re using python-virtualenv, you might be in for a surprise: The default access to the global site-packages modules is reversed between virtualenv 1.6.x and 1.7. When you were used to finding your apt-get installed python modules like python-mysqldb and python-psycopg2 in your new virtualenv environment, now they’re suddenly unavailable. The culprit: --no-site-packages Ignored (the default). Don´t give access to the global site-packages modules to the virtual environment. /blog/2012/mysql-replicating-repair-table/ Mon, 27 Feb 2012 00:00:00 +0000 /blog/2012/mysql-replicating-repair-table/ From the MySQL 5.1 manual: 15.4.1.16. Replication and REPAIR TABLE When used on a corrupted or otherwise damaged table, it is possible for the REPAIR TABLE statement to delete rows that cannot be recovered. However, any such modifications of table data performed by this statement are not replicated, which can cause master and slave to lose synchronization. For this reason, in the event that a table on the master becomes damaged and you use REPAIR TABLE to repair it, you should first stop replication (if it is still running) before using REPAIR TABLE, then afterward compare the master’s and slave’s copies of the table and be prepared to correct any discrepancies manually, before restarting replication. /blog/2012/indirect-scp-bypass-remote-firewall-rules/ Fri, 17 Feb 2012 00:00:00 +0000 /blog/2012/indirect-scp-bypass-remote-firewall-rules/ Suppose I’m on machine DESKTOP and I want to copy files from server APPLE to server BANANA. DESKTOP has access to both, but firewalls and/or missing ssh keys prevent direct access between APPLE and BANANA. Regular scp(1) will now fail. It will attempt to do a direct copy and then give up. This is where this indirect scp wrapper (view) comes in: First, it tries to do the direct copy. /blog/2012/mysql-replication-relay-log-pos/ Wed, 25 Jan 2012 00:00:00 +0000 /blog/2012/mysql-replication-relay-log-pos/ So, hardware trouble caused a VPS to go down. This VPS was running a MySQL server in a slave setup. Not surprisingly, the unclean shutdown broke succesful slaving. There are several possibly causes for slave setup breakage. This time it was the local relay log file (mysqld-relay-bin.xxxx) that was out of sync. SHOW SLAVE STATUS\G looked like this: ... Master_Log_File: mysql-bin.001814 <-- remote/master file (IO thread) Read_Master_Log_Pos: 33453535 <-- remote/master pos (IO thread) Relay_Log_File: mysqld-relay-bin. /blog/2011/mysql-slow-queries-sample/ Mon, 12 Dec 2011 00:00:00 +0000 /blog/2011/mysql-slow-queries-sample/ Sometimes you’re in a situation where you know that a database is more heavily loaded than it should be. Time to figure out which queries are stressing it the most. The standard thing to do with a MySQL database would be to enable query logging with general_log_file. Or, to get only slow queries and those not using indexes, the log_slow_queries. But, if this is a mission critical and heavily loaded database, adding expensive logging may be just enough to give it that final push to become overloaded. /blog/2011/postgres-alter-column-look-closer/ Sun, 11 Dec 2011 00:00:00 +0000 /blog/2011/postgres-alter-column-look-closer/ Just now, I tried to convert an integer column in a PostgreSQL database to one of type VARCHAR. I knew you had to do an explicit cast, so I was a bit stumped that I still wasn’t allowed to perform the ALTER TABLE. mydb=> ALTER TABLE mytable ALTER COLUMN mycolumn TYPE VARCHAR(31) USING mycolumn::text; ERROR: operator does not exist: character varying >= integer HINT: No operator matches the given name and argument type(s). /blog/2011/fixing-symptoms-not-problems/ Fri, 02 Dec 2011 00:00:00 +0000 /blog/2011/fixing-symptoms-not-problems/ Some people seem to think that fixing the symptom is fixing the problem. import random def return_one_of(list): return list[random.randint(0, len(list))] def say_something(): try: print return_one_of(["Hello World!", "Hi!", "How you doin'?"]) except: return say_something() say_something() Gah! This is obviously an example, but there are people who do this and claim to have “fixed the problem”. Let me reiterate: the fact that your code does not raise any exceptions does NOT mean that it is not broken code! /blog/2011/django-mongodb-manage-dbshell/ Tue, 15 Nov 2011 00:00:00 +0000 /blog/2011/django-mongodb-manage-dbshell/ The current django-mongodb-engine doesn’t seem to ship with a working manage dbshell command yet. Right now it returns this: $ ./manage.py dbshell ... File "/home/walter/.virtualenvs/myproject/lib/python2.6/site-packages/django/core/management/commands/dbshell.py", line 21, in handle connection.client.runshell() File "/home/walter/.virtualenvs/myproject/lib/python2.6/site-packages/django_mongodb_engine/base.py", line 108, in __getattr__ raise AttributeError(attr) AttributeError: client The fix is simple, patch your django_mongodb_engine with this: --- django_mongodb_engine/base.py.orig 2011-11-15 11:53:47.000000000 +0100 +++ django_mongodb_engine/base.py 2011-11-15 11:54:07.000000000 +0100 @@ -7,6 +7,7 @@ from pymongo.connection import Connection from pymongo.collection import Collection +from . /blog/2011/certificate-verify-fail-crt-bundle/ Thu, 20 Oct 2011 00:00:00 +0000 /blog/2011/certificate-verify-fail-crt-bundle/ So. SSL certificates are still black magic to me. Especially when they cause trouble. Like when one of the sysadmins has forgotten to add the certificate bundle to the apache2 config. Then you get stuff like this: $ hg pull -u abort: error: _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed Most web browsers do not notice this as they already have the intermediate CA files, but /etc/ssl/certs/ca-certificates.crt seemingly doesn’t. The problem in this case was not that I was missing any certificates locally. /blog/2011/backtrace-without-debugger/ Mon, 12 Sep 2011 00:00:00 +0000 /blog/2011/backtrace-without-debugger/ You may not always have gdb(1) at hand. Here are a couple of other options at your disposal. #1 Use addr2line to get the crash location $ cat badmem.c void function_c() { int *i = (int*)0xdeadbeef; *i = 123; } // <-- line 1 void function_b() { function_c(); } void function_a() { function_b(); } int main() { function_a(); return 0; } $ gcc -g badmem.c -o badmem $ ./badmem Segmentation fault No core dump? /blog/2011/gdb-backtrace-running-process/ Tue, 06 Sep 2011 00:00:00 +0000 /blog/2011/gdb-backtrace-running-process/ Sometimes you want a backtrace or a core dump from a process that you do not want to stall. This could concern a multithreaded application of which some threads are still doing important work (like handling customer calls). Firing up gdb would halt the process for as long as you’re getting info, and raising a SIGABRT to get a core dump has the negative side-effect of killing the process. Neither is acceptable in a production environment. /blog/2011/sip-six-digit-port-number-invalid/ Sun, 04 Sep 2011 00:00:00 +0000 /blog/2011/sip-six-digit-port-number-invalid/ While looking through opensips logs of a customer, sometimes we would see the following: ERROR:core:parse_via: invalid port number <110900> ERROR:core:parse_via: <SIP/2.0/UDP 1.2.3.4:110900;branch=z9hG4bKabcdef... ERROR:core:parse_via: parsed so far:<SIP/2.0/UDP 1.2.3.4:110900;branch=z9hG4bKabcdef... ERROR:core:get_hdr_field: bad via As you can see, that 6-digit port number is invalid. Furthermore, when sniffing this traffic, we could see that the port number is almost right. The traffic came from port 11090 (one less zero at the end). Not only the Via header, the Contact header too had the extra appended zero. /blog/2011/openswan-klips-install-modules/ Wed, 13 Jul 2011 00:00:00 +0000 /blog/2011/openswan-klips-install-modules/ If you want to be able to sniff your IPsec traffic with OpenSwan, you’ll need to get KLIPS instead of the default NETKEY IPsec protocol stack. Installing that on Ubuntu/Karmic should be a matter of: ~# apt-get install openswan-modules-source ~# cd /usr/src /usr/src# tar jxvf openswan-modules.tar.bz2 /usr/src# cd modules/openswan /usr/src/modules/openswan# make KERNELSRC=/lib/modules/`uname -r`/build module module_install But it’s not. Right now, we’re running the default Linux kernel 2.6.31-23-server on this Karmic machine. /blog/2011/mocp-random-enqueue/ Tue, 12 Jul 2011 00:00:00 +0000 /blog/2011/mocp-random-enqueue/ After disk failure on our company music server, I lost my enqueue-some-random-music-script. That shan’t happen again. So here, for my own enjoyment: autoenq.sh #!/bin/sh enqueue_app="mocp -a" music_glob="*.mp3" music_path="`dirname "$0"`" list_path="$music_path/.autoenq.list" if [ "$*" = "-c" ]; then # Create list of all files find . -type f -iname "$music_glob" > "$list_path.tmp" 2>/dev/null # no lost+found # Create list of all dirs that have files cat "$list_path.tmp" | sed -e 's/\/[^\/]*$//' | sort | uniq > "$list_path" exit 0 fi args="`echo "$*" | sed -e "s/['\\\\]//g"`" # no backslashes and single quotes please args="`echo "$args" | sed -e 's/[[:blank:]]\+/. /blog/2011/asterisk-dialplan-peculiarities-regex-with-eqtilde/ Mon, 04 Jul 2011 00:00:00 +0000 /blog/2011/asterisk-dialplan-peculiarities-regex-with-eqtilde/ In the Asterisk PBX dialplan, expressions can be formed using the $[...] syntax. Addition, subtraction, comparison and so on are defined. As is a regex operator: =~ Unfortunately, the documentation about the details of the implementation is hard to find. Here, a breakdown of my findings: static struct val * op_eqtilde is defined in main/ast_expr2.y It uses the REG_EXTENDED flag when calling regcomp: so extended regular expression syntax is used. /blog/2011/executing-remote-command-ssh-extra-escaping/ Thu, 09 Jun 2011 00:00:00 +0000 /blog/2011/executing-remote-command-ssh-extra-escaping/ If you use ssh to run commands remotely, you may have run into the problem that you need an extra layer of escaping. Let’s say you have application myapp that for some reason only runs on host myserver. If you have functional ssh keys to log onto myserver it can be helpful to create a myapp wrapper on your desktop. After all, this: $ myapp myargs … is far more convenient than doing this: /blog/2011/django-query-expression-negate/ Wed, 08 Jun 2011 00:00:00 +0000 /blog/2011/django-query-expression-negate/ Suppose you have an is_enabled boolean in your Django model. class Rule(models.Model): is_enabled = models.BooleanField(blank=True) # other exciting fields here And now imagine you want to negate the is_enabled values. Something you would easily do in SQL, with: UPDATE myapp_rule SET is_enabled = NOT is_enabled; The Django F-syntax is nice, and looks like it should be up for the task. Let’s sum up a couple of attempts: Rule.objects.update(is_enabled=(not F('is_enabled'))) No! You get this: /blog/2011/mysql-issue-warnings-cron/ Mon, 16 May 2011 00:00:00 +0000 /blog/2011/mysql-issue-warnings-cron/ I’ve previously written about MySQL pain in the behind issues involving views with SECURITY DEFINER and bad client collation selection. For the former problem, I wrote a script that you could call periodically to warn you of potential problems with your views. Now I’ve extended it to warn you about collation issues as well. Put warn-mysql-issues.sh (view) in your cron tab and run it periodically. It’ll save you from production-time errors that you get when attempting to compare a string of one collation with another. /blog/2011/linux-canon-mf8350-printer-driver/ Thu, 28 Apr 2011 00:00:00 +0000 /blog/2011/linux-canon-mf8350-printer-driver/ Getting printer drivers for the Canon MF8350 to work under Ubuntu is a big pain in the behind. (Installation using custom scripts that abuse both /usr/lib and /usr/local/lib and ultimately fail to compile for obscure reasons.) My colleague found that the easiest way to get it to work, was converting the RPM to DEB using alien(1). For your enjoyment, here are the two debian packages needed for Ubuntu 10.04 (amd64): /blog/2011/diff-memory-exhausted-udiff/ Thu, 07 Apr 2011 00:00:00 +0000 /blog/2011/diff-memory-exhausted-udiff/ Sometimes when I’m unsure what a button in a database-driven application does, I simply click it and check the differences in the database before and after the click. If the database dumped is (somewhat) sorted and with one line per row (for MySQL use –skip-extended-insert), this can be an easy method of verifying that your application action does exactly what you expect and nothing more. Create a pre.sql before the click, click the button and watch the action happen. /blog/2011/pcap-capture-fragments-udp/ Thu, 31 Mar 2011 00:00:00 +0000 /blog/2011/pcap-capture-fragments-udp/ When dealing with internet protocols that operate on top of UDP, fragmenting suddenly becomes a lot less uncommon. Normally, you would only encounter fragments on TCP connections when the MTU on the sending host is larger then the MTU in any of the next hops. Hosts usually attempt to avoid fragmentation for obvious reasons. (Inefficiëncy, extra reassembly work.) For connectionless UDP packets this is a different matter. Protocols over UDP expect packets to be single entities. /blog/2011/no-sql-security-definer-please/ Tue, 15 Mar 2011 00:00:00 +0000 /blog/2011/no-sql-security-definer-please/ Have you ever had it happen that you removed a MySQL user and suddently parts of your application stopped working? Not because you removed the user that was connecting, but because you removed the user that defined the particular view or function that you were using. I have, and it was quite stressful ;-) We moved a slave machine to a different IP, I updated the mysql.user host column, and BAM, the application running on the master mysql stopped working. /blog/2011/build-error-unixodbc-debian-squeeze/ Thu, 10 Mar 2011 00:00:00 +0000 /blog/2011/build-error-unixodbc-debian-squeeze/ Building unixodbc-2.2.14p2 on debian/squeeze which you just fetched through apt-get source unixodbc. ~/src/unixodbc-2.2.14p2$ ./configure ... works fine ... ~/src/unixodbc-2.2.14p2$ make ... make[1]: Entering directory `/home/walter/src/unixodbc-2.2.14p2/odbcinst' make[1]: *** No rule to make target `libltdl/libltdlc.la', needed by `libodbcinst.la'. Stop. make[1]: Leaving directory `/home/walter/src/unixodbc-2.2.14p2/odbcinst' make: *** [install-recursive] Error 1 The fix: add top_build_prefix to the environment. ~/src/unixodbc-2.2.14p2$ top_build_prefix=`pwd`/ make ... success! /blog/2011/asterisk-nat-keepalive-round-robin-dns/ Sun, 27 Feb 2011 00:00:00 +0000 /blog/2011/asterisk-nat-keepalive-round-robin-dns/ Current Asterisk (telephony software) version 1.6.2.x (and probably 1.4 and 1.8), has an odd quirk with the qualify option. The qualify option enables a function that checks the response times of the SIP peer. By default, it sends an OPTIONS SIP packet every 60 seconds. The quirk here, is that it sends the packet to the first A-record resolved for this peers hostname at startup (or sip reload). This works fine in most cases when the host has only one A-record. /blog/2011/pruning-old-data-mysql-csv/ Fri, 21 Jan 2011 00:00:00 +0000 /blog/2011/pruning-old-data-mysql-csv/ It is not uncommon to have a database with records that just accumulate and accumulate over time. Think of log files, telephony billing records, traffic data and so forth. The chances that you’ll ever need this data again are very slim. And letting your database grow indefinitely is not particularly smart. Time to prune! Two things you need to worry about while pruning your data: Throwing it all away without a backup doesn’t feel right. /blog/2011/port-forwarded-ssh-port-22/ Fri, 14 Jan 2011 00:00:00 +0000 /blog/2011/port-forwarded-ssh-port-22/ Sometimes you need to access your source code repository-server from a new server which hasn’t been whitelisted yet. You check out the source over port 22, but you can’t, because traffic from new-server to 22 is rejected. The quick solution, you know this, is ssh port forwarding. Connect to old-server and forward connections to repository-server from there. $ ssh old-server -L1234:repository-server:22 That works. For mercurial, at least. $ hg clone ssh://walter@localhost:1234//srv/hg/myproject myproject walter@localhost's password: requesting all changes . /blog/2010/faxable-images-asterisk-pbx/ Tue, 30 Nov 2010 00:00:00 +0000 /blog/2010/faxable-images-asterisk-pbx/ Because creating images that the open source PBX Asterisk(tm) will properly fax using the SendFAX() application, was a pain in the ass, I’d like to share my findings. The HOWTO for creating TIFF images that are laid out so the spandsp fax back-end in asterisk, is embedded in the images2fax.sh shell script, below. In short, you need to have images of 1728x2292, in 2 colors, with the correct DPI (204x196) and the right compression. /blog/2010/nat-switch-external-source-port/ Tue, 23 Nov 2010 00:00:00 +0000 /blog/2010/nat-switch-external-source-port/ When reproducing an issue with IP phones speaking SIP, I ran into the question of how to switch source port on my linux NAT router. The problem the clients were having, were a result of a failing (or reset) NAT-gateway. The NAT-gateway would change the external source port mid-dialog (some SIP dialogs can persist for quite a long time). So, how do you go about switching source port on an UDP connection on your Linux NAT router without resetting it (or disturbing anyone else using it)? /blog/2010/sip-totag-grandstream-register/ Mon, 30 Aug 2010 00:00:00 +0000 /blog/2010/sip-totag-grandstream-register/ SIP Question: The Grandstream GXP2000 1.1.2.23 sends SIP REGISTER requests with a To tag. Is my proxy wrong in refusing the request? REGISTER sip:server SIP/2.0 Via: SIP/2.0/UDP 1.2.3.4:5074;branch=z9hG4bK0b90873d634698eb From: "phone 123" <sip:123@server>;tag=c29eb9104c6a5a86 To: <sip:123@server>;tag=as77984b6 Contact: <sip:123@1.2.3.4:5074;transport=udp> Supported: path Authorization: Digest username="123", realm="server", algorithm=MD5, uri="sip:server", nonce="0997652c", response="3b91afb768c11ae0a0405e1bed41bc23" Call-ID: 3033205eb0b5d203@192.168.4.117 CSeq: 56349 REGISTER Expires: 3600 User-Agent: Grandstream GXP2000 1.1.2.23 Max-Forwards: 70 Allow: INVITE,ACK,CANCEL,BYE,NOTIFY,REFER,OPTIONS,INFO,SUBSCRIBE,UPDATE,PRACK,MESSAGE Content-Length: 0 Answer: no, the proxy is right. RFC 3261 says this about it. /blog/2010/nxclient-locale-passing/ Wed, 04 Aug 2010 00:00:00 +0000 /blog/2010/nxclient-locale-passing/ So, I achieved victory on getting the compose key to work in the NX session. On to get a proper English language setting on our Terminal Server. The configuration suffers from two problems: (1) /etc/environment had values set (LANG=nl_NL.UTF-8 and LANGUAGE=nl_NL:nl). (2) nxssh does not pass the LANG/LC_* environment variables. If I were to remove the /etc/environment variables and configure everything like in a previous post of mine, everyone gets the POSIX locale (nxssh doesn’t pass anything). /blog/2010/altgr-nxclient-compose-key/ Tue, 03 Aug 2010 00:00:00 +0000 /blog/2010/altgr-nxclient-compose-key/ Like various reports on the internet suggest, the AltGr compose key doesn’t work properly or not at all from an NXClient connected to an NXServer (FreeNX in my case). Note that this is a different issue from the one where Alt_R (and Super_L, Super_R, Ctrl_R en Menu) remains pressed after which no normal typing is possible. That issue is described in Alt Gr keeps stuck and involves a new int sendKey = 0; in nxagent that should be reverted. /blog/2010/python2-6-features-python2-5/ Sun, 20 Jun 2010 00:00:00 +0000 /blog/2010/python2-6-features-python2-5/ Today I'll show you some quick and dirty python2.5 compatibility fixes. Of course you're developing on python2.6 or even python3.x, but your customer still lives in the dark ages. Here are two fixes that might come in handy. ImportError: No module named ssl Falling back to python2.5 socket.SSL if there is no python2.6 ssl through a small wrap_socket replacement: import socket try: from ssl import wrap_socket except ImportError: class wrap_socket: def __init__(self, socket): self. /blog/2010/uninitialized-globals-c-language/ Wed, 16 Jun 2010 22:00:00 +0000 /blog/2010/uninitialized-globals-c-language/ As per the C language spec., uninitialized globals are initialized to zero (0). Nandu310 tells us why on his blog about the memory areas in the C language. Data segment: the data segment is to hold the value of those variables that need to be available throughout the life time of the program. […] There are two parts in this segment. The initialized data segment and uninitialized data segment. When variables are initialized to some value (other than 0 or which is different value), they are allocated in the initialized segment (. /blog/2010/unexpanded-tabs-mercurial-web-diff/ Fri, 07 May 2010 00:00:00 +0000 /blog/2010/unexpanded-tabs-mercurial-web-diff/ The hgweb mercurial web interface on current Debian/Squeeze (mercurial-common 1.5.1-2) lists tab characters as-is in the diff view. Every line is prefixed not only by a plus or a minus (unified diff), but also by file and line numbers. This can cause a tab (0x9) character to appear as a single space. This does not look nice. The following patch can be applied to expand the tab character so the intentation looks right again. /blog/2010/mysql-utf8-collation-conversion/ Fri, 09 Apr 2010 00:00:00 +0000 /blog/2010/mysql-utf8-collation-conversion/ On a clean MySQL install — on a Debian or Ubuntu system at least — the MySQL server gets the latin1_swedish_ci with latin1 character set by default. Every time you set up a new machine, you must remember to either fix the defaults in my.cnf config file or to supply character set and collation options when creating databases. Of course you’ll opt to set this by default in my.cnf first: