Blog

Blog

kubectl / broken terminal / ipython

Just now I ran into an IPython interpreter inside a Docker container inside Kubernetes misbehaving: After starting ipython inside a kubectl for a second time, IPython wouldn’t show the input prompt. It only showed the output prompt. Turns out it was due to the terminal settings. For some reasons, after logging out of kubectl exec, the next exec would get 0 rows and 0 columns; as if someone had run stty rows 0 on the terminal.

Read more

vimrc / debian stretch

In Debian/Stretch, the default ViM settings have been changed — for the worse, in my opinion. However, undoing the bad settings is not a matter of fixing them in your ~/.vimrc, because when that file is detected no defaults at all are set. The quick fix is to create a custom /etc/vim/vimrc.local file with the following settings: " Instead of auto-sourcing this afterwards, source it now. source $VIMRUNTIME/defaults.vim let g:skip_defaults_vim = 1 " Now we undo the "wrong" settings.

Read more

core file / docker image / auplink

A while, I’ve been looking at a stray /core file in some of our daily Xenial Docker images. Time to find out where it comes from. Tracing with a few well placed RUN ls -l /core || true, tells us that the dump appeared after a large RUN statement and not during one. Running gdb on the core revealed that it was a dump of auplink, a part of Docker. Opening the core on a Xenial machine with docker installed, showed the following backtrace:

Read more

Goodbye Unity

Welcome GNOME-SHELL After having gotten used to Unity on the Ubuntu desktop, with Ubuntu Artful it is time to say goodbye. When Ubuntu first added the Unity shell with just the sidebar with big buttons, in favor of the more traditional GNOME with its Windows 95 style interface, many were skeptical, me included. But removing the clutter was good, and I’ve happily worked with it for years. And you really don’t want to waste time tweaking your desktop away from the OS provided defaults.

Read more

ubuntu bionic / crashing gdm / eglgetdisplay

After upgrading from Ubuntu 17.10 to Ubuntu 18.04, and rebooting, the GNOME Display Manager (gdm) went into a restart loop. No promised speed gains. Instead, I got an unusable desktop. Being quick with CTRL+ALT+F3, I could enter my username and password in the text console after a couple attempts — the gdm restart would continuously steal console/tty focus — after which a sudo systemctl stop gdm was possible. This left me with a shell and plenty of time to examine the situation.

Read more

checking client ssl certificate / from python

A quick howto on checking SSL/TLS client certificates from Django/Python. Generally, when you want to use client certificates, you’ll let the HTTPS server (e.g. NGINX) do the certificate validation. For NGINX you’d add this config, and be done with it. # TLS server certificate config: ... # TLS client certificate config: ssl_verify_client on; # or 'optional' ssl_client_certificate /PATH/TO/my-ca.crt; ... location ... { ... # $ssl_client_s_dn contains: "/C=.../O=.../CN=...", where you're # generally interested in the CN-part (commonName) to identify the # "who".

Read more

docker application placement / paths

Where do you place the application inside the Docker image? In the past, when deploying Python/Django projects, we’d put them in /srv/django-projects/APPNAME on a (possibly shared) machine. The python-virtualenv that came with it, went into /srv/virtualenvs/APPNAME. Now that we’re dockerizing many projects, we don’t need the virtualenv (there is only one environment) and we don’t need the APPNAME either (there is only one application). So, where should we place the project?

Read more

ubuntu / goodbye unity / welcome gnome-shell

After having gotten used to Unity on the Ubuntu desktop, with Ubuntu Artful it is time to say goodbye. When Ubuntu first added the Unity shell with just the sidebar with big buttons, in favor of the more traditional GNOME with its Windows 95 style interface, many were skeptical, me included. But removing the clutter was good, and I’ve happily worked with it for years. And you really don’t want to waste time tweaking your desktop away from the OS provided defaults.

Read more

screen / wipes copy buffer

A mismash of bugs and workarounds causes the copy buffer (X selection) to get wiped some of the time in my recent desktop environment. And that in a seemingly unpredictable manner. The following bug is mostly in play: GNOME VTE soft reset wipes selection That bug causes: reset(1) to wipe the middle-mouse (primary) buffer (although this differs per system — could not put my finger on this); reset(1) to wipe the clipboard buffer, but only if the reset was called from window that originated the current clipboard buffer contents; GNU screen(1) initialization to misbehave as reset does, as described above — even through an ssh session — by wiping the buffer, if TERM=xterm-256color.

Read more

dovecot / roundcube / mail read error

Today we ran into a dovecot/imap crash on a Xenial box. The Dovecot in question was the patched dovecot-2.2.22. Due to an as of yet unexplained cause, reading mail through Thunderbird mail client worked fine, but when opening a message with Roundcube (webmail), most messages would give an odd error about a “message that could not be opened”. An IMAP trace of Roundcube revealed that the IMAP server stopped responding after the client A0004 UID FETCH command.

Read more

Meltdown & Spectre attacks

Information regarding Meltdown and Spectre attacks. Current state Waiting for software patch availability. Patched ubuntu kernels are available for testing. Updates: 20180104: Created blogpost 20180105: Added new information/links 20180105: Status update 20180108: Added information from Redhat about performance impact from patches. 20180108: Updated links list. 20180108: Status update Links https://spectreattack.com / https://meltdownattack.com (same site) https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-every-modern-processor-has-unfixable-security-flaws/ https://arstechnica.com/gadgets/2018/01/meltdown-and-spectre-heres-what-intel-apple-microsoft-others-are-doing-about-it/ https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown https://www.raspberrypi.org/blog/why-raspberry-pi-isnt-vulnerable-to-spectre-or-meltdown/ High level description CVE’s Spectre - CVE-2017-5715 Spectre - CVE-2017-5753 Meltdown - CVE-2017-5754 As described on: https://spectreattack.

Read more

Recap 2017

2017 ISO27001 certified + NEN7510 Ewout joined our team as an SRE More projects opensourced at github: https://github.com/ossobv Uradesign designed logo’s for several of our open source projects Started providing Kubernetes as a Service / Managed Kubernetes Lots of interesting stuff

Read more

Availability during holiday December 2017

Starting the 16th of December we are on leave. We return to the office on the 2nd of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability.

Read more

reprepro / multiversion / build recipe

We used to use reprepro (4.17) to manage our package repository. However, it did not support serving multiple versions of the same package. The Benjamin Drung version from GitHub/profitbricks/reprepro does. Here’s our recipe to build it. $ git clone -b 5.1.1-multiple-versions https://github.com/profitbricks/reprepro.git $ cd reprepro It lacks a couple of tags, so we’ll add some lightweight ones. $ git tag 4.17.1 2d93fa35dd917077e9248c7e564648da3a5f1fe3 && git tag 4.17.1-1 0c9f0f44a84f67ee5f14bccf6507540d4f7f8e39 && git tag 5.

Read more

Maintenance network Mediacentrale Nov 1st 2017 - 22:00

Maintenance network Mediacentrale On November 1st 2017 after 22:00 we will upgrade our network in the Mediacentrale. Due to roadworks around Julianaplein in Groningen that will impact our current connections, we will move our network traffic to an upgraded router and fiber path, thus minimizing downtime related to these roadworks. Furthermore, this maintenance also results in an upgrade in our capacity to the Mediacentrale, as we will upgrade from a 1G to 10G infrastructure.

Read more

linux / process uptime / exact

How to get (semi)exact uptime values for processes? If you look at the ps faux listing, you’ll see a bunch of values: walter 27311 0.8 1.8 5904852 621728 ? SLl sep06 61:05 \_ /usr/lib/chromium-browser/... walter 27314 0.0 0.2 815508 80852 ? S sep06 0:00 | \_ /usr/lib/chromium-brow... walter 27316 0.0 0.0 815508 14132 ? S sep06 0:01 | | \_ /usr/lib/chromium-... That second value (27311) is the PID, the tenth (61:05) how much CPU time has been spent.

Read more

sudo / cron / silence logging / authlog

Do you use sudo for automated tasks? For instance to let the Zabbix agent access privileged information? Then your auth.log may look a bit flooded, like this: Aug 30 10:51:44 sudo: zabbix : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/iptables -S INPUT Aug 30 10:51:44 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Aug 30 10:51:44 sudo: pam_unix(sudo:session): session closed for user root Or, if you run periodic jobs by root from cron, you get this:

Read more

powerdns / pdnsutil / remove-record

The PowerDNS nameserver pdnsutil utility has an add-record, but no remove-record. How can we remove records programmatically for many domains at once? Step one: make sure we can list all domains. For our PowerDNS 4 setup, we could do the following: $ list_all() { ( for type in master native; do pdnsutil list-all-zones $type; done ) | grep -vE '^.$|:' | sort -V; } $ list_all domain1.tld domain2.tld ... Step two: filter the domains where we want to remove anything.

Read more

gdb / debugging asterisk / ao2_containers

One of our Asterisk telephony machines appeared to “leak” queue member agents. That is, refuse to ring them because they were supposedly busy. When trying to find the cause, there weren’t any data dumping functions for the container I wanted to inspect in the CLI. In this case the pending_members which is of type struct ao2_container. So, we had to resort to using gdb to inspect the data. The struct ao2_container container data type itself looks like this:

Read more

letsencrypt / expiry mails / unsubscribe

Today I got one of these Letsencrypt Expiry mails again. It looks like this: Your certificate (or certificates) for the names listed below will expire in 19 days (on 21 Jun 17 19:38 +0000). Please make sure to renew your certificate before then, or visitors to your website will encounter errors. [domain here] ... If you want to stop receiving all email from this address, click [link here] (Warning: this is a one-click action that cannot be undone) I don’t need this particular domain anymore.

Read more

puppet / pip_version / facter

Every once in a while I have to deal with machines provisioned by puppet. I can’t seem to get used to the fact that --test not only tests, but actually does. It displays what it does though output, which is nice. To test without applying, you need the --noop flag. But, today I wanted to bring up the quick fix to this old warning/error: Error: Facter: error while resolving custom fact "pip_version": undefined method `[]' for nil:NilClass The cause of the issue is an old version of pip(1) which has no --version parameter.

Read more

ubuntu zesty / apt / dns timeout / srv records

Ever since I updated from Ubuntu/Yakkety to Zesty, my apt-get(1) would sit and wait a while before doing actual work: $ sudo apt-get update 0% [Working] Madness. Let’s see what it’s doing… $ sudo strace -f -s 512 apt-get update ... [pid 5603] connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 ... [pid 5603] sendto(3, "\1\271\1\0\0\1\0\0\0\0\0\0\5_http\4_tcp\3ppa\tlaunchpad\3net\0\0!\0\1", 46, MSG_NOSIGNAL, NULL, 0) = 46 [pid 5603] poll([{fd=3, events=POLLIN}], 1, 5000 <unfinished ...> ... [pid 5600] select(8, [5 6 7], [], NULL, {0, 500000}) = 0 (Timeout) .

Read more

squashing old git history

You may have an internal project that you wish to open source. When starting the project, you didn’t take that into account, so it’s likely to contain references to private data that you do not wish to share. Step one would be to clean things up. If this is a slow process, this can take time, while in the mean time the project gets updates. Now, at one point you’re confident that at commit X1000, the project contains only non-private data.

Read more

Loadbalancer maintenance 22nd february 2017

In the night of Wednesday (22nd of Febr. 2017) to Thursday (23rd of Febr. 2017) between 23:45 and 03:00 we will perform maintenance on our loadbalancers. ####Maintenance window 22-02-2017 23:45 - 23-02-2017 03:00 ####Description Loadbalancers will be upgraded to increase throughput and enable new capabilities. ####Impact When maintenance starts, we’ll reroute all traffic to the secondary loadbalancer. Customers that have a multi-site setup should therefore not have any service interruption.

Read more

detect invisible selection / copy buffer / chrome

In Look before you paste from a website to terminal the author rightly warns us about carelessly pasting any input from a web page into the terminal. This LookBeforePaste Chrome Extension is a quick attempt at trying to warn the user. Example output when pressing CTRL-C on the malicious code: Heuristics are defined as follows. They could certainly be improved, but it’s a start. function isSuspicious(node) { if (node.nodeType == node.

Read more