Blog

On February 16, 2016 details on a vulnerability in glibc were released (CVE-2015-7547). The vulnerability is remotely exploitable and affects a lot of systems. More info will be added later when more information is available. We started emergency patch procedures for our environments and managed customer environments. Summary Classification: Critical. Remote exploitation possible. Impact: Wide impact, all services that use glibc and perform dns resolving are vulnerable. upstream description The glibc DNS client side resolver is vulnerable to a stack-based buffer overflow when the getaddrinfo() library function is used.

So, while developing a server application for a client, my colleague Harm decided it would be a waste of our programming time to add TCP server code. Inetd and friends can do that really well. The amount of new connects to the server would be minimal, so the overhead of spawning a new Python process for every connect was negligible. Using xinetd as an inetd server wrapper is simple. The config would look basically like this:

In the night of Friday (Feb. 12th) to Saturday (Feb. 13th) between 1:00 and 2:00 our co-location provider will perform maintenance on the PDU's in one of our racks. Customers we consider to be directly affected (bare metal servers) will receive an additional notification. Outside of that it's mostly OSSO infrastructure services that are affected. ####Maintenance window 01:00-02:00 on 13th of Februari 2016. ####Description The PDU's of one of the racks will be taken into service one by one (A and B feed).

I recently set up psdiff on a few of my servers as a basic means to monitor process activity. It disclosed that my SaltStack master daemon — which I'm running as a non-privileged user — was losing a single child, exactly 24 hours after I had ran salt commands. This seemed to be a recurring phenomenon. The salt server — version 0.17.5+ds-1 on Ubuntu Trusty — was running these processes:

Polyglot XHTML: Serving pages that are valid HTML and valid XML at the same time. A number of documents have been written on the subject, which I shall not repeat here. My summary: HTML5 is not going away. XHTML pages validate in the browser. If you can get better validation during the development of your website, then you'll save yourself time and headaches. Thus, for your development environment, you'll set the equivalent of this: DEFAULT_CONTENT_TYPE = 'application/xhtml+xml' (and a workaround for Django pages if they're out of your control) But for your production environment, you'll still use text/html.

From 24th of December we are on leave and return to the office on the 4th of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability.

In the night of Monday (Dec. 7th) to Tuesday (Dec. 8th) between 1:00 and 6:00 we will upgrade the router at our RUG Rekenhal POP. Impact is limited to IP Access locations and IP Transit customers on this POP. ####Maintenance window 01:00-04:00 on 8th of December 2015. ####Description We will upgrade the router to allow planned network upgrades. Impact The router at the RUG Rekenhal POP will be unavailable for 30min-60min.

Getting the Asterisk PBX CLI to work more like you're used to from the (readline) bash shell can, be a time-saver. For example, you may want reverse-i-search (^R), backward word deletion (^W) and word skipping (^<arrow-left> and ^<arrow-right>). It can be done, but you must configure the editline library in a similar manner as you would configure .inputrc. Support for the .editrc configuration file was added in May 2011 (git commit d508a921).

In the night of Thursday (Nov. 26th) to Friday (Nov. 27th) between 1:00 and 6:00 we will perform maintenance to the virtual server infrastructure at location TCN. ####Maintenance window 01:00-06:00 on 27th of November 2015 ####Description Virtual server infrastructure will be upgraded to a new major software release. Due to the major version upgrade and incompatibility between versions servers will experience downtime for 15-60 minutes Impact Virtual servers will be offline for ~15 minutes (up to 60 minutes worst case).

We wanted to use EncFS to be able to store encrypted backups. The requirements for that are: The backup server initiates the backup. That's where we configure which hours are safe (resource wise) and which files need backing up (etc, home, root, srv, …). And it means the backup server can safely be placed behind a gateway disallowing all incoming connections. The backup server cannot know the passwords of files.

I ran into an obscure Could not link against ! error when configuring EncFS: ~/src$ apt-get source encfs ... ~/src$ cd encfs-1.7.4/ ~/src/encfs-1.7.4$ ./configure ... configure: WARNING: BOOST_CPPFLAGS -I/usr/include checking whether the Boost::Serialization library is available... yes configure: error: Could not link against ! That's odd. And not immediately obvious how to fix. For starters we need all the dependencies that Debian defines: ~/src/encfs-1.7.4$ sed -e '/^Build-Depends: /!d;s/^[^:]*: //;s/([^)]*)//g;s/,//g' \ debian/control debhelper librlog-dev librlog5 libfuse-dev libssl-dev pkg-config libboost-serialization-dev libboost-filesystem-dev quilt dh-autoreconf ~/src/encfs-1.

Service disruptions on one of the core routers location (CR1) in TCN. 5 Nov 2015 - 01:00 CR1 Supervisor placement Following up the maintenance finished on Monday, we will add an extra Router Supervisor in CR1 for extended redundancy in case of main Supervisor failure. This maintenance will be carried out tonight, at 01:00 on the 5th of November. There is no expected impact. This maintenance is a followup on the hardware replacement of the Supervisor in CR1.

In the night of Thursday (Nov. 5th) to Friday (Nov. 6th) between 1:00 and 3:00 we will perform network maintenance at our TCN co-location. ####Maintenance window 01:00-03:00 on 6th of November 2015 ####Description We will perform changes in the network configuration which will cause a change of mac address of the gateway for each subnet. Impact Depending on the device it may take some time to pick up this change. Generally, busy servers will pick up the changes almost instantly and servers which are mostly idle may take a while.

A few days ago, the Scapy project was brought to my attention. Scapy is an internet packet manipulation library for Python2. It can be used to sniff and decode packets, or to generate your own custom packets. In the most basic form, it runs on raw sockets, sniffing and decoding traffic like tcpdump. See the sniff() examples and the send(IP(dst="1.2.3.4") / ICMP()) example for sending a simple packet. But just as easily, it works on regular datagram sockets — those that you don't need CAP_NET_RAW powers for.

To syntax check Python code before executing, I use flake8. And when coding in the Vim editor, I use the vim-flake8 plugin that allows me to hit <F7> to quickly check for errors in the file I'm currently working in. But, there are currently two common flavors of Python: python2 and python3. And therefore flake8 comes in two flavors as well — you guessed it — a python2 and a python3 flavor.

While I was writing a Python tool to wrap C Gdb so I could fetch some info out of it automatically, I ran into the issue that it reads the terminal size (lines x columns) to adjust its output. I wanted consistent machine readable output, so I enlarged the terminal size programmatically: now row based output would not get wrapped by Gdb. Later I noticed that it would cease to use the terminal size — in fact, use the default 80 columns — if I also redirected stderr to a non-tty.

As of this writing, Debian testing (stretch) contains Asterisk version 13.1.0. The Debian source as GIT repository is here: https://anonscm.debian.org/git/pkg-voip/asterisk.git (browse) Packaging a newer version is not that hard, if we start out with the debian/ directory kindly supplied by the Debian maintainers. Hints to get things running: Use a local git repository By using a local git repository in your unpacked Asterisk dir, you can quickly restart from scratch any time you mess anything up.

I was wondering how easy it was to encrypt files before rsyncing them away to the backup machine. A quick search turned up the suggestion to use encfs by the user Thor on ServerFault. That looks like a decent solution. Let's figure out if it meets our needs. The idea is that we do this: # mount read-only encrypted virtual copy of unencrypted local data: encfs --reverse -o ro ~/data/ ~/.

Here, an awesome shell one-liner to find which process uses the most files, relative to its max-open-files soft limit. $ for x in /proc/[0-9]* do fds=0 max=`awk '/^Max open files/ {print $4}' $x/limits 2>/dev/null` && for t in $x/fd/*; do fds=$((fds+1)); done && echo $((fds*100/max)) ${x##*/} done | sort -rn | while read l do pid=${l##* }; echo "$l `readlink /proc/$pid/exe`"; break; done 57 16674 /usr/lib/dovecot/imap-login So, my imap-login (pid 16674) apparently uses 57% percent of its allowed max open files.

Okay, so we all know that printers are sent from hell, but we still need to use them from time to time. Today, we were trying to print a PDF document with bar codes on it. Amazingly enough, the text on the PDF looked fine, but the bar codes (images) appeared as if they were wrapped at the wrong place. Luckily, convert(1) from ImageMagick came to the rescue: $ convert -density 300 -define pdf:fit-page=A4 input.

As I mentioned the other day, my VM was slow, so I needed a way to figure out which VM guests were causing the heavy load on our Proxmox platform. I hacked up proxtop to enumerate the top resource users: $ ./proxtop -t day proxmox.example.com monitor@pve Password:<enter password> SORTED BY: cpu, avg ... SORTED BY: diskread, avg ------------------ #0: 3.1 MiB/s pve10 (acme-bugs-bunny) #1: 1.3 MiB/s pve07 (customerX-private) #2: 992.3 KiB/s pve10 (acme-road-runner) .

So, my VM was slow, and I needed to know which VM guest was eating all the resources. These VM containers are all managed by Proxmox; which is great, but it doesn't show which VM guest is eating all the resources. Luckily, Proxmox provides an API to get that info. The docs pointed to two API modules for Python, my language of choice for these kinds of jobs: proxmoxer and pyproxmox.

Today, my choice of Python modules to Interface with Zabbix. They are all pretty similar, so that made it harder to choose. Here the six modules, as mentioned on the Zabbix wiki are, in the order of my preference. Note that second and third came close, but I favor clean documented code and fewer dependencies. The last ones didn't get tested because of my Python3 requirement. zabbix-client # pip: zabbix-client # pep: 99% # last-update: Aug.

Even after writing plenty of Asterisk PBX dialplan, I occasionally get bitten by the unintuitiveness of the parser. A few rules to avoid mistakes, are: Always use double quotes on no side of the expression, or better yet, on both if there is a chance that the value is empty: $[${HANGUPCAUSE}=17] or $["${value_which_may_be_empty}"="somevalue"] Otherwise try to avoid double quotes (and semi-colons, and backslashes) whenever possible. If you need to escape them, it's too easy to get it wrong.

A high risk security issue in glibc was disclosed last night. Because of the potential high impact we started our emergency patch procedures for osso managed environments and notify customers with self managed environments. Ghost vulnerability details Qualys discovered a buffer overflow in dns resolve functions in the GNU C library (glibc). They created a proof of concept exploit for exim and dubbed the vulnerability "GHOST". All processes that might do dns lookups are susceptible to attacks when using a vulnerable glibc version.

While we do Python VirtualEnv stuff every day, we rarely do Ruby environments. And after the Ubuntu dist-upgrade, the Ruby dependencies for our GitLab were broken — as was expected. This happens for Python pip installed packages as well. They're linked against older system libraries, which have been removed by the upgrade. How to fix the Gitlab dependencies? Browse through the upgrade docs to find a bundle install command. # cd /home/git/gitlab # sudo -u git -H bundle install \ --without development test postgres --deployment # for MySQL That did… absolutely nothing — again, as was expected.

Tired of the Fail2ban start and stop e-mails? Especially after a manual fail2ban restart, the [Fail2Ban] vsftpd: stopped on HOSTNAME and [Fail2Ban] vsftpd: started on HOSTNAME mail tuple is too spammy. Quick fix to disable them: Create a new file, named /etc/fail2ban/actions.d/sendmail-no-start-stop.local: diff --git /etc/fail2ban/action.d/sendmail-no-start-stop.local /etc/fail2ban/action.d/sendmail-no-start-stop.local new file mode 100644 index 0000000..cb7ecb9 --- /dev/null +++ /etc/fail2ban/action.d/sendmail-no-start-stop.local @@ -0,0 +1,3 @@ +[Definition] +actionstart = +actionstop = And — you're using mta = sendmail right?

When trying to keep up with all the TLS/SSL security changes, you need to modify your nginx config every now and then. The good TLS config may look like this: # nginx.conf: http { ssl_certificate /etc/ssl/MY_DOMAIN.pem; ssl_certificate_key /etc/ssl/MY_DOMAIN.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA; ssl_session_cache shared:SSL:5m; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security "max-age=15768000; includeSubDomains"; And the above config is accompanied by a fairly good A grade from the Qualys SSL Labs Analyzer.

Storing an UUID in MySQL efficiently: DROP FUNCTION IF EXISTS uuidbin; CREATE FUNCTION uuidbin(uuid_val varchar(36)) RETURNS varbinary(16) DETERMINISTIC SQL SECURITY INVOKER RETURN CONCAT(UNHEX(LEFT(uuid_val,8)),UNHEX(MID(uuid_val,10,4)), UNHEX(MID(uuid_val,15,4)),UNHEX(MID(uuid_val,20,4)), UNHEX(RIGHT(uuid_val,12))); DROP FUNCTION IF EXISTS uuidstr; CREATE FUNCTION uuidstr(uuid_val varbinary(16)) RETURNS varchar(36) DETERMINISTIC SQL SECURITY INVOKER RETURN LOWER(CONCAT(HEX(LEFT(uuid_val,4)),'-',HEX(MID(uuid_val,5,2)), '-',HEX(MID(uuid_val,7,2)),'-',HEX(MID(uuid_val,9,2)), '-',HEX(RIGHT(uuid_val,6)))); Now you can create your uuid columns with type binary(16). And conversion is easy: mysql> select uuidstr(uuidbin(uuidstr(uuidbin(uuidstr(uuidbin( 'a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f')))))) as uuid_back_and_forth; +--------------------------------------+ | uuid_back_and_forth | +--------------------------------------+ | a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f | +--------------------------------------+

Django makemessages can be quite slow on larger projects. $ time python ../manage.py makemessages -lnl -ddjango processing language nl real 0m8.203s user 0m2.670s sys 0m5.763s Why does it take so long? Well, it's system call heaven: $ strace -f python ../manage.py makemessages -lnl -ddjango \ >tmp.log 2>&1 $ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log | sort | uniq -c | sort -n | tail -n10 10893 rt_sigaction 16179 stat 16819 fcntl 22875 access 27833 read 32469 open 33650 fstat 40891 mprotect 69181 mmap 1267039 close For every file, a call to xgettext(1) is made.

Sometimes, after a stray copy operation, your filesystem times may reflect the time the files were copied instead of when the file was actually last altered. For example this image folder here: $ ls -l phone2013 total 320856 -rw-rw-r-- 1 walter walter 1524591 nov 17 21:52 2012-10-28 08.54.58.jpg -rw-rw-r-- 1 walter walter 1534840 nov 17 21:52 2012-10-28 08.55.04.jpg -rw-rw-r-- 1 walter walter 1635908 nov 17 21:52 2012-10-28 08.55.09.jpg ... -rw-rw-r-- 1 walter walter 1600504 nov 17 21:52 2013-10-22 11.

So, I was really simply trying to figure out why talking to my OpenSIPS instance over a datagram unix socket failed. If I had bothered to check the server logs, I would immediately have seen that it was a simple stupid permission issue. Instead, I ended up reimplementing recvfrom and sendto in Python using the ctypes library. Which was completely useless, since Python socket.recvfrom and socket.sendto already work properly. To let the time spent on that not go to a complete waste, I give you (and myself) an example of ctypes usage.

Syslog generally works fine as it is, so I don't need to poke around in it often. That also means that I forget how to tweak it. How did you move those every-5-minutes cron jobs out of /var/log/syslog? The rules (selection + action) look like this in the Debian default config: *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log The manual has this to say about it: You can specify multiple facilities with the same priority pattern in one statement using the comma (,) operator.

One of our datacenter locations (TCN Telehouse) will have major maintenance on its power infrastructure this month. They scheduled 4 maintenance windows of 1,5 hours each during which either the A or B feed will be powerless. In the last weeks we've double checked our infrastructure and this week we will finish our last preparations. All equipment that is not equipped with dual power supplies is connected to an ATS (Automatic Transfer Switch) to achieve power redundancy.

While I was battling an obscure Ubuntu shutdown issue — more about that later — I noticed that daemonized jobs started from my X session were not reparented to PID 1 init, but to a custom init --user, owned by me. What? I cannot start daemon that outlive my X session? That's right, I cannot. Check this out: $ sh -c 'sleep 61 &' $ ps faxu | egrep 'init|sleep 61' root 1 .

Today's git question: does git reset undo a merge or only parts of it? TL;DR: It undoes the entire merge. If you think about it logically, it must, since an object describes the entire state of the repository. But it can feel awkward and unexpected that older items than the object that we're resetting to, are removed as well. Let's just try it. Set up a repository with two branches:

Recently I wrote about cherry picking upgrades. Sometimes you'll want to do the inverse. For that purpose there exists apt-mark hold (and its counterpart apt-mark unhold). For example, you may to delay the mysql upgrade I mentioned, for now. In that case you do: # apt-mark hold mysql-client-5.5 mysql-common mysql-server-5.5 mysql-server-core-5.5 Now you can apt-get upgrade all the other packages while the mysql packages stay on hold. Note that these are shown in the held list every time you run upgrade, so you won't forget about them.

SquirrelMail on Debian/Wheezy (2:1.4.23~svn20120406-2) stopped showing (none) for e-mails that lack a subject. Now I cannot open any subject-less mail because there is nothing to click on. The quick fix: --- /usr/share/squirrelmail/functions/mailbox_display.php.orig 2014-08-15 10:37:37.000000000 +0200 +++ /usr/share/squirrelmail/functions/mailbox_display.php 2014-08-15 10:38:27.000000000 +0200 @@ -268,6 +268,9 @@ function printMessageInfo($imapConnectio $title = str_replace('"', "''", $title); $td_str .= " title=\"$title\""; } + if (!$subject) { + $subject = '(none)'; + } $td_str .= ">$flag$subject$flag_end</a>$bold_end"; echo html_tag( 'td', $td_str, 'left', $hlt_color ); break;

Ever needed to restore only one database on a MySQL server and found out you only had one SQL dump containing all databases? Its quite common to dump all databases in one SQL file (mysqldump –all-databases or -A). But when using multiple databases on one MySQL instance you often need to restore just one of them. The minimal effort solution: mysql --one-database desired_db_name < alldatabases.sql fix!

Transcript follows: [him] did I mention I'll be off from work earlier today because I'm having dinner with friends. I'll be off earlier today because I'm having dinner with friends. [me] where did you say you were going? [him] I'll be having dinner at the Grand Cafe Apparently the irony was lost on him. I should've used emoticons. But! Instead of emoticons, one may also use the irony punctuation: ⸮

So, doing an apt-get upgrade on a Debian or Ubuntu machine sometimes does more than you want at once. See this upgrade example I encountered just now: # apt-get upgrade ... The following packages will be upgraded: curl dpkg ifupdown iproute libcurl3 libcurl3-gnutls libgnutls26 libmysqlclient18 libsnmp-base libsnmp15 libssl1.0.0 libxml2 linux-firmware linux-generic-lts-quantal mysql-client-5.5 mysql-client-core-5.5 mysql-common mysql-server mysql-server-5.5 mysql-server-core-5.5 openssh-client openssh-server openssl tzdata update-manager-core whoopsie 26 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

Did you ever wonder what the '<,'> characters mean when you CTRL-V visual block select text in vim? For example: you press CTRL-V and select a bit of text. Then type : (colon). Instead of just the colon, you see: :'<,'>. You append s/^/#/ hit enter. As requested, the selected block is now “commented out”. That's a nice feature, but why the funny characters? In order to understand that, we remind you of the % (percent sign) that we use to select the entire file.

My .vimrc usually starts out with this. Syntax highlighting is super, and my terminals always have a black background. The modeline option enables me and others to set certain options for certain files only. Like: {# vim: syntax=htmldjango: #} to mark a .html file as using the django html syntax instead of regular html syntax. See also my Inserting vim modelines tip. syn on set bg=dark set modeline Second, since I develop a lot in Python, I enable the vim-flake8 python source code checker plugin:

I always forget how easy it is to upgrade postgresql on Ubuntu (from 9.1 to 9.3 this time). It seems like a pain to have to manually upgrade the cluster, but when it comes down to it, it's self-documenting and quick. My shell session basically went like this: $ sudo apt-get install postgresql-9.3 ... The following extra packages will be installed: postgresql-client-9.3 ... $ sudo /etc/init.d/postgresql stop * Stopping PostgreSQL 9.

Recently I tried to open a connection to a remote server over SSH at a new location. The connection opened just fine, but it seemed that a few bytes kept getting buffered. It looked like this first animated gif you see. After a long wait, you realise that the data you're wating just won't come. First after pressing a key, you get the data. This isn't workable… Enumerating the possible culprits, there could really only be the wifi-nat-modem — a Thomson TG789vn, Telia device — doing extra buffering, possibly conflicting with the Nagle algorithm (TCP_NODELAY).

On my recently upgraded Ubuntu Trusty (14.04) machine, git diff started producing colorized output. That's nice, but it'd be even nicer if it recognised that I'm using a dark background. Put this in your ~/.gitconfig. This colorscheme is the one you're used to from vim. [color "diff"] meta = green bold frag = yellow bold old = red bold new = cyan bold

Last night an important security vulnerability was made public with corresponding security updates. It risks exposing private keys when vulnerable. OpenSSL was vulnerable starting from their OpenSSL 1.0.1 release on 14th of March 2012 till OpenSSL 1.0.1g released on 7th of April 2014. Two security teams independently reported this issue and it's safe to assume others did as well. On top of that it's not possible to trace whether you were successfully exploited.

This morning we experienced packet loss on one of our links. After rerouting the traffic the problem was resolved. Only routes over the GN-IX were affected. timeline 12:15 first notification of the packetloss. 12:36 - 12:40 Hickups in the internet connectivity due to rerouting of traffic. after 12:40 No more issues for our customers. Offhour followup re-enable temporary disabled links and rebalance traffic. Background information The issue occurred because GN-IX hit its capacity limit on their links between Groningen and Amsterdam.

When you're monitoring security update availability using Zabbix or some other monitoring tool, you'll need a method to discern regular updates from security updates. I've seen my collegues do this: $ /usr/lib/update-notifier/apt-check --human-readable | grep security | awk '{print $1}' But that requires an install of the update-notifier-common package. (Note the -common. The main package has tons of requirements you don't need.) In the quest for less dependencies — less installed packages — I used aptitude to get the info.

After switching a virtual IP around with keepalived we experienced a locking issue. Some client process on the server had some locks which did not get released but we could not see which query caused it. This broke our MySQL replication in this case, it was waiting on the locks to be released while executing the binlogs. We located the locks with mysql> show engine innodb status\G; The specific transaction did not show any query details but did show us a whole list of locks.