Blog

Here, an awesome shell one-liner to find which process uses the most files, relative to its max-open-files soft limit. $ for x in /proc/[0-9]* do fds=0 max=`awk '/^Max open files/ {print $4}' $x/limits 2>/dev/null` && for t in $x/fd/*; do fds=$((fds+1)); done && test "${max:-0}" -gt 0 && echo $((fds*100/max)) ${x##*/} done | sort -rn | while read l do pid=${l##* }; echo "$l`readlink /proc/$pid/exe`"; break; done 57 16674 /usr/lib/dovecot/imap-login So, my imap-login (pid 16674) apparently uses 57% percent of its allowed max open files.

Okay, so we all know that printers are sent from hell, but we still need to use them from time to time. Today, we were trying to print a PDF document with bar codes on it. Amazingly enough, the text on the PDF looked fine, but the bar codes (images) appeared as if they were wrapped at the wrong place. Luckily, convert(1) from ImageMagick came to the rescue: $ convert -density 300 -define pdf:fit-page=A4 input.

As I mentioned the other day, my VM was slow, so I needed a way to figure out which VM guests were causing the heavy load on our Proxmox platform. I hacked up proxtop to enumerate the top resource users: $ ./proxtop -t day proxmox.example.com monitor@pve Password:<enter password> SORTED BY: cpu, avg ... SORTED BY: diskread, avg ------------------ #0: 3.1 MiB/s pve10 (acme-bugs-bunny) #1: 1.3 MiB/s pve07 (customerX-private) #2: 992.3 KiB/s pve10 (acme-road-runner) .

So, my VM was slow, and I needed to know which VM guest was eating all the resources. These VM containers are all managed by Proxmox; which is great, but it doesn’t show which VM guest is eating all the resources. Luckily, Proxmox provides an API to get that info. The docs pointed to two API modules for Python, my language of choice for these kinds of jobs: proxmoxer and pyproxmox.

Today, my choice of Python modules to Interface with Zabbix. They are all pretty similar, so that made it harder to choose. Here the six modules, as mentioned on the Zabbix wiki are, in the order of my preference. Note that second and third came close, but I favor clean documented code and fewer dependencies. The last ones didn’t get tested because of my Python3 requirement. zabbix-client # pip: zabbix-client # pep: 99% # last-update: Aug.

Even after writing plenty of Asterisk PBX dialplan, I occasionally get bitten by the unintuitiveness of the parser. A few rules to avoid mistakes, are: Always use double quotes on no side of the expression, or better yet, on both if there is a chance that the value is empty: $[${HANGUPCAUSE}=17] or $["${value_which_may_be_empty}"="somevalue"] Otherwise try to avoid double quotes (and semi-colons, and backslashes) whenever possible. If you need to escape them, it’s too easy to get it wrong.

A high risk security issue in glibc was disclosed last night. Because of the potential high impact we started our emergency patch procedures for osso managed environments and notify customers with self managed environments. Ghost vulnerability details Qualys discovered a buffer overflow in dns resolve functions in the GNU C library (glibc). They created a proof of concept exploit for exim and dubbed the vulnerability "GHOST". All processes that might do dns lookups are susceptible to attacks when using a vulnerable glibc version.

While we do Python VirtualEnv stuff every day, we rarely do Ruby environments. And after the Ubuntu dist-upgrade, the Ruby dependencies for our GitLab were broken — as was expected. This happens for Python pip installed packages as well. They’re linked against older system libraries, which have been removed by the upgrade. How to fix the Gitlab dependencies? Browse through the upgrade docs to find a bundle install command. # cd /home/git/gitlab # sudo -u git -H bundle install \ --without development test postgres --deployment # for MySQL That did… absolutely nothing — again, as was expected.

Tired of the Fail2ban start and stop e-mails? Especially after a manual fail2ban restart, the [Fail2Ban] vsftpd: stopped on HOSTNAME and [Fail2Ban] vsftpd: started on HOSTNAME mail tuple is too spammy. Quick fix to disable them: Create a new file, named /etc/fail2ban/actions.d/sendmail-no-start-stop.local: diff --git /etc/fail2ban/action.d/sendmail-no-start-stop.local /etc/fail2ban/action.d/sendmail-no-start-stop.local new file mode 100644 index 0000000..cb7ecb9 --- /dev/null +++ /etc/fail2ban/action.d/sendmail-no-start-stop.local @@ -0,0 +1,3 @@ +[Definition] +actionstart = +actionstop = And — you’re using mta = sendmail right?

When trying to keep up with all the TLS/SSL security changes, you need to modify your nginx config every now and then. The good TLS config may look like this: # nginx.conf: http { ssl_certificate /etc/ssl/MY_DOMAIN.pem; ssl_certificate_key /etc/ssl/MY_DOMAIN.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA; ssl_session_cache shared:SSL:5m; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security "max-age=15768000; includeSubDomains"; And the above config is accompanied by a fairly good A grade from the Qualys SSL Labs Analyzer.

Storing an UUID in MySQL efficiently: DROP FUNCTION IF EXISTS uuidbin; CREATE FUNCTION uuidbin(uuid_val varchar(36)) RETURNS varbinary(16) DETERMINISTIC SQL SECURITY INVOKER RETURN CONCAT(UNHEX(LEFT(uuid_val,8)),UNHEX(MID(uuid_val,10,4)), UNHEX(MID(uuid_val,15,4)),UNHEX(MID(uuid_val,20,4)), UNHEX(RIGHT(uuid_val,12))); DROP FUNCTION IF EXISTS uuidstr; CREATE FUNCTION uuidstr(uuid_val varbinary(16)) RETURNS varchar(36) DETERMINISTIC SQL SECURITY INVOKER RETURN LOWER(CONCAT(HEX(LEFT(uuid_val,4)),'-',HEX(MID(uuid_val,5,2)), '-',HEX(MID(uuid_val,7,2)),'-',HEX(MID(uuid_val,9,2)), '-',HEX(RIGHT(uuid_val,6)))); Now you can create your uuid columns with type binary(16). And conversion is easy: mysql> select uuidstr(uuidbin(uuidstr(uuidbin(uuidstr(uuidbin( 'a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f')))))) as uuid_back_and_forth; +--------------------------------------+ | uuid_back_and_forth | +--------------------------------------+ | a89e6d7b-f2ec-11e3-bcfb-5c514fe65f2f | +--------------------------------------+

Django makemessages can be quite slow on larger projects. $ time python ../manage.py makemessages -lnl -ddjango processing language nl real 0m8.203s user 0m2.670s sys 0m5.763s Why does it take so long? Well, it’s system call heaven: $ strace -f python ../manage.py makemessages -lnl -ddjango \ >tmp.log 2>&1 $ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log | sort | uniq -c | sort -n | tail -n10 10893 rt_sigaction 16179 stat 16819 fcntl 22875 access 27833 read 32469 open 33650 fstat 40891 mprotect 69181 mmap 1267039 close For every file, a call to xgettext(1) is made.

Sometimes, after a stray copy operation, your filesystem times may reflect the time the files were copied instead of when the file was actually last altered. For example this image folder here: $ ls -l phone2013 total 320856 -rw-rw-r-- 1 walter walter 1524591 nov 17 21:52 2012-10-28 08.54.58.jpg -rw-rw-r-- 1 walter walter 1534840 nov 17 21:52 2012-10-28 08.55.04.jpg -rw-rw-r-- 1 walter walter 1635908 nov 17 21:52 2012-10-28 08.55.09.jpg ... -rw-rw-r-- 1 walter walter 1600504 nov 17 21:52 2013-10-22 11.

So, I was really simply trying to figure out why talking to my OpenSIPS instance over a datagram unix socket failed. If I had bothered to check the server logs, I would immediately have seen that it was a simple stupid permission issue. Instead, I ended up reimplementing recvfrom and sendto in Python using the ctypes library. Which was completely useless, since Python socket.recvfrom and socket.sendto already work properly. To let the time spent on that not go to a complete waste, I give you (and myself) an example of ctypes usage.

Syslog generally works fine as it is, so I don’t need to poke around in it often. That also means that I forget how to tweak it. How did you move those every-5-minutes cron jobs out of /var/log/syslog? The rules (selection + action) look like this in the Debian default config: *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log The manual has this to say about it: You can specify multiple facilities with the same priority pattern in one statement using the comma (,) operator.

One of our datacenter locations (TCN Telehouse) will have major maintenance on its power infrastructure this month. They scheduled 4 maintenance windows of 1,5 hours each during which either the A or B feed will be powerless. In the last weeks we’ve double checked our infrastructure and this week we will finish our last preparations. All equipment that is not equipped with dual power supplies is connected to an ATS (Automatic Transfer Switch) to achieve power redundancy.

While I was battling an obscure Ubuntu shutdown issue — more about that later — I noticed that daemonized jobs started from my X session were not reparented to PID 1 init, but to a custom init --user, owned by me. What? I cannot start daemon that outlives my X session? That’s right, I cannot. Check this out: $ sh -c 'sleep 61 &' $ ps faxu | egrep 'init|sleep 61' root 1 .

Today’s git question: does git reset undo a merge or only parts of it? TL;DR: It undoes the entire merge. If you think about it logically, it must, since an object describes the entire state of the repository. But it can feel awkward and unexpected that older items than the object that we’re resetting to, are removed as well. Let’s just try it. Set up a repository with two branches:

Recently I wrote about cherry picking upgrades. Sometimes you’ll want to do the inverse. For that purpose there exists apt-mark hold (and its counterpart apt-mark unhold). For example, you may to delay the mysql upgrade I mentioned, for now. In that case you do: # apt-mark hold mysql-client-5.5 mysql-common mysql-server-5.5 mysql-server-core-5.5 Now you can apt-get upgrade all the other packages while the mysql packages stay on hold. Note that these are shown in the held list every time you run upgrade, so you won’t forget about them.

SquirrelMail on Debian/Wheezy (2:1.4.23~svn20120406-2) stopped showing (none) for e-mails that lack a subject. Now I cannot open any subject-less mail because there is nothing to click on. The quick fix: --- /usr/share/squirrelmail/functions/mailbox_display.php.orig 2014-08-15 10:37:37.000000000 +0200 +++ /usr/share/squirrelmail/functions/mailbox_display.php 2014-08-15 10:38:27.000000000 +0200 @@ -268,6 +268,9 @@ function printMessageInfo($imapConnectio $title = str_replace('"', "''", $title); $td_str .= " title=\"$title\""; } + if (!$subject) { + $subject = '(none)'; + } $td_str .= ">$flag$subject$flag_end</a>$bold_end"; echo html_tag( 'td', $td_str, 'left', $hlt_color ); break;

Ever needed to restore only one database on a MySQL server and found out you only had one SQL dump containing all databases? Its quite common to dump all databases in one SQL file (mysqldump –all-databases or -A). But when using multiple databases on one MySQL instance you often need to restore just one of them. The minimal effort solution: mysql --one-database desired_db_name < alldatabases.sql fix!

Transcript follows: [him] did I mention I'll be off from work earlier today because I'm having dinner with friends. I'll be off earlier today because I'm having dinner with friends. [me] where did you say you were going? [him] I'll be having dinner at the Grand Cafe Apparently the irony was lost on him. I should’ve used emoticons. But! Instead of emoticons, one may also use the irony punctuation: ⸮

So, doing an apt-get upgrade on a Debian or Ubuntu machine sometimes does more than you want at once. See this upgrade example I encountered just now: # apt-get upgrade ... The following packages will be upgraded: curl dpkg ifupdown iproute libcurl3 libcurl3-gnutls libgnutls26 libmysqlclient18 libsnmp-base libsnmp15 libssl1.0.0 libxml2 linux-firmware linux-generic-lts-quantal mysql-client-5.5 mysql-client-core-5.5 mysql-common mysql-server mysql-server-5.5 mysql-server-core-5.5 openssh-client openssh-server openssl tzdata update-manager-core whoopsie 26 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

Did you ever wonder what the '<,'> characters mean when you CTRL-V visual block select text in vim? For example: you press CTRL-V and select a bit of text. Then type : (colon). Instead of just the colon, you see: :'<,'>. You append s/^/#/ hit enter. As requested, the selected block is now “commented out”. That’s a nice feature, but why the funny characters? In order to understand that, we remind you of the % (percent sign) that we use to select the entire file.

My .vimrc usually starts out with this. Syntax highlighting is super, and my terminals always have a black background. The modeline option enables me and others to set certain options for certain files only. Like: {# vim: syntax=htmldjango: #} to mark a .html file as using the django html syntax instead of regular html syntax. See also my Inserting vim modelines tip. syn onset bg=darkset modelineSecond, since I develop a lot in Python, I enable the vim-flake8 python source code checker plugin:

I always forget how easy it is to upgrade postgresql on Ubuntu (from 9.1 to 9.3 this time). It seems like a pain to have to manually upgrade the cluster, but when it comes down to it, it’s self-documenting and quick. My shell session basically went like this: $ sudo apt-get install postgresql-9.3 ... The following extra packages will be installed: postgresql-client-9.3 ... $ sudo /etc/init.d/postgresql stop * Stopping PostgreSQL 9.

Recently I tried to open a connection to a remote server over SSH at a new location. The connection opened just fine, but it seemed that a few bytes kept getting buffered. It looked like this first animated gif you see. After a long wait, you realise that the data you’re wating just won’t come. First after pressing a key, you get the data. This isn’t workable… Enumerating the possible culprits, there could really only be the wifi-nat-modem — a Thomson TG789vn, Telia device — doing extra buffering, possibly conflicting with the Nagle algorithm (TCP_NODELAY).

On my recently upgraded Ubuntu Trusty (14.04) machine, git diff started producing colorized output. That’s nice, but it’d be even nicer if it recognised that I’m using a dark background. Put this in your ~/.gitconfig. This colorscheme is the one you’re used to from vim. [color "diff"] meta = green bold frag = yellow bold old = red bold new = cyan bold

Last night an important security vulnerability was made public with corresponding security updates. It risks exposing private keys when vulnerable. OpenSSL was vulnerable starting from their OpenSSL 1.0.1 release on 14th of March 2012 till OpenSSL 1.0.1g released on 7th of April 2014. Two security teams independently reported this issue and it’s safe to assume others did as well. On top of that it’s not possible to trace whether you were successfully exploited.

This morning we experienced packet loss on one of our links. After rerouting the traffic the problem was resolved. Only routes over the GN-IX were affected. timeline 12:15 first notification of the packetloss. 12:36 - 12:40 Hickups in the internet connectivity due to rerouting of traffic. after 12:40 No more issues for our customers. Offhour followup re-enable temporary disabled links and rebalance traffic. Background information The issue occurred because GN-IX hit its capacity limit on their links between Groningen and Amsterdam.

When you’re monitoring security update availability using Zabbix or some other monitoring tool, you’ll need a method to discern regular updates from security updates. I’ve seen my collegues do this: $ /usr/lib/update-notifier/apt-check --human-readable | grep security | awk '{print $1}' But that requires an install of the update-notifier-common package. (Note the -common. The main package has tons of requirements you don’t need.) In the quest for less dependencies — less installed packages — I used aptitude to get the info.

After switching a virtual IP around with keepalived we experienced a locking issue. Some client process on the server had some locks which did not get released but we could not see which query caused it. This broke our MySQL replication in this case, it was waiting on the locks to be released while executing the binlogs. We located the locks with mysql> show engine innodb status\G; The specific transaction did not show any query details but did show us a whole list of locks.

For me, using FreeBSD is still a bit like eating soup with a fork. Everything seems to make perfect sense but when I get to work I feel crippled and get annoyed by small differences with a GNU/Linux environment. A post to reduce the horror. The examples below work for a clean FreeBSD 10 install. This post is mostly useful for Linux users. I welcome additional tips. software management You can now use pkg which is quite friendly.

Today we received a DDoS on our network which caused a service interruption for our customers for about 20 minutes. This blogpost is a short report on the impact and nature of the attack. Impact Impact was network wide and caused degraded service for our customers between 14:45 and 15:08, a little over 20 minutes. The graph shows the impact as seen from our UK monitoring node (off net). Incoming traffic All our uplinks were saturated.

The Python xml.dom.minidom parseString silently skips over unknown entities. The only entities it does know, are &lt;, &gt;, &amp;, &apos; and &quot; and of course the numeric entities &#nn; and &#xhh;. That’s obvious, because those are the only ones defined in the XML 1.0 spec. However, if you’re parsing XHTML documents, it’s not nice that the entity references to special characters silently get dropped. Other people have stubled on the same issue, like in parsing xml containing &entities; with minidom and Problem with minidom and special chars in HTML.

A simple script to convert BSON data to JSON data: bson2json.py (view) Example usage: $ bson2json.py /var/backups/mongodb/all-dbs.mon/graylog2/streams.bson --pretty [ { "_id": "506ed227dc1d710c0700000e", "additional_columns": [], "alarm_active": true, "alarm_callbacks": [ "org.graylog2.emailalarmcallback.callback.EmailAlarmCallback", "org.graylog2.execalarmcallback.callback.ExecAlarmCallback" ], "alarm_limit": 80, "alarm_period": 5, "alarm_timespan": 5, "created_at": "2012-10-05T12:27:19Z", ...

Apparently I’m not the only one who randomly selects text as they read. My colleagues complained about this issue too. If you click Reply in Thunderbird Mail, only the text you recently selected is included in the new message. That’s not what I wanted! Luckily the Mozilla developers realised this too. Go to about:config and flip the switch. mailnews.reply_quoting_selection = false

Today we got a suspiciously good looking e-mail in the inbox. Someone who supposedly got a reminder about an unpaid invoice from us. The mail contained a zip-file with two scans. The first was a PDF, the second was an executable (a virus obviously). So.. where was the Amavis virus/spam scanner in all this? Show headers revealed that something was detected: X-Amavis-Alert: BANNED, message contains .exe,scan2/HP scan scan =?iso-8859-1?Q?HYJKIOPH5600002.=E2=80=AEfdp.exe?= Then why weren’t we informed?

After the upgrade of my desktop to Ubuntu Raring (13.04) my gnome-calculator’s menu bar had become unreachable. I don’t need the menu, except that it went into default BASIC mode. And I need the PROGRAMMING mode. The configuration seemed to be okay (accessible through gconf-editor): $ gconftool /apps/gcalctool --dump | grep -B1 -A4 mode <entry> <key>mode</key> <value> <string>PROGRAMMING</string> </value> </entry> <entry> <key>modetype</key> <value> <string>PROGRAMMING</string> </value> </entry> But that was apparently the old config.

A quick rundown on installing TeamViewer without a gazillion ia32-libs. The problem: if you attempt to install teamviewer_linux_x64.deb on your 64-bit machine, the ia32-libs dependency tries to install more than 200 packages. That not only feels like overkill, it takes a hell of a long time too. The solution: alter the dependency list in the .deb and create a small metapackage that references only the required libs. What follows, is the steps how.

Voilà, a MySQL function to count occurrences of a character (or a string of characters). DROP FUNCTION IF EXISTS OCCURRENCES; delimiter // CREATE FUNCTION OCCURRENCES (`needle` VARCHAR(255), `hackstack` TEXT) RETURNS INT NOT DETERMINISTIC READS SQL DATA SQL SECURITY INVOKER BEGIN DECLARE `result` INT DEFAULT -1; DECLARE `pos` INT DEFAULT 0; DECLARE `skip` INT DEFAULT LENGTH(`needle`); REPEAT SET `pos` = (SELECT LOCATE(`needle`, `hackstack`, `pos` + `skip`)); SET `result` = `result` + 1; UNTIL `pos` = 0 END REPEAT; RETURN `result`; END; // delimiter ; Now you can do things like this:

MySQL has many odd quirks. One that bit us recently was this: regression: >=mysql-5.4 utf8 collations are marked as not ascii compatible When using the utf8_unicode_ci collation, datetime column comparisons against strings would ignore any indexes. The lack of working indexes obviously caused huge performance degradation. Our bug report was ignored in Februari. Apparently a new bug was opened in March: Datetime field comparisons do not work properly with utf8_unicode_ci collation

Mozilla Thunderbird uses an odd max line length of 999 + CRLF: 1001 characters. When using DKIM preprocessing, this can result in DKIM validation failure. To reproduce, we would send a mail that didn’t wrap well with line lengths in excess of 999 characters. Like this mail with 1000 'x' characters: Thunderbird splits that into 999 times 'x', and CRLF and another line with a single 'x'. However, that first line gets split again.

Numerous articles have been written about why you want to install Pillow instead of PIL to get the Python Imaging tools. Like Problems with PIL? Use Pillow instead! (Find more by searching for “IOError: decoder zip not available”.) This note concerns something more insidious: a seemingly broken Pillow installation after the removal of PIL. ~$ mkvirtualenv piltest (piltest)~$ pip install PIL (piltest)~$ pip freeze | grep -i pil PIL==1.1.7 Now this should work:

Caveats arbiter - 10.10.10.1 node1 - 10.10.10.10 node1 - 10.10.10.11 Installing the nodes update /etc/hosts to make sure all the machines are resolvable, perform this on all 3 hosts (2 active mysql nodes and the machine the arbiter resides on). Update the hostnames and addresses accordingly. $ cat << EOF >> /etc/hosts 10.10.10.1 arbiter 10.10.10.10 node1 10.10.10.11 node2 EOF Setup APT to use the MariaDB repositories

Just a quick post with the most straight forward way to configure http webservers/proxies in regard to ssl certificate chains. I’m open for improvements if it improves readability or reduces linecount. :-) pound In /etc/pound/pound.cfg you only need one line in the HTTPS listener: ListenHTTPS Cert "/etc/pound/file-with-key-and-cert-chain.pem" apache2 SSLCertificateFile /path/to/file-with-key-and-cert-chain.pem SSLCertificateChainFile /path/to/file-with-key-and-cert-chain.pem

Most browser trusted SSL certificates work with intermediate certificates nowadays. The CA only ships the root certificates for inclusion with the browsers and because they do this your certificate can’t be verified if you don’t include the intermediate certificates in your webserver configuration. chain order So you’re in the bottom in a chain of trust. Your certificate is signed by a certificate which is signed by another certificate and so on.

Of course you do daily backups of your data. For your databases, you generally need a dump of the data for it to be useful. For your and my reference, here are a few database dump scripts. Make sure /var/backups is backed up daily. Observe that we keep extra backups around. I’ve found that the need to access an old database is far more common than accessing old files (from the backup storage).

So, there have been numerous reports around the internet that the HP ProBook 4510s has fan speed issues. In my case, the following happens: Turn on the laptop: no problem. Using the laptop while on AC: no problem. Switch on the laptop from suspend mode without AC plugged in: problem! The fan goes into overdrive and stays there. It's related to the temperature measurements, somehow. According to the sensors(1) tool, it's the temp6 value that's off.

If you add new disks to a virtual machine you don’t have to reboot to be able to use them. I assume this works the same for normal hotswappable disks as well but that just doesn’t happen too often these days. Didn’t test for all drivers/use cases but the example below is in a VMWARE environment. It has to be handled as scsi by the kernel obviously. check for current disks: