Blog

Blog

salt master losing children

I recently set up psdiff on a few of my servers as a basic means to monitor process activity. It disclosed that my SaltStack master daemon — which I’m running as a non-privileged user — was losing a single child, exactly 24 hours after I had ran salt commands. This seemed to be a recurring phenomenon. The salt server — version 0.17.5+ds-1 on Ubuntu Trusty — was running these processes:

Read more

polyglot xhtml

Polyglot XHTML: Serving pages that are valid HTML and valid XML at the same time. A number of documents have been written on the subject, which I shall not repeat here. My summary: HTML5 is not going away. XHTML pages validate in the browser. If you can get better validation during the development of your website, then you’ll save yourself time and headaches. Thus, for your development environment, you’ll set the equivalent of this:

Read more

Availability during holiday december 2015

From 24th of December we are on leave and return to the office on the 4th of January. During this period we are available 24/7 for incident response and other urgent matters as usual. If you already know of any urgent requests which needs to be handled during this period, please inform us in advance so we can plan the required availability.

Read more

Planned maintenance - router upgrade RUG RH POP (01:00-04:00 8 DEC 2015)

In the night of Monday (Dec. 7th) to Tuesday (Dec. 8th) between 1:00 and 6:00 we will upgrade the router at our RUG Rekenhal POP. Impact is limited to IP Access locations and IP Transit customers on this POP. ####Maintenance window 01:00-04:00 on 8th of December 2015. ####Description We will upgrade the router to allow planned network upgrades. Impact The router at the RUG Rekenhal POP will be unavailable for 30min-60min.

Read more

asterisk / editline / key bindings

Getting the Asterisk PBX CLI to work more like you’re used to from the (readline) bash shell can, be a time-saver. For example, you may want reverse-i-search (^R), backward word deletion (^W) and word skipping (^<arrow-left> and ^<arrow-right>). It can be done, but you must configure the editline library in a similar manner as you would configure .inputrc. Support for the .editrc configuration file was added in May 2011 (git commit d508a921).

Read more

Planned maintenance virtual servers TCN (01:00-06:00 27 NOV 2015)

In the night of Thursday (Nov. 26th) to Friday (Nov. 27th) between 1:00 and 6:00 we will perform maintenance to the virtual server infrastructure at location TCN. ####Maintenance window 01:00-06:00 on 27th of November 2015 ####Description Virtual server infrastructure will be upgraded to a new major software release. Due to the major version upgrade and incompatibility between versions servers will experience downtime for 15-60 minutes Impact Virtual servers will be offline for ~15 minutes (up to 60 minutes worst case).

Read more

encfs / recursion into itself

We wanted to use EncFS to be able to store encrypted backups. The requirements for that are: The backup server initiates the backup. That’s where we configure which hours are safe (resource wise) and which files need backing up (etc, home, root, srv, …). And it means the backup server can safely be placed behind a gateway disallowing all incoming connections. The backup server cannot know the passwords of files.

Read more

encfs / configure / libboost

I ran into an obscure Could not link against ! error when configuring EncFS: ~/src$ apt-get source encfs ... ~/src$ cd encfs-1.7.4/ ~/src/encfs-1.7.4$ ./configure ... configure: WARNING: BOOST_CPPFLAGS -I/usr/include checking whether the Boost::Serialization library is available... yes configure: error: Could not link against ! That’s odd. And not immediately obvious how to fix. For starters we need all the dependencies that Debian defines: ~/src/encfs-1.7.4$ sed -e '/^Build-Depends: /!d;s/^[^:]*: //;s/([^)]*)//g;s/,//g' \ debian/control debhelper librlog-dev librlog5 libfuse-dev libssl-dev pkg-config libboost-serialization-dev libboost-filesystem-dev quilt dh-autoreconf ~/src/encfs-1.

Read more

core router service disruption [UPDATED 5 Nov 2015]

Service disruptions on one of the core routers location (CR1) in TCN. 5 Nov 2015 - 01:00 CR1 Supervisor placement Following up the maintenance finished on Monday, we will add an extra Router Supervisor in CR1 for extended redundancy in case of main Supervisor failure. This maintenance will be carried out tonight, at 01:00 on the 5th of November. There is no expected impact. This maintenance is a followup on the hardware replacement of the Supervisor in CR1.

Read more

Planned network maintenance (01:00-03:00 6 NOV 2015)

In the night of Thursday (Nov. 5th) to Friday (Nov. 6th) between 1:00 and 3:00 we will perform network maintenance at our TCN co-location. ####Maintenance window 01:00-03:00 on 6th of November 2015 ####Description We will perform changes in the network configuration which will cause a change of mac address of the gateway for each subnet. Impact Depending on the device it may take some time to pick up this change. Generally, busy servers will pick up the changes almost instantly and servers which are mostly idle may take a while.

Read more

scapy / dns server / snippet

A few days ago, the Scapy project was brought to my attention. Scapy is an internet packet manipulation library for Python2. It can be used to sniff and decode packets, or to generate your own custom packets. In the most basic form, it runs on raw sockets, sniffing and decoding traffic like tcpdump. See the sniff() examples and the send(IP(dst="1.2.3.4") / ICMP()) example for sending a simple packet. But just as easily, it works on regular datagram sockets — those that you don’t need CAP_NET_RAW powers for.

Read more

flake8 / vim / python2 / python3

To syntax check Python code before executing, I use flake8. And when coding in the Vim editor, I use the vim-flake8 plugin that allows me to hit <F7> to quickly check for errors in the file I’m currently working in. But, there are currently two common flavors of Python: python2 and python3. And therefore flake8 comes in two flavors as well — you guessed it — a python2 and a python3 flavor.

Read more

python / subprocess / winch

While I was writing a Python tool to wrap C Gdb so I could fetch some info out of it automatically, I ran into the issue that it reads the terminal size (lines x columns) to adjust its output. I wanted consistent machine readable output, so I enlarged the terminal size programmatically: now row based output would not get wrapped by Gdb. Later I noticed that it would cease to use the terminal size — in fact, use the default 80 columns — if I also redirected stderr to a non-tty.

Read more

debian / packaging asterisk 13

As of this writing, Debian testing (stretch) contains Asterisk version 13.1.0. The Debian source as GIT repository is here: https://anonscm.debian.org/git/pkg-voip/asterisk.git (browse) Packaging a newer version is not that hard, if we start out with the debian/ directory kindly supplied by the Debian maintainers. Hints to get things running: Use a local git repository By using a local git repository in your unpacked Asterisk dir, you can quickly restart from scratch any time you mess anything up.

Read more

on-the-fly encrypted backups

I was wondering how easy it was to encrypt files before rsyncing them away to the backup machine. A quick search turned up the suggestion to use encfs by the user Thor on ServerFault. That looks like a decent solution. Let’s figure out if it meets our needs. The idea is that we do this: # mount read-only encrypted virtual copy of unencrypted local data: encfs --reverse -o ro ~/data/ ~/.

Read more

monitoring / process open files / limit

Here, an awesome shell one-liner to find which process uses the most files, relative to its max-open-files soft limit. $ for x in /proc/[0-9]* do fds=0 max=`awk '/^Max open files/ {print $4}' $x/limits 2>/dev/null` && for t in $x/fd/*; do fds=$((fds+1)); done && test "${max:-0}" -gt 0 && echo $((fds*100/max)) ${x##*/} done | sort -rn | while read l do pid=${l##* }; echo "$l`readlink /proc/$pid/exe`"; break; done 57 16674 /usr/lib/dovecot/imap-login So, my imap-login (pid 16674) apparently uses 57% percent of its allowed max open files.

Read more

converting unprintable pdf / imagemagick

Okay, so we all know that printers are sent from hell, but we still need to use them from time to time. Today, we were trying to print a PDF document with bar codes on it. Amazingly enough, the text on the PDF looked fine, but the bar codes (images) appeared as if they were wrapped at the wrong place. Luckily, convert(1) from ImageMagick came to the rescue: $ convert -density 300 -define pdf:fit-page=A4 input.

Read more

proxmox / resource usage

As I mentioned the other day, my VM was slow, so I needed a way to figure out which VM guests were causing the heavy load on our Proxmox platform. I hacked up proxtop to enumerate the top resource users: $ ./proxtop -t day proxmox.example.com monitor@pve Password:<enter password> SORTED BY: cpu, avg ... SORTED BY: diskread, avg ------------------ #0: 3.1 MiB/s pve10 (acme-bugs-bunny) #1: 1.3 MiB/s pve07 (customerX-private) #2: 992.3 KiB/s pve10 (acme-road-runner) .

Read more

proxmox api / python module

So, my VM was slow, and I needed to know which VM guest was eating all the resources. These VM containers are all managed by Proxmox; which is great, but it doesn’t show which VM guest is eating all the resources. Luckily, Proxmox provides an API to get that info. The docs pointed to two API modules for Python, my language of choice for these kinds of jobs: proxmoxer and pyproxmox.

Read more

zabbix api / python module

Today, my choice of Python modules to Interface with Zabbix. They are all pretty similar, so that made it harder to choose. Here the six modules, as mentioned on the Zabbix wiki are, in the order of my preference. Note that second and third came close, but I favor clean documented code and fewer dependencies. The last ones didn’t get tested because of my Python3 requirement. zabbix-client # pip: zabbix-client # pep: 99% # last-update: Aug.

Read more

asterisk / dialplan / variable expansion / security

Even after writing plenty of Asterisk PBX dialplan, I occasionally get bitten by the unintuitiveness of the parser. A few rules to avoid mistakes, are: Always use double quotes on no side of the expression, or better yet, on both if there is a chance that the value is empty: $[${HANGUPCAUSE}=17] or $["${value_which_may_be_empty}"="somevalue"] Otherwise try to avoid double quotes (and semi-colons, and backslashes) whenever possible. If you need to escape them, it’s too easy to get it wrong.

Read more

GHOST: glibc gethostbyname buffer overflow

A high risk security issue in glibc was disclosed last night. Because of the potential high impact we started our emergency patch procedures for osso managed environments and notify customers with self managed environments. Ghost vulnerability details Qualys discovered a buffer overflow in dns resolve functions in the GNU C library (glibc). They created a proof of concept exploit for exim and dubbed the vulnerability "GHOST". All processes that might do dns lookups are susceptible to attacks when using a vulnerable glibc version.

Read more

gitlab / upgrade / ruby / bundle

While we do Python VirtualEnv stuff every day, we rarely do Ruby environments. And after the Ubuntu dist-upgrade, the Ruby dependencies for our GitLab were broken — as was expected. This happens for Python pip installed packages as well. They’re linked against older system libraries, which have been removed by the upgrade. How to fix the Gitlab dependencies? Browse through the upgrade docs to find a bundle install command. # cd /home/git/gitlab # sudo -u git -H bundle install \ --without development test postgres --deployment # for MySQL That did… absolutely nothing — again, as was expected.

Read more

fail2ban / started / e-mail / disable

Tired of the Fail2ban start and stop e-mails? Especially after a manual fail2ban restart, the [Fail2Ban] vsftpd: stopped on HOSTNAME and [Fail2Ban] vsftpd: started on HOSTNAME mail tuple is too spammy. Quick fix to disable them: Create a new file, named /etc/fail2ban/actions.d/sendmail-no-start-stop.local: diff --git /etc/fail2ban/action.d/sendmail-no-start-stop.local /etc/fail2ban/action.d/sendmail-no-start-stop.local new file mode 100644 index 0000000..cb7ecb9 --- /dev/null +++ /etc/fail2ban/action.d/sendmail-no-start-stop.local @@ -0,0 +1,3 @@ +[Definition] +actionstart = +actionstop = And — you’re using mta = sendmail right?

Read more

git / gnutls / handshake failed / nginx ciphers

When trying to keep up with all the TLS/SSL security changes, you need to modify your nginx config every now and then. The good TLS config may look like this: # nginx.conf: http { ssl_certificate /etc/ssl/MY_DOMAIN.pem; ssl_certificate_key /etc/ssl/MY_DOMAIN.key; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA; ssl_session_cache shared:SSL:5m; ssl_session_timeout 5m; ssl_prefer_server_ciphers on; add_header Strict-Transport-Security "max-age=15768000; includeSubDomains"; And the above config is accompanied by a fairly good A grade from the Qualys SSL Labs Analyzer.

Read more