Recently we were bitten by an old kernel bug on Ubuntu that only rarely triggers. Finding out where the problem was is easier if you know where to look.
We had no kernel logs to go on. Only a hanging machine with no output. And a hunch that the running kernel version linux-image-5.4.0-122 was the likely culprit. Was there another way than meticulously reading all changelogs and changes to find out which bug we're dealing with?
For issues that are already fixed in a newer kernel, the following recipe seems to work. (Note that we would like to use commit ids instead of string matching, but this won't work because Ubuntu cherry-picks the commits, altering the commit ids.)
Step 1: get the Ubuntu kernel repo:
$ git clone git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal \ linux-ubuntu-focal
Step 2: check which versions have been released:
$ git -c tag.sort=-v:refname tag | grep Ubuntu-5.4.0 Ubuntu-5.4.0-144.161 Ubuntu-5.4.0-143.160 ... Ubuntu-5.4.0-123.139 Ubuntu-5.4.0-122.138 Ubuntu-5.4.0-121.137 Ubuntu-5.4.0-120.136 ...
Step 3: pick which versions we want to check. In this case, we're
assuming that the bug was introduced in
we'll plug in
Ubuntu-5.4.0-121.137..Ubuntu-5.4.0-122.138. And we'll
Ubuntu-5.4.0-122.138..Ubuntu-5.4.0-144.161 so we get all
versions to the end.
$ git log Ubuntu-5.4.0-121.137..Ubuntu-5.4.0-122.138 --format=%s | grep -v ^UBUNTU: | while read -r l; do res=$( git log Ubuntu-5.4.0-122.138..Ubuntu-5.4.0-144.161 \ --grep "Fixes:.*$(printf '%s' "$l" | sed -e 's/[^A-Za-z0-9 ]/./g')" \ --oneline) test -n "$res" && echo "$res (fixes: $l)" done 97842ea930e0 tcp: make sure treq->af_specific is initialized (fixes: tcp: md5: incorrect tcp_header_len for incoming connections)
$ git show 97842ea930e0 commit 97842ea930e0eb94bfdb87beaf87d56224c1e8ad Author: Eric Dumazet <firstname.lastname@example.org> Date: Sun Apr 24 13:35:09 2022 -0700 tcp: make sure treq->af_specific is initialized ... Fixes: 5b0b9e4c2c89 ("tcp: md5: incorrect tcp_header_len for incoming connections")
Nice. One result. Seems to be the one we're looking for.
$ git -c tag.sort=-v:refname tag --contains 97842ea930e0 | tail -n1 Ubuntu-5.4.0-123.139
And it was included in the next release already.