django / makemessages / slow

django / makemessages / slow

  • Written by
    Walter Doekes
  • Published on

Django makemessages can be quite slow on larger projects.

$ time python ../ makemessages -lnl -ddjango
processing language nl

real    0m8.203s
user    0m2.670s
sys     0m5.763s

Why does it take so long? Well, it’s system call heaven:

$ strace -f python ../ makemessages -lnl -ddjango \
    >tmp.log 2>&1
$ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log |
    sort | uniq -c | sort -n | tail -n10
  10893 rt_sigaction
  16179 stat
  16819 fcntl
  22875 access
  27833 read
  32469 open
  33650 fstat
  40891 mprotect
  69181 mmap
1267039 close

For every file, a call to xgettext(1) is made.

Can we speed this up?

Yes we can, by calling xgettext with 100 files at once. And by not copying the javascript files before reading them. (That second change has been in Django master since November 11 2014.)

$ cdsitepackages
$ patch -p0 < ~/django-1.4.15-makemessages-speedup.patch
patching file django/core/management/commands/
$ cd -
$ time python ../ makemessages -lnl -ddjango
processing language nl

real    0m1.505s
user    0m1.320s
sys     0m0.183s

We now only do a fraction of the system calls:

    635 mprotect
   1166 getdents
   1992 munmap
   2483 lstat
   2946 mmap
   3422 fstat
   5664 read
   7590 stat
   7859 open
  20565 close

That’s much better, with 61 times fewer calls to close(2) calls and 4 times fewer calls to open(2).

Note that that’s especially important for Docker users, where the speedup can be as dramatic as from 120 to less than 8 seconds.

Get the the patch for Django 1.4.15: django-1.4.15-makemessages-speedup.patch (view)

Extra tip: use venvpatch(1) to conveniently patch your virtual env.

Back to overview Newer post: uuid / storage / mysql Older post: photo exif timestamp / filesystem mtime