django / makemessages / slow
Django makemessages can be quite slow on larger projects.
$ time python ../manage.py makemessages -lnl -ddjango
processing language nl
real 0m8.203s
user 0m2.670s
sys 0m5.763s
Why does it take so long? Well, it’s system call heaven:
$ strace -f python ../manage.py makemessages -lnl -ddjango \
>tmp.log 2>&1
$ sed -e 's/(.*//;s/^\[[^]]*\] //;/^ \?</d;/,/d;/^+/d' tmp.log |
sort | uniq -c | sort -n | tail -n10
10893 rt_sigaction
16179 stat
16819 fcntl
22875 access
27833 read
32469 open
33650 fstat
40891 mprotect
69181 mmap
1267039 close
For every file, a call to xgettext(1) is made.
Can we speed this up?
Yes we can, by calling xgettext with 100 files at once. And by not copying the javascript files before reading them. (That second change has been in Django master since November 11 2014.)
$ cdsitepackages
$ patch -p0 < ~/django-1.4.15-makemessages-speedup.patch
patching file django/core/management/commands/makemessages.py
$ cd -
/PATH/TO/PROJECT
$ time python ../manage.py makemessages -lnl -ddjango
processing language nl
real 0m1.505s
user 0m1.320s
sys 0m0.183s
We now only do a fraction of the system calls:
635 mprotect
1166 getdents
1992 munmap
2483 lstat
2946 mmap
3422 fstat
5664 read
7590 stat
7859 open
20565 close
That’s much better, with 61 times fewer calls to close(2) calls and 4
times fewer calls to open(2).
Note that that’s especially important for Docker users, where the speedup can be as dramatic as from 120 to less than 8 seconds.
Get the the patch for Django 1.4.15: django-1.4.15-makemessages-speedup.patch (view)
Extra tip: use venvpatch(1) to conveniently patch your virtual env.