Interestingly enough the slowdown appears to be caused by the modesetting driver the new Xorg uses for Intel cards by default!
I explicitly configured the old driver in my xorg.conf and it seems to work fine again!
Section "Device" Identifier "Card0"
Driver "intel"
BusID "PCI:0:2:0"
EndSection
Apparently there is some bad interaction between the modesetting and DisplayLink drivers. This should be addressed as the modesetting thing is now default in many distributions.