If you used "displaylink-debian" to install it and if the driver dir is still there, you can do this by just running the script and selecting "uninstall".
Otherwise, get the version of driver you installed, extract the source, i.e:
Code:
./displaylink-driver-1.0.138.run --keep --noexec
cd displaylink-driver-1.0.138/
sudo ./displaylink-installer.sh uninstall