1+ second lag when I switch the focused window between monitors

[knoa]

When I switch the focused window on a different monitor, the mouse cursor will get stuck for 1+ seconds, even if both windows are on native displays.
If I quit the DisplayLink Manager, the lag disappears.

How can I avoid the issue?

--
MacBook Pro 2018 15" (In clamshell mode)
macOS Sonoma 14.6.1 (Latest)
Native monitors: 3840x2160 + 1920x1080 +1600x1200
DisplayLink monitor: 1024x768 (Samsung SyncMaster 152X)
DisplayLink Manager 1.10.3 (Build 34) (Latest)

keywords for search:
lag, laggy, delay, stuck, halt, stop, freeze
mouse cursor, keyboard input
monitor, display, screen, window
between, another, switch, focus, activate, click, change