Had this issue for a long time, and found a way to work around it.
1. Install DisplayPlacer
```sh
brew tap jakehilborn/jakehilborn && brew install displayplacer
```
2. Configure the displays to be on the definitive arrangement you want it to be
3. Run `displayplacer list`. This will print a long output but what matters to us is at the end:
```
...
mode 47: res:3360x1890 hz:60 color_depth:4 scaling: on
mode 48: res:3840x2160 hz:60 color_depth:4 scaling: on
mode 49: res:4096x2160 hz:60 color_depth:4 scaling: on
Execute the command below to set your screens to the current arrangement:
displayplacer "id:65CF52D2-4D65-9866-C792-BAE1233F7DFE res:1680x1050 color_depth:4 scaling

n origin

0,0) degree:0" "id:ED61CFA3-165E-4A21-2F94-B100084444A1 res:1080x1920 hz:60 color_depth:8 scaling

ff origin

3600,-840) degree:90" "id:0000410C-0000-08CC-0000-0CEE00000000 res:1920x1200 hz:60 color_depth:4 scaling

ff origin

1680,-1200) degree:0" "id:00004C2D-0000-0D20-4357-413400000000 res:1920x1080 hz:60 color_depth:4 scaling

ff origin

1680,0) degree:0"
```
Now we can use the command provided there to restore the display arragement.
4. Save the last emitted line to a script and run that script whenever the screens are misconfigured.
I've linked this script to a shortcut so whenever screens are out of place I press the shortcut and they automagically restore themselves.