what you’re describing is what I got when I tried to used with the screen off. Maybe you can try to plug in a screen. I looked into this problem and there are ways to demand Xorg to keep drawing the screen all the time, but I found it hard to achieve and hence overhead for my case
Very interesting! thanks for sharing :)