Are you maybe just giving too much grace? I do that thing where i basically accept it when someone’s behaviour makes sense given their circumstances, but it is still negatively affecting me and eventually they do something a bit too far and can no longer “hide” behind material reality having shaped that behaviour. Then the pent up annoyedness and the valid reaction to that situation gets released at once.

Of course they are not actually hiding or excusing their behaviour, but that’s the word that came to me.