If you're running Debian lenny (testing) and since the last upgrade it seems that Xfce freezes randomly, you're not crazy, it's a real bug (#449050 and #449134 already reported). xfwm4 freezes in certain condition with gtk 2.12 wich has entered testing yesterday.

xfwm4 4.4.1-3 in unstable fixes the problem, and it'll reach testing really soon now (it just built on arm and should transition in the next days). In the meanwhile you should be able to take the unstable package and install it on testing without problem.

Sorry for the mess, from the pkg-xfce team.