So I don't forget it, since recent sudo package resets environment even for $HOME, the working sudoers (5) for me is:
Cmnd_Alias PBUILDER = /usr/sbin/pbuilder,
/usr/sbin/cowbuilder
Defaults!PBUILDER env_keep+=HOME
You may have to adjust the commands, YMMV.