So my newly upgraded from 2012 R2 DPM kept failing on doing backups of some VMs which have the new RCT style backup capabilities.
After quite a lot of troubleshooting, I found this article which solved the problem (thanks!):
https://jtpedersen.com/2016/12/dpm-2016-backup-failes-when-creating-a-new-backup-of-virtual-machine-with-generic-error/?unapproved=42564&moderation-hash=8f4d2b7ae2e586925bc52aa315606379#comment-42564