This works fine on most sites, but in
Document Center sites, where the document ID feature is already activated document sets in the site suddenly show the web parts on the welcome page twice. I debugged it, and it turns out that this happens when the document ID feature is the culprit. However, this doesn't happen if I activate the ID feature using powershell.
My only solution at the moment is to check if the feature is already activated before activating it. This stopped the problem, but doesn't explain it.