For little GUI-changes in own ArcMap-customizations normally it is enough to start the application and look that every entry is visible. Especially for refactoring of an own toolbar it was still enough to verify that the toolbar is already shown in the choice-list.

But this is a fallacy: the entries there comes from a cache. You could verify this by dumping informations in the toolbar constructor. The constructor is only called if the toolbar is activated! Otherwise you see the toolbar name, but this name comes out of a cache.

The cache is stored in the registry under:

HKCU\Software\ESRI\ArcMap\Settings\CommandBarNameCache