This doesn’t make any difference, as you already discovered, because all directives are loaded by default (and this is currently the expected behavior):
https://github.com/driftyco/ionic/issues/6092
AFAIK currently there’s no way to achieve this. But your idea is very valid, especially for the mobile web. I’m also looking forward for such a possibility because it will make the use of the Ionic framework in (mobile) web applications reasonable (and justifiable).
Check out the following post for more details about the Ionic’s PWA efforts: