Moving from ionic 3 to 4 should not be a big deal - just check the breaking changes doc on their repo. I did so earlier and wasnt a big problem.
I think what could become a concern is the angular upgrades you need. Maybe you should look those breaking chances as leading element for considering app migration instead of Ionic. And of course all other plugins and libraries you are dependent on who rely on certain versions of angular. Some libs require quite an overhaul of the app going from angular a to angular b.
Ionic 4 to 5 should be a breeze then as well - for their part of course only
Both 4 and 5 have stable versions and ionic 6 now has an rc - making the api stable too