I have a feeling it’s the plugin overhead, as I’ve used Google Maps with custom directives in Ionic before with no issues. I guess it depends on what you’re trying to do as well.
How does it perform if you just embed a map in your project and build with Cordova?
EDIT: Thinking about it now, there was a noticeable speed difference on Android < 4.4 using the Google Maps Javascript API. Looks like there is a plugin: https://github.com/imhotep/MapKit that lets you use the Android API. Never used it before, but in theory should offer some improved performance.