As mentioned by @trevor62590, the issue is the ionic keyboard plugin only calculates the scrollable view height for the one input field only. When you navigate to the next field, the plugin does not expand the scroll height.
My workaround is just disabling keyboard scroll
cordova.plugins.Keyboard.disableScroll(true);
That way we can keep the native scrolling and the accessory bar.
Reference: Click a input field. whole app jumps down and back to the original place