I had the same questions as you. I used to ship XWalk with my app too but finally I don't do it anymore because my app target user using Android >= 5
I tested my app on actualized Android 6 and Android 7 real phone and the overall performance were still cool and as @rapropos noticed, the app binary size is really smaller now.
But each app are different, consider having some tests on real phones too first
P.S.: If your app is already in store and if you decide to remove XWalk and if after you push your new version to store you notice that your clients aren't updated, no worries, the solution is there https://forum.ionicframework.com/t/solved-android-store-update-successfully-pushed-but-not-noticed-on-devices