Announcing the Capacitor ML Kit Barcode Scanning Plugin

I’m very excited to introduce you to the brand new Capacitor ML Kit Barcode Scanning plugin. This plugin is part of the new Capacitor ML Kit project by Capawesome, which aims to bring the powerful ML Kit SDKs to Capacitor.

The plugin allows you to scan and decode various types of barcodes, including QR codes and UPC codes. For a complete list of supported barcodes, see BarcodeFormat. The scanning is done directly on the device and does not require a network connection. The plugin supports Android and iOS, and it allows multiple barcodes to be scanned at once. It also has torch and autofocus support, and an optional ready-to-use interface without the need for webview customizations.

Here you can find the full blog post: Announcing the Capacitor ML Kit Barcode Scanning Plugin - Capawesome

Let me know if you have any questions!

5 Likes

Finally and thank you for your contributions!!!

1 Like

Great job!
But it seems me a bit unstable.
I couldn’t run it using --livereload and the scann doesn’t work.
Please, which version of NodeJS, Ionic and Capacitor do you recommend to have a best result?
Thanks!

Both should work fine, livereload as well as the two scan methods. Please check out the demo app: GitHub - robingenz/capacitor-mlkit-plugin-demo: ⚡️ Simple Ionic Angular app to demonstrate the use of certain Capacitor ML Kit plugins.
Feel free to create a GitHub issue with a reproducible example and I’ll take a closer look.

1 Like

Can I use Capacitor 4.7 for that?

Yes, if you use plugin version 0.0.3:

npm i  @capacitor-mlkit/barcode-scanning@0.0.3

does it work for ios platform?

The plugin supports Android and iOS, and it allows multiple barcodes to be scanned at once.

Yes

Hi. Could you help me again. I tried using version 0.0.3 and it works however I have an error for samsung S21.

Android version 13

When I try to scan a barcode I have an error

scan failed.
com.google.mlkit.common.MlKitException: Failed to scan code.
at com.google.mlkit.vision.codescanner.internal.zze.zzc(com.google.android.gms:play-services-code-scanner@@16.0.0-beta3:4)
at com.google.mlkit.vision.codescanner.internal.zzf.onActivityResult(com.google.android.gms:play-services-code-scanner@@16.0.0-beta3:5)
at androidx.activity.result.ActivityResultRegistry.doDispatch(ActivityResultRegistry.java:409)
at androidx.activity.result.ActivityResultRegistry.dispatchResult(ActivityResultRegistry.java:366)
at androidx.activity.ComponentActivity.onActivityResult(ComponentActivity.java:712)
at android.app.Activity.dispatchActivityResult(Activity.java:8951)
at android.app.ActivityThread.deliverResults(ActivityThread.java:5987)
at android.app.ActivityThread.handleSendResult(ActivityThread.java:6033)
at android.app.servertransaction.ActivityResultItem.execute(ActivityResultItem.java:67)
at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2574)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8757)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)

Probably something happens here

  static void zzc(@Nullable Barcode var0, int var1) {
        Pair var2 = (Pair)zza.getAndSet((Object)null);
        if (var2 != null) {
            if (var0 != null) {
                ((TaskCompletionSource)var2.first).setResult(var0);
            } else if (var1 == 201) {
                ((CancellationTokenSource)var2.second).cancel();
            } else {
                Object var3 = var2.first;
                TaskCompletionSource var4 = (TaskCompletionSource)var3;
                MlKitException var5 = new MlKitException("Failed to scan code.", var1);
                var4.setException(var5);
            }
        } else {
            Log.e("GmsBarcodeScannerImpl", "Scanning task source doesn't exist when setting back result.");
        }
    }

Also tried using iPhone 11 and Samsung A70 and it works for both

It seems like it’s related to this open issue: Google Issue Tracker

There is a workaround mentioned in this thread. I will give it a try as soon as i have some time. If it works, I will release an update. I have created a new GitHub issue for this: bug: · Issue #38 · capawesome-team/capacitor-mlkit · GitHub

In the meantime, you can use the startScan(…) method. This works without any problems.

I will try, thank you a lot

Hello. As for startScan() method. Do we need to have some additional configuration to use it?

My method is

    await BarcodeScanner.requestPermissions();

    document.querySelector('body')?.classList.add('barcode-scanner-active');

    const listener = await BarcodeScanner.addListener(
      'barcodeScanned',
      async result => {
        console.log(result.barcode);
      },
    );

When I call it the error is showed up

startScan failed.
java.lang.NullPointerException: Attempt to invoke virtual method ‘androidx.camera.core.Preview$SurfaceProvider androidx.camera.view.PreviewView.getSurfaceProvider()’ on a null object reference
at io.capawesome.capacitorjs.plugins.mlkit.barcodescanning.BarcodeScanner.lambda$startScan$0$io-capawesome-capacitorjs-plugins-mlkit-barcodescanning-BarcodeScanner(BarcodeScanner.java:100)
at io.capawesome.capacitorjs.plugins.mlkit.barcodescanning.BarcodeScanner$$ExternalSyntheticLambda8.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:942)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8757)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)

No, it should work without any modifications.
Can you please provide a minimal, reproducible example so i can debug the issue?
Feel free to use this app template: GitHub - capawesome-team/.capacitor-app: ⚡️ Capacitor app template to create a minimal, reproducible example.

2 Likes

yeah, The method looks like

async getBarcodeForAndroid(): Promise<string> {
    await BarcodeScanner.requestPermissions();

    document.querySelector("body")?.classList.add("barcode-scanner-active");

    const listener = await BarcodeScanner.addListener(
      "barcodeScanned",
      async (result) => {
        console.log(result.barcode);
      }
    );
}

When I call it the error is occured

message: “Attempt to invoke virtual method ‘androidx.camera.core.Preview$SurfaceProvider androidx.camera.view.PreviewView.getSurfaceProvider()’ on a null object reference”

startScan failed.
java.lang.NullPointerException: Attempt to invoke virtual method ‘androidx.camera.core.Preview$SurfaceProvider androidx.camera.view.PreviewView.getSurfaceProvider()’ on a null object reference
at io.capawesome.capacitorjs.plugins.mlkit.barcodescanning.BarcodeScanner.lambda$startScan$0$io-capawesome-capacitorjs-plugins-mlkit-barcodescanning-BarcodeScanner(BarcodeScanner.java:100)
at io.capawesome.capacitorjs.plugins.mlkit.barcodescanning.BarcodeScanner$$ExternalSyntheticLambda8.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:942)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8757)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1067)

it shows me this method

 public void startScan(ScanSettings scanSettings, StartScanResultCallback callback) {
        // Stop the camera if running
        stopScan();
        // Hide WebView background
        hideWebViewBackground();

        this.scanSettings = scanSettings;

        BarcodeScannerOptions options = buildBarcodeScannerOptions(scanSettings);
        barcodeScannerInstance = BarcodeScanning.getClient(options);

        ImageAnalysis imageAnalysis = new ImageAnalysis.Builder().setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build();
        imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(plugin.getContext()), this);

        ListenableFuture<ProcessCameraProvider> cameraProviderFuture = ProcessCameraProvider.getInstance(plugin.getContext());
        cameraProviderFuture.addListener(
            () -> {
                try {
                    processCameraProvider = cameraProviderFuture.get();

                    CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(this.scanSettings.lensFacing).build();

                    previewView = plugin.getActivity().findViewById(R.id.preview_view);

                    Preview preview = new Preview.Builder().build();

preview.setSurfaceProvider(previewView.getSurfaceProvider());// This place shows the error

// Start the camera
                    camera =
                        processCameraProvider.bindToLifecycle((LifecycleOwner) plugin.getContext(), cameraSelector, preview, imageAnalysis);
                    callback.success();
                } catch (Exception exception) {
                    callback.error(exception);
                }
            },
            ContextCompat.getMainExecutor(plugin.getContext())
        );
    }

Samsung S21,
Android version 13
@capacitor-mlkit/barcode-scanning”: “^0.0.3”

Please create a minimal, reproducible example using the app template so I can reproduce your issue.

plugin.getActivity().findViewById(R.id.preview_view)–>function is returns always null value

cameraProviderFuture.addListener(
() → {
try {
processCameraProvider = cameraProviderFuture.get();

                CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(this.scanSettings.lensFacing).build();
        
                previewView = plugin.getActivity().findViewById(R.id.preview_view);

                  // previewView.setScaleType(PreviewView.ScaleType.FILL_CENTER);

                  Preview preview = new Preview.Builder().build();
                  preview.setSurfaceProvider(previewView.getSurfaceProvider());

                  // Start the camera
                  camera =
                    processCameraProvider.bindToLifecycle((LifecycleOwner) plugin.getContext(), cameraSelector, preview, imageAnalysis);

                callback.success();
            } catch (Exception exception) {
                callback.error(exception);
            }
        },
        ContextCompat.getMainExecutor(plugin.getContext())
    );

robingenz - we are getting an error when updating v5.0.0 to v5.0.3.

We tried the pod repo update with no success. We found this in github issues: bug(barcode-scanning): Cocoapods dependency conflicts between MLKit and Firebase · Issue #23 · capawesome-team/capacitor-mlkit · GitHub. We are using the @capacitor-firebase as well. It also did not work.

How to resolve?

You need to run „pod install —repo-update“ in the „iOS/App/„ directory of your project as described in the error message. This definitely works.

Still not working :frowning_face:

heres what my podfile looks like:

LGTM. In this case please provide a Minimal, Reproducible Example using this template in a public GitHub repository so I can debug the issue.