Working with WebRTC in Ionic

Hi I am creating a VoIP application in ionic. I am using WebRTC which asks for permissions to access camera and microphone. It works perfectly fine when I run my web app in browser.

I have written code to read permissions from user in Android as well. It asks for permissions to user, yet even after allowing all permissions, I am getting error of not able to access media devices.

I have also tried to allow permissions manually to my app from settings. Yet, I am unable to resolve this.

I am not sure if I need to allow browser related permissions as well for Android?

I have really invested a lot of time into this, and would really appreciate if anyone can help me with it. Thanks

1 Like
import { NgxPermissionsService } from 'ngx-permissions';
………
perm = ["ADMIN", "MEDIA", "VIDEO_CAPTURE", "AUDIO_CAPTURE"];
.......
this.permissionsService.hasPermission(this.perm).then(success => {
:::::  



[app.module.ts]


:::;
imports:
...
NgxPermissionsModule.forChild({
      permissionsIsolate: true,
      rolesIsolate: true}),

It didn’t solve my issue. It is still throwing error: “Failed to get local media device”. Even though I has all permissions

@mhartington Hi Mike, Could you look into this? Would really appreciate. Thanks!

ngOnInit() {
    
 
    this.permissionsService.loadPermissions(this.perm)
   
  }

This code probably help you…

Thanks for your reply @peterkhang , but permissions are already granted. That is not the issue. I also tried allowing permissions manually from settings for my app.
Problem is that I am trying webRTC in my ionic project. Not sure if It is looking for web permissions even in android?

webrtc is web browser protocol : This protocol needs web browser permission. I have tested it with kurento-tutorital-java/kurento-hello-world-recording demo server.

<ion-header>
  <ion-toolbar color="--bory-blue">
    <ion-buttons slot="start">
      <ion-menu-button color="light"></ion-menu-button>
    </ion-buttons>
    <ion-title>
      SpeechMate::{{auth.title}}
    </ion-title>
  </ion-toolbar>
</ion-header>

<ion-content padding>
  <div class="videoIn">
    <div class="col-md-5">
      <h3>Local stream</h3>
      <video id="videoInput" autoplay width="480px" height="360px"
        poster="{{auth.root}}assets/png/webrtc.png"></video>
    </div>
  </div>

  <div class="row">
    <ion-grid>
      <ion-row>
        <ion-col size="12">
          <ion-button class="start" color="white" (click)='startAfterCheckPermission()' color="primary" *ngIf="status === NO_CALL">start</ion-button>    
          <ion-button class="start" color="white" (click)='stop()' color="danger" *ngIf="status === POST_CALL || status === IN_CALL || status === IN_PLAY">stop</ion-button>     
          <ion-button class="start" color="white" (click)='play()' color="primary" *ngIf="status === NO_CALL">play</ion-button>
        </ion-col>
      </ion-row>
    </ion-grid>     
  </div>
  <div class="videoOut">
    <h3>Remote stream</h3>
    <video id="videoOutput" autoplay width="480px" height="360px"
      poster="{{auth.root}}assets/png/webrtc.png"></video>
  </div>
  <div class="row">
    <div class="col-md-12">
      <label class="control-label" for="console">Console</label><br>
      <br>
      <div id="console" class="democonsole">
        <ul></ul>
      </div>
    </div>
  </div>

  <div id="layer" *ngIf="show"><img src="{{auth.root}}assets/imgs/loading.gif"/></div>

</ion-content>


import { Component, OnInit,ViewChild,ElementRef, HostListener, NgZone  } from '@angular/core'
import { AuthServiceProvider } from 'src/app/providers/auth-service/auth-service'
import { v4 as uuidv4 } from 'uuid'
import { environment } from 'src/environments/environment'
import { WebRtcPeer } from 'kurento-utils'
import { QueueingSubject } from 'queueing-subject'
import { Subscription, Observable, timer } from 'rxjs'
import { Timer } from 'interval-timer'
import { share, switchMap, retryWhen, delay, map } from 'rxjs/operators'
import { PushEvent, Status } from 'src/app/providers/common/common'
import { AndroidPermissions } from '@ionic-native/android-permissions/ngx';
import { NgxPermissionsService } from 'ngx-permissions';

import makeWebSocketObservalbe, {
  GetWebSocketResponses,
  normalClosureMessage,
  WebSocketOptions,
} from 'rxjs-websockets'
import { runInThisContext } from 'vm'

//https://github.com/webhacking/WebRTC-Example-With-Typescript/blob/aa5b76c054db9e099e24d4787cd01320de6ae916/src/main.ts
@Component({
  selector: 'app-webrtc',
  templateUrl: './webrtc.page.html',
  styleUrls: ['./webrtc.page.scss'],
})
export class WebrtcPage implements OnInit {
  static self
  isRecording = false
  // @ViewChild('videoInput',{read: ElementRef,static: true}) videoInElement: ElementRef<HTMLElement>
  // @ViewChild('videoOutput',{read: ElementRef,static: true}) videoOutElement: ElementRef<HTMLElement>
  // @ViewChild('layer',{read: ElementRef,static: true}) layer: ElementRef<HTMLElement>
  videoInElement: HTMLElement
  videoOutElement: HTMLElement
  layer: HTMLElement
  webRtcPeer: WebRtcPeer = null
  input: QueueingSubject<string> = null
  websocket: Observable<GetWebSocketResponses<string>> = null
  messages: Observable<string | ArrayBuffer | Blob> = null
  messagesSubscription: Subscription = null
  isSubscribed: boolean = false
  status: Status = Status.NO_CALL
  NO_CALL = 0
  IN_CALL = 1
  POST_CALL = 2
  DISABLED = 3
  IN_PLAY = 4
  perm = ["ADMIN", "MEDIA", "VIDEO_CAPTURE", "AUDIO_CAPTURE"];
  show: Boolean = false


  constructor(public auth: AuthServiceProvider, 
    public permissionsService: NgxPermissionsService,
    public androidPermissions: AndroidPermissions) { 
    
      this.auth.platform.ready().then(res => {
        
        this.videoInElement = <HTMLElement>document.getElementById("videoInput")
        this.videoOutElement = <HTMLElement>document.getElementById("videoOutput")
        this.layer = <HTMLElement>document.getElementById("layer")
        WebrtcPage.self = this
      })
      
  }

  ngOnInit() {
    
 
    this.permissionsService.loadPermissions(this.perm)
    
    //  this.auth.ahttp.get('url').subscribe((permissions: any[]) => {
    //    //const perm = ["ADMIN", "EDITOR"]; example of permissions
    //    this.permissionsService.loadPermissions(permissions);
    // })
  }

  showLoading() {
    this.show = true
  }

  hideLoading() {
    this.show = false
  }

  ionViewDidEnter() {

    if( this.webRtcPeer ) {
      this.stop()
    }

    if( this.isConnected ) {
      this.close()
    }

    this.connect()
  }

  ionViewWillLeave() {
    

    if( this.webRtcPeer ) {
      this.stop()
    }

    if( this.isConnected() ) {
      this.close()
    }
  }

  isConnected(): boolean {
    return this.messagesSubscription && !this.messagesSubscription.closed
  }

  send(message: any): void {
    try {
      const jsonMessage = JSON.stringify(message)
      console.log('Sending message: ' + jsonMessage);
      WebrtcPage.self.input.next(jsonMessage)
    } catch(e) {
      console.log('socket send error : ' + JSON.stringify(e))
      WebrtcPage.self.closeWebsocket()
    }
  }

  closeWebsocket() {
    try { WebrtcPage.self.messagesSubscription.unsubscribe() } catch(e) {}
    WebrtcPage.self.messagesSubscription = null
    WebrtcPage.self.messages = null
    WebrtcPage.self.websocket = null
    try { WebrtcPage.self.input.unsubscribe() } catch(e) {}
    WebrtcPage.self.input = null
    WebrtcPage.self.isSubscribed = false
  }

  startResponse(message) {
    WebrtcPage.self.status = Status.IN_CALL
    console.log("SDP answer received from server. Processing ...");
  
    WebrtcPage.self.webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
        return console.error(error);
    });
  }

  playResponse(message) {
    WebrtcPage.self.status = Status.IN_PLAY
    WebrtcPage.self.webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
        return console.error(error);
    });
  }

  playEnd() {
    WebrtcPage.self.status = Status.POST_CALL 
    WebrtcPage.self.hideLoading()
  }

  connect() {

    if( this.isConnected() ) {
      this.closeWebsocket()
    }

    this.isSubscribed = false
    // this.eventList = []
    // this.eventList.unshift({path: 'SEND', message: PushEvent.CONNECT})
    this.input = new QueueingSubject<string>()
    this.websocket = makeWebSocketObservalbe(''wss://kurento-hello-world-recording-server:8080/recording')
    this.messages = this.websocket.pipe(
      switchMap((getResponses: GetWebSocketResponses) => {
        console.log('websocket opened')
        //webrtc client connect to mirror server
        //this.webRtcPeer.generateOffer(this.onOffer); 
        return getResponses(this.input)
      }),
      retryWhen((errors) => {
        errors.subscribe(sourceError => {
          console.log(JSON.stringify(sourceError))
        })
        return Observable.create(obs => obs.error(PushEvent.DISCONNECTED))
      }),
      share(),
    )
    this.messagesSubscription = this.messages.subscribe(
      (message: string | ArrayBuffer | Blob ) => {

        try {
          let received: any = null
          if( message instanceof ArrayBuffer ) {
            received = JSON.parse(String.fromCharCode.apply(null, new Uint16Array(message)))
          } else if( message instanceof Blob ) {
            throw new Error('Blob message is not allowed')
          } else {
            console.log('received message:', message)
            if( message === PushEvent.CONNECTED ) {
              //this.webRtcPeer.generateOffer(this.onOffer); 
              return
            }
            received = JSON.parse(message)
          }
          var parsedMessage = received
          console.info('Received message: ' + message)
          switch (parsedMessage.id) {
            case 'startResponse':
                this.startResponse(parsedMessage);
                break;
            case 'playResponse':
              this.playResponse(parsedMessage);
              break;
            case 'playEnd':
              this.playEnd()
              break
            case 'error':
   
                this.status = Status.NO_CALL
                console.log("Error message from server: " + parsedMessage.message);
                break;
            case 'iceCandidate':
                this.webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
                  if (error) {
                      console.error("Error adding candidate: " + error);
                      return;
                  }
                });
                break;
            case 'stopped':
              this.auth.presentAlert(parsedMessage.stt + ":" + parsedMessage.path)
              break;
            case 'paused':
              break;
            case 'recording':
              break;
            default:
                this.status = Status.NO_CALL
                console.log('Unrecognized message', parsedMessage);
                console.error('Unrecognized message', parsedMessage);
            }
        } catch(e) {
          console.log(JSON.stringify(e))
          console.error('Unrecognized message');
        }
      },
      (error: Error) => {
        const { message } = error
        if (message === normalClosureMessage) {
          //this.eventList.unshift({path: 'RECV' , message: PushEvent.UNSUBSCRIBE}) 
          console.log('server closed the websocket connection normally : ') 
          this.closeWebsocket()
        } else {
          //this.eventList.unshift({path: 'RECV' , message: PushEvent.DISCONNECTED}) 
          console.log('socket was disconnected due to error:', message)          
          this.closeWebsocket()          
        }
      },
      () => { //complete
        console.log('the connection was closed in response to the user')
        // this.eventList.unshift({path: 'RECV', message: PushEvent.CLOSED})
        this.closeWebsocket()
      }
    )
  }
  
  close() {
    
    try {
      if( this.isSubscribed ) {
        //this.unsubscribe()
      }

      const localTimer = new Timer({
        startTime: 300,
        endTime: null,
        updateFrequency: null,
        selfAdjust: true,
        countdown: false,
        animationFrame: false
      })

      localTimer.on('start', () => {
        // this.eventList.unshift({path: 'RECV', message: PushEvent.CLOSED})
        this.closeWebsocket()
      })

      localTimer.start()

    } catch(e) {}
  }

  onOffer(none, offerSdp) {
    console.info('Invoking SDP offer callback function ' + location.host);
    var message = {
       id : 'start',
       mode : 'video-and-audio',
       sdpOffer : offerSdp
    }
    WebrtcPage.self.send(message);
  }

 
  onIceCandidate(candidate) {
    console.log("Local candidate" + JSON.stringify(candidate));

    var message = {
      id: 'onIceCandidate',
      candidate: candidate
    };
    WebrtcPage.self.send(message);
  }

  async startAfterCheckPermission() {

    this.permissionsService.hasPermission(this.perm).then(success => {
      if( success ) {

        if( this.auth.platform.is("android") ) {
    
          this.androidPermissions.requestPermissions([this.androidPermissions.PERMISSION.CAMERA, 
            this.androidPermissions.PERMISSION.RECORD_AUDIO, 
            this.androidPermissions.PERMISSION.READ_EXTERNAL_STORAGE,
            this.androidPermissions.PERMISSION.WRITE_EXTERNAL_STORAGE]).then( (result: any) => {
            if( result.hasPermission )
              this.start() 
            else
              this.auth.presentAlert('Media Device Permission Not Granted')
          }, err => {
              this.auth.showError(err)
          })
        } else {
          this.start()
        }
      } else {
        this.permissionsService.loadPermissions(this.perm)
        this.permissionsService.hasPermission(this.perm).then(success => {
          if( success ) {
            this.start()
          } else {
            this.auth.presentAlert('Permisssion Error')
          }
        })
      }
    })
  }

  getConstraints(): any {
    return  {
      audio: true,
      video: true
    }
  }

  start() {
    console.log("Starting video call ...")
    this.status = Status.DISABLED
    this.showLoading()
    
    console.log("Creating WebRtcPeer and generating local sdp offer ...");
 
    var constraints = {
      audio: true,
      video: true
    }
    var options = {
        audio: true,
        localVideo: this.videoInElement,
        remoteVideo: this.videoOutElement,
        mediaConstraints : constraints,
        onicecandidate: this.onIceCandidate
      }

    this.webRtcPeer = new WebRtcPeer.WebRtcPeerSendrecv(options, error => {
      if( error ) {
        return false 
      }
      WebrtcPage.self.webRtcPeer.generateOffer(this.onOffer)
    })

    if( ! this.webRtcPeer ) {
      this.auth.presentAlert('Cannot create webRtc adapter')
      this.hideLoading()
    } 
  }

  stop() {
    console.log("Stopping video call ...");
    var stopMessageId = (this.status === Status.IN_CALL) ? 'stop' : 'stopPlay';

    if (this.webRtcPeer) {
      this.webRtcPeer.dispose();
      this.webRtcPeer = null;
  
      var message = {
        id : stopMessageId
      }
      this.send(message);
    }

    this.hideLoading()
    this.status = Status.NO_CALL
  }

  play() {
    console.log("Starting to play recorded video...");
  
    // Disable start button
    this.status = Status.DISABLED
    this.showLoading()
  
    console.log('Creating WebRtcPeer and generating local sdp offer ...');
  
    var options = {
        remoteVideo : this.videoOutElement,
        mediaConstraints : this.getConstraints(),
        onicecandidate : this.onIceCandidate
    }
  
    this.webRtcPeer = new WebRtcPeer.WebRtcPeerRecvonly(options, error => {
      if (error)
        return console.error(error);
      
    });
    this.webRtcPeer.generateOffer(this.onPlayOffer);
  }
  
  onPlayOffer(error, offerSdp) {
    if (error)
      return console.error('Error generating the offer');
    console.info('Invoking SDP offer callback function ' + location.host);
    var message = {
        id : 'play',
        sdpOffer : offerSdp
    }
    WebrtcPage.self.send(message);
  }
}

My app’s webrtc page worked fine on browser but not on android native also to me :woozy_face:
I suspect chronium don’t support webrtc requestPermission :
https://chromium.googlesource.com/chromium/src/+/master/content/public/browser/web_contents_delegate.cc : 216 line

I have got this error on android logcat:
E/chromium: [ERROR:web_contents_delegate.cc(218)] WebContentsDelegate::CheckMediaAccessPermission: Not supported.

I can now resolve this issue in native android. After changing minSdkVersion from 23 to 27, this issue was resolved. Probably webrtc protocol is not supported by low minSdkVersion.

I’m sorry but my parter’s feedback is wrong: I have mis-understood . They have sent feedback to me again that the issue was not resolved yet also after upgrading minSdkVersion.

https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-quickstart
https://github.com/calvinckho/capacitor-jitsi-meet
We have plan to try to use jitsi client and server: I will feedback it to you after testing this module.

@peterkhang Ill post a reply in a hour or so now I am driving :smile: … Had a simular issue recently and had to do some minor tweaks before using getUserMedia inside the Android Webview. Are you using Angular?

Also just as a note: getuserMedia for iOS is not available outside the Safari Browser… so in iOS you must use the safari browser or create a custom plugin or use ionic native camera(enterprise edition). Normal edition will allow you to access the camera but it is like if you have another Activity or screen in Android and will only let you capture images or record videos not the implementation I believe you have with getuserMedia now.