Working with WebRTC in Ionic

Hi I am creating a VoIP application in ionic. I am using WebRTC which asks for permissions to access camera and microphone. It works perfectly fine when I run my web app in browser.

I have written code to read permissions from user in Android as well. It asks for permissions to user, yet even after allowing all permissions, I am getting error of not able to access media devices.

I have also tried to allow permissions manually to my app from settings. Yet, I am unable to resolve this.

I am not sure if I need to allow browser related permissions as well for Android?

I have really invested a lot of time into this, and would really appreciate if anyone can help me with it. Thanks

1 Like
import { NgxPermissionsService } from 'ngx-permissions';
………
perm = ["ADMIN", "MEDIA", "VIDEO_CAPTURE", "AUDIO_CAPTURE"];
.......
this.permissionsService.hasPermission(this.perm).then(success => {
:::::  



[app.module.ts]


:::;
imports:
...
NgxPermissionsModule.forChild({
      permissionsIsolate: true,
      rolesIsolate: true}),

It didn’t solve my issue. It is still throwing error: “Failed to get local media device”. Even though I has all permissions

@mhartington Hi Mike, Could you look into this? Would really appreciate. Thanks!

ngOnInit() {
    
 
    this.permissionsService.loadPermissions(this.perm)
   
  }

This code probably help you…

Thanks for your reply @peterkhang , but permissions are already granted. That is not the issue. I also tried allowing permissions manually from settings for my app.
Problem is that I am trying webRTC in my ionic project. Not sure if It is looking for web permissions even in android?

webrtc is web browser protocol : This protocol needs web browser permission. I have tested it with kurento-tutorital-java/kurento-hello-world-recording demo server.

<ion-header>
  <ion-toolbar color="--bory-blue">
    <ion-buttons slot="start">
      <ion-menu-button color="light"></ion-menu-button>
    </ion-buttons>
    <ion-title>
      SpeechMate::{{auth.title}}
    </ion-title>
  </ion-toolbar>
</ion-header>

<ion-content padding>
  <div class="videoIn">
    <div class="col-md-5">
      <h3>Local stream</h3>
      <video id="videoInput" autoplay width="480px" height="360px"
        poster="{{auth.root}}assets/png/webrtc.png"></video>
    </div>
  </div>

  <div class="row">
    <ion-grid>
      <ion-row>
        <ion-col size="12">
          <ion-button class="start" color="white" (click)='startAfterCheckPermission()' color="primary" *ngIf="status === NO_CALL">start</ion-button>    
          <ion-button class="start" color="white" (click)='stop()' color="danger" *ngIf="status === POST_CALL || status === IN_CALL || status === IN_PLAY">stop</ion-button>     
          <ion-button class="start" color="white" (click)='play()' color="primary" *ngIf="status === NO_CALL">play</ion-button>
        </ion-col>
      </ion-row>
    </ion-grid>     
  </div>
  <div class="videoOut">
    <h3>Remote stream</h3>
    <video id="videoOutput" autoplay width="480px" height="360px"
      poster="{{auth.root}}assets/png/webrtc.png"></video>
  </div>
  <div class="row">
    <div class="col-md-12">
      <label class="control-label" for="console">Console</label><br>
      <br>
      <div id="console" class="democonsole">
        <ul></ul>
      </div>
    </div>
  </div>

  <div id="layer" *ngIf="show"><img src="{{auth.root}}assets/imgs/loading.gif"/></div>

</ion-content>


import { Component, OnInit,ViewChild,ElementRef, HostListener, NgZone  } from '@angular/core'
import { AuthServiceProvider } from 'src/app/providers/auth-service/auth-service'
import { v4 as uuidv4 } from 'uuid'
import { environment } from 'src/environments/environment'
import { WebRtcPeer } from 'kurento-utils'
import { QueueingSubject } from 'queueing-subject'
import { Subscription, Observable, timer } from 'rxjs'
import { Timer } from 'interval-timer'
import { share, switchMap, retryWhen, delay, map } from 'rxjs/operators'
import { PushEvent, Status } from 'src/app/providers/common/common'
import { AndroidPermissions } from '@ionic-native/android-permissions/ngx';
import { NgxPermissionsService } from 'ngx-permissions';

import makeWebSocketObservalbe, {
  GetWebSocketResponses,
  normalClosureMessage,
  WebSocketOptions,
} from 'rxjs-websockets'
import { runInThisContext } from 'vm'

//https://github.com/webhacking/WebRTC-Example-With-Typescript/blob/aa5b76c054db9e099e24d4787cd01320de6ae916/src/main.ts
@Component({
  selector: 'app-webrtc',
  templateUrl: './webrtc.page.html',
  styleUrls: ['./webrtc.page.scss'],
})
export class WebrtcPage implements OnInit {
  static self
  isRecording = false
  // @ViewChild('videoInput',{read: ElementRef,static: true}) videoInElement: ElementRef<HTMLElement>
  // @ViewChild('videoOutput',{read: ElementRef,static: true}) videoOutElement: ElementRef<HTMLElement>
  // @ViewChild('layer',{read: ElementRef,static: true}) layer: ElementRef<HTMLElement>
  videoInElement: HTMLElement
  videoOutElement: HTMLElement
  layer: HTMLElement
  webRtcPeer: WebRtcPeer = null
  input: QueueingSubject<string> = null
  websocket: Observable<GetWebSocketResponses<string>> = null
  messages: Observable<string | ArrayBuffer | Blob> = null
  messagesSubscription: Subscription = null
  isSubscribed: boolean = false
  status: Status = Status.NO_CALL
  NO_CALL = 0
  IN_CALL = 1
  POST_CALL = 2
  DISABLED = 3
  IN_PLAY = 4
  perm = ["ADMIN", "MEDIA", "VIDEO_CAPTURE", "AUDIO_CAPTURE"];
  show: Boolean = false


  constructor(public auth: AuthServiceProvider, 
    public permissionsService: NgxPermissionsService,
    public androidPermissions: AndroidPermissions) { 
    
      this.auth.platform.ready().then(res => {
        
        this.videoInElement = <HTMLElement>document.getElementById("videoInput")
        this.videoOutElement = <HTMLElement>document.getElementById("videoOutput")
        this.layer = <HTMLElement>document.getElementById("layer")
        WebrtcPage.self = this
      })
      
  }

  ngOnInit() {
    
 
    this.permissionsService.loadPermissions(this.perm)
    
    //  this.auth.ahttp.get('url').subscribe((permissions: any[]) => {
    //    //const perm = ["ADMIN", "EDITOR"]; example of permissions
    //    this.permissionsService.loadPermissions(permissions);
    // })
  }

  showLoading() {
    this.show = true
  }

  hideLoading() {
    this.show = false
  }

  ionViewDidEnter() {

    if( this.webRtcPeer ) {
      this.stop()
    }

    if( this.isConnected ) {
      this.close()
    }

    this.connect()
  }

  ionViewWillLeave() {
    

    if( this.webRtcPeer ) {
      this.stop()
    }

    if( this.isConnected() ) {
      this.close()
    }
  }

  isConnected(): boolean {
    return this.messagesSubscription && !this.messagesSubscription.closed
  }

  send(message: any): void {
    try {
      const jsonMessage = JSON.stringify(message)
      console.log('Sending message: ' + jsonMessage);
      WebrtcPage.self.input.next(jsonMessage)
    } catch(e) {
      console.log('socket send error : ' + JSON.stringify(e))
      WebrtcPage.self.closeWebsocket()
    }
  }

  closeWebsocket() {
    try { WebrtcPage.self.messagesSubscription.unsubscribe() } catch(e) {}
    WebrtcPage.self.messagesSubscription = null
    WebrtcPage.self.messages = null
    WebrtcPage.self.websocket = null
    try { WebrtcPage.self.input.unsubscribe() } catch(e) {}
    WebrtcPage.self.input = null
    WebrtcPage.self.isSubscribed = false
  }

  startResponse(message) {
    WebrtcPage.self.status = Status.IN_CALL
    console.log("SDP answer received from server. Processing ...");
  
    WebrtcPage.self.webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
        return console.error(error);
    });
  }

  playResponse(message) {
    WebrtcPage.self.status = Status.IN_PLAY
    WebrtcPage.self.webRtcPeer.processAnswer(message.sdpAnswer, function(error) {
      if (error)
        return console.error(error);
    });
  }

  playEnd() {
    WebrtcPage.self.status = Status.POST_CALL 
    WebrtcPage.self.hideLoading()
  }

  connect() {

    if( this.isConnected() ) {
      this.closeWebsocket()
    }

    this.isSubscribed = false
    // this.eventList = []
    // this.eventList.unshift({path: 'SEND', message: PushEvent.CONNECT})
    this.input = new QueueingSubject<string>()
    this.websocket = makeWebSocketObservalbe(''wss://kurento-hello-world-recording-server:8080/recording')
    this.messages = this.websocket.pipe(
      switchMap((getResponses: GetWebSocketResponses) => {
        console.log('websocket opened')
        //webrtc client connect to mirror server
        //this.webRtcPeer.generateOffer(this.onOffer); 
        return getResponses(this.input)
      }),
      retryWhen((errors) => {
        errors.subscribe(sourceError => {
          console.log(JSON.stringify(sourceError))
        })
        return Observable.create(obs => obs.error(PushEvent.DISCONNECTED))
      }),
      share(),
    )
    this.messagesSubscription = this.messages.subscribe(
      (message: string | ArrayBuffer | Blob ) => {

        try {
          let received: any = null
          if( message instanceof ArrayBuffer ) {
            received = JSON.parse(String.fromCharCode.apply(null, new Uint16Array(message)))
          } else if( message instanceof Blob ) {
            throw new Error('Blob message is not allowed')
          } else {
            console.log('received message:', message)
            if( message === PushEvent.CONNECTED ) {
              //this.webRtcPeer.generateOffer(this.onOffer); 
              return
            }
            received = JSON.parse(message)
          }
          var parsedMessage = received
          console.info('Received message: ' + message)
          switch (parsedMessage.id) {
            case 'startResponse':
                this.startResponse(parsedMessage);
                break;
            case 'playResponse':
              this.playResponse(parsedMessage);
              break;
            case 'playEnd':
              this.playEnd()
              break
            case 'error':
   
                this.status = Status.NO_CALL
                console.log("Error message from server: " + parsedMessage.message);
                break;
            case 'iceCandidate':
                this.webRtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
                  if (error) {
                      console.error("Error adding candidate: " + error);
                      return;
                  }
                });
                break;
            case 'stopped':
              this.auth.presentAlert(parsedMessage.stt + ":" + parsedMessage.path)
              break;
            case 'paused':
              break;
            case 'recording':
              break;
            default:
                this.status = Status.NO_CALL
                console.log('Unrecognized message', parsedMessage);
                console.error('Unrecognized message', parsedMessage);
            }
        } catch(e) {
          console.log(JSON.stringify(e))
          console.error('Unrecognized message');
        }
      },
      (error: Error) => {
        const { message } = error
        if (message === normalClosureMessage) {
          //this.eventList.unshift({path: 'RECV' , message: PushEvent.UNSUBSCRIBE}) 
          console.log('server closed the websocket connection normally : ') 
          this.closeWebsocket()
        } else {
          //this.eventList.unshift({path: 'RECV' , message: PushEvent.DISCONNECTED}) 
          console.log('socket was disconnected due to error:', message)          
          this.closeWebsocket()          
        }
      },
      () => { //complete
        console.log('the connection was closed in response to the user')
        // this.eventList.unshift({path: 'RECV', message: PushEvent.CLOSED})
        this.closeWebsocket()
      }
    )
  }
  
  close() {
    
    try {
      if( this.isSubscribed ) {
        //this.unsubscribe()
      }

      const localTimer = new Timer({
        startTime: 300,
        endTime: null,
        updateFrequency: null,
        selfAdjust: true,
        countdown: false,
        animationFrame: false
      })

      localTimer.on('start', () => {
        // this.eventList.unshift({path: 'RECV', message: PushEvent.CLOSED})
        this.closeWebsocket()
      })

      localTimer.start()

    } catch(e) {}
  }

  onOffer(none, offerSdp) {
    console.info('Invoking SDP offer callback function ' + location.host);
    var message = {
       id : 'start',
       mode : 'video-and-audio',
       sdpOffer : offerSdp
    }
    WebrtcPage.self.send(message);
  }

 
  onIceCandidate(candidate) {
    console.log("Local candidate" + JSON.stringify(candidate));

    var message = {
      id: 'onIceCandidate',
      candidate: candidate
    };
    WebrtcPage.self.send(message);
  }

  async startAfterCheckPermission() {

    this.permissionsService.hasPermission(this.perm).then(success => {
      if( success ) {

        if( this.auth.platform.is("android") ) {
    
          this.androidPermissions.requestPermissions([this.androidPermissions.PERMISSION.CAMERA, 
            this.androidPermissions.PERMISSION.RECORD_AUDIO, 
            this.androidPermissions.PERMISSION.READ_EXTERNAL_STORAGE,
            this.androidPermissions.PERMISSION.WRITE_EXTERNAL_STORAGE]).then( (result: any) => {
            if( result.hasPermission )
              this.start() 
            else
              this.auth.presentAlert('Media Device Permission Not Granted')
          }, err => {
              this.auth.showError(err)
          })
        } else {
          this.start()
        }
      } else {
        this.permissionsService.loadPermissions(this.perm)
        this.permissionsService.hasPermission(this.perm).then(success => {
          if( success ) {
            this.start()
          } else {
            this.auth.presentAlert('Permisssion Error')
          }
        })
      }
    })
  }

  getConstraints(): any {
    return  {
      audio: true,
      video: true
    }
  }

  start() {
    console.log("Starting video call ...")
    this.status = Status.DISABLED
    this.showLoading()
    
    console.log("Creating WebRtcPeer and generating local sdp offer ...");
 
    var constraints = {
      audio: true,
      video: true
    }
    var options = {
        audio: true,
        localVideo: this.videoInElement,
        remoteVideo: this.videoOutElement,
        mediaConstraints : constraints,
        onicecandidate: this.onIceCandidate
      }

    this.webRtcPeer = new WebRtcPeer.WebRtcPeerSendrecv(options, error => {
      if( error ) {
        return false 
      }
      WebrtcPage.self.webRtcPeer.generateOffer(this.onOffer)
    })

    if( ! this.webRtcPeer ) {
      this.auth.presentAlert('Cannot create webRtc adapter')
      this.hideLoading()
    } 
  }

  stop() {
    console.log("Stopping video call ...");
    var stopMessageId = (this.status === Status.IN_CALL) ? 'stop' : 'stopPlay';

    if (this.webRtcPeer) {
      this.webRtcPeer.dispose();
      this.webRtcPeer = null;
  
      var message = {
        id : stopMessageId
      }
      this.send(message);
    }

    this.hideLoading()
    this.status = Status.NO_CALL
  }

  play() {
    console.log("Starting to play recorded video...");
  
    // Disable start button
    this.status = Status.DISABLED
    this.showLoading()
  
    console.log('Creating WebRtcPeer and generating local sdp offer ...');
  
    var options = {
        remoteVideo : this.videoOutElement,
        mediaConstraints : this.getConstraints(),
        onicecandidate : this.onIceCandidate
    }
  
    this.webRtcPeer = new WebRtcPeer.WebRtcPeerRecvonly(options, error => {
      if (error)
        return console.error(error);
      
    });
    this.webRtcPeer.generateOffer(this.onPlayOffer);
  }
  
  onPlayOffer(error, offerSdp) {
    if (error)
      return console.error('Error generating the offer');
    console.info('Invoking SDP offer callback function ' + location.host);
    var message = {
        id : 'play',
        sdpOffer : offerSdp
    }
    WebrtcPage.self.send(message);
  }
}

My app’s webrtc page worked fine on browser but not on android native also to me :woozy_face:
I suspect chronium don’t support webrtc requestPermission :
https://chromium.googlesource.com/chromium/src/+/master/content/public/browser/web_contents_delegate.cc : 216 line

I have got this error on android logcat:
E/chromium: [ERROR:web_contents_delegate.cc(218)] WebContentsDelegate::CheckMediaAccessPermission: Not supported.