File transfer to have a priority based file download

Ionic Framework ionic-v3
#1

Hi,

I need to build a offline ionic app. App api has many serve url images, about 255 datas. I used file transfer plugin to download the images, to externalApplicationStorageDirectory, and store it in sqlite database. But the images are not downloaded in accordance with the api data order. The datas are downloaded in different order and inserted differently in database.

I need to download the datas according to the api order and to be inserted like the same order to database.

Do anyone have a different idea to do so.

Thanks.

#2

I would use rxjs for this.

// Ionic 3 with new rxjs imports
import { from } from 'rxjs/observable/from';
import { bufferCount } from 'rxjs/operators/bufferCount';
import { concatMap } from 'rxjs/operators/concatMap';
import { map } from 'rxjs/operators/map';

downloadData(): Observable<any> {
  return from(this.data).pipe( // Pipe each entry of the url-array through the observable stream
    bufferCount(8), // Group the entries in arrays of 8 -> 8 simultanious downloads.
    map(arr => arr
      .map(url => this.fileTransfer.download(url, 'target') // Download files for the 8 entries.
        .catch(err => console.log(err)))), // Catch errors, so that one download failure does not stop all downloads
    concatMap(arr => Promise.all(arr)) // Wait until the 8 downloads are done, then start the next package.
  );
}

// With old rxjs imports
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/observable/from';
import 'rxjs/add/operator/bufferCount';
import 'rxjs/add/operator/concatMap';
import 'rxjs/add/operator/map';

downloadData(): Observable<any> {
  return Observable
    .from(this.data) // Pipe each entry of the url-array through the observable stream
    .bufferCount(8) // Group the entries in arrays of 8 -> 8 simultanious downloads.
    .map(arr => arr
      .map(url => this.fileTransfer.download(url, 'target') // Download files for the 8 entries.
        .catch(err => console.log(err)))) // Catch errors, so that one download failure does not stop all downloads
    .concatMap(arr => Promise.all(arr)); // Wait until the 8 downloads are done, then start the next package.
 }

If you want to download one by one use this:

downloadData(): any {
    return Observable
      .from(this.data) // Pipe each entry of the url-array through the observable stream
      .concatMap(url => this.fileTransfer.download(url, 'target') // Wait until the download is done, then start the next download.
        .catch(err => console.log(err))); // Catch failures to prevent one download failure stopping all downloads.
  }

I haven’t tried if it works. So if somebody finds a mistake, please tell here.

#3

Hi @Nexi,
Thanks for the reply. Let me check and feedback you.

Thanks.

#4

Hi @Nexi,

I tried, but i get this error,

ERROR TypeError: object is not observable
at Function.webpackJsonp.306.FromObservable.create [as from]

My code is as follows,

  			for(let item of this.edition){
				const fileTransfer: FileTransferObject = this.EditionDownload.create();
				const url = encodeURI(item.imgsrc);
			  	Observable.from(item) 
			    .bufferCount(8) 
			    .map(arr => arr
			      .map(url => {
			      	fileTransfer.download(url, this.file.externalApplicationStorageDirectory + item.cover_image).then((location) => {
			      		this.cover_image_native_path = location.nativeURL();
			      	});
			   	}))
				this.database.executeSql('INSERT INTO Cover VALUES(?, ?)', [ JSON.stringify(this.cover_image_native_path) , JSON.stringify(item)]);
			}

Is the code is correct? Can you please suggest me the right way.

Thanks.

#5 
Observable.from(this.edition)
  .bufferCount(8) 
  .map(arr => arr
    .map(item => {
      const fileTransfer: FileTransferObject = this.EditionDownload.create();
      return fileTransfer.download(
        encodeURI(item.imgsrc),
        this.file.externalApplicationStorageDirectory + item.cover_image
      ).then(
      ).then((location) => {
        return this.database.executeSql('INSERT INTO Cover VALUES(?, ?)', [ JSON.stringify(location.nativeURL()) , JSON.stringify(item)]);
      }).catch(error => return undefined;
    })
  );
}

But if you are not that familiar with Observables you can use async await for that. For easy use cases it is much simpler:

@Injectable()
export class DownloadProvider {

  // ...
  // attributes, constructor and other methods
  // ...

  async downloadFiles(edition): Promise<void> {
    for (const item of edition) {
      try {
        const fileTransfer: FileTransferObject = this.EditionDownload.create();
        const url = encodeURI(item.imgsrc);
        const target = this.file.externalApplicationStorageDirectory + item.cover_image;

        const location = await fileTransfer.download(
          url,
          target
        );
        // Do you really need to stringify it? Isn't it already a string?
        const filePath = JSON.stringify(location.nativeURL());
        // If executeSql does not return a promise, remove await
        await this.database.executeSql('INSERT INTO Cover VALUES(?, ?)', [ filePath , JSON.stringify(item)]);
      } catch (error) {
        console.error('Item could not be downloaded', item);
      }
    }
  }
}
#6

Hi @Nexi,
Thanks for the solution. It worked. Thanks a lot :grinning: :grinning:

#7

If you want simultanious downloads but keep the order of your items in the database, you can try this example:

@Injectable()
export class DownloadProvider {

  // ...
  // attributes, constructor and other methods
  // ...

  async downloadFiles(edition: Array<any>): Promise<void> {
    // Maximal simultanious downloads
    const MAX = 4;
    // Creates an array of arrays with maximal MAX entries
    const items: Array<Array<any>> = edition.reduce((acc, item, index) => {
      const i = Math.floor(item / MAX);
      return acc[i] ? acc[i].push(item) : acc[i] = [item];
    }, []);

    // For each array with MAX entries do simultanious downloads
    for (const entry of items) {
      const downloads = entry.map((item) => {
        const fileTransfer: FileTransferObject = this.EditionDownload.create();
        const url = encodeURI(item.imgsrc);
        const target = this.file.externalApplicationStorageDirectory + item.cover_image;

        const locationPromise: Promise<any> = fileTransfer.download(
          url,
          target
        ).catch(error => undefined); // For check if download succeded at #1
        return { item, locationPromise };
      });

      const downloadedItems = await Promise.all(downloads.map(({ item, locationPromise }) => locationPromise
        .then(location => ({ item, location }))));

      for (const { item, location } of downloadedItems) {
        if (location) { // #1: Checks if download succeded
          // Do you really need to stringify it? Isn't it already a string?
          const filePath = JSON.stringify(location.nativeURL());
          try {
            // If executeSql does not return a promise, remove await
            await this.database.executeSql('INSERT INTO Cover VALUES(?, ?)', [filePath, JSON.stringify(item)]);
          } catch (error) {
            console.error('Item could be saved in database', item);
          }
        } else {
          console.error('Item could not be downloaded', item);
        }
      }
    }
  }
}