Optimizing Storage bottleneck with 100s of items

Ionic Framework ionic-v3
#1

There are times when I want to get all the objects that are stored in Ionic Storage for a subset of keys.

I have implemented a wrapper around Storage that mirrors the functionality so that I can substitute it out as @rapropos suggested in this post.

Right now I am testing my application on an iPhone 7+ and with a data set of 435 entries it is taking 6+ seconds to iterate through the keys, filter the keys based on some prefix, and then get the objects associated with those keys.

Perhaps i am doing something wrong in this code that will be obvious to you guys but I am not sure what it is.
I am considering switching to SQLLite but would rather stick with Ionic Storage if possible.

Here is the offending function:

constructor(private storage: Storage) {
console.log(Storage Provider: created with driver = ${storage.driver})
}

public getAll(keyPrefix:string) : Observable<any> {
    let keysObs : Observable<string[]> =         
        Observable.fromPromise( <Promise<string[]>>this.storage.keys());
    let flatKeys : Observable<string> = keysObs.mergeMap( (keys:string[]) => {
        return Observable.from(keys);
    });
    let filteredKeys: Observable<string> = flatKeys.filter( (key:string) => {
        let prefixIndex:number = key.indexOf(keyPrefix);
        return prefixIndex == 0 
        }
    );

    let itemPromises: Observable<Observable<any>> = filteredKeys.map( 
        (key:string) => {
            //console.debug("Retrieving from storage with key = ", key);
            return Observable.fromPromise(this.storage.get(key)) } 
        );

    return itemPromises.mergeAll();
}

At a higher level I am not even sure how to profile my code to see which 20% of the code is taking up 80% of the time. It might not even be happening in this block. If this were a C# or Java project then I would just fire up the profiler, run the application a couple of times and see where the hot spots are. What do people out there do to profile their code?

I looked into using XCode instruments but am skeptical that it will give me meaningful function level reporting given that my XCode project shows just the output from the ionic build and not the source Typescript and Javascript that is used to generate the XCode project.

Any thoughts/suggestions would be most helpful.

-Harley

1 Like
#2

As an experiment I created a branch where I replaced my StorageProvider with a version that goes directly to SQLite and gets the keys and values in 1 query. The time went from ~6 seconds to ~194 ms the first time I request the data set and then averages at about ~50ms for subsequent requests of the same dataset.

In the old setup Storage was acting as a front for SQLite.
It would make 1 query to get all the keys (400+) in this test.
Then it would make N more queries to get the objects associated with the key.

Now I am just making 1 query and getting the keys and the values and it works much faster.

#3

Ionic Storage is a simple hammer.

You could of course put all keys in one key as subvalues, but that would probably mess with your other use cases and patterns I assume.

Maybe you are beyond the use cases for Ionic Storage…
(although I am not sure there might not be a better way to do what you did)

#4

The benefit of Observables is that they are nonblocking, not that they’re fast. I’m sure Array.prototype.filter is much more optimized than Observable.filter. If you need speed, I think it makes the most sense to use an Observable for a very simple query, and then to perform all the logic with optimized JS once the data is in memory. It sounds as though that’s what you did with your second post.

But to be honest, I’m kinda questioning your app logic. Do you really need to know 400+ pieces of information at one time? You can’t display that much on a screen. There might be a way to change your app logic so you only request information from storage if the user requests it somehow.

#5

That’s good feedback Aaron…

I had a new toy (Observables) and perhaps was trying to overuse it when it
wasn’t fit for purpose since my dataset is bounded. I’ll think about
whether I can adjust the design.

I am not necessarily showing all the information to the user, but in order
to filter and sort the items I am using the full dataset. By moving to
SQLite I can now leverage SQL to filter and sort on retrieval and work on
less rows.

On the view side I am using VirtualScroll to limit the number of items
rendered… although in line with your feedback I could potentially
leverage the InfiniteScroll.