Where to store large and permanent offline data

Ionic Framework ionic-v3
1

I am currently developing an app that help students to practice past exam questions for a matriculation exam. The catch here is to beat other apps already doing this by making questions and solutions of the app totally available offline.

My challenge now is the data is large. About #12,000 questions/solutions approximately. Where do I put this data?

Can I save it as a JSON file in the front end?.. And would it not affect performance? Apart from having to upload an heavy app file to app store.

Are there other possible solutions for making these questions/solutions available offline on ionic app. Thanks

1 Like
2

Hi

You need plugins I would say

Sqlite could be an option (ionic storage)

Or file storage may even be better

Maybe try firestore from Google, but I dont know about its limitation

Dont forget to encrypt at application level if the data is sensitive

Tom

1 Like
3

I’ve used Ionic Storage (with only the IndexedDB-driver) to store a whole JSON-tree.
Instead of fetching the data via HTTP, I just fetch it from the database and do the filitering locally.

I guess this isn’t the best solution, but in my case, it did the trick

1 Like
4

Hi. Was that on native device or pwa?

Interested to hear about the limitations

5

Great!!! I would search on this info, thanks

6

Thanks MattE.

This sounds much like what I want. Because there will be no crud operations. Only read operations. And any update will be done manually (annually) after recurring exam.

The question now is, how large was ur JSON file? Can indexedDB store as much?

7

No PWA. Just a hybrid Android & iOS app

8

My biggest JSON was about 3 MB or something like that.
But the total of all the data would be around 8 MB or something like that.

IndexedDB was the best option to store large JSON-trees, because it doesn’t have a limit per record in the database.

9

Thanks a lot, one more thing please, from your answer, it means one can store data in separate files, like different batches? Just incase I want to have storage for different subjects… And I hope u wud be helpful if I have difficulties while implementing… :slightly_smiling_face: thanks

10

You can choose under which ID you store which data.
The only thing is that you can retrieve the ID somewhere, to fetch the data later.

Maybe the image underneath can help you understand the concept

image

11

Ionic Storage can be backed by SQLite (see the docs) which can store magnitutes of more data. This should not be a problem at all.

1 Like
12

yeah, I have one app that is using 70k records and 50k records , no problem here

4 Likes
13

Thanks a lot. I’m sure about this now

14

I believe it can handle extreme strings, right?

I have stringified a large object which then gets CryptoJS.AES() encrypted.

Any limit expectations ypu see?

15

I basically have the same problem except I would like to store MP3 files rather than JSON. Does anyone have any idea how I would go about this?

16

Easy. File plugin.

Rhdz

Tom

17

Thanks but I can’t even get it to store a text file at the minute.

this.file.writeFile(this.file.dataDirectory, "test.txt", "text")
.then((success) => {
  this.fileData = this.file.dataDirectory;
}, (error) => {
  this.fileData = 'error';
});

That should write a small text file with “text” in it but the following code only gives me a “TypeError”

this.file.readAsText(this.file.dataDirectory, "test.txt")
.then((success) => {
  this.fileData = success;
}, (error) => {
  this.fileData = error;
});

}

18

Use catch to catch the error not the comma

(U r not handling the promise correctly)

Then see if there is an error

Console log success and error

Then maybe it is easier to debug

19

Also please open a new topic for new problems. Thanks.

20

I believe it can handle extreme strings, right?

I have stringified a large object which then gets CryptoJS.AES() encrypted.

Any limit expectations ypu see?

It’s these edge use-cases that throw a spanner into our assumptions.

I had exactly this problem. I had a payload that needed encryption using the exact same library you mention. The output was approximately a 20K long encrypted string.

For whatever reason, there were android phones that would simply freeze when doing an SQL lite write using local forage as a front end wrapper. It wasn’t a transaction freeze - it would just get locked and required an app kill. I just couldn’t replicate it on my android phones but some of my app users just could not use my app. Given that I couldn’t debug, and the users were not really super technical, so asking them to monitor adb logs was a no go, I gave up and used indexed DB for android and sqlite for iOS in my order preferences. Given I couldn’t debug, I can’t really say if it was a core library problem or a problem with my code. No problem in iOS though.