Have a relative large data set and i dont know how to store it


#1

Hi there,

I’m doing an app since some time ago, and i’m facing a problem relative to storing data, and i don’t really know what is the best way to face the problem.

My app is a transports app for my city, we have some bus lines, each with their respective bus stops, and each line have two or more trajectories. More or less, i there are 1500 bus stops which i need to store in some way and filter them quickly to suggest the user as he types the stop name.

Each stop has a stop ID (int), stop name, latitude, longitude and lines that go in said stop.

I have tried to store the stops hardcoded in a service, and filter them, which worked nice. The problem is that said stops change on time, as well as new stops are created, and lines have different departure times according to the weather.

After seeing that i had to update the hardcoded object and update the app constantly, i tried to store the stops in my server and retrievr them by the app yo store them inside the phone using SQLite. The problem with SQLite is that for writing / updating 1500 rows, the phone needs ~40seconds, and can’t be done in background, so i need the user to perform a tedious and slow action, which i dont like a bit. Also, even if only one stop has changed, i need to update the whole database or do a pretty hard versioning system.

After that, i’ve thought in firebase. A blazing fast database where i could store the stops and update them via Curl api. The issue is that firebase does not support queries “like”, eg: if i do search for “ion” i don’t get as a result “ionic”, firebase only supports exact value search).

I have thought about storing the stops in firebase and fetch them on app start , and store them in some $rootscope var, but this is ugly and also i’ll need to wait for the stops before the user can start using the app… which is not user friendly. Also, the favorites system, recent stops system and most visited stops system, need the stop information to work, and i cannot wait to get them, not even once at start.

I also have tried to store the stops in my server and create an api that filters the data on real time, but is not fast enough

What would be the best way to face this issue?

Sorry for the wall of text.


#2

Few remarks:

  • 1500 rows is not a large data set, using a database may be overkill especialy if your queries do not vary.
  • your server could export it as json and your app could store it using localstorage
  • in some cases, I suppose that you could query your server about the stops/lines that can reached from user’s current position which should result in smaller subset.
  • if you go for a database, rather than re-creating the full database each time, could not you send just updates either by coding it yourself or by using database sync features (maybe not free)?
  • why could not you do your database update in background?