Should I use separate a database to store my active and historic data

#1

Hi guys,

A couple of weeks ago I went to Google to attend a talk about Firebase and in the talk they advised to not use Firebase for storing historical data like logs, stats and not actively changing datasets.

The advice from the speaker
It is usually a good idea to separate your active data from the historical data. If you’re only showing the last 30 days worth of data in your app, your application performance will be best if you put the older data in a different location from the active data. This doesn’t just apply to Firebase, nor even only to NoSQL solutions: even in SQL you see developers creating separate tables for active and for historical data. In Firebase this can be as simple as segmenting the data into month-based buckets.

The second point depends more on your needs. What I often see is that developers keep historical data to do reporting/analysis on. While it is definitely possible to do such analysis in Firebase, there are solutions that are more tailored to this use-case. BigQuery is a great example of such a solution as are many other analytics systems.

Do I need to setup a separate database in MySQL (I already master MySQL so it would be my first alternative) to store these data sets or would a Firebase database be good for the time being?

#2

I am not a friend of those services. Because the basic pricing plans are nothing^^.
If you are using firebase for logging data you will quickly reach the limits.

For that money you can get an own v- or small rootServer and you can do anything you want :).
So you could use CouchDB with PouchDB at the frontend for offline storage and built in synchronisation.

#3

How fast is quickly? How many active users should my app have to reach those limits?

And if you should choose between Couch DB and Pouch DB which one should you use, also isn’t MySQL an option?

#4

CouchDB is the databasesystem and PouchDB is a wrapper around it to use it very easy on clientside.

Sure you can use MySQL but you have to do a little bit more work to get all the functionalities Couch/PouchDB provides

#5

Alright good to know! What are the advantages of CouchDB in comparison to MySQL? Because you said there where some features provided by CouchDB that MySQL didn’t have.

#6

things like: buildin offline syncing, nice syntax in your app code, it is documentbased (good if you want to store massive data, introduced to work very good with JavaScript

#7

Oh those things are pretty awesome, I’ve got one question about the document based data storage: aren’t those files going to be very big and use a lot of server disk space?

#8

I’m using Firebase mainly for its realtime capabilities and I’m very happy with it.

But I’m also storing images (pictures) in my app and for that I’m not using Firebase but Amazon S3 because as @bengtler says you will quickly outgrow your plan’s limits.

For the same reason if I have “historic” or “logging” data (non-active data) I would probably not store it in Firebase but set up a cheap server on Digital Ocean (can be had for USD 5 per month) or whatever and use a MEAN stack or PostgreSQL (or CouchDB/CouchBase) to store it there.

If you do it like this then you won’t very quickly outgrow a moderately-priced Firebase “Candle” plan (50 USD per month), unless your app gets a huge number of users. If I’d ever get to that scale then I’d probably leave Firebase and set up all of the backend stuff myself.

1 Like
#9

I can not say this. They can grow faster because you often store redundant data to avoid populations (in mysql called joins), because there are no “foreign keys” you know from e.g. mysql.

If you use for example mongodb 3 with storageengine wiredtiger you have a really good and fast data compression.

The only thing that costs a little bit more space are the db-images to restore last db states after a an error or system crash (round about 65mb for medium sized dbs).

If it is to much for you to build up all on your own. Follow the advices of @leob. “Grow up” with your app :smile: .
If you want logging or error tracking --> you can easy “abuse” services like google analytics for that --> you want to log something special --> track it as a custom event in analytics, same with error cases.

For hard javascript errors you can use other services, if you need it.

#10

Exactly, I skipped setting up my own backend server just because I want to get my app published quickly.

And yes with Firebase you also store data redundantly (denormalized), they even recommend that and it makes sense (it’s NoSQL).

I’ll just wait and see if my app gets any traction, if it doesn’t then I won’t outgrow my Firebase plan, if it does become popular then all the better, I can spend some time to build my own backend.

Yes and for logging/analytics I’m also considering Google Analytics (or maybe Ionic.io but that’s “alpha” software so I don’t know if that would be wise).

#11

Thanks for all the information guys! I’ll outgrow my Firebase first and will fetch my analytics through Google Analytics. I also added the advice from the speaker of the talk that I attended :grinning: