Firebase Denormalization and huge data requests

Hi everyone.
I’m going to use NoSQL database for a huge project but I’ve worked since 2002 with SQL databases so I’m quite “closed” about this kind of things.

I have a quite complex data tree and I’m wondering how to flat the data structure to make it work well and fast, even with millions of data loaded.

That’s the data part I’m worried about:
registered users can post things in the app. Latitude and longitude is saved into Firebase. A category and the creation data too.
Now.
I need to make a “query” like “SELECT * FROM posts WHERE category = {category} AND latitude > {latBig} AND latitude < {latSmall} AND (same for longitude) ORDER BY date DESC”

I know that I need to make indexes to make the JSON work well but I’m stuck here because there are lots of things to check.

My idea was to make posts as always, plus add an indexes of categories (they’re just 10 so it’s not so hard to make).
The user can filter things based on categories and/or position (posts near him) and THIS is the hard part.

QUESTION (finally):
How can I get just the last 100 created post near me of a specific category?

DOUBT:
Get all the posts based on data should be reeeeeeally heavy for the system or not?
In SQL less fetch data = more speed.
In NoSQL is that the same? Because I’ve read about no issue on getting so much data in a single call but It’s the opposite of what I’ve learned and used since now so I’m a little bit scared to make a complex app based on a service that I have to change in the future :grimacing:

Thanks for all the tips you’re going to tell me. It’s important for me to have all clear before touch a single line of code.

Hi there,

You have GeoFire for that :

Look here https://www.firebase.com/blog/2013-09-25-location-queries-geofire.html (pretty old but…)
It will do the job for you !

So you can only query data around one location !

Best,
Thibaut

Thanks gensollen.
I’ve read all the Firebase blog, forum and documentation.
My needs are not only for the location. I need a way to do similar sql things in Firebase OR (and this one should be the solution) a new way to manage database data with NoSQL.

In this particular case I’ve done all the app and it was quite finished BUT It was a little bit slow and I need something better to manage all the data (but first of all FASTER). That’s why NoSQL.

Now I’m stuck on the deep search of records.

In your case, I would do:

Create “node” of the category, and but data in it. So you will do a geoquery in the node

var firebaseRef = new Firebase("https://<your-firebase>.firebaseio.com/THE_CATEGORY");
// Create a GeoFire index
var geoFire = new GeoFire(firebaseRef);

Thus you only query posts from the category and in the good location

Or maybe I still does not understand your need ?

What I need is hard to explain to myself either :neutral_face:

I need to make the query I’ve wrote in the first post BUT “flatten” it to be used in Firebase’s NoSQL.

I need to understand if I fetch thousand of results and filter it (via AngularJS’s ngRepeat filter) is better than ask for a “complex” query to Firebase (plus I really don’t know IF it’s possible to ask for some deep search to Firebase).

The base of NoSQL is that you have to think about your query and you do not have to care about the data. You can have duplicate data (and you have to !). Query are then easier. But it is more difficult to change data ( cause you have different changes).

Thus I would create :
POSTS --> Category 1
–> Category 2
–> …
And then use geofire for querying in these categories

Does it make sense ?

You can fetch all the post and then filter them in your app, but you will download a lot of data, and it is not really scalable. Thus, you cannot really do a complex query, but you can do a complex structure in firebase, and then have simple query.

Does it make sense ?

That’s the clue I’m right now :sweat_smile: and you’ve get it. Thanks :smile:

I still need a “click” as I call it. It’s the “how I can make this all work as it should creating which data structure”.
I have, for example, the “all posts” page that must show all the post ordered by data and with a limit of 100 posts.
The example you’ve done it’s ok if you have to choose a category but if you’re not?

It will depend on what you want to display.

As I told you, you have to duplicate content, and store ID.

So you can create a node “all” in your posts, and with firebase you can only query the last 100 items.

For the category you can only store ID.

Does it make sense ? Take some readings about NoSQL concepts.

I’ve read all about NoSQL, that’s why I’m asking, because I can’t get how to certain things.
I get how to flatten data but I still don’t get how to filter it from the Firebase side.

SELECT * FROM posts ORDER BY data DESC
This one is ok: …url/posts").orderBy("-data")

SELECT * FROM postsByCat WHERE key = {cat} ORDER BY data DESC
This one is ok too: …url/posts").child({cat}).orderBy("-data")

SELECT * FROM postsByCat WHERE key = {cat} AND age =>18 AND age <= 22 ORDER BY data DESC
Well, this one is much more complicated to me at the moment.
I know how to get the most of the data I want but when I have to filter the data in 2 or plus variables the things are much more complicated.
I need a logical explanation on how to do it with NoSQL. I really don’t get how to do that.

For multiple filtering it is really complicated, but sometimes you can filter by child (like category).

For the age, I would try with “startAt() and endAt()” of firebase.

Best.

That one should be the solution until you realize you must use that with the orderBy directive :expressionless:
So I can’t order it by date Desc AND by startAt & endAt. So THIS is the clue!
How we can go around that?

BTW thanks for your time gensollen :smile:

I guess, if you cannot do it on the server side, you have to do it on ionic and angular.

But you can ask Firebase for this particular case.

Best