Writing to two Firebase instances in parallel algorithm


#1

I’ve been thinking how to implement this functionality for like a week now, and still haven’t managed to find a more or less decent solution. Here the problem:

I need to store various locations in one node (Country+State+City) to general a list with checkboxes. This node looks like this. The “count” parameters are there to know how many users added a given City (is it’s zero, then the city was removed by all users and should also be removed from Firebase).

At the same time each user of my App needs to store his/her locations selection in their own node, such that the App would remember their last selected Cities.

Here is what makes this problem difficult, the list of locations is dynamically updated each time someone (anyone) added/removed a city. I’d really not like to update all user nodes each time this happens, but instead only when the user either opens the app or checks his/her selected cities. So based on this I think we need two sets of nodes (one for dynamic updates - one node) and the other set - individual nodes of each user (many).

Any advice/recommendations on how to implement such functionality would be highly appreciated as I just don’t know anymore what to do! Many thanks!


#2

You need to flatten your data otherwise it’s not going to be maintainable which will result with questions like you are asking all the time.

With firebase you can handle object association via the automated generated keys.

Split the Countries, States and Cities into their own nodes and let firebase generate a unique key for each of them (if you are not entering them manually).

In a country node you could have a list of state keys and same thing for the states (list of city keys).

You could also have a reference to the parent (state knowing in what county it’s in etc…) if that’s needed.

For the association with your users, I imagine that you have a node for them and their uid to identify them so you can do the same thing here too.

Here is a small example

{
    "coutries": {
        "-JiGh_31GA20JabpZBfa": {
            "name": "United States",
            "states": {
                "-JiGh_31GA20JabpZBfb": true
            }
        }
    },
    "states": {
        "-JiGh_31GA20JabpZBfb": {
            "name": "Florida",
            "country": "-JiGh_31GA20JabpZBfa",
            "cities": {
                "-JiGh_31GA20JabpZBfc": true
            }
        }
    },
    "cities": {
        "-JiGh_31GA20JabpZBfc": {
            "name": "Miami",
            "state": "-JiGh_31GA20JabpZBfb",
            "visited_by": {
                "P1yjH4GgcFQcJUHULmTgMLC68w64": true
            }
        }
    },
    "users": {
        "P1yjH4GgcFQcJUHULmTgMLC68w64": {
            "cities_visited": {
                "-JiGh_31GA20JabpZBfc": true
            }
        }
    }
}

With this example when a user visits a city you just have to update the two node related to this action (the city and the user). When you want to know how many people visited the city you can count the visited_by list in the city node and when you want to know what cities your user visited you can consult the cities_visited list in your user node.

Firebase documentation on how you should structure your data

Let me know if you have any questions :slight_smile:


#3

WOW!! Thanks so much! I’m pretty sure this will work! Thank you so much, this problem was killing me for over a week! I’ll code it up now and will let you know if it works, but I’m 99.9% sure that it will!

Much appreciated!!!


#4

I use the uuid library to create a uuid for literally everything, and I manipulate uuids in much the same way as described in the above post. I decided not to use the Firebase keys, so that in case I wanted to move my data away from Firebase, I controlled the id creation. But yeah, you have to think with a lot of indirection if you want flat (and constant-size) data. Your “objects” become lists of strings.


#5

Thanks :blush:

Valid point there… I never thought about that. Again, I only use firebase with my personal projects so I don’t mind it dealing with the ids but if you’re working on a bigger project I would follow the advice of @AaronSterling.


#6

Works like a charm! Many thanks!
Maybe you could be able to advise how to sort such data in a clever way?

Basically I have a bunch of node with location data stored in chronological order in Firebase, and I need to display only the nodes which have the right location (country+state+city). I can use a unique ID for each city (so that to avoid also checking country+state), but how to deal with many selected cities, when a user selects more than just one city?

Many thanks for your awesome help again!!!


#7

I’m confused now…

What are those locations? city location? user location?

What does this means? What is the “right” location?

The list of visited city keys are in user.cities_visited. If you want to retrieve the city objects associated, you would have to iterate though this list and get them one by one.


#8

Let me explain the overall setup. My App will offer sales listings to users, kinda of a classified app. Users will be able to select cities from which to show listings. Users are able to select multiple cities at once to update the shown listings. Listings are shown in an infinite scroll manner, where Firebase requests 5-10 listings (nodes) at a time, and then the App checks if these are within the given cities or not, and should show only within the selected cities. If not enough listings (3) are found within the first batch of 5-10, Firebase fetched the difference, and the routine repeats itself until we show 3 listings from the selected cities.

Firebase allows only one criteria for sorting/selection, and there I already have No. of listings to fetch at one time. I can’t load all, there are thousands.

I’m wondering what would be the best way to do the sorting of the fetched batch of listings.

I can do it via a For Loop with an If Statement inside. But, if we have 5 fetched listings and 10 selected cities, this gives a lot of loops and if statements. If there any other “more clever way” to manipulate this data?

I know it’s not an easy setup, but I hope I’ve given some further clarity to this.

Many thanks again for your time and help! Really appreciated! I’m rather new to coding and Ionic, been at it for a few months now, so sorry if some questions may sound stupid.


#9

This goes beyond my experience… maybe some others people could help you achieve this (I would suggest the creation of another thread even).

A few points:

  • Look more closely into queries because I thought you could sort by multiple value (might no be the case)
  • If there is no such thing then you could do the sorting in your page controller so that only the filtered list of sales listings is used with ngFor (do not filter them in the template).

I’m sorry I can’t help you more… I don’t have a lot of experience with queries so I can’t really give you a clear answer (maybe someone else here will hopefully)


#10

Your number of listings is in the thousands? Are all filters equally important? Or is there a ranking: most important city, second-most-important, etc? These questions get fancy fast, and you might want to read about map queries in a book on computational geometry.

But here’s a simple approach.

When you save or update a listing, you also update the list of “items in City X.” For each city, you maintain a list of unique IDs of listings in that city. Then when a query comes, you pull items from the list in city X, and check for existence in cities Y and Z. If you fail, keep sampling.

There are much better ways to do this, but they require more sophisticated data structures. The basic idea is that you pay a cost in space (redundant copies) in return for speed in search. Fortunately, since your data relations are represented in spatial two dimensions (a map), algorithms for these queries exist and are very fast, at scale.


#11

This might be relevant to what you’re inquiring about. Though, it is an old bit of code and a little messy. The gist of it is, I have a firebase database that is split into two main groups.

  1. Images
  2. Keywords that can be added to images

this bit of code has keywordToRemove (deleting a Keyword from the app), and imagesAttached (an array of firebase keys that belong to each image that the Keyword was attached to). Every time I add a Keyword to an image, I add that images firebase key to the Keywords ‘image keys’ array with a value of true. This function evaluates the Keyword, and deletes said Keyword from each image it was previously attached to.

Christ that makes no sense reading it, but, it works. Maybe you can glean something from the sample code.

the ‘Galleries’ ref is a reference to specific galleries of images according to users and dates.
the ‘Keywords’ ref is a reference to app-wide keywords that can be attached to any image.
‘this.masterKeys.userKey’ is the specific user’s id in fireabse database.
‘dateKey’ (your first snapshot resulting in snapshot.key) is each gallery of images (held in place by a date)
From Component

 deleteKeyword(keyword, i){
  let keywordKey = keyword[0] //this equates to the keyword's firebase key
  let imagesAttached = Object.keys(keyword[1]) //this equates to the array of firebase keys belonging to images the keyword is assigned to
  let len = imagesAttached.length - 1; //ignore this.
     imagesAttached.splice(len, 1); //ignore this
   this.keyService.deleteKeyword(keywordKey, imagesAttached)
   this.keywords = this.keyService.returnMainKeywords();
   }

In Provider (keyService)

   deleteKeyword( keywordToRemove, imagesAttached ) {
  //keywordToRemove is actually the Keyword's firebase key.
    let specificImage = "";
    let UserGallery = firebase.database().ref('Galleries').child(this.masterKeys.userKey);    
    
    for (let i = 0; i < imagesAttached.length; i++) {
      specificImage = imagesAttached[i];
      UserGallery.on("child_added", (snapshot) => {
         let dateKey = snapshot.key;
         UserGallery.child(dateKey).child(specificImage).child(keywordToRemove).remove();
      })    
    }
   firebase.database().ref('Keywords').child(this.masterKeys.userKey).child(keywordToRemove).remove()
  }

Again, this is dated, I did a poor job of naming, and it honestly makes little sense to me at this moment, but it works, and is fairly quick. It’s from my initial attempts at doing anything semi-complicated with firebase, the format needs to be flattened, etc., but if an idea comes from it, very well.

Further, It is not writing to, but deleting. I believe the concept will carry over, if what I provided has any relevance to your goal.


#12

Thanks a lot everybody for your awesome help! That’s pretty much how I’ve just implemented it. So far it works fine, but let’s see what happens when I get more data in (if I do that is…)