Fetching many records from backend

Hello guys,

I’ve been using infinite scroll component from ionic. This is my situation : I have search field and below it is infinite scroll which shows 1st 20 records. After user changes search input infinite scroll resets with that for records that only match that search input (e.g only records that start with ‘Ae’). I’ve done this by fetching all data from backend initially and then using array functions like filter to change what is displayed in infinite scroll. This all work very well when there are not too much records in database (around 150 from what i tested) but as i think application will grow i think i will have more records than that and i tried with 1000 records, time for loading the page is drastically changing and when i test on phone it loads forever.
So my question is can you suggest me better way to do this. I know it is bad to fetch all data when you dont use it all. I thought it would be better to do filtering inside vue with javascript functions then hitting backend with request every time search input changes(maybe this is bad?). Other thing i thought of is to make filtering on backend side and then just get chunks for infinite scroll to be populated?
Idk can somebody experienced point me in right direction, i just need the patter to use i will do reasearch on it myself, i just dont know what is best practice and what will not stress my app so much when there are too many records in databse. Thanks!

ion-virtual-scroll is not supported in Vue.

Maybe theres an alternative somewhere specifically for vue. If you do a search online, more than likely you will find what you seek.

I’ve decided to stick with infinite scroll and only load number of records that i actually use in my component at the time. Not taking all data but for exaample only first 20 will be shown, when users activates infinity scroll by scrolling down i append records to existing array of records(the initial 20 records) with new 20 records that i get.
Basically i made my backend just return stuff i need ,i even specified only columns that are being used and only number of records shown are return each time endpoint is getting hit.

Would you consider this a good solutin to given problen? Will it scale well?
Thanks for your help!

did you check this out? vue3-virtual-scroller - npm

Do you consider this to be better solution than what i proposed above?
I will look into it if you think so, but if they will give same performance or similar i might just stick to what i started testing.
Thanks for reply! Appreciate it

my attitude is if you have a working solution stick with it until next time!!

Generally speaking, I think this is a great solution. Backends tend to have way more resources than client devices do, and it’s also easy for you to throw more backend resources at the problem if that becomes necessary. That tends to go over much better than telling your users to buy faster phones.

There is one caveat, though: state management. HTTP is a stateless protocol, and REST APIs shine the best when they too can be stateless. If you find yourself needing to keep track on the server of which client has seen which records, and basically reinventing cookies or session IDs, I would tweak the design to do my best to avoid that.

@aaronksaunders @rapropos

Thanks for tips guys! Really appreciate it!

To sum this thread up for somebody that will maybe read it in future.

How to load your data from backend? -Make backend endpoints that will return ONLY data that you will use at that time. Example : If you want to show ,like me, initial 20 records of data DONT load all records that exist in your database. Load only 20! And after user scroll or click button or whatever action is performed to show more data then and ONLY then load more data but not more than needed.

This shrunk load on my application significantly(and more than that :D), also if in future you add 10k+ records or even more you dont need to worry because you only take data you need from backend and not all of it.

I didn’t use pagination as people here suggested but only because i needed chunks of data to show and i would never use pagination in my app. And pagination also sends its data in JSON so i thought it would be inefficient to use it. I made custom static method on my Eloquent model( i explained above what it does).

Since virtual scroll is not currently working with infinite scroll in Ionic v6.0.3, another option is lazy rendering of components in the list so their markup is removed when not in view leaving just a single div with the calculated component height to store it’s place on the list. The solution is explained here <Lazy> rendering in Vue to improve performance | by Martin Malinda | Vue.js Developers | Medium

To make this work well for my purposes I had to reduce the unrender delay and I am also disabling/enabling unrendering when routing to a detail page and coming back to the list so the scroll position doesn’t change. To do that I use the onIonViewWillLeave and onIonViewWillEnter events.