I have to make lots of HTTP requests for data to an API. I have a service with functions that give me the expected results via promises. My initial thought was to put the requests information in an array, then loop over it and call my service function to get the data, which then writes the data in another array (array2). After the long time the loop will take to get all the data I then write that array2 into storage for further usage.
Those of you that already got all the async stuff will (rightfully) laugh now, as of course this doesn’t work at all: As the promises are aync, the loop is finished in a few miliseconds and an empty array is written to storage. Meanwhile my browser (and debug API server!) almost crashes because of the thousands of HTTP requests I just sent out at the exact same moment. Yeah, async is hard for some of us
After I realized my mistake I thought about the problem a bit more and realized, that I need to limit the number of calls that are done at the same time. Probably my API service could handle that internally and start a waiting list of calls to it and only x requests are then fire in parallel.
But that of course won’t help with the problem that the loop still finishes instantly and I write my empty array to storag. So I thought that there has to be a way to check if a list of promises is finished. It does, it is called
promise.all still will DDOS my server and kill the browser.
More googling brought me to
async.mapLimit: https://caolan.github.io/async/docs.html#mapLimit Parallel execution, limited to y, collects all the responses, perfect.
But is this the way I want to go? Googling for mor information I of course ended up in the pit of 2015/16 nodejs posts about the different implementations (bluebird?) of everything and … UAH! I have no idea if this will even work and is not a monumental mistake.
So my question (finally):
I have x tasks to execute. What should I use to execute y of them in parallel until they are all finished (some not working should be handled by just continuing, not stopping all work)?