I’m making a new app that uses a map as it’s core and the user can import some geographic data from KMLs and KMZs. Most of the time the app is intended to be used without internet connection. The user can import some zip files (KMZ) that contain geographic data which will be used within the map, some of these zips are unzipped in a single text file (KML) that have 300MB+, the data inside these files are then read to be load inside the map.
In the beginning I used Ionic Capacitor Filesystem plugin for reading the this unzipped text file and writing a new modified file in the a specific device’s FS folder, but I started to get some Out of Memory errors while doing this. After some research, I manage to solve the writing problem using a chunk-writing method, which uses way less memory.
My main problems now are:
1 – Most of the alternatives are using different types of data, like
File object, which are not easy to obtain with Capacitor FileSystem.
2 – Because of this I am forced to do some conversions, which involve reading the file as a whole, which always going to lead to an out of memory crash.
This is what I’m trying to do now, based on this Gist:
The problem with this approach is that I need the file as a Blob to read with chunks, but to convert the file to Blob I need to read it first, which return me to out-of-memory error.
I have tried some different approaches too:
1 - A plugin called read-chunk, but wasn’t exactly what I needed, the plugin read a part of the file using a start position and file’s length, at this point I tried to use it to read the file piece by piece, but the write algorithm needs a whole object to be able to write chunk by chunk, and this object need to be
File. That generate one more problem, the read-chunk plugin returns a
Buffer that I need to convert before writing, and I can’t convert a single piece and send do the write algorithm, because in the end I will have many files and not a whole file.