I have an array of nested objects populated in a loop.
Something like:
var myArray = [
object1: {
//lots of nesting
},
...
]
Now I run a loop over this and want to overwrite from index 1 after every loop iteration, example:
function getNestedObject(i) {
//some object creation code
}
for (i = 0 ; i< 50000 ; i++) {
var item = _.cloneDeep(getNestedObject(i));
myArray.push (item);
if (myArray.length > 20) {
//delete all elements and start pushing from scratch
//Do i need to additionally destroy the items within each of these 20 objects being overwritten to avoid memory overflow?
myArray.splice(0,20);
}
}
This is required to avoid heap overflow due to heap space being gobbled up by the object array.
Do i need to additionally destroy the items within each of these 20 objects being overwritten to avoid memory overflow or automatic gc would happen within this scope?
I am not sure how big your actual objects in the array are but you can try lodash chunk method to divide an array into multiple arrays with the specified length if you already have an array of objects present like below.
To do that use lodash chunk method.
_.chunk(['a', 'b', 'c', 'd'], 2);
// => [['a', 'b'], ['c', 'd']]
_.chunk(['a', 'b', 'c', 'd'], 3);
// => [['a', 'b', 'c'], ['d']]
_.chunk(myArray, 20);
// [[arr1],[arr2] ... ]
// This will produce your resultantn arrays
Now, to answer your question about the process memory is that if the size of myArray is going to big and you split it into smaller arrays of size 20. In your process of the creation of an array, Even if you remove or splice the original array, at the end of the splitting process you still have the objects which have been split in memory, thus there is no cost cutting to say so. And you may still end up with memory overflow.
Try writing the split arrays to a temp file and read it back when needed.
for (i = 0 ; i< 50000 ; i++) {
var item = _.cloneDeep(getNestedObject(i));
myArray.push (item);
if (myArray.length > 20) {
//appeand the array temp file
fs.writeFileSync('myTemp.json', json, 'utf8');
//clear the temp array
myArray = [];
}
//nothing subastanital in memory at this point
}
note* above is pseudo code for explanation may need to change as per requirements
I would like to add the solution may vary depending on when you plan to use these objects or process them in your logic.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments