Problem
I have an AngularJS code to perform upload of files. At first I thought it works OK but when I’ve tried to upload bigger files (e.g. 5 files, 10 MB each) then I saw that this code has very poor performance regarding memory handling.
For example: when I attach 5 files (10 MB each = 50 MB in total) then peak browser process memory demand reaches 2 GB!
I’ve tested it in Firefox, Chrome and IE. Additionally, in Chrome when browser process reaches 3 GB of memory committed then the page crashes.
In Firefox I can see an error in console when trying to upload more than 80 MB at once:
out of memory
I’ve tried to modify the parts where the actual request is made and removed data: JSON.stringify(formattedData)
which was the first bottleneck. And it turned out that stringify
ing is redundant there.
But the second bottleneck still exists which is the function binArrayToJson
which takes the ArrayBuffer and reads the data into the byte array.
var binArrayToJson = function (binArray) {
var str = [binArray.length];
for (var i = 0, binLength = binArray.length; i < binLength; i++) {
str[i] = binArray[i];
}
return str;
}
var applyAttachments = function (data1) {
angular.forEach(data1,
function (value, key) {
if (key === "Attachments") {
for (var i = 0; i < value.length; i++) {
if (value[i].file) { //already saved item don't contain 'file' property
var reader = new FileReader();
reader.onloadend = (function (f) {
return function (result) {
var arrayBuffer = result.target.result;
value[f].fileContent = binArrayToJson(new Uint8Array(arrayBuffer));
value[f].isUploaded = true;
};
})(i);
reader.readAsArrayBuffer(value[i].file);
}
}
}
});
};
var createMVCModel = function (output) {
var defaultStringValue = "";
return {
id: output.id,
Name: output.name || defaultStringValue,
Status: output.status || "Draft",
Date: output.date || null
Attachments: output.attachments || []
};
};
var saveModel = function (data, url) {
var formattedData = createMVCModel(data);
var deferred = $q.defer();
applyAttachments(formattedData);
var check = function () {
if (allNewFilesUploaded(formattedData) === true) {
$http({
url: url,
method: "POST",
data: formattedData,
headers: { 'Content-Type': undefined }
})
.then(function (result) {
deferred.resolve(result);
},
function (result) {
deferred.reject(result);
});
} else {
setTimeout(check, 1000);
}
}
check();
return deferred.promise;
};
I omitted the parts where the following requirements are being met:
- checking number of files (limit is 10)
- checking size of each file (limit is 10 MB each)
- checking permitted file extensions
- sending rest of the (text) data along with files to the server (ASP.NET MVC method)
Solution
Marshaling the file contents as a byte array is obviously problematic.
Browsers can easily upload files through HTML forms, with
<form method="POST" enctype="multipart/form-data">
<input type="file">
</form>
Adapt the server to handle file uploads using that mechanism, then develop a replacement AJAX client.