So the plan was simple…
1, Find interface to handle upload/downloads
2, Use a cloud storage provider to host the files
3, Make it work with curl
4, Job done
As a previous user of Amazon AWS S3 I didn’t have any real problems with and liked features such as Cloudfront, but for the Shunt project the price point was to high especially as people may potentially upload large files share them then leave them sitting dormant and forgotten while taking up costly disk space. As I already use the Backblaze B2 storage service for backups and with their very reasonable storage and bandwidth prices I decided to use them for shunt. Some work was done with nodejs and I was eventually able to upload to B2 via curl and have the download link returned. “You should be happy” I hear you say but alas the actual transfer mechanism became the stumbling block. So I had my node server running on a remote machine and from a local one I could do
curl -F ‘file=@/tmp/test.file’ IP ADDRESS OF SERVER/files
and once completed the download link would be returned. The issue with this implementation was that the file had to first be transferred to the node server and then again from the node server to backblaze and depending on the connection speeds this effectively doubled (or more) the time it took to upload a file. The second issue was that the file was uploaded to the servers ram and not the HDD, the advantage of this was that it was written very quickly and only a small HDD would be required BUT it meant that my test server with 4GB of memory would fill up very quickly if people were doing large simultaneous uploads and servers with large memory 64GB+ don’t come cheap. The download link went straight to backblaze so there was no issue downloading files. Then came the second problem, from the time the file was successfully uploaded to my server it took a further 25 seconds before it would start upload to backblaze and as well all know 25 seconds is an eternity in networking time. Even though shunt will be a free service this kind of upload time frames are unacceptable.
