So after all the hassle trying to write code for third party storage providers i’ve decided to back to square one and use server based storage along with a custom script. This may take some time.

So the plan was simple…

1, Find interface to handle upload/downloads
2, Use a cloud storage provider to host the files
3, Make it work with curl
4, Job done

As a previous user of Amazon AWS S3 I didn’t have any real problems with and liked features such as Cloudfront, but for the Shunt project the price point was to high especially as people may potentially upload large files share them then leave them sitting dormant and forgotten while taking up costly disk space. As I already use the Backblaze B2 storage service for backups and with their very reasonable storage and bandwidth prices I decided to use them for shunt. Some work was done with nodejs and I was eventually able to upload to B2 via curl and have the download link returned. “You should be happy” I hear you say but alas the actual transfer mechanism became the stumbling block. So I had my node server running on a remote machine and from a local one I could do

curl -F ‘file=@/tmp/test.file’ IP ADDRESS OF SERVER/files

and once completed the download link would be returned. The issue with this implementation was that the file had to first be transferred to the node server and then again from the node server to backblaze and depending on the connection speeds this effectively doubled (or more) the time it took to upload a file. The second issue was that the file was uploaded to the servers ram and not the HDD, the advantage of this was that it was written very quickly and only a small HDD would be required BUT it meant that my test server with 4GB of memory would fill up very quickly if people were doing large simultaneous uploads and servers with large memory 64GB+ don’t come cheap. The download link went straight to backblaze so there was no issue downloading files. Then came the second problem, from the time the file was successfully uploaded to my server it took a further 25 seconds before it would start upload to backblaze and as well all know 25 seconds is an eternity in networking time. Even though shunt will be a free service this kind of upload time frames are unacceptable.

shunt.io what does it mean? what made me decide on the name?

When trying to decide on a name for this service I decided on three things it should be….

Relevant – No point it being totally unrelated

Memorable – When someone thinks about transferring a file it needs to be associated with memorable word/s

Short – UploadFilesFromTheCommandLine.com just isn’t going to work

I went throught the usual words, upload, send, transfer, etc and as expected all the top TLDS had been taken. I then began to consider other words that meant moving something from one place to another and thanks to my train anorak days I quickly came up with “shunt”. Again all the top tlds were taken but then I remembered the .io extension which is popular with tech start-ups and websites. A quick search showed shunt.io was available for registration so I quickly snapped it up. The added advantage of this tld is that IO is an abbreviations for input/output which makes it a perfect fit for this service.

noun

noun: shunt; plural noun: shunts

an act of pushing or shoving something.

shunt verb (MOVE)

to move someone or something from one place to another

So this new project of mine centres on the ability to be able to easily upload files from the command line. WHY? As a daily linux user I find it frustrating that in order to share files I with have to open a browser, type a url, click upload which is fine for a local system but for a remote headless system it’s just not viable.

Another issue is that some file hosters require special software to be installed in order to upload. After working with may different linux/unix flavours as well as routers os’s like openwrt and busybox all of which come with a basic set of software tools I found curl to be the most common in base installs hence the reason shunt.io is based around it.