Raspberry Pi Experiments - File Server using Hard Drive

For long I wanted to host a cloud file server for my needs. As my file collection was growing bulky and taking a lot of space on my laptop and on my wife's hard disk. It was absolutely necessary that I would have to buy a external hard disk. There were two features that I wanted in my cloud file server solution.
1. To be hosted from my home
2. To be accessible from any place

For point 2 I restored to opening a ssh tunnel from home Raspberry Pi to my AWS instance. For more details check my blog http://priyabgeek.blogspot.in/2016/08/raspberry-pi-experiment-ssh-reverse.html

Having read about the how a hard disk can be connected to Raspberry pi and can be mounted like in any other linux system I got a Seagate 2 TB SATA Barracuda 3.5" internal Desktop Hard Disk - ST2000DM006 from eBay and got a local 3.5 Inch USB 2 Hard disk case. The important thing here is that I used a Hard disk case with external power supply as this is important because the Hard disk can not work by taking current from Raspberry Pi via USB. And with Barracuda hard disk I also got an external fan which really made it a great deal as now I do not have to worry of cooling the hard disk. Now the tricky part was the USB connector. The case did not have one but the compatible USB cable that comes with Arduino UNO did work well and after connecting did not have any issues.

Now coming to the commands that I used to setup the file system.

First check if hard disk is detected

pi@raspberrypi:~ $ lsusb
Bus 001 Device 004: ID 152d:2329 JMicron Technology Corp. / JMicron USA Technology Corp. JM20329 SATA Bridge
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

If you detect a SATA Bridge driver it means your disk is detected.

For my setup I used the instructions given in https://devtidbits.com/2013/03/21/using-usb-external-hard-disk-flash-drives-with-to-your-raspberry-pi/.

Though there are known issues with NTFS on Raspberry PI and Raspbian I chose the file system because of the sheer portability and support I can get out of NTFS as a network files system which I did not expect out of a fat32 file system because of its formatting issues and its max 2gb limit for the file size. The issue with ext4 or similar linux file formats is that they are not supported natively on Mac or Windows. Hence I went ahead with NTFS as the file system for my external hard disk.

List your file systems:

sudo fdisk -l 

Device     Boot Start        End    Sectors  Size Id Type
/dev/sda1        2048 3907029167 3907027120  1.8T 83 Linux

Before formatting unmount the USB drive to be safe:

sudo umount /dev/sda1

For more clarity on the disk used and partition details I use test disk which can be installed as given below:

sudo apt-get update
sudo apt-get install testdisk

Launch testdisk command as given in the below screencast:

sudo testdisk

 

Now we need to install the ntfs drivers:

sudo apt-get install ntfs-3g

After installing it we can format the hard disk as below

sudo mkfs.ntfs /dev/sda1 -f -v -I -L disk-name 

We can check the disk details using the command:

pi@raspberrypi:~ $ sudo blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="22E0-C711" TYPE="vfat" PARTUUID="4bad8776-01"
/dev/mmcblk0p2: UUID="202638e1-4ce4-45df-9a00-ad725c2537bb" TYPE="ext4" PARTUUID="4bad8776-02"
/dev/sda1: LABEL="segate-dash" UUID="0EEB5BXXXXXXXX" TYPE="ntfs"
/dev/mmcblk0: PTUUID="4bad8776" PTTYPE="dos"

Next activity was to setup the file server. For this I needed very basic HTTP Server of showing the list of files and a basic service of uploading some files. And to expose this server via ssh tunnel I used the instructions given in http://priyabgeek.blogspot.in/2016/08/raspberry-pi-experiment-ssh-reverse.html

For the file server I used one of my projects https://github.com/bobquest33/gofileserver. You can download it using git and install the golang from the binaries at https://golang.org/dl/ and download the latest ARM version of go binaries. As the scope of how to install go and compile the above project is out of scope of this blog I will cover it in a different post or leave you to explore it for your self. Finally what you get is a file server. Now its not mandatory that you use my approach you can also use your own approach as well.

What I did however was to first mount the formatted hard disk to the below path:

sudo mkdir /mnt/usbdrive
sudo mount /dev/sda1 /mnt/usbdrive
ls /mnt/usbdrive

Then pointed the gofileserver to the given path /mnt/usbdrive and added it to crontab:

@reboot nohup sudo /home/pi/code/gofileserver/gofileserver :8001 /mnt/usbdrive & 

Now this setup is subjective and you can do a better job but this is what I felt as the best option given the circumstances.

And finally I was able to see the files:

Now for most of the cases it works fine and I am also having a interface where I can upload the files but that is still work in progress and I am trying to fix. But at the end what matters most is that where ever I am I am getting a read only version of my file which is great. Now the only thing I want to do next it to add https support and more user friendly UI which I will take as my future project.

Coming to the issues, one of the biggest issues which I faced was if I try to use a file upload program like scp or html multi-part file upload the file upload is fast but the server hangs. Also some times the server goes offline. Now I do not know if this is because of the NTFS driver or some other reason. But as a start now I can access my files from anywhere and use it.

If you have any better suggestions of feedback please do comment below.