1. Retrieving data

This document describes the method to download data from URL and from commonly-used repositories.

Using curl or wget

Both curl and wget commands can be used to download files or documents from command line using a URL. Do note that wget is only available on Linux distros.

curl -O <url>
wget <url>

Using git

git is a distributed version control system that tracks changes in computer files locally or on a remote server such as GitHub. Our lab use git routinely to download repositories from GitHub and to collaborate on software development and deployment.

To download packages, run:

git clone <repository>

You can version track on existing directories/projects by running:

cd /path/to/existing/codes
git init

Once initialised, any changes made in the directory and its subfolders will be tracked. The diagram below describes the general git workflow.

1. New files need to be added to the staging area:

git add <path to files> #or
git add .               #add all files

2. Changes in existing files will automatically be added to the staging area.
3. These new files/changes need to be committed before it is "saved". Try to do this in small chunks and provide a simple description of the commit:

git commit <file> -m "Commit message"  #or
git commit -am "Commit message"  #to commit all files

4. Once a bunch of commits have been gathered or when you wish to store the changes onto a remote server (GitHub), you need to push the changes

git push origin <branch name>

5. Make sure that your local repository is up-to-date with the remote before pushing by running both of these commands:

git fetch --all
git pull --all

If you wish to pull specific branches, replace --all with origin <branch name>.

For each project, 2 mandatory branches need to exist:

• main: public deployment branch. Code needs to be fully functional.
• stage: testing branch before deployment. Checks are done here Apart from these, unlimited number of development branches or GH-pages branch may exist in the repository for specific purposes. Keep these other branches neat.

Below are some functions when working with branches:

git checkout -b <new branch name>    #create new branch
git checkout -b <new branch name> <diff branch>   #create new branch based on <diff branch>
git checkout <branch name>           #switch branches
git merge <diff branch>              #merge <diff branch> into current branch

SRA (Sequence Read Archive)

It is highly recommended to create a delimted text file (CSV or TSV) containing at least a column that lists out the SRA IDs for download. The code below parses through the text file and downloads the data from SRA using fastq-dump:

cd /PATH/TO/PROJECT/originals
mkdir -p FOLDERNAME && cd $_

while IFS=$'\t' read -r SRA COLUMN2 COLUMN3;do
    fastq-dump --gzip --split-files $SRA
done < /PATH/TO/TEXTFILE

Notes:

• Make sure the input text file contain no header names
• Change the paths to project diretory and input text file accordingly
• Change $'\t' to $',' for comma-delimited input files
• SRA,COLUMN2 and COLUMN3 are newly-created variables containing a vector for each column

FTP servers

The code below is commonly-used to download entire directories from FTP servers:

# create data directory
cd /PATH/TO/PROJECT/originals
mkdir -p FOLDERNAME && cd $_


sftp user@ftpserver   # password will be prompted
get -r ./folder_name  # to download entire folder
exit

Alternatively, use secure copy (scp) to move files from servers into the workstation. This was usually done when downloading raw files from King's Genomic Facility. Download instructions are usually provided, which looks something like this:

scp -r user@sheba.genetics.kcl.ac.uk:/data/S697 ./.

Make sure to perform MD5 Checksum on each downloaded file and compare it with the given checksum. This is one way to perform MD5 checksum:

cd /PATH/TO/DIRECTORY
for files in *.md5;do
    fastq=${files%.md5}
    cat $files
    md5sum $fastq
done

Checksum values will be printed in command-line and visual inspection can be done.

Google Drive

The code below can be used to download file from Google Drive

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=FILEID" -O FILENAME && rm -rf /tmp/cookies.txt

Notes:

• Replace FILEID with the file ID which can be found in the sharing link of file
• Replace FILENAME with the name of the file

For example, if you'd like to download the file in the following link

https://drive.google.com/file/d/17-_VGhgAPt3St-X3dVT7trJ45rOoeZQH/view?usp=sharing

Then, the FILEID is '17-_VGhgAPt3St-X3dVT7trJ45rOoeZQH' and the FILENAME is 'M0_2017-10-13_000.tif' .

wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=17-_VGhgAPt3St-X3dVT7trJ45rOoeZQH' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=17-_VGhgAPt3St-X3dVT7trJ45rOoeZQH" -O M0_2017-10-13_000.tif && rm -rf /tmp/cookies.txt