Linux shell intro

StitchIt is targeted at a Linux analysis PC and some aspects, particularly syncAndCrunch aren't tested on Windows and there are no plans to do this. Basic Linux command line (shell) skills are therefore important but pretty easy to learn. The following is a bare-minimum list of Linux commands that users need to know.

Regular users: basic commands

• cd- for changing directory

• ls - for listing directories

• less and cat for displaying text file contents

• rm including rm -fr for removing files. Must understand why rm -fr is dangerous!

• mv - for moving and renaming

• mkdir - make directories

Regular users: remote working

• Using tmux for persistent shell sessions: tutorial.

• How to ssh into a Linux machine. How to log into the analysis machine remotely (including from off-site locations) is helpful to know.

• Using nano to edit text files. This may be useful for remote working.

More advanced users: useful tools to know

• The htop "task manager" for checking system CPU and RAM usage. You should understand what the load average means and how to tell if the machine is low on free RAM.

• Understand how to check disk space with ncdu.

Compressing raw data

All users will need to know how to compress the raw data directory. The administrator of the analysis PC may choose to make available easy scripts for this. If these aren't available, then users need to to understand commands of this form:

tar -I lzip2 -cvf rawData_sample123.tar.bz ./rawData

Backing up data to the server

It has been our experience that GUI-based file browsers on all operating systems sometimes do weird things with networked storage. For this reason it's better to copy files from the analysis machine to a storage server using the rsync command. For a tutorial see here. The general syntax you will need is:

rsync -av ./sampleDir_123 /mnt/server/lab_name/user_name/path/to/data

Understand the importance of adding a trailing slash in the source directory name (i.e. ./sampleDir_123 vs ./sampleDir_123/).

Note that it is common for centrally-managed servers to have tape backup systems that automatically backup new files. Files may well be classed as new if they are renamed or moved. Therefore it is good practice to copy files to what can reasonably be considered a final location on the server. In other words, don't make a habit of placing files in a temporary location on the server then moving them. Try to avoid a future situation where very large directories need to be moved or renamed.

Shell Examples

• Look at a recipe file at the command line.

  $ less recipe_FM_mar234_210314_121236.yml

• Extract all the lines listing how long a section took to acquire from the acquistion log file.

  $ grep 'FINISHED section' acqLog_FM_mar234.txt

• rsync data from a remote machine to the current directory

  rsync -av --progress [email protected]:/mnt/data/someSample ./