Multihops on Tramp

For Emacs users, we can streamline our workflow by configuring TRAMP to use multi-hops. Add the following to your config (usually .emacs, or .emacs.d/init.el; TRAMP is installed with Emacs, no additional packages required):

(add-to-list 'tramp-default-proxies-alist 
             '("gpsc" 
               nil 
               "/ssh:USERNAME@<LOCAL HOSTNAME>:"))

That allows me to open files on GPSC in my local Emacs via: C-x C-f /ssh:gpsc:FILENAME.

Execute code blocks with Org and Babel

You can use orgmode to run code blocks from a file on your local machine on GPSC. To do this, use a header like this in your local .org file (which assumes you’ve set up your .ssh/config files as described above):

#+PROPERTY: header-args:bash :results output :dir /ssh:gpsc:./path/to/working/directory

I ran into transient permission issues executing code blocks. I’ve fixed this by configuring org-babel to use my home directory for temp files. It seems to have problems writing to /tmp on GPSC, although I should have permission to do so? Here’s the code from my emacs init.el:

(setq org-babel-remote-temporary-directory "~/")

So far the temp files babel generates are cleaned up automatically, so this hasn’t created any detritus. You could use a subdirectory under ~/, but the directory must exist (i.e., you need to create it yourself) before org can find it - it won’t make it for you.

With this done, you can submit code blocks from your org file directly to slurm on GPSC. Here’s my template:

#+BEGIN_SRC bash :results output
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=slurm_test_emacs
  #SBATCH --output=slurm_test_emacs.log
  #SBATCH --open-mode=append
  #SBATCH --partition=standard
  #SBATCH --time=0:01:00
  #SBATCH --ntasks=1
  #SBATCH --cpus-per-task=1
  #SBATCH --mem-per-cpu=1G
  #SBATCH --comment="<SUBMIT_COMMENT>"
  #SBATCH --account=<ACCOUNT_NAME>

  date
  echo
  echo hello
  echo

  SUBMITSCRIPT
#+END_SRC

Everything between <<SUBMITSCRIPT and SUBMITSCRIPT will get passed to sbatch as if it were a separate script file.

If you do use this approach, note that you must escape any variables!. This won’t work:

#+BEGIN_SRC bash :results output
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=bad_HEREDOC
  <...>

  NAME=TYLER
  echo $NAME

  SUBMITSCRIPT
#+END_SRC

But this will (note the backslash in front of the $):

#+BEGIN_SRC bash :results output
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=good_HEREDOC
  <...>

  NAME=TYLER
  echo \$NAME

  SUBMITSCRIPT
#+END_SRC

A further wrinkle: this uses a heredoc, and heredocs need to write a temp file. By defaul this goes in /tmp, and in some cases /tmp may be full. If that happens, you’ll get an error

cannot create temp file for here-document: No space left on device

All users can write to /tmp, so you can’t fix this yourself. You can change the location where your heredoc gets written, though:

#+BEGIN_SRC bash :results output
  export TMPDIR=/path/to/your/personal/tmp/directory
  sbatch <<SUBMITSCRIPT
  #!/bin/bash
  #SBATCH --job-name=good_HEREDOC
  <...>

  NAME=TYLER
  echo \$NAME

  SUBMITSCRIPT
#+END_SRC

You may set this permanently by adding the export TMPDIR=... line to your .bashrc on GPSC.

Once a job is submitted, the ID will appear in your local .org file, e.g.:

#+RESULTS:
: Submitted batch job 1385545

You can retrieve info on the run with:

#+BEGIN_SRC bash :results output 
sacct --jobs=1385545 --format=jobid,jobname,state,elapsed,MaxRSS
#+END_SRC

In my case, this generated:

#+RESULTS:
: JobID           JobName      State    Elapsed     MaxRSS 
: ------------ ---------- ---------- ---------- ---------- 
: 1385545      slurm_tes+  COMPLETED   00:00:07 
: 1385545.bat+      batch  COMPLETED   00:00:07      2096K 
: 1385545.ext+     extern  COMPLETED   00:00:07      2088K 

Use the format argument to request which details to report. MaxRSS is the maximum amount of RAM used during the run. Other options are available via sacct --helpformat, with more details in the slurm manual.

Note that GPSC doesn’t provide the seff command, which you may have used on other clusters.

Linking to files on the remote server

It may be more convenient to manage more complex scripts as stand-alone .sh files on the GPSC. If the configuration above works for you, you can insert a link to remote files in your org file with the syntax:

[[file:/ssh:gpsc:~/path/to/script.sh][My Script]]

This will create a ‘hotlink’ in your file you can click to open that file in your local Emacs. The default keyboard shortcut for entering links is C-c C-l.

That will allow you to edit the remote file as if it were local. When you’re ready to submit it, you can do that from your local .org file:

#+BEGIN_SRC bash :results output
sbatch script.sh
#+end_src

Note that the location of your script will be relative to the directory you listed in your file header.

I also use org links for data files I might need to refer to or edit (i.e., sample indices), and files generated by the jobs themselves.