Beginner's guides
Getting Started Resources
Table of Contents: Tips & tricks on getting programming help • Introduction to HPC • Linux command line • Git version control • Building software with EasyBuild • Managing Cluster Instability for Users • Scientific documents and static site generators • Emacs talks
“Tips & tricks on getting programming help”
Webinar (2020-11-13) by Marie-Hélène Burle
Introduction to HPC
- Supercomputing in Canada: An Introduction (PDF slides)
- High Performance Computing: Short Introduction - Univ. of Manitoba perspective (PDF slides)
“How to submit and run jobs on Compute Canada HPC systems”
Webinar (2018-Sep-19) - Introduction to working on our HPC systems for new users.
“Best practices in HPC/HTC environments”
Webinar (2024-Oct-15) by Roman Baranowski
Most current Digital Research Alliance of Canada systems, as well as many local advanced research computing (ARC) systems and clusters support high-performance computing (HPC) and high-throughput computing (HTC) workflows. HPC workflows tend to focus on tightly coupled parallel jobs and, as such, they must execute within a particular site with low-latency interconnects. Conversely, HTC workflows consist of independent, mostly sequential jobs that can be individually scheduled on many different computing resources. In this webinar, we guide you through the best practices for deploying your HPC or HTC computations on our clusters:
- Interacting with the available hardware (login and data transfer nodes, compute nodes, interconnect etc.)
- Using the File Systems (where to store your data)
- Working with the available software and installing/building your own
Interacting with the scheduler (testing your jobs scripts, submitting jobs)
- PDF slides
Linux command line
For a basic introduction to Bash command line – as well as programming in Python or R, and version control with Git – please check out a Software Carpentry workshop closest to you. For an online tutorial, we highly recommend their Introduction to using the shell in a HPC context.
“Fun tools to simplify your life in the command line”
Webinar (2020-02-19) by Marie-Hélène Burle
“A few of our favourite tools”
Webinar (2022-03-16) by Marie-Hélène Burle and Alex Razoumov
In this webinar we talk about the following command-line tools:
- lazygit a wonderful terminal UI for Git,
- bat: a great syntax highlighter,
- ripgrep: a fast alternative to grep,
- fd: a really fast alternative to find, and
- pass: a command line password manager.
In the presentation, we also use a few other neat command-line tools such as hyperfine - for sophisticated benchmarking - and diff-so-fancy - which makes your diffs a lot more readable. For the Emacs users, we finish the workshop with two Emacs utilities:
- TRAMP: a remote file access system,
- Helm: a “framework for incremental completions and narrowing selections”.
“Turbo boost your interactive experience on the cluster with tmux”
Webinar (2022-11-23) by Chris Want
Have you experienced issues with unstable SSH connections that prevent you from being productive on your favourite cluster? Do you sometimes wish you had an extra terminal open so that you could run additional programs while waiting for tasks to complete? tmux is a terminal multiplexer (installed on the clusters) that can provide solutions to both connectivity and productivity sore points. It keeps your work session alive (even under the worst network conditions) while also providing you with the power to turn your terminal into a tiled window interface. This talk walks through some of the best features that will help you get work done on the cluster.
“Lesser known but very useful Bash features”
Webinar (2022-12-07) by Alex Razoumov and Marie-Hélène Burle
Knowing basic Linux commands is essential for researchers using remote systems such as HPC clusters. Bash is the most commonly used Linux shell, which you will use by default on most Alliance hardware. Although we teach Bash basics in various online schools and in-person workshops many times a year, there are some useful Bash features and tricks that we never get to teach, due to our usual time constraints. Finally we can share some of them with you in this webinar!
In this presentation, we talk about running commands in a subshell, subsetting string variables, Bash arrays, modifying separators with IFS, running Python code from inside self-contained Bash functions, editing your command history, running unaliased versions of commands, handy use of bracket expansion, and a few other topics.
Git version control
“Lazygit”
Webinar (2024-Oct-29) by Marie-Hélène Burle
While it is important to know how to use Git from the command line, this makes for an austere experience: a series of commands are required to gather information on the state of the working tree, see changes to files, or get a schematic of the commit history. Committing sections of files interactively and other operations are just awkward affairs. On the other hand, the many graphic interfaces for Git are often buggy, slow, and limiting.
One option is to write exciting functions with tools such as fzf to make things more friendly and visual. A simpler and more polished option is to use an already built user interface for Git that runs directly in the command line. Lazygit is one such open source tool. After years of development, it is a mature, beautiful tool that allows to perform any Git operation in the command line in a convenient, fast, and visual fashion.
In this webinar, I demo how I use lazygit in my daily workflow to run routine as well as more complex Git commands.
Building software with EasyBuild
“Installing software using EasyBuild in your cluster home directory”
Webinar (2021-12-08) by Ata Roudgar
“Building software on Compute Canada clusters using EasyBuild”
Webinar (2020-04-15) by Ali Kerrache
Managing Cluster Instability for Users
Presentation on dealing with hardware and software instabilities on Cedar from a user’s perspective: causes, workarounds, bad workflows, and best practices. It was part of a larger WestGrid’s March 2019 Town Hall – the embedded video below starts at the relevant section.
- Combined PDF slides from all presenters from WestGrid’s March 2019 Town Hall
Authoring scientific documents and static site generators
“The new R Markdown: authoring dynamic scientific documents with Quarto”
Webinar (2023-04-11) by Marie-Hélène Burle
This webinar shows how you can easily create beautiful publications – webpages, PDFs, websites, presentations, books, etc, complete with formatted text, dynamic code and figures – with Quarto. Quarto is the successor to R Markdown. By combining the powers of Jupyter or knitr with Pandoc, it works with R, but also with Python and Julia code blocks, adding new functionality to the old tool.
Emacs
“Emacs as a programming IDE for Python, R, and Julia”
Webinar (2023-Nov-14) by Marie-Hélène Burle
Once upon a time (not that long ago), powerful text editors such as Vim and Emacs were the only nice interfaces to work with code. Nowadays, there are countless sleek and more GUI-oriented tools such as VS Code, RStudio, or JupyterLab that provide amazing IDEs, without the learning curve. So why would one still use Emacs as a programming IDE? What does that even look like? In this webinar, I show some of the many reasons why I can’t let go of Emacs, then show how it can be used as a programming IDE for Python, R, and Julia.
“Modern Emacs: all those new tools that make Emacs better and faster”
Webinar (2024-Mar-05) by Marie-Hélène Burle
Emacs might have been created in the 70s, but its development is alive and well. 10 years ago version 24 brought huge speedups with lexical binding. In 2022, version 28 added – among other things – just-in-time native compilation for elisp code for improved performance. Version 29 last year brought countless exciting new additions such as official tree-sitter support and built-in Eglot and use-package. In addition to Emacs itself, a profusion of modern packages have emerged over the past few years (e.g. the vertico/consult/orderless/marginalia/embark completion system; corfu and cape for at point completion) bringing great speed and sleekness to the user experience.
“Understanding Emacs modes”
Webinar (2024-Sep-17) by Marie-Hélène Burle
At the core of Emacs functioning are modes: major modes set the appearance and behaviour of Emacs for various types of texts (e.g. a Python script and a Markdown document display different syntax highlighting and have different functions activated), while minor modes provide additional features that can be turned on or off (e.g. spell checking). Understanding how modes work is key to customizing Emacs and exploiting its strengths. In this webinar, I explain the functioning of Emacs modes and show how to manage and even customize them. Finally, I demo how the package Polymode allows to embed sections of a type of text in another type (e.g. snippets of code in a Markdown document).