From c12bbdfc8849da5b0ff3e737a13d2d8010585946 Mon Sep 17 00:00:00 2001
From: patmjen <patmjen@dtu.dk>
Date: Wed, 16 Nov 2022 10:55:30 +0100
Subject: [PATCH] Simplify Python guide
---
HPC_Python_Guide.md | 87 ++++++++++-----------------------------------
1 file changed, 18 insertions(+), 69 deletions(-)
diff --git a/HPC_Python_Guide.md b/HPC_Python_Guide.md
index 33cb0dc..1270fd1 100644
--- a/HPC_Python_Guide.md
+++ b/HPC_Python_Guide.md
@@ -23,78 +23,27 @@ sxm2sh
a100sh
```
-## II. First time virtualenv setup
+## II. Setup Virtualenv
+1. Get to a CPU or GPU node on the cluster (see [Section I](#i-preliminaries)).
-1. Navigate to your project folder.
+2. Navigate to your project folder.
-2. Load modules for Python by entering:
- ```
- module load python3/3.9.14
- module load numpy/1.23.3-python-3.9.14-openblas-0.3.21
- module load scipy/1.9.1-python-3.9.14
- module load matplotlib/3.6.0-numpy-1.23.3-python-3.9.14
- module load cuda/11.6
- ```
- We load `numpy`, `scipy`, and `matplotlib` as modules, because the HPC team have made optimized versions for the HPC cluster.
-
- > __NOTE:__ This guide uses Python 3.9 and CUDA 11.6 but other versions are available.
-
-3. Create a virtualenv by running:
- ```
- virtualenv <VENV_NAME>
- ```
-
-4. Activate the virtualenv by running:
- ```
- source <VENV_NAME>/bin/activate
- ```
-
-You should now be able to install packages with pip install <PACKAGE> as normal.
-
-> __Troubleshooting:__ If pip doesn't work, you may need to manually install it with:
-> ```
-> easy_install pip
-> ```
-
-## III. Virtualenv activation
-
-> __NOTE:__ These steps must be done every time. Also if you change from a login node to a GPU node (e.g. by calling `sxm2sh`)
-
-1. Navigate to your project folder.
-
-2. Load modules for Python by entering:
- ```
- module load python3/3.9.14
- module load numpy/1.23.3-python-3.9.14-openblas-0.3.21
- module load scipy/1.9.1-python-3.9.14
- module load matplotlib/3.6.0-numpy-1.23.3-python-3.9.14
- module load cuda/11.6
- ```
- We load `numpy` and `scipy` as modules, since the HPC team have made optimized versions for the HPC cluster.
+3. Download `scripts/init.sh` and place it in your project folder. This **only** needs to be done the first time.
+ > **Tip:** You can do this by calling
+ > ```bash
+ > wget https://lab.compute.dtu.dk/patmjen/hcp_tutorials/-/raw/main/scripts/init.sh
+ > ```
+ > in the terminal.
-3. Activate the virtualenv by running:
+4. Call
```
- source <VENV_NAME>/bin/activate
+ source init.sh
```
+ This will setup and activate your virtualenv. You must **do this every time** you log in or change node (e.g. by calling `sxm2sh`)!
+ > **Tip:** You can configure the Python version, CUDA version, and name and location of the virtualenv by changing the variables at the top of `init.sh`.
-> __Pro tip:__ To make life easy, put these commands in a bash script called `init.sh`:
-> ```
-> #!/bin/bash
-> module load python3/3.9.14
-> module load numpy/1.23.3-python-3.9.14-openblas-0.3.21
-> module load scipy/1.9.1-python-3.9.14
-> module load matplotlib/3.6.0-numpy-1.23.3-python-3.9.14
-> module load cuda/11.6
->
-> source <VENV_NAME>/bin/activate
-> ```
->
-> which you can then run by entering:
-> ```
-> source init.sh
-> ```
->
-> and this will prepare everything
+5. Your are done! You can now install packages with `pip install <PACKAGE>` and run python3 code with `python`.
+ > __Troubleshooting:__ If pip doesn't work, you may need to manually install it with: `easy_install pip`
## IV. Jupyter notebooks on ThinLinc
@@ -108,7 +57,7 @@ You should now be able to install packages with pip install <PACKAGE> as normal.
a100sh -X
```
-3. Navigate to your project folder, load modules, and activate the virtualenv. Same steps as in section III.
+3. Navigate to your project folder and activate the virtualenv. Same steps as in [section II](#ii-setup-virtualenv).
4. Install Jupyter with:
```
@@ -135,7 +84,7 @@ You should now be able to install packages with pip install <PACKAGE> as normal.
1. Open a terminal on the cluster, either through ThinLinc or ssh.
-2. Call `sxm2sh` or `linuxsh`, as described in section I, so you are not on a login node.
+2. Call `sxm2sh` or `linuxsh`, as described in [section I](#i-preliminaries), so you are not on a login node.
3. Start a tmux session by running:
```
@@ -153,7 +102,7 @@ You should now be able to install packages with pip install <PACKAGE> as normal.
echo $HOSTNAME
```
-5. Navigate to your project folder, load modules, and activate the virtualenv. Same steps as in section III.
+5. Navigate to your project folder, and activate the virtualenv. Same steps as in [section II](#ii-setup-virtualenv).
6. Start a Jupyter lab or Jupyter notebook server by entering one of the following:
```
--
GitLab