Setup Instructions
All handson activity will be on ThetaGPU. Here are instructions that are common for all the lessons.
Using ThetaGPU
- Log into Theta, then ssh from there to one of the GPU service nodes
- Use secure shell with trusted X11 forwarding enabled
ssh -l <username> -Y theta.alcf.anl.gov ssh -Y thetagpusn1 # or thetagpusn2
- Use secure shell with trusted X11 forwarding enabled
- Copy Installed Software
- Once you are logged into ThetaGPU, please execute the following instruction
to create a local, editable copy of numerical package software.
cd ~ rsync -a /eagle/ATPESC2023/EXAMPLES/track-5-numerical .
- Note 1: do not include a trailing slash,
/
in the path argument. - Note 2: You may be asked periodically throughout the day to re-execute this command to update your local copy if we discover changes are necessary.
- Note 1: do not include a trailing slash,
- Once you are logged into ThetaGPU, please execute the following instruction
to create a local, editable copy of numerical package software.
- Schedue a ThetaGPU compute node for compiles and runs
qsub -I -q single-gpu -n 1 -t 60 -A ATPESC2023 --attrs filesystems=home,eagle
- Note 1:
single-gpu
queue provides a single GPU and 16 cores - for (max) 60 minutes, we would need to rerun qsub as needed. - Note 2: One cannot (compile or) run the binaries on the frontends theta, thetagpusn1 or thetagpusn2. Use the allocated node for such usage.
- Note 3: ThetaGPU job scheduling policies document
- Note 4: To enable X windows for visualization on the compute node, you can open a new terminal and login to the allocated compute node by doing
ssh -Y thetagpuXY
(thetagpuXY
is your compute node id)
- Note 1:
- Load the required MPI, blas, lapack modules
module load openmpi/openmpi-4.1.4_ucx-1.12.1_gcc-9.4.0 aocl/blis/blis-3.2 aocl/libflame/libflame-3.2
Miscellaneous ssh instructions
Usig ssh control master
feature helps with repeated access to theta
. For this, one can add (say on your laptop) the following to ~/.ssh/config
Host theta.alcf.anl.gov
Compression yes
ForwardX11 yes
ForwardX11Trusted yes
ControlMaster auto
ControlPersist 12h
ControlPath ~/.ssh/%r@theta.alcf.anl.gov:%p
With this setup - the first time you login theta.alcf.anl.gov - you need to provide passwd. But subsequent ssh/scp/sftp will go through this control master - and not ask for passwd
Beyond that, you may also want to have a look at…