Instructions to complete prior to August 4th.

Setup Instructions

In the introductory talk, tomorrow, August 4th, we’ll provide additional details about this web site and Zoom rooms and Slack channels for our parallel sessions. Here, now, we provide instructions for setting up your Cooley login enviornment, reserving one node for the day for hands-on lessons and setting up visualization tools.

Instructions here are divided into required and optional steps. We expect everyone to, minimally, complete all required steps here. The optional steps are likely to improve your experience by simplifying or improving performance of certain operations.

Required Steps

Please complete the following required steps prior to the beginning of the session on Tuesday, August 4th.

Log Into Cooley
- Use secure shell with compression, and trusted X forwarding enabled
```
ssh -l <username> -C -Y cooley.alcf.anl.gov
```
Copy Installed Software
- Once you are logged into Cooley, please execute the following instruction to create a local, editable copy of numerical package software.
```
cd ~
rsync -a /projects/ATPESC2020/MathPackagesTraining2020/HandsOnLessons .
```
  - Note 1: do not include a trailing slash, / in the path argument.
  - Note 2: You may be asked periodically throughout the day to re-execute this command to update your local copy if we discover changes are necessary.
Setup to use appropriate MPI and GCC
- The above software is built with gcc-8.2 and corresponding MPI. Please update your `~/.soft.cooley to have
```
@visit
@paraview
+gcc-8.2.0
+mvapich2-2.2-gcc820
@default
```
- And run
```
resoft
```
Confirm you can compile and run an example
- As a test case, use an example from hypre to confirm you can compile and run an example
```
qsub -I -n 1 -t 5 -A ATPESC2020 -q training
cd HandsOnLessons/hand_coded_heat
make mpi_test
mpicc mpi_test.c -o mpi_test
mpiexec -n 4 ./mpi_test
Size=4, Rank=0
Size=4, Rank=1
Size=4, Rank=2
Size=4, Rank=3
exit
```
- The qsub command reserves a cooley node for interactive work for 5 minutes. You may have to wait a moment for the interactive prompt on the reserved node to return.
- The above commands produce makefile and execution output. In particular the last echo command should produce a 0 response.
As soon after 9:30am, Tuesday , August 4th as possible, allocate an interactive node on cooley. The following command allocates a single Cooley node (-n 1) for 300 minutes (-t 300) using the ATPESC2020 allocation (-A ATPESC2020) and the queue reservation (-q training):
```
qsub -I -n 1 -t 300 -A ATPESC2020 -q training
```
The command blocks until the node is ready. Until the allocation expires (300 minutes in this example), all commands executed in the returned session will run on the allocated compute node; mpiexec can be used directly instead of going through qsub.
- Note 1: Please DO NOT run MPI jobs on the login nodes. Instead, run them on an allocated compute node.
- Note 2: Be aware, however, that any running job will be terminated when your allocation expires.
- Note 4: To enable X windows for visualization on the compute node, you can open a new terminal and login to the allocated compute node by doing ssh -Y cc0xx (cc0xx is your compute node id)

Visualization Tool Setup

By far, where visualization tools are concerned, the easiest thing to do is to install the tools on your local laptop/desktop and then using them in client/server mode to connect to and display data from Cooley.

Local Installations

Results from various applications we use today may involve visualization with VisIt, ParaView or other visualization tools. By far, the simplest and most reliable way to setup any of these tools is to create a local installation on your laptop and then transfer data files from cooley to visualize them locally. In track 4, you will have alread learned how to do this. Nonetheless, for your convenience links to instructions for installing these tools locally are provided here…

For VisIt, go here to find a suitable bundled executable installation for your system and then download and install it.
For ParaView, go here to find a suitable bundled executable installation for your system and then download and install it.

Once you have installed these tools, you can run them in client/server mode to connect to cooley and visualize here data there or you may manually transfer data between Cooley and your desktop/laptop system. These two modes of use are described briefly in the remaining two sections.

Using Local Installations in Client-Server Mode

A benefit from installing these tools locally is that once you have them installed locally, you can also configure them to run client-server where you run an instance locally but use that instance to log into a remote resource, such as cooley, and visualize data there without having to manually transfer it locally. To setup these tools for client-server operation…

Follow these instructions to setup and run VisIt client-server to Cooley.
- Note: VisIt versions 3.0, 3.1 and 3.2 are also installed even though the above instructions don’t mention that.
Follow these instructions to setup and run ParaView client-server to Cooley.

Using Local Installations and Manually Moving Data

Manually logging in to move data files each time you need to can become combersome. You can use a single scp command to copy many files using either file globbing or the -r recursive command-line option to copy whole directory trees as for example…

scp "cooley:/tmp/imag00*.png" .

scp -r cooley:/foo/bar/tree .

Beyond that, you may also want to have a look at…

to simplify the process of manually moving data over many iterations of examples and tests. For ex: to easily login to cooley - one can do:

add the following to ~/.ssh/config
```
Host cooley.alcf.anl.gov
  Compression yes
  ForwardX11 yes
  ForwardX11Trusted yes
  ControlMaster auto
  ControlPersist 12h
  ControlPath ~/.ssh/%r@cooley.alcf.anl.gov:%p
```
With this - the first time you login cooley.alcf.anl.gov - you need to provide passwd. But subsequent ssh/scp/sftp will go through this control master - and not ask for passwd

The instructions beyond this point are not necessary and are for experts only

Optional Steps

Using Remote Visualization Tools Through VNC

If you don’t want to bother with installing anything locally or if you run into problems with that and/or if you don’t want to constantly move data between cooley and your local system or setup passwordless ssh for that, a final option is to use VNC. Setting up a VNC connection involves a complex set of steps including SSH tunneling and port forwarding. It is described in some detail here. However, we have created a shell script for you to run to simplify this process.

But, a few challenges in using VNC on cooley are…

It is likely to be useful to you only if you are on OSX or Linux or have Cygwin on Windows.
You must have some vnc client installed on your local system
- It is built into OSX
- On linux, you will need either vncviewer or vinagre
Only the twm window manager is available
Windows can be sized way too large for the VNC wrapper window in which they are placed
Cutting and pasting between VNC windows and other windows may not work as expected.

With all those warnings and caveats aside, in addition, the setup script we provide does take the following actions to your login setup on cooley…

It adds some lines to your ~/.ssh/config file (for SSH control master)
It changes your ~/.vnc/xstartup If you do not wont these files changed…please do not run this script.

Here are the steps

Download the vnc setup script to a suitable directory on your local machine

atpesc_cooley_vnc_setup.sh

Run the script [but first fix file permissions]

chmod 755 ./atpesc_cooley_vnc_setup.sh
./atpesc_cooley_vnc_setup.sh <cooley-username>

It will prompt you for your cooley login password. Enter it as you normally would.

You may observer output such as…

Password: 
Warning: No xauth data; using fake authentication data for X11 forwarding.
Warning: No xauth data; using fake authentication data for X11 forwarding.

It will then ask you for a temporary VNC password. Use 8 chars, upper and lower case and at least one digit. Please do not use the password used here in this example.
```
Create temporary VNC Password: Hello246
```

It will ask you to confirm your password

You have entered "Hello246", is this correct?
1) Yes
2) No

Enter 1 for Yes or 2 for no and a chance to re-entry a password.

It will then attempt to allocate (e.g. reserve a node on cooley). While the node reservation request is processing, it will periodically print a messaging indicating it is still waiting for the allocation…

Warning: No xauth data; using fake authentication data for X11 forwarding.
Checking for allocation completion
process_mux_new_session: tcgetattr: Operation not supported by device
Warning: No xauth data; using fake authentication data for X11 forwarding.
Checking for allocation completion
Checking for allocation completion
Checking for allocation completion
Checking for allocation completion
Checking for allocation completion
Got allocation at cc006

Note, it can take a few minutes before an allocation is created. When the allocation is ready, it prints the node id. Above, that is cc006.

It then sets up a tunneling VNC connection to that node on port 22590.

Warning: No xauth data; using fake authentication data for X11 forwarding.
[1] 2479360
Warning: No xauth data; using fake authentication data for X11 forwarding.
Attempting to connect VNC to localhost:22590 - If this fails you can reattempt this manually

If all goes well, you will be prompted to enter your vnc password to connect to the vnc instance just created. Enter the password you created above (e.g. Hello246 in this example). Then, you should see a VNC window appear like so with an xterm running in it on the allocated node.
From here, you can
```
soft add +visit
soft add +paraview
```
to add VisIt and ParaView to your environment.
Finally, remember that part of the process of using VNC is to acquire a reserved allocation on cooley. So, you should not then also reserve any other cooley nodes apart from the one reserved with VNC through this script.