mk notes

• each node has 2 CPUs (2 cores each) and 2 GPUs (240 cores each)
• name convention: G0-... or G1-... for GPU jobs
• qacct -j <JID> -o < username > (usage report)
• qalter -N <newname> $JOB_ID (changes job name during submission of a job - within script)
• qhost (show status of all hosts)
• qsub -p 100 .... (low priority of job, qalter could also be used)
• qrsh -pe smp 8 (reserves the whole node until logout)
• qstat -u '*' (all users) [see also online]
• qstat -f (full output only for myself)
• qstat -r | sed '/Master/ ,+d' | sed '/Hard Resources/ ,+1d' (show complete job names)
• qstat -s z (list of completed jobs)
• cexec "command" (execute command on all nodes, except node01)
• script: #!/bin/bash; cexec $1 && echo "--------- node01---------" && $1 (execute on all nodes including node01)
• measure turnaround time via 'time' or get_ow_usage ??
• mpicc (mpi compiler)
• mpirun -np

• #$ -cwd (starts job in current working directory)
• #$ -pe mpi 2
• $HOME home directory on execution machine
• $USER user ID of job owner
• $JOB_ID current job ID
• $JOB_NAME current job name (see -N option)
• $HOSTNAME name of the execution host
• $TASK_ID array job task index number
• qalter -N <newname> $JOB_ID (changes job name during submission of a job - within script)

• nvcc example.cu -o example.out
• ./example.out
• nvcc Options:
• -deviceemu (enables device emulation (needed for cuda-gdb))
• -ptax-options=-v (displays register and shared memory usage)
• --keep --opencc-options -LIST:source=on (keeps ptx-code with
  commented source aside)

• GPU handling, without changing the scheduler. Idea no 01
• Script qresub if required GPUs are not available. If available, qalter -N Gx and proceed. This should ideally accompanied by a bad nice value for CPU-demanding jobs.

execute from node or within script: /scratch/cuda_memtest-1.1: ./sanity_check.sh [0 or 1 or nothing] -> echo $? yields 0 if ok [test 0 before and after program executing]

todo: /opt/NVIDIA_GPU_Computing_SDK/C/bin/linux/release/deviceQuery says Compute mode: .. multiple .. [J proposes: Exclusive mode, no GPU numbers anymore, ideal solution?]

gpu reservation [gpu=1] node reservation [gpu=2]

#!/bin/sh
#$ -N g0_jobname
#$ -S /bin/sh
#$ -cwd
#$ -l gpu=1
#$ -q all.q@node01
sleep 10d

mdgpu - a molecular dynamic simulation program using GPUs by Julian Engel