Submit multiple jobs from a file (NetworkComputer)
When using NC, you usually submit jobs one at a time, from the command line or within a script. Sometimes you may want to submit large numbers of similar or different jobs at once.
There are several ways to accomplish this for the following different kinds of jobs:
- Jobs with the same command line and resources
- Jobs with the same resources, but different command lines
- Jobs where the resources and command lines differ
- Jobs with order dependencies
Jobs with the same command line and resources
These jobs can often be handled by a job array, using the -array option of nc run. The array index may have a start, end and incremental value, but the default is set with a range from 1..N. The symbolic variables @INDEX@, @JOBID@, and @ARRAYID@ are substituted in the env, wd, toolname jobname and output filenames. For example:
% cd ~/tmp
% nc gui &
% nc run -r unix -e BASE -array 2000 hostname
Jobs with the same resources, but different command lines.
These jobs may be handled with the -f option of nc run. Prepare a file containing the job command lines, each on a separate line, and pass the name of the file to the -f option. For example:
% nc run -r unix -e BASE -f my-cmds.txt
Jobs where resources and command line differ
Create a file that consists of a collection of nc run command lines, and run it as a shell script. This gives flexibility, but the job submission is slower because each job involves starting a new instance of vovsh and interpreting all the Tcl code of the nc run command.
For a collection of jobs where both the resources and command lines differ, you can use a FlowTracer FDL (Flow Description Language) file in the context of NC, with the vovbuild command. A significant difference between NC and FlowTracer is that NC does not have runtime tracing, so your FDL file must be complete and not rely on runtime tracing to discover I/O dependencies.
This method enables you to specify the parameters of each job separately, but economically because of the way FDL works. You can specify the command line, environment and resources, and you can organize jobs into sets for easy tracking.
Jobs that have order dependencies
If your jobs have order dependencies, you can use -dep on the command lines of a file of nc run commands, or you can use FDL as described above, using the FDL I and O statements to specify the files (or phantom dependencies) that link the jobs.