Convert a Makefile into FDL (FlowTracer)
To convert a makefile into FDL, perform the following steps:
1. Review a fragment of a real Makefile. The original Makefile contained about eight steps, each with a different name. Here are the first two steps:
SHELL = /bin/csh WORKPATH = ./make STARTDIR = $(shell pwd) all: step1 step2 step1: mkdir -p $(WORKPATH); \ cd step1; \ csh run_step1.csh; \ cd $(STARTDIR); \ date > $(WORKPATH)/step1 step2: step1 mkdir -p $(WORKPATH); \ cd step2; \ csh run_step2.csh; \ cd $(STARTDIR); \ date > $(WORKPATH)/step2 2. Convert the instructions into FDL using SERIAL and TASK:
set WORKPATH ./make SERIAL { indir –create $WORKPATH {} SERIAL { indir step1 { TASK ./run_step1.csh } TASK date > $WORKPATH/step1 } indir –create $WORKPATH {} SERIAL { indir step2 { TASK ./run_step2.csh } TASK date > $WORKPATH/step2 } }
3. The files “$WORKPATH/step1” and “$WORKPATH/step2” are not really useful, so use this simple flow description:
set WORKPATH ./make indir –create $WORKPATH {} SERIAL { indir step1 { TASK ./run_step1.csh } indir step2 { TASK ./run_step2.csh } } 4. Use the foreach statement in Tcl to arrive at following flow description:
​SERIAL { foreach step [list step1 step2 ] { indir $step { TASK ./run_${step}.csh } } }
5. In this way, you reproduce substantially the same behavior that is implemented in the makefile. The additional benefit is that, you have a beginning of dependency management, while the example makefile does not really handle neither status nor dependencies. In fact, with the makefile, the repeated command triggers the execution of both step1 and step2.
% make step2 % make step2 % make step2 FlowTracer always manages status and dependencies so that the command will run the jobs in step1 first and then step2
% vsr step2 In this case, this command will “run all jobs in directory step2”. When called the first time, it will run the jobs in step1 first and then step2. But if you repeat the order again, FlowTracer will do nothing because it knows that the jobs are already done (i.e. “green”).
% vsr step2 % vsr step2 % vsr step2 6. Add attributes to the flow description. (This is only the beginning of the conversion. You can add a name to each job in the flow, as with the following example.
SERIAL { foreach step [list step1 step2 ] { indir $step { N “job_$step” TASK ./run_${step}.csh } } } Or you can decide describe the resources such as licenses and RAM required by each job, and perhaps also the expected duration of the jobs, as with the following example.
SERIAL { foreach step [list step1 step2 ] { indir $step { R “RAM/1500 License:synth” ; # Need 1.5GB of RAM and a license X 2h; # The job is expected to take 2 hours N “This is job $step” TASK ./run_${step}.csh } } } You can seed put all jobs into a named set as well.
S “my:named:set” { SERIAL { foreach step [list step1 step2 ] { indir $step { R “RAM/1500 License:synth” X 2h N “This is job $step” TASK ./run_${step}.csh } } } } 7. Add dependencies to the various steps in the flow.
S “my:named:set” { SERIAL { foreach step [list step1 step2 ] { indir $step { R “RAM/1500 License:synth” X 2h N “This is job $step” TASK ./run_${step}.csh > $step.log I –sticky ./run_${step}.csh O –sticky ./$step.log } } } }
8. After you learn more about the input/output behavior of each step, turn the FDL around once again and lose the SERIAL/TASK procedures that we have used so far and let FlowTracer figure out the dependencies based on the I/O behavior of the jobs.
For example, assuming that step2 uses as input step1.out:
R “RAM/1500 License:synth” X 2h S “my:named:set” { indir step1 { T vov ./run_step1.csh > step1.log I ./run_step1.csh O step1.out } indir step2 { T vov ./run_step2.csh > step2.log I ../step1/step1.out I ./run_step2.csh O ./step2.log } } } }
This flow description is a more complicated than the one based on TASK/SERIAL, but it is also more representative of the actual FDL used by experienced designers.