🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Using Timer element to retry when errors occur

User: "Tony_20992"
Altair Community Member
Updated by Tony_20992

Hi, 

We have a visual process within Automator that errors out due to a prior process importing a large file. 

Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.

It was mentioned in a training session to use a retry feature.  I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element.  Is the below process correct or should I have both Export elements feed into one timer/retry?  

Please note that both Dataprep Export processes is where the error mentioned above occurs.

image

Thanks in advance
Tony

------------------------------
Tony Ruffy
Sr Analyst

------------------------------

Find more posts tagged with

Sort by:
1 - 1 of 11
    User: "Steve_Caiels"
    Altair Employee
    Accepted Answer
    Updated by Steve_Caiels

    Hi Tony,

    We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that.  Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.

    The retry would not create a new log file.  All the 'check' and 'retry' events would appear in the log for the first run.  The 2nd run is triggering because, as far as Automator is concerned, there are two files to process.  MyFile.txt.Partial and MyFile.txt.

    You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work.  I think you will still see the creation of the temporary .partial file before it gets renamed to its final name.  If it happens very quickly, you may not see the .Partial file, of course. 

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 10-13-2020 10:57 AM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Hi Steve,

    The above is correct except the BOT is simply taking a file and saving it to a folder location.  The BOT is not renaming the file during this transfer.  I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing.  The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.

    I will perform additional testing including Mo's example leveraging a HUB and Timer element.

    Thanks again,
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst
    ------------------------------

    Original Message:
    Sent: 10-13-2020 05:00 AM
    From: Steve Caiels
    Subject: Using Timer element to retry when errors occur

    Hi Tony,

    In response to post number 12, here is what I think is happening . . .

    Run 1.
    1.a BOT initiates the download with a .partial file extension.
    1.b Automator process is triggered by the arrival of the .partial file.  The internal Automator work list is created and expecting to run everything against the .partial file.
    1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
    1.d The BOT renames the .partial file to .txt.  This is seen by Automator as a new file arrival and triggers Run 2.  Automator is unaware that the BOT has renamed the input file.  It is still expecting to be able to work with the .partial file.
    1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure

    Run 2
    2.a The .txt file is created by step 1.d, above.
    2.b Automator process is triggered by the "arrival" of the .txt file.  The internal Automator work list is created and expecting to run against the .txt.
    2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
    2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.

    You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs.  I think it will be the .partial file for run 1 and the .txt file for run 2.

    The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention.  Your existing process should work and you will not get any rogue log failures.

    If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two.  The first process, for example, could attempt to move the input file into a separate location.  It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it.  On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.

    Regards,
    Steve.


    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 10-12-2020 01:20 PM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Thank you Mo, I will make some adjustments with your above recommendations to test.

    Thanks
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst

    Original Message:
    Sent: 10-09-2020 08:16 PM
    From: Mahmoud Abdolrahim
    Subject: Using Timer element to retry when errors occur

    Tony,
    The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.
    image
    image
    I set the timer 'Retry at most' to 3.  You may play with the Timer and Check input settings and figure which settings are best for your Visual process.  I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer.  You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
    To simulate the process, I opened the input file in Monarch Classic, and started the process manually.  After the first try, I closed the input file, and noticed the process was completed.
    Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input.  However, because the Check Input failed once it will trigger an alert.  If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).
    image
    Regards
    Mo

    ------------------------------
    Mahmoud Abdolrahim
    Senior Implementation & Integration Engineer
    Altair Engineering

    Original Message:
    Sent: 10-01-2020 11:23 AM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Hi,

    We have a visual process within Automator that errors out due to a prior process importing a large file.

    Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.

    It was mentioned in a training session to use a retry feature.  I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element.  Is the below process correct or should I have both Export elements feed into one timer/retry?

    Please note that both Dataprep Export processes is where the error mentioned above occurs.

    image
    Thanks in advance
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst

    ------------------------------
    Hi Tony,

    Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.

    They also came up with what looks like a very neat solution!

    I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.

    The work flow would then be:

    1. The BOT starts the download of the file to MyFile.txt.partial.  As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
    2. The file is renamed to MyFile.txt when the download is complete.
    3. The Automator process is triggered by the arrival of the .txt file

    This would be a much neater workflow.  You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.

    Regards,
    Steve.


    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 10-13-2020 11:42 AM
    From: Steve Caiels
    Subject: Using Timer element to retry when errors occur

    Hi Tony,

    We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that.  Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.

    The retry would not create a new log file.  All the 'check' and 'retry' events would appear in the log for the first run.  The 2nd run is triggering because, as far as Automator is concerned, there are two files to process.  MyFile.txt.Partial and MyFile.txt.

    You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work.  I think you will still see the creation of the temporary .partial file before it gets renamed to its final name.  If it happens very quickly, you may not see the .Partial file, of course.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 10-13-2020 10:57 AM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Hi Steve,

    The above is correct except the BOT is simply taking a file and saving it to a folder location.  The BOT is not renaming the file during this transfer.  I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing.  The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.

    I will perform additional testing including Mo's example leveraging a HUB and Timer element.

    Thanks again,
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst

    Original Message:
    Sent: 10-13-2020 05:00 AM
    From: Steve Caiels
    Subject: Using Timer element to retry when errors occur

    Hi Tony,

    In response to post number 12, here is what I think is happening . . .

    Run 1.
    1.a BOT initiates the download with a .partial file extension.
    1.b Automator process is triggered by the arrival of the .partial file.  The internal Automator work list is created and expecting to run everything against the .partial file.
    1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
    1.d The BOT renames the .partial file to .txt.  This is seen by Automator as a new file arrival and triggers Run 2.  Automator is unaware that the BOT has renamed the input file.  It is still expecting to be able to work with the .partial file.
    1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure

    Run 2
    2.a The .txt file is created by step 1.d, above.
    2.b Automator process is triggered by the "arrival" of the .txt file.  The internal Automator work list is created and expecting to run against the .txt.
    2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
    2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.

    You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs.  I think it will be the .partial file for run 1 and the .txt file for run 2.

    The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention.  Your existing process should work and you will not get any rogue log failures.

    If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two.  The first process, for example, could attempt to move the input file into a separate location.  It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it.  On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.

    Regards,
    Steve.


    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 10-12-2020 01:20 PM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Thank you Mo, I will make some adjustments with your above recommendations to test.

    Thanks
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst

    Original Message:
    Sent: 10-09-2020 08:16 PM
    From: Mahmoud Abdolrahim
    Subject: Using Timer element to retry when errors occur

    Tony,
    The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.
    image
    image
    I set the timer 'Retry at most' to 3.  You may play with the Timer and Check input settings and figure which settings are best for your Visual process.  I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer.  You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
    To simulate the process, I opened the input file in Monarch Classic, and started the process manually.  After the first try, I closed the input file, and noticed the process was completed.
    Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input.  However, because the Check Input failed once it will trigger an alert.  If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).
    image
    Regards
    Mo

    ------------------------------
    Mahmoud Abdolrahim
    Senior Implementation & Integration Engineer
    Altair Engineering

    Original Message:
    Sent: 10-01-2020 11:23 AM
    From: Tony Ruffy
    Subject: Using Timer element to retry when errors occur

    Hi,

    We have a visual process within Automator that errors out due to a prior process importing a large file.

    Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.

    It was mentioned in a training session to use a retry feature.  I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element.  Is the below process correct or should I have both Export elements feed into one timer/retry?

    Please note that both Dataprep Export processes is where the error mentioned above occurs.

    image
    Thanks in advance
    Tony

    ------------------------------
    Tony Ruffy
    Sr Analyst

    ------------------------------