Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.
Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Best Answer
-
Steve Caiels_21881 said:
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.
I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------Hi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0
Altair EmployeeAnswers
-
Hi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Steve,Steve Caiels_21881 said:Hi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
------------------------------
-------------------------------------------
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Steve,Tony_20992 said:Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
------------------------------
-------------------------------------------
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
------------------------------
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Tony_20992 said:
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
------------------------------
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------Hi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Thanks Steve, I will make a few adjustments as you mentioned and test the process.Steve Caiels_21881 said:Hi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Tony,Steve Caiels_21881 said:Hi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.
Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Mahmoud,Mahmoud said:Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
-------------------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Tony,Tony_20992 said:Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
It can be difficult to be sure without knowing the details of your process (both business and Automator!) and the options used. However, my feeling is that in terms of the "file in use" issue, you can drop the retry and just use the 'Check Input' node to make sure the process is only run when the file has stopped growing. I'm thinking that you no longer see the "file in use" issue, right? If that's the case, it backs up my suspicion.
The timer element only controls its own connections. In your original process, it will retry the specific export if there is a failure. So while it might no longer be relevant for the "file in use", it could still be useful if the export destination is temporarily unavailable.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Tony_20992 said:
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------Hi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Steve,Steve Caiels_21881 said:Hi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Sorry for the confusion, the file name is "BP_AUD200G1-50W_10_01_2020.txt" where each week, the date reference will adjust accordingly. Currently there are no areas of this process which renames the file. I believe when "BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial" is referenced in the log, it is when the process is attempting to import the growing file. This name is given as a temporary filename while the file is growing during transfer. I hope this paints a better picture and answers your prior questions.
I will make another adjustment to add the Check input element and link it to both export processes. I accidently left a timer element within my process which wasn't linked and didn't realize it still runs. I will run a better test to confirm. I appreciate the time and feedback regarding this process.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 11:57 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Hi Steve,Tony_20992 said:Hi Steve,
Sorry for the confusion, the file name is "BP_AUD200G1-50W_10_01_2020.txt" where each week, the date reference will adjust accordingly. Currently there are no areas of this process which renames the file. I believe when "BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial" is referenced in the log, it is when the process is attempting to import the growing file. This name is given as a temporary filename while the file is growing during transfer. I hope this paints a better picture and answers your prior questions.
I will make another adjustment to add the Check input element and link it to both export processes. I accidently left a timer element within my process which wasn't linked and didn't realize it still runs. I will run a better test to confirm. I appreciate the time and feedback regarding this process.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 11:57 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
I've provided some screenshots and data below from my initial test. Can you confirm what causes or initiates the 2nd run referenced within below screenshots?
Updated ProcessThank you
File was transferred by BOT to folder location which initiated process.
First run that started at 11:13am completed with error due to partial file
Second run that automatically started had one expected warning but did complete with expected results.
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 12:29 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Sorry for the confusion, the file name is "BP_AUD200G1-50W_10_01_2020.txt" where each week, the date reference will adjust accordingly. Currently there are no areas of this process which renames the file. I believe when "BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial" is referenced in the log, it is when the process is attempting to import the growing file. This name is given as a temporary filename while the file is growing during transfer. I hope this paints a better picture and answers your prior questions.
I will make another adjustment to add the Check input element and link it to both export processes. I accidently left a timer element within my process which wasn't linked and didn't realize it still runs. I will run a better test to confirm. I appreciate the time and feedback regarding this process.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 11:57 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
-------------------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Thank you Mo, I will make some adjustments with your above recommendations to test.Mahmoud said:Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
-------------------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Reply moved to end of chain . . .Tony_20992 said:Hi Steve,
I've provided some screenshots and data below from my initial test. Can you confirm what causes or initiates the 2nd run referenced within below screenshots?
Updated ProcessThank youFile was transferred by BOT to folder location which initiated process.
First run that started at 11:13am completed with error due to partial file
Second run that automatically started had one expected warning but did complete with expected results.
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 12:29 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Sorry for the confusion, the file name is "BP_AUD200G1-50W_10_01_2020.txt" where each week, the date reference will adjust accordingly. Currently there are no areas of this process which renames the file. I believe when "BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial" is referenced in the log, it is when the process is attempting to import the growing file. This name is given as a temporary filename while the file is growing during transfer. I hope this paints a better picture and answers your prior questions.
I will make another adjustment to add the Check input element and link it to both export processes. I accidently left a timer element within my process which wasn't linked and didn't realize it still runs. I will run a better test to confirm. I appreciate the time and feedback regarding this process.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 11:57 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,I have a feeling that we are chasing the wrong options in this instance.If the error messages are related to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', then I still think there may be a separate issue that you will not be able to overcome with either the Timer or Retry nodes.If this is the case, then the issue is that the file is getting renamed when the download is complete and the steps below will be taking place . .1. File 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' first arrives and the process is started. The file that Automator is working on is set to 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial', for example.
2. The Check input waits because the file is still growing.
3. The download is finished, and your transfer program renames the file to ''BP_AUD200G1-50W_10_01_2020.txt'
4. The Automator process will now fail because the file it is expecting to work on no longer exists. Depending on timing of the rename, it could fail on the file check node or the Export node. In either case, the error message will reference the 'partial' file.
Is this what appears to be happening?
If so, I think there are a few solutions:
If you have control over the download process:
i) Not use the temporary naming. The 'check input' node will stop it being used before it is fully download.
ii) Download to a temporary folder, then rename and move into a new location that Automator is monitoring.
iii) Add something specific to the 'completed' file, such as Done_'BP_AUD200G1-50W_10_01_2020.txt;, so that you can reference this in the monitoring options.
If you can't change the process, then you may need to have a separate feeder process that recreates options ii) or iii) above using a Script.For example. An initial process can be triggered on *.* in the download folder. A 'check input' node will wait for the download to complete. The process will fail in the same way as your current process, but that doesn't matter. A script on the failure path could then copy the completed file to an "automator download" folder where your existing process doesn't have to worry about the .partial naming issues. The advantage of using a script is that it will not be restricted to working with the initial file name.
If I've misread the situation, then sorry for any confusion!
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-09-2020 10:56 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Mahmoud,
I am now wondering if I need to use the Timer element to retry this process. As mentioned in my original post, we have a BOT that transfers the source file into a specified folder location where the trigger is referencing. During the process where the BOT is transferring the file, this process is triggered. The file is fairly large and still locked by the BOT to where the process errors with "...because it is being used by another process.". The file actually becomes available shortly after this error when the file is fully transferred to the folder location.
My question, with two workspaces and data exports within this process, where in the process should the timer element be placed if I were to add it? Does the timer element have the entire process retry or just a specified portion? I believe with the check input element, the process will error without attempting to retry the process.
I hope that makes sense, please let me know if any further information is needed.
Thanks
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:56 AM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Hi Tony,
Your Visual Process contains two Workspaces and two exports, which means two job in one Visual Process. The way you designed it both jobs runs concurrent. As Steve mentioned you need to connect the Check Input object to the second Data Prep Export object or add another Check input and connect it to the second Data Prep Export. I would just connect the first Check Input and connect it to the second Data Prep Export.
You can the Check Input red button to a new email distribution, per Steve suggestion.
Another way to stop the process and generate an alert, without connecting the Check Input object to a Distribution object is to create a local alert in the Visual process. To do that follow these steps:
Click on the bell icon on the tool bar
Click on the + button in the code pick list select the SystemAlertProcessElementFailed option.
Check the Ignore Global Alert box (if the Check ignore global alert is not shown, you need to enable it in System Preferences>Automator)Click on the element Type pick list and select Check Input.Select email, and enter email information, click on the Ok button, and then save button.
You can create a global alert instead of a local alert, so when any check input in any process fails, the global alert will send an alert.
Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-08-2020 11:16 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
You can't stop the process running per se, but you can control which nodes run.
The general concept is that a workflow item such as an export or distribution will only run if all the inputs paths to it are complete. You can see this in action in your logfile screen shot.
Note how the 'DataPrep Export' does not run (0 events) because the path is never taken from the 'check input' node. Therefore, there is an unused path into the Export node.
However, the 'DataPrep Export (1)' is not dependant on the 'check Input' node, so despite there being no inputs, it will attempt to run. To stop it running, you can connect the same (or different) 'Check Input' node to the 2nd export.
Finally, right click on the canvas and select 'Display Advanced Connectors". You can now connect a distribution node to the red 'error' path of the 'Check Input' node to send your email.
I think that should do it.Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-08-2020 10:44 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Regarding the Check Input element, if no file is present as seen in the test scenario below, is there a way I can stop the process from running and simply sending an email notification that no file was present?
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-02-2020 01:51 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Thanks for the information, I modify and test my process using the Check input element. Since this file is being used by to flows within this process, would I need to setup only one Check Input as seen below? Steps leading up to this, we have a reporting tool which a BOT pulls the input file, saves as .TXT to where the trigger location is referenced. The wildcard was placed due to the date stamp added to the file name each day, but the actual file name is what you listed above as
"BP_AUD200G1-50W_10_01_2020.txt". I believe that partial temporary name file is only being referenced as the file is still being placed into the folder location. Its about a 16MB file being imported.
I'll perform testing early next week and provide any issues. Thanks again for your help!
------------------------------
Tony Ruffy
Sr Analyst
Gap Inc Store Labs
San Francisco CA
505-462-0613
Original Message:
Sent: 10-02-2020 05:31 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
I think the 'Check Input' would be the best option to look at, in this scenario. But the 'partial' in the file name makes me wonder about a possible complication. More on that after the screenshot!
Ignoring the possible 'partial' complication for a moment, the process below has an input file that takes a few minutes to grow and get finalised. Without any checks, the trigger will fire as soon as the file arrives and will attempt to process the incomplete file. Depending on factors such as file type and whatever application is creating the incoming file, this could result in either the process running against an incomplete file or the file locking issue you have seen.
I should mention that this is not generally an issue. In most cases Automator and the OS work out what is going on with exclusive modes etc and the process will only trigger when the incoming file is ready. But we do see this occasionally when files are arriving via some ftp clients or are located on a Windows share from a Unix box. Hence the check input node.
The 'Check Input' node has an option to cycle through a specified number of checks to make sure the file is no longer growing. The process will only run when the specified number of checks gives a consistent result. For example, if you know your application writes data at least every 30 seconds, then you should check it every 30 seconds and wait for 2 consecutive checks. Of course, you may want to build in a safety margin.
On the potential '.partial' issue; I'm guessing that the file name 'BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' will change to 'BP_AUD200G1-50W_10_01_2020.txt' once the download is complete?The wildcards are expanded into the worklist of files when the process is launched, so if this is the case, you will get an error. The process will be triggered on the 'partial' name and be expecting that as the workspace input. However, that file will be missing by the time the consistency check is complete and it tries to run the workspace.
Is it possible to either:
- a) Not use the temporary file naming for the download. This is often an option for ftp but can pose a risk if the download fails and the file is incomplete.
- b) Change the download script so that the .partial file goes into a temporary folder and gets moved to the 'inbox' when the download is complete.
Steve,
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------0 -
Tony_20992 said:
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
------------------------------
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------Hi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Hi Steve,Steve Caiels_21881 said:Hi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Hi Tony,Tony_20992 said:Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Steve Caiels_21881 said:
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------Hi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Hi Steve,Steve Caiels_21881 said:Hi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Interesting, ok, I will make this update to the monitoring setup. Would the below example meet this recommendation?
Current monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*
Updated monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*.txt
Thanks,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-14-2020 04:25 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Hi Tony,Tony_20992 said:Hi Steve,
Interesting, ok, I will make this update to the monitoring setup. Would the below example meet this recommendation?
Current monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*
Updated monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*.txt
Thanks,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-14-2020 04:25 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Yes, that looks fine. Don't forget to make the same change to the input file mask in the 'CRJ Weekly File Input' node, as well.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-14-2020 09:53 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Interesting, ok, I will make this update to the monitoring setup. Would the below example meet this recommendation?
Current monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*
Updated monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*.txt
Thanks,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-14-2020 04:25 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0 -
Hi Steve,Steve Caiels_21881 said:Hi Tony,
Yes, that looks fine. Don't forget to make the same change to the input file mask in the 'CRJ Weekly File Input' node, as well.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
-------------------------------------------
Original Message:
Sent: 10-14-2020 09:53 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Interesting, ok, I will make this update to the monitoring setup. Would the below example meet this recommendation?
Current monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*
Updated monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*.txt
Thanks,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
Original Message:
Sent: 10-14-2020 04:25 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
After a few test runs, the below process completed with no errors or warnings. Thank you as well as Mo's time and input on this process. It is in a much better place than where we started.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
-------------------------------------------
Original Message:
Sent: 10-14-2020 10:15 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
Yes, that looks fine. Don't forget to make the same change to the input file mask in the 'CRJ Weekly File Input' node, as well.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
------------------------------
Original Message:
Sent: 10-14-2020 09:53 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
Interesting, ok, I will make this update to the monitoring setup. Would the below example meet this recommendation?
Current monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*
Updated monitor file setup: CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W*.txt
Thanks,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-14-2020 04:25 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,
Although I was already pretty sure, I checked with product management team about Automator having nothing to do with the .partial naming and they confirmed that was the case.
They also came up with what looks like a very neat solution!
I believe the issue is caused by the .partial file being 'worked on' by Automator and disappearing half way through. You can avoid this by changing your monitoring in input file masks from *.* to *.txt.
The work flow would then be:1. The BOT starts the download of the file to MyFile.txt.partial. As this does not comply with a *.txt mask, Automator will ignore the file and the process will not start.
2. The file is renamed to MyFile.txt when the download is complete.
3. The Automator process is triggered by the arrival of the .txt file
This would be a much neater workflow. You would only get one job. You could almost certainly get rid of the check input and retry nodes in your process.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 11:42 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occur
Hi Tony,
We don't perform any file renaming on the input file as part of our processing, unless there is a distribution defined specifically to do that. Assuming you don't have any copy or move distributions (in other processes, maybe?) that are specifically set up to use the .partial renaming on files in that location, then I am pretty sure it is your download process that is generating the .partial file name.
The retry would not create a new log file. All the 'check' and 'retry' events would appear in the log for the first run. The 2nd run is triggering because, as far as Automator is concerned, there are two files to process. MyFile.txt.Partial and MyFile.txt.
You should be able to check this by disabling the monitoring and/or scheduling on this process so that it does not run, then watch the input folder while your BOT does its work. I think you will still see the creation of the temporary .partial file before it gets renamed to its final name. If it happens very quickly, you may not see the .Partial file, of course.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-13-2020 10:57 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi Steve,
The above is correct except the BOT is simply taking a file and saving it to a folder location. The BOT is not renaming the file during this transfer. I believe we are seeing the partial file naming extension due to the automator process looking for the file while it is still growing. The 2nd run referenced was the retry attempt which I feel was triggered by the Check Input, but please correct me if I am incorrect here.
I will perform additional testing including Mo's example leveraging a HUB and Timer element.
Thanks again,
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-13-2020 05:00 AM
From: Steve Caiels
Subject: Using Timer element to retry when errors occurHi Tony,In response to post number 12, here is what I think is happening . . .
Run 1.
1.a BOT initiates the download with a .partial file extension.
1.b Automator process is triggered by the arrival of the .partial file. The internal Automator work list is created and expecting to run everything against the .partial file.
1.c Automator enters the 'Check Input' node and periodically looks at the .partial file to see if it is still growing
1.d The BOT renames the .partial file to .txt. This is seen by Automator as a new file arrival and triggers Run 2. Automator is unaware that the BOT has renamed the input file. It is still expecting to be able to work with the .partial file.
1.e The 'Check Input' node is no longer able to find the .partial file and generates the log failure
Run 2
2.a The .txt file is created by step 1.d, above.
2.b Automator process is triggered by the "arrival" of the .txt file. The internal Automator work list is created and expecting to run against the .txt.
2.c Automator enters the 'Check Input' node and periodically looks at the .txt file to see if it is still growing
2.d Once the .txt. passes the required number of 'Check Input' consistency checks, the process continues as expected.
You should be able to verify this by checking the filespec of the 'CRJ Weekly File Input' node in the logs. I think it will be the .partial file for run 1 and the .txt file for run 2.
The neatest way to overcome this would be to change the BOT so that it doesn't use the .partial naming convention. Your existing process should work and you will not get any rogue log failures.
If you are not able to change the BOT behaviour, then it may be beneficial to split the process into two. The first process, for example, could attempt to move the input file into a separate location. It would still suffer from the disappearing .partial file, but as you are expecting this failure, you could ignore it. On the successful runs, it would move the .txt file into a new location where your 'main' process would be working.
Regards,
Steve.
------------------------------
Steve Caiels
Professional Services
Altair
Original Message:
Sent: 10-12-2020 01:20 PM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Thank you Mo, I will make some adjustments with your above recommendations to test.
Thanks
Tony
------------------------------
Tony Ruffy
Sr Analyst
Original Message:
Sent: 10-09-2020 08:16 PM
From: Mahmoud Abdolrahim
Subject: Using Timer element to retry when errors occur
Tony,
The following workflow shows how to add a retry by a combination of a Hub, Timer, and Check Input.I set the timer 'Retry at most' to 3. You may play with the Timer and Check input settings and figure which settings are best for your Visual process. I suggest you start with 60 second and 2 for number of checks for the Check Input, and 5 sec and Indefinitely for the Timer. You can check the job log (sort it started Ascending) and determine the duration time when the file is released, and adjust the Timer and Check Input settings.
To simulate the process, I opened the input file in Monarch Classic, and started the process manually. After the first try, I closed the input file, and noticed the process was completed.
Please note that if after 60 seconds the file is not released, the process goes through the Timer and then back to Check Input. However, because the Check Input failed once it will trigger an alert. If the file is released after the first try, then it should go through normal execution flow (after the Check Input time is expired).Regards
Mo
------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Altair Engineering
Original Message:
Sent: 10-01-2020 11:23 AM
From: Tony Ruffy
Subject: Using Timer element to retry when errors occur
Hi,
We have a visual process within Automator that errors out due to a prior process importing a large file.
Error: The process cannot access the file 'Reports\CRJ Weekly\Downloads\Monarch Input\BP_AUD200G1-50W_10_01_2020.txt.x1xhc4b.partial' because it is being used by another process.
It was mentioned in a training session to use a retry feature. I've added it to the below process but was not sure if the retry element has the entire process restart or just a specific element. Is the below process correct or should I have both Export elements feed into one timer/retry?
Please note that both Dataprep Export processes is where the error mentioned above occurs.Thanks in advance
Tony
------------------------------
Tony Ruffy
Sr Analyst
------------------------------
0