Automation Runs Very Slow when Reading Large Data AS400 Data Sets via ODBC

Question

I have created automation using Auto Com (Command Line automation) where Monarch 15.x is reading large data sets from the AS400 via ODBC connection.  The file sizes range from a few hundred records to over 3 million (15 to 30 fields).  As the number of records get above 50,000 the automation began to crawl.  The automation always complete successfully, but the wait time is painfully long.  The automation is set up to run during off hours, which is ok, but these automation scripts are also set up to run manually, on demand.

Note that I am using Monarch Classic to the the ODBC linking and processing.  My question is, firstly, could I integrate Data Prep Studio into the automation (again, we only have the Auto Com license).  If DPS could read the AS400 tables much faster, and create the required .csv file exports, hopefully that would speed up the automation.

Any advice would be helpful.

------------------------------
Al Rice
------------------------------

Al_22614 · Answer

Thanks Mahmoud:

I will do as you suggest.  Additionally, I will run the high volume processes on scheduler during off hours.

Thanks,
Al

------------------------------
Al Rice
------------------------------
-------------------------------------------
Original Message:
Sent: 11-12-2019 07:51 AM
From: Mahmoud Abdolrahim
Subject:  Automation Runs Very Slow when Reading Large Data AS400 Data Sets via ODBC

Hi Al,
There is no com automation for the DPS.
Even though there is no com automation for the DPS,  I would create the same job in DPS and test it manually, and see if DPS is faster than Classic or about the same.
If DPS is faster, then your other option would be to use the Monarch Server Automator, which you can automate the DPS jobs.
You also need to install the ODBC drivers for the AS400 on the Automator server.
Regards
Mo

------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Datawatch Corporation
MA
(978) 935-3840
------------------------------

Original Message:
Sent: 11-09-2019 07:13 PM
From: Al Rice
Subject: Automation Runs Very Slow when Reading Large Data AS400 Data Sets via ODBC

I have created automation using Auto Com (Command Line automation) where Monarch 15.x is reading large data sets from the AS400 via ODBC connection.  The file sizes range from a few hundred records to over 3 million (15 to 30 fields).  As the number of records get above 50,000 the automation began to crawl.  The automation always complete successfully, but the wait time is painfully long.  The automation is set up to run during off hours, which is ok, but these automation scripts are also set up to run manually, on demand.

Note that I am using Monarch Classic to the the ODBC linking and processing.  My question is, firstly, could I integrate Data Prep Studio into the automation (again, we only have the Auto Com license).  If DPS could read the AS400 tables much faster, and create the required .csv file exports, hopefully that would speed up the automation.

Any advice would be helpful.

------------------------------
Al Rice
------------------------------
"

Mo Abdolrahim · Answer

Hi Al,
There is no com automation for the DPS.  
Even though there is no com automation for the DPS,  I would create the same job in DPS and test it manually, and see if DPS is faster than Classic or about the same.
If DPS is faster, then your other option would be to use the Monarch Server Automator, which you can automate the DPS jobs.
You also need to install the ODBC drivers for the AS400 on the Automator server.
Regards
Mo

------------------------------
Mahmoud Abdolrahim
Senior Implementation & Integration Engineer
Datawatch Corporation
MA
(978) 935-3840
------------------------------
-------------------------------------------
Original Message:
Sent: 11-09-2019 07:13 PM
From: Al Rice
Subject: Automation Runs Very Slow when Reading Large Data AS400 Data Sets via ODBC

I have created automation using Auto Com (Command Line automation) where Monarch 15.x is reading large data sets from the AS400 via ODBC connection.  The file sizes range from a few hundred records to over 3 million (15 to 30 fields).  As the number of records get above 50,000 the automation began to crawl.  The automation always complete successfully, but the wait time is painfully long.  The automation is set up to run during off hours, which is ok, but these automation scripts are also set up to run manually, on demand.

Note that I am using Monarch Classic to the the ODBC linking and processing.  My question is, firstly, could I integrate Data Prep Studio into the automation (again, we only have the Auto Com license).  If DPS could read the AS400 tables much faster, and create the required .csv file exports, hopefully that would speed up the automation.

Any advice would be helpful.

------------------------------
Al Rice
------------------------------
"