How to trap data from thousands of report files?

Altair Forum User
Altair Forum User
Altair Employee
edited November 2019 in Community Q&A
I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

------------------------------
Akitake Oguchi
Technical Manager
Altair Engineering Inc.
CHUO-KU
81362255826
------------------------------
Tagged:

Answers

  • Steve_Caiels
    Steve_Caiels
    Altair Employee
    edited November 2019
    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file. 

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • CPorthouse
    CPorthouse
    Altair Employee
    edited November 2019
    Steve Caiels_21881 said:

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file. 

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "

    Just to add on to Steve's answer, if you are using Monarch Classic (and have the appropriate license) you can use COM automation where you can call monarch from a command line.  This can be used in a batch file where you can process each file individually and append the output to a common export.

    ------------------------------
    Chris Porthouse
    Senior Implementation & Integration Engineer
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • Altair Forum User
    Altair Forum User
    Altair Employee
    edited November 2019
    Steve Caiels_21881 said:

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file. 

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?
    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • Altair Forum User
    Altair Forum User
    Altair Employee
    edited November 2019
    Chris Porthouse said:

    Just to add on to Steve's answer, if you are using Monarch Classic (and have the appropriate license) you can use COM automation where you can call monarch from a command line.  This can be used in a batch file where you can process each file individually and append the output to a common export.

    ------------------------------
    Chris Porthouse
    Senior Implementation & Integration Engineer
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "

    Hi Chris,

    Thank you for your advise. Actually I have not only PDF reports but also some Excel files, which I want to combine. However, I think COM automation is worth trying. Couldn't find any documents in my latest Monarch 15 installation but found in Monarch 14. I will look into it.

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    Japan
    81362255826
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 08:59 AM
    From: Chris Porthouse
    Subject: How to trap data from thousands of report files?

    Just to add on to Steve's answer, if you are using Monarch Classic (and have the appropriate license) you can use COM automation where you can call monarch from a command line.  This can be used in a batch file where you can process each file individually and append the output to a common export.

    ------------------------------
    Chris Porthouse
    Senior Implementation & Integration Engineer
    Altair
    ------------------------------

    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • Steve_Caiels
    Steve_Caiels
    Altair Employee
    edited November 2019
    Altair Forum User said:

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?

    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
    Hi Akitake,

    Drag and Drop should do it.  Just grab the files and make sure you drop them into the green highlight that will show up when you get the mouse over the original file.
    image
    Regards,
    Steve,




    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-20-2019 12:47 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?
    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------

    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • Altair Forum User
    Altair Forum User
    Altair Employee
    edited November 2019
    Steve Caiels_21881 said:

    Hi Akitake,

    Drag and Drop should do it.  Just grab the files and make sure you drop them into the green highlight that will show up when you get the mouse over the original file.

    image
    Regards,
    Steve,




    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-20-2019 12:47 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?
    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------

    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
    Thank you for your advice Steve. It works perfectly in my case. I love Monarch!

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    Japan
    81362255826
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-21-2019 03:08 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    Drag and Drop should do it.  Just grab the files and make sure you drop them into the green highlight that will show up when you get the mouse over the original file.
    image
    Regards,
    Steve,




    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-20-2019 12:47 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?
    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826

    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
  • Mahmoud
    Mahmoud
    Altair Employee
    edited November 2019
    Altair Forum User said:

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?

    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair
    ------------------------------

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "
    Hi Akitake,
    Copy all PDF files in one folder.  Launch DPS, select your model, in the prepare page under the table name (which should be the name of the model) select report option, select Add Report, navigate to the folder where all PDF files resides, select the first file, scroll to the bottom, hold the shift key, select the last file.  This should select all files,  then click on the Open button.  The model will be applied to all PDF files that are opened in DPS and the table will display all extracted data from all PDF document.
    The limit for open reports in one Monarch session for classic is 1024.  You can check the Monarch specifications by clicking the following links:
    Monarch Classic specifications: http://docs.datawatch.com/monarch/desktop/ht_cui_specs.htm
    Data Prep Specifications: http://docs.datawatch.com/monarch/desktop/DPS_Technical_Specifications.htm

    Regards
    Mo

    ------------------------------
    Mahmoud Abdolrahim
    Senior Implementation & Integration Engineer
    Datawatch Corporation
    MA
    (978) 935-3840
    ------------------------------
    -------------------------------------------
    Original Message:
    Sent: 11-20-2019 12:47 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    Hi Steve,

    Thank you for your reply. Yes, a single model is fine to me, but how can I easily add thousands of report files to a table? I know by selecting the menu 'Report Options' > 'Add Report' of the table, I can add a report to the table. But I'm afraid I can't add multiple files at once. Is there a good alternative?
    image


    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------

    Original Message:
    Sent: 11-14-2019 03:45 AM
    From: Steve Caiels
    Subject: How to trap data from thousands of report files?

    Hi Akitake,

    If a single model or work space is suitable for all reports, then Monarch can process around 1000 reports in a single session.  Of course, it may take a long time to load all of these - especially PDF files.  For a few thousand, you should be able to run several batches and use the Append function when exporting to create a single output file.

    For many thousands of files, or if this is an ongoing requirement, I would suggest using the server version (Automator).  You build the model or workspace on a sample, then deploy it onto the server version.  This will process as many files as you need and is only limited by the resources of the server.

    Regards,
    Steve.

    ------------------------------
    Steve Caiels
    Professional Services
    Altair

    Original Message:
    Sent: 11-13-2019 12:41 AM
    From: Akitake Oguchi
    Subject: How to trap data from thousands of report files?

    I have thousands of PDF report files and want to trap data from them and combine into one. I know it's easy to trap data from one or a few report files with Monarch, but how can I get scalability to handreds or thousands of files?

    ------------------------------
    Akitake Oguchi
    Technical Manager
    Altair Engineering Inc.
    CHUO-KU
    81362255826
    ------------------------------
    "

Categories