Mass transaction upload - TM to S3 interface

 9 Replies
 1 Subscribed to this topic
 45 Subscribed to this forum
Sort:
Author
Messages
bearcat32
Basic Member
Posts: 12
Basic Member
    What is your approach to uploading mass transactions (ie Merit or pay rate changes) into TM and having them reflect in S3? After a few tests it seems the delivered flows take hours/days to simply interface the data to the LT holding tables. There has to be a more efficient solution.

    Thanks for any advice.
    Srini Rao
    Veteran Member
    Posts: 149
    Veteran Member
      Hello, Last week we processed 9700 pay rate change transactions and it took 53 minutes to export to LT11.

      You check couple of things...
      1. Change process flow's log levels to "No Debug"
      2. Check out if you have any user exists on Lt11.
      Woozy
      Veteran Member
      Posts: 709
      Veteran Member
        The interface is really the best and only option that won't cause headaches down the road.

        Srini Rao is absolutely correct. I'm guessing you have logging turned on, which greatly decreases performance. We keep logging completely off for all of our batch flows unless there is something very specific we are troubleshooting - and then only with a very small batch.

        We do turn on some logging for short duration single record flows - i.e. Action flows - so that we track what happened in each event.

        Good Luck!
        Kelly Meade
        J. R. Simplot Company
        Boise, ID
        bearcat32
        Basic Member
        Posts: 12
        Basic Member
          Thank you both very much for your responses.
          If I may ask a few additional questions in terms of your process.

          *Do you use spreadsheet designer to load the actual pay rates to TM or another utility?

          *Do you commit the data to TM in batches (e.g 1,000 at a time) or in a single load (e.g. all 9,700 transactions at once)?

          *Do you use process immediately for these or do you run the export flows to interface all the records under one work unit?

          Thanks again!
          Woozy
          Veteran Member
          Posts: 709
          Veteran Member
            In our case, we generally use LSD when we need to do these loads. Sometimes we do them all at once, and sometimes we break them up - but this is usually because of the data source or calculation differences rather than performance reasons. It isn't unusual for us to upload 5000+ records at once with LSD, and we frequenly do tens of thousands - though it is very slow.

            Our TM to S3 interface runs on a schedule twice-a-day, and we generally don't run it manually, even for these types of processes. We just let the interface process pick up whatever is available when it runs.
            Kelly Meade
            J. R. Simplot Company
            Boise, ID
            Srini Rao
            Veteran Member
            Posts: 149
            Veteran Member
              We uploaded 9700 records in two batches. After uploading we schedule LT101 and export flow.

              bearcat32
              Basic Member
              Posts: 12
              Basic Member
                Thanks Kelly.

                In terms of benchmarking could you give a ballpark figure on how long 10,000 records would take you to upload via LSD
                as well as how long it would take for 10,000 records to interface to S3 from TM via the export flows? We are getting varying definitions of "slow"
                so trying to ascertain what other clients are experiencing.

                Hope to see you @ Inforum!
                Woozy
                Veteran Member
                Posts: 709
                Veteran Member
                  Unfortunately, I really don't know. I'm on the development side, so I'm not the one doing the uploads or running the interface.

                  I know the upload speed really depends on how many fields, which business classes, etc. I think we generally see LSD uploads of 1-5 records per second, but it could be significantly slower if the server is busy or if the upload is "wide" or complicated. By "slow" I mean slow in comparison to real batch processing engines like SQL.

                  On the TM to S3 interface side, when we do a mass pay change it may cause the interface to run for several hours instead of a few minutes. Again, I can't really give you a good estimate. Sorry.
                  Kelly Meade
                  J. R. Simplot Company
                  Boise, ID
                  Woozy
                  Veteran Member
                  Posts: 709
                  Veteran Member
                    By the way - unfortunately we won't be at Inforum this year. We were planning to be going live with our upgrade to LMRK/TM 10 this weekend. However, that has now been bumped out to next month. Lots to do.

                    Have fun anyway! Say "hi" to everyone for me.
                    Kelly Meade
                    J. R. Simplot Company
                    Boise, ID
                    Peter O
                    Veteran Member
                    Posts: 69
                    Veteran Member
                      I see this thread is old, but thought I would add my two cents anyways regarding how we perform merits/uploads:

                      -We use LSD for transactions that are approx 10,000 records which will take a few hours. It depends on if we're doing an upload during peak utilization as well.
                      -Merits are generally done with LSD as well, but anything that is done semi-regularly we develop into an IPA flow and run it which generally gives us better time (depending on the transactions).
                      -Our Exports generally take only take a few minutes and we schedule them to run daily (anywhere from a few hundred ~1000 transactions)
                      -We had the issue with exports taking a very long time as well (hours to days), but found that the issue was that our debugging was wide open (as stated above) - once we turned that off, our exports would run in a few hours for mass exports and a few minutes for the daily transactions.