Prev
Next
Forums Infor / Lawson Platforms S3 Systems Administration

Upgrade 9.0.0 to 9.0.1 - Slow MS Addins Performance on PROD only

 23 Replies
 0 Subscribed to this topic
 27 Subscribed to this forum
Sort:
Author
Messages
Scott Whitfield
Basic Member
Posts: 7
Basic Member
11/1/2011 3:02 PM
    This is my first post and I thank everyone in advance for their time.


    We recently did a fresh new install of Lawson 9.0.1 along with the latest websphere, LUU etc...data was exported from our exissting data from a 9.0.0 env

    At this point, everything is fine except with Addins performance.

    The new environment is Virtual with Windows 2008 Server std x64 and SQL 2008 Server R2 x64 with separate PROD and TEST environments.

    The environments are equal except for PROD is using Raw Data Maps for dedicated space.


    Performance numbers for a 1000 record update from Excel 2010 using the latest MS Addins for Lawson ----

    TEST - 1 min

    PROD - 20 min

    If we do a direct upload/change/add query to a temporary table in SQL on PROD we get the 1 min result


    Any thoughts, ideas, suggestions are welcomed.


    Thank you,




    The addins error log is as follows:

    "DATE: 10/6/2011  TIME: 3:32:44 PM
    "
    "ACTUAL CALL STRING THAT GENERATED THE ERROR:
    "
    "http://fhcplawprod.fhcp.local/servlet/Router/Transaction/Erp?_PDL=PROD&_TKN=GL40.1&_EVT=CHG&_RTN=DATA&_TDS=IGNORE&FC=Change&_ADDINS=TRUE&_f1=C&_f39=10&_f44=2011&_f45=9&_f46=GL&_f48=N&_f49=1000&_f50=&_f67r0=&_f68r0=A&_f69r0=10&_f70r0=10600200&_f71r0=77000&_f75r0=0&_OUT=XML&_EOT=TRUE"
    "
    ---------------
    "
    "ERROR KEY = TRANSACTION_ERROR
    "
    "ERROR MESSAGE =  Error occurred while performing transaction

    "
    "ERROR DETAILS:
    "
    " com.lawson.tesla.agent.TransactionException: TRANSACTION_ERROR
     at com.lawson.tesla.exec.Tesla.invoke(Tesla.java:374)
     at com.lawson.tesla.agent.ErpTransactionAgent.processRequest(ErpTransactionAgent.java:184)
     at com.lawson.ios.agent.container.AgentContainerImpl$AgentWrapper.processRequest(AgentContainerImpl.java:479)
     at com.lawson.tesla.broker.TransactionBroker.processRequest(TransactionBroker.java:78)
     at com.lawson.ios.agent.container.AgentContainerImpl$AgentWrapper.processRequest(AgentContainerImpl.java:479)
     at com.lawson.ios.agent.container.AgentContainerImpl.processRequest(AgentContainerImpl.java:255)
     at com.lawson.ios.servlet.Router.doGet(Router.java:212)
     at com.lawson.ios.servlet.Router.doPost(Router.java:293)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
     at com.lawson.servlet.TransformFilter.doFilter(TransformFilter.java:106)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
     at com.lawson.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:96)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
     at com.lawson.servlet.XSSValidatorFilter.doFilter(XSSValidatorFilter.java:99)
     at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
     at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
     at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:934)
     at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
     at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
     at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
     at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
     at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
     at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:445)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:504)
     at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:301)
     at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
     at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
     at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
     at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
     at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
     at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
     at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
     at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
    Caused by: java.lang.IllegalArgumentException: Whole value too large
     at java.lang.Throwable.(Throwable.java:67)
     at com.lawson.rdtech.crtio.CrtioNumber.set(Unknown Source)
     at com.lawson.rdtech.crtio.CrtioNumber.set(Unknown Source)
     at com.lawson.rdtech.crtio.CrtioServiceMessageBuffer.setTypeNumericInt(Unknown Source)
     at com.lawson.rdtech.crtio.DynamicCrtioMsg.setFldValue(Unknown Source)
     at com.lawson.rdtech.crtio.DynamicCrtioMsg.setFlds(Unknown Source)
     at com.lawson.tesla.exec.Tesla.createCrtioBuffer(Tesla.java:1077)
     at com.lawson.tesla.exec.Tesla.invoke(Tesla.java:314)
     ... 42 more
     "
    "
    "



    Alex Tsekhansky
    Veteran Member
    Posts: 92
    Veteran Member
    11/2/2011 1:24 PM
      USUALLY such slow performance is caused by WebSphere setup issues, lack of RAM, or missing indexes in the database. I would check the latter first. To check whether it's WAS or the database, try exporting data out using command line tools (e.g. rngdbdump) and import them back into a table where addins are slow. If rngdbdump/importdb is fast, then the issue is on a web side of things.
      SP
      Veteran Member
      Posts: 122
      Veteran Member
      11/2/2011 2:37 PM
        Have you ran the "verifyxxxx" util (where "xxxx" equals appropriate db
        platform)?

        I would start there. Could be corrupt/missing indexes.
        SP
        Veteran Member
        Posts: 122
        Veteran Member
        11/2/2011 2:37 PM
          How as the data exported/imported? Was this with native db tools, Lawson
          tools, etc...??
          SP
          Veteran Member
          Posts: 122
          Veteran Member
          11/2/2011 2:42 PM
            Scott,

            How is performance with 1 record on PROD? 5 records? 50 records?
            100?..???

            Does anything appear in the ladb.log?

            Also, after running verifymsf2000.exe, you might want to look at your
            maintenance plans on the new db and make sure that weekly, they include a
            db integrity check, index reorg, and update stats.
            Scott Whitfield
            Basic Member
            Posts: 7
            Basic Member
            11/3/2011 7:59 PM
              Thank you for the quick responses... I will have the team take each of your recommendations and pull the requested metrics.

              Scott Whitfield
              Basic Member
              Posts: 7
              Basic Member
              11/8/2011 12:48 PM
                I moved the virtual test environment to the same host and blade center as prod ... this move did not affect addins in TEST

                I ran bldmsf2000ddl -URIq PROD and after completion ran addins ... 1000 records still takes 20 minutes in PROD

                I'm thinking maybe it's a setting/config within websphere?
                John Henley
                Posts: 3353
                11/8/2011 1:11 PM
                  Is query wizard slow, or just upload wizard?
                  Thanks for using the LawsonGuru.com forums!
                  John
                  John Henley
                  Posts: 3353
                  11/8/2011 1:14 PM
                    Also, make sure you don't have tracing turned on in WAS; DO NOT look at the settings via WAS console (I don't trust it)...look for the traces/log files in the WAS directory tree...if they are huge, you have tracing turned on in WAS.
                    Thanks for using the LawsonGuru.com forums!
                    John
                    Jimmy Chiu
                    Veteran Member
                    Posts: 641
                    Veteran Member
                    11/8/2011 1:41 PM
                      The same portal server is hosting both your PROD and TEST portal right?
                      SP
                      Veteran Member
                      Posts: 122
                      Veteran Member
                      11/8/2011 4:54 PM
                        What were the results of the verifymsf2000?

                        If you run verifysmf2000 on both TEST and PROD, do they report the same
                        numbers for tables, indexes, etc...?
                        Scott Whitfield
                        Basic Member
                        Posts: 7
                        Basic Member
                        11/8/2011 7:31 PM
                          yes... same number of indexes, tables, and no inconsistencies found.
                          John Henley
                          Posts: 3353
                          11/8/2011 8:57 PM
                            Are your test and prod running in same websphere instance and lawson environment ?
                            Thanks for using the LawsonGuru.com forums!
                            John
                            SP
                            Veteran Member
                            Posts: 122
                            Veteran Member
                            11/8/2011 9:48 PM
                              Do you have multiple productlines in the production environment (e.g. Train, test, etc...)?

                              Did you recompile PROD (cobcmp) after the LSF upgrade?
                              Scott Whitfield
                              Basic Member
                              Posts: 7
                              Basic Member
                              11/9/2011 12:08 PM
                                there is a test and a prod websphere instance

                                yes, a cobcmp was run post LSF upgrade
                                John Henley
                                Posts: 3353
                                11/9/2011 1:41 PM
                                  Going back to the beginning, you included an error message from the Addins. Are you getting that for each row that you are trying to upload? If you're getting the error, are you sure the addins upload is actually working?
                                  Thanks for using the LawsonGuru.com forums!
                                  John
                                  Greg Moeller
                                  Veteran Member
                                  Posts: 1498
                                  Veteran Member
                                  11/9/2011 4:26 PM
                                    We are noticing some of the same issues just with job performance after our MSP 5/ESP 7 upgrade. Weird thing is TEST and PROD are now very close to the same levels, but TEST can run a GL190 importing 27,000 JE in about 3 minutes and in PROD the same job takes 2+ hours.
                                    John Henley
                                    Posts: 3353
                                    11/9/2011 5:14 PM
                                      That's interesting since batch is not using websphere. Are they in the same environment?
                                      Thanks for using the LawsonGuru.com forums!
                                      John
                                      Karen Ploof
                                      Veteran Member
                                      Posts: 118
                                      Veteran Member
                                      11/9/2011 5:33 PM
                                        Are the security rules in TEST the same as those in PROD? I know that my TEST environments tend to have less complex security since access to that environment is limited. If security is more complex in PROD, and an update job runs faster in TEST than PROD, could it be because the PROD job (batch or addins) has to read through and resolve all the security rules, to be sure the user who submitted the task has proper authority, before processing the action?

                                        Greg Moeller
                                        Veteran Member
                                        Posts: 1498
                                        Veteran Member
                                        11/9/2011 6:07 PM
                                          No, different environments, different servers, but the exact same security (if the lsdump/lsload utilities worked correctly). No guarantees there.

                                          Maybe this should have been a separate topic, but I thought it fit here, too.
                                          SP
                                          Veteran Member
                                          Posts: 122
                                          Veteran Member
                                          11/9/2011 9:17 PM
                                            Greg,

                                            If you are running LSF901, you should look at the following: JT-241140,
                                            JT-241140, JT-264451

                                            Examine the joblogs and check to see if the jobs are spending an inordinate
                                            amount of time actually moving from active to completed status.


                                            On Wed, Nov 9, 2011 at 1:15 PM, wrote:

                                            > [image: LawsonGuru.com Logo] <https://www.lawsonguru.com/> S3 Systems
                                            > Administration Forum Notification A message was posted to a thread you
                                            > are tracking. *RE: Upgrade 9.0.0 to 9.0.1 - Slow MS Addins Performance
                                            > on PROD only* Posted by: *Greg Moeller*
                                            > 11/09/2011 01:07 PM No, different environments, different servers, but
                                            > the exact same security (if the lsdump/lsload utilities worked correctly).
                                            > No guarantees there.
                                            >
                                            > Maybe this should have been a separate topic, but I thought it fit here,
                                            > too.
                                            > ------------------------------
                                            >
                                            > To view the complete thread and reply via your browser, please visit:
                                            >
                                            > https://www.lawsonguru.co...rmance-on-prod-only/
                                            >
                                            > You were sent this email because you opted to receive email notifications
                                            > when someone posted and/or responded to a message on this forum.
                                            > To unsubscribe to this thread please visit your user profile page and
                                            > change your subscription options.
                                            >
                                            > Thank you,
                                            > LawsonGuru.com
                                            >

                                            Scott Whitfield
                                            Basic Member
                                            Posts: 7
                                            Basic Member
                                            12/9/2011 7:43 PM
                                              The users who are running the addins query are using LAUA ... we even created a 2nd websphere instance to rule out a 'bad' install ... we have the same results. PROD is 10x slower than TEST 1027 records in our control file takes 2m in TEST and 20m in PROD
                                              Greg Moeller
                                              Veteran Member
                                              Posts: 1498
                                              Veteran Member
                                              12/12/2011 3:31 PM
                                                Scott: Did you ever achieve resolution to your slow add-ins experience? If so, would you mind sharing what it was?
                                                Scott Whitfield
                                                Basic Member
                                                Posts: 7
                                                Basic Member
                                                6/28/2012 10:43 AM
                                                  My apologies for the long delay...

                                                  Ultimately, between a certified lawson vendor and lawson support no solution could be found.  We rebuilt our app server and migrated the data over and addins performance was up to par.

                                                  Our theory is that the original upgrade used the 'cygwin' utility which may have caused some unknown issue during the migration.  We did not use this tool with the 2nd app server... this is the only difference between our upgrade process.


                                                  Regards