RSS slowness

Hey all,

Sorry if this is long, I just wanted to give as much information as I could...any help would be greatly appreciated.

Several months ago, we installed MSP11 on our Production system which required an upgrade to RSS. Over the past few weeks, some requesters have reported slowness during checkout in Shopping, some having to wait up to 10 minutes to checkout. The reports have been scattered by location as well as time of day, but it does not seem to be affecting all users, at least not to this degree. Also, a buyer using Receiving has reported the system taking about 2 minutes to process an approval.

We have a few customizations, hence little support from Lawson on this. There are 2 basic types of customizations we made- 1) aesthetic changes updating the XML, XSL, or HTML, and 2) additional validation using JavaScript on checkout, mostly to make sure the accounting unit/account/sub account combinations are valid. These new JavaScript functions call the Lawson DME object using the same structure as existing standard RSS. We've also verified the indices are being populated correctly and the number of records each query returns is very low. We have made no changes to Receiving which is also running slow.

We had a tough time in our testing systems stabilizing RSS. After many tests looking at memory and system settings (listed below), we changed the 'max_thread_proc' kernel setting from 64 to 256 and the system has not crashed since. Anytime the thread count for the tomcat process hit 64 or above (using glance), RSS would crash shortly after. Our production system is set to 256 and the highest thread count we've seen the past few days is about 180, and it has not crashed at all, it’s just running slowly.

We've activated Trace just prior to checkout (ctrl+shift+t) and see that the checkout seems to take minutes to get through our additional validation, but I don't see any errors in Tomcat logs or system logs.

We've not really been able to duplicate the issue on a test system since the issues have been reported, and the longest wait time we've had is 10-15 seconds.

My questions -- has anyone else experienced any issues with RSS running very slowly? Does what I’ve described sound like the slowness is being caused by our customizations in checking out (since trace shows they’re taking a long time to run) or does it sound more like that may just be a result of RSS itself running slow because of perhaps the thread count coming somewhat close to the limit (180 / 256) - could bumping this number up again solve the problem? Or from a system setting? Or from specific PC's (Lawson has said numerous times this version of RSS is PC-intensive)? Recommendations on proceeding?

Here are the additional steps we tried in our original testing with no success prior to changing max_thread_proc:
- Edit processflow.properties
- Change minDbconnection
- Change maxDbconnection
- Change Thread sizes
- touch $CGIDIR/dme.log
- change thread limits in /opt/tomcat/jakarta-tomcat-5.0.28/prod/conf/server.xml
- Removed customizations and disabled select processflows in WF05
- tomcat memory settings
- changed to 512min 512max
- Edit PoolMgrProperties
- Add initconns_dme = 5
- add maxconns_dme = 20
- change logging type (was debug)

-------------------
UNIX, HPUX, RSS 8.1.0.53, 8.0.3 MSP11, 8.0.3 ESP6, Portal 3.1.7.286