Thursday, August 14, 2008

How to Clear the Application Server Cache Without Shutting it Down

Updated 4.9.2008: It is often necessary to clear the physical cache files on the PeopleSoft application server. You would think that after all the length of time that PeopleTools has been around, that development would have sorted out the problems in object version numbering. Sometimes, recently migrated changes do not take effect until the cache is cleared. Global Support will nearly always ask you to clear the cache if you are experiencing any kind of problem in the PIA.

Normally, you would shut down the application server, delete the physical cache directories and restart the application server.

From PeopleTools 8.48, each Application server process can only use one dedicated set of physical cache files that is in a directory whose name includes the name and ID number of the server, (previously the server process could use any unlock set cache directories). This allowed PeopleSoft to add an option to the psadmin utility to trigger each PSAPPSRV process to clear out its own physical cache (my thanks to the anonymous question asked after the original version of this posting that reminded me). This option also causes each server process to recycle. It is the recommended and supported way to clear the physical cache files.

This can also be invoked from the command line

psadmin -c purge -d <domain> [-noarch | -arch <archive_directory>] [-log <"log_comments">]
where 'domain' specifies domain name in PS_HOME and
'archive_directory' specifies location to which to quarantine the purged cache,
'log_comments' specifies any comments to be added to the purge cache log entry

However, even in previous versions, there has always been a way to invalidate all physical cache files. Any cached object older than the value of LASTREFRESHDTTM on the table PSSTATUS (it was on a different table prior to PeopleTools 8) is purged from the cache when the process that references that cache starts. Therefore, if that value is updated to the current time, the entire cache will be purged.

UPDATE PSSTATUS
SET LASTREFRESHDTTM = SYSDATE
/
COMMIT
/
Sometimes, developers also have to clear the physical cache on their clients used by Application Designer. Updating PSSTATUS also clears two-tier client caches. In fact, this behaviour is left over from the days when PeopleTools was a two-tier application, and it was necessary to clear cache files on users' desktop computers.
  • If you have multiple Application Servers on a single database, then you can shut each one down in turn without any loss of server. The users will fail over to the surviving servers. However, this can result in the load being unevenly distributed and could overload one server.

  • In small environments, and this includes most development and test systems, there is only a single application server. Shutting down an application server requires downtime and can be disruptive.
If an application server has at least two instances of a server process then each process can be recycled one by one, and there is no loss of service because the other processes can handle incoming requests. You could do this manually by entering the stop and boot commands in tmadmin (I discussed this in Chapter 13 of PeopleSoft for the Oracle DBA, but didn't combine it with the idea of clearing the Application Server cache).

I have produced a script, tuxcycle.sh (available from the Go-Faster website) that uses the Tuxedo command line interface utility, tmadmin, to find out which PSAPPSRV processes are running, and then sequentially recycles each of them. I don't think I would completely automate these two operations into a single process, nor would I run them unattended, but you can now clear cache files without taking a system away from users or developers.

Remember, that users will experience a reduction in performance after clearing the cache because it needs to be rebuilt by querying the PeopleTools table.

The Process Scheduler, Master Process Scheduler, Distribution Server, and Application Engine processes also have physical caches. These are also cleared when LASTREFRESHDTTM is updated and the processes restarted. However, you should not shut down the Process Scheduler while processes are executing. The server process will shutdown and the Application Engine will be killed if using an Application Engine server process (PSAESRV). Any stand-alone processes, including psae, will be killed by the Process Scheduler when it is restarted.

9 comments :

Anonymous said...

Hello David

U mentioned "Any cached object older than the value of LASTREFRESHDTTM on the table PSSTATUS (it was on a different table prior to PeopleTools 8) is purged from the cache when the process that references that cache starts"

Q)What kind of process reference cache.Is it Online activity which reference the cache?
Q2)Does Purge cache option in Appserver Options do the same.
Q3)What is the advantage of recycling App Server Processes.Your script just recycles it.Do we have any perf benefit from it?

David Kurtz said...

1. Application Server processes including the PSAPPSRV, and the various PUBSUB servers. The Process Scheduler and Application Engine process also use physical cache.

2. I have to admit that I had forgotten about the 'Purge Cache' option in psadmin when I wrote this posting. This option was introduced in PeopleTools 8.48, and is, of course, the officially recommended way to purge the cache. Code built into the application server initialisation routine is invoker to purge both the memory cache and the physical cache. However, I am not sure how this is trigger for some restarts only.

3. Recycling the Application Server in this way saves having to shut it down. There is no performance benefit. Clearing the cache will almost certainly degrade performance until it has been rebuilt from the PeopleTools tables. Clearly the cache at all is a necessary evil.

Dan Kibler said...

David

You don't mention Shared Cache. If you delete the shared cache files you would affect all the running processes in that instance of the application server. As far as I know, the only way to replace the shared cache files are to rebuild them through the online process.

Regards,
Dan

David Kurtz said...

Correct. This posting is purely about dedicated cache. Shared cache files are not updated by the server processes, so the issue never arises. The in-memory cache is cleared when the process recycles.

Marcus said...

I know some people up north, who will love you for this!

Anonymous said...

Correct - clearing the cache is a neccesary evil. Does anyone have tips and advice to 'prebuild' the cache ? In our environment, we have 20 app servers and after weekly migrations we bounce the servers and clear cache. The first users in take an awful hit until cache is generated. Any tips on this ?

Iouri Chadour said...

To answer the prebuild cache question. Please look up the LOADCACHE process in PeopleBooks. It has the step by step instructions. This process pre-builds cache into stage directory in process scheduler server directories. You can run the process and then copy all of the files into individual App Server Process Directories, however this would require you to bring down the app server first.

Iouri Chadour said...

We have huge problems with Shared Cache. For some reason after certain amount of time, all the queries that have tree node refernces in them start failing when users try to modify ,resave or click View SQL tab in Query Manager. In App Server log we get ICPanel Out of Memory error. After rerun LOADCACHE and replace the cache the problem goes away for sometime. We are running on AIX/DB2. Thanks in advance.

Parthasarathy said...

Hi All,

My question is, I could see some messages in IB which is in the "new" status for a long time and get processed late. And some time we need to bounce the pub/sub server to process the messages. Please let me know if this because of the cache which slows the performance of the pub/sub server