Friday, September 10, 2010

PeopleSoft Run Control Purge Utility

Run Control records are used to pass parameters into processes scheduled processes. These tables tend to grow, and are rarely purged. Generally, once created a run control is not deleted.  When operator accounts are deleted, the Run Controls remain, but are no longer accessible to anyone else.

I have worked on systems where new Run Controls, whose IDs contain either a date or sequence number, are generated for each process. The result is that the Run Control tables, especially child tables, grow quickly and if not regularly managed will become very large. On one system, I found 18 million rows on one table!

RECNAME         FIELDNAME            NUM_ROWS     BLOCKS
--------------- ------------------ ---------- ----------
TL_RUN_CTRL_GRP RUN_CNTL_ID          18424536     126377
AEREQUESTPARM   RUN_CNTL_ID           1742676      19280
AEREQUESTTBL    RUN_CNTL_ID            333579       3271
XPQRYRUNPARM    RUN_CNTL_ID            121337       1630
TL_TA_RUNCTL    RUN_CNTL_ID            112920        622
…

I have written a simple Application Engine process, GFC_RC_ARCH, that purges old Run Controls from these tables.  It can be downloaded from my website.

Run Control records are easily identified. They are characterised by:
  • the first column of these tables is always OPRID, and the second is either RUNCNTLID or RUN_CNTL_ID,
  • these two columns are also the first two columns of the unique key,
  • the Run Control tables appear on pages of components that are declared as the process security component for that process.
I have decided that if the combination of OPRID and RUN_CNTL_ID does not appear in the process scheduler request table, PSPRCSRQST, then the Run Control record should be deleted. Thus, as the delivered Process Scheduler Purge process, PRCSPURGE, deletes rows from the Process Scheduler tables, so my purge process will delete rows from the Run Control tables.

I have chosen to make these two Application Engine processes mutually exclusive, so the Process Scheduler will not run both at the same time, but that configuration cannot be delivered in an Application Designer project.

Monday, July 12, 2010

Announcing the Co-Operative PeopleTools Table Reference

Update 20.6.2019: I have published on Github the code used to generate the PeopleTools Table reference described here.  The element of co-operation is should be easier via updates to the metadata scripts in Github.  See blog post PeopleTools Table Reference Generator.

I have created a reference to the PeopleTools tables and views on my website. 

In the course of my work on PeopleSoft, I spend a lot of time looking at the PeopleTools tables. They contain meta-data about the PeopleSoft application. Much of the application is stored in various tables that are maintained by Application Designer. Some tables provide information about the Data Model. Others contain configuration data that is maintained via PeopleTools components in the PIA.

Many of my utility scripts query information from PeopleTools tables and some also update them. Of course, that is strictly not supported, but if you understand how the tables fit together it can be done relatively safely.  So, it is very helpful to be able to understand what is in these tables.

In PeopleSoft for the Oracle DBA, I discussed some of the PeopleTools tables that are of regular interest. I included the tables that correspond to the database catalogue, and I discussed what happens during the PeopleSoft login procedure, submission of process requests to the Process Scheduler and PS/Query. The tables that are maintained by the process scheduler are valuable because they contain information about who ran what process when, and how long they ran.

I am not the only person to have started to document the PeopleTools tables on their website or blog, most people have picked a few tables that are of particular interest. However, I want to tackle the problem in a slightly different way. There are over 3000 PeopleTools tables and views (as defined by the PeopleTools object security group in PSOBJGROUP). Tackling all of them manually would be a monumental task.

Nevertheless, I do want a complete reference. So, I have written code to dynamically generate a page for each PeopleTools table and view, and I have put as much information about these records as I can find in the PeopleTools tables themselves. Reference to related objects, including objects referenced in the text of views, appear as links to those pages.

I have started to manually add my own annotation to the generated pages.  So far I have only added descriptions to a few tables (marked with an asterisk). However, I would like to make this a collaborative project, and I have already had updates to some pages.
  • There is a page for each PeopleTools table and view. If you save that page, add descriptions, and return it to me, I will upload it to the site.
  • You can also add links to related websites and blog pages to your entries.
  • Please put your name and, if you wish, a link to your website to the bottom of the pages you author.
  • Let me know if you think you have found a mistake. 
I hope you find it useful.

    Thursday, June 17, 2010

    Configuring Large PeopleSoft Application Servers

    Occasionally, I see very large PeopleSoft systems running on large proprietary Unix servers with many CPUs.  In an extreme case, I needed to configure application server domains with up to 14 PSAPPSRV processes per domain (each domain was on a virtual server with 8 CPU cores, co-resident with the Process Scheduler).

    The first and most important point to make is don't have too many server processes.  If you run out of CPU or if you fully utilise all the physical memory and start to page memory from disk, then you have too many server processes.  It is better to queue on a Tuxedo queue rather than the CPU run queue, or disk queue during paging.

    Multiple APPQ/PSAPPSRV Queues

    A piece of advice that I originally got from BEA (prior to their acquisition by Oracle) was that you should not have more than 10 server processes on a single queue in Tuxedo.  Otherwise, you are likely to suffer from contention on the IPC queue structure because processes must acquire exclusive access to the queue in order to enqueue a service request to the queue or dequeue a request from it.  Instead multiple queues should be configured that are both serviced by the same server processes and so advertise the same services. 

    If you look at the 'large' template delivered by PeopleSoft, you will see that it produces a domain that runs between 9 and 15 PSAPPSRV processes.  This does not conform to the advice I received from BEA.  I repeated this advice in PeopleSoft for the Oracle DBA.  Though I cannot now find the source for it, I stand by it.  I have recently been able to conduct some analysis to confirm it on a real production system.  Domains with two queues of 8 PSAPPPSRV server process each out performed domains with only a single queue.

    Load Balancing Across Queues

    If the same service is advertised on multiple queues, then Tuxedo recommended that you should specify realistic service loads and use Tuxedo load balancing to determine where to enqueue requests.  I want to emphasise that I am talking about load balancing across queues within a Tuxedo domain, and not about load balancing across Tuxedo domains in the web server.

    This is what the Tuxedo documentation says about load balancing:

    "Load balancing is a technique used by the BEA Tuxedo system for distributing service requests evenly among servers that offer the same service. Load balancing avoids overburdening some servers while leaving others idle or infrequently used. Before sending a request to a service routine, the BEA Tuxedo system identifies all servers capable of handling the request and selects the one most appropriate for maintaining a balanced load across all the servers in the configuration.


    You can control whether a load-balancing algorithm is used on the system as a whole. Such as algorithm should be used only when necessary, that is, only when a service is offered by servers that use more than one queue. Services offered by only one server, or by multiple servers in a Multiple Server, Single Queue (MSSQ) do not need load balancing. The LDBAL parameter for these services should be set to N. In other cases, you may want to set LDBAL to Y."

    It doesn't state that load balancing is mandatory for multi-queue domains, and only hints that it might improve performance.  If load balancing is not used, the listener process puts the messages on the first empty queue (one where no requests are queued).  If all queues have requests the listener round-robins between the queues.

    You could consider giving ICScript, GetCertificate and other services with small service times a higher Tuxedo Service priority.  This means they jump the queue 9 times out of 10.  ICScript is generally used during navigation, GetCertificate is used at log on.  Giving these services higher priority will mean they perform well even when the system is busy.  Users often need to do several mouse clicks to navigate around the system, but these services are usually quick.  This will improve the user experience without changing the overall performance of the system.

    Data

    I have recently been able to test the performance of a domains with up to 14 PSAPPSRVs on a single IPC queue, versus domains with two queues with up to 7 PSAPPSRVs each, both with and without Tuxedo queue balancing.   These results come from a real production system where the multiple queue configuration was implemented on 2 of the 4 application servers.  The system has a short-lived weekly peak period of on-line processing.  During that time Tuxedo spawns additional PSAPPSRV processes, and so I get different sets of times for different numbers of process. 

    The timings are produced from transactions sampled by PeopleSoft Performance Monitor.  I capture the number of spawned processes using the Tuxmon scripts on my website that use tmadmin to collect Tuxedo metrics.

    1 Queue2 Queue

    Server Processes per Queue

    Number of Services

    Mean ICPanel Service Time

    Server Processes per Queue

    Number of Services

    Mean ICPanel Service Time

    6

    2,616

    1.33

    3

    6945

    1.05

    7

    1,949

    0.97




    8

    1,774

    1.06

    4

    7595

    1.16

    9

    1,713

    1.02




    10

    1,553

    1.25

    5

    4629

    1.17

    11

    1,250

    1.30




    12

    969

    1.32

    6

    3397

    1.16

    13

    427

    1.21




    14

    1,057

    1.10

    7

    3445

    1.13

     Total

    13,308

    1.17


    26011

    1.13

    The first thing to acknowledge is that this data is quite noisy because it comes from a real production system, and the effects we are looking for are quite small.

    I am satisfied that the domains with two PSAPPSRV queues generally perform better under high load, than those under 1.  Not only does the queue time increase on the single queue domain, the service time also increases.

    However, I cannot demonstrate that Tuxedo Load Balancing makes a significant difference in either direction.

    My results suggest that domains with multiple queues for requests handled by PSAPPSRV process perform slightly better without load balancing if there is no queue of requests, but perform slightly better if there is a queue of pending requests.  However, the difference is small.  It is not large enough to be statistically significant in my test data.

    Conclusion

    If you have a busy system with lots of on-line users, and sufficient hardware to resource it, then you might reach a point when you need more than 10 PSAPPSRVs.  In which case, I recommend that you configure multiple Tuxedo queues.

    On the whole, I would recommend that Tuxedo Load Balancing should be configured.  I would not expect it to improve performance, but it will not degrade it either.

    Friday, June 11, 2010

    Life Cycle of a Process Request

    Oracle's Flashback Query facility lets you query a past version of a row by using the information in the undo segment.  The VERSIONS option lets you seen all the versions that are available. Thus, it is possible to write a simple query to retrieve the all values that changed on a process request record through its life cycle.

    The Oracle parameter undo_retention determines how long that data remains in the undo segment. In my example, it is set to 900 seconds, so I can only query versions in the last 15 minutes. If I attempt to go back further than this I will get an error.

    column prcsinstance heading 'P.I.' format 9999999
    column rownum heading '#' format 9 
    column versions_starttime format a22
    column versions_endtime format a22
    
    SELECT rownum, prcsinstance
    , begindttm, enddttm
    , runstatus, diststatus
    , versions_operation, versions_xid
    , versions_starttime, versions_endtime
    FROM sysadm.psprcsrqst
    VERSIONS BETWEEN timestamp
    systimestamp - INTERVAL '15' MINUTE AND
    systimestamp 
    WHERE prcsinstance = 2920185
    /
    
    #     P.I. BEGINDTTM           ENDDTTM             RU DI V VERSIONS_XID     VERSIONS_STARTTIME    VERSIONS_ENDTIME
    -- -------- ------------------- ------------------- -- -- - ---------------- --------------------- ----------------------
     1  2920185 10:51:13 11/06/2010 10:52:11 11/06/2010 9  5  U 000F00070017BD63 11-JUN-10 10.52.10 AM
     2  2920185 10:51:13 11/06/2010 10:52:11 11/06/2010 9  5  U 001A002C001CB1FF 11-JUN-10 10.52.10 AM 11-JUN-10 10.52.10 AM
     3  2920185 10:51:13 11/06/2010 10:52:11 11/06/2010 9  7  U 002C001F000F87C0 11-JUN-10 10.52.10 AM 11-JUN-10 10.52.10 AM
     4  2920185 10:51:13 11/06/2010 10:52:11 11/06/2010 9  1  U 000E000A001771CE 11-JUN-10 10.52.10 AM 11-JUN-10 10.52.10 AM
     5  2920185 10:51:13 11/06/2010 10:52:02 11/06/2010 9  1  U 002A000F00125D89 11-JUN-10 10.52.01 AM 11-JUN-10 10.52.10 AM
     6  2920185 10:51:13 11/06/2010                     7  1  U 0021000B00132582 11-JUN-10 10.51.10 AM 11-JUN-10 10.52.01 AM
     7  2920185                                         6  1  U 0004002000142955 11-JUN-10 10.51.10 AM 11-JUN-10 10.51.10 AM
     8  2920185                                         5  1  I 0022002E0013F260 11-JUN-10 10.51.04 AM 11-JUN-10 10.51.10 AM

    Now, I can see each of the committed versions of the record. Note that each version is the result of a different transaction ID.
    Reading up from the last and earliest row in the report, you can see the history of this process request record.

    • At line 8 it was inserted (the value of psuedocolumn VERSION_OPERATION is 'I') at RUNSTATUS 5 (queued) by the component the operator used to submit the record.
    • At line 7, RUNSTATUS was updated to status 6 (Initiated) by the process scheduler.
    • At line 6 the process begins and updates the BEGINDTTM with the current database time, and sets RUNSTATUS to 7 (processing).
    • At line 5 the process completes, updates ENDDTTM to the current database time, and sets RUNSTATUS to 9 (success).
    • At line 4 the ENDDTTM is updated again. This update is performed by the Distribution Server process in the Process Scheduler domain as report output is posted to the report repository.  Note that the value is 1 second later than the VERSIONS_ENDTIME, therefore this time stamp is based on the operating system time for the host running the process scheduler. This server's clock is slightly out of sync with that of the database server.
    • At lines 3 to 1 there are 3 further updates as the distribution status is updated twice more.

    For me, the most interesting point is that ENDDTTM is updated twice; first with the database time at which the process ended, and then again with the time at which any report output was successfully completed.

    I frequently want measure the performance of a processes. I often write script that calculate the duration of the process as being the difference between ENDDTTM and BEGINDTTM, but now it is clear that this includes the time taken to post the report and log files to the report repository.

    For Application Engine processes, you can still recover the time when the process ended. If batch timings are enabled and written to the database, the BEGINDTTM and ENDDTTM are logged in PS_BAT_TIMINGS_LOG.

    select * from ps_bat_timings_log where process_instance = 2920185
    
    PROCESS_INSTANCE PROCESS_NAME OPRID                          RUN_CNTL_ID
    ---------------- ------------ ------------------------------ ------------------------------
    BEGINDTTM           ENDDTTM             TIME_ELAPSED TIME_IN_PC TIME_IN_SQL TRACE_LEVEL
    ------------------- ------------------- ------------ ---------- ----------- -----------
    TRACE_LEVEL_SAM
    ---------------
             2920185 XXX_XXX_XXXX 52630500                       16023
    10:51:12 11/06/2010 10:52:02 11/06/2010        49850      35610       13730        1159
                128

    You can see above that ENDDTTM is the time when the process ended.

    That gives me some opportunities. For Application Engine programs, I can measure the amount of time taken to posting report content, separately from the process execution time.  This query shows me that this particular process took 49 seconds, but the report output took a further 9 seconds to post.

    SELECT r.begindttm begindttm
    , NVL(l.enddttm, r.enddttm) enddttm
    , (NVL(l.enddttm, r.enddttm)-r.begindttm)*86400 exec_secs
    , r.enddttm posttime
    , (r.enddttm-NVL(l.enddttm, r.enddttm))*86400 post_secs
    FROM sysadm.psprcsrqst r
        LEFT OUTER JOIN sysadm.ps_bat_timings_log l
        ON l.process_instance = r.prcsinstance
    WHERE r.prcsinstance = 2920185
    
    BEGINDTTM           ENDDTTM              EXEC_SECS POSTTIME             POST_SECS
    ------------------- ------------------- ---------- ------------------- ----------
    10:51:13 11/06/2010 10:52:02 11/06/2010         49 10:52:11 11/06/2010          9  

    For more detail on the Flashback Query syntax see the Oracle SQL Reference.

    Wednesday, March 31, 2010

    Oracle Plan Stability (Stored Outlines) in PeopleSoft Global Payroll

    A longer version of this posting, with experimental results, is available on my website.

    In PeopleSoft for the Oracle DBA, I wrote a page (p. 291) explaining why stored outlines were not suitable for use in PeopleSoft.  Five years later, my view has not significantly changed.  Essentially, stored outlines work best with shared SQL, and there isn't much shared SQL in PeopleSoft, because a lot of it is dynamically generated.
    • Code generated by the component processor is dynamically generated.  At save time, only fields that have changed are updated.
    • PeopleCode can written in such a way that where clauses are dynamically assembled
    • nVision reports have variable numbers of criteria on literal tree node IDs in the queries.
    • By default in Application Engine, bind variables are converted to literals before the SQL is submitted to the database.  Even if this is overridden by enabling ReUseStatement in Application Engine or by using Cursor Sharing in the database, the code still wouldn’t be sharable.  Different instances of Application Engine executing the same program can use different instances on non-shared temporary records, so the tables in otherwise identical SQL statements are not the same.  You would get one version of the statement per temporary table instance.
    However, there are very limited exceptions to this rule (otherwise I wouldn't have a story to tell).  The SQL in COBOL and SQR programs are more likely to be shareable.   Although some programs are coded to generate SQL dynamically, bind variables are passed through to SQL statements, and they use regular tables for working storage and not PeopleSoft temporary records.
    A Global Payroll customer came to me with a problem where the payroll calculation (GPPDPRUN) would usually run well, but sometimes, the execution plan of a statement would change and the calculation would take additional several hours.  It is significant that the Global Payroll engine is written in COBOL.  My usual response to this sort of problem in Global Payroll is to add a hint to the stored statement.  Usually, I find that only a few statements that are affected.  However, after this happened a couple of times in production, it was clear that we couldn't continue to react to these problems. We needed to proactively stop this happening again.  This is exactly what stored outlines are designed to do.

    Using Stored Outlines in the PeopleSoft GP Engine

    Earlier I said that we could apply stored outlines to the Global Payroll calculation engine (GPPDPRUN) because it generally doesn’t use dynamic code with embedded literal values.  
    While outlines are being created, the following privilege needs to be granted.  It can be revoked later.

    GRANT CREATE ANY OUTLINE TO SYSADM;

    We can create a trigger to collect the stored outlines for a payroll calculation, thus:
    • The trigger fires when a payroll calculation process starts or finishes. 
    • At the start a payroll process it starts collecting stored outlines in a category called the same as the process; GPPDPRUN.
    • When the process finishes, outline collection is disabled by setting it back to false.
    CREATE OR REPLACE TRIGGER sysadm.gfc_create_stored_outlines
    BEFORE UPDATE OF runstatus ON sysadm.psprcsrqst
    FOR EACH ROW
    WHEN (new.prcsname = 'GPPDPRUN' AND (new.runstatus = 7 OR old.runstatus = 7))
    DECLARE
     l_sql VARCHAR2(100);
    BEGIN
      l_sql := 'ALTER SESSION SET create_stored_outlines = ';
      IF :new.runstatus = 7 THEN
        EXECUTE IMMEDIATE l_sql||:new.prcsname;
      ELSIF :old.runstatus = 7 THEN
        EXECUTE IMMEDIATE l_sql||'FALSE';
      END IF;
    EXCEPTION WHEN OTHERS THEN NULL; --because I dont want to crash the process scheduler  
    END;
    /

    The exact number of outlines that are collected during this process will vary depending upon configuration, and which employees are processed as different payroll rules are invoked.
    If no more outlines are to be collected the CREATE ANY OUTLINE privilege can be revoked.  This does not prevent the outlines from being used.
    REVOKE CREATE ANY OUTLINE FROM SYSADM;
    Then, the category of outlines can be used in subsequent executions by replacing the trigger above with the one below, and the execution plans cannot change so long as the SQL doesn’t change.
    CREATE OR REPLACE TRIGGER sysadm.gfc_use_stored_outlines
    BEFORE UPDATE OF runstatus ON sysadm.psprcsrqst
    FOR EACH ROW
    WHEN (new.prcsname = 'GPPDPRUN' AND (new.runstatus = 7 OR old.runstatus = 7))
    DECLARE
     l_sql VARCHAR2(100);
    BEGIN
      l_sql := 'ALTER SESSION SET use_stored_outlines = ';
      IF :new.runstatus = 7 THEN
    EXECUTE IMMEDIATE l_sql||:new.prcsname;
      ELSIF :old.runstatus = 7 THEN
        EXECUTE IMMEDIATE l_sql||'FALSE';
      END IF;
    EXCEPTION WHEN OTHERS THEN NULL; --because I dont want to crash the process scheduler 
    END;
    /

    After running an identify-and-calc and a cancel, we can see how many of the outlines are actually used.
    SELECT category, count(*) outlines
    , sum(decode(used,'USED',1,0)) used
    FROM user_outlines
    GROUP BY category
    ORDER BY 1
    /

    I have a large number of unused outlines because of additional recursive SQL generated because OPTIMIZER_DYNAMIC_SAMPLING was set 4. This does not occur if this parameter is set to the default of 2.
    CATEGORY                         OUTLINES       USED
    ------------------------------ ---------- ----------
    GPPDPRUN                              572        281

    I can then remove the unused outlines.
    EXECUTE dbms_outln.drop_unused;

    Used flags on the outlines can be reset, so we later we can see the outlines being used again.
    BEGIN
     FOR i IN (SELECT * FROM user_outlines WHERE category = 'GPPDPRUN') LOOP
      dbms_outln.clear_used(i.name);
     END LOOP;
    END;
    /

    If I want to go back running without outlines, I just disable the trigger
    ALTER TRIGGER sysadm.stored_outlines DISABLE;

    To re-enable outlines, just re-enable the trigger.
    ALTER TRIGGER sysadm.stored_outlines ENABLE;

    Conclusions

    Stored Outlines have very limited application in a PeopleSoft system.  However, they can easily be collected and used with the PeopleSoft Global Payroll engine.  It is just a matter of granting a privilege and using the database triggers on the process request table.
    Testing that they actually have the desired effect is quite difficult, because you are trying to prove a negative.  I don’t think it is adequate simply to say that the outline has been used.
    • First you would need an environment where payroll calculation performs well, where you could collect outlines.
    • Then you would need a payroll calculation that performs poorly because the execution plan for at least one SQL statement is different
      • Either, on a second environment with exactly the same code.
      • Or in the same environment on a different set of data.
    • Then, it would be possible to demonstrate that applying the outline causes the execution plan to revert and restores the performance.  This can be confirmed by comparison of the ASH data for the various scenarios.
    Even if you don’t want to use a stored outline immediately, it might be advantageous to collect them, and have them available when you do encounter a problem.