Showing posts with label Temporary Records. Show all posts
Showing posts with label Temporary Records. Show all posts

Tuesday, January 28, 2025

SQL Profiles for Application Engine Steps that Reference Temporary Records via the %Table() Meta-SQL

Oracle generally advises using SQL Profiles/Patches/Baselines rather than introducing hints into application code.  Using one of these forms of plan stability saves you from having to alter the code, and then having to test and verify that the change is functionally neutral and release it to production. It also saves repeating that whole process if you ever choose to remove or change the hint.  

I generally use SQL profiles with PeopleSoft, because they are the only plan stability mechanism that can force match a statement.  That is to say, a force-matching SQL profile will match other SQL statements that differ only in embedded literal values.  PeopleSoft code makes extensive use of literal values. SQL patches and baselines only exactly match statements (so they match the SQL ID and not the force matching signature). Note that SQL Profiles require that the Tuning Pack is licenced, and that is only available on Enterprise Edition or Oracle.

I have written previously about using SQL Profiles to inject hints into dynamically generated code.  If any part of the SQL (other than literal values) changes, then a different SQL profile is needed for each variation.  

When generating SQL profiles for dynamically generated code, you have to be able to predict every possible variation in the code and generate a SQL profile for every combination of every variation.  You end up writing code that mimics the dynamic code generation in the application

The same is true of Application Engine steps that reference temporary records via the %Table() meta-SQL but are otherwise static.  This variation is a special case because it is a part of PeopleTools' behaviour.  It is easy to determine which tables could be referenced by querying some of the PeopleTools tables.  Although, the table name itself can be set dynamically!

Let's take an example statement.  I identified it as performing poorly during an analysis of ASH data.  I have a SQL_ID, a plan hash value, and ACTION indicates the name of the Application Engine step that generated it (because Application Engine instrumentation is enabled by setting EnableAEMonitoring - see https://blog.go-faster.co.uk/2015/03/undocumented-application-engine.html).

  • SQL ID = bk98x60cspttj
  • SQL Plan Hash Value = 113493817
  • Action = XX_TREESEL.iDetVal.iDetVal.S

This is the statement and execution plan report generated by dbms_xplan.display_workload_repository (formatted and edited slightly).  I have the statement, the execution plan and the outline of hints that describes the plan.

SELECT * FROM table(dbms_xplan.display_workload_repository('bk98x60cspttj',113493817,'ADVANCED +ADAPTIVE'));

SQL_ID
-------------
INSERT INTO PS_TSEL_P_TAO12 (CHARTFIELD, CHARTFIELD_VALUE,
PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
FROM PS_TSEL_R30_TAO12 A, PS_GL_ACCOUNT_TBL DV 
WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 
AND DV.SETID = '12345' 
AND DV.EFFDT = (SELECT MAX(EFFDT) 
 	FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT 
 	AND EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) 
AND DV.ACCOUNT >= A.RANGE_FROM_30
AND DV.ACCOUNT <= A.RANGE_TO_30

Plan hash value: 113493817

--------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------------------
|   0 | INSERT STATEMENT                   |                   |       |       | 38754 (100)|          |
|   1 |  LOAD TABLE CONVENTIONAL           | PS_TSEL_P_TAO17   |       |       |            |          |
|   2 |   SORT UNIQUE                      |                   |     1 |   161 | 38754   (1)| 00:00:02 |
|*  3 |    FILTER                          |                   |       |       |            |          |
|   4 |     MERGE JOIN                     |                   | 24219 |  3807K|     8  (50)| 00:00:01 |
|   5 |      SORT JOIN                     |                   |  1138 |   151K|     3  (34)| 00:00:01 |
|*  6 |       INDEX STORAGE FAST FULL SCAN | PSATSEL_R30_TAO17 |  1138 |   151K|     2   (0)| 00:00:01 |
|*  7 |      FILTER                        |                   |       |       |            |          |
|*  8 |       SORT JOIN                    |                   |  8513 |   207K|     3  (34)| 00:00:01 |
|*  9 |        INDEX STORAGE FAST FULL SCAN| PS_GL_ACCOUNT_TBL |  8513 |   207K|     2   (0)| 00:00:01 |
|  10 |     SORT AGGREGATE                 |                   |     1 |    25 |            |          |
|  11 |      FIRST ROW                     |                   |     1 |    25 |     2   (0)| 00:00:01 |
|* 12 |       INDEX RANGE SCAN (MIN/MAX)   | PS_GL_ACCOUNT_TBL |     1 |    25 |     2   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------------------

Query Block Name / Object Alias (identified by operation id):
-------------------------------------------------------------
   1 - SEL$1
   6 - SEL$1 / A@SEL$1
   9 - SEL$1 / DV@SEL$1
  10 - SEL$2
  12 - SEL$2 / B@SEL$2

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('19.1.0')
      DB_VERSION('19.1.0')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$2")
      OUTLINE_LEAF(@"SEL$1")
      OUTLINE_LEAF(@"INS$1")
      FULL(@"INS$1" "PS_TSEL_P_TAO17"@"INS$1")
      INDEX_FFS(@"SEL$1" "A"@"SEL$1" ("PS_TSEL_R30_TAO17"."PROCESS_INSTANCE"
              "PS_TSEL_R30_TAO17"."CHARTFIELD" "PS_TSEL_R30_TAO17"."RANGE_FROM_30"
              "PS_TSEL_R30_TAO17"."RANGE_TO_30"))
      INDEX_FFS(@"SEL$1" "DV"@"SEL$1" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT"
              "PS_GL_ACCOUNT_TBL"."EFFDT"))
      LEADING(@"SEL$1" "A"@"SEL$1" "DV"@"SEL$1")
      USE_MERGE(@"SEL$1" "DV"@"SEL$1")
      PQ_FILTER(@"SEL$1" SERIAL)
      INDEX(@"SEL$2" "B"@"SEL$2" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT"
              "PS_GL_ACCOUNT_TBL"."EFFDT"))
      END_OUTLINE_DATA
  */
…

Below is the underlying Application Engine step that generated the SQL.  There is other dynamic code in this step that is driven by configuration data as well as the %Table() meta-SQLs.  Other reasons not to introduce hints into the statement include:

  • different dynamic variations of the code might require different hints, and then I would have to write more code to generate the hint dynamically.  However, in this case, I am only going to deal with a single variation, and I am only going to produce one set of SQL Profiles,
  • this step and the code generation are delivered by PeopleSoft, any change in these areas would be considered as a customisation.

%InsertSelect(DISTINCT
, %Bind(FT_TSEL_AET.RECNAME_SEL_TBL, NoQuotes)
, %Bind(FT_TSEL_STR_AET.DTL_RECNAME, NoQuotes
) DV
, %Bind(FT_TSEL_AET.FIELDNAME_CF, NoQuotes) = %Bind(FT_TSEL_STR_AET.DTL_FIELDNAME)
, %Bind(FT_TSEL_AET.FIELDNAME_VALUE, NoQuotes) = DV.%Bind(FT_TSEL_STR_AET.DTL_FIELDNAME, NoQuotes)
, PROCESS_INSTANCE = %Bind(PROCESS_INSTANCE)
, TREE_NAME = %Bind(FT_TSEL_AET.TREE_NAME)
, TREE_NODE = A.TREE_NODE
, TREE_NODE_NUM = A.TREE_NODE_NUM
, TREE_LEVEL_NUM = A.TREE_LEVEL_NUM, SETCNTRLVALUE =%Bind(FT_TSEL_AET.SETCNTRLVALUE)
, %Bind(FT_TSEL_GEN_AET.FIELDNAME_FROM, NoQuotes) = A.%Bind(FT_TSEL_GEN_AET.FIELDNAME_FROM, NoQuotes)
, %Bind(FT_TSEL_GEN_AET.FIELDNAME_TO, NoQuotes) = A.%Bind(FT_TSEL_GEN_AET.FIELDNAME_TO, NoQuotes)
, SETID_TREE = %Bind(FT_TSEL_GEN_AET.SETID_TREE)
, EFFDT = %Bind(FT_TSEL_STR_AET.EFFDT)
, CFV_SET = %Bind(FT_TSEL_AET.CFV_SET)) 
FROM %Table(%Bind(FT_TSEL_GEN_AET.RECNAME_SEL_TBL, NoQuotes)) A
, %Table(%Bind(FT_TSEL_STR_AET.DTL_RECNAME, NoQuotes)) DV 
WHERE A.CHARTFIELD = %Bind(FT_TSEL_AET.FIELDNAME) 
AND A.PROCESS_INSTANCE = %Bind(PROCESS_INSTANCE) 
%Bind(FT_TSEL_WRK_AET.WHERE_TXT_LONG, NoQuotes)
This step references two temporary records.  It generates data in one temporary table TSEL_P_TAO based on data in another, TSEL_R30_TAO.  It happens to use instance 12 of both records.  This is really just a coincidence.  Temporary table instances are allocated and deallocated in a group when an application engine starts and ends, so it is common for one step to use the same instance of different tables, but there is nothing to stop different table instances from being used.  That can occur when concurrently executing multiple instances of a program or different programs that allocate just some of the same tables.

I now need to produce SQL profiles for each permutation.  I will start by generating a script to create a single profile for a single SQL statement using Oracle's coe_xfr_sql_profile.sql script that is delivered as a part of the SQLT utility.  It is available from the Oracle support website (All About the SQLT Diagnostic Tool (Doc ID 215187.1)).  
It produces a SQL script such as the one below (I have removed the comments).  The SQL text is in a CLOB variable, and all the hints required to reproduce the same execution plan are in an array.  The script generates a signature for the statement and then creates a SQL profile from that information.
I have already customised my version of coe_xfr_sql_profile.sql so that FORCE_MATCH => TRUE in the generated scripts.
WHENEVER SQLERROR EXIT SQL.SQLCODE;
VAR signature NUMBER;
DECLARE
sql_txt CLOB;
h       SYS.SQLPROF_ATTR;
BEGIN
sql_txt := q'[
INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678
FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
]';
h := SYS.SQLPROF_ATTR(
q'[BEGIN_OUTLINE_DATA]',
q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
q'[OPTIMIZER_FEATURES_ENABLE('19.1.0')]',
q'[DB_VERSION('19.1.0')]',
q'[ALL_ROWS]',
q'[OUTLINE_LEAF(@"SEL$683B0107")]',
q'[OUTLINE_LEAF(@"SEL$C772B8D1")]',
q'[UNNEST(@"SEL$2")]',
q'[OUTLINE_LEAF(@"INS$1")]',
q'[OUTLINE(@"SEL$2")]',
q'[OUTLINE(@"SEL$7511BFD2")]',
q'[OUTLINE(@"SEL$1")]',
q'[FULL(@"INS$1" "PS_TSEL_P_TAO17"@"INS$1")]',
q'[NO_ACCESS(@"SEL$C772B8D1" "VW_SQ_1"@"SEL$7511BFD2")]',
q'[INDEX(@"SEL$C772B8D1" "DV"@"SEL$1" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT" "PS_GL_ACCOUNT_TBL"."EFFDT"))]',
q'[FULL(@"SEL$C772B8D1" "A"@"SEL$1")]',
q'[LEADING(@"SEL$C772B8D1" "VW_SQ_1"@"SEL$7511BFD2" "DV"@"SEL$1" "A"@"SEL$1")]',
q'[USE_NL(@"SEL$C772B8D1" "DV"@"SEL$1")]',
q'[USE_MERGE(@"SEL$C772B8D1" "A"@"SEL$1")]',
q'[PARTIAL_JOIN(@"SEL$C772B8D1" "A"@"SEL$1")]',
q'[INDEX_FFS(@"SEL$683B0107" "B"@"SEL$2" ("PS_GL_ACCOUNT_TBL"."SETID" "PS_GL_ACCOUNT_TBL"."ACCOUNT" "PS_GL_ACCOUNT_TBL"."EFFDT"))]',
q'[END_OUTLINE_DATA]');

:signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);

DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
sql_text    => sql_txt,
profile     => h,
name        => 'coe_g7wz6ctquwjcy_2476903986',
description => 'coe g7wz6ctquwjcy 2476903986 '||:signature||'',
category    => 'DEFAULT',
validate    => TRUE,
replace     => TRUE,
force_match => TRUE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
END;
/
WHENEVER SQLERROR CONTINUE
SET ECHO OFF;
PRINT signature
PRO
PRO ... manual custom SQL Profile has been created
PRO
SET TERM ON ECHO OFF LIN 80 TRIMS OFF NUMF "";
SPO OFF;
PRO
PRO COE_XFR_SQL_PROFILE_g7wz6ctquwjcy_2476903986 completed
In Application Engine, different tables will be used if multiple programs referencing the same table run concurrently, or if a process crashes and holds its allocation to a particular table instance.  Therefore, I need to create a set of similar SQL profiles, one for each statement that could be generated by this Application Designer step.  I need to iterate through all possibilities, so I have added some additional pieces to the script.  They are explained in the footnotes below.
REM coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal.sql
SPO coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal.log;
WHENEVER SQLERROR CONTINUE
SET serveroutput on ECHO ON TERM ON LIN 2000 TRIMS ON NUMF 99999999999999999999;
clear screen
ALTER SESSION SET CURRENT_SCHEMA=SYSADM;1
…
WHENEVER SQLERROR EXIT SQL.SQLCODE;
VAR signature NUMBER;

DECLARE
  l_recname1 VARCHAR2(15) :=    'TSEL_R30_TAO';2
  l_table1   VARCHAR2(18) := 'PS_TSEL_R30_TAO17';
  l_recname2 VARCHAR2(15) :=    'TSEL_P_TAO';
  l_table2   VARCHAR2(18) := 'PS_TSEL_P_TAO17';
  l_name     VARCHAR2(30);
  sql_txt CLOB;
  h       SYS.SQLPROF_ATTR;

  e_no_sql_profile EXCEPTION;
  PRAGMA EXCEPTION_INIT(e_no_sql_profile, -13833);
BEGIN
FOR i IN (3
  WITH v4  AS (SELECT rownum-1 row_number FROM dual CONNECT BY LEVEL <= 100) 
  SELECT DISTINCT 
         v1.row_number id1, r1.recname recname1, t1.table_name table_name1
  ,      v2.row_number id2, r2.recname recname2, t2.table_name table_name2
  ,      o.TEMPTBLINSTANCES
  FROM   psrecdefn r1
  ,      psrecdefn r2
  ,      pstemptblcntvw5 i1
  ,      pstemptblcntvw i2
  ,      all_tables t1
  ,      all_tables t2
  ,      psoptions o 
  ,      ps.psdbowner p 
  ,      v v1
  ,      v v2
  WHERE  r1.rectype = 7 AND r1.recname = i1.recname AND r1.recname = l_recname1
  AND    r2.rectype = 7 AND r2.recname = i2.recname AND r2.recname = l_recname2
  AND    v1.row_number <= i1.temptblinstances + o.temptblinstances6 --up to total number of instances
  AND    v2.row_number <= i2.temptblinstances + o.temptblinstances  --up to total number of instances
  AND    (v1.row_number = 0 OR v1.row_number > o.temptblinstances)7 --omit online temp tables
  AND    (v2.row_number = 0 OR v2.row_number > o.temptblinstances)  --omit online temp tables
  and    t1.owner = p.ownerid AND t1.table_name 
         = DECODE(r1.sqltablename,' ','PS_'||r1.recname,r1.sqltablename) 
         ||DECODE(v1.row_number,0,'',LTRIM(TO_NUMBER(v1.row_number)))8 --derive table table
  and    t2.owner = p.ownerid AND t2.table_name 
         = DECODE(r2.sqltablename,' ','PS_'||r2.recname,r2.sqltablename) 
         ||DECODE(v2.row_number,0,'',LTRIM(TO_NUMBER(v2.row_number)))  --derive table table
  AND    ABS(v1.row_number-v2.row_number)<=19  --max variation in instance numbers
  ORDER BY id1, id2
) LOOP
  -----------123456789012345678901234567890
  l_name := 'XX_TREESEL.iDetVal.iDetVal'||i.id1||i.id2;10
  
sql_txt := q'[
INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678
FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
]';

h := SYS.SQLPROF_ATTR(
q'[BEGIN_OUTLINE_DATA]',
q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
q'[ALL_ROWS]',
q'[LEADING(@"SEL$1" "A"@"SEL$1")]',11
q'[END_OUTLINE_DATA]');

sql_txt := REPLACE(sql_txt, l_table1,i.table_name1);12
sql_txt := REPLACE(sql_txt, l_table2,i.table_name2);
:signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
dbms_output.put_line(l_name||':'||:signature||':'||sql_txt);

for n in 1 .. h.count LOOP13
  if h(n) LIKE '%'||l_table1||'%' THEN
    h(n) := REPLACE(h(n), l_table1,i.table_name1);
    dbms_output.put_line(n||':'||h(n));
  end if;
  if h(n) LIKE '%'||l_table2||'%' THEN
    h(n) := REPLACE(h(n), l_table2,i.table_name2);
    dbms_output.put_line(n||':'||h(n));
  end if;
end loop;

BEGIN --drop profile if already exists
  DBMS_SQLTUNE.drop_SQL_PROFILE(name => l_name);
  EXCEPTION WHEN e_no_sql_profile THEN NULL;
END;

DBMS_SQLTUNE.IMPORT_SQL_PROFILE ( --create new profile
sql_text    => sql_txt,
profile     => h,
name        => l_name,
description => 'coe XX_TREESEL.iDetVal.iDetVal '||l_name||' '||:signature||'',
category    => 'DEFAULT',
validate    => TRUE,
replace     => TRUE,
force_match => TRUE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
  END LOOP;
END;
/
WHENEVER SQLERROR CONTINUE
…
SPO OFF;
PRO
PRO coe_xfr_sql_profile_XX_TREESEL.iDetVal.iDetVal completed

  1. Set CURRENT_SCHEMA to specify PeopleSoft owning schema, SYSADM.
  2. For each temporary record in the original SQL statement, add pairs of variables to specify the name of PeopleSoft record and Oracle temporary table instance referenced in the statement.  I could derive the record name from the table name, but it is easier just to hard-code it.
  3. The SQL statement produces all combinations of temporary records that could appear in the SQL statement.  I will put it in an implicit cursor loop, and then for each row returned, the script will create a SQL profile.
  4. Common table expression V returns 100 rows, numbered 0 to 99.  Irrespective of the number of temporary table instances specified in each Application Engine program, there can only be non-shared 99 table instances for each PeopleTools record, plus the shared instance (that doesn't have a suffix number).  
  5. PSTEMPTBLCNTVW returns the number of non-shared batch (i.e. not online) instances of each temporary record that needs to be built.  This is in addition to the number of online temporary table instances.
  6. The query will return a row for each instance of each temporary table up to the number of instances required by the application engines plus the number of online table instances, but not exceeding the 99 rows returned by CTE V.
  7. Most Application Engines do not run online in the component processor, therefore there is no need to build SQL profiles on these instances.  There are exceptions, such as some as the journal and voucher edit and post processes in Financials, in which case these criteria should be removed.
  8. This expression joins the record and instance number to the table in the database.  Instance 0 will be used to refer to the shared instance.
  9. The instance numbers of the temporary records is permitted to vary by up to one in either direction.  Thus the script generates profiles with instance 12 of one table, and instances 11 to 13 of the other.  It is rare, but possible, for there to be any difference in instance numbers between tables.  It is possible, but rarer for the difference to be greater than one.
  10. SQL Profile names are limited to 30 characters.  I have specified a meaningful name based on the Application Engine step, up to 26 characters, and then the two IDs (which can be up to 2 digits each).
  11. In this example, I am not going to use the full set of hints in the captured profile.  I just want to introduce a single leading hint.
  12. I will substitute each table name in the SQL text with the specific table instance name.
  13. Tables are generally referenced in hints via the row source alias.  However, sometimes the table name appears in the hints, and must also be replaced with the specific table name.  So I also work through all the hints in array h and substitute any table names that may be there.  Indexes are not referenced by their names but by a list of indexed columns.
The result is a set of profiles for each possible SQL statement.
XX_TREESEL.iDetVal.iDetVal00:11943215885789639839:
INSERT INTO PS_TSEL_P_TAO (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
FROM PS_TSEL_R30_TAO A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30

XX_TREESEL.iDetVal.iDetVal1717:15747497907378648788:
INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30

XX_TREESEL.iDetVal.iDetVal1718:12015611546030583918:
INSERT INTO PS_TSEL_P_TAO18 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
FROM PS_TSEL_R30_TAO17 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30

XX_TREESEL.iDetVal.iDetVal1817:14883898515022367531:
INSERT INTO PS_TSEL_P_TAO17 (CHARTFIELD, CHARTFIELD_VALUE, PROCESS_INSTANCE) SELECT DISTINCT 'ACCOUNT', DV.ACCOUNT, 12345678 
FROM PS_TSEL_R30_TAO18 A, PS_GL_ACCOUNT_TBL DV WHERE A.CHARTFIELD = 'ACCOUNT' AND A.PROCESS_INSTANCE = 12345678 AND DV.SETID 
= 'GLOBE' AND DV.EFFDT = (SELECT MAX(EFFDT) FROM PS_GL_ACCOUNT_TBL B WHERE SETID = DV.SETID AND ACCOUNT = DV.ACCOUNT AND 
EFFDT <= TO_DATE('2025-01-31','YYYY-MM-DD')) AND DV.ACCOUNT >= A.RANGE_FROM_30 AND DV.ACCOUNT <= A.RANGE_TO_30
…
Note
  • If the number of non-shared instances of a table in an Application Engine is increased, you need to build add table instances with Application Designer.
  • If the number of online temporary instances is increased, you need to build additional instances for every temporary record in the database.  If the number is reduced some tables will cease to be used, and they ought to be dropped.  
  • If either the number of application engine table instances or online temporary table instances are changed, then you will need to rerun the script to create additional SQL profiles.

Tuesday, March 20, 2018

Resetting High Water Marks on On-line Temporary Table Instances

PeopleSoft has always used regular database tables for temporary working storage in batch processes.   Up to PeopleTools 7.x working storage tables were shared by all instances of a program.  That led to consistent read contention when multiple processes concurrently used the same table, resulting in much higher high water marks that increased durations of full scans.
From PeopleTools 8, many copies of each temporary working storage table are created.  Application Engines that run on the Process Scheduler are allocated exclusive use of a particular copy of the table.  This avoids the inter-process contention.  They start by truncating each allocated table, which resets the high-water mark.
Some delivered processing uses batch programs that are run apparently synchronously from the PIA.  On-line edit and post in Financials General Ledger is a common example.  Up to PeopleTools 7, the application server would synchronously spawn a batch process and wait for it to complete.  From PeopleTools 8 the process is submitted to the process scheduler, and the PIA polls the Scheduler tables waiting for the process to complete.  However, Application Engine can be run within the component processor.  In Financials General Ledger, this can be chosen by a setting an installation configuration option.  The truly on-line method can perform better because you are no longer waiting for the process scheduler to pick up the process request.  A separate process Application Engine is not spawned, but the Application Engine program is executed by the PSAPPSRV application server process.  One of the limitations is that the Application Engine program cannot commit.  Committing after steps or sections is suppressed, and the %TruncateTable macro generates a delete statement instead.  Therefore, on-line temporary table instances are never truncated by any process and their high-water marks can be raised by processes that handle larger volumes of data.  This can have impacts for subsequent processes with smaller data volumes but that still have to full-scan working storage tables up to their high water marks.

Truncating On-line Temporary Table Instances

The answer is to implement a periodic process that truncates working storage tables, but only doing so when the table is not currently being used by a process.  Every on-line Application Engine program is allocated a temporary table instance number, it locks the corresponding row on the table PS_AEONLINEINST.  If it is allocated to instance 1, it locks the row where CURTEMPINSTANCE is 1 and uses instance 1 of each temporary record that it needs.  
Therefore the proposed truncate process must also lock the row on PS_AEONLINEINST that corresponds to each table that is to be truncated.  The truncate must be done in an autonomous transaction so that the implicit commit does not release that lock.  The lock can be released after the truncate completes.  Thus, the truncate process waits for any online process to complete before truncating a table with the same instance number, and no process can start while the truncate process is holding the lock.  However, each truncate will be very quick, and so each lock will only be held briefly, and it will have only a minimal effect on any online process that may be running at the time.  

I have written a PL/SQL packaged procedure (to perform this process for all temporary records.  It is available on Github as a part of my repository of miscellaneous PeopleSoft scripts.

Package Usage

Usually, the package will be run without any parameters. The default behaviour will be to truncate tables with more than a single extent.  Information on what the package does is emitted to the server output.
Set serveroutput on 
EXECUTE xx_onlineinsthwmreset.main;
The package can be run in test mode when it will list the commands without executing them.  Thus you can see what it will do without actually doing it.
EXECUTE xx_onlineinsthwmreset.main(p_testmode=>TRUE);
The package can optionally deallocate any physical storage. Storage will be reallocated next time the table is used.
EXECUTE xx_onlineinsthwmreset.main(p_drop_storage=>TRUE, p_min_extents=>0);
The package can be run for certain tables that match a particular pattern.
EXECUTE xx_onlineinsthwmreset.main(p_recname_like=>'JP%');
I recommend that the package is run daily. However, it can be run safely while the users are doing on-line edit/post processing, but it would be sensible to choose a quiet time.

Friday, March 25, 2016

Interview with PeopleSoft Administrator Podcast

I recently recorded an interview with Dan Iverson and Kyle Benson for the PeopleSoft Administrator Podcast. It has been spread over three episodes. There is lots of other good stuff on the website and other episodes that are well worth listening to.

(25 March 2016) #21 - Temporary Tables


(8 April 2016) #23 - The Application Server


(15 April 2016) #24 - Application Server Tuning

You can listen to the podcast on psadmin.io, or subscribe with your favourite podcast player, or in iTunes.

Thursday, February 17, 2011

Unlocking Temporary Table Instances from Deleted Process Requests

In a previous blog entry, I discussed how to detect whether an Application Engine process had run out of non-shared Temporary Table instances by querying the message log table for the entries written when this happens.

This can happen because processes that have locked all available non-shared table instances are currently processing. However, if a restartable Application Engine program terminates with an error, the tables remain locked to that process instance so that it can be restarted. If that process instance is neither cancelled nor restarted so that it subsequently runs to success then the tables will remain locked, and in time the Process Scheduler archive process may purge the process request record.

However, that archive process does not unlock the temporary table instances (by deleting the registration of the table with the instance from the record AETEMPTBLMGR), and once the process request has been deleted it cannot be cancelled in the Process Monitor. Therefore, there is no way for an operator to unlock the tables via the web interface.  Over time this can build up and new processes are forced to use the shared instance.  In my experience this is more common on development and test environments than production.

The problem is that is a silent and insidious problem that can gradually degrade batch performance.  Forcing processes onto the shared instance forces them to delete rather than truncate working storage tables.  So there is additional redo, and high water marks are not reset.  Concurrent processes can contend as they share the same physical table.  There may be more work to preserve read consistency, and more activity on the undo segment.

The way to prevent this problem is, of course, to make sure you have sufficient instance of temporary table, but before you create new instances check that there an no existing instances that should not still be locked. 

Update 7.3.2011: Phil's comment below correctly reminds me that PeopleSoft provided a way to remove locks without issuing SQL directly at PeopleTools -> Application Engine -> Review Temp Table Usage.  This component clears records from PS_AETEMPTBLMGR, PS_AERUNCONTROL and PS_AERUNCONTROLPC.  This feature is also described at PeopleSoft Tips & Notes: PS_AERUNCONTROL.

However, you can only remove locks for one process instance at a time.  If the problem has got out of hand, you might still chose to delete the rows from these three table where there is no longer a process request record. 

DELETE FROM ps_aetemptblmgr t
WHERE NOT EXISTS(
 SELECT 'x'
 FROM psprcsrqst r
 WHERE r.prcsinstance = t.process_instance)
/
DELETE FROM ps_aeruncontrol t
WHERE NOT EXISTS(
 SELECT 'x'
 FROM psprcsrqst r
 WHERE r.prcsinstance = t.process_instance)
/
DELETE FROM ps_aeruncontrolpc t
WHERE NOT EXISTS(
 SELECT 'x'
 FROM psprcsrqst r
 WHERE r.prcsinstance = t.process_instance)
/ 

Wednesday, May 13, 2009

Wednesday, April 22, 2009

Reducing Unnecessary Instances of Temporary Records

In a previous posting, I recommended moving temporary records, that are very frequently truncated by Application Engine programs, to a tablespace with a 32Kb block size, and using a larger uniform extent size (I chose 1Mb).

However, continuing the example for my last posting on this subject, TL_TIMEADMIN has 150 temporary records, that have 173 indexes (in HCM8.9). So you get 323 segments for every instance set in the Application Engine properties, and that would consume at least 323Mb of the 32Kb tablespace. If that space consumption is not a problem, then stop reading now.

However, I noticed that some temporary records are used by several Application Engine programs. This is usually because one program calls another and the temporary records are referenced in both. However, if both programs have a number of instances of temporary records defined in their properties, then temporary tables will be built for both.

Lets take an example TL_PUB_TM_AE calls TL_PUB_TM1. They are both delivered with 5 instances.

AE_APPLID    TEMPTBLINSTANCES
------------ ----------------
TL_PUB_TM1                  5
TL_PUB_TM_AE                5

They share 8 temporary records in common.

SELECT a.recname, a.ae_applid, b.ae_applid
FROM  psaeappltemptbl a
FULL OUTER JOIN psaeappltemptbl b
ON  a.recname = b.recname
AND b.ae_applid = 'TL_PUB_TM_AE'
WHERE a.ae_applid = 'TL_PUB_TM1'
ORDER BY 1
/

RECNAME         AE_APPLID    AE_APPLID
--------------- ------------ ------------
TL_PROF_LIST    TL_PUB_TM1
TL_PROF_WRK     TL_PUB_TM1
WRK_PROJ1_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ2_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ3_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ4_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ5_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ6_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ7_TAO   TL_PUB_TM1   TL_PUB_TM_AE
WRK_PROJ_TAO    TL_PUB_TM1   TL_PUB_TM_AE

5 temporary records are built by Application Designer for each Application Engine program. But TL_PUB_TM1 is never run on its own. So do you need the extra instances of those 8 temporary records? The temporary records defined on TL_PUB_TM_AE are a subset of TL_PUB_TM1. If you reduced the number of instances on TL_PUB_TM_AE to 0, you would still have the 5 instances defined on TL_PUB_TM_TM1. But that would enable you to drop 40 tables and their indexes.

So, I started to wonder if there was a general principle here. If the temporary tables on an Application Engine program are a subset of those on another program, then providing you make ensure the number of instances on the superset is not less than those of the subset, you could reduce the number of instances on the subset to 0.

This view reports Application Engine programs whose temporary records are a subset of those on another program, and also counts the number of records in the subset.

CREATE OR REPLACE VIEW gfc_aetemptbl_hier AS
SELECT
sup.ae_applid sup_applid, supa.temptblinstances sup_instances
,  sub.ae_applid sub_applid, suba.temptblinstances sub_instances
, (SELECT COUNT(*)
FROM   psaeappltemptbl supc, psaeappltemptbl subc
WHERE  supc.ae_applid = sup.ae_applid
AND    subc.ae_applid = sub.ae_applid
AND    subc.recname = supc.recname) num_records
FROM (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sup
, (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sub
, psaeappldefn supa
, psaeappldefn suba
WHERE sup.ae_applid != sub.ae_applid
AND supa.ae_applid = sup.ae_applid
AND suba.ae_applid = sub.ae_applid
AND EXISTS( /*a temporary record in common*/
SELECT 'x'
FROM psaeappltemptbl sup1
, psaeappltemptbl sub1
WHERE sub1.ae_applid = sub.ae_applid
AND sup1.ae_applid = sup.ae_applid
AND sup1.recname = sub1.recname
AND ROWNUM = 1)
/*there is no record in the subset that is not in the superset*/
AND NOT EXISTS(
SELECT  'x'
FROM psaeappltemptbl sub2
WHERE   sub2.ae_applid = sub.ae_applid
AND NOT EXISTS(
SELECT  'x'
FROM psaeappltemptbl sup2
WHERE   sup2.ae_applid = sup.ae_applid
AND sub2.recname = sup2.recname
AND ROWNUM = 1)
AND ROWNUM = 1)
/*there is a record in the subset that is not in the subset - so there is a difference*/
AND EXISTS(
SELECT  'x'
FROM psaeappltemptbl sup2
WHERE   sup2.ae_applid = sup.ae_applid
AND NOT EXISTS(
SELECT  'x'
FROM psaeappltemptbl sub2
WHERE   sub2.ae_applid = sub.ae_applid
AND sup2.recname = sub2.recname
AND ROWNUM = 1)
AND ROWNUM = 1)
ORDER BY 1,2;

This is the output from the view for the Application Engine programs in the example.

SUP_APPLID   SUP_INSTANCES SUB_APPLID   SUB_INSTANCES NUM_RECORDS
------------ ------------- ------------ ------------- -----------
…
TL_PUB_TM1               5 TL_PUB_TM_AE             5           8
TL_PUB_TM1               5 TL_PY_PUB_TM             5           5
TL_PUB_TM_AE             5 TL_PY_PUB_TM             5           5
…

I found that some Application Engine programs have identical sets of temporary records. This can happen when a program is cloned, which some customers do when they want to customise a vanilla program. This view reports on them.

CREATE OR REPLACE VIEW gfc_aetemptbl_eq AS
SELECT sup.ae_applid sup_applid, supa.temptblinstances sup_instances
, sub.ae_applid sub_applid, suba.temptblinstances sub_instances
, (SELECT COUNT(*)
FROM   psaeappltemptbl supc, psaeappltemptbl subc
WHERE  supc.ae_applid = sup.ae_applid
AND    subc.ae_applid = sub.ae_applid
AND    subc.recname = supc.recname) num_records
FROM (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sup
, (SELECT DISTINCT ae_applid FROM psaeappltemptbl) sub
, psaeappldefn supa
, psaeappldefn suba
WHERE sup.ae_applid < ae_applid =" sup.ae_applid" ae_applid =" sub.ae_applid" ae_applid =" sub.ae_applid" ae_applid =" sup.ae_applid" recname =" sub1.recname" rownum =" 1)" ae_applid =" sub.ae_applid" ae_applid =" sup.ae_applid" recname =" sup2.recname" rownum =" 1)" rownum =" 1)" ae_applid =" sup.ae_applid" ae_applid =" sub.ae_applid" recname =" sub2.recname" rownum =" 1)" rownum =" 1)">

Here, three programs share the same set of temporary records.

SUP_APPLID   SUP_INSTANCES SUB_APPLID   SUB_INSTANCES NUM_RECORDS
------------ ------------- ------------ ------------- -----------
…
ELEC_TSCRPT             20 E_TSCRPT_BAT            20           2
ELEC_TSCRPT             20 E_TSCRPT_LIB            20           2
E_TSCRPT_BAT            20 E_TSCRPT_LIB            20           2
…

I can use these view to set the instances on the subsets to 0 and increase the instances on the supersets as necessary. There are examples of both subsets within subsets and more than two Application Engine programs that share the same set of temporary tables. So I do this repeatedly until all the subsets have zero instances.

This PL/SQL script makes the updates to the Application Engine programs (including maintaining PeopleSoft version numbers), and also creates an Application Designer project called GFC_TTI with all the programs and records. This project can then be used to migrate the Application Engine programs to another environment.

DECLARE
l_any BOOLEAN;
l_projectname VARCHAR2(30 CHAR) := 'GFC_TTI';
l_version_aem INTEGER;
l_version_pjm INTEGER;

PROCEDURE projitem(objecttype   INTEGER
,objectid1    INTEGER
,objectvalue1 VARCHAR2) IS
BEGIN
INSERT INTO psprojectitem
(projectname ,objecttype
,objectid1 ,objectvalue1 ,objectid2 ,objectvalue2
,objectid3 ,objectvalue3 ,objectid4 ,objectvalue4
,nodetype ,sourcestatus ,targetstatus ,upgradeaction ,takeaction ,copydone)
VALUES
(l_projectname,objecttype
,objectid1, objectvalue1, 0, ' '
, 0, ' ', 0, ' '
,0,0,0,0,1,0);
EXCEPTION WHEN dup_val_on_index THEN NULL;
END;

BEGIN
UPDATE psversion
SET version = version+1
WHERE objecttypename IN('SYS','AEM','PJM');

UPDATE pslock
SET version = version+1
WHERE objecttypename IN('SYS','AEM','PJM');

SELECT version
INTO   l_version_aem
FROM   psversion
WHERE  objecttypename = 'AEM';

SELECT version
INTO   l_version_pjm
FROM   psversion
WHERE  objecttypename = 'PJM';

l_any := TRUE;
WHILE l_any LOOP
l_any := FALSE;
FOR i IN(
SELECT *
FROM gfc_aetemptbl_hier a
WHERE a.sub_instances > 0
AND NOT EXISTS(
SELECT 'x'
FROM   gfc_aetemptbl_hier b
WHERE  b.sup_applid = a.sub_applid
AND    b.sub_instances > 0
AND    ROWNUM = 1)
ORDER BY 1
) LOOP
UPDATE psaeappldefn x
SET    temptblinstances =     
GREATEST(x.temptblinstances
,i.sub_instances,i.sup_instances)
,      version = l_version_aem
,      lastupddttm = SYSDATE
WHERE  ae_applid = i.sup_applid; 

projitem(33,66,i.sup_applid);

UPDATE psaeappldefn x
SET    temptblinstances = 0
,    version = l_version_aem
,      lastupddttm = SYSDATE
WHERE  ae_applid = i.sub_applid;

projitem(33,66,i.sub_applid);
l_any := TRUE;
END LOOP;
END LOOP;

l_any := TRUE;
WHILE l_any LOOP
l_any := FALSE;
FOR i IN(
SELECT *
FROM  gfc_aetemptbl_eq a
WHERE a.sub_instances > 0
AND NOT EXISTS(
SELECT 'x'
FROM   gfc_aetemptbl_eq b
WHERE  b.sup_applid = a.sub_applid
AND    b.sub_instances > 0
AND    ROWNUM = 1)
ORDER BY 1
) LOOP
UPDATE psaeappldefn x
SET    temptblinstances =     
GREATEST(x.temptblinstances
,i.sub_instances,i.sup_instances)
,      version = l_version_aem
,      lastupddttm = SYSDATE
WHERE  ae_applid = i.sup_applid;

projitem(33,66,i.sub_applid);

UPDATE psaeappldefn x
SET    temptblinstances = 0
,      version = l_version_aem
,      lastupddttm = SYSDATE
WHERE  ae_applid = i.sub_applid;

projitem(33,66,i.sub_applid);
l_any := TRUE;
END LOOP;
END LOOP;
END;

INSERT INTO psprojectitem
(projectname ,objecttype
,objectid1 ,objectvalue1 ,objectid2 ,objectvalue2
,objectid3 ,objectvalue3 ,objectid4 ,objectvalue4
,nodetype ,sourcestatus ,targetstatus ,upgradeaction ,takeaction ,copydone)
SELECT DISTINCT
l_projectname,0
, 1, recname, 0, ' '
, 0, ' ', 0, ' '
, 0,0,0,0,1,0
FROM   psaeappltemptbl t
,      psprojectitem i
WHERE  i.projectname  = l_projectname
AND    i.objecttype   = 33
AND    i.objectid1    = 66
AND i.objectvalue1 = t.ae_applid
AND NOT EXISTS(
SELECT 'x'
FROM   psprojectitem i1
WHERE  i1.projectname = l_projectname
AND    i1.objecttype  = 0
AND    i1.objectid1   = 1
AND    i1.objectvalue1 = t.recname
);

BEGIN
INSERT INTO psprojectdefn
(projectname,version,projectdescr,tgtservername,tgtdbname
,tgtoprid,tgtopracct,comprelease,srccompreldttm,tgtcompreldttm
,compreldttm,keeptgt,tgtorientation,comparetype,commitlimit
,reportfilter,maintproj,lastupddttm,lastupdoprid,releaselabel
,releasedttm,objectownerid,descrlong)
VALUES
(l_projectname,l_version_pjm,'Temporary Table Instances',' ',' '
,' ',' ',' ',NULL,NULL
,NULL,31,0,1,50
,16232832,0,SYSDATE,'PS',' '
,NULL,' ','Application Engine programs, and related Temporary Records, '
||'whose number of temporary table instances have been changed');
EXCEPTION WHEN dup_val_on_index THEN
UPDATE psprojectdefn
SET    version = (SELECT version FROM psversion
WHERE  objecttypename = 'PJM')
,      lastupddttm = SYSDATE
WHERE  projectname = l_projectname;
END;
END;


Conclusion

The effect on my demo HCM8.9 system was to reduce the total number of temporary table instances from 5942 to 5106, a 14% reduction. However, when I tried this on an HCM9.0 system, I got a reduction of only 7%. This shows that PeopleSoft has been more careful about specifying the number of temporary tables in the later version.

Then you can use the script in an earlier posting to remove the excess tables.

Monday, April 06, 2009

Statistics Management for PeopleSoft Temporary Records in Application Engine Programs

(Updated 11.7.2014) Last year, I wrote about Oracle Optimizer Statistics and Optimizer Dynamic Sampling with PeopleSoft Temporary Records. Earlier this year, I encountered a situation where Optimizer Dynamic Sampling was not sufficient, and I needed properly gathered statistics on an object. I modified my PL/SQL packaged procedure wrapper that can be called by the %UpdateStats PeopleCode macro via a customised DDL model to collects statistics.

I still recommend locking statistics on PeopleSoft Temporary Record, so that table is omitted from schema-wide or database-wide operations to refresh statistics. If the statistics on a table are locked, and it is not a Global Temporary Table, then the wrapper package will force collection and update of statistics on the table (previously it suppressed gathering of statistics on tables with locked statistics).

However, when the Application Engine program completes, any statistics collected on those temporary tables are no longer needed. Worse, the statistics refer to data that will be deleted and replaced by some future program, and if the table were not reanalysed, the statistics would be misleading and could cause the database to produce an inappropriate execution plan. Some temporary records are shared by multiple programs, so you cannot guarantee that statistics will always be refreshed when the table is next used.

When an Application Engine program completes successfully, or when the process request is cancelled, specific instances of temporary records that were allocated when the program began are deallocated by deleting the row from PS_AETEMPTBLMGR. Therefore, I propose the following trigger that will delete the statistics for that record when that row is deleted.
CREATE OR REPLACE TRIGGER sysadm.gfc_deletetemptablestats
AFTER INSERT ON sysadm.ps_aetemptblmgr
FOR EACH ROW
WHEN (new.curtempinstance > 0)
DECLARE
 PRAGMA AUTONOMOUS_TRANSACTION;
 l_table_name VARCHAR2(30) := '';
 l_last_analyzed DATE := '';
 l_stattype_locked VARCHAR2(5) := '';
 table_doesnt_exist EXCEPTION;
 PRAGMA EXCEPTION_INIT(table_doesnt_exist,-20001);
BEGIN
 SELECT r.table_name, t.last_analyzed
 INTO   l_table_name, l_last_analyzed
 FROM ( 
        SELECT r.recname
        ,      DECODE(r.sqltablename,' ','PS_'||r.recname,r.sqltablename)||:new.curtempinstance table_name
        FROM   psrecdefn r
        ) r
        LEFT OUTER JOIN user_tables t
        ON t.table_name = r.table_name
 AND t.temporary = 'N'
 WHERE  r.recname = :new.recname;

 SELECT s.stattype_locked
 INTO   l_stattype_locked
 FROM   user_tab_statistics s
 WHERE  s.table_name = l_table_name
 AND    s.object_type = 'TABLE';

 IF l_last_analyzed IS NOT NULL THEN --only delete statistics if they exist
  dbms_stats.delete_table_stats(ownname=>'SYSADM',tabname=>l_table_name,force=>TRUE);
 END IF;
 IF l_stattype_locked IS NULL THEN --stats need to be locked, 21,11,2009
  dbms_stats.lock_table_stats(ownname=>user,tabname=>l_table_name);
 END IF;
 
EXCEPTION
  WHEN no_data_found THEN NULL;
  WHEN table_doesnt_exist THEN NULL;
END;
/
show errorss

NB: The trigger must use an autonomous transaction because dbms_stats also commits its updates.

You can test the trigger like this: First I will populate the control table with a dummy record, and collect statistics

INSERT INTO ps_aetemptblmgr
(PROCESS_INSTANCE, RECNAME, CURTEMPINSTANCE, OPRID, RUN_CNTL_ID, AE_APPLID
,RUN_DTTM, AE_DISABLE_RESTART, AE_DEDICATED, AE_TRUNCATED)
VALUES
(0,'TL_EXCEPT_WRK',24,'PS','Wibble','TL_TIMEADMIN',sysdate,' ', 1,0)
/
execute dbms_stats.gather_table_stats(ownname=>'SYSADM',tabname=>'PS_TL_EXCEPT_WRK24',force=>TRUE);

column table_name format a18
SELECT table_name, num_rows, last_analyzed
FROM user_tables where table_name = 'PS_TL_EXCEPT_WRK24'
/

TABLE_NAME           NUM_ROWS LAST_ANALYZED
------------------ ---------- -------------------
PS_TL_EXCEPT_WRK24          0 14:36:12 06/04/2009

Now I will delete the row, and the trigger will delete the statistics for me.

DELETE FROM ps_aetemptblmgr
WHERE process_instance = 0
and curtempinstance = 24
and recname = 'TL_EXCEPT_WRK'
/

SELECT table_name, num_rows, last_analyzed
FROM user_tables where table_name = 'PS_TL_EXCEPT_WRK24'
/

TABLE_NAME           NUM_ROWS LAST_ANALYZED
------------------ ---------- -------------------
PS_TL_EXCEPT_WRK24

Thursday, February 26, 2009

Do You Need More Temporary Table Instances?

When an Application Engine loads a program prior to execution, it attempts to allocate an instance of each temporary record specified in the program to itself. If the allocation of a table fails because there are no available instances, Application Engine is will use the shared instance (unless the program is configured to abort if non-shared tables cannot be assigned). In this case it will write an entry to the message log to warn that the shared instance of the record has been used.

When processes use the shared tables performance is likely to be degraded by contention on the table. The %TruncateTable() PeopleCode macro generates a DELETE by process instance on the shared table instead of a TRUNCATE command.
The problem is that unless you look in the message log, you will not know that this is happening. However, it easy to write a query that will look at the message log table and report whenever this has occurred.

REM tr_moreinst.sql
select  p.message_parm recname, r.prcsname
,  count(*) occurances
,  max(l.dttm_stamp_sec) last_occurance
,  max(p.process_instance) process_instance
from  ps_message_log l
,  ps_message_logparm p
 left outer join psprcsrqst r
 on r.prcsinstance = p.process_instance
where  l.message_set_nbr = 108
and    l.message_nbr = 544
and    p.process_instance = l.process_instance
and    p.message_seq = l.message_seq
and    l.dttm_stamp_sec >= sysdate - 7
group by p.message_parm, r.prcsname
order by 1,2
/
This report tells you which programs failed to allocated instances of which record, how many times that has happened within the last 7 days.
            Processes Unable to Allocate Non-Shared Temporary Record

                                                                  Last
Record          Process                  Last                   Process
Name            Name         Occurrences Occurrence            Instance
--------------- ------------ ----------- ------------------- ----------
TL_ABS_WRK      TL_TIMEADMIN           4 08:24:39 01/01/2009      12345
TL_ATTND_HST1   TL_TIMEADMIN          10 08:23:40 01/01/2009      12345
TL_COMP_BAL     TL_TIMEADMIN          11 08:23:40 01/01/2009      12345
...
NB: Just because an Application Engine could not allocate a non-shared table, does not automatically imply that you need more instances of that record. It could be that other processes had failed, but the temporary tables are still allocated to the process until the process is either restarted and completes successfully, or the process is deleted or cancelled.
You might choose to create some spare instances of records to allow for failed processes, but if you do not clear failed processes you will eventually run out of instances.

Friday, January 30, 2009

Managing Changes to the Number of Instances of Temporary Tables used in Application Engine Programs

When you run multiple copies of either the same Application Engine program, or different Application Engines that happen to use the same temporary work records, you need to worry about how many instances of the program or programs are likely to run concurrently, and hence how many instances of the temporary table to build.

But how do you manage these tables when you change the number of instances in the Application Engine properties?

How many tables are built for each Temporary Record?

The number of instances of each Application Engine program is set in the properties for that program (and stored on the PeopleTools table PSAEAPPLDEFN in the column TEMPTBLINSTANCES).

This screenshot is the Application Engine Program Properties for TL_TIMEADMIN. It is delivered configured for up to 10 concurrent instances.


The number of tables that are built for each Temporary Record is the sum of the instances in all the Application Engine programs in which the record is used, plus the number of Global Instances (in set up on the PeopleTools Options page.

Let’s take the record TL_PROF_LIST as an example. It is defined as a Temporary Record in 8 different Application Engine programs (in my HCM8.9 demo system).

SELECT a.recname, a.ae_applid, b.temptblinstances
FROM   psaeappltemptbl a, psaeappldefn b
WHERE  a.ae_applid = b.ae_applid
AND    a.recname = 'TL_PROF_LIST'
ORDER BY a.recname
/

RECNAME         AE_APPLID    TEMPTBLINSTANCES
--------------- ------------ ----------------
TL_PROF_LIST    TL_AGG_SECTN                1
TL_PROF_LIST    TL_OUTTCD                   5
TL_PROF_LIST    TL_PUB_TM1                  5
TL_PROF_LIST    TL_SCHHRSRPT                5
TL_PROF_LIST    TL_SCHRES_AE                9
TL_PROF_LIST    TL_ST_LIB                   5
TL_PROF_LIST    TL_TIMEADMIN               10
TL_PROF_LIST    TL_TRPROFILE               10

So, across all the programs 50 temporary tables are required.

SELECT * FROM pstemptblcntvw 
WHERE  recname = 'TL_PROF_LIST'
/

RECNAME         TEMPTBLINSTANCES
--------------- ----------------
TL_PROF_LIST                  50

The system has three global instances.

SELECT temptblinstances
FROM   psoptions
/

TEMPTBLINSTANCES
----------------
               3

So, Application Designer will build 54 tables based on this record.
  • 3 ‘global’ instances for Application Engines without a Process Instance number. The table names will be suffixed with numbers 1 to 3.
  • 50 ‘private’ instances for the number of requested instances of the Application Engines. The table names will be suffixed with numbers 4 to 53
  • 1 ‘shared’ version without a suffix (in other words the usual name) which is used if there is no instance no already allocated to a process instance.
And this is the top and bottom of the script to build the tables only generated by Application Designer.

CREATE TABLE PS_TL_PROF_LIST (PROCESS_INSTANCE DECIMAL(10) NOT NULL,
EMPLID VARCHAR2(11) NOT NULL,
EMPL_RCD SMALLINT NOT NULL,
START_DT DATE,
END_DT DATE) TABLESPACE TLWORK STORAGE (INITIAL 40000 NEXT 100000
MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10 PCTUSED 80
/
…
CREATE TABLE PS_TL_PROF_LIST53 (PROCESS_INSTANCE DECIMAL(10) NOT NULL,
EMPLID VARCHAR2(11) NOT NULL,
EMPL_RCD SMALLINT NOT NULL,
START_DT DATE,
END_DT DATE) TABLESPACE TLWORK STORAGE (INITIAL 40000 NEXT 100000
MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10 PCTUSED 80
/


Are there any temporary tables that have not been built but that should be built?

If you increase the number of instances of temporary tables on an Application Engine program then you may need to build the extra temporary tables. This query reports the missing tables.



I dropped PS_TL_PROF_LIST42, but I also increased the number of instance of TL_TIMEADMIN from 10 to 11. So the query reports that instances 42 and 54 of this table are missing, and all the other temporary tables for TL_TIMEADMIN also report a missing instance.

RECNAME           INSTANCE TEMPTBLINSTANCES
--------------- ---------- ----------------
…
TL_PMTCH_TMP1           19               19
TL_PMTCH_TMP2           19               19
TL_PROF_LIST            42               54
TL_PROF_LIST            54               54
TL_PROF_WRK             38               38
TL_PT_FINAL             29               29
…


The remedy is very simple. Application Designer will build a script for just the missing tables.

CREATE TABLE PS_TL_PROF_LIST42 (PROCESS_INSTANCE DECIMAL(10) NOT NULL,
EMPLID VARCHAR2(11) NOT NULL,
EMPL_RCD SMALLINT NOT NULL,
START_DT DATE,
END_DT DATE) TABLESPACE TLWORK STORAGE (INITIAL 40000 NEXT 100000
MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10 PCTUSED 80
/

Are there any temporary tables built that should not be built?

If you have reduced the number of temporary tables, then you may need to drop the excess tables.

This query reports tables that are beyond the number required.




I built the missing tables for the previous example, but then I reduced the number of instances on TL_TIMEADMIN back to 10. Now, the query reports that there is an extra table for each record beyond the number defined.

RECNAME         INSTANCE TEMPTBLINSTANCES TABLE_NAME
------------- -------- ---------------- ------------------
…
TL_PMTCH_TMP1       19               18 PS_TL_PMTCH_TMP119
TL_PMTCH_TMP2       19               18 PS_TL_PMTCH_TMP219
TL_PROF_LIST        54               53 PS_TL_PROF_LIST54
TL_PROF_WRK         38               37 PS_TL_PROF_WRK38
TL_PT_FINAL         29               28 PS_TL_PT_FINAL29
…


The problem is that Application Designer will not generate the DDL to drop any of these tables. If you reduce the number of temporary table instances, you will have to drop these tables yourself. However, now, it is easy to adjust the above query to produce the DDL to drop the tables.

SELECT 'DROP TABLE '||t.table_name||' PURGE;' cmd
FROM   …

The query then produces these commands.

…
DROP TABLE PS_TL_PMTCH_TMP119 PURGE;
DROP TABLE PS_TL_PMTCH_TMP219 PURGE;
DROP TABLE PS_TL_PROF_LIST54 PURGE;
DROP TABLE PS_TL_PROF_WRK38 PURGE;
DROP TABLE PS_TL_PT_FINAL29 PURGE;
…


The scripts in this posting can be downloaded from my website.

Tuesday, June 03, 2008

Oracle Optimizer Statistics and Optimizer Dynamic Sampling with PeopleSoft Temporary Records

PeopleSoft Temporary Records are used for working storage during Application Engine programs. Typically, AE programs truncate and repopulate the tables before using them. PeopleSoft recognised the need to keep the statistics on these tables in line with the data that they contain, and so used the %UpdateStats macro in many places in delivered programs to update the statistics.

However, frequently gathering statistics on even small tables can become time consuming. Recently, I have been working on PeopleSoft Time and Labor. This makes heavy use of temporary records. In a single execution of TL_TIMEADMIN, several tables associated with temporary records are truncated, repopulated and analyzed many times. I discussed the problem with excessive use of truncate elsewhere giving rise to Local Write Wait.

Oracle also recognised this problem, and in version 9 of the database they introduced Optimizer Dynamic Sampling, where the database samples the data to generate statistics at statement parse time.

I am still testing, but on Oracle 10gR2 (version 10.2.0.3) I have obtained improvements in performance and stability of T&L AE processes by:
  1. Deleting optimizer statistics on tables associated with temporary record in order to force the optimizer to sample at parse time
  2. Locking optimizer statistics to prevents the %UpdateStats macro from putting them back on. Tables with locked statistics are also omitted by GATHER_SCHEMA_STATS and GATHER_DATABASE_STATS (unless the force option is specified) and hence also by the delivered maintenance window job to refresh stale statistics.
  3. Implementing alternative DDL model that uses a PL/SQL packaged function to suppress the error when attempting to collect statistics on table whose statistics are locked (see %UpdateStats() -v- Optimizer Dynamic Sampling). This also addresses the the mix-up in the DDL models
  4. The final piece of the puzzle has been to set OPTIMIZER_DYNAMIC_SAMPLING to 4 at instance level. I certainly have had problems with this parameter set to the default of 2.
The dynamic sampling levels are described in the Performance Tuning Guide 14.5.6.4.
  • Level 2: Apply dynamic sampling to all unanalyzed tables.
  • Level 3: As Level 2, plus all tables for which standard selectivity estimation used a guess for some predicate that is a potential dynamic sampling predicate.
  • Level 4: As Level 3, plus all tables that have single-table predicates that reference 2 or more columns.
So the next stage is to identify working storage records and their associated tables.
I started off looking for tables that had recently been analysed.


The following script identifies all instances of temporary tables associated with temporary records, and then deletes and locks the statistics. I started by restricting it to list of specific tables, but I think it would be perfectly reasonable to take this approach with all temporary records.


BEGIN
 FOR x IN (
  SELECT /*+LEADING(o i r v)*/ t.table_name, t.last_analyzed, t.num_rows
  ,      s.stattype_locked
  FROM pstemptblcntvw i
    INNER JOIN psrecdefn r
    ON r.recname = i.recname
    AND r.rectype = '7'
  , psoptions o
  , user_tables t
     LEFT OUTER JOIN user_tab_statistics s
     ON  s.table_name = t.table_name
     AND s.partition_name IS NULL
  , (SELECT rownum row_number
     FROM   psrecdefn 
     WHERE  ROWNUM <= 100) v                 
  WHERE  v.row_number <= i.temptblinstances + o.temptblinstances
  AND    t.table_name = DECODE(r.sqltablename,' ','PS_'||r.recname,r.sqltablename)
                      ||DECODE(v.row_number*r.rectype,100,'',LTRIM(TO_NUMBER(v.row_number))) 
/*---------------------------------------------------------------------            
--AND    r.recname IN('TL_PMTCH1_TMP' --TL_TA000600.SLCTPNCH.STATS1.S…
--                   ,'TL_PMTCH2_TMP' --TL_TA000600.CALC_DUR.STATS1.S…)
-----------------------------------------------------------------------*/
  AND   (/*  t.num_rows        IS NOT NULL --not analyzed 
        OR   t.last_analyzed   IS NOT NULL --not analyzed
        OR*/ s.stattype_locked IS     NULL --stats not locked
        ) 
) LOOP
  IF x.last_analyzed IS NOT NULL THEN --delete stats
   dbms_output.put_line('Deleting Statistics on '||user||'.'||x.table_name);
   dbms_stats.delete_table_stats(ownname=>user,tabname=>x.table_name,force=>TRUE);
  END IF;
  IF x.stattype_locked IS NULL THEN --lock stats
   dbms_output.put_line('Locking Statistics on '||user||'.'||x.table_name); 
   dbms_stats.lock_table_stats(ownname=>user,tabname=>x.table_name);
  END IF;
 END LOOP;
END;
/


Updated 11.2.2009: I have updated my advice on the use of Optimiser Dynamic Sampling (see %UpdateStats -v- Optimizer Dynamic Sampling. I still consider this to be a useful feature, but I have found scenarios where Oracle has not chosen a better plan that it did choose with explicitly gathered statistics. Therefore, I still suggest locking statistics on temporary working storage record, but where batch programs have been coded to explicitly update statistics then dbms_stats should be called with the force option to override the lock. I have updated my
DDL model wrapper script accordingly.

The scripts in this posting can be downloaded from my website

Wednesday, January 09, 2008

Global Temporary Tables and PeopleSoft Temporary Records

At a round table at the UKOUG 2007 conference, the question of reducing redo during Application Engine batch programs was mentioned. For example, In both Global Payroll and Time & Labor, AE programs will shuffle data through a series of working storage tables before putting them into the final tables.

Using any RDBMS (not just Oracle) as a temporary working store is not efficient. The database will certainly store that data, but it will go to a lot of time and trouble to make sure that it is stored securely and can be recovered in the event of a disaster. In an Oracle database, every insert into these working storage tables will generate redo, and the user process must wait for the redo to be written to disk when the session commits. Excessive redo generation can become a performance issue, it can also impact on a Data Guard DR solution, and on the time taken to perform incremental backup and recovery. However, it is not necessary to be able to recover the data in the working storage tables, only the final result data.

Effect of Frequently Truncating Tables

I have observed significant amounts of time spent truncating working storage tables during batch programs, especially when concurrent process frequently truncate working storage tables. This is a typical example from a Statspack report.
Top 5 Timed Events                                Avg %Total
~~~~~~~~~~~~~~~~~~                               wait   Call
Event                           Waits Time (s)   (ms)   Time
----------------------------- ------- -------- ------ ------
local write wait                5,756    3,955    687   52.6
enq: RO - fast object reuse     1,500    2,685   1790   35.7

From Metalink, Oracle Note 334822.1 sums up the situation: "Processes that involve temporary tables being truncated and repopulated in multiple, concurrent batch streams may present this situation. The underlying problem is we have to write the object's dirty buffers to disk prior to actually truncating or dropping the object. This ensures instance recoverability and avoids a stuck recovery. It seems at first glance perfectly reasonable to simply truncate a temporary table, then repopulate for another usage. And then to do the temporary populate/truncate operations in concurrent batches to increase throughput. However, in reality the concurrent truncates get bogged down as dbwr gets busy flushing those dirty block buffers from the buffer cache. You will see huge CI enqueue waits. The multiple truncate operations in concurrent streams absolutely kill throughput. This is specially critical with large buffers.

"There was also a discussion in Bug: 4147840 (non-publish) where a PeopleSoft process was causing this behaviour because of the above explanation and they seemed to fix it by changing some PeopleSoft code to implement delete rather than truncate on small temporary tables."

But if we go back to deleting rows from these tables we will copy the contents to the redo log. Furthermore, deleting data will not reset the high watermark on the table.

GTTs

One option is to recreate working storage tables as Oracle Global Temporary Tables (GTT). There are two major advantages to GTTs.
  • They do not generate redo, although they do generate undo. Most of the time batch processes will be inserting data into these tables. The undo on the insert is just to delete the data and so is much smaller. If the table cleared by truncation rather rather than deleting the data, there is virtually no undo generated. The problem with local write wait and contention on the RO enqueue does not occur with GTTs.
  • For each session that references the same GTT, Oracle will create a separate physical copy of the table in the temporary segment. The Global Payroll engine is written in COBOL, and so it uses ordinary PeopleSoft records for working storage. If multiple 'streams' are used to run different sets of data simultaneously, the processes share the same working storage tables. Each session will have to scan through and discard working storage data from other sessions, and when updating or deleting from temporary tables the Oracle may have to generate read consistent copies of these data blocks. The same happens in Application Engine when there are no unlocked instances of working storage tables available. This effect can severely restrict the ability to scale the number of concurrent processes. If these tables were recreated as GTTs, then these problems are resolved, and the number of payroll streams can be scaled without creating any contention between them.
There are some other aspects of GTTs that it is important to understand
  • Because a copy of the GTT is created for each referencing database session, GTTs cannot be used to pass data between database sessions, nor can they be used for any on-line process in PeopleSoft because there is no guarantee which application server process will handle which service request. So, for example, you cannot use GTTs to hold dynamic nVision selectors if you plan to run nVision windows client in 3-tier mode. The service request that populates the dynamic selector may be executed by one PSQRYSRV process, but the service to run the query may be run by another.
  • The GTT will persist until either the end of the transaction or the session depending on how the GTT is created. In PeopleSoft, GTTs should be created PRESERVE ON COMMIT because we do not want to loose the data when process commits.
  • Although it is possible to collect optimizer statistics on GTTs, only one set can be stored on each table. So one session could end up using statistics collected by another session (see demonstration of this behaviour). It is probably better to remove statistics on from the GTTs, and rely upon Optimizer Dynamic Sampling (I discussed this in my article in Issue 31 of Oracle Scene Magazine.
GTTs work well in batch programs in PeopleSoft. They can be intrduced easily into SQR and COBOL processes because the processes run and terminate. When the process terminates, the database session is closed and Oracle automatically drops the physical copy of the GTT from the temporary segment relinquishing the space consumed. In Application Engine the position is a little more complicated.
  • Some AE programs can be restarted after a failure. They save their state information in the database as they progress, and can be restarted from the point at which they last committed. But, working storage data will not be preserved if it is held in a GTT, either because the database session will have terminated or because it will be restarted on a different PSAESRV server process. Therefore, restart must be disabled before GTTs can be introduced. This involves a customisation to the program.
  • Ideally, AE programs that use GTTs should run by a stand-alone psae executable, not a PSAESRV server process. PSAESRV server processes where introduced in PeopleTools 8.44 when the Process Scheduler became a fully fledged Tuxedo domain. These Tuxedo server processes each maintain a persistent connection to the database, so they do not have to sign into the database for every AE program request. They are only recommended for systems where there are a very large number of short AE program request, such as CRM. However, it is advantageous for the GTTs to be cleared out at the end of each process, otherwise they will continue to have an overhead in the temporary segment. Even delivered AE programs are not always conscientious about this.
There are operational aspects to disabling restart on Application Engine programs. Often rows of data are allocated to a specific process by writing the process instance on those rows. When a process fails it can leave these rows allocated to that process, and require some manual tidying up. This is a significant discouragement to disabling restart and implementing GTTs, which leaves us with the 'local write wait' problem. PeopleTools Temporary Records PeopleSoft recognised the contention problems caused by different processes sharing the same working storage table and introduced Temporary Record in PeopleTools 8.0. On Oracle it merely degrades performance due to the overhead of producing read-consistent copies of data block, but on other database platforms, read consistency is achieved by using page-level locks (although the subsequent release of SQL Server 2005 with its multi-versioning option gave it similar functionality to Oracle). Version 8 applications also saw a move away from Cobol towards Application Engine, with its new capability to run PeopleCode, for batch processing (although Global Payroll was an exception to this). Temporary records were designed for use in AE processes. Each Application Engine program must declare which temporary tables it will use (stored on PSAEAPPLTEMPTBL), and how many instances of it will be run concurrently (stored on PSAEAPPLDEFN). A PeopleTools Temporary Record is created as an ordinary database table, but Application Designer creates multiple copies of the table. It uses the view PSTEMPTBLCNTVW to calculate the total number of instances of the temporary table required by all AE programs that require it, and then adds on the number of Global Instances (stored on PSOPTIONS).
Take for example a Time & Labor working storage record TL_IPT3. The build script generated by Application Designer creates the table PS_TL_IPT3 as expected, but it used by Application Engine program TL_TIMEADMIN that specifies 10 instances, and there are 3 global instances, so it also created PS_TL_IPT31 through PS_TL_OPT313. So I get 14 copies of the table and its indexes. It will therefore come as no surprise that Temporary Records are the one of the major reasons why there are so many tables in a PeopleSoft database. In a vanilla demo HR database, there are over 6400 temporary tables out of a total of 17600 tables! Now, what if, having met the above criteria, I want to recreate make my Temporary Records as Global Temporary Tables? It is tempting to think that there is no longer any need to have multiple instances of the record, but there is some special behaviour for temporary records coded into PeopleTools. If you run out of instances of temporary tables, you use the table without the numeric suffix, and this treated as a shared temporary table. Thus temporary records should have the field PROCESS_INSTANCE as a part of their unique key, and if they do not you get a warning when you try to save them in Application Designer.
However, there is also a change in behaviour to the %TruncateTable macro. It is fast and efficient to clear a working storage table with TRUNCATE rather than DELETE. It generates only a trivial amount of redo and resets the high water mark on the table. The PeopleCode macro %TruncateTable evaluates to a TRUNCATE command on database platforms that support this command.
%TruncateTable (%Table(TL_IPT3))
If you are working on a temporary instance of a table then you don't have to worry about anyone else, and this macro evaluates, as expected to a truncate command (this test was performed on an Oracle database).
-- 11.44.57 .......(TL_TA_RULES.C1_00200.Step02) (SQL)
TRUNCATE TABLE SYSADM.PS_TL_IPT34
/
-- Row(s) affected: 0
COMMIT
/
However, if you have run out of temporary table instances and you are working on the shared table, %TruncateTable exhibits a special behaviour, it evaluates to DELETE and it automatically appends the criteria to specify the current process instance number.
-- 11.52.30 .......(TL_TA_RULES.C1_00200.Step02) (SQL)
DELETE FROM PS_TL_IPT3 WHERE PROCESS_INSTANCE = 1085
/
-- Row(s) affected: 0
COMMIT
/
Putting it all Together As is always the case in PeopleSoft, the challenge is to use an advantageous database feature without an unacceptable degree of customisation in PeopleSoft. I have considered and discounted the possibility of converting the Temporary Records to permanent records. That would require customisation of every working storage table and would change the behaviour of the %TruncateTable macro preventing multiple instances of the AE program from being able to execute concurrently. There is no doubt that it is nearly always preferable to clear out a working storage table by truncating it rather than deleting it. When you delete data you end up copying it as undo to the redo logs, and to the undo segment which as generates redo. So, we need to keep all the instances of the Temporary Records, so that we keep the normal behaviour of %TruncateTable, but we need to make them all into GTTs, including the shared instance, so that we save the redo overhead and we can still truncate them to release space back to the temporary segment and reset the high water mark. The following query lists the Temporary Records where there is no referencing Application Engine program where restart is enabled, the number of instances of each table, including the global instances.
SELECT r.recname
,      o.temptblinstances+NVL(c.temptblinstances,0) temptblinstances
FROM   psoptions o
,      psrecdefn r
LEFT OUTER JOIN pstemptblcntvw c ON c.recname = r.recname
WHERE  r.rectype = 7
AND NOT EXISTS( --omit tables where restart not disabled on referencing AE
SELECT 'x'
FROM   psaeappltemptbl t
,      psaeappldefn a
WHERE  t.recname = r.recname
AND    a.ae_applid = t.ae_applid
AND    a.ae_disable_restart = 'N' --restart not disabled
AND    ROWNUM = 1)
DDL for GTTs in App Designer So now we turn to the question of how to generate the DDL to build the GTTs. It is probably not a good idea to implement this in development and unit test databases. One of the problems with GTTs is that you cannot see what somebody else stored in their copy of a GTT. This can make debugging rather difficult because you cannot see the temporary data. So I suggest that this should only be done in configuration and user acceptance test and performance test databases before implementing in production. It is possible to get Application Designer to do this, but it requires both changes to the DDL Models to enable customisations to the record to comment out the physical storage clauses and to insert the GLOBAL TEMPORARY keywords into the DDL. However, it does require a customisation to each record to specify DDL overrides. The DDL Models to create tables and indexes on Oracle should be changed as follows. 1: Create Table:
CREATE **GLOBTEMP** TABLE [TBNAME] ([TBCOLLIST]) **GTSPCOM1** TABLESPACE [TBSPCNAME] STORAGE (INITIAL **INIT** NEXT **NEXT** MAXEXTENTS **MAXEXT** PCTINCREASE **PCT**) PCTFREE **PCTFREE** PCTUSED **PCTUSED** **GTSPCOM2**;
2: Create Index
CREATE [UNIQUE] **BITMAP** INDEX [IDXNAME] ON [TBNAME] ([IDXCOLLIST]) **GTSPCOM1** TABLESPACE **INDEXSPC** STORAGE (INITIAL **INIT** NEXT **NEXT** MAXEXTENTS **MAXEXT** PCTINCREASE **PCT**) PCTFREE **PCTFREE** PARALLEL NOLOGGING **GTSPCOM2**;
The three new parameters (in bold) should default to a blank string in the DDL model.
  • GLOBTEMP will be used to specify the keyword GLOBAL TEMPORARY.
  • GTSPCOM1 will be used to start a comment in front on the physical table attributes. It is not legitimate to specify these parameters on a GTT. This variable will also be used to specify the GTT clause 'ON COMMIT PRESERVE ROWS'.
  • GTSPCOM2 will be used to close the comment at the end of the physical table attributes.
DDL overrides need to be specified on the table
And all of its indexes.
It would be very tedious to add all these overrides manually, so I have produced two PL/SQL scripts that update the PeopleTools tables directly.
  • globtemp.sql: implements GTTs for all temporary records where there is no restartable AE program that references the table.
  • globtemp-undo.sql: reverts the record to normal tables where there is no restartable AE program (so that changes can be deliberately undone for testing).
Both scripts maintain PeopleTools object version numbers to cause the tables to cache properly in Application Designer. The scripts also maintain the items in an Application Designer project GLOBTEMP. The project should be created manually before the script is run. The scripts do not commit themselves, so that the person running them can decide whether to commit or rollback the changes. Sample output - this shows only the shareable table
DROP TABLE PS_GPCHAL074_TMP
/
CREATE GLOBAL TEMPORARY TABLE PS_GPCHAL074_TMP (PROCESS_INSTANCE
DECIMAL(10) NOT NULL,
FROMDATE DATE,
TO_DATE DATE) ON COMMIT PRESERVE ROWS /* TABLESPACE GPAPP STORAGE
(INITIAL 40000 NEXT 100000 MAXEXTENTS UNLIMITED PCTINCREASE 0)
PCTFREE 10 PCTUSED 80 */
/
CREATE UNIQUE  INDEX PS_GPCHAL074_TMP ON PS_GPCHAL074_TMP
(PROCESS_INSTANCE,
FROMDATE,
TO_DATE) /* TABLESPACE PSINDEX STORAGE (INITIAL 40000 NEXT 100000
MAXEXTENTS UNLIMITED PCTINCREASE 0) PCTFREE 10  PARALLEL NOLOGGING */
/
ALTER INDEX PS_GPCHAL074_TMP NOPARALLEL LOGGING
/

Unfortunately, the last ALTER INDEX command in the script raises an Oracle error - ORA-14451: unsupported feature with a temporary table.
But this command is added automatically to the create index commands and cannot be removed. This error should simply be ignored.