Startseite 5 Schlagwort: Oracle E-Business Suite
Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

Theoretically, starting and stopping an Oracle E-Business Suite environment is trivial:

source /u01/install/APPS/EBSapps.env run;
adstpall.sh -mode=allnodes

However, in practice, doing so envelops some issues:

  • You'll have to enter the apps and WebLogic password,
  • running concurrent processes might cause issues.

Passing apps and WebLogic password

If you want to entirely automate the shutdown, you don't want to enter the apps and WebLogic password manually. This is especially necessary should you decide to gracefully stop your dev/test environment every night to safe on Infrastructure and Licensing costs on OCI, as described in this previous blog post. As described both over there and with some more details regarding the OCI Vault service here, that can be solved easily with some lines as follows:

adstpall.sh -mode=allnodes apps/$XX_APPS_PWD << EOF
$XX_WEBLOGIC_PWD
EOF

Stopping E-Business Suite Concurrent Managers

A more complex issue I've seen in a lot of customer environments is that just stopping the "internal concurrent manager" might be too aggressive. Think of a concurrent request starting other requests and waiting for them. If you shut down the environment without further measures before such child requests have kicked off, they will never start and the parent request will wait forever, preventing the shutdown of the concurrent manager.

A possible solution for this is to put the parent and the child request to different concurrent managers. That way, you can first stop the concurrent manager running the parent program and only stop the manager that runs the child processes after all the parent requests are completed.

With that, you're able to define an "order" in which you can shut down the concurrent managers so that the process reproducible succeeds with arbitrary starting points.

Automating the stopping of Concurrent Managers

The procedure shown above requires quite the manual effort. I've seen customers that had 5-10 "stages" in which the various concurrent managers had to be stopped. Unfortunately, by using some "semi-documented" APIs, it's possible to stop (and later start) these concurrent managers through scripts.

I've implemented this with the following bash helper function:

function stopManager {
stop_initiated=0
for (( ; ; ))
do
res=`sqlplus -s apps/$XX_APPS_PWD << EOF
set pages 0
set head off
set feed off
SELECT
running_processes
FROM
fnd_concurrent_queues
WHERE
concurrent_queue_name ='$2'
;
EXIT;
EOF`
echo running processes for $2 is: $res
if [ "$res" -eq "0" ]; then
break
fi
if [ "$stop_initiated" -eq "0" ]; then
CONCSUB apps/$XX_APPS_PWD SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND DEACTIVATE $1 $2
stop_initiated=1
fi
sleep 5
done
}

The function runs in a loop and has the following core components:

  • It checks if the manager is (still) running using a sqlplus select.
  • It stops the manager using CONCSUB.

The function is called with the Application Short Name and the name of the manager to stop.

By using this helper function, you can then have a main script as follows:

source /home/oracle/credsEnv.sh
source /u01/install/APPS/EBSapps.env run;

stopManager XXIS XX_RESTART_NP_MANAGER
stopManager FND XX_NP_MANAGER

#next in parallel:
stopManager FND XX_NP_MANAGER_1 &
stopManager FND XX_NP_MANAGER_2 &
stopManager FND XX_NP_MANAGER_3 &
wait
stopManager FND XX_MMP_MANAGER

This process first stops XX_RESTART_NP_MANAGER, after this is completed, it stops XX_NP_MANAGER.

Then, XX_NP_MANAGER_1, XX_NP_MANAGER_2 and XX_NP_MANAGER_3 are stopped in parallel (to save time).

Finally, we stop XX_MMP_MANAGER.

Bringing everything back up

Restarting the concurrent managers is straightforward, and usually, it is not necessary to wait until each stage is completed:

CONCSUB apps/$XX_APPS_PWD SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ACTIVATE XXIS XX_RESTART_NP_MANAGER
CONCSUB apps/$XX_APPS_PWD SYSADMIN 'System Administrator' SYSADMIN CONCURRENT FND ACTIVATE FND XX_NP_MANAGER

Summary

With the above-mentioned script and especially by leveraging CONCSUB (documented in 147449.1), it is easily possible to automate a "proper and robust" and yet "graceful" way of shutting down Oracle E-Business Suite. This is handy in many automation scenarios - no matter whether it is stopping dev/test to save money or automating patching.

Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Upgrading E-Business Suite 12.2 to use Analytics Publisher 12c

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

For almost a year already, it has been possible to change the document generation engine of an Oracle E-Business Suite 12.2 instance from BI Publisher 10g to Analytics Publisher 12c. Since so far, this isn’t widely used among E-Business Suite customers, let's have a closer look.

Motivation and procedure

The main reasons to do this are both increased stability, long-term support and especially the possibilities to create proper Office 2007+ xlsx files. This produces much smaller files especial for large reports.

The procedure to use Analytics Publisher 12c basically consists of applying some patches and then changing an autoconfig variable from xdo10g to xdo12c. The My Oracle Support (MOS) note "Using Oracle Analytics Publisher with Oracle E-Business Suite Release 12.2 (Doc ID 2964098.1)" describes this in full detail.

Issues with templates - Type safety

One issue we found was that some reports could no longer be generated with an XPTY0004 error. The reason for this is also described in 2964098.1: Analytics Publisher is stricter regarding type safety. While with 10g it didn't matter if you declared a variable as string and assigned a number (or vice versa), this now causes issues. While once identified, this is easy to fix, it can be awful to find all the places where the fix has to be applied.

Issues with templates - Matching closing tags

We had some more issues with more strict validations regarding matching opening and closing tags as well. Most of those problems are shown in the "XML Publisher Template Re-Generator" logfile. So, be sure to review that log for any errors or warnings with your custom reports.

Issues with fonts and the € symbol

In some reports we were printing the € symbol into a PDF file. After upgrading to xdo12c, that symbol was missing in all those PDFs. The situation was similar to what is described in "Euro Symbol is Displayed as Question Mark '?' in XML Publisher Report (Doc ID 387698.1)". However, in our 12.2.12 instance those Albany Fonts were available. And they were picked up when using xdo10g.

Setting the BI Publisher into debug mode as described in 286962.1 - "How to Obtain Debug Information for XML Publisher Version 5.6?" brought us to find this undocumented bug: Analytics Publisher 12c is relying on GoNotoCurrent*.ttf fonts that are not available in 12.2.12.

You can either wait for Oracle to fix that bug or apply the 12.2.14 patch or - if that is a bit of an overkill - just extract the GoNotoCurrent fonts from the 12.2.14 patch and put the files into $AF_JRE_TOP/lib/fonts.

I'll update this blog post once Oracle has published an official fix for this.

Summary

While typically, some adjustments are necessary in pre-existing custom BI Publisher reports, upgrading to Analytics Publisher 12c is definitely worth it. The technical procedure is straight forward – and on issues easy to undo –, and the new capability to create proper xlsx files was long waited for.

Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Upgrading E-Business Suite to 12.2.14 - FND_VAULT again

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

Mid of September Oracle released the latest E-Business Suite 12.2 RUP 12.2.14.

Tons of new features

As described in the official release notes, the new release contains tons of new functionalities in almost all application modules as well as a lot of new Enterprise Command Centers. Finally, some technical issues with Analytics Publisher 12c fonts got fixed.

So far, we just found a single regression bug with 12.2.14.

FND_VAULT issues again

Similar to when upgrading to 12.2.13 last year with a different customer, we unfortunately ran into some issues with FND_VAULT again. This is especially problematic since the package behind this is not only wrapped but also highly obfuscated. That makes any "private" bug hunting difficult, and you're typically completely in the hands of Oracle development for any analysis as well as a fix (even Oracle Support does nothing besides forwarding messages). And that may take time.

The error we ran into while applying the 12.2.14 patch was while applying AFVLTPCY.sql:

ORA-06502: PL/SQL: numeric or value error
ORA-06512: at "APPS.FND_VAULT", line 288
ORA-06512: at "APPS.FND_VAULT", line 260
ORA-06512: at "APPS.FND_VAULT", line 2203
ORA-06512: at "APPS.FND_VAULT", line 2325
ORA-06512: at "APPS.FND_VAULT", line 2338
ORA-06512: at line 220

A first attempt to resolve this was to apply 36842768 (merge=yes) together with the 12.2.14 patch (36026788) to get the latest version of FND_VAULT related packages:

/* $Header: AFSCVL8B.pls 120.8.12020000.24 2024/08/05 23:58:18 emiranda ship $

This didn't help unfortunately though.

With the help of some special debug patch we found out, that there seems to be a regression with fnd_vault entries longer than 30 characters. By removing those as follows (from the run edition):

exec fnd_vault.del('FND_SMTP','JOHANNES.FRIEDRICH.MICHLER@PROMATIS.DE');
commit;

With that, we were able to successfully restart the failed worker with adctrl.

You can find problematic entries by temporarily removing the policies preventing selects on fnd_vault_data and querying in that table for too long records (lenghtb>30).

Of course, you have to be sure that this data is not so important and at least not necessary for first tests. In my case, only FND_SMTP used as part of FND Output delivery was affected; furthermore, talking to the users showing up confirmed that they hadn't been using this functionality on purpose.

Summary

While Oracle has not fixed the issue (Bug 37083953 - 12.2.14 RUP FAILURE: ORA-06502 ERROR IN FND_VAULT DURING PATCH APPLICATION) as of today (beginning of October 2024), I've at least been able to identify the cause and find kind of an (ugly and not ready for production) workaround. With that, you can at least complete the patching cycle and check if everything else is working.

Other than that, all testing with 12.2.14 looks very smooth; so far, we haven’t run into any functional issues during testing.

Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Patching E-Business Suite with more comfort - part 2 notifying with PuTTY

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

Every quarter Oracle releases a bunch of Critical Patch Updates that have to be applied to (often many) E-Business Suite instances. While the patch application is highly automated through the adop utility, there are two (little) things that always annoyed me when patching:

  • You have to provide the apps, ebs_system and weblogic password on every adop command
  • You have to re-check your terminal connection to check if a certain step completed

This first part of this blog post series covered how we can easily pass the passwords to adop; in this second part I will show how notifications can be brought to your attention quickly once the patching is completed.

Notifications to PuTTY

It is often inconvenient that after kicking off the adop process (or something similar long-running) you have to re-check the PuTTY terminal over and over again to check if the process completed. Luckily PuTTY has the capability to "flash the task bar" as well as "play a sound" whenever a so call "bell-event" is produced on the remote session. The behavior of putty is controlled through those settings:


Settings for PuTTY to play the standard system alert sound as well as flash the task bar

With those setting in place you should be able to perform a test with the following command:

sleep 3; echo -e "\a"

Then quickly navigate away from the putty window and you should see the Putty Window flashing as well as a "dong" sound ringing:


Flashing Putty Window

Bell with screen

Typically I like to not start adop (or other long running processes) directly but I use a "screen" (https://www.gnu.org/software/screen/) session to start the process. In that way should my internet or VPN connection break down the process continues and I can reattach to the session.

Unfortunately screen does some "special" handling of the "bell" event which in the original configuration I found in my "jump host" from where I connect to the E-Business Suite environments screwed the putty notification of the bell event. To fix this I created a ~/.screenrc file as follows:

[opc@ebscm2 ~]$ cat .screenrc
bell_msg "Bell in window %n ^G"
vbell off

The first command sets the bell_msg to both contain a visual hint in which of potential multiple (background) windows of you screen session a bell event happened. The final ^G makes sure that another "bell" event is raised through that notification. With this setting you make sure that a background bell is actually passed on to PuTTY.

The second setting controls the foreground screen session. By disabling the visual bell (flashing of screen window) a regular bell is passed to the terminal.

The combination of those settings allow bell signals to be propagated to the PuTTY terminal both in foreground and background screen windows. Before continuing, it is a good idea to test both with the above mentioned echo command.

Creating a custom myadop.sh script

Based on this I extended the myadop.sh script from my first blog post by the last line:

[oracle@prod122app01 ~]$ cat /home/oracle/myadop.sh
source /home/oracle/credsEnv.sh
{ echo ${XX_APPS_PWD};echo ${XX_EBS_SYSTEM_PWD};echo ${XX_WEBLOGIC_PWD} ; } | adop "$@"
#now alert the terminal:
echo -e "\a"

Using this simple command I'm then able to run something like:

 ./myadop.sh phase=apply patches=36117775,36117775_D:u36117775.drv

This will result in a bell signal and a flashing putty taskbar logo as soon as the adop process completed.

Summary

Using this procedure I no longer have to check (eventually multiple) screen windows manually every few minutes if my patching completed - either successfully or in error. The procedure can obviously also be used for a lot of more long-running commands.

Building better stop scripts for Oracle E-Business Suite - gracefully stopping managers

Automating Code Signing with latest E-Business Suite Signing Improvements - Part 4

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

Last week, Oracle released a major re-work of its article on Signing JAR Files for Oracle E-Business Suite Release 12 (Doc ID 1591073.1). Taking into consideration that all major public Code Signing CAs now require Hardware Security Module (HSM) based storage of the private key this was long awaited. In previous parts of this blog series I've covered how we can workaround the limitations of jarsigning in E-Business Suite, see part 1, part 2 and part 3.

In this 4th part we'll cover how this procedure can be improved by using these latest patches to E-Business Suite.

Installation of required patches

To take advantage of the new procedures you have to apply a couple of patches (see 1591073.1) for more details:

adop phase=prepare,apply,finalize,cutover,cleanup patches=36492441
adop phase=prepare,apply,actualize_all,finalize,cutover,cleanup patches=36407980,36505379,36555711,36499096 finalize_mode=full mtrestart=no cleanup_mode=full

Note: The Support note explicitly states that you have to apply 36492441 first, so it seems that this has to be done in 2 patching cycles.

Implementation of a customjarsign.sh for Certum

Without doing anything in addition applying those patches still signs the jars in KEYSTORE mode (the same way as before). However now you have the possibility to write your own/modified customarsign.sh script that calls more advanced signing commands. In my case I've changed that file to have the main custom signing method as follows:

signjar_custom () # $1 name of 1 JAR file
{
echo "INFO: Signing JAR '$1' with certum..."
zip -d ${jar} 'META-INF/*.SF' 'META-INF/*.RSA'
scp $1 certum@172.31.2.199:/tmp/signing-dummy.jar
retry_count=0
while true; do
ssh certum@172.31.2.199 "/home/certum/SS-9.1.11.0-dist/jre/bin/jarsigner -keystore NONE -tsa \"http://time.certum.pl\" -certchain /home/certum/mychain.pem -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -storepass 12345 /tmp/signing-dummy.jar 4F4F410D0356A9110B16AC9C73BD6F59" | tee -a ~/customsign.log
if [ ${PIPESTATUS[0]} -eq 0 ]; then
break
fi
if [ $retry_count -eq 0 ]; then
ssh certum@172.31.2.199 "echo \"Please start certum on Cloud Manager\" | mailx -s \"Please start and sign in to Certum for patching/signing!\" XX@promatis.de"
fi
retry_count=$((retry_count + 1))
echo `date`":in retry-loop for $1 , iteration $retry_count" >> ~/customsign.log
echo "signing failed; will try again in 5 seconds. Press q to abort"
read -t 5 -n 1 key
if [ "$key" = 'q' ]; then
echo "Loop aborted by user."
return 1
fi
JAR_EXISTS_FILE=/home/oracle/NO_JAR_SIGNING
if [ -f "$JAR_EXISTS_FILE" ]; then
echo "/home/oracle/NO_JAR_SIGNING exists. Exit with error" | tee -a ~/customsign.log
return 1
fi
done
scp certum@172.31.2.199:/tmp/signing-dummy.jar $1
echo "`date`signing of $1 succesful" >> ~/customsign.log
return 0
}

Now let us walk through this step by step:

  • first of all, this relies on the idea of part 3 of the blog series to copy the jar file to be signed (residing in $1) to the cloud manager VM on 172.31.2.199.
  • Before doing this, we remove any old signatures in the jar file - even though this may not always be necessary; it shouldn't harm either
  • Then we copy the jar to be signed to /tmp/signing-dummy.jar on the signing compute instance
  • after that we call the jarsigner over there through ssh
  • and finally if everything worked fine we copy the signed /tmp/singing-dummy.jar back to the original place and return as successful

As you have probably noticed there is some additional code though:

  • First of all logging is not working that nice when just handing out everything to stdout, so that's why I've decided to put everything relevant to ~/customsign.log
  • Furthermore, the process to sign using the Certum tools requires an up-to-date login/token. Since that may or may not be available I've decided to query for the signing result status and then go into a loop checking again every 5 seconds.

Since there is no real way to interact with the user through adjss / adadmin/adop I've decided to have the following workarounds:

  1. On the first fail of signing I write an E-Mail to myself asking myself to start the signing tool
  2. For testing purposes and when calling customsign.sh directly I allow to abort the endless loop by pressing "q"
  3. For PROD scenarios where I'm somehow unable to get a token and still want to continue the overall cycle (of course then without signing the jars) I check for a ~/NO_JAR_SIGNING file and if that exists I exit the script with an error.

Unit Testing the script

The support note describes an easy way to unit-test the script; keep in mind to do some testing both with the Certum tool started and not started. At the end you should have a result similar to:


successful test calling

Using the new CUSTOM method

Once unit testing is successful you have to copy the customjarsign.sh file to $APPL_TOP_NE/ad/admin/customjarsign.sh.

Afterwards you can change the signing mode to CUSTOM:

adjss -mode CUSTOM

Once that is done you can test in a more realistic way by calling:

adjss -sign

That should produce a result as follows:

Successful test using adjss

Keep in mind to also do some negative testing (stopped and only later on started SignTool) as well.

Re-Signing using adadmin

After all the tests were successful you can simply call adadmin (or apply a patch) which should call your customjarsign.sh script and sign modified .jar files "on the fly".

Summary

The new jar-sign-tooling released by Oracle makes Code Signing a lot more convenient again. Beside of using Certum there is an even more convenient "partial implementation" already provided in the script from Oracle that uses the digicert signing which should work entirely without user interaction. If you can spare the additional $$ it is probably easier to go for digicert instead of certum thus.