Last week, I've upgraded the E-Business Suite of a first customer to the latest E-Business Suite RUP 12.2.13, Enterprise Command Centers V11 as well as Database 19.21.
A comprehensive list of the new features is available over here: https://www.oracle.com/a/ocom/docs/applications/ebusiness/ebs-12-2-13-and-ecc-november-2023.pdf
Symptoms of Issue while applying the patch
While/after applying the main 12.2.13 patch (34776655) unfortunately the instance broke down terribly. The issues that we were seeing were:
- Oracle Forms could no longer be opened. The following error was shown:
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
- Enterprise Command Centers can no longer be used. In the eccManaged Server we see logs as follows:
Caused by: java.lang.RuntimeException: java.sql.SQLException: ORA-01017: invalid username/password; logon denied at oracle.apps.fnd.security.AppsConnectionManager.makeGuestConnection(AppsConnectionManager.java:920) at oracle.apps.fnd.security.DBConnObj.getLabelledConnection(DBConnObj.java:298)
- The same error can also be seen with trivial dbc-based connection tests
- Since all that suggests there are issues with the APPS-Password, I then tried to change the APPS/APPLSYS/APPS_NE password using AFPASS. That gives:
[oracle@master230703app01 ~]$ date; AFPASSWD -s APPLSYS; date Wed Nov 22 13:11:02 CET 2023 Enter the ORACLE password of Application Object Library 'APPSUSER': Connected successfully to APPS. Error in password verification for APPS AFPASSWD completed with errors. Wed Nov 22 13:11:06 CET 2023
- Using the “older” tool to change the passwords FNDCPASS was similar:
oracle@master230703app01 ~]$ FNDCPASS APPS/AbCdEfG 0 Y EBS_SYSTEM/Secret1234 SYSTEM APPLSYS MYNEWPWD Log filename : L858096.log Report filename : O858096.out [oracle@master230703app01 ~]$ cat L858096.log +---------------------------------------------------------------------------+ Application Object Library: Version : 12.2 Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved. FNDCPASS: +---------------------------------------------------------------------------+ Current system time is 21-NOV-2023 00:19:40 +---------------------------------------------------------------------------+ Arguments FNDCPASS EBS_SYSTEM/***** SYSTEM APPLSYS ***** +----------------------------------------------------------------------------+ Working... Error in password verification for APPS +---------------------------------------------------------------------------+ Concurrent request completed Current system time is 21-NOV-2023 00:19:40 +---------------------------------------------------------------------------+
- Checking if the E-Business Suite Concurrent Managers are up also fails:
[oracle@master230703app01 ~]$ adcmctl.sh status You are running adcmctl.sh version 120.19.12020000.7 Enter the APPS username : apps Enter the APPS password : Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error. Internal Concurrent Manager status could not be determined. adcmctl.sh: exiting with status 0 adcmctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/MAST_mastebsapp01/logs/appl/admin/log/adcmctl.txt for more information ...
All those issues suggest, that something is broken with the Vault.
Cause
All those issues started even during the application of the patch; so even before the cutover phase the “run” edition of the E-Business Suite got broken.
A lot of analysis together with the E-Business Suite development team at Oracle revealed, that the problem is related with us having:
- The sec_case_sensitive_logon database parameter is set to FALSE. So this means, that password checking on the DB is not done case sensitive
- However, the APPS/APPLSYS/APPS_NE password on that instance has previously been set to a mixed case Password (aBcDeFgH instead of ABCDEFGH). While this is not a problem in normal 12.2.12 and AD/TXK 15 operation, somehow the patch stumbles over this.
Be aware: We've been "rather lucky" that the instance were we first realized this was not PROD. Even if you do all your testing first on a Dev/Test instance you might hit this issue only on PROD. Just think about having APPS as the APPS-Password on Dev/Test (I've heard of that ? ) and then having something FaNcY as the PROD password. Then the Dev/Test works perfect but on PROD you break the PROD even before you start your planned downtime for the ADOP Cutover.
Solution A: Fix before upgrade
If you’ve got a valid backup that you can easily restore probably the easiest approach to fix this is:
- Before applying the 12.2.13 patch change the password to something “all uppercase” (e.g. MYPWD) using AFPASSWD. See https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T174305.htm#522325 for all details.
- Optionally ( I did this, I don’t think though that it is necessary) enable sec_case_sensitive_logon to TRUE and (further) optionally change the password back to something with mixed case for better security.
- Only then apply the patch and it should go through without issues.
Solution B: If you're already screwed up
If your instance is already patched and “now broken” you can probably file a SR and mention that you’re running into the bug 36040277; the dev team can then provide you a temporary package that allows to change the APPS Password and thus repair the instance / the Vault.
Summary
Beside of this, all the patches as part of 12.2.13, AD/TXK 15 and ECC V11 could be applied without any issues – which proves again that the entire process and system in the meantime is very stable and those yearly RUPs are really not too big a deal to apply.