This week, Oracle released the version 23.3.1 of Cloud Manager. More details can be found in my previous blog post New E-Business Suite Cloud Manager Release 23.3.1 as well as in https://blogs.oracle.com/ebsandoraclecloud/post/introducing-ebs-cloud-manager-2331
The new release finally provides the possibility to change the apps, ebs_system and weblogic password during cloning. This is great, and for most of my customers it works "out-of-the box".
However with one customer I ran into the issue as follows:
[10.22.14.8] * ruby_block[20231223_02-50-20 : Start Admin Server on Run file system : execution] action run [10.22.14.8] [2023-12-23T02:50:20+01:00] INFO: Processing ruby_block[20231223_02-50-20 : Start Admin Server on Run file system : execution] action run (oracle-ebs::alter-weblogic-password line 105) [10.22.14.8] ***** Start of STDOUT ***** E-Business Suite Environment Information ---------------------------------------- RUN File System : /u01/install/APPS/fs2/EBSapps/appl PATCH File System : /u01/install/APPS/fs1/EBSapps/appl Non-Editioned File System : /u01/install/APPS/fs_ne DB Host: prodebsora.intern.dns Service/SID: UPG Sourcing the RUN File System ... You are running adadminsrvctl.sh version 120.10.12020000.11 Starting WLS Admin Server... Refer /u01/install/APPS/fs2/inst/apps/UPG_prodebsapp01/logs/appl/admin/log/adadminsrvctl.txt for details AdminServer logs are located at /u01/install/APPS/fs2/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/logs adadminsrvctl.sh: exiting with status 1 adadminsrvctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/UPG_prodebsapp01/logs/appl/admin/log/adadminsrvctl.txt for more information ... ***** End of STDOUT *****
Digging into the AdminServer.out logfiles I realized that cloud manager starts and stops the Admin Server multiple times. The first shutdown of the AdminServer didn't work cleanly/fast enough though. Then Cloud Manager did ignore this and tried to bring the AdminServer up again - which failed obviously due to the parallel shutdown progress.
Investigating the AdminServer logs during the shutdown revealed Stuck Threads around the EmbeddedLdap server. This brought me to "Admin Server Take a Long Time to Shutdown (Doc ID 2992284.1)" which fits a Cloning Situation quite good.
This lead me to modifying the Oracle provided script to change the passwords as follows:
[oracle@ebscm2 cloudmanager_custom]$ diff /u01/install/APPS/apps-unlimited-ebs/resources/chef-repo/cookbooks/oracle-ebs/recipes/change-datasource-passwords.rb /u01/install/APPS/apps-unlimited-ebs/resources/chef-repo/cookbooks/oracle-ebs/recipes/change-datasource-passwords.rb.org
44c44
< startAdminServerCmd = "#{sourceEnvCmd}; rm /u01/install/APPS/fs?/FMW_Home/user_projects/domains/EBS_domain/servers/AdminServer/data/ldap/conf/replicas.prop; sh $INST_TOP/admin/scripts/adadminsrvctl.sh start -nopromptmsg"
---
> startAdminServerCmd = "#{sourceEnvCmd}; sh $INST_TOP/admin/scripts/adadminsrvctl.sh start -nopromptmsg"
So I just delete the replicas.prop before the initial startup (on RUN and PATCH Filesystem). This will be fixed during startup anyways.
Then just do:
knife cookbook upload -a -V -o /u01/install/APPS/apps-unlimited-ebs/resources/chef-repo/cookbooks
After that, also on this environment cloning worked again with Cloud Manager 23.3.1.
I guess that this might be related with this being a Multi-Zone environment with many managed servers on multiple nodes.
I hope Oracle will incorporate this into the next Cloud Manager release; it shouldn't harm to recreate that file during Cloning.