Automating Code Signing with multiple E-Business Suite instances - Part 3

Automating Code Signing with multiple E-Business Suite instances - Part 3

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development

 

As you are probably aware by now, since June 1st 2023 all well-known / public Certificate Authorities (CA) no longer provide Code Signing Certificates using pure software based private keys (see https://www.linkedin.com/posts/johannes-michler-099892ab_code-signing-key-storage-requirements-will-activity-7090432157688492032-jGvC).

Since I prefer using such a trusted / public CA to sign Java Applets (that are still crucial for Oracle E-Business Suite or Oracle Forms) I've recently had a look into how we can now sign those Java JAR files. Part 1 of this blog series introduced the topic and an available "Cloud Based" Code Signing Certificate provider: Signing EBS/Forms - Part 1

In a second part I covered how the code signing can be done on an E-Business Suite Application Server running on Oracle Linux 7 on Oracle Cloud Infrastructure (OCI).

This third post will look how we can further automate this by installing the Certum tools onto the E-Business Suite Cloud Manager VM. First, we'll cover the latest changes from Certum, then we'll look into some scripts that can be used on multiple E-Business Suite Application servers to send the .jar files for signing to that central signing instance.

Certum Tool updates (April 2024)

Back in the previous post, I've complained about the incomplete translation of the Certum tools still revealing a lot of polish error messages. While it seems this is fixed at least partially, I realized that the 2.9.9 versions available over there https://files.certum.eu/software/SimplySignDesktop/Linux-RedHat/ leads to fatal crashes (segmentation fault). That is why for now I stuck with the 2.9.8 release.

Installing Certum SimplySignDesktop as a non-root user

When installing the SimplySignDesktop tool according to the official documentation it is necessary to do so globally/as the root user. Since I didn't like the tool to modify my cloud manager VM in that massive way, I've investigated what the installer actually does. With that I was able to get the tool running with a way less privileged user (that I call certum). Run the following as root:

yum install https://rpmfind.net/linux/epel/8/Everything/x86_64/Packages/s/stalonetray-0.8.3-15.el8.x86_64.rpm
yum install libxslt.x86_64 pulseaudio-libs-glib2.x86_64 libwebp.x86_64 xkeyboard-config
useradd certum
sudo su – certum
mkdir .ssh
vi .ssh/authorized_keys
# add the SSH public key(s) of your oracle@ebs-appserver
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

Then connect a SSH Session with X-Forwarding as certum:

wget https://files.certum.eu/software/SimplySignDesktop/Linux-RedHat/2.9.8-9.1.6.0/SimplySignDesktop-2.9.8-9.1.6.0-x86_64-prod-centos.bin
sh SimplySignDesktop-2.9.8-9.1.6.0-x86_64-prod-centos.bin --target /home/certum/
cp /home/certum/SSD-2.9.8-dist/SimplySignDesktop.xml /home/certum/

Create a /home/certum/provider_simplysign.cfg file as follows:

name=SimplySignDesktop/SimplySignPKCS
library=/home/certum/SSD-2.9.8-dist/SimplySignPKCS_64-MS-1.0.20.so
slot=-1

Furthermore, create a script startGUI.sh as follows:

export LD_LIBRARY_PATH=/home/certum/SSD-2.9.8-dist/
export QT_QPA_PLATFORM_PLUGIN_PATH=/home/certum/SSD-2.9.8-dist/plugins
export OPENSSL_CONF=/etc/ssl/
stalonetray &
/home/certum/SSD-2.9.8-dist/SimplySignDesktop

Finally start the Script and sign in with a one-time-token.

Do a test as follows (in new SSH Session):

/home/certum/SS-9.1.6.0-dist/jre/bin/keytool -list -keystore NONE -storetype PKCS11 -providerclass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -v

This will provide an alias, in our case: 4F4F410D1234A9110B16DA9C83BD6F59

Furthermore, create a /home/certum/mychain.pem file as described in the previous episode.

Passing the jars

On the E-Business Apps-Server first create a ~/sign_1.sh script as follows:

folderstamp=$(date +%Y-%m-%d-%H:%M)
mkdir -p /home/oracle/sign_bkp/${folderstamp}
jar=$1
# Remove Signature from jar files created through ADADMIN in EBS
echo " ** Removing EBS signature from: ${jar} "
cp -i ${jar} /home/oracle/sign_bkp/${folderstamp}/
zip -d ${jar} 'META-INF/*.SF' 'META-INF/*.RSA'
scp ${jar} certum@10.1.2.199:/tmp/signing-dummy.jar

ssh certum@10.1.2.199 "/home/certum/SS-9.1.6.0-dist/jre/bin/jarsigner -keystore NONE -tsa \"http://time.certum.pl\" -certchain /home/certum/mychain.pem -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -storepass 12345 /tmp/signing-dummy.jar 4F4F410D1234A9110B16DA9C83BD6F59"

scp certum@10.1.2.199:/tmp/signing-dummy.jar ${jar}

The script first creates a backup of the jar, then un-signs the .jar files and copies it to the cloud-manager VM (in my case with IP 10.1.2.199). There the jar is signed and finally the signed .jar is copied back to the E-Business Suite Apps Tier.

This allows signing a single .jar file; the script may be helpful when applying a patch with "options=nojarsigning". Then in there should be a file such as /u01/install/APPS/fs_ne/EBSapps/log/adop/176/20240327_132920/apply/mastebsapp01/36177213/log/jarlist.txt containing all the .jar files that require re-signing.

For the initial signing the procedure in the previous episode can be combined with the copying of the .jar to the Cloud Manager VM.

Verifying and patching

As an alternative to signing "just" the files in $NE_BASE/EBSapps/log/adadmin/log/jarlist.txt I found it useful to just sign all .jar files under $JAVA_TOP. For this the following script proved helpful:

folderstamp=$(date +%Y-%m-%d-%H:%M)
mkdir -p /home/oracle/sign_bkp/${folderstamp}
# Select the jar files from jarlist.txt
for jar in $(find $JAVA_TOP/oracle/apps -name \*.jar)
do
# Remove Signature from jar files created through ADADMIN in EBS
echo " ** Removing EBS signature from: ${jar} "
cp -i ${jar} /home/oracle/sign_bkp/${folderstamp}/
zip -d ${jar} 'META-INF/*.SF' 'META-INF/*.RSA'
scp ${jar} certum@10.1.2.199:/tmp/signing-dummy.jar
ssh certum@10.1.2.199 "/home/certum/SS-9.1.6.0-dist/jre/bin/jarsigner -keystore NONE -tsa \"http://time.certum.pl\" -certchain /home/certum/mychain.pem -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/certum/provider_simplysign.cfg -storepass 12345 /tmp/signing-dummy.jar 4F4F410D1234A9110B16DA9C83BD6F59"
scp certum@10.1.2.199:/tmp/signing-dummy.jar ${jar}
done

It is helpful to first to a check if the $jar is already signed as follows:

result=`jarsigner -verify -certs ${jar}| tr -d '[:space:]'`
if [[ "jarverified." != "$result" ]]
then
echo ${jar} needs re-sign; $result
# put the signing here
fi

Summary

Using above scripts, it is amazingly easy to sign all .jar files both initially as well as after applying a patch. The version using a "find" on $JAVA_TOP may sign "a bit more than needed", but in my experience that does not do any harm.

I am still hoping that Oracle will provide a way to "Hook" a script such as sign_1.sh into the signing process called during patching or through adadmin. This would probably be announced in "Signing EBS Jar Files With HSM (Hardware Security Module) - (Doc ID 2806640.1)".

Oracle Recognized as a Leader in Configure, Price, Quote by Independent Research Firm

Oracle has been recognized as a Leader in The Forrester Wave™: Configure, Price, Quote Solutions, Q2 2023. The report, published by Forrester Research in June 2023, evaluated 14 vendors based on 24 criteria across three categories: current offering, strategy, and market presence. Of the 14 providers recognized, Oracle Fusion Cloud Customer Experience (CX) was one of five Leaders and received the highest score in the current offering category.

For more information on the achievement, read the full article here.

Automating Code Signing with multiple E-Business Suite instances - Part 3

Turn off all the lights at night - Reducing costs by automatically pausing EBS@OCI instances

Johannes Michler PROMATIS Horus Oracle


Executive Vice President – Head of Platforms & Development


Costs Analysis dashboard in OCI

Running services on a public cloud platform, even when running the service 24/7, is often cheaper than running these services on premise regarding Total Cost of Ownership (TCO). Especially for development and testing instances, this cost advantages get even better when taking into consideration the following aspects:

  • The number of environments required is often volatile: There are phases when you need many environments with high performance (e.g. during a UAT of a major "new feature" or an "upgrade" implementation), but then there are many other months where a single "small" testing instance to troubleshoot issues might be enough.
  • Even during peak periods, such instances are often not needed 24/7 but only during usual business hours.

When running Oracle E-Business Suite on Oracle Cloud Infrastructure (OCI), this advantage is even more significant than with other cloud vendors: OCI allows to easily "rent" licenses for the database and application service required for E-Business Suite in a PaaS model "by the hour". This means: When reducing the average number of CPU cores used throughout the year (by one or both methods shown above), the cost savings can be very significant.

Let's have a look into the second aspect and see how we can easily shutdown E-Business Suite instances on OCI during night or weekend times.

Clean E-Business Suite shutdown

First of all, to automate pausing instances (and billing) during nighttime, you first have to cleanly shut down the E-Business Suite application server and middle tier. At least until today, there is no way of really "pausing" /"freezing" an instance without shutting down everything. Even though this is on the OCI roadmap, we will need to see if a complex environment such as an E-Business Suite will survive such a hibernate. I suggest the following stopEBS.sh script. You will obviously replace APPS_PWD and WEBLOGIC_PWD with your values:

source /u01/install/APPS/EBSapps.env run
adstpall.sh -mode=allnodes apps/APPS_PWD << EOF
WEBLOGIC_PWD
EOF

sh /home/oracle/stop_apex122.sh

echo -n "Waiting for Concurrent Manager to go down"

while true; do
$FND_TOP/bin/FNDSVCRG STATUS > /tmp/icmstatus2.txt
cat /tmp/icmstatus2.txt
if test `cat /tmp/icmstatus2.txt | grep "Internal Concurrent Manager is Active" | wc -l` -eq 0 ; then
echo
echo -n "Concurrent Manager is down now";
break;
fi
sleep 10;
done
ps -fu oracle
sleep 60
ps -fu oracle

Stopping of the OCI infrastructure (and billing)

With that, you can initiate the shutdown of the VMs (in this case with Base Database on VM and a single Apps Tier on Compute) using a script stopInstance.sh to which you need to pass the environment name:

COMPARTMENT_ID=ocid1.compartment.oc1..XXXXXXX
CONFIG_FILE=/u01/install/APPS/.oci/johannes.michler@promatis.de
ENV_NAME=$1

echo "Instance Name:"$ENV_NAME
HOST_APPS=${1,,}app01
IP_APPS=`dig +short ${HOST_APPS}.appssubnet.ebsnetwork.oraclevcn.com`
echo "IP Address Apps:"$IP_APPS
HOST_DB=${1,,}db
IP_DB=`dig +short ${HOST_DB}.dbsubnet.ebsnetwork.oraclevcn.com`
echo "IP Address DB:"$IP_DB

OCID_APPS=$(oci compute instance list --compartment-id $COMPARTMENT_ID --query "data [?\"display-name\" == '${ENV_NAME}app01'].id|join(',',@)" --config-file $CONFIG_FILE | tr -d '\"')
echo "OCID-APPS:"$OCID_APPS
OCID_DB_SYS=$(oci db system list --compartment-id $COMPARTMENT_ID --config-file $CONFIG_FILE --query "data [?\"display-name\" == '${ENV_NAME}'].id|join(',',@)"| tr -d '\"')
echo "OCI-DB-Sys:"$OCID_DB_SYS

OCID_DB_NODE=$(oci db node list --db-system-id $OCID_DB_SYS --config-file $CONFIG_FILE --compartment-id $COMPARTMENT_ID --query "data[].id|join(',',@)"| tr -d '\"')
echo "OCI-DB-Node:"$OCID_DB_NODE

echo Stopping apps tier
ssh $IP_APPS "./stopEBS.sh"
echo Stopping VM-DB
ssh $IP_DB "srvctl stop database -d \$ORACLE_UNQNAME -stopoption IMMEDIATE"
oci db node stop --config-file $CONFIG_FILE --db-node-id $OCID_DB_NODE

echo Now stopping Apps
oci --config-file $CONFIG_FILE compute instance action --action STOP --instance-id $OCID_APPS

The script makes use of the OCI Command Line Interface (CLI) to stop the database and the compute instance(s).

If you have the database on compute, you can use commands like the ones to stop the compute instance for the apps tier.

Keep in mind that you might need to disable advanced monitoring as well to stop billing of that OCI Database Management.

Bringing everything up again

Starting everything again works similar: I'm using a startInstance.sh script as follows:

echo "starting shutdown"
COMPARTMENT_ID=ocid1.compartment.oc1..XXXXX
CONFIG_FILE=/u01/install/APPS/.oci/johannes.michler@promatis.de
ENV_NAME=$1
echo "Instance Name:"$ENV_NAME
HOST_APPS=${1,,}app01
IP_APPS=`dig +short ${HOST_APPS}.appssubnet.ebsnetwork.oraclevcn.com`
echo "IP Address Apps:"$IP_APPS
HOST_DB=${1,,}db
IP_DB=`dig +short ${HOST_DB}.dbsubnet.ebsnetwork.oraclevcn.com`
echo "IP Address DB:"$IP_DB

OCID_APPS=$(oci compute instance list --compartment-id $COMPARTMENT_ID --query "data [?\"display-name\" == '${ENV_NAME}app01'].id|join(',',@)" --config-file $CONFIG_FILE | tr -d '\"')
echo "OCID-APPS:"$OCID_APPS
OCID_DB_SYS=$(oci db system list --compartment-id $COMPARTMENT_ID --config-file $CONFIG_FILE --query "data [?\"display-name\" == '${ENV_NAME}'].id|join(',',@)"| tr -d '\"')
echo "OCI-DB-Sys:"$OCID_DB_SYS

OCID_DB_NODE=$(oci db node list --db-system-id $OCID_DB_SYS --config-file $CONFIG_FILE --compartment-id $COMPARTMENT_ID --query "data[].id|join(',',@)"| tr -d '\"')
echo "OCI-DB-Node:"$OCID_DB_NODE

echo Starting DB
oci db node start --config-file $CONFIG_FILE --db-node-id $OCID_DB_NODE --wait-for-state AVAILABLE
echo Now Starting Apps
oci --config-file $CONFIG_FILE compute instance action --action START --instance-id $OCID_APPS --wait-for-state RUNNING &
wait
echo "started DB and Apps"
sleep 30
until ssh $IP_DB "echo da" 2> /dev/null
do
echo "not ready, waiting 5"
sleep 5
done
echo "sleeping another 30, dann start"
sleep 30

until ssh $IP_DB "srvctl start database -d \$ORACLE_UNQNAME"
do
echo "not ready, waiting 5"
sleep 5
done

until ssh $IP_APPS "echo da" 2> /dev/null
do
echo "not ready, waiting 5"
sleep 5
done
echo "sleeping another 30, dann start"
sleep 30
echo "Just in case the mountpoint is not yet there"
ssh opc@$IP_APPS sudo mount /u01

ssh $IP_APPS ./startEBS.sh

More fine-grained scaling - even without downtimes and maybe on PROD

Of course, the above approach can be down in less harsh manner. Instead of stopping the entire environment, you could, for example, simply stop some of the application tiers during nighttime; e.g. if you need 6 oacore servers to handle the daily work, but maybe only one during the night, then you can shut down five servers every night. And since in this way, the system stays available, you could even do so on production. OCI has dynamic scaling of CPU and memory on the roadmap, so maybe this gives even more flexibility in the future.

Summary

Using the above scripts and some crontab entries (e.g. on the E-Business Suite Cloud Manager machine) you can easily stop most of the costs for E-Business Suite Dev and Test Instances if they're not needed.
There are many reasons to run E-Business Suite on OCI as I've shown in previous blog posts. By dynamically scaling down the infrastructure during low or no-use periods costs can be dropped significantly! If you're interested to try it out, maybe look at the Free Trial for OCI and the E-Business Suite on OCI Hands On Lab - see my previous posts on things to consider while doing so with brand new tenancies.

Oracle Recognized as a Leader in Human Capital Management by Independent Research Firm

Oracle has been recognized as a Leader in The Forrester Wave™: Human Capital Management, Q2 2023. The report, published by Forrester Research in May 2023, evaluated 11 vendors based on 28 criteria across three categories: current offering, strategy and market presence. Of the 11 cloud HCM providers recognized, Oracle Fusion Cloud Human Capital Management (HCM) was one of only three Leaders and received the highest score in the current offering category.

For more information on the achievement, read the full article here.

PROMATIS receives Certificate: Expertise in Oracle E-Business Suite Applications to Oracle Cloud in Western Europe

PROMATIS receives Certificate: Expertise in Oracle E-Business Suite Applications to Oracle Cloud in Western Europe

We’re delighted to announce that we’ve achieved Service Expertise in Oracle E-Business Suite Applications to Oracle Cloud in Western Europe. This further demonstrates our commitment to provide holistic and seamless Oracle Cloud implementations.

This award not only testifies to our high level of expertise in Oracle E-Business Suite, but it also presents the satisfaction of our customers.

The Expertise Initiative was launched by Oracle to provide more transparency on the specific capabilities of Oracle partner companies. To receive a certification, an Oracle partner company must meet strict criteria: On the one hand, the requirements include the availability of enough certified consultants in the company, and on the other hand, corresponding customer references must be submitted, which confirm the company’s own project successes for the respective focus field.