While moving a customer from On-Premises to Oracle Cloud Infrastructure, we recently conducted some performance testing before the final cutover. For the E-Business Suite database server, a physical box with 16 CPU cores was migrated to a VMStandard E3.Flex VM with 12 OCPU. From a CPU performance perspective, the new VM was much faster than the old box. Unfortunately, this was not the case on the I/O performance side. The first blog post (part 1) contains some general information about I/O performance in Oracle Cloud Infrastructure. In Part 2 I have covered how we can use ASM to overcome the limitations.

In this part we'll see how we can migrate an E-Business Suite on an OCI database instance to a LVM "RAID0" volume (striping) across a bunch of smaller disks.

Logical Volume Manager LVM

Wikipedia gives a good explanation on the purpose of Linux LVM:

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. LVM is used for the following purposes:

• Creating single logical volumes of multiple physical volumes or entire hard disks (somewhat similar to RAID 0, but more similar to JBOD), allowing for dynamic volume resizing.
• Managing large hard disk farms by allowing disks to be added and replaced without downtime or service disruption, in combination with hot swapping.
• On small systems (like a desktop), instead of having to estimate at installation time how big a partition might need to be, LVM allows filesystems to be easily resized as needed.
• Performing consistent backups by taking snapshots of the logical volumes.
• Encrypting multiple physical partitions with one password.
  LVM can be considered as a thin software layer on top of the hard disks and partitions, which creates an abstraction of continuity and ease-of-use for managing hard drive replacement, repartitioning and backup.

For our purpose we are mainly interested in the first aspect of LVM.

Creating a new Logical Volume

Unfortunately, E-Business Suite instances provisioned through Oracle Cloud Manager (be it new installations or be it the restore of a backup) "on Compute" only use a single "Block Volume" to store all the data and the Oracle-Home of the Database (the whole structure beginning with /u01). Fortunately, this can be changed with the following sequence of steps outlined below:

1. In the Oracle Cloud Infrastructure Console create 9 new Block Volumes (lvm1-lvm9) with 500 GB each and attach them to the database compute instance.
2. Run the iscsi-attach commands shown in OCI console for each of the volumes (making them available in the linux vm).
3. Convert the LUNs into LVM physical volumes using:
pvcreate /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk
4. Extend the volume group to use those disks:
vgextend ebs_vg /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk
5. Create a new logical volume:
lvcreate -i9 -I4M -l100%FREE -n ebs_lv3 ebs_vg
Note: I have made good experience using a stripe size of 4 Megabyte, however this may depend on your payload and DB Block size.
6. Verify with lvdisplay -m that the new logical volume is available and appropriately striped.

Moving data to the new Volume

The easiest approach is to do an "offline" copy of all the data to the new logical volume. For this I did:

1. Stop everything and unmount /u01
2. Move everything to the new volume:
dd if=/dev/dm-0 of=/dev/dm-1 bs=128M status=progress oflag=direct iflag=direct
Double-check to take the appropriate source and target. If you want to resize to a smaller target in the same process you can also use filesystem copy (while preserving all links and permissions). The copy worked with appx. 600MByte/s so took a bit more than 2 hours for our database.
3. Change /etc/fstab to mount ebs_lv2 instead of ebs_lv and remount /u01
4. Bring the database up again.

It is also possible to do an online-migration using an approach as described here:

1. Instead of step (5) of the lvcreation do a lvconvert to create a raid1 between the existing LUN and the new LUNs:
lvconvert --type mirror -m 1 --mirrorlog core --stripes 8 /dev/ebs_vg/ebs_lv /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
2. If this gives an error like "Number of extents requested (1165823) needs to be divisible by 8." first extend the logical volume by 1-7 extends to have something dividable.
3. Then break the mirror:
lvconvert -m0 /dev/ebs_vg/ebs_lv /dev/sdb
4. Reduce the VG by the old LUN:
vgreduce /dev/ebs_vg /dev/sdb
5. Remove the PV:
pvremove /dev/sdb
6. Detach the old 4.5TB Volume using the iscsi Commands, then using the OCI Cloud Console. Finally destroy the old/large volume.

Performance

Measuring Performance gives us an I/O-Performance that is by a factor of 7.7 (out of the theoretical maximum of 9) above the performance with a single drive. The throughput is a bit more than factor 3 higher than with a single drive:
Max IOPS = 178.571
Max MBPS = 1.409

This poor increase in throughput is due to the fact that we have hit the networking speed of our 12 OCPU VM: With "only" 12 GBps of network bandwidth we can reach no more than appx. 1400 MByte per second.

To check if this scales up even further with the added Network Bandwidth, change the number of OCPUs from 12 to 20, which gives us:

max_iops = 213.391 (theoretical max. by Storage: 225K)
max_mbps = 2.115 (theoretical max. by Storage 4.320; by Network 2.500 MByte/S)

This is again close to the physical limit of the network interface regarding the throughput, and as we see, the IOPS went a bit higher as well.

Overall, the performance was more or less the same compared to ASM. Note that for ASM I chose to use only 7 LUNs (since that was enough to contain all the data; then in addition, I had a LUN with 1 TB for the Oracle Home/...) while in the case with LVM everything went to the same logical volume requiring 9 LUNs (and thus providing an even better performance).

Summary

Using multiple Block Storage devices behind a single LVM logical volume with striping provides a convenient and reliable way to store the contents of an Oracle Database. In Oracle Cloud Infrastructure, more than a single block volume can easily be used to store the data. By striping of the data across multiple volumes, one can get a much better I/O performance than by using a single block volume, as provided by oracle "out of the box". For a database with multiple terabytes of storage, this provides a massive improvement in performance without additional OCI costs.

There are only two major drawbacks of this approach:

• Unfortunately, so far, it is not possible to perform a fast-clone using E-Business Suite Cloud manager. There is an Enhancement Request planning to add support for a similar functionality.
• Creating a backup and restoring this backup (e.g. for Cloning production to test) using Cloud Manager Tooling is possible. However, the target will also just have a single block volume. This leads to the testing environment being (slightly, hopefully this has only a performance impact) different from production.

To round up, I would recommend going for the LVM approach compared to ASM for the time being - especially if you do not use ASM for all of your "other" Oracle Databases in use.

The numbers are so excellent using this approach, that for now I do not give an "LVM Cache" on top of DenseIO instances with locally attached NVMe storage a try (as I had initially thought of). According to the documentation, this would at least not help much for an 8 OCPU instance much, given that DenseIO would then also "only" provide 250.000 IOPS. And it adds a lot of complexity and additional costs, especially since no "DenseIO E3 shape" is currently available.

While moving a customer from On-Premises to Oracle Cloud Infrastructure, we recently conducted some performance testing before the final cutover. For the E-Business Suite database server, a physical box with 16 CPU cores was migrated to a VMStandard E3.Flex VM with 12 OCPU. From a CPU performance perspective, the new VM was much faster than the old box. Unfortunately, this was not the case on the I/O performance side.

Let's take a look at how we measured the I/O performance and what the problem was. Then, in 2 more blog posts, I will look at two different approaches to solving this problem and their corresponding performance.

OCI I/O-Performance

Oracle gives a fairly detailed description of the I/O performance of block volumes (and that's all we are considering at the moment, since that’s the default location of the database) in OCI.

Basically there are 3 factors to consider for both throughput and IOPS:

• Network-Speed of the Compute Instance (for E3 this is 1GBit/OCPU)
• Performance Class can either be "Lower Cost" (1TB costs 26$ per month), "Balanced" (43$/TB) or "Performance" (60$/TB). The more expensive option you choose, the better the performance
• Size of the volume (up to a specific cut-off size)

The exact relationships / numbers are listed in the following table:

For the "Balanced" Block Volume type this means (depending on the volume size):

E-Business Suite Cloud Manager provisioned Database (on Compute)

When creating a compute instance to run the Oracle database (12c or 19c) for an E-Business Suite instance (12.1 and 12.2), Oracle creates only one large block volume containing both the Oracle home database and all of the data from archive log files. For one customer this was a volume of 4.5 TB. Unfortunately, this means that "only" a maximum throughput of 480MByte/s at 25,000 IOPS is available. Of course, this is only true if the compute instance has enough network bandwidth. In the case of E3.Flex instances, this means that at least 4 OCPUs are required (better 6 to have some buffer).

Measuring performance

There are many ways to measure the I/O Performance on Linux / for Oracle Databases. A convenient way is to let the database measure the performance on its own: This has the advantage that it also works for ASM based storage, as we'll see in the next episode.

The following program performs a short performance test:

SET SERVEROUTPUT ON
DECLARE
l_latency PLS_INTEGER;
l_iops PLS_INTEGER;
l_mbps PLS_INTEGER;
BEGIN
DBMS_RESOURCE_MANAGER.calibrate_io (num_physical_disks => 7,
max_latency => 20,
max_iops => l_iops,
max_mbps => l_mbps,
actual_latency => l_latency);
DBMS_OUTPUT.put_line('Max IOPS = ' || l_iops);
DBMS_OUTPUT.put_line('Max MBPS = ' || l_mbps);
END;
/

This gives us the numbers we would expect in several successive runs (on a 6 OCPU instance with a single 3.5 TB block volume of type "balanced performance"):

Max IOPS = 23833
Max MBPS = 448

When changing from "Balanced" to "Higher Performance" the numbers get a bit better (even though I expected the IOPS to increase even more):

Max IOPS = 28426
Max MBPS = 468

Get better

The obvious approach to improving this is to split up a block volume into possibly smaller block volumes. When using "Balanced Performance", it should be sufficient to reduce each block volume to about 1 TB in order to achieve "maximum throughput". To achieve maximum IOPS throughput, it might be even better to divide the single block volume into pieces of 500 GB each.

In two more blog posts, I'll describe how this can be achieved with either Oracle Automatic Storage Management (ASM) or by using the Linux Volume Management (LVM). Spoiler: With both methods the performance can be accelerated by a factor of 3-5 very easily and without additional costs.