An incident occurred with a customer of mine during running 3 Oracle Patches in a single node database (19.13 version).
The customer had automated scripts to run patches in Oracle databases. So, he entered as input three Oracle Patches and let the script run automatically to deploy those patches.
Unfortunately, this script was not running any prerequisites and did not have proper error handling to stop when the patch fails. The patches were 33515361 (Database Release Update 19.14.0.0.220118), 33561310 (OJVM Release Update 19.14.0.0.220118) and 33455225 (MVIEW refresh is getting failed with ORA-01740).
The outcome of that was, the RU patch (33515361) was deployed successfully, then the OJVM patch failed with an error and then the script continued to run the patch 33455225 which also failed. No datapatch -verbose was executed though since it is done manually.
So we are in a state where 2 patches have failed concurrently.
Both patches when the failed, they had the statement:
The following make actions have failed :
Re-link fails on target «ioracle».
OPatch failed to restore OH ‘/u01/app/oracle/product/19.0.0/dbhome_1’. Consult OPatch document to restore the home manually before proceeding.
NApply was not able to restore the home. Please invoke the following scripts:
- restore.[sh,bat]
- make.txt (Unix only)
to restore the ORACLE_HOME. They are located under
«/u01/app/oracle/product/19.0.0/dbhome_1/.patch_storage/NApply/2022-02-10_10-35-09AM»
UtilSession failed: Re-link fails on target «ioracle».
Log file location: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2022-02-10_10-35-09AM_1.log
OPatch failed with error code 73
I checked after that, the lsinventory, and it showed the first and last patch in the list. Of course, they could not be trusted after those failures.
I checked also if the Oracle home was configured correctly with opatch:
$ opatch prereq CheckOracleHome
After a little research, I decided to proceed with the Oracle recommendation to restore the Oracle home. So I executed the below:
cd /u01/app/oracle/product/19.0.0/dbhome_1/.patch_storage/NApply/2022-02-10_10-35-09AM
./restore.sh
sh make.txt
I restored the Oracle home only for the the first patch failure. I had the concern if it needed to run again for the second patch failure, but I thought that since the it would restore the Oracle home to a previous state then it would not need to run again for the other patch.
The restore and make ran successfully and restored the Oracle home and relinked it.
So then the last action would be to rollback the patches.
When I tried though to rollback, the opatch gave me an error:
ERROR : oracle_home/inventory/oneoffs/33515361 corrupted
It seems, that the restore had purged the directories of the patches that are used there from opatch. So no 33515361 or 33561310 existed there. Only the previous patches installed.
I also tried again to run lsinventory but failed again with the same error.
Then I researched the new error, and the only answers I could find were in RAC environments, where the patch had succeeded in one node and not in the other. So the dbas there, just copied the files from the one node to the directory missing in the other node. But unfortunately, this could not work for our case.
Then I assumed that these files in the directory, could possibly be the same as the ones from the original patch folder from the zip file. So I checked the other folders of the oneoffs directory with the previous patches. So, I found out that basically, two files are being used there, the actions.xml and the inventory.xml.
After that, I compared those xmls with the ones existing in the original patch folders of the previous patches and they seemed almost the same. (Only some beautification and a tag for standalone was different)
So, the assumption was verified that those files could be used too.
I created the two directories missing and I pasted the xmls from the orginal patch folders.
Afterwards, I checked and lsinventory was working correctly.
So I proceeded, with rolling back the patches and restoring the database. I did not run datapatch -verbose since it didn’t run the first time.
After rolling back the patches, I checked the lsinventory and the listed patches were removed. Then I started up the database, it opened successfully and I checked for patch violations and there were none:
select name, cause, type, message, status from pdb_plug_in_violations where type = ‘ERROR’ and status !=’RESOLVED’;
So the database was fully restored to its previous state as before the patches.
Written by Nikos Minaidis
11/02/2022
Nikos Minaidis 