ORA-65028 when plugging in a PDB from a PDB archive into a RAC database

2020-04-28 Off By Markus Flechtner

I’m currently testing all the different PDB features like local and remote cloning or plugging in a PDB on a 19.6 Container-Database in a Maximum Availability Architecture (MAA), i.e. a setup with RAC, ASM and DataGuard in place. Everything went fine, except for the attempt to plugin a PDB from a PDB archive, which always resulted in an ORA-65028.

All attempts failed with the following error stack:

ORA-65028: Unable to open plugin data file at path +DATA
ORA-17503: ksfdopn:2 Failed to open file +DATA
ORA-15045: ASM file name '+DATA' is not in reference form

When checking the files in ASM, I found out that all tablespaces except SYSTEM had been copied to the diskgroup, occupying disk space and leaving an unusable PDB. As the PDB is unusable, it’s a good idea to drop it and to release the diskspace in the diskgroup.

My servers are running Grid Instrastructure and RDBMS 19.6:

oracle@geri:~/ [CDB1] $ORACLE_HOME/OPatch/opatch lspatches
30484981;OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.
oracle@geri:~/ [CDB1] +ASM1
oracle@geri:~/ [+ASM1] $ORACLE_HOME/OPatch/opatch lspatches
30655595;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30655595)
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
30489632;ACFS RELEASE UPDATE 19.6.0.0.0 (30489632)
30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

OPatch succeeded.

I did some other tests which worked fine, like

  • plugging in the same PDB in a CDB running on ASM in a Oracle Restart environment (Oracle 19.6)
  • plugging in the same PDB in my RAC database with one instance running

So I could rule out that the PDB archive file was somehow corrupt.

And a colleague told me that he could not reproduce the problem in his 19c environment.

So another aspect came into my mind: RDBMS compatibility on diskgroup level:

I’ve got a 11.2.0.4 database running on the same cluster, which is using the same diskgroups (“DATA” and “FRA”) like the 19c database.

In consequence, the RDBMS compatiblity level was set to “11.2.0.4” for “DATA” and “10.1.0.0.0” for “FRA”, because the compatibility level determines the lowest database version which can use the diskgroup.

So I moved the 11.2.0.4 database to another diskgroup and changed the RDBMS compatibility level of my diskgroups “DATA” and “FRA” to “12.2.0.0.0”.

SQL> column name format a10
SQL> column DATABASE_COMPATIBILITY format a30
SQL>  select name,DATABASE_COMPATIBILITY from v$asm_diskgroup;

NAME	   DATABASE_COMPATIBILITY
---------- ------------------------------
DATA	   12.2.0.0.0
FRA	   12.2.0.0.0

And – voilĂ  – I could plug in the PDB without any issues:

SQL> create pluggable database pdb2 
  2  using '/mnt/data/pdb1.pdb' 
  3  create_file_dest='+DATA' standbys=none;

Pluggable database created.

Perhaps setting the compatibility to “12.1.0.0.0” would have been sufficient, but I chose “12.2.0.0.0” because PDB archives were introduced with Oracle Database 12.2

Thanks to my colleagues Christian Gohmann and Ludovico Caldara for their valuable input when working on this issue.

Update:

A former colleague told me that he had the same problem, but changing the RDBMS compatibility did not solve the issue. So there must be another root cause for this issue.

References & more information