Currently there is a status of UNRESOLVABLE GAP
on database DBFSXD01 and DBFSXD02
[code language=”sql”]
mrcche1de [DBFSXD021]> sql
SQL*Plus: Release 12.1.0.2.0 Production on Mon Jul 23 14:54:50 2018
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 8589934592 bytes
Fixed Size 7663544 bytes
Variable Size 1862271048 bytes
Database Buffers 6710886400 bytes
Redo Buffers 9113600 bytes
ORA-01105: mount is incompatible with mounts by other instances
ORA-01677: standby file name conversion parameters differ from other instance
[/code]
status for listener on this node
[code language=”sql”]
Service "DBFSXD02.mrcconsulting.com" has 3 instance(s).
Instance "DBFSXD021", status BLOCKED, has 1 handler(s) for this service…
Instance "DBFSXD022", status READY, has 1 handler(s) for this service…
Instance "DBFSXD023", status READY, has 1 handler(s) for this service…
BLOCKED
[/code]
During rolling patching of the cluster
First stop the service, then disable the service, then shutdown the instance
Non-rolling we will stop database
To stop service and instance on cluster
[code language=”sql”]
srvctl stop service -d <DB_NAME> -s <SERVICE_NAME> -i <INSTANCE_NAME>
srvctl stop service -d DBFSXB01 -s <SERVICE_NAME> -i DBFSXB011
srvctl disable service -d <DB_NAME> -s <SERVICE_NAME> -i <INSTANCE_NAME>
srvctl stop instance -d <DB_NAME> -n <HOSTNAME>
[/code]
To start service and instance on cluster
[code language=”sql”]
srvctl start instance -d <DB_NAME> -n <HOSTNAME>
srvctl enable service -d <DB_NAME> -s <SERVICE_NAME> -i <INSTANCE_NAME>
srvctl start service -d <DB_NAME> -s <SERVICE_NAME> -i <INSTANCE_NAME>
[/code]
Then we stop cluster as well
This is for rolling patching only
For non-rolling commands
Stop DB and service on the cluster
[code language=”sql”]
srvctl stop service -d <DB_NAME>
srvctl disable service -d <DB_NAME> -s <SERVICE_NAME>
srvctl stop database -d <DB_NAME>
[/code]
To start DB and instance on cluster
[code language=”sql”]
srvctl start database -d <DB_NAME>
srvctl enable service -d <DB_NAME> -s <SERVICE_NAME>
srvctl start service -d <DB_NAME>
[/code]
After stopping database/service we can stop CRS (cluster).
and is this on node for MRP
[code language=”sql”]
alter database recover managed standby database cancel;
[/code]
to disconnect don’t run anything just stop service and DB
DG is looking for thread1 archive 926 which is applied earlier and got deleted
so we need to re-image
As we don’t have backup we need to restore DG from primary
we just have it doing DBFS
And increase archive retention from 7 days to 15 days so that it will kept for long
it may take 2 hr est if no issues
[code language=”sql”]
sqlplus "/as sysdba"
startup nomount pfile=’/oracle/admin/DBFSXB01/pfile/initstdby.ora’
and then
rman target sys/password@DBFSXD01 auxiliary sys/password@DBFSXB01
duplicate target database for standby from active database;
[/code]
RMAN completed
now both Primary and standby are in sync
Primary : DBFSXD01
Standby : DBFSXB01
We already recreated standby for D01 to correct the issue
Rebuilt standby from primary using RMAN duplicate
if we have a failure how long DB duplicate from active will take
Actual copy took less than 30 mins. as DB is very small
Total time it took around 2 hrs.
i think that if we have the message we got at first we will have to rebuild that database
in the future and to prevent it from happening again
we will need to make sure we shutdown and startup properly
increase archive retention to 15 days. so that we don’t need to rebuild as long as archives are not deleted in that 15 days.
or we make sure they are not deleted before being applied to the standby side
current script already checking applied and more than 7 days old
in this case it was applied and got deleted after 7 days was applied?
due to instance crash, it is looking for instance1 archive while recovering instance2 archive for crash recovery
but instane1 archive got deleted as it was applied previously
increasing archive retention to 15 days will avoid this issue
on node 1 of VA
I should first take down the mrp process on the standby side
And make sure no lag between primary and standby before switchover
First we need to stop all instances except first node on each side that is both primary and standby, normally MRP will run on standby first node.
After stopping other instances on both primary and standby
Run this command on primary :
[code language=”sql”]
SELECT DATABASE_ROLE from v$DATABASE;
SELECT SWITCHOVER_STATUS FROM V$DATABASE;
ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY WITH SESSION SHUTDOWN;
[/code]
after this is done we need to run below commands on standby side
[code language=”sql”]
SELECT SWITCHOVER_STATUS FROM V$DATABASE;
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;
alter database open;
[/code]
and then modify CRS configuration with SRVCTL as primary and standby.
This command we need to run on primary
[code language=”sql”]
srvctl modify database -d <new_primary> -r primary -s open
[/code]
Now we need to start instances on all nodes of primary with SRVCTL
on new standby we need to stop existing instance and CRS configuration as standby by using below command
[code language=”sql”]
srvctl modify database -d <new_standby_db_name> -r physical_standby -s open
[/code]
Now start DB with SRVCTL command
Recap again:
unmount all DBFS filesystems first on each node
[code language=”sql”]
fusermount -uz /dbfs_test01
[/code]
First on the standby:
[code language=”sql”]
srvctl stop instance -d DBFSXB01 -i DBFSXB012,DBFSXB013
[/code]
Second on the primary
[code language=”sql”]
srvctl stop instance -d DBFSXD01 -i DBFSXD012,DBFSXD013
[/code]
once you have started all instances with SRVCTL
then we need to start MRP on standby
[code language=”sql”]
srvctl modify database -d DBFSXB01 -r physical_standby -s open
[/code]
Here we are modifying the shell of what the name was previously
[code language=”sql”]
srvctl modify database -d DBFSXD01 -r physical_standby -s open
[/code]
D01 and B02
D01 primary for B01
D02 primary for B02
i had failed over D01
D01 Primary for B01
we don’t need to any thing with B01 on TX side just we need to make D01 as standby with this command
Previously this was the primary database
[code language=”sql”]
srvctl modify database -d DBFSXD01 -r physical_standby -s open
[/code]
then
[code language=”sql”]
srvctl start database -d DBFSXD01
[/code]
then we need to start MRP on just one node
And we need to open new primary with below command once open DB and then stop
[code language=”sql”]
srvctl modify database -d DBFSXB01 -r primary -s open
[/code]
then we need to start DB on all nodes with SRVCTL
then we need to start MRP on Standby side
mrche1de [DBFSXB011]-> srvctl status database -d DBFSXB01
Instance DBFSXB011 is running on node mrche1de
Instance DBFSXB012 is running on node mrche1df
Instance DBFSXB013 is running on node mrche1dg
this is on VA cluster
Then we can start MRP on node 1 in TX in D01
[code language=”sql”]
alter database recover managed standby database using current logfile disconnect from session;
[/code]