处理Oracle集群Asm磁盘阵列断网问题

加入收藏设为首页

备份与恢复: 处理Oracle集群Asm磁盘阵列断网问题
时间：2014/11/14 20:10:23  作者：solgle  来源：www.solgle.com  查看：3828  评论：0
内容摘要：今天刚到公司，发现Oracle数据库连接不上，马上远程登陆Linux服务器查看情况...本文出自：http://www.solgle.com/news/124.html[grid@solgle.com-2 bin]$ ./crs_stat -v -tCRS-0184: Canno...
今天刚到公司，发现Oracle数据库连接不上，马上远程登陆Linux服务器查看情况...
本文出自：http://www.solgle.com/news/124.html
[grid@solgle.com-2 bin]$ ./crs_stat -v -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@solgle.com-2 bin]$ ./crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

[grid@solgle.com-2 bin]$ ./crsctl start crs

CRS-4563: Insufficient user privileges.

CRS-4000: Command Start failed, or completed with errors.

[grid@solgle.com-2 bin]$

----查案磁盘

[grid@solgle.com-1 bin]$ oracleasm listdisks;

[grid@solgle.com-1 bin]$ oracleasm scandisks;

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[grid@solgle.com-1 bin]$ oracleasm scandisks;

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[grid@solgle.com-1 bin]$ oracleasm listdisks;

[grid@solgle.com-1 bin]$

-----检查网络

[grid@solgle.com-1 bin]$ ping 192.16.3.100

PING 192.16.3.100 (192.16.3.100) 56(84) bytes of data.

From 192.16.3.50 icmp_seq=1 Destination Host Unreachable

From 192.16.3.50 icmp_seq=2 Destination Host Unreachable

From 192.16.3.50 icmp_seq=3 Destination Host Unreachable

From 192.16.3.50 icmp_seq=15 Destination Host Unreachable

From 192.16.3.50 icmp_seq=16 Destination Host Unreachable

From 192.16.3.50 icmp_seq=17 Destination Host Unreachable

From 192.16.3.50 icmp_seq=18 Destination Host Unreachable

From 192.16.3.50 icmp_seq=19 Destination Host Unreachable

64 bytes from 192.16.3.100: icmp_seq=20 ttl=128 time=2007 ms

64 bytes from 192.16.3.100: icmp_seq=21 ttl=128 time=1004 ms

64 bytes from 192.16.3.100: icmp_seq=22 ttl=128 time=2.43 ms

------网线掉了，现在网络已经恢复，重启rac服务

-----磁盘阵列掉线，重新挂载磁盘

[root@solgle.com-1 ~]# /etc/init.d/iscsid restart

Stopping iscsid:

Starting iscsid: [ OK ]

[root@solgle.com-1 ~]# /etc/init.d/iscsi restart

Stopping iscsi: [ OK ]

Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target, portal: 2002:c010:364::c010:364,3260].

iscsiadm: initiator reported error (8 - connection timed out)

iscsiadm: Could not log into all portals [ OK ]

[root@solgle.com-1 ~]#

-----设置挂载

[root@solgle.com-1 ~]# iscsiadm -m discovery -t sendtargets -p 192.16.3.100

192.16.3.100:3260,1 iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target

[root@solgle.com-1 ~]# fdisk -l

... ...

Disk /dev/sdb: 10.7 GB, 10737418240 bytes

64 heads, 32 sectors/track, 10240 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0xe591bcf6

Device Boot Start End Blocks Id System

/dev/sdb1 1 10240 10485744 83 Linux

Disk /dev/sde: 43.2 GB, 43243274240 bytes

64 heads, 32 sectors/track, 41240 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x87f7ea70

Device Boot Start End Blocks Id System

/dev/sde1 1 41240 42229744 83 Linux

Disk /dev/sdd: 96.7 GB, 96720650240 bytes

255 heads, 63 sectors/track, 11758 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x374f77b9

Device Boot Start End Blocks Id System

/dev/sdd1 1 11758 94446103+ 83 Linux

[root@solgle.com-1 ~]#

----磁盘已经挂载进来

[root@solgle.com-1 ~]# oracleasm scandisks;

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "DISK3"

Instantiating disk "DISK1"

Instantiating disk "DISK2"

Instantiating disk "DISK4"

[root@solgle.com-1 ~]#

----再次登录到其它节点做磁盘挂载

[grid@solgle.com-2 bin]$ /etc/init.d/iscsid restart

Stopping iscsid:

rm: cannot remove `/var/run/iscsiuio.pid': Permission deniedFAILED]

rm: cannot remove `/var/run/iscsiuio.pid': Permission denied

rm: cannot remove `/var/lock/subsys/iscsid': Permission denied

Starting iscsid: [FAILED]

touch: cannot touch `/var/lock/subsys/iscsid': Permission denied

[grid@solgle.com-2 bin]$

[grid@solgle.com-2 bin]$ su - root

Password:

[root@solgle.com-2 ~]# /etc/init.d/iscsid restart

Stopping iscsid:

Starting iscsid: [ OK ]

[root@solgle.com-2 ~]#

[root@solgle.com-2 ~]# /etc/init.d/iscsi restart

Stopping iscsi: [ OK ]

Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target, portal: 2002:c010:364::c010:364,3260].

iscsiadm: initiator reported error (8 - connection timed out)

iscsiadm: Could not log into all portals [ OK ]

[root@solgle.com-2 ~]#

[root@solgle.com-2 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "DISK1"

Instantiating disk "DISK3"

Instantiating disk "DISK4"

Instantiating disk "DISK2"

[root@solgle.com-2 ~]#

-----执行启动服务

[grid@solgle.com-1 bin]$ ./crs_stat -v -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@solgle.com-1 bin]$ ./crsctl start crs

CRS-4563: Insufficient user privileges.

CRS-4000: Command Start failed, or completed with errors.

[grid@solgle.com-1 bin]$ su - root

Password:

[root@solgle.com-1 bin]# ./crsctl start cluster

CRS-2503: Resource 'ora.cssd' is in UNKNOWN state and must be stopped first

CRS-2679: Attempting to clean 'ora.cssd' on 'solgle.com-1'

CRS-2672: Attempting to start 'ora.diskmon' on 'solgle.com-1'

CRS-2676: Start of 'ora.diskmon' on 'solgle.com-1' succeeded

CRS-2681: Clean of 'ora.cssd' on 'solgle.com-1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'solgle.com-1'

CRS-2676: Start of 'ora.cssd' on 'solgle.com-1' succeeded

CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'solgle.com-1'

CRS-2672: Attempting to start 'ora.ctssd' on 'solgle.com-1'

CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'solgle.com-1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'solgle.com-1'

CRS-2676: Start of 'ora.ctssd' on 'solgle.com-1' succeeded

CRS-2672: Attempting to start 'ora.evmd' on 'solgle.com-1'

CRS-2676: Start of 'ora.evmd' on 'solgle.com-1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'solgle.com-1' succeeded

CRS-2679: Attempting to clean 'ora.asm' on 'solgle.com-1'

CRS-2681: Clean of 'ora.asm' on 'solgle.com-1' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'solgle.com-1'

CRS-2676: Start of 'ora.asm' on 'solgle.com-1' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'solgle.com-1'

CRS-2676: Start of 'ora.crsd' on 'solgle.com-1' succeeded

CRS-5702: Resource 'ora.crsd' is already running on 'solgle.com-1'

CRS-4000: Command Start failed, or completed with errors.

[root@solgle.com-1 bin]#

---同理启动solgle.com-2

... ...

CRS-2681: Clean of 'ora.asm' on 'solgle.com-2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'solgle.com-2'

CRS-2676: Start of 'ora.asm' on 'solgle.com-2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'solgle.com-2'

CRS-2676: Start of 'ora.crsd' on 'solgle.com-2' succeeded

CRS-5702: Resource 'ora.crsd' is already running on 'solgle.com-2'

CRS-4000: Command Start failed, or completed with errors.

[root@solgle.com-2 bin]# ./crsctl stat res -t

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Status failed, or completed with errors.

---过后几分钟

[root@solgle.com-2 bin]# ./crs_stat -v -t

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE solgle.com-1

ora.DATA2.dg ora....up.type 0/5 0/ ONLINE ONLINE solgle.com-1

ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE solgle.com-1

ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE solgle.com-1

ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE solgle.com-1

ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE solgle.com-1

ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE

ora.solgle.db ora....se.type 0/2 0/1 ONLINE OFFLINE

ora....network ora....rk.type 0/5 0/ ONLINE ONLINE solgle.com-1

ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE solgle.com-1

ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE solgle.com-1

ora....SM1.asm application 0/5 0/0 ONLINE ONLINE solgle.com-1

ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE solgle.com-1

ora.solgle.com-1.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.solgle.com-1.ons application 0/3 0/0 ONLINE ONLINE solgle.com-1

ora.solgle.com-1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE solgle.com-1

ora....SM2.asm application 0/5 0/0 ONLINE ONLINE solgle.com-2

ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE solgle.com-2

ora.solgle.com-2.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.solgle.com-2.ons application 0/3 0/0 ONLINE OFFLINE

ora.solgle.com-2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE solgle.com-2

ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE solgle.com-1

[root@solgle.com-2 bin]# ./crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA1.dg

ONLINE ONLINE solgle.com-1

ONLINE ONLINE solgle.com-2

ora.DATA2.dg

ONLINE ONLINE solgle.com-1

ONLINE ONLINE solgle.com-2

ora.LISTENER.lsnr

ONLINE ONLINE solgle.com-1

ONLINE ONLINE solgle.com-2

ora.asm

ONLINE ONLINE solgle.com-1 Started

ONLINE ONLINE solgle.com-2 Started

ora.gsd

OFFLINE OFFLINE solgle.com-1

OFFLINE OFFLINE solgle.com-2

ora.net1.network

ONLINE ONLINE solgle.com-1

ONLINE ONLINE solgle.com-2

ora.ons

ONLINE ONLINE solgle.com-1

ONLINE ONLINE solgle.com-2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE solgle.com-1

ora.cvu

1 ONLINE ONLINE solgle.com-1

ora.solgle.db

1 ONLINE OFFLINE Instance Shutdown,STARTING

2 ONLINE OFFLINE STARTING

ora.oc4j

1 ONLINE ONLINE solgle.com-1

ora.solgle.com-1.vip

1 ONLINE ONLINE solgle.com-1

ora.solgle.com-2.vip

1 ONLINE ONLINE solgle.com-2

ora.scan1.vip

1 ONLINE ONLINE solgle.com-1

[root@solgle.com-2 bin]#

----最终实例solgle.com-1没有启动起来

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE solgle.com-1

ora.cvu

1 ONLINE ONLINE solgle.com-1

ora.solgle.db

1 ONLINE OFFLINE Instance Shutdown

2 ONLINE ONLINE solgle.com-2 Open

ora.oc4j

1 ONLINE ONLINE solgle.com-1

ora.solgle.com-1.vip

1 ONLINE ONLINE solgle.com-1

ora.solgle.com-2.vip

1 ONLINE ONLINE solgle.com-2

----单独启动实例solgle.com-1

[grid@solgle.com-1 bin]$ ./srvctl start instance -d solgle -n solgle.com-1

PRCR-1013 : Failed to start resource ora.solgle.db

PRCR-1064 : Failed to start resource ora.solgle.db on node solgle.com-1

CRS-5017: The resource action "ora.solgle.db start" encountered the following error:

ORA-00205: error in identifying control file, check alert log for more info

. For details refer to "(:CLSN00107:)" in "/u01/app/grid/11.2.0/log/solgle.com-1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.solgle.db' on 'solgle.com-1' failed

[grid@solgle.com-1 bin]$

[grid@solgle.com-1 bin]$ ./srvctl start instance -d solgle -n solgle.com-1

PRCR-1013 : Failed to start resource ora.solgle.db

PRCR-1064 : Failed to start resource ora.solgle.db on node solgle.com-1

CRS-5017: The resource action "ora.solgle.db start" encountered the following error:

ORA-00603: ORACLE server session terminated by fatal error

ORA-27504: IPC error creating OSD context

ORA-27300: OS system dependent operation:sendmsg failed with status: 105

ORA-27301: OS failure message: No buffer space available

ORA-27302: failure occurred at: sskgxpsnd2

Process ID: 0

Session ID: 0 Serial number: 0

. For details refer to "(:CLSN00107:)" in "/u01/app/grid/11.2.0/log/solgle.com-1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.solgle.db' on 'solgle.com-1' failed

[grid@solgle.com-1 bin]$ su - root

[grid@solgle.com-1 bin]$ cat /etc/sysctl.conf 显示都不正常了

-------------------一下为解决该问题-----------------------------------------------------------------------------

--重启了linux服务器，并重新挂载了磁盘重复了上面的操作解决了

... ...

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE solgle.com-2

ora.cvu

1 ONLINE ONLINE solgle.com-2

ora.solgle.db

1 ONLINE ONLINE solgle.com-1 Open

2 ONLINE ONLINE solgle.com-2 Open

ora.oc4j

1 ONLINE ONLINE solgle.com-2

ora.solgle.com-1.vip

1 ONLINE ONLINE solgle.com-1

ora.solgle.com-2.vip

1 ONLINE ONLINE solgle.com-2

ora.scan1.vip

1 ONLINE ONLINE solgle.com-2

[grid@solgle.com-1 bin]$

标签：Oracle集群Asm磁盘阵列断网的恢复


solgle.com 版权所有，欢迎分享！！！

上一篇：逻辑DataGuard重新部署逻辑备用数据库
下一篇：RMAN-06207:警告:由于状态不匹配,所以不能删除1对象(对于DISK通道)

相关文章

相关评论

本类更新

本类推荐

本类排行

: 国家数据 | 中证指数 | Oracle官方文档 | 沪深行情 | 流量统计 | 深交所 | 上交所 | 中国证券登记 | 同业拆放利率 | 巨潮资讯 | 指数估值 | 看财报 |

联系我们 - 关于我们 - 网站同盟