备份与恢复

处理Oracle集群Asm磁盘阵列断网问题

时间:2014-11-14 20:10:23  作者:solgle  来源:www.solgle.com  查看:3145  评论:0
内容摘要:今天刚到公司,发现Oracle数据库连接不上,马上远程登陆Linux服务器查看情况...本文出自:http://www.solgle.com/news/124.html[grid@solgle.com-2 bin]$ ./crs_stat -v -tCRS-0184: Canno...
今天刚到公司,发现Oracle数据库连接不上,马上远程登陆Linux服务器查看情况...
本文出自:http://www.solgle.com/news/124.html
[grid@solgle.com-2 bin]$ ./crs_stat -v -t
CRS-0184: Cannot communicate with the CRS daemon.
 
[grid@solgle.com-2 bin]$ ./crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
 
[grid@solgle.com-2 bin]$ ./crsctl start crs
CRS-4563: Insufficient user privileges.
CRS-4000: Command Start failed, or completed with errors.
[grid@solgle.com-2 bin]$ 
 
 
----查案磁盘
[grid@solgle.com-1 bin]$ oracleasm listdisks; 
[grid@solgle.com-1 bin]$ oracleasm scandisks;
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[grid@solgle.com-1 bin]$ oracleasm scandisks;
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[grid@solgle.com-1 bin]$ oracleasm listdisks;
[grid@solgle.com-1 bin]$ 
 
-----检查网络
[grid@solgle.com-1 bin]$ ping 192.16.3.100
PING 192.16.3.100 (192.16.3.100) 56(84) bytes of data.
From 192.16.3.50 icmp_seq=1 Destination Host Unreachable
From 192.16.3.50 icmp_seq=2 Destination Host Unreachable
From 192.16.3.50 icmp_seq=3 Destination Host Unreachable
From 192.16.3.50 icmp_seq=15 Destination Host Unreachable
From 192.16.3.50 icmp_seq=16 Destination Host Unreachable
From 192.16.3.50 icmp_seq=17 Destination Host Unreachable
From 192.16.3.50 icmp_seq=18 Destination Host Unreachable
From 192.16.3.50 icmp_seq=19 Destination Host Unreachable
64 bytes from 192.16.3.100: icmp_seq=20 ttl=128 time=2007 ms
64 bytes from 192.16.3.100: icmp_seq=21 ttl=128 time=1004 ms
64 bytes from 192.16.3.100: icmp_seq=22 ttl=128 time=2.43 ms
 
------网线掉了,现在网络已经恢复,重启rac服务
 
-----磁盘阵列掉线,重新挂载磁盘
[root@solgle.com-1 ~]# /etc/init.d/iscsid restart
Stopping iscsid: 
Starting iscsid:                                           [  OK  ]
[root@solgle.com-1 ~]# /etc/init.d/iscsi restart
Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target, portal: 2002:c010:364::c010:364,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals    [  OK  ]
[root@solgle.com-1 ~]# 
 
-----设置挂载
[root@solgle.com-1 ~]# iscsiadm -m discovery -t sendtargets -p 192.16.3.100
192.16.3.100:3260,1 iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target
 
[root@solgle.com-1 ~]# fdisk -l
... ...
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xe591bcf6
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1       10240    10485744   83  Linux
 
Disk /dev/sde: 43.2 GB, 43243274240 bytes
64 heads, 32 sectors/track, 41240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x87f7ea70
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1       41240    42229744   83  Linux
 
Disk /dev/sdd: 96.7 GB, 96720650240 bytes
255 heads, 63 sectors/track, 11758 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x374f77b9
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       11758    94446103+  83  Linux
[root@solgle.com-1 ~]# 
----磁盘已经挂载进来
[root@solgle.com-1 ~]# oracleasm scandisks;
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DISK3"
Instantiating disk "DISK1"
Instantiating disk "DISK2"
Instantiating disk "DISK4"
[root@solgle.com-1 ~]# 
 
 
----再次登录到其它节点做磁盘挂载
[grid@solgle.com-2 bin]$ /etc/init.d/iscsid restart
Stopping iscsid: 
rm: cannot remove `/var/run/iscsiuio.pid': Permission deniedFAILED]
rm: cannot remove `/var/run/iscsiuio.pid': Permission denied
rm: cannot remove `/var/lock/subsys/iscsid': Permission denied
Starting iscsid:                                           [FAILED]
touch: cannot touch `/var/lock/subsys/iscsid': Permission denied
[grid@solgle.com-2 bin]$ 
[grid@solgle.com-2 bin]$ su - root
Password: 
[root@solgle.com-2 ~]# /etc/init.d/iscsid restart
Stopping iscsid: 
Starting iscsid:                                           [  OK  ]
[root@solgle.com-2 ~]# 
[root@solgle.com-2 ~]# /etc/init.d/iscsi restart 
Stopping iscsi:                                            [  OK  ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.1991-05.com.microsoft:solgle.com-1-lin-server-data-target, portal: 2002:c010:364::c010:364,3260].
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not log into all portals   [  OK  ]
[root@solgle.com-2 ~]# 
[root@solgle.com-2 ~]# oracleasm scandisks       
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "DISK1"
Instantiating disk "DISK3"
Instantiating disk "DISK4"
Instantiating disk "DISK2"
[root@solgle.com-2 ~]# 
 
 
 
-----执行启动服务
[grid@solgle.com-1 bin]$ ./crs_stat -v -t
CRS-0184: Cannot communicate with the CRS daemon.
[grid@solgle.com-1 bin]$ ./crsctl start crs
CRS-4563: Insufficient user privileges.
CRS-4000: Command Start failed, or completed with errors.
 
[grid@solgle.com-1 bin]$ su - root
Password: 
[root@solgle.com-1 bin]# ./crsctl start cluster
CRS-2503: Resource 'ora.cssd' is in UNKNOWN state and must be stopped first
CRS-2679: Attempting to clean 'ora.cssd' on 'solgle.com-1'
CRS-2672: Attempting to start 'ora.diskmon' on 'solgle.com-1'
CRS-2676: Start of 'ora.diskmon' on 'solgle.com-1' succeeded
CRS-2681: Clean of 'ora.cssd' on 'solgle.com-1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'solgle.com-1'
CRS-2676: Start of 'ora.cssd' on 'solgle.com-1' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'solgle.com-1'
CRS-2672: Attempting to start 'ora.ctssd' on 'solgle.com-1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'solgle.com-1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'solgle.com-1'
CRS-2676: Start of 'ora.ctssd' on 'solgle.com-1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'solgle.com-1'
CRS-2676: Start of 'ora.evmd' on 'solgle.com-1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'solgle.com-1' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'solgle.com-1'
CRS-2681: Clean of 'ora.asm' on 'solgle.com-1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'solgle.com-1'
CRS-2676: Start of 'ora.asm' on 'solgle.com-1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'solgle.com-1'
CRS-2676: Start of 'ora.crsd' on 'solgle.com-1' succeeded
CRS-5702: Resource 'ora.crsd' is already running on 'solgle.com-1'
CRS-4000: Command Start failed, or completed with errors.
[root@solgle.com-1 bin]# 
 
---同理启动solgle.com-2
... ...
CRS-2681: Clean of 'ora.asm' on 'solgle.com-2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'solgle.com-2'
CRS-2676: Start of 'ora.asm' on 'solgle.com-2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'solgle.com-2'
CRS-2676: Start of 'ora.crsd' on 'solgle.com-2' succeeded
CRS-5702: Resource 'ora.crsd' is already running on 'solgle.com-2'
CRS-4000: Command Start failed, or completed with errors.
[root@solgle.com-2 bin]# ./crsctl stat res -t  
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
 
---过后几分钟
[root@solgle.com-2 bin]# ./crs_stat -v -t    
Name           Type           R/RA   F/FT   Target    State     Host        
----------------------------------------------------------------------
ora.DATA1.dg   ora....up.type 0/5    0/     ONLINE    ONLINE    solgle.com-1        
ora.DATA2.dg   ora....up.type 0/5    0/     ONLINE    ONLINE    solgle.com-1        
ora....ER.lsnr ora....er.type 0/5    0/     ONLINE    ONLINE    solgle.com-1        
ora....N1.lsnr ora....er.type 0/5    0/0    ONLINE    ONLINE    solgle.com-1        
ora.asm        ora.asm.type   0/5    0/     ONLINE    ONLINE    solgle.com-1        
ora.cvu        ora.cvu.type   0/5    0/0    ONLINE    ONLINE    solgle.com-1        
ora.gsd        ora.gsd.type   0/5    0/     OFFLINE   OFFLINE               
ora.solgle.db  ora....se.type 0/2    0/1    ONLINE    OFFLINE               
ora....network ora....rk.type 0/5    0/     ONLINE    ONLINE    solgle.com-1        
ora.oc4j       ora.oc4j.type  0/1    0/2    ONLINE    ONLINE    solgle.com-1        
ora.ons        ora.ons.type   0/3    0/     ONLINE    ONLINE    solgle.com-1        
ora....SM1.asm application    0/5    0/0    ONLINE    ONLINE    solgle.com-1        
ora....C1.lsnr application    0/5    0/0    ONLINE    ONLINE    solgle.com-1        
ora.solgle.com-1.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
ora.solgle.com-1.ons   application    0/3    0/0    ONLINE    ONLINE    solgle.com-1        
ora.solgle.com-1.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    solgle.com-1        
ora....SM2.asm application    0/5    0/0    ONLINE    ONLINE    solgle.com-2        
ora....C2.lsnr application    0/5    0/0    ONLINE    ONLINE    solgle.com-2        
ora.solgle.com-2.gsd   application    0/5    0/0    OFFLINE   OFFLINE               
ora.solgle.com-2.ons   application    0/3    0/0    ONLINE    OFFLINE               
ora.solgle.com-2.vip   ora....t1.type 0/0    0/0    ONLINE    ONLINE    solgle.com-2        
ora.scan1.vip  ora....ip.type 0/0    0/0    ONLINE    ONLINE    solgle.com-1 
 
[root@solgle.com-2 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA1.dg
               ONLINE  ONLINE       solgle.com-1                                         
               ONLINE  ONLINE       solgle.com-2                                         
ora.DATA2.dg
               ONLINE  ONLINE       solgle.com-1                                         
               ONLINE  ONLINE       solgle.com-2                                         
ora.LISTENER.lsnr
               ONLINE  ONLINE       solgle.com-1                                         
               ONLINE  ONLINE       solgle.com-2                                         
ora.asm
               ONLINE  ONLINE       solgle.com-1                     Started             
               ONLINE  ONLINE       solgle.com-2                     Started             
ora.gsd
               OFFLINE OFFLINE      solgle.com-1                                         
               OFFLINE OFFLINE      solgle.com-2                                         
ora.net1.network
               ONLINE  ONLINE       solgle.com-1                                         
               ONLINE  ONLINE       solgle.com-2                                         
ora.ons
               ONLINE  ONLINE       solgle.com-1                                         
               ONLINE  ONLINE       solgle.com-2                                         
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.cvu
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.db
      1        ONLINE  OFFLINE                               Instance Shutdown,STARTING             
      2        ONLINE  OFFLINE                               STARTING            
ora.oc4j
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.com-1.vip
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.com-2.vip
      1        ONLINE  ONLINE       solgle.com-2                                         
ora.scan1.vip
      1        ONLINE  ONLINE       solgle.com-1                                         
[root@solgle.com-2 bin]# 
 
----最终实例solgle.com-1没有启动起来
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.cvu
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.db
      1        ONLINE  OFFLINE                               Instance Shutdown   
      2        ONLINE  ONLINE       solgle.com-2                     Open                
ora.oc4j
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.com-1.vip
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.com-2.vip
      1        ONLINE  ONLINE       solgle.com-2                                         
 
 
----单独启动实例solgle.com-1
[grid@solgle.com-1 bin]$ ./srvctl start instance -d solgle -n solgle.com-1
PRCR-1013 : Failed to start resource ora.solgle.db
PRCR-1064 : Failed to start resource ora.solgle.db on node solgle.com-1
CRS-5017: The resource action "ora.solgle.db start" encountered the following error: 
ORA-00205: error in identifying control file, check alert log for more info
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/11.2.0/log/solgle.com-1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
 
CRS-2674: Start of 'ora.solgle.db' on 'solgle.com-1' failed
[grid@solgle.com-1 bin]$ 
 
[grid@solgle.com-1 bin]$ ./srvctl start instance -d solgle -n solgle.com-1
PRCR-1013 : Failed to start resource ora.solgle.db
PRCR-1064 : Failed to start resource ora.solgle.db on node solgle.com-1
CRS-5017: The resource action "ora.solgle.db start" encountered the following error: 
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/11.2.0/log/solgle.com-1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
 
CRS-2674: Start of 'ora.solgle.db' on 'solgle.com-1' failed
[grid@solgle.com-1 bin]$ su - root
[grid@solgle.com-1 bin]$ cat /etc/sysctl.conf  显示都不正常了
 
-------------------一下为解决该问题-----------------------------------------------------------------------------
--重启了linux服务器,并重新挂载了磁盘重复了上面的操作解决了
... ...
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       solgle.com-2                                         
ora.cvu
      1        ONLINE  ONLINE       solgle.com-2                                         
ora.solgle.db
      1        ONLINE  ONLINE       solgle.com-1                     Open                
      2        ONLINE  ONLINE       solgle.com-2                     Open                
ora.oc4j
      1        ONLINE  ONLINE       solgle.com-2                                         
ora.solgle.com-1.vip
      1        ONLINE  ONLINE       solgle.com-1                                         
ora.solgle.com-2.vip
      1        ONLINE  ONLINE       solgle.com-2                                         
ora.scan1.vip
      1        ONLINE  ONLINE       solgle.com-2                                         
[grid@solgle.com-1 bin]$ 
 
 
标签:Oracle集群Asm磁盘阵列断网的恢复 

solgle.com 版权所有,欢迎分享!!!

相关文章
    相关评论
       Copyright © 2013-2020 solgle.com,All rights reserved.[solgle.com] 公安机关备案号:51010802000219
    Email:solgle@solgle.com; weixin:cd1008610000 ICP:蜀ICP备14011070号-1