This article demonstrates how ASM behaves when adding disks, removing disks, and running out of free space.

ASM version 12.1.0.2.190716
Configuration 1: A disk group with two disks and Normal redundancy.

Creating a group with Normal redundancy:

SQL> CREATE DISKGROUP testkks1 normal REDUNDANCY DISK 'ORCL:DISK1','ORCL:DISK2';
Diskgroup created.

Checking the result:

https://docs.oracle.com/database/121/OSTMG/GUID-E775902B-6461-4E61-BC43-231D1701FBDA.htm#OSTMG94549

• Total_MB — total raw space capacity.
• Free_MB — available raw space.
• Usable_file_MB — available (protected) space considering redundancy.
• Req_mir_free_MB — the amount of space required to restore full redundancy after the most severe failure that the disk group can tolerate.

1.1 Adding a disk to the group:

SQL> ALTER DISKGROUP testkks1 ADD DISK 'ORCL:DISK3';
Diskgroup altered.

Checking the result:

What do we see here:

• Total_MB — increased by the size of one disk.
• Free_MB — increased by the size of one disk.
• Req_mir_free_MB — increased by the size of one disk.
• Usable_file_MB — remained unchanged at 5042.

1.2 Removing a disk:

We’re back to where we started.

1.3 Adding two disks to the disk group:

SQL> ALTER DISKGROUP testkks1 ADD DISK 'ORCL:DISK3','ORCL:DISK4' REBALANCE POWER 9 WAIT;
Diskgroup altered.

Checking the result:

What do we see here:

• Total_MB — increased by the size of two disks.
• Free_MB — increased by the size of two disks.
• Req_mir_free_MB — equals the size of one disk.
• Usable_file_MB — changed, increasing by half the size of one disk.

Adding one more disk:

SQL> ALTER DISKGROUP testkks1 ADD DISK 'ORCL:DISK5' REBALANCE POWER 9 WAIT;	
Diskgroup altered.

• Total_MB — increased by the size of one disk.
• Free_MB — increased by the size of one disk.
• Req_mir_free_MB — equals the size of one disk (unchanged).
• Usable_file_MB — changed, increasing by half the size of one disk.

Now, let’s conclude that:

• Total_MB — represents the total amount of raw space.
• Free_MB — represents the total amount of free raw space.
• Req_mir_free_MB — equals the size of one disk if Normal redundancy is specified, the number of disks is three or more, and no Failgroup is explicitly assigned for each disk during disk group creation.

From this, it follows that with Normal redundancy and three or more disks, Usable_file_MB will be equal to:

Usable_file_MB=(Free_MB — Req_mir_free_MB)/2

For 5 disks, with Normal redundancy:

Usable_file_MB=(25438-5119)/2= 10 159

For 4 disks, with Normal redundancy:

Usable_file_MB=(20321-5119)/2=7 601

For 3 disks, with Normal redundancy:

Usable_file_MB=(15204-5119)/2= 5 042

For 2 disks, with Normal redundancy:

 Usable_file_MB=(10136-0)/2= 5 068

The edge case for 3 disks shows that, starting from this number of disks, ASM begins reserving 1 disk to restore full redundancy after the most severe failure that can be tolerated by the disk group.

Configuration 2: A disk group with two disks, Normal redundancy, and its filling.

1.1 Initial state:

Filling the disk group:

SQL> create tablespace kkstbs1 datafile '+TESTKKS1' size 5050M autoextend on next 10m;
Tablespace created.

Result:

Expanding the data file by another 10MB:

SQL> alter database datafile '+TESTKKS1/kksdb/DATAFILE/KKSTBS1.256.1046190953' resize 5060M;
Database altered.

Result:

Expanding by another 10MB:

SQL> alter database datafile '+TESTKKS1/kksdb/DATAFILE/KKSTBS1.256.1046190953' resize 5070M;
alter database datafile '+TESTKKS1/kksdb/DATAFILE/KKSTBS1.256.1046190953' resize 5070M
*
ERROR at line 1:
ORA-01237: cannot extend datafile 5
ORA-01110: data file 5: '+TESTKKS1/KKSDB/DATAFILE/kkstbs1.256.1046190953'
ORA-17505: ksfdrsz:1 Failed to resize file to size 648960 blocks
ORA-15041: diskgroup "TESTKKS1" space exhausted

Not so fast.

1.2 Adding a disk:

SQL> ALTER DISKGROUP testkks1 ADD DISK 'ORCL:DISK3' REBALANCE POWER 9 WAIT;
Diskgroup altered.

At this stage, we conclude that Usable_file_MB went negative due to one disk being reserved for restoring full redundancy after the most severe failure that can be tolerated by the disk group. Otherwise, we would encounter ORA-15041, which indicates that the disk group has run out of space.

Additionally, we note that data started being written to this disk long before Usable_file_MB became negative.

Obviously, further filling of the disk group is pointless since, at this stage, the maximum negative value we can reach is Usable_file_MB = -5119.

REQUIRED_MIRROR_FREE_MB indicates the amount of space that must be available in a disk group to restore full redundancy after the worst failure that can be tolerated by the disk group without adding additional storage. This requirement ensures that there are sufficient failure groups to restore redundancy. Also, this worst failure refers to a permanent failure where the disks must be dropped, not the case where the disks go offline and then back online.

According to this description, in the current situation (Usable_file_MB is negative, and the so-called «reserved» disk already contains more data than it should), if we suddenly lose one disk, something very interesting will happen.

Checking the scenario: Losing a disk.

SQL> alter diskgroup testkks1 set attribute 'DISK_REPAIR_TIME'='10m';
Diskgroup altered.

— Here, I changed the device type from NVMe to SCSI (which changed the Failgroup names) because I couldn’t find a proper way to disconnect an NVMe disk on a virtual machine. 😆 —

[root@kks grid12]# ./bin/kfed read /dev/sdf1 | grep dskname
kfdhdb.dskname:                   DISK1 ; 0x028: length=5

Removing the disk:

[root@kks grid12]# echo 1 > /sys/block/sdf/device/delete

Alert log from database:

• NOTE: updating disk modes to 0x5 from 0x7 for disk 3 (DISK1) in group 2 (TESTKKS1): lflags 0x0   
• NOTE: disk 3 (DISK1) in group 2 (TESTKKS1) is offline for reads
• NOTE: updating disk modes to 0x1 from 0x5 for disk 3 (DISK1) in group 2 (TESTKKS1): lflags 0x0   
• NOTE: disk 3 (DISK1) in group 2 (TESTKKS1) is offline for writes

Rebalance has not started yet; less than 10 minutes have passed since the disk was lost:

Let’s try expanding the data file by another 1024MB:

SQL> alter database datafile '+TESTKKS1/kksdb/DATAFILE/KKSTBS1.256.1046190953' resize 7098M;
Database altered.

Here are two peculiarities:

1. Strangeness #1: Even though the lost disk «disk 3 (DISK1) in group 2 (TESTKKS1) is offline for writes,» its Free_MB decreased from 1012MB to 332MB.
2. Strangeness #2: The maximum possible Usable_file_MB in this configuration is 5042MB, yet the data file size at the moment of disk removal was 6074MB.

After the DISK_REPAIR_TIME = '10m' elapsed, the disk was successfully removed:

SUCCESS: alter diskgroup TESTKKS1 drop disk DISK1 force /* ASM SERVER */

However, the disk remained, renamed as _DROPPED_0003_TESTKKS1.

Attempts to rebalance or add a 10MB file were unsuccessful.

ORA-15041: diskgroup "TESTKKS1" space exhausted
WARNING: Rebalance encountered ORA-15041; continuing


SQL> alter tablespace kkstbs1 add datafile '+TESTKKS1' size 10m;                    
alter tablespace kkstbs1 add datafile '+TESTKKS1' size 10m
*
ERROR at line 1:
ORA-01119: error in creating database file '+TESTKKS1'
ORA-17502: ksfdcre:4 Failed to create file +TESTKKS1
ORA-15041: diskgroup "TESTKKS1" space exhausted

Summary of the Current Situation:

• A disk group with Normal redundancy consisting of three disks has effectively turned into a two-disk group.
• With two or three disks, the Usable_file_MB of the disk group is 5,042MB.
• When a disk is lost and Usable_file_MB goes negative, redundancy is lost for the data that caused the negative value. Currently, the tablespace size is 7,098MB, exceeding the usable limit.
• If one more disk fails, it will lead to either tablespace loss or complete disk group failure.

Checking the scenario:

[root@kks grid12]# ./bin/kfed read /dev/sdi1 | grep dskname
kfdhdb.dskname:                   DISK2 ; 0x028: length=5

[root@kks grid12]# echo 1 > /sys/block/sdi/device/delete

ASM alertlog:

• WARNING: Write Failed. group:2 disk:0 AU:1 offset:1044480 size:4096
• path:ORCL:DISK2
• SQL> alter diskgroup TESTKKS1 dismount force /* ASM SERVER:816479400 */
• SUCCESS: diskgroup TESTKKS1 was dismounted
• SUCCESS: alter diskgroup TESTKKS1 dismount force /* ASM SERVER:816479400 */
• SUCCESS: ASM-initiated MANDATORY DISMOUNT of group TESTKKS1

ASMCMD> lsdg -g testkks1
ASMCMD-8001: diskgroup ‘testkks1’ does not exist or is not mounted

The disk group is lost.