PRCA-1022 : ACFS file system resource already exists for disk group

I was recently playing around with ACFS and ADVM, trying different things on my test cluster. I had created an ASM volume and mounted it using ASMCA, and then deleted it and tried to re-create the same thing with the command line.

Creating the diskgroup, the volume, all worked fine through the CLI. However when trying to add with srvctl using the following command:

srvctl add filesystem -d /dev/asm/dbhome-326 -g 'DBHOME' -v DBHOME -m /u01/app/oracle/product/11.2.0/dbhome_1 -u oracle
I got the following error:

PRCA-1022 : ACFS file system resource already exists for disk group dbhome and volume orahome

Hm. That's weird. So I tried running: 

srvctl remove filesystem -d /dev/asm/dbhome-326

And get an error saying the filesystem doesn't exist. But the dbhome diskgroup exists, but I couldn't seem to find the link between the two.

So here's the real lesson for the day. I usually run Putty in little rectangular windows, and so when I do a crsctl stat res -t, the output scrolls by and I usually just get the last 10 entries or so, and assume everything above is fine. Except when it's not. Like in this case.

It looks like the ASMCA doesn't clean up everything quite right, as the ACFS resource was still listed in the registry, as evidenced by this entry in the crsctl stat output: (truncated somewhat for readability) 

...

ora.dbhome.dbhome.acfs 

 OFFLINE OFFLINE ol6-112-rac1 volume 

 OFFLINE OFFLINE ol6-112-rac2 volume
So it was just a matter of deleting the resource:

[root@ol6-112-rac1 bin]# ./crsctl delete resource ora.dbhome.dbhome.acfs
And upon deletion, the srvctl add filesystem command run successfully. So: Run your terminal windows longer vertically! (or pay closer attention to the status output of crsctl stat).