Unable to delete files with full disk quota on ZFS

Problem

When I was removing old build logs and source from my ZFS file serever to create for fresh builds, i tried removing old bits.

bash# rm gmake-optimize-domestic.log.1

rm: cannot remove file `gmake-optimize-domestic.log.1': Disk quota exceeded

Solution

You will need to copy /dev/null into file that is taking up space.

Example:
bash# ls -la

drwxr-xr-x   2 svbld    staff          8 Sep 26  2010 ./
drwxr-xr-x   3 svbld    staff          3 Sep 24  2010 ../
-rw-r--r--   1 svbld    staff       2707 Sep 26  2010 20100924.1.rep.1
-rw-r--r--   1 svbld    staff    1748129 Sep 24  2010 cvs-get.log.1
-rw-r--r--   1 svbld    staff        388 Sep 26  2010 email-mailx.log.1
-rw-r--r--   1 svbld    staff    3593895 Sep 26  2010 gmake-optimize-domestic.log.1
-rw-r--r--   1 svbld    staff      40709 Sep 24  2010 rt.log.1
-rw-r--r--   1 svbld    staff      43369 Sep 24  2010 sour.conf

bash# cp /dev/null gmake-optimize-domestic.log.1
bash#ls -la
total 10943
drwxr-xr-x   2 svbld    staff          8 Sep 26  2010 ./
drwxr-xr-x   3 svbld    staff          3 Sep 24  2010 ../
-rw-r--r--   1 svbld    staff       2707 Sep 26  2010 20100924.1.rep.1
-rw-r--r--   1 svbld    staff    1748129 Sep 24  2010 cvs-get.log.1
-rw-r--r--   1 svbld    staff        388 Sep 26  2010 email-mailx.log.1
-rw-r--r--   1 svbld    staff          0 Nov 22  2011 gmake-optimize-domestic.log.1
-rw-r--r--   1 svbld    staff      40709 Sep 24  2010 rt.log.1
-rw-r--r--   1 svbld    staff      43369 Sep 24  2010 sour.conf

bash# rm gmake-optimize-domestic.log.1

bash# ls -lah gmake-optimize-domestic.log.1
/bin/ls: gmake-optimize-domestic.log.1: No such file or directory

As you can see, the file is 3593895 KBs in size, I then make the file zero bytes, and then I'm able to remove it. Once enough files have been removed in this manner, you should be able to use the rm command again.

What causes this?

This is due to how ZFS functions. ZFS is a Copy On Write filesystem, so a file deletion actually takes slightly more space on disk before a file is actually deleted, as it writes the metadata involved with the file deletion before it removes the allocation for the file being deleted. This is how ZFS is able to always be consistent on disk, even in the event of a crash.