When you unset the reference, you just break the binding between variable name and variable content. This does not mean that variable content will be destroyed. For example:
<?php $a = 1; $b =& $a; unset($a); ?>
won't unset $b, just $a.
Again, it might be useful to think about this as analogous to the Unix unlink call.
PHP General