We've all been thankful that Windows makes it a little hard to eliminate a file on a hard drive. There are four different actions usually necessary: deleting the file in the first place, confirming the deletion in a dialog box, emptying the Recycle Bin (or deleting the file from the Recycle Bin), and then confirming that decision in a dialog box. This process can occasionally be a lifesaver, it can also be tedious (particularly if you are trying to get rid of sensitive files that you don't want hanging around in the Recycle Bin).
* If you let the Recycle Bin get full, however, the oldest recycled files will be lost when new ones are recycled.
NOTE: Even deleting a file from the Recycle Bin doesn't destroy the information right away. Windows makes the file's disk space available for reassignment, but doesn't immediately write over that disk space. People with the proper tools could still read the file. To prevent this, you need file-deletion software that is not part of Windows.