file Storage Method Cleanup: Added Files

The file storage method checks the database data/plugins/file for recently modified (added or removed) files.

The normal sequence of events for adding a file to the database is:

1. DB transaction begins.
   1. Create the file--the details are outside of the scope of this section.

   2. Determine the Hash of the incoming file.

   3. fsync() the incoming file (if not done already).

   4. If not already so named, rename the file to its Hash-determined name in the hash cache.

   5. Determine that the file does not already exist in blobs, nor in data/plugins/file. Obtain a write lock in the process of verifying the latter.

   6. Add the file to data/plugins/file, key = Hash, value = `A' (for Add).

   7. Commit the DB transaction.

   If the transaction fails or is aborted, the file is not inserted into the database at all. Note that this DB transaction often involves many related changes to the database.

2. DB transaction begins.
   1. Get a write lock on the recently added file in data/plugins/file.

   2. Copy or link the file to the blobs directory.

   3. Delete the file's entry in data/plugins/file.

   4. Commit the DB transaction.

   If the transaction fails or is aborted, the file is still not inserted into the database, but another insertion attempt will be made when recovery is run in the future.

The recovery procedure consists of repeating the second part of this sequence.

Note that in normal mode, the file's Hash has been `verified' (actually it was empirically determined recently) and in recovery mode, the cache recovery procedure verifies that all files in the cache have correct contents and removes (either deletes or moves into a holding area) those that don't. Thus it is not necessary to do an extra validation step in the second part of this sequence.