Description

The Content Directory Monitor allows files placed in a monitored directory to be ingested into the DDF Content Repository and/or the Metadata Catalog (MDC). A monitored directory is a directory configured to be polled by DDF periodically (typically every one second) for any new files added to the directory that should be ingested into the Content Framework.

The typical execution flow of the Directory Monitor is:

1. A new file is detected in the monitored directory, 
2. The file's contents are passed on to the Content Framework and processed based on whether the monitored directory's processing directive was:
   1. Configured to just store the file in the DDF Content Repository,
   2. Configured to just process the file's metadata and ingest it into the MDC, or 
   3. Configured to both store the file in the Content Repository and ingest it into the MDC.
3. If the response from the Content Framework is successful, indicating the content was stored and/or processed, then the file in the monitored directory is either deleted (default behavior), or copied to a sub-directory called .ingested (see below for how to configure this behavior). If the response from the Content Framework was unsuccessful or a failure occurred, then the file is moved from the monitored directory to a sub-folder named .errors, allowing easy identification of the ingested files that had problems.

Multiple monitored directories can be configured, each monitoring different directories.

Usage

The Content Directory Monitor provides the capability to easily create content in the DDF Content Repository and Metacards in the MDC by simply placing a file in a directory that has been configured to be monitored by DDF. For example, this would be useful for copying files from a hard drive (or directory) in a batch-like operation to the monitored directory and having all of the files processed by the Content Framework.

Sample Usage Scenarios

Scenario 1: Monitor single directory for storage and processing, with no file backup

• The Content Directory Monitor is configured with:
  • the relative path of inbox for the Directory Path, 
  • Processing Directive set to Store and Process, and 
  • the Copy Ingested Files is not checked
• As files are placed in the monitored directory <DDF_INSTALL_DIR>/inbox, the files are ingested into the Content Framework.
  • The Content Framework generates a GUID for the create request for this ingested file
  • Since the Store and Process directive was configured the ingested file is passed on to the Content File System Storage Provider which:
    • creates a sub-directory in the Content Repository using the GUID, and 
    • places the ingested file into this GUID sub-directory using the file name provided in the request
  • The Content Framework then invokes the Catalog Content Plugin which:
    • looks up the Input Transformer associated with the ingested file's mime type
    • this Input Transformer creates a Metacard based on the ingested file's contents
    • invokes the Catalog Framework, which inserts the metacard into the MDC
  • Content Framework sends back a successful status to the Camel route that was monitoring the directory
  • Camel route completes and deletes the file from the monitored directory

Scenario 2: Monitor single directory for storage with file backup

• The Content Directory Monitor is configured with:
  • the absolute path of /usr/my/home/dir/inbox for the Directory Path, 
  • Processing Directive set to Store only, and 
  • the Copy Ingested Files is checked
• As files are placed in the monitored directory /usr/my/home/dir/inbox, the files are ingested into the Content Framework.
  • The Content Framework generates a GUID for the create request for this ingested file
  • Since the Store directive was configured the ingested file is passed on to the Content File System Storage Provider which:
    • creates a sub-directory in the Content Repository using the GUID, and 
    • places the ingested file into this GUID sub-directory using the file name provided in the request
  • Content Framework sends back a successful status to the Camel route that was monitoring the directory
  • Camel route completes and moves the file from the monitored directory to its sub-directory /usr/my/home/dir/inbox/.ingested

Scenario 3: Monitor multiple directories for processing only, with file backup - errors encountered on some ingests

• Two different Content Directory Monitors are configured with:
  • the relative path of inbox and inbox2 for the Directory Path, 
  • Processing Directive on both directory monitors set to Process, and 
  • the Copy Ingested Files is checked for both directory monitors
• As files are placed in the monitored directory <DDF_INSTALL_DIR>/inbox, the files are ingested into the Content Framework.
  • The Content Framework generates a GUID for the create request for this ingested file
  • Since the Process directive was configured the ingested file is passed on to the  Catalog Content Plugin which:
    • looks up the Input Transformer associated with the ingested file's mime type, but no Input Transformer is found
    • an exception is thrown
  • Content Framework sends back a failure status to the Camel route that was monitoring the directory
  • Camel route completes and moves the file from the monitored directory to the .errors sub-directory
• As files are placed in the monitored directory <DDF_INSTALL_DIR>/inbox2, the files are ingested into the Content Framework.
  • The Content Framework generates a GUID for the create request for this ingested file
  • The Content Framework then invokes the Catalog Content Plugin which:
    • looks up the Input Transformer associated with the ingested file's mime type
    • this Input Transformer creates a Metacard based on the ingested file's contents
    • invokes the Catalog Framework, which inserts the metacard into the MDC
  • Content Framework sends back a successful status to the Camel route that was monitoring the directory
  • Camel route completes and  moves the file from the monitored directory to its .ingested sub-directory

Installation and Uninstallation

The Content Directory Monitor is packaged as the content-core-directorymonitor feature and is part of the content-core-app. It is installed by default.

It can be installed and uninstalled using the normal processes described in the Configuration section of the Administrator's Guide.

Note that the content-core-catalogerplugin feature must be installed for the metacards to be created and inserted into the MDC. This feature provides the linkage between the Content Framework and the Catalog Framework. If the client attempts a STORE_AND_PROCESS or a PROCESS only without this feature installed a failure will be returned.

Configuration

This component can be configured using the normal processes described in the Administrator's Guide's Configuration section.

The configurable properties for the Content Directory Monitor are accessed from the Content Directory Monitor Configuration in the Admin Console.

Configuring Content Directory Monitors

Managed Service Factory PID

ddf.content.core.directorymonitor.ContentDirectoryMonitor

Configurable Properties

Known Issues

None