Updates to Support Async Processing

PostProcess Plugin Forwarder

• Create PostProcessPlugin that implements PostCreateStoragePlugin and PostUpdateStoragePlugin. The PostProcessPlugin will lookup a service implementation for PostProcessService implementation.

PostProcess Service Implementation (JMS)

• This plugin will process the ByteSource data and save it in a local/shared filesystem as a temporary file (to allow async post processing of data not permanently stored by DDF.
• JMS Queue Listener takes request message off of the Queue and calls the PostProcessPluginProcessor for processing the request.

PostProcess Service Implementation (Default)

• Default implementation that will asynchronously process items through a thread pool configured within the container.

PostProcess Plugin Processor

• In both instances, the PostProcess Plugin Processor sends each PostProcess Plugin the request for processing similar to how the CreateOperations and UpdateOperations processing is currently done.
• Once the processing is complete, the updated Metacard will be passed back to the PostProcess Service updateMetacard() method. This method will then forward the metacard to update over to the OSGI event queue for internal updates. In the case of JMS the response is posted to a JMS update queue that has a JMS listener on the instance running the catalog framework. The Catalog Framework is a subscriber to the OSGI event queue and process the updates without running PostXXXPlugins to avoid data looping.
• Store Updated Metacard - Last update wins for a metacard.

Parallel processing of simultaneous conflicting updates 

1. Should this design account for multiple events on the same piece of data being processed in order. For instance if we receive and update for Product A and while that update is processing a delete event comes through. If the delete is processed out of order from the update what should happen? Potential solutions could include a persistent Processing Manager to help coordinate requests coming in, but it would also add complexity to the design
   1. To support user updates that can occur during long-running PostProcess plugins where the metacard has changed since it was handed off to the PostProcess plugins, partial updates of plugin changed data need to be supported.  This processing will involve saving an initial metacard and comparing with the resulting metacard to determine the differences generated by the PostProcess plugins. Once the differences are determined, the latest version of the metacard will be fetched and the differences applied before submission back to the Catalog Framework as a metacard update.

Decision

Created an asynchronous processing framework prototype.

https://github.com/codice/ddf/tree/2.22.x/catalog/async

Due to lack of interest, removed in DDF 2.23.