/
Developing a Source

Developing a Source

Owned by Shaun Morris
Last updated: May 24, 2013Version comment

Prerequisites

  1. Understand the desired component for development as described in the DDF Catalog section.
    • Review existing implementations in the source code and application documentation.
  2. Have an IDE and the ability to create OSGi bundles.
  3. Understand the Use of the Whiteboard Design Pattern section and how to publish services to the OSGi service registry.

Overview

Sources are components that enable DDF to talk to back-end services. They let DDF do query and ingest operations on catalog stores, and query operations on federated sources. Sources reside in the Sources area of the DDF Overview.

Creating a New Source

Implement a Source Interface

There are three types of sources that can be created. All of these types of sources can perform a Query operation. Operating on queries is the foundation for all sources. All of these sources must also be able to return their availability and the list of content types currently stored in their back-end data stores.

  1. Catalog Provider - ddf.catalog.source.CatalogProvider
    Used to communicate with back-end storage. Allows for Query and Create/Update/Delete operations.
  2. Federated Source - ddf.catalog.source.FederatedSource
    Used to communicate with remote systems. Only allows query operations.
  3. Connected Source - ddf.catalog.source.ConnectedSource
    Similar to a Federated Source with the following exceptions:
    • Queried on all local queries
    • SiteName is hidden (masked with the DDF sourceId) in query results
    • SiteService does not show this Source's information separate from DDF's.

Implementing any of the source types follows a similar format:

  1. Create a new class that implements the specified Source interface.
  2. Implement the required methods.
  3. Create OSGi descriptor file to communicate with the OSGi registry (described in the Working with OSGi  section).
    1. Import DDF packages
    2. Register source class as service to OSGi registry
  4. Deploy to DDF

Catalog Provider

  1. Create java class that implements CatalogProvider

    public class TestCatalogProvider implements ddf.catalog.source.CatalogProvider

  2. Implement the required methods from the ddf.catalog.source.CatalogProvider interface

    public CreateResponse create(CreateRequest createRequest) throws IngestException;

public UpdateResponset update(UpdateRequest updateRequest) throws IngestException;

public DeleteResponse delete(DeleteRequest deleteRequest) throws IngestException;

  3. Import the DDF interface packages to the bundle manifest (in addition to any other required packages):

    Import-Package: ddf.catalog, ddf.catalog.source

  4. Export the service to the OSGi registry:

    Blueprint example
    <service ref="[[TestCatalogProvider]]" interface="ddf.catalog.source.CatalogProvider" />

The DDF Integrator's Guide provides details on the following Catalog Providers that come with DDF out of the box:

A code example of a Catalog Provider delivered with DDF is the Catalog Solr Embedded Provider.

Federated Source

  1. Create Java class that implements FederatedSource

    public class TestFederatedSource implements ddf.catalog.source.FederatedSource

  2. Implement the required methods of the ddf.catalog.source.FederatedSource interface.


     

  3. Import the DDF interface packages to the bundle manifest (in addition to any other required packages):

    Import-Package: ddf.catalog, ddf.catalog.source

  4. Export the service to the OSGi registry:

    Blueprint example
    <service ref="[[TestFederatedSource]]" interface="ddf.catalog.source.FederatedSource" />

The DDF Integrator's Guide provides details on the following Federated Sources that come with DDF out of the box:

A code example of a Federated Source delivered with DDF can be found in ddf.catalog.source.solr

Connected Source

  1. Create Java class that implements ConnectedSource

    public class TestConnectedSource implements ddf.catalog.source.ConnectedSource

  2. Implement the required methods of the ddf.catalog.source.ConnectedSource interface.


     

  3. Import the DDF interface packages to the bundle manifest (in addition to any other required packages):

    Import-Package: ddf.catalog, ddf.catalog.source

  4. Export the service to the OSGi registry:

    Blueprint example
    <service ref="[[TestConnectedSource]]" interface="ddf.catalog.source.ConnectedSource" />

Please Note

In some Providers that are created, there is a need to make Web Service calls through JAXB clients. It is best NOT to create your JAXB client as a global variable. We have seen intermittent failures with the creation of Providers and federated sources when clients are created in this manner. Create your JAXB clients every single time within the methods that require it in order to avoid this issue.

Exception Handling

In general, sources should only send information back related to the call, not implementation details.

Examples:
  • "Site XYZ not found" message rather than the full stack trace with the original site not found exception.
  • The caller issues a malformed search request. Return an error describing the right form, or specifically what was not recognized in the request. Do not return the exception and stack trace where the parsing broke.
  • The caller leaves something out. Do not return the null pointer exception with a stack trace, rather return a generic exception with the message "xyz was missing".
External Resources

Three Rules for Effective Exception Handling