Metrics Collecting Application
- Shaun Morris
- Hugh Rodgers (Unlicensed)
Description
The Metrics Collecting Application collects data for all of the pre-configured metrics in DDF and stores them in custom JMX Management Bean (MBean) attributes. Samples of each metric's data is collected every 60 seconds and stored in the <DDF_INSTALL_DIR>/data/metrics directory, with each metric stored in its own .rrd file. Refer to the Metrics Reporting Application for how the stored metrics data can be viewed.
Do not remove the <DDF_INSTALL_DIR>/data/metrics directory or any files in it. If this is done, then all existing metrics data will be permanently lost.
Also note that if DDF is uninstalled/re-installed that all existing metrics data will be permanently lost.
The metrics currently being collected by DDF are:
| Metric | JMX MBean Name | MBean Attribute Name | Description |
|---|---|---|---|
| Catalog Exceptions | ddf.metrics.catalog:name=Exceptions | Count | A count of the total number of exceptions, of all types, thrown across all catalog queries executed. |
| Catalog Exceptions Federation | ddf.metrics.catalog:name=Exceptions.Federation | Count | A count of the total number of Federation exceptions thrown across all catalog queries executed. |
| Catalog Exceptions Source Unavailable | ddf.metrics.catalog:name=Exceptions.SourceUnavailable | Count | A count of the total number of SourceUnavailable exceptions thrown across all catalog queries executed. These exceptions occur when the source being queried is currently not available. |
| Catalog Exceptions Unsupported Query | ddf.metrics.catalog:name=Exceptions.UnsupportedQuery | Count | A count of the total number of UnsupportedQuery exceptions thrown across all catalog queries executed. These exceptions occur when the query being executed is not supported or is invalid. |
| Catalog Ingest Created | ddf.metrics.catalog:name=Ingest.Created | Count | A count of the number of catalog entries created in the Metadata Catalog. |
| Catalog Ingest Deleted | ddf.metrics.catalog:name=Ingest.Deleted | Count | A count of the number of catalog entries updated in the Metadata Catalog. |
| Catalog Ingest Updated | ddf.metrics.catalog:name=Ingest.Updated | Count | A count of the number of catalog entries deleted from the Metadata Catalog. |
| Catalog Queries | ddf.metrics.catalog:name=Queries | Count | A count of the number of queries attempted. |
| Catalog Queries Comparison | ddf.metrics.catalog:name=Queries.Comparison | Count | A count of the number of queries attempted that included a string comparison criteria as part of the search criteria, e.g., PropertyIsLike, PropertyIsEqualTo, etc. |
| Catalog Queries Federated | ddf.metrics.catalog:name=Queries.Federated | Count | A count of the number of federated queries attempted. |
| Catalog Queries Fuzzy | ddf.metrics.catalog:name=Queries.Fuzzy | Count | A count of the number of queries attempted that included a string comparison criteria with fuzzy searching enabled as part of the search criteria. |
| Catalog Queries Spatial | ddf.metrics.catalog:name=Queries.Spatial | Count | A count of the number of queries attempted that included a spatial criteria as part of the search criteria. |
| Catalog Queries Temporal | ddf.metrics.catalog:name=Queries.Temporal | Count | A count of the number of queries attempted that included a temporal criteria as part of the search criteria. |
| Catalog Queries Total Results | ddf.metrics.catalog:name=Queries.TotalResults | Mean | An average of the total number of results returned from executed queries. This total results data is averaged over the metric's sample rate. |
| Catalog Queries Xpath | ddf.metrics.catalog:name=Queries.Xpath | Count | A count of the number of queries attempted that included a Xpath criteria as part of the search criteria. |
| Catalog Resource Retrieval | ddf.metrics.catalog:name=Resource | Count | A count of the number of products retrieved. |
| Services Latency | ddf.metrics.services:name=Latency | Mean | The response time from receipt of the request at the endpoint until the response is about to be sent to the client from the endpoint. This response time data is averaged over the metric's sample rate. |
Source Metrics
Metrics are also collected on a per Source basis for each configured Federated Source and Catalog Provider. When the Source is configured, the metrics listed in the table below are automatically created. With each request that is either an enterprise query or a query that lists the source(s) to query, these metrics are collected. When the Source is deleted, the associated metrics' MBeans and Collectors are also deleted. However, the RRD file in the data/metrics directory containing the collected metrics remain indefinitely and remain accessible from the Metrics tab in the Admin console.
In the table below, the metric name is based on the Source's ID (indicated by <sourceId>).
| Metric | JMX MBean Name | MBean Attribute Name | Description |
|---|---|---|---|
| Source <sourceId> Exceptions | ddf.metrics.catalog.source:name=<sourceId>.Exceptions | Count | A count of the total number of exceptions, of all types, thrown from catalog queries executed on this Source. |
| Source <sourceId> Queries | ddf.metrics.catalog.source:name=<sourceId>.Queries | Count | A count of the number of queries attempted on this Source. |
| Source <sourceId> Queries Total Results | ddf.metrics.catalog.source:name=<sourceId>.Queries.TotalResults | Mean | An average of the total number of results returned from executed queries on this Source. This total results data is averaged over the metric's sample rate. |
Usage
The Metrics Collecting Application is used when collection of historical metrics data, such as catalog query metrics or message latency type of data, is desired.
Installation and Uninstallation
The Metrics Collecting Application is installed by default. It should not be uninstalled because the JMX Collector will not be removed. The next time the system is restarted, the JMX Collector will try to find the JMX MBean but it won't be created since the metrics bundle was uninstalled. It will timeout after a few minutes.
Configuration
No configuration is made for the Metrics Collecting Application. All of the metrics that it collects data on are either pre-configured in DDF out of the box, or dynamically created
as Sources are created or deleted.
Known Issues
- None