Ingest FMV Metadata

Owned by Joseph North
Dec 18, 2015
4 min read

This page concerns the ingestion of FMV metadata into DDF. Specifically, the scope is currently limited to extracting Unmanned Air System (UAS) Local Data Set (LDS) KLV metadata from STANAG 4609-compliant MPEG-2 transport streams. 

Parsing the MPEG-2 Transport Stream

We had success with the mpegts-streamer library (license: Apache, v2.0), a pure Java solution that provides the ability to perform packet-by-packet processing of MPEG-2 transport streams. It provides data structures that interpret the packet headers for you, so you don’t have to read bits and bytes manually. It’s also easy to use and processes transport streams quickly. This is the only open source, pure Java library we could find for processing MPEG-2 transport streams, but we recommend its use nonetheless.

Detecting MPEG-2 Transport Streams

To get any metadata from a transport stream, we have to know when we are dealing with one. Detecting a transport stream is somewhat difficult, since a transport stream is just a sequence of packets. The jcodec library (license: FreeBSD) provides a format detector that is supposedly capable of detecting MPEG program streams and transport streams. We tried it on a few sample MPEG transport streams, and while it correctly detected some of them, it was unable to detect the ones that we retrieved from this site. These samples were obtained from TV broadcasts, and the site notes that the transport streams weren’t deliberately cut on packet boundaries, so there may be partial packets. We weren’t able to find any other Java libraries that provide transport stream detection, so we may have to modify this one, develop our own with the help of the mpegts-streamer library, or investigate other options.

Unpacking the KLV                 

The mpegts-streamer library makes it easy to grab the payload associated with a metadata packet, but the payload is still just an array of bytes that contains KLV data. The STANAG 4609 standard specifies how the KLV is structured. The top-level LDS KLV is identified by a 16-byte Universal Key, and its value portion contains the individual LDS metadata KLVs (i.e., the actual metadata). The specification says that the nested LDS KLVs do not have 16-byte keys. Instead, they have BER-encoded (i.e. variable-length) keys that map to Universal Keys (this is done to avoid wasting bandwidth on keys). There’s a KLV library called KLV.java (license: public domain), but it doesn’t support BER-encoded keys. However, we could easily modify it to support them.

Finding the Right Data

A transport stream can contain one or more programs (multimedia streams). Specific tables (the Program Association Table and the Program Map Tables) in the transport stream contain the information necessary to tell which packets in the transport stream correspond to which elementary stream (e.g. a video stream, an audio stream, or a KLV stream) and to which program. These tables are transmitted in packets inside the transport stream along with all the elementary stream packets, and they don’t appear at any specific locations in the stream. Depending on how the transport stream was cut, there may be elementary stream packets that appear before the PAT and PMTs. But when those packets are first encountered, we can’t know what they are since we don’t have the information from the PAT and PMTs. Therefore, it is necessary to make an initial pass over the transport stream and find the PAT and PMTs so we know what packets in the transport stream to look for. But it’s not necessary to analyze the entire transport stream. The PAT tells you what programs are in the transport stream, so once you find the PMTs that correspond to them, you can stop looking. At this point, you have all the information about which packets contain the data for the programs’ elementary streams. 

Once you have this information, you can go back through the transport stream and extract KLV metadata from any of the programs that have KLV elementary streams. Since a transport stream can have more than one program, it may make sense to create a metacard for each program. This would be relatively easy to implement as long as we store the program’s identifier in the metacard.

Extracting and Interpreting KLV

MISB ST 0601.8 (the UAS LDS standard) lists 95 KLV metadata elements, some of which STANAG 4609 says are mandatory. It seems reasonable to provide mappings for these 95 keys by default, such that when they are encountered in the KLV, their respective values are extracted (other keys would just be ignored). These mappings would include the key names in addition to information about how each value should be interpreted (since each raw value is just a sequence of bytes). For example, in ST 0601.8, floating point values are encoded in the KLV as integers. We have to convert the byte sequence to an integer and then convert that to a floating point number using a formula that maps it to the appropriate range.

In our prototype, we implemented these mappings with a static Map in memory. We specified all the KLV keys, names, and interpretation information directly in the code, but it would probably be preferable to retrieve them from a configuration file.

A New Input Transformer

It probably makes sense to create a new input transformer that is only responsible for handling video files. Upon receiving the input file, it would attempt to determine the format. If the file is a transport stream, then it could be handled in the manner described in the preceding sections.

Candidate Tickets for Implementing This Capability

  • Create a new input transformer responsible for handling video files (perhaps just using Tika for metadata extraction initially).
  • Update the new video input transformer to detect MPEG transport streams and parse any KLV according to the STANAG 4609 and MISB ST 0601.8 standards.
    • Initially, this would probably just involve extracting all the metadata and writing it out to the metacard’s metadata field as XML.
    • Perhaps this could be limited to just one program in the transport stream, with a later ticket expanding it to all programs.
  • Update the new video input transformer to map MISB ST 0601.8 KLV elements to basic metacard attributes where possible.
  • Update the new video input transformer to generate a thumbnail for an MPEG transport stream.