Windows Compatibility

Java and the JVM is cross-platform technology, but far too often important coding practices that go ignored can effectively negate this benefit. This article will list some best-practices to be observed when programming in Java and working with the native file system to preserve the flexibility to build and run DDF on UNIX and Windows systems.

 

 

 

Preface

This guide is to serve as a single source of truth for writing Windows-compliant Java code, including but not limited to production implementations, testing logic, and testing resources. Please keep this page up to date as new issues that are not documented here occur while working tickets. The end goal is to maintain a code-base that can be used, iterated on, built, and tested on various Windows environments. It is not sufficient to just support the end-product and end-users on Windows, but also any developers that are using Windows


 

Getting Started

In order to debug Windows issues, an environment will be needed. Refer to our vagrant repo for setup instructions. Some notes to consider: 

  • Sharing files between the VM and host environment will slow things down unbearably (especially if it's maven related)
  • It helps to provide the VM with access to 100% of system resources and choose to only develop on the VM or host (not both) at any given time
  • Multithreaded builds do not work on Windows (see bottom of page)
  • If needed, the host system should be able to handle two concurrent builds: one single threaded build locally and one single threaded build on the VM
    • Beyond that, browsers become difficult to use

 

File Paths: The Primary Cause for Cross-Platform Failure

The key issue is that most built-in Java methods for resolving paths and URLs are very well behaved on UNIX but do not perform precisely as expected on Windows. The prime example is Paths.get which will succeed on a Mac and happily throw an exception on a PC. It is important to determine, when processing a string that represents a system resource, at which stage of processing it is safe to pass to any appropriate Java method within the relevant classes. Our core production code is rarely the main problem.  

The biggest offenders here are unit tests. When hard-coding path strings to be used for testing, make sure all literals are passed to Paths.get upon initialization and not in the test method itself. Also, when loading resources, there are several quirks to be aware of that only occur on Windows. 

Examples of Valid and Invalid File Paths

File Path conventions differ based on operating system. The following shows key examples of the different rule-sets between Windows and UNIX systems:

Valid Windows Path: 
C:\Documents and Settings\Christopher\My Documents\My Pictures\funny.jpg
Valid Unix Path: 
/var/home/Christopher/documents/pictures/funny.jpg
Most Common Issue: 

The following specific example is a major offender. When passed to Paths.get it will always yield the error message that follows. 

\C:\Sample\sample.txt
java.nio.file.InvalidPathException: Illegal char <:> at index 2: \C:\Sample\sample.txt

The problem is the leading \ in the path which is often appended when fetching a URL to a Java resource. The most straight-forward way to fix it is to plug it into the File object constructor and extract a path from that. 

 

Examples of Valid and Invalid URIs

Universal Resource Identifiers carry strict conventions that can often throw an error during a call to Paths.get. While UNIX is very forgiving, Windows is not: 

Invalid Windows URI Examples:
  1. file://D:Program FilesViewerstartup.html
  2. C:Program FilesMusicWeb Sysmain.html?REQUEST=RADIO
  3. file:////applib/products/a%2Db/ abc%5F9/4148.920a/media/start.swf
Corresponding Correct Windows URI Examples: 
  1. file:///D:/Program%20Files/Viewer/startup.htm
  2. file:///C:/Program%20Files/Music/Web%20Sys/main.html?REQUEST=RADIO
  3. file://applib/products/a-b/abc_9/4148.920a/media/start.swf

These illustrate many more concepts than will be addressed here. It suffices to just note the differences between the paths and the URIs, particularly the count of leading slashes. 


 

Resource Coding Practices: How to Avoid File Pitfalls

The basics begin with clear declaration of the file system variables at the top of the file; especially for static final fields in unit tests. If the field can be stored as a Path instead of a String, all the better. Keeping state as a Path offers more options as test classes are extended. 

Best Case Scenario: What you should be doing 99% of the time

Whenever possible, do not handle or manipulate paths with regard to test resources. Simply use the class or class loader to return the resource as a stream and consume it into a String or any preferred data type that is needed. 

Get the Resource as a Stream

Resource loading should occur as follows whenever possible: 

Get a Resource as a Stream
InputStream resourceStream = getClass().getClassLoader().getResourceAsStream("resourceName");
String resourceData = IOUtils.toString(resourceStream, "UTF-8");

Also acceptable is to work with the URL: 

Get a Resource as a Stream
URL resourceUrl = getClass().getClassLoader().getResource("resourceName");
InputStream resourceStream = resourceUrl.openStream();
String resourceData = IOUtils.toString(resourceStream, "UTF-8");

Do not rely on path manipulation. It eliminates the problem entirely. 

Also, avoid the following style if possible. It shows very inconsistent behavior on Windows due to our OSGi environment depending on the module or bundle the code is a part of. 

Don't do this
 getClass().getResourceAsStream("/resourceName");

Using java.io.File

If you absolutely need to sanitize a bad URL that is returned from getResource() then the constructor of java.io.File can help, but it is not the best practice. 

java.io.File on other JDKs

 We are lucky that the current implementation of the File object on our current JDK can sanitize an arbitrary path across UNIX and Windows; however, that will not always be the case. Our solution should always abstract the path separator symbol when possible so our code is portable to any OS with any JDK. For now, using File is acceptable only as a band-aid.

Fix Relative Path Issues with AbsolutePathResolver

Within org.codice.ddf.configuration there is a handy helper class called AbsolutePathResolver that can make life a lot easier when there is no other choice but to manipulate paths. It functions as such: 

Use of AbsolutePathResolver
String pathValue = new AbsolutePathResolver("etc/keystores/serverTruststore.jks").getPath();

Just like that, a relative path has been transformed into its absolute counterpart. 

Case-by-Case Specific Solutions

The following sections will explore actual errors found in DDF and how they were resolved for Windows. They are not guaranteed to work in all cases but can assist with trouble-shooting the problem. 

Formatting Resource Files with File Extensions

Windows allows extension-less files. Take for example any artifact and open it in a respectable editor (such as Sublime Text) and the file contents will be rendered somehow; be it binary, metadata, or just text. The problem occurs in the JVM implementation for Windows. When loading resources for testing purposes or otherwise, with class.getResource(), class.getClassLoader().getResourceAsStream(), or any variation of resource loading, that resource file must have a valid file extension. While the JVM will load extension-less files on UNIX without a problem, it will fail on Windows and always yield a NullPointerException

Load Resources with Valid File Extensions
this.getClass().getClassLoader().getResource("/first_subfolder_in_resources_folder/second_subfolder/myResource.xml");

Loading Resources

Too often a unit test will incorrectly retrieve the path to a resource. It may look like any variation of this: 

Improper resource loading
for (String schematronFile : schematronFiles) {
	// On Windows, the resource loader returns an invalid String for getPath(). There's an extra preceding '/' in the string. 
	String resourcePath = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile).getPath();
	schemaFiles.add(resourcePath);
}

Not even a qualifying call to Paths.get() will fix the issue: 

Improper resource loading with Paths.get()
for (String schematronFile : schematronFiles) {
	// This is still a problem. The preceding '/' is not filtered out. 
	String resourcePath = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile).getPath();
	resourcePath = Paths.get(resourcePath).toString();
	schemaFiles.add(resourcePath);
}

Instead, it is important to build up the resource through the Java File object like so: 

Correct loading of a resource
for (String schematronFile : schematronFiles) {
	URL schematronResource = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile);
	if (schematronResource == null) {
		throw new NullPointerException(
			"The Schematron Resource came back null. Did you remove the resources folder?");
	}
	String resourcePath = new File(schematronResource.getFile()).getAbsolutePath();
	schemaFiles.add(resourcePath);
}

Building URIs

Sometimes the wrong object is built up manually, and the resulting resource is inaccurate: 

Incorrectly formed URI
@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    uri = new URI(FILE_SCHEME_PLUS_SEP + ABSOLUTE_PATH + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

In this case, not even calls to Paths.get can improve the situation. The fundamental problem here is the URI constructor is being used incorrectly: 

Incorrectly formed URI with Paths.get calls
@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    uri = new URI(FILE_SCHEME_PLUS_SEP + ABSOLUTE_PATH 
            + Paths.get(TEST_PATH).toString() 
            + Paths.get(JPEG_FILE_NAME_1).toString());

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

Finally, the problem is solved by converting a properly constructed path to a URI on its own: 

Correctly building a URI from a path
@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    Path pathForUri = Paths.get(ABSOLUTE_PATH, TEST_PATH, JPEG_FILE_NAME_1);
	uri = pathForUri.toUri();

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

Rule of Thumb

These are some fairly subtle examples. Overtime, they have accumulated into a lot of technical debt and the pattern is always the same. To keep things as easy as possible, here is a simple rule of thumb to following when working with paths and URIs in Java: 

If you are building up a known path, use Paths.get().

If you are presented a string representing a resource that you didn't build up yourself, then get its path through the File object.

Whenever possible, if a URI or URL is needed, start with a Path and use Path.toUri() and other provided converters when necessary.

References

There are several prior incidents regarding Java and the JVM's underlying file system. 

Stack Overflow: getResource on Windows

Stack Overflow: Exception caused by preceding forward slash


 

Blueprint

Be very careful when working with paths in blueprint files. Remember that the java.io.File constructor is able to accurately parse paths cross-platform regardless of the delimiter you choose. Do not use Paths.get in a blueprint by way of the factory-method attribute. Instead, use a File object and invoke toPath() on it within a Java class, like so: 

ExportCommand.java
// Example taken from admin-core-migration-commands
// Note that defaultExportDirectory is a Path object
public ExportCommand(ConfigurationMigrationService configurationMigrationService,
            Security security, File defaultExportDirectoryAsFile) {
        this.configurationMigrationService = configurationMigrationService;
        this.security = security;
        this.defaultExportDirectory = defaultExportDirectoryAsFile.toPath();
    }

where the blueprint will look like: 

blueprint.xml
	<command-bundle xmlns="http://karaf.apache.org/xmlns/shell/v1.1.0">
        ...
        <command>
            <action class="org.codice.ddf.migration.commands.ExportCommand">
                <argument ref="configurationMigrationService"/>
                <argument ref="security"/>
                <argument ref="defaultExportDirectoryPath"/>
            </action>
        </command>
    </command-bundle>

    ...

    <bean id="defaultExportDirectoryPath" class="java.io.File">
        <argument value="${ddf.home}/etc/exported"/>
    </bean>

Refactoring may be required so that dependency injection (and the Java object's constructor) uses a File as an argument, not a Path


 

Symbol Errors

Line Termination

When unit tests require hard-coded metadata, the cleanest and easiest way to define this for a cross-platform setting is with a format string. 

Correct line termination
private static final String PRETTY_XML_NODE = String.format(
        "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>%n"
                + "<node>Valid-XML</node>%n");

It is not advisable to use termination literals or even System.lineSeparator(). The format string symbol %n will substitute the appropriate terminator based on underlying operating system. 

Foreign Character Encoding

Windows does not handle foreign character literals gracefully like UNIX. Granted it can accurately compare equality between String literals of the appropriate encoding, but is not always able to display the values of these literals. This issue is still being researched and it is highly recommended that any String operations that occur outside the default Charset are also given the desired encoding for the operation as well. This is typically UTF-8 for most cases. 

There are several instances where Latin, Chinese, or other encodings will get formatted backwards on Windows. 


 

Coding Patterns and Practices

Try-With-Resources

Always place your test streams in a try-with-resources block. On UNIX, the file system doesn't care if something is being used; it will delete, mutate, or move it without issue. Windows is very picky and if a file lock is acquired, operations will fail. 

Streams must be in Try-With-Resources
String updatedFileContents = null;
	try (InputStream is = item.getInputStream()) {
		updatedFileContents = IOUtils.toString(is);
    }

Managing Executors in Unit Tests

One last issue worth noting. Any set of unit tests may work consistently on a UNIX system, but exhibit undefined behavior on Windows. In the following case we see an executor service that was not being refreshed between unit tests. For some reason, the test behavior was acting "apparently deterministic" on UNIX (there's no way this can be attributed to luck), but completely unpredictable on Windows. This could be attributed to the underlying implementation for threading. 

Notice that, in the bad implementation below, the same executor is used across the entire test class. 

Bad candidates to be "static" and "final"
	private static final long SHORT_TIMEOUT = 25;

    private static final long LONG_TIMEOUT = 100;

    private static final FilterFactory FILTER_FACTORY = new FilterFactoryImpl();

    private static final ExecutorService EXECUTOR = Executors.newFixedThreadPool(2);

    private static final Logger LOGGER =
            LoggerFactory.getLogger(FederationStrategyTest.class.getName());

 

This is the correct way to implement the same test solution: 

Corrected use of test variables
	private static final long SHORT_TIMEOUT = 25;

    private static final long LONG_TIMEOUT = 100;

    private static final Logger LOGGER =
            LoggerFactory.getLogger(FederationStrategyTest.class.getName());

    private FilterFactory filterFactory;

    private ExecutorService executor;

    @Before
    public void setup() throws Exception {
        filterFactory = new FilterFactoryImpl();
        refreshExecutor();
    }

    private void killAndWaitForExecutor() throws Exception {
        if (executor != null && !executor.isShutdown()) {
            executor.shutdown();
            executor.awaitTermination(1L, TimeUnit.MINUTES);
        }
    }

    private void refreshExecutor() throws Exception {
        killAndWaitForExecutor();
        executor = Executors.newFixedThreadPool(2);
    }

One should also take special precautions if tests are using real threads and validating or asserting on the results of asynchronous operations. Before calling any critical test statements, ensure the executor has finished its tasks. 


 

Deploying DDF on Windows Virtual Machines

When setting up a running instance of DDF on a Windows box, particularly a virtualized one, the following error (or similar) is often encountered: 

1 Interrupted Action

 Error 0x80010135: Path too long

Here is the known solution: 

  1. Ensure that the name of the DDF zip file is as short as possible, regardless of its version or other metadata. Use ddf.zip and nothing longer. 
  2. Unzip DDF in the root directory of the entire VM, usually the "C" drive. The path calculation takes everything into account and this helps minimize damage. 
  3. DDF should unzip without issue and run fine. 

 

Java IPv6 on Windows

Working with Java on Windows in an IPv6 use case requires several extra considerations that are not necessary on a UNIX flavored environment. Particularly with DDF, the etc\hosts file must be properly configured. 

  • On Windows, the interface routing is required. It looks like this: feco::a00:27ff:fed0:422b%1 user.local
    • Note that %1 means network interface 1 which is normally redundant information. Not for Windows. 
    • If VPN or an intranet is setup, then that interface needs to be specified
  • On UNIX, the following would suffice: feco::a00:27ff:fed0:422b user.local


Outstanding Issues Pending Investigation

  • Parallel builds do not work on Windows due to the maven front-end plug-in and NPM
  • Catalog-app hard-coded dependency results in odd behavior and a broken DDF container startup on Windows when the VM is disconnected from the internet (see line 281):
catalog-app features.xml
<feature name="catalog-transformer-xml" install="auto" version="${project.version}"
	description="XML MetacardTransformer and InputTransformer">
	<feature prerequisite="true">catalog-app</feature>
	<bundle>mvn:ddf.catalog.transformer/catalog-transformer-xml/${project.version}</bundle>
	<bundle>
		mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.xstream/${xstream.bundle.version}
	</bundle>
	<bundle>
		mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.xpp3/${cxf.xpp3.bundle.version}
	</bundle>
	<bundle>mvn:commons-collections/commons-collections/3.2.1</bundle>
</feature>

 

Errata

This guide is far from exhaustive and is open for feedback. It is merely a compilation of the key issues encountered during DDF-1919, among others since then.