Windows Compatibility

Preface

This guide is to serve as a single source of truth for writing Windows-compliant Java code, including but not limited to production implementations, testing logic, and testing resources. Please keep this page up to date as new issues that are not documented here occur while working tickets. The end goal is to maintain a code-base that can be used, iterated on, built, and tested on various Windows environments. It is not sufficient to just support the end-product and end-users on Windows, but also any developers that are using Windows. 

Getting Started

In order to debug Windows issues, an environment will be needed. Refer to our vagrant repo for setup instructions. Some notes to consider: 

• Sharing files between the VM and host environment will slow things down unbearably (especially if it's maven related)
• It helps to provide the VM with access to 100% of system resources and choose to only develop on the VM or host (not both) at any given time
• Multithreaded builds do not work on Windows (see bottom of page)
• If needed, the host system should be able to handle two concurrent builds: one single threaded build locally and one single threaded build on the VM
  • Beyond that, browsers become difficult to use

File Paths: The Primary Cause for Cross-Platform Failure

The key issue is that most built-in Java methods for resolving paths and URLs are very well behaved on UNIX but do not perform precisely as expected on Windows. The prime example is Paths.get which will succeed on a Mac and happily throw an exception on a PC. It is important to determine, when processing a string that represents a system resource, at which stage of processing it is safe to pass to any appropriate Java method within the relevant classes. Our core production code is rarely the main problem.  

The biggest offenders here are unit tests. When hard-coding path strings to be used for testing, make sure all literals are passed to Paths.get upon initialization and not in the test method itself. Also, when loading resources, there are several quirks to be aware of that only occur on Windows. 

Examples of Valid and Invalid File Paths

File Path conventions differ based on operating system. The following shows key examples of the different rule-sets between Windows and UNIX systems:

Valid Windows Path: 

C:\Documents and Settings\Christopher\My Documents\My Pictures\funny.jpg

Valid Unix Path: 

/var/home/Christopher/documents/pictures/funny.jpg

Most Common Issue: 

The following specific example is a major offender. When passed to Paths.get it will always yield the error message that follows. 

\C:\Sample\sample.txt

java.nio.file.InvalidPathException: Illegal char <:> at index 2: \C:\Sample\sample.txt

The problem is the leading \ in the path which is often appended when fetching a URL to a Java resource. The most straight-forward way to fix it is to plug it into the File object constructor and extract a path from that. 

Examples of Valid and Invalid URIs

Universal Resource Identifiers carry strict conventions that can often throw an error during a call to Paths.get. While UNIX is very forgiving, Windows is not: 

Invalid Windows URI Examples:

1. file://D:Program FilesViewerstartup.html

2. C:Program FilesMusicWeb Sysmain.html?REQUEST=RADIO

3. file:////applib/products/a%2Db/ abc%5F9/4148.920a/media/start.swf

Corresponding Correct Windows URI Examples: 

1. file:///D:/Program%20Files/Viewer/startup.htm

2. file:///C:/Program%20Files/Music/Web%20Sys/main.html?REQUEST=RADIO

3. file://applib/products/a-b/abc_9/4148.920a/media/start.swf

These illustrate many more concepts than will be addressed here. It suffices to just note the differences between the paths and the URIs, particularly the count of leading slashes. 

Resource Coding Practices: How to Avoid File Pitfalls

The basics begin with clear declaration of the file system variables at the top of the file; especially for static final fields in unit tests. If the field can be stored as a Path instead of a String, all the better. Keeping state as a Path offers more options as test classes are extended. 

Best Case Scenario: What you should be doing 99% of the time

Whenever possible, do not handle or manipulate paths with regard to test resources. Simply use the class or class loader to return the resource as a stream and consume it into a String or any preferred data type that is needed. 

Get the Resource as a Stream

Resource loading should occur as follows whenever possible: 

InputStream resourceStream = getClass().getClassLoader().getResourceAsStream("resourceName");
String resourceData = IOUtils.toString(resourceStream, "UTF-8");

Also acceptable is to work with the URL: 

URL resourceUrl = getClass().getClassLoader().getResource("resourceName");
InputStream resourceStream = resourceUrl.openStream();
String resourceData = IOUtils.toString(resourceStream, "UTF-8");

Do not rely on path manipulation. It eliminates the problem entirely. 

Also, avoid the following style if possible. It shows very inconsistent behavior on Windows due to our OSGi environment depending on the module or bundle the code is a part of. 

 getClass().getResourceAsStream("/resourceName");

Using java.io.File

If you absolutely need to sanitize a bad URL that is returned from getResource() then the constructor of java.io.File can help, but it is not the best practice. 

Fix Relative Path Issues with AbsolutePathResolver

Within org.codice.ddf.configuration there is a handy helper class called AbsolutePathResolver that can make life a lot easier when there is no other choice but to manipulate paths. It functions as such: 

String pathValue = new AbsolutePathResolver("etc/keystores/serverTruststore.jks").getPath();

Just like that, a relative path has been transformed into its absolute counterpart. 

Case-by-Case Specific Solutions

The following sections will explore actual errors found in DDF and how they were resolved for Windows. They are not guaranteed to work in all cases but can assist with trouble-shooting the problem. 

Formatting Resource Files with File Extensions

Windows allows extension-less files. Take for example any artifact and open it in a respectable editor (such as Sublime Text) and the file contents will be rendered somehow; be it binary, metadata, or just text. The problem occurs in the JVM implementation for Windows. When loading resources for testing purposes or otherwise, with class.getResource(), class.getClassLoader().getResourceAsStream(), or any variation of resource loading, that resource file must have a valid file extension. While the JVM will load extension-less files on UNIX without a problem, it will fail on Windows and always yield a NullPointerException. 

this.getClass().getClassLoader().getResource("/first_subfolder_in_resources_folder/second_subfolder/myResource.xml");

Loading Resources

Too often a unit test will incorrectly retrieve the path to a resource. It may look like any variation of this: 

for (String schematronFile : schematronFiles) {
	// On Windows, the resource loader returns an invalid String for getPath(). There's an extra preceding '/' in the string. 
	String resourcePath = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile).getPath();
	schemaFiles.add(resourcePath);
}

Not even a qualifying call to Paths.get() will fix the issue: 

for (String schematronFile : schematronFiles) {
	// This is still a problem. The preceding '/' is not filtered out. 
	String resourcePath = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile).getPath();
	resourcePath = Paths.get(resourcePath).toString();
	schemaFiles.add(resourcePath);
}

Instead, it is important to build up the resource through the Java File object like so: 

for (String schematronFile : schematronFiles) {
	URL schematronResource = SchematronValidationServiceTest.class.getClassLoader().getResource(schematronFile);
	if (schematronResource == null) {
		throw new NullPointerException(
			"The Schematron Resource came back null. Did you remove the resources folder?");
	}
	String resourcePath = new File(schematronResource.getFile()).getAbsolutePath();
	schemaFiles.add(resourcePath);
}

Building URIs

Sometimes the wrong object is built up manually, and the resulting resource is inaccurate: 

@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    uri = new URI(FILE_SCHEME_PLUS_SEP + ABSOLUTE_PATH + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

In this case, not even calls to Paths.get can improve the situation. The fundamental problem here is the URI constructor is being used incorrectly: 

@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    uri = new URI(FILE_SCHEME_PLUS_SEP + ABSOLUTE_PATH 
            + Paths.get(TEST_PATH).toString() 
            + Paths.get(JPEG_FILE_NAME_1).toString());

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

Finally, the problem is solved by converting a properly constructed path to a URI on its own: 

@Test
public void testHTTPReturnsFileNameWithoutPath() throws Exception {
    URI uri = new URI(HTTP_SCHEME_PLUS_SEP + HOST + TEST_PATH + JPEG_FILE_NAME_1);

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);

    Path pathForUri = Paths.get(ABSOLUTE_PATH, TEST_PATH, JPEG_FILE_NAME_1);
	uri = pathForUri.toUri();

    verifyFileFromURLResourceReader(uri, JPEG_FILE_NAME_1, JPEG_MIME_TYPE);
    }

Rule of Thumb

These are some fairly subtle examples. Overtime, they have accumulated into a lot of technical debt and the pattern is always the same. To keep things as easy as possible, here is a simple rule of thumb to following when working with paths and URIs in Java: 

References

There are several prior incidents regarding Java and the JVM's underlying file system. 

Stack Overflow: getResource on Windows

Stack Overflow: Exception caused by preceding forward slash

Blueprint

Be very careful when working with paths in blueprint files. Remember that the java.io.File constructor is able to accurately parse paths cross-platform regardless of the delimiter you choose. Do not use Paths.get in a blueprint by way of the factory-method attribute. Instead, use a File object and invoke toPath() on it within a Java class, like so: 

// Example taken from admin-core-migration-commands
// Note that defaultExportDirectory is a Path object
public ExportCommand(ConfigurationMigrationService configurationMigrationService,
            Security security, File defaultExportDirectoryAsFile) {
        this.configurationMigrationService = configurationMigrationService;
        this.security = security;
        this.defaultExportDirectory = defaultExportDirectoryAsFile.toPath();
    }

where the blueprint will look like: 

	<command-bundle xmlns="http://karaf.apache.org/xmlns/shell/v1.1.0">
        ...
        <command>
            <action class="org.codice.ddf.migration.commands.ExportCommand">
                <argument ref="configurationMigrationService"/>
                <argument ref="security"/>
                <argument ref="defaultExportDirectoryPath"/>
            </action>
        </command>
    </command-bundle>

    ...

    <bean id="defaultExportDirectoryPath" class="java.io.File">
        <argument value="${ddf.home}/etc/exported"/>
    </bean>

Refactoring may be required so that dependency injection (and the Java object's constructor) uses a File as an argument, not a Path. 

Symbol Errors

Line Termination

When unit tests require hard-coded metadata, the cleanest and easiest way to define this for a cross-platform setting is with a format string. 

private static final String PRETTY_XML_NODE = String.format(
        "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>%n"
                + "<node>Valid-XML</node>%n");

It is not advisable to use termination literals or even System.lineSeparator(). The format string symbol %n will substitute the appropriate terminator based on underlying operating system. 

Foreign Character Encoding

Windows does not handle foreign character literals gracefully like UNIX. Granted it can accurately compare equality between String literals of the appropriate encoding, but is not always able to display the values of these literals. This issue is still being researched and it is highly recommended that any String operations that occur outside the default Charset are also given the desired encoding for the operation as well. This is typically UTF-8 for most cases. 

There are several instances where Latin, Chinese, or other encodings will get formatted backwards on Windows. 

Coding Patterns and Practices

Try-With-Resources

Always place your test streams in a try-with-resources block. On UNIX, the file system doesn't care if something is being used; it will delete, mutate, or move it without issue. Windows is very picky and if a file lock is acquired, operations will fail. 

String updatedFileContents = null;
	try (InputStream is = item.getInputStream()) {
		updatedFileContents = IOUtils.toString(is);
    }

Managing Executors in Unit Tests

One last issue worth noting. Any set of unit tests may work consistently on a UNIX system, but exhibit undefined behavior on Windows. In the following case we see an executor service that was not being refreshed between unit tests. For some reason, the test behavior was acting "apparently deterministic" on UNIX (there's no way this can be attributed to luck), but completely unpredictable on Windows. This could be attributed to the underlying implementation for threading. 

Notice that, in the bad implementation below, the same executor is used across the entire test class. 

	private static final long SHORT_TIMEOUT = 25;

    private static final long LONG_TIMEOUT = 100;

    private static final FilterFactory FILTER_FACTORY = new FilterFactoryImpl();

    private static final ExecutorService EXECUTOR = Executors.newFixedThreadPool(2);

    private static final Logger LOGGER =
            LoggerFactory.getLogger(FederationStrategyTest.class.getName());

This is the correct way to implement the same test solution: 

	private static final long SHORT_TIMEOUT = 25;

    private static final long LONG_TIMEOUT = 100;

    private static final Logger LOGGER =
            LoggerFactory.getLogger(FederationStrategyTest.class.getName());

    private FilterFactory filterFactory;

    private ExecutorService executor;

    @Before
    public void setup() throws Exception {
        filterFactory = new FilterFactoryImpl();
        refreshExecutor();
    }

    private void killAndWaitForExecutor() throws Exception {
        if (executor != null && !executor.isShutdown()) {
            executor.shutdown();
            executor.awaitTermination(1L, TimeUnit.MINUTES);
        }
    }

    private void refreshExecutor() throws Exception {
        killAndWaitForExecutor();
        executor = Executors.newFixedThreadPool(2);
    }

One should also take special precautions if tests are using real threads and validating or asserting on the results of asynchronous operations. Before calling any critical test statements, ensure the executor has finished its tasks. 

Deploying DDF on Windows Virtual Machines

When setting up a running instance of DDF on a Windows box, particularly a virtualized one, the following error (or similar) is often encountered: 

Here is the known solution: 

1. Ensure that the name of the DDF zip file is as short as possible, regardless of its version or other metadata. Use ddf.zip and nothing longer. 
2. Unzip DDF in the root directory of the entire VM, usually the "C" drive. The path calculation takes everything into account and this helps minimize damage. 
3. DDF should unzip without issue and run fine. 

Java IPv6 on Windows

Working with Java on Windows in an IPv6 use case requires several extra considerations that are not necessary on a UNIX flavored environment. Particularly with DDF, the etc\hosts file must be properly configured. 

• On Windows, the interface routing is required. It looks like this: feco::a00:27ff:fed0:422b%1 user.local
  • Note that %1 means network interface 1 which is normally redundant information. Not for Windows. 
  • If VPN or an intranet is setup, then that interface needs to be specified

• On UNIX, the following would suffice: feco::a00:27ff:fed0:422b user.local

Outstanding Issues Pending Investigation

• Parallel builds do not work on Windows due to the maven front-end plug-in and NPM
• Catalog-app hard-coded dependency results in odd behavior and a broken DDF container startup on Windows when the VM is disconnected from the internet (see line 281):

<feature name="catalog-transformer-xml" install="auto" version="${project.version}"
	description="XML MetacardTransformer and InputTransformer">
	<feature prerequisite="true">catalog-app</feature>
	<bundle>mvn:ddf.catalog.transformer/catalog-transformer-xml/${project.version}</bundle>
	<bundle>
		mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.xstream/${xstream.bundle.version}
	</bundle>
	<bundle>
		mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.xpp3/${cxf.xpp3.bundle.version}
	</bundle>
	<bundle>mvn:commons-collections/commons-collections/3.2.1</bundle>
</feature>

Errata

This guide is far from exhaustive and is open for feedback. It is merely a compilation of the key issues encountered during DDF-1919, among others since then.