Use zipTree as source set for building and static compiling

Question

Background

Project Alice generates Java source code, stores it in sources.jar, then uploads it to a Maven repository. Project Bob pulls sources.jar down and needs to use it when compiling. Bob does not know that Alice exists, only where to find sources.jar.

Versions: JDK 11, Gradle 7.3.1, IntelliJ IDEA 2021.3.1

Problem

Making gradle (and IntelliJ's IDEA) build using source files embedded in a JAR file. To be clear, the JAR file contents resemble:

$ jar -tvf sources.jar 
     0 Thu Feb 03 08:38:56 PST 2022 META-INF/
    52 Thu Feb 03 08:38:56 PST 2022 META-INF/MANIFEST.MF
     0 Thu Feb 03 08:38:30 PST 2022 com/
     0 Thu Feb 03 08:38:32 PST 2022 com/domain/
     0 Thu Feb 03 08:38:30 PST 2022 com/domain/package/
   938 Thu Feb 03 08:38:32 PST 2022 com/domain/package/SourceCode.java

Solutions that extract the .java files from the .jar file introduce knock-on effects we'd like to avoid, including:

• Editable. Extracted source files can be edited in the IDE. We'd like them to be read-only. We could add a task that sets the files read-only, but that feels like solving the wrong problem (added complexity).
• Synchronization. When a newly generated sources.jar is pulled down, we'll have to delete the extraction directory to remove any stale .java files that were preserved. If there was a way to avoid extraction, then the act of pulling down the new sources.jar file would ensure correctness (no added complexity). By unarchiving the .java files, it's possible to enter an inconsistent state:

$ jar -tvf sources.jar | grep java$ | wc -l
61
$ find src/gen -name "*java" | wc -l
65

If there was a way to treat sources.jar as a source directory without extracting the files, these knock-on effects disappear.

Attempts

A number of approaches have failed.

sourceSets

Changing sourceSets doesn't work:

sourceSets.main.java.srcDirs += "jar:${projectDir}/sources.jar!/"

The error is:

Cannot convert URL 'jar:/home/user/dev/project/sources.jar!/' to a file.

Using a zipTree with sourceSets doesn't work, although the error message is telling:

sourceSets.main.java.srcDirs += zipTree(file: "${projectDir}/sources.jar")

Error:

Cannot convert the provided notation to a File or URI.

The following types/formats are supported:

• A URI or URL instance.

This was expected. What was unexpected was that URL instances are allowed, but seemingly not if embedded within a JAR file.

The following allows building Bob, but the IDE is unable to find SourceCode.java:

sourceSets.main.java.srcDirs += zipTree("${projectDir}/sources.jar").matching {
  include "com"
}

build task

Modifying the build task to extract the generated code first partially works:

task codeGen {
  copy {
    from( zipTree( "${projectDir}/sources.jar" ) )
    into( "${buildDir}/src/gen/java" )
  }

  sourceSets.main.java.srcDirs += ["${buildDir}/src/gen/java"]
}

build { doFirst { codeGen } }

The issue is that removing the build directory then prevents static compiles (because IDEA cannot find the generated source files). In any case, we don't want to extract the source files because of all the knock-on problems.

compile task

The following snippet also does not compile:

tasks.withType(JavaCompile) {
  source = zipTree(file: "${projectDir}/sources.jar")
}

Not updating sourceSets means that the IDE cannot discover the source files.

sync task

We could extract the files into the main source directory, instead, such as:

def syncTask = task sync(type: Sync) {
    from zipTree("${projectDir}/sources.jar")
    into "${projectDir}/src/gen/java"

    preserve {
        include 'com/**'
        exclude 'META-INF/**'
    }
}

sourceSets.main.java.srcDir(syncTask)

While that addresses the clean issue, we’re left with the original problems that we’d like to avoid.

Content Root

Setting the Content Root and marking the Source Folder from within IntelliJ IDEA works. The IDE updates .idea/misc.xml to include:

  <component name="FrameworkDetectionExcludesConfiguration">
    <file type="web" url="file://$PROJECT_DIR$" />
    <file type="web" url="jar://$PROJECT_DIR$/project/sources.jar!/" />
  </component>

In theory, the idea plugin may have the ability to set this value.

Question

How would you instruct Gradle to reference and build using source files that are stored in an external Java archive file when compiling a project (without extracting the archive) such that IDEA can statically resolve the source files, as well?

Answer 1

We decided that extracting the jar file would be the best approach after all:

apply plugin: 'idea'

final GENERATED_JAR = "${projectDir}/sources.jar"
final GENERATED_DIR = "${buildDir}/generated/sources"

task extractSources(type: Sync) {
  from zipTree(GENERATED_JAR)
  into GENERATED_DIR
}

sourceSets.main.java.srcDir extractSources
clean.finalizedBy extractSources

idea.module.generatedSourceDirs += file(GENERATED_DIR)

This:

• Retains the generated sources after clean
• Issues a warning in the IDE when modifying generated sources
• Couples the build process to extracting source files
• Reuses the build/generated/sources path
• Keeps .jar file and .java files in synchronization

In effect, the following behaviours work as expected:

• ./gradlew clean -- leaves the extracted .java files intact, effectively
• ./gradlew build -- re-synchronizes .java files with .jar contents