How do I get an aggregated list of external dependencies of my maven project?

Question

I have a large multimodule maven project that has a large number of dependencies. I would like to generate a complete, duplicate-filtered list of third-party dependencies (that is, all dependencies not using the group id of the project) this project has.

I have tried using mvn dependency:list -DexcludeGroupIds=org.example.projectx for this purpose, but it seems unable to aggregate the output into a single list. When I run the command from the project root command line, I get output as follows:

[...]
[INFO] ------------------------------------------------------------------------
[INFO] Building ProjectX: ModuleA - Datatypes 4.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ projectx-moda-datatypes ---
[INFO] 
[INFO] The following files have been resolved:
[INFO]    org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO]    org.hamcrest:hamcrest-core:jar:1.3:test
[INFO]    junit:junit:jar:4.12:test
[INFO]    com.google.guava:guava:jar:18.0:compile
[INFO] 
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ProjectX: ModuleB - Binary 4.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ projectx-modb-binary ---
[INFO] 
[INFO] The following files have been resolved:
[INFO]    org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO]    com.google.guava:guava:jar:18.0:compile

..etc, for every single submodule. Not only is this not a single list (but a separate list for each submodule), but as you can see it contains duplicates. Moreover, the actual output I'm interested in is buried in a torrent of other Maven output (download messages, etc).

The -DoutputFile=<file.txt> option does not offer a solution either. The result of running mvn dependency:list -DoutputFile=deps.txt from my project root is not a single file listing all dependencies, but multiple separate files, one in each submodule-directory.

I can of course redirect the maven console output to a file (mvn [options] > output.txt) and try some clever regexing in vi to filter it down to the list I want. However, I was hoping there was a way to get what I need using just Maven, either using the dependency plugin or some other reporting plugin that I'm not aware of.

Answer 1

I was trying out a similar exercise and could not find any configurations / goals to dependency plugin that can help in this regard. But fortunately, merging few commands would make it work well to provide a list of all unique and sorted list of all the dependencies.

mvn dependency:list | grep "\[INFO\][^:]*:[^:]*:[^:]*:[^:]*:.*" | cut -d] -f2- | sort | uniq

Being specific to this question, the below should work.

mvn dependency:list -DexcludeGroupIds=org.example.projectx | grep "\[INFO\][^:]*:[^:]*:[^:]*:[^:]*:.*" | cut -d] -f2- | sort | uniq

Answer 2

I needed to extract the list of libraries/test libraries to feed them to the sonar scan application. We are not using the maven sonar-scanner plugin and I needed to have a simple way to avoid the developers to define the list of libraries manually. Also, I wanted to avoid to generally pass the whole maven cache to the scanner, as it would slow the scan down.

With the following commands, I can extract the list of dependencies and format as sorted, unique, comma separate list of full paths:

# This will generate the dependency list for each module and store it in separate files (#1)
mvn dependency:list -DoutputAbsoluteArtifactFilename=true -DoutputFile=dependencies.txt -Dsort=true --offline

# To print the test dependencies (#2)
{ find . -name dependencies.txt -exec grep ":test:" {} \;; } | sort | uniq | cut -d":" -f 6 | paste -d , -s

# To print the list of all dependencies except the test ones (#3)
{ find . -name dependencies.txt -exec grep -E ":.+:" {} \;; } | grep -v ":test:" | sort | uniq | cut -d":" -f 6 | paste -d , -s

Description:

1. use the dependency plugin to generate all the dependencies and store them in files named dependencies.txt which will be stored in each module's folder. Also, it sorts the dependencies and prints the full path of the artifact (i.e. org.springframework.boot:spring-boot-autoconfigure:jar:2.5.6:compile:/usr/share/maven/ref/repository/org/springframework/boot/spring-boot-autoconfigure/2.5.6/spring-boot-autoconfigure-2.5.6.jar)

2. find, in the current folder and subfolders, all the files named dependencies.txt and pass each found file to grep. Then filter only the lines containing ":test:", then sort all the lines (each line is a dependency), filter out duplicates, extract the 6th field (which correspond to the file name with full path), then paste it as a sequence of comma separated entries

3. is similar to 1 but grep will filter all the lines with at least 2 ":" (some lines are descriptive and contain only 1), then the result is passed through grep again to filter out all the lines containing the ":test:" string, leaving only the runtime, compile etc dependencies.

Other info

• the external curly brackets ("{ ... }") are used to create a sub shell
• the ones together ("{}") are a placeholder for the find command and will be substituted with the path of each found file
• the first ";" in "\;;" is to end the find command and the second is to end the command in the subshell