Scan projects

• 1: Scan for open source risks
• 1.1: Java
• 1.2: Kotlin
• 1.3: Go
• 1.4: Python
• 1.5: JavaScript/TypeScript
• 1.6: Ruby
• 1.7: .NET
• 1.8: Bazel
• 1.9: Swift/Objective-C
• 1.10: Scala
• 1.11: PHP
• 1.12: Rust
• 2: Scanning strategies
• 3: Perform incremental scans
• 4: Scoping scans
• 5: Working with monorepos
• 6: Repository Security Posture Management
• 7: Rescan projects
• 8: Perform scans in CI pipelines
• 8.1: Set up keyless authentication
• 8.2: Scanning with GitHub Actions
• 8.3: Scanning in Bitbucket Pipelines
• 8.4: Scanning in GitLab Pipelines
• 8.5: Scanning with CircleCI
• 8.6: Scanning with Google Cloud Build
• 8.7: Scanning with Jenkins
• 8.8: Scanning in Azure Pipelines
• 9: Scan from your IDE
• 10: Manage build tools (Beta)
• 11: Scan artifacts and binaries
• 12: Scan containers (Beta)

1 - Scan for open source risks

Endor Labs supports the following major capabilities to help teams reduce the risk and expense of software dependency management across the lifecycle of software reuse.

• SCA - Software composition analysis is the identification of the bill of materials for first-party software packages and the mapping of vulnerabilities to these software component versions. SCA helps teams to maintain compliance and get visibility into the risks of their software inventory.
• Endor Scores - Endor Labs provides a holistic risk score that includes the security, quality, popularity and activity of a package. Risk scores help in identifying leading indicators of risk in addition to if a software component is outdated, or unmaintained. Risk analysis helps teams to go beyond vulnerabilities and approach the risk of their software holistically.
• Reachability Analysis - Reachability analysis is Endor Labs’ capability to perform static analysis on your software packages to give context to how each vulnerability may be reached in the context of your code. This includes mapping vulnerabilities back to vulnerable functions so that deep static analysis can target vulnerabilities with higher levels of granularity as well as the identification of unused software dependencies.
• Upgrade Impact Analysis - Upgrade impact analysis allows security teams to set better expectations with their development teams by identifying breaking changes associated with an update of a direct dependency.

The resource requirements, both minimum and recommended, for build runners or workers executing scans using endorctl are listed here.

Note: Large applications may require additional resources to complete or enhance the scan performance.

Minimum Resources

Recommended Resources

Supported languages

The following table shows Endor Labs language coverage:

* For JavaScript, reachability is currently limited to the package level; function-level reachability will be supported in the future.

Complete support matrix

The following comprehensive matrix lists the supported languages, build tools, manifest files, and supported requirements.

See the detailed procedure for all supported languages:

1.1 - Java

Java is a high-level, object-oriented programming language widely used by developers. Endor Labs supports scanning and monitoring of Java projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

Endor Labs requires the following prerequisites in place for successful scans.

• Install JDK versions between 11 and 22
  • For JDK 8, see Scan the projects on JDK version 8
• Make sure your repository includes one or more files with .java extension.
• Install any version of the Maven Package Manager if your project uses Maven.
• Install Gradle build system version 6.0.0 and higher, if your project uses Gradle. To support lower versions of Gradle, see Scan projects on Gradle versions between 4.7 and 6.0.0.
• For projects not using Maven or Gradle, make sure that your project is set up properly to scan without the pom.xml file. See Scan projects without pom.xml for more information.

Build Java projects

You must build your Java projects before running a scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

Use Gradle

To analyze your software built with Gradle, Endor Labs requires that the software be able to be successfully built. To perform a quick scan, dependencies must be located in the local package manager cache. The standard $GRADLE_USER_HOME/caches or /User/<username>/.gradle/caches must exist and contain successfully downloaded dependencies. To perform a deep scan the target artifact must be generated on the file system as well.

To build your project with Gradle, use the following procedure:

1. If you would like to run a scan against a custom configuration, specify the Gradle configuration by setting an environment variable.

      export endorGradleJavaConfiguration="<configuration>"
   

   The default configuration is runtimeClasspath if none is provided.

2. Ensure that you can resolve the dependencies for your project without errors by running the following command:

   For Gradle wrapper:

      ./gradlew dependencies
   

   For Gradle:

      gradle dependencies
   

3. Run ./gradlew assemble or gradle assemble to resolve dependencies and to create an artifact that may be used for deep analysis.

Override subproject level configuration

In a multi-build project, if you set the environment variable endorGradleJavaConfiguration=[GlobalConfiguration], the specified configuration is used for dependency resolution across all projects and subprojects in the hierarchy below.

\--- Project ':samples'
     +--- Project ':samples:compare'
     +--- Project ':samples:crawler'
     +--- Project ':samples:guide'
     +--- Project ':samples:simple-client'
     +--- Project ':samples:slack'
     +--- Project ':samples:static-server'
     +--- Project ':samples:tlssurvey'
     \--- Project ':samples:unixdomainsockets'

To override the configuration only for the :samples:crawler and :samples:guide subprojects, follow these steps:

1. Navigate to the root workspace, where you execute endorctl scan, and run ./gradlew projects to list all projects and their names.

2. Run the following command at the root of the workspace:

   echo ":samples:crawler=testRuntimeClasspath,:samples:guide=macroBenchMarkClasspath" >> .endorproperties
   

   This creates a new file named .endorproperties in your root directory. This enables different configurations for the specified subprojects in the file.

3. Run endorctl scan as usual.

At this point, all other projects will adhere to the GlobalConfiguration. However, the :samples:crawler subproject will use the testRuntimeClasspath configuration, and the :samples:guide subproject will use the macroBenchMarkClasspath configuration.

Use Maven

To analyze your software built with Maven, Endor Labs requires that the software be able to be successfully built. To perform a quick scan, dependencies must be located in the local package manager cache. The standard .m2 cache must exist and contain successfully downloaded dependencies. To perform a deep scan the target artifact must be generated on the file system as well.

To build your project with Maven, use the following procedure:

1. Ensure that you can resolve the dependencies for your project without error by running the following command.

   mvn dependency:tree
   

2. Run mvn install and make sure the build is successful.

   info

   If you want to skip the execution of tests during the build, you can use -DskipTests to quickly build and install your projects.

   mvn install -DskipTests
   

3. If you have multiple Java modules not referenced in the root pom.xml file, make sure to run mvn install separately in all the directories.

Configure Maven private registries

If you have a private registry and internal dependencies on other projects, you must configure private registries. In addition to scanning public Java projects and repositories, Endor Labs provides the support to integrate with private registries.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Maven and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When deep analysis is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans.

Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --disable-private-package-analysis

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Scan projects without pom.xml (Beta)

Endor Labs supports projects that do not use Maven or Gradle, and have no pom.xml in the following cases.

Uber jars

If there is an uber jar (fat jar) that contains all application classes and dependency jars of the project, you can set the environment variable, ENDOR_JVM_USE_ARTIFACT_SCAN as true and run the scan.

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
endorctl scan --quick-scan --package --path=<jar/ear/war location> --project-name=<project name>

For example:

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
endorctl scan --quick-scan --package --path=/Users/johndoe/projects/project21.jar --project-name=Project21

Application dependencies in classpath

If you do not have an uber jar with dependencies, but only have application dependency files (like jar, war, or ear), you can set the path to these files in the environment variable, ENDOR_JVM_USE_ARTIFACT_SCAN_CLASSPATH and run the scan.

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
export ENDOR_JVM_USE_ARTIFACT_SCAN_CLASSPATH=<path that contains application depedencies>
endorctl scan --quick-scan --package --path=<jar/ear/war location> --project-name=<project name>

For example:

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
export ENDOR_JVM_USE_ARTIFACT_SCAN_CLASSPATH=/Users/johndoe/caches/modules/files-2.1
endorctl scan --quick-scan --package --path=/Users/johndoe/projects/project21.jar --project-name=Project21

Application classes and dependencies extracted as first-party class files

If application class files and dependency jar files are extracted as first-party class files, you can provide the first-party files in an environment variable, ENDOR_JVM_FIRST_PARTY_PACKAGE.

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
export ENDOR_JVM_FIRST_PARTY_PACKAGE="<dependency/application 1>,<dependency/application 2>,...,<dependency/application N>"
endorctl scan --quick-scan --package --path=<jar/ear/war location> --project-name=<project name>

For example:

Your project jar has the following structure where com.org.doe and com.org.deer are application class files and dependencies.

fawn.jar
├── com.org.doe
│   ├── A.class
│   └── B.class
├── com.org.deer
│   ├── Util.class
│   └── Utilities.class
├── com.org.dep1
│   ├── Dep1.class
│   └── Dep2.class
└── com.org.dep2
    ├── 2Dep1.class
    └── 2Dep2.class

export ENDOR_JVM_USE_ARTIFACT_SCAN=true
export ENDOR_JVM_FIRST_PARTY_PACKAGE="com.org.doe,com.org.deer"
endorctl scan --quick-scan --package --path=/Users/johndoe/projects/fawn.jar --project-name=Fawn

Scan the projects on JDK version 8

Endor Labs supports JDK versions between 11-22, however, you can scan projects on JDK 8 using the following procedure:

1. Build your Java project on JDK 8.

2. After building, switch your Java home to JDK 11 or higher versions.

   export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home
   

3. Run a scan

Scan projects on Gradle versions between 4.7 and 6.0.0

To scan Java projects on Gradle versions between 4.7 and 6.0.0, make sure to

1. Check the version of your project using:

   ./gradlew --version
   

2. The project must have a Gradle wrapper. You can generate the Gradle wrapper using:

   --gradle-version <your required version>.
   

   Endor Labs prioritizes Gradle wrapper over Gradle and it is a recommended best practice to use Gradle Wrapper.

3. Before executing the endorctl scan, ensure the project can be built in your required version.

   Execute ./gradlew assemble.
   

4. Use --bypass-host-check during endorctl scan to execute scans on projects that have Gradle versions lower than 6.0.0.

Understand the scan process

Endor Labs analyzes your Java code and dependencies to detect known security issues, including open-source vulnerabilities and generates call graphs.

How Endor Labs resolves dependencies for package versions

Endor Labs resolves the dependencies for Java packages based on the following factors:

• For packages built using Maven, it leverages the Maven cache in the .m2 directory of your file system to resolve the package’s dependencies and mirrors Maven’s build process for the most accurate results.
• For Maven, Endor Labs respects the configuration settings present in the settings.xml file. If this file is included in your repository, you need not provide any additional configuration.
• For packages built using Gradle, it uses Gradle and Gradle wrapper files to build packages and resolve dependencies.
• Endor Labs supports EAR, JAR, RAR, and WAR files.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis on the Java code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known limitations

• If a package can not be successfully built in the source control repository, static analysis will fail.
• Spring dependencies are analyzed based on spring public entry points to reduce the impact of Inversion of Control (IOC) frameworks. Dependencies and functions are identified as reachable and unreachable in the context of a spring version and its entry points.
• Annotation processing is limited only to the usage of the code they annotate.
• Static analysis of reflection and callbacks are not supported.
• Endor Labs requires JDK 11 to generate call graphs for Java projects. Gradle versions lacking JDK 11 support are not compatible.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:

  • Java is not installed or not present in the PATH environment variable. Install Java and try again. See Java documentation for more information.
  • The installed version of Java is lower than the required version. Install JDK versions between 11-22 and try again.
  • Java is installed but Maven or Gradle is not installed. In such cases, the dependency resolution may not be complete.

• Unresolved dependency errors: Maven is not installed properly or the system is unable to build root pom.xml. Run mvn dependency:tree in the root of the project and try again. In such cases, the dependency resolution may not be complete.

• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

• Gradle variant incompatibility message: Gradle performs JVM toolchain checks for subprojects or dependencies and may raise errors indicating a Java version mismatch between dependencies declared in Gradle manifest and Java home setup.

  Example error message:

  Incompatible because this component declares a component for use during compile-time, compatible with Java version 21 and the consumer needed a component for use during runtime, compatible with Java version 17 - To resolve this and taking advantage of Java’s backward compatibility, instruct Gradle to use the higher version of JDK detected in the error message. For example, for the message above, specify org.gradle.java.home=<path of java> in .gradle/gradle.properties. The path needs to be to the root of the directory bin/java. For example, if your Java is at /Users/Downloads/jdk-21/Contents/Home/bin/java, specify org.gradle.java.home=/Users/Downloads/jdk-21/Contents/Home. - If you are scanning a purely Java 8 Gradle project and if you encounter the above error, set org.gradle.java.home to point to Java 8 home, before you execute the endorctl scan. - A general guideline for determining which Java version to use, is to match the Java version specified in .gradle/gradle.properties with the one used for building your Gradle project.

• Call graph errors: - The project can not be built because a dependency cannot be located in the repository. - Sometimes, the project is not built, if a Java version discrepancy exists between the required repository version and the version on the system running the scan. For example, the Java required version is 1.8 but the system has 12 installed. Install the required version and try again.

• If you have a private registry and internal dependencies on other projects, you must configure the credentials of the registry. See Configure Maven private registries.

• If you have a large repository or if the scan fails with out-of-memory issues, you may need to increase the JVM heap size before you can successfully scan. Export the ENDOR_SCAN_JVM_PARAMETERS environment variable with additional JVM parameters before performing the scan as shown below:

  export ENDOR_SCAN_JVM_PARAMETERS="-Xmx32G"
  

• If you use a remote repository configured to authenticate with a client-side certificate, you must add the certificate through an endorctl parameter. Export the ENDOR_SCAN_JVM_PARAMETERS parameter before performing a scan. See Maven documentation for details.

  export ENDOR_SCAN_JVM_PARAMETERS="-Xmx16G,-Djavax.net.ssl.keyStorePassword=changeit,
  -Djavax.net.ssl.keyStoreType=pkcs12,
  -Djavax.net.ssl.keyStore=/Users/myuser/Documents/nexustls/client-cert1.p12"
  

1.2 - Kotlin

Kotlin is a statically typed programming language that runs on the Java Virtual Machine (JVM), known for its concise syntax, null safety, and seamless integration with Java. Endor Labs supports scanning and monitoring of Kotlin projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

To ensure a successful scan using Endor Labs, it is essential to have the following prerequisite software installed:

• Maven Package Manager: Version 3.6.1 and higher

• Gradle Build System: Version 6.0.0 and higher

• Java Runtime Environment: JDK versions between 11 and 19

• Kotlin: version 1.4 through 2.0

  For users on JDK 8, specific instructions can be found in the section Scan the Projects on JDK Version 8.

• Make sure your repository includes one or more files with .kt extension.

Build Kotlin projects

Before initiating a scan with Endor Labs, ensure that your Kotlin projects are built successfully. Additionally, ensure that the packages are downloaded into local package caches and build artifacts are present in their standard locations. Follow the guidelines to use Gradle and Maven:

Use Gradle

To analyze your software built with Gradle, Endor Labs requires:

• The software must be successfully built with Gradle.
• For quick scans, dependencies must be located in the local package manager cache. The standard $GRADLE_USER_HOME/caches or /User/<<username>>/.gradle/caches cache must exist.
• For deep scans, the target artifact must be generated on the filesystem.

To build your project with Gradle, run the following commands:

1. Specify the Gradle configuration by setting an environment variable.

export endorGradleKotlinConfiguration="compileClasspath"

If you need to override the default configuration, use the command:

export endorGradleKotlinConfiguration="<configuration>"

The default configuration is `runtimeClasspath` if none is provided.

For Android projects, you can set the configuration using:

export endorGradleAndroidConfiguration="<configuration>"

The default configuration for an Android application or library uses the same structure established by Android Studio. For applications, all possible combinations of application variants are examined. For libraries, all possible combinations of library variants are examined. The first variant from the alphabetically sorted list is then selected for runtimeClasspath.

If neither approach provides a value, the default configuration is set to releaseRuntimeClasspath.

2. Confirm an error-free dependency resolution for your project.

gradle dependencies

or, with a Gradle wrapper.

./gradlew dependencies

3. Generate the artifact for deep analysis.

gradle assemble

or, with a Gradle wrapper.

./gradlew assemble

Override sub project level configuration

In a multi-build project, if you set the environment variable endorGradleKotlinConfiguration=[GlobalConfiguration] and/or endorGradleAndroidConfiguration=[GlobalConfiguration], the specified configuration is used for dependency resolution across all projects and sub-projects in the hierarchy below.

    \--- Project ':samples'
         +--- Project ':samples:compare'
        +--- Project ':samples:crawler'
        +--- Project ':samples:guide'
        +--- Project ':samples:simple-client'
        +--- Project ':samples:slack'
        +--- Project ':samples:static-server'
        +--- Project ':samples:tlssurvey'
        \--- Project ':samples:unixdomainsockets'

To override the configuration only for the :samples:crawler and :samples:guide sub-projects, follow these steps:

1. Navigate to the root workspace, where you execute endorctl scan, and run ./gradlew projects to list all projects and their names.

2. Run the following command at the root of the workspace:

echo ":samples:crawler=testRuntimeClasspath,:samples:guide=macroBenchMarkClasspath" >> .endorproperties

This creates a new file named .endorproperties in your root directory. This enables different configurations for the specified sub-projects in the file.

3. Run endorctl scan as usual.

At this point, all other projects will adhere to the GlobalConfiguration. However, the :samples:crawler sub-project will use the testRuntimeClasspath configuration, and the :samples:guide sub-project will use the macroBenchMarkClasspath configuration.

Use Maven

To analyze your software built with Maven, Endor Labs requires:

• The software must be successfully built with Maven.
• For quick scans, dependencies must be located in the local package manager cache. The standard .m2 cache must exist.
• For deep scans, the target artifact must be generated on the filesystem.

To build your project with Maven, run the following commands:

1. Confirm an error-free dependency resolution for your project.

mvn dependency:tree

2. Run mvn install and ensure the build is successful.

   info

   If you want to skip the execution of tests during the build, you can use -DskipTests to quickly build and install your projects.

mvn install -DskipTests

3. If you have multiple Kotlin modules not referenced in the root pom.xml file, ensure to run mvn install separately in each directory.

Configure Maven private registries

If you have a private registry with internal dependencies on other projects, you must configure private registries. Endor Labs not only supports scanning public Kotlin projects and repositories but also allows integration with private registries.

Follow these steps to configure Maven private registries:

1. Sign in to Endor Labs and navigate to Integrations under Manage from the left sidebar.
2. In the Package Managers > Maven section, click Manage.
3. On the Maven integrations page, click Add Package Manager.
4. Specify a package manager URL.
5. (Optional) To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and specify the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

To scan your repositories with Endor Labs, you can use the following options after building your Kotlin projects.

Option 1 - Quick scan

To quickly gain insight into your software composition, initiate a quick scan using the following command:

endorctl scan --quick-scan

This scan offers a quick overview without performing reachability analysis, helping you prioritize vulnerabilities.

Save local results

To scan a Git project repository from the root directory and save the results locally in the results.json file, use the following command:

endorctl scan --quick-scan -o json | tee /path/to/results.json

This generates comprehensive results and analysis information, accessible from the Endor Labs user interface.

Access results

To access and review detailed results, sign in to the Endor Labs user interface. Navigate to Projects on the left sidebar, and locate your project for a thorough examination of the scan results.

Option 2 - Deep scan

To perform dependency resolution and reachability analysis, use deep scan with Endor Labs. This option is recommended only after successfully completion of quick scan.

endorctl scan

Save local scan results

To save the local results to a results.json file, use the following flag.

endorctl scan -o json | tee /path/to/results.json

This generates comprehensive results and analysis information, accessible from the Endor Labs user interface.

Analyze private packages

During deep analysis, Endor Labs thoroughly analyzes all private software dependencies that have not been previously scanned. While this initial operation may slow down scans, subsequent scans remain unaffected.

If your organization does not own specific software parts and related findings are non-actionable, you can choose to disable this analysis using the disable-private-package-analysis flag. Disabling private package analysis enhances scan performance but may result in a loss of insights into how applications interact with first-party libraries.

To disable private package analysis, use the following command flag:

endorctl scan --disable-private-package-analysis

Access scan results

To access and review detailed results, sign in to the Endor Labs user interface. Navigate to Projects on the left sidebar, and locate your project for a thorough examination of the scan results.

Scan the projects on JDK version 8

While Endor Labs primarily supports JDK versions between 11-22, you can still scan projects on JDK 8 by following these steps:

1. Build your Java project on JDK 8.
2. After successful build, switch your Java home to JDK 11 or higher versions.

export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home

3. Run a scan.

Understand the scan process

Endor Labs analyzes your Kotlin code and dependencies to identify known security issues, including open-source vulnerabilities.

How Endor Labs resolves dependencies for package versions

Endor Labs resolves Kotlin package dependencies by considering the following factors:

• For packages built with Maven, it leverages the Maven cache in the .m2 directory of your file system. This mirrors Maven’s build process for precise results.
• For packages built with Maven, it respects the configuration settings present in the settings.xml file. If the file is included in your repository, any additional configuration is not necessary.
• For packages built with Gradle, it leverages Gradle and Gradle wrapper files to build and resolve dependencies.
• Endor Labs supports AAR, EAR, JAR, RAR, and WAR files.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis on the code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known limitations

• If a package can not be successfully built in the source control repository, static analysis will fail.
• Spring dependencies are analyzed based on spring public entry points to reduce the impact of Inversion of Control (IOC) frameworks. Dependencies and functions are identified as reachable and unreachable in the context of a spring version and its entry points.
• Annotation processing is limited only to the usage of the code they annotate.
• Static analysis of reflection and callbacks are not supported.
• If Endor Labs fails to resolve dependencies using default Kotlin configurations, the Kotlin configuration must be specified.
• Static analysis for Kotlin projects using Gradle is only supported when the Kotlin plugin for Gradle versions 1.5.30 to 1.9.x

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:
  • Java is not installed or not present in the PATH environment variable. Install Java and try again. See Java documentation for more information.
  • For android applications, $ANDROID_HOME must be specified as an environment variable.
  • The installed version of Java is lower than the required version. Install JDK versions between 11-22 and try again.
  • Java is installed but Maven or Gradle is not installed. In such cases, the dependency resolution may not be complete.
• Unresolved dependency errors: Maven is not installed properly or the system is unable to build root pom.xml. Run mvn dependency:tree in the root of the project and try again. In such cases, the dependency resolution may not be complete.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.
• Call graph errors:
  • If the project is not compiled, call graphs are not generated. Run gradlew compileKotlin or gradlew compileReleaseKotlin for android based projects before running the scan.
  • Sometimes, the project is not compiled, if a Kotlin version discrepancy exists between the required repository version and the version on the system running the scan. For example, the Kotlin required version is 1.4 but the system has lower version installed. Install the required version and try again.
• If you have a private registry and internal dependencies on other projects, you must configure the credentials of the registry. See Configure Maven private registries.
• If you use a remote repository configured to authenticate with a client-side certificate, you must add the certificate through an endorctl parameter. Export the ENDOR_SCAN_JVM_PARAMETERS parameter before performing a scan. See Maven documentation for details.

export ENDOR_SCAN_JVM_PARAMETERS="-Xmx16G,-Djavax.net.ssl.keyStorePassword=changeit,
-Djavax.net.ssl.keyStoreType=pkcs12,
-Djavax.net.ssl.keyStore=/Users/myuser/Documents/nexustls/client-cert1.p12"

1.3 - Go

Go or Golang is a software development programming language widely used by developers. Endor Labs supports scanning and monitoring of Go projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

• Make sure that you have Go 1.12 or higher versions.
• Make sure your repository includes one or more files with .go extension.

Build Go projects

You must build your Go projects before running the scan. Additionally, ensure that the packages are downloaded into the local package caches and that go.mod file well formed and is available in the standard location.

To ensure that your go.mod file is well formed, run the following command:

go mod tidy

go get ./

This removes any dependencies that are not required by your project and ensures to resolve the dependencies without errors.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs resolves your Golang-based dependencies by leveraging built-in Go commands to replicate the way a package manager would install your dependencies.

To discover package names for Go packages Endor Labs uses the command:

go list -e -mod readonly -json -m

To analyze the dependency graph of your package Endor Labs uses the command:

go list -e -deps -json -mod readonly all

To assess external dependencies, specifically third-party packages or libraries that your Go project relies on, Endor Labs uses the command:

go list -e -deps -json -mod vendor all

These commands allow us to assess packages’ unresolved dependencies, analyze the dependency tree, and resolve dependencies for your Go projects.

Known Limitations

Endor Labs creates go.mod files for you when projects do not have a go.mod file. This can lead to inconsistencies with the actual package created over time and across versions of the dependencies.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors:

  • Go is not installed or not present in the PATH environment variable. Install Go and try again.
  • The installed version of Go is lower than 1.12. Install Go version 1.12 or higher and try again.

• Resolved dependency errors:

  • A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.
  • If the go.mod file is not well-formed then dependency resolution may return errors. Run go mod tidy and try again.

• Call graph errors:

  These errors often mean the project won’t build. Please ensure any generated code is in place and verify that go build ./... runs successfully.

1.4 - Python

Python is a high-level, interpreted programming language widely used by developers. Endor Labs supports the scanning and monitoring of Python projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

Ensure that the following prerequisites are complete:

• Install Python 3.6 or higher versions. See the Python documentation on how to install Python.
• Ensure that the package manager pip, Poetry, or PDM is used by your projects to build your software packages. From Python 3.12, install setuptools if you use pip.
• Set up any build, code generation, or other dependencies that are required to install your project’s packages.
• Organize the project as one or more packages using setup.py, setup.cfg, pyproject.toml, or requirements.txt package manifest files.
• Make sure your repository includes one or more files with .py extension or pass either one of requirements.txt, setup.py, setup.cfg or pyproject.toml using the --include-path flag. See Scoping scans.

Build Python projects

You must create a virtual environment and build your Python projects before running the endorctl scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Configure any private repositories
   • If you use dependencies from a PyPI compatible repository other than pypi.org, configure it in the Integrations section of the Endor Labs web application. See Configure Python private repositories for more details.
2. Clone the repository and create a virtual environment inside it

   1. Clone the repository using git clone or an equivalent workflow.

   2. Enter the working copy root directory that’s created.

   3. Create a virtual environment based your package manager:

      For pip or setuptools

      • Use python3 -m venv venv. Set up the virtual environment in the root folder that you want to scan and name it venv or .venv.
      • Install your project’s dependencies using venv/bin/python -m pip install -r requirements.txt or venv/bin/python -m pip install.
      • If the virtual environment is created outside the project, use one of the ways defined in Virtual environment support to specify the path of the Python virtual environment to Endor Labs.

      For Poetry projects

      • Install your project’s dependencies using poetry install.

      For PDM projects

      • Install your project’s dependencies using pdm sync.

Virtual environment support

For Poetry and PDM, virtual environments endorctl automatically picks the virtual environments.

For pip, you need to use one of the following ways to specify the virtual environment details of your Python projects for both quick and deep scans.

• Set up the virtual environment in the root folder that you want to scan and name it venv or .venv, it is automatically picked up by the Endor Labs application.

export PYTHONPATH=/usr/tmp/venv:/usr/tmp/another-venv

• Set the environment variable ENDOR_SCAN_PYTHON_VIRTUAL_ENV to the path of the virtual environment of your Python project.

export ENDOR_SCAN_PYTHON_VIRTUAL_ENV=/usr/tmp/venv

• Set the environment variable ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES to true to indicate that a virtual environment is not present and Endor Labs can use the system-wide Python installation packages and modules.

export ENDOR_SCAN_PYTHON_GLOBAL_SITE_PACKAGES=true

If you do not set up the virtual environment, Endor Labs attempts to set it up with all the code dependencies, however, we recommend that you install all dependencies in a virtual environment for the most accurate results.

If you are using custom scripts without manifest files to assemble your dependencies, make sure to set up the virtual environment and install the dependencies.

Configure private Python repositories

In addition to scanning public Python projects, Endor Labs provides support to fetch and scan private Python repositories. Endor Labs will fetch the resources from the authenticated endpoints and perform the scan, and you can view the dependencies and findings.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select PyPI and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully. The deep scan performs the following operations for the Python projects.

• Discovers explicitly declared dependencies,
• Discovers project dependent OSS packages present in the venv/global and scope/python.
• Performs reachability analysis and generates call graphs.
• Detects dependencies used in source code but not declared in the package’s manifest files called phantom dependencies.

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans.

Organizations might not own some parts of the software internally and the related findings are not actionable by them. They can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs uses the following two methods to analyze your Python code.

• Dependency resolution using manifest files
• Dependency resolution using static analysis

Endor Labs uses the results from both these methods to perform superior dependency resolution, identify security issues, detect open-source vulnerabilities, and generate call graphs.

Dependency resolution using manifest files

In this method, Endor Labs analyzes the manifest files present in a project to detect and resolve dependencies. The manifest files are analyzed in the following priority.

For Poetry and pip, if both lock and toml files are available, both are analyzed based to detect and resolve dependencies.

For pip, the dependency resolution is as follows, where the first available file in the priority list is analyzed to detect and resolve dependencies, and others are ignored.

On initialization of a scan, Endor Labs identifies the package manager by inspecting files such as the pyproject.toml, poetry.lock, pdm.lock, setup.py, and requirements.txt. When the files, poetry.lock or pyproject.tomlfiles are discovered, Endor Labs will use the Poetry package manager to build the project. When the files, pdm.lock or pyproject.toml files are discovered, Endor Labs will use the PDM package manager. Otherwise, it will use pip3.

Example

This is an example that demonstrates scanning a Python repository from GitHub on your local system using the endorctl scan. Here we are assuming that you are running the scan on a Linux or Mac operating system environment and that you have the following Endor Labs API key and secret stored in the environment variables. See endorctl flags and variables.

• ENDOR_API_CREDENTIALS_KEY set to the API key
• ENDOR_API_CREDENTIALS_SECRET set to the API secret
• ENDOR_NAMESPACE set to your namespace (you can find this when logged into Endor Labs by looking at your URL: https://app.endorlabs.com/t/NAMESPACE/...; it is typically a form of your organization’s name)

pip

git clone https://github.com/HybirdCorp/creme_crm.git
cd creme_crm
python3 -m venv venv
source venv/bin/activate
venv/bin/python3 -m pip install
endorctl scan

Poetry

git clone https://github.com/HybirdCorp/creme_crm.git
cd creme_crm
poetry lock
endorctl scan

PDM

git clone https://github.com/HybirdCorp/creme_crm.git
cd creme_crm
pdm sync
endorctl scan

The scan for this repository is expected to be completed in a few minutes depending on the size of the project. You can now visit app.endorlabs.com, navigate to Projects, and choose the helloflas/flask-examples project to see your scan results.

Dependency resolution using static analysis

All Python projects do not always include manifest files. A project can be a series of install statements that are assembled by custom scripts. Even when manifest files are present, the dependency information and version declared in the manifest file may be drastically different from what is used in a project.

To solve this problem, Endor Labs has developed a unique method for dependency resolution by performing a static analysis on the code, giving you complete visibility of what is used in your code.

• Endor Labs enumerates all Python packages and recognizes the import statements within the project. An import statement is a Python code statement that is used to bring external modules or libraries into your Python script.
• It performs a static analysis of the code to match the import statements with the pre-installed packages and recursively traverses all files to create a dependency tree with the actual versions that are installed in the virtual environment.
• It detects the dependencies at the system level to identify which ones are resolved and retrieves the precise name and version information from the library currently in use.
• Also, it gives you accurate visibility into your project components and helps you understand how the components depend on one another.

Through this approach, Endor Labs conducts comprehensive dependency management, assesses reachability, and generates integrated call graphs.

Known Limitations

• Endor Labs specifically looks for the requirements.txt file for a Python project using pip. If you use a different file name, it won’t be automatically discovered.
• Python versions older than 3.7 are not supported but may work as expected.
• If a virtual environment is not provided, Python version constraints are not assumed based on the runtime environment of CI. Dependencies are shown for all possible versions of Python at runtime. If a virtual environment is provided, Endor Labs respects what is installed in the virtual environment.
• Symbolic links into manifest files may result in the same package being duplicated in the project.
• If a dependency is not available in the PyPI repository or in a configured private package repository, Endor Labs will be unable to build the software and scans may fail without first building the package in the local environment successfully.

Call Graph Limitations

• Function calls using dispatch table calls might not be included in the call graph.
• Function calls using unresolved variables might not be included in the call graph.
• Dynamically modified or extended function calls used to declare methods or attributes at run time might not be included in the call graph.
• Functions called indirectly through a function pointer and not by their direct name, might not be included in the call graph.
• Type stubs that provide hints or type annotations for functions, methods, and variables in your Python modules or libraries have to be installed manually before performing a scan.
• If your project has a pyproject.toml file that includes tools.pyright section, it overrides Endor Labs settings for Pyright and may result in incorrect call graph results. You will need to remove the tools.pyright section from the pyproject.toml file.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Virtual environment errors: You can identify the errors that may occur during virtual environment installation by looking for the following message in the error logs; failed to create virtual environment or failed to install dependencies.
• Missing environment dependency: If your code depends on packages such as psycopg2, environment dependencies such as PostgreSQL are also required. The endorctl scan may fail if the environment where it is running does not have PostgreSQL installed.
• Incompatible Python version: The default Python version in the environment where the endorctl scan is running is incompatible with one or more of the dependencies that are needed by the code.
• Incompatible architecture: One or more dependencies are not compatible with the operating system architecture of the local system on which you are running the endorctl scan. For example, projects with dependency on PyObjC can be run on Mac-based systems, but not Linux systems. A few Python libraries are incompatible with x32 architectures and can only be run on x64 architectures.
• Resolved dependency errors: A version of a dependency does not exist, or it cannot be found. It may have been removed from the repository.
• Call graph errors: These errors come if pip or Poetry are unable to build the project because a required dependency cannot be located.

1.5 - JavaScript/TypeScript

JavaScript is a high-level, interpreted programming language primarily used for creating interactive and dynamic web content widely used by developers. Endor Labs supports the scanning and monitoring of JavaScript projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

• Endor Labs requires the following pre-requisite software to be installed to successfully perform a scan:
  • Yarn: Any version
  • npm: 6.14.18 or higher versions
  • pnpm: 3.0.0 or higher versions
• Make sure your repository includes one or more files with .js or .ts extension.

Build JavaScript projects

You can choose to build your JavaScript projects before running a scan. This will ensure that either a package-lock.json, yarn.lock, or pnpm-lock.yaml file is created enhancing the scan speed.

Ensure your repository has package.json and run the following command making sure it builds the project successfully.

For npm:

npm install

For Yarn:

yarn install

For pnpm:

pnpm install

If the project is not built, endorctl builds the project during the scan and generate either package-lock.json, yarn.lock, or pnpm-lock.yaml file. Make sure that either npm, Yarn, or pnpm is installed on your system. If your repository includes a lock file, endorctl uses the existing file for dependency resolution and does not create it again.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

To detect dependencies when using pnpm:

If you are using pnpm, set the environment variable ENDOR_PNPM_ENABLED to true and then run the scan.

export ENDOR_PNPM_ENABLED=true

endorctl scan

To enable dependency reachability:

To enable dependency reachability that detects dependencies used in source code but not declared in the package’s manifest files, set the flag --call-graph-languages with javascript,typescript.

endorctl scan --call-graph-languages=javascript,typescript

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Dependency analysis tools analyze the lock file of an npm, yarn, or pnpm based package and attempt to resolve dependencies. To resolve dependencies from private repositories, the settings of the .npmrc file in the repository is considered.

Endor Labs surpasses mere manifest file analysis by expertly resolving JavaScript dependencies and identifies:

• Dependencies listed in the manifest file but not used by the application
• Dependencies used by the application but not listed in the manifest file
• Dependencies listed in the manifest as transitive but used directly by the application
• Dependencies categorized as test in the manifest, but used directly by the application

Developers can eliminate the false positives, false negatives, and easily identify test dependencies with this analysis. The dependencies used in source code but not declared in the package’s manifest files are tagged as Phantom.

Endor Labs also supports npm, Yarn, and pnpm workspaces out-of-the-box. If your JavaScript frameworks and packages use workspaces, Endor Labs will automatically take the dependencies from the workspace to ensure that the package successfully builds.

Scan speed is enhanced if the lock file exists in the repository. endorctl does not perform a build and uses the existing files in the repository for analysis.

Known Limitations

• Endor Labs doesn’t currently support local package references
• If a dependency can not be resolved in the lock file, building that specific package may be unsuccessful. This package may have been removed from npm or the .npmrc file is not properly configured. Other packages in the workspace are scanned as usual.
• Call graph generation is not supported for JavaScript

Troubleshoot errors

• Unresolved dependency errors: The manifest file package.json is not buildable. Try running npm install, yarn install, or pnpm install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.6 - Ruby

Ruby is a widely used open-source programming language. Endor Labs supports scanning and monitoring of Ruby projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Software prerequisites

The following prerequisites must be fulfilled:

• All applications monitored by Endor Labs must be on Ruby versions 2.6 or higher.
• A Gemfile or a *.gemspec file must be present in your Ruby project.
• Make sure your repository includes one or more files with .rb extension.

Build Ruby projects

You can choose to build your Ruby projects before running a scan. This will ensure thatgemfile.lock is created.

Ensure your repository has Gemfile and run the following command making sure it builds the project successfully.

bundler install

If the project is not built, endorctl will build the project during the scan and generate Gemfile.lock. If the repository includes a Gemfile.lock, endorctl uses this file for dependency resolution and does not create it again.

Configure Ruby private Gem sources

If you have a private registry and internal dependencies on other projects, you must configure private registries. Endor Labs can fetch the gems that are hosted in a private gem repository which are not available publicly after authenticating the endpoints.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Ruby Gems and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs looks for Gemfile, *.gemspec, and Gemfile.lock files to find and monitor the dependency activity.

• A Gemfile is a configuration file used in Ruby projects to specify the required RubyGems (libraries or packages) for the project’s dependencies.
• A *.gemspec file is a RubyGems specification file used to define the metadata and dependencies for a RubyGem.
• The Gemfile.lock file is automatically generated by Bundler and is used to lock down the specific versions of gems and their dependencies installed in your project’s environment.

If the Gemfile.lock is not present in your project, Endor Labs generates this file and stores it in a temp directory. The file is deleted after extracting dependency information.

Endor Labs’ dependency resolution mechanism assesses multiple factors, including compatibility, stability, and availability, to determine the most suitable version for usage. The resolved dependency version is used during the build or execution of your Ruby project. By utilizing the dependency graph, you can access significant information regarding the dependencies. This includes determining whether a dependency is direct or transitive, checking its reachability, verifying source availability, and more. The dependency graph provides a visual representation that allows you to examine the graphical details of these dependencies.

Known limitations

• Call graphs are not supported for Ruby projects.
• If a dependency can not be resolved in the Gemfile, building that specific package may not be successful. This package may have been removed from the Gem package manager. Other packages in the workspace are scanned.

Troubleshoot errors

• Unresolved dependency errors: The Gemfile is not buildable. Try running bundler install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.7 - .NET

.NET is a free, cross-platform, open-source developer platform for building different types of applications. Endor Labs supports the scanning and monitoring of projects built on the .NET platform.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

The following prerequisites must be fulfilled:

• Make sure your repository includes one or more files with .cs extension.
• One or more *.csproj files must be present in your repository.
• The .NET command or Nuget command must be installed and available on the host system.
• At least one .NET SDK installed on the system must be compatible with the project’s global.json file settings.

Call graph requirements

• You must install .NET 7.0.1 (SDK 7.0.101) or later on the host system.
• Endor Labs’ call graph support for .NET is based on Microsoft’s Common Intermediate Language (CIL). Artifacts such as .exe or .dll files must be available in the project’s standard workspace through a build and restore or a restored cache.

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

You must restore your .NET projects before running a quick scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Run the following commands to resolve dependencies and create the necessary files to scan your .NET project.

To ensure that the build artifacts project.assets.json file is generated and dependencies are resolved run:

dotnet restore

If you use Nuget instead run:

nuget restore

To create a packages.lock.json file if your project uses a lock file run:

dotnet restore --use-lock-file

If project.assets.json or packages.lock.json are not present and if the project is buildable, endorctl will restore the project and create a project.assets.json or a packages.lock.json file to resolve dependencies. 2. You can run a quick scan with the following commands:

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

You must restore and build your .NET projects before running a deep scan. Additionally, ensure that the packages are downloaded into the local package caches and that the build artifacts are present in the standard locations.

1. Run the following commands to restore and build your project. This may vary depending on your project’s configuration.

dotnet restore
dotnet build

2. You can run a deep scan with the following commands:

endorctl scan

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans.

Organizations might not own some parts of the software internally and findings are actionable by another team. These organizations can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --disable-private-package-analysis

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Configure Nuget private repositories

You can configure Endor Labs to fetch dependencies from private NuGet feeds after authenticating the endpoints.

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Nuget and click Manage.
3. Click Add Package Manager.
4. Enter a package manager URL.
5. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the username and password of your private package manager repository.
6. Click Add Package Manager to save your configuration.

Understand the scan process

A *.csproj file is an XML-based C# project file that contains information about the project, such as its source code files, references, build settings, and other configuration details. The dependencies and findings are listed individually for every csproj file. The scan discovers all *.csproj** files and uses these files to resolve the appropriate dependency graph of your project.

(Beta) Endor Labs scans the .NET projects that are using the Central Package Management feature of Nuget for the packages declared as:

• Package references in Directory.Build.props or Directory.Packages.props files.
• Package references in any *.props file and the prop file is imported in the csproj file.
• Package references in *.Targets file.

Endor Labs enriches your dependency graph to help you understand if your dependencies are secure, sustainable, and trustworthy. This includes Endor Labs risk analysis and scores, if a dependency is direct or transitive, and if the source code of the dependency is publicly auditable.

Software composition analysis for .NET is performed in the following ways:

• Using Project.assests.json
• Using package.lock.json

How does dependency resolution happen using Project.assets.json

The Project.assets.json file is used in .NET projects to store metadata and information about the project’s dependencies and assets.

Endor Labs fetches resolved package versions, paths to the dependencies’ assets, such as assemblies and resources, and other related information from this file. If a project does not include a project.assets.json file, it is generated through the dotnet restore or the nuget restore command. This command uses all the configured sources to restore dependencies as well as project-specific tools that are specified in the project file.

How does dependency resolution happen using package.lock.json

The package.lock.json file is used in .NET projects to lock dependencies and their specific versions. It is a snapshot of the exact versions of packages installed in a project, including their dependencies and sub-dependencies, requested versions, resolved versions, and contenthash. The lock file provides a more dependable, uniform, and accurate representation of the dependency graph.

In Endor Labs’ dependency management, the resolution of dependencies is primarily based on package.lock.json, which takes precedence over projects.assets.json to resolve dependencies.

Endor Labs fetches the dependency information from package.lock.json and creates a comprehensive dependency graph. The vulnerabilities associated with the dependencies are listed on the Endor Labs’ user interface.

If the package.lock.json file is not present in the repository, Endor Labs triggers the restore process to generate the package.lock.json file and uses it to perform the dependency scans.

How Endor Labs performs static analysis on the code (Beta)

Endor Labs performs static analysis on the C# code based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs looks for the project’s .dll files typically located within the bin directory.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• The following .NET programming languages are not currently supported for dependency resolution or call graph generation.
  • Projects written in F#
  • Projects written in Visual Basic
• When using the GitHub app, either resolve all the private and internal dependencies, or Configure Nuget private repositories before running a scan.
• Call graph binary discovery may fail if you use environment variables to define the name of the package in your *.csproj file.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors: .NET or Nuget is not installed or not present in the PATH environment variable. Install Nuget and try again.
• Unresolved dependency errors: This error occurs when the .csproj file can not be parsed or if it has syntax errors.

1.8 - Bazel

Bazel is an open-source build and test tool, which is commonly used in monorepos. Bazel is used to quickly build software across multiple languages.

Using Endor Labs, organizations relying on Bazel can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for deep scan

Software prerequisites

Ensure that the following prerequisites are in place for a successful scan.

• One or more WORKSPACE files must be present in your repository.
• The bazel command must be installed and available on the host system. To install Bazel, see the Bazel documentation.
• Bazel versions 5.x.x and 6.x.x are supported.

Select and build your Bazel targets

Endor Labs supports scanning targets using the following language-specific Bazel rules:

• Bazel Rules for Java support included for 4.1 or higher versions.

  • java_library

• Bazel Rules for Python support included for 0.9.0 or higher versions.

  • py_binary
  • py_library

• Bazel Rules for Golang support included for 0.40.1 or higher versions with Bazel versions 5.x.x, 6.x.x. 0.42.0 or higher versions with Bazel versions 7.x.x.

  • go_binary
  • go_library
  • Bazel with Gazelle in vendored mode

• Bazel Rules for Scala support is available for versions 5.0.0 to 6.6.0, including 5.0.0 and 6.6.0.

  • scala_binary
  • scala_library

• To scan with Endor Labs you’ll need to specify a list of targets to scan or a Bazel query for which all targets matching your query criteria are scanned.

• To select your Bazel targets you can search using the Bazel query language for specific targets to scan, or scan all targets that match a given query.

Here are some examples.

• To get all targets created with a java_binary build rule you can use the query:

bazel query 'kind(java_binary, //...)'

• To get all targets created with a py_binary build rule you can use the query:

bazel query 'kind(py_binary, //...)'

• To get all targets created with a go_binary build rule under the golang directory you can use the query:

bazel query 'kind(go_binary, //...)'

• To get all targets created with a scala_binary build rule under the scala directory you can use the query:

bazel query 'kind(scala_binary, //...)'

In general, refine your query to select the most important targets in your monorepo and align with your existing build workflows.

You can choose to build the targets before running the scan. Use the bazel build commands to do this by passing a comma-separated list of targets. For example, //:test and //:test2 run bazel build //:test,//:test2. endorctl will attempt a scan if the targets are not built. endorctl uses bazel build //:test and bazel query 'deps( //:test)' --output graph to build your targets.

Run a scan

Use the following options to scan your repositories. Perform a scan after building the projects. See

Once you’ve selected your targets you can define the targets for scanning using one of three command line arguments:

To include or exclude specific targets, you can pass a comma-separated list of targets to --bazel-exclude-targets or --bazel-include-targets.

To scan a specific list of targets, use the command:

endorctl scan --use-bazel --bazel-include-targets=//your-target-name

To scan a list of targets using the Bazel query language, use the following command and replace the following query with your own.

endorctl scan --use-bazel --bazel-targets-query='kind(java_binary, //...)

If your workspace file is not located at the root of the repository you must define the WORKSPACE file location for the targets you would like to scan.

For example:

endorctl scan --use-bazel --bazel-targets-query='kind(java_binary, //...) --bazel-workspace-path=./src/java

For Go projects, if you use Bazel with Gazelle in vendored mode, you need to provide the path of your go.mod file in the command option, --bazel-vendor-manifest-path.

For example:

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --bazel-vendor-manifest-path=<path to go.mod>

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition. This scan won’t perform reachability analysis to help you prioritize vulnerabilities.

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

You can run a deep scan with the following commands:

endorctl scan --use-bazel --bazel-include-targets-query=//your-target-name

Use the following flags to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --use-bazel --bazel-include-targets-query=//your-target-name -o json | tee /path/to/results.json

When a deep scan is performed all private software dependencies are completely analyzed by default if they have not been previously scanned. This is a one-time operation and will slow down initial scans, but won’t impact subsequent scans.

Organizations might not own some parts of the software internally and findings are actionable by another team. These organizations can choose to disable this analysis using the flag disable-private-package-analysis. By disabling private package analysis, teams can enhance scan performance but may lose insights into how applications interact with first-party libraries.

Use the following command flag to disable private package analysis:

endorctl scan --use-bazel --bazel-include-targets=//your-target-name --disable-private-package-analysis

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

To understand the scan process for Java projects, see Endor Labs for Java.

To understand the scan process for Python projects, see Endor Labs for Python.

To understand the scan process for Go projects, see Endor Labs for GO.

Known limitations

• Scanning Java binaries using Bazel is not supported.

1.9 - Swift/Objective-C

CocoaPods is a popular package manager for Swift and Objective-C. It simplifies the integration of Swift and Objective-C dependencies through Podfile declaration and automated installation.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Software prerequisites

The following prerequisites must be fulfilled:

• All applications monitored by Endor Labs must be on CocoaPods versions 0.9.0 or higher.
• A Podfile and a Podfile.lock must be present in your CocoaPods project.
• Make sure your repository includes one or more files with .swift, .h, or .m extension.

Build CocoaPods projects

If the Podfile.lock is not present in your repository, run the following command to create the Podfile.lock for your Podfile.

pod install

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies. Use the flag --languages=swift along with the endorctl scan to discover Swift and Objective-C dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs looks for the Podfile and Podfile.lock files to discover the dependencies used by an application.

• A Podfile is a configuration file used in CocoaPods projects to specify the required libraries or packages for the project’s dependencies.
• A Podfile.lock file is a CocoaPods specification file used to define the metadata and dependencies.

To successfully discover Swift and Objective-C dependencies, both Podfile and Podfile.lock files must be present in your project for each Podfile.

Known limitations

• Call graphs are not supported for the CocoaPods projects.
• If a Podfile.lock file is not present, Endor Labs will skip analyzing the project and present a warning that the package was skipped.

1.10 - Scala

Scala is a general-purpose and scalable programming language widely used by developers. Endor Labs supports the scanning and monitoring of Scala projects managed by the interactive build tool sbt.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for scan

Make sure that your system has a minimum 8-core processor with 32 GB RAM to successfully scan Scala projects.

Software prerequisites

The following prerequisites must be fulfilled:

• A manifest file for the Scala build tool, build.sbt must be present in your repository.
• Make sure your repository includes one or more files with .sc or .scala extension.
• If your sbt version is lower than 1.4, you must install the sbt-dependency-graph plugin. The sbt-dependency-graph plug-in is by default integrated into the sbt versions 1.4 and higher.
• Make sure that the project/build.properties file has the required sbt version.

Option 1 - Quick scan

You must be able to build your Scala projects before running a scan. The standard .sbt cache must exist and contain successfully downloaded dependencies.

1. Ensure your repository has a build.sbt file and run the following commands making sure it builds the project successfully.

   sbt compile
   

   sbt projects
   

2. Make sure sbt dependencyTree runs successfully inside the project directory.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

You must be able to build your Scala projects before running a scan. The build artifact as well as the standard .sbt cache must exist and contain successfully downloaded dependencies.

1. Ensure your repository has a build.sbt file and run the following commands making sure it builds the project successfully.

   sbt package
   

   sbt projects
   

2. Make sure sbt dependencyTree runs successfully inside the project directory.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs scans Scala projects by executing sbt plugins and inspecting the build.sbt file to retrieve information about direct and transitive dependencies.

• The build.sbt file is a configuration file used in Scala projects with sbt to define project settings, dependencies, and build tasks. This file provides the necessary configuration and instructions to sbt on resolving and managing project dependencies.

• The sbt dependency graph plugin visualizes the dependencies between modules in a Scala project.

Endor Labs analyzes information from both these methods to determine different components, binary files, manifest files, images, and more in the Scala codebase and presents finding policy violations, identifying, and resolving dependencies.

Using Endor Labs users can gain significant insights into the structure and relationships of their Scala project’s dependencies, aiding in managing dependencies effectively, identifying potential issues, and ensuring a well-organized and maintainable codebase.

How Endor Labs performs static analysis on the code

Endor Labs performs static analysis based on the following factors:

• Call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• Software composition analysis for Scala on Microsoft Windows operating systems is currently unsupported.
• Scala packages built with Gradle are not currently supported.

Troubleshoot errors

Here are a few error scenarios that you can check for and attempt to resolve them.

• Host system check failure errors: These errors occur if:
  • If sbt is not installed or present in the path variable. Install sbt 1.4 or higher versions and try again.
  • If the sbt version mentioned in the project or the build.properties file is lower than 1.4 and the sbt-dependency-graph plug-in is not installed. Install the sbt-dependency-graph plug-in and try again.
• Dependency graph errors - Scala by default imports MiniDependencyTreePlugin which is a mini version of the sbt-dependency-graph plugin and it supports only dependencyTree command. To get complete features of the sbt-dependency-graph plugin, add DependencyTreePlugin to your project/plugins.sbt file and run the scan again. See Scala documentation for details.

1.11 - PHP

PHP is a popular server-side scripting language primarily used for web development. Endor Labs supports the scanning and monitoring of PHP projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

Software prerequisites

• One of the following prerequisites must be fulfilled:
  • The PHP project must contain a composer.json file. If the project includes the composer.lock file it is beneficial, but this is not a mandatory requirement.
  • If the composer.lock file is not present in the repository, it is necessary to have PHP and Composer installed before running a scan on your local system.
• Make sure your repository includes one or more files with .php extension.
• The following versions are supported for PHP and Composer:
  • PHP 5.3.2 and higher versions
  • Composer 2.2.0 and higher versions

Build PHP projects

You can choose to build your PHP projects before running a scan. This will ensure that composer.lock is created.

Ensure your repository has composer.json and run the following command making sure it builds the project successfully.

composer install

If the project is not built, endorctl will build the project during the scan and generate composer.lock. If the repository includes a composer.lock, endorctl uses this file for dependency resolution and does not create it again.

Configure private Composer package repositories

If you have a private registry and internal dependencies on other projects, you must configure private registries.

To configure private Composer package repositories:

1. Sign in to Endor Labs and select Integrations under Manage from the left sidebar.
2. From Package Managers, select Packagist and click Manage.
3. Click Add Package Manager.
4. In PACKAGE MANAGER HOST, enter the host domain of the package manager.
5. From the available options, choose the authentication method for private repositories.
6. To enable Endor Labs to authenticate to your registry, select Authenticate to this registry and enter the required details of your private package manager repository.
7. Click Add Package Manager to save your configuration.

Run a scan

Perform a scan to get visibility into your software composition and resolve dependencies.

endorctl scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs discovers all composer.json files in your PHP project and uses these files to resolve the dependencies of your packages. Composer is a PHP dependency management tool that enables you to specify the libraries your project relies on and manages the process of installing or updating them. The dependencies and findings are listed in the Endor Labs application individually for every composer.json file.

In Endor Labs’ dependency management, the resolution of dependencies is based on both composer.json and composer.lock files. The composer.lock file is generated by Composer and includes information such as resolved versions, package information, transitive dependencies, and other details. Using the composer.lock file ensures deterministic dependency installation by recording the exact versions of installed dependencies and their transitive dependencies. If the composer.lock file is not present in the repository, Endor Labs generates the composer.lock file, and uses it to analyze the operational and security risks associated with your package’s dependencies. Endor Labs fetches the dependency information and creates a comprehensive dependency graph.

Known Limitations

Call graphs are not supported for PHP projects.

Troubleshoot errors

• Unresolved dependency errors: The composer.json is not buildable. Try running composer install in the root project to debug this error.
• Resolved dependency errors: A version of a dependency does not exist or it cannot be found. It may have been removed from the repository.

1.12 - Rust

Rust is a software programming language widely used by developers. Endor Labs supports scanning and monitoring of Rust projects.

Using Endor Labs, developers can:

• Test their software for potential issues and violations of organizational policy
• Prioritize vulnerabilities in the context of their applications
• Understand the relationships between software components in their applications

System specifications for scan

Make sure that you have a minimum system requirement specification of an 8-core processor with 32 GB RAM.

Use a system equipped with either Mac OS X or Linux operating systems to perform the scans.

Software prerequisites

• Make sure the following prerequisites are installed: - Package Manager Cargo - Any version - Rust - Any version,
• Make sure your repository includes one or more files with .rs extension.
• Install Rust using the latest rustup tool.

Build Rust projects

Ensure your repository has Cargo.toml file and run the following command making sure it builds the project successfully.

cargo build

If the project is not built, endorctl will build the project during the scan and generate the Cargo.lock file. If the repository includes a Cargo.lock file, endorctl uses this file for dependency resolution and does not create it again.

Run a scan

Use the following options to scan your repositories. Perform the endorctl scan after building the projects.

Option 1 - Quick scan

Perform a quick scan to get quick visibility into your software composition and perform dependency resolution. It discovers dependencies that the package has explicitly declared. If the package’s build file is incomplete then the dependency list will also be incomplete. This scan won’t perform the reachability analysis to help you prioritize vulnerabilities.

endorctl scan --quick-scan

You can perform the scan from within the root directory of the Git project repository, and save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

endorctl scan --quick-scan -o json | tee /path/to/results.json

You can sign into the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Option 2 - Deep scan

Use the deep scan to perform dependency resolution, reachability analysis, and generate call graphs. You can do this after you complete the quick scan successfully.

endorctl comes bundled as a Docker image for portable usage and integration into continuous integration (CI) pipelines. To configure the endorctl Docker image, you must perform the following steps:

1. Install Docker
2. Get Endor Labs API credentials
3. Export your environment variables
4. Scan Rust projects

Export your environment variables

endorctl requires four environment variables for your source control system and your Endor Labs tenant.

These environment variables are:

• ENDOR_API_CREDENTIALS_KEY - The API key used to authenticate against the Endor Labs API.
• ENDOR_API_CREDENTIALS_SECRET - The API key secret used to authenticate against the Endor Labs API.
• ENDOR_NAMESPACE - The Endor Labs namespace you want to scan against. Locate the namespace from the top left-hand corner, under the Endor Labs logo on the Endor Labs application,.
• SOURCE_PATH - The path to your source code that may be mounted to the Docker container.

To export your environment variables run the following commands and insert the environment variable values in each command.

export ENDOR_NAMESPACE=name-of-your-namespace
export ENDOR_API_CREDENTIALS_KEY=endorlabs-api-key
export ENDOR_API_CREDENTIALS_SECRET=endorlabs-api-secret
export SOURCE_PATH=/path/to/your/source/code

Scan Rust projects

To scan your rust project run the following command:

docker run -it --rm \
 -e ENDOR_NAMESPACE=$ENDOR_NAMESPACE \
 -e ENDOR_API_CREDENTIALS_KEY=$ENDOR_API_CREDENTIALS_KEY \
 -e ENDOR_API_CREDENTIALS_SECRET=$ENDOR_API_CREDENTIALS_SECRET \
 -v $SOURCE_PATH:/root/endorlabs \
 us-central1-docker.pkg.dev/endor-ci/public/endorctl:latest \
 scan --path=/root/endorlabs

Use the following command to save the local results to a results.json file. The results and related analysis information are available on the Endor Labs user interface.

docker run -it --rm \
 -e ENDOR_NAMESPACE=$ENDOR_NAMESPACE \
 -e ENDOR_API_CREDENTIALS_KEY=$ENDOR_API_CREDENTIALS_KEY \
 -e ENDOR_API_CREDENTIALS_SECRET=$ENDOR_API_CREDENTIALS_SECRET \
 -v $SOURCE_PATH:/root/endorlabs \
 us-central1-docker.pkg.dev/endor-ci/public/endorctl:latest \
 scan --path=/root/endorlabs -o json | tee /path/to/results.json

You can sign in to the Endor Labs user interface, click the Projects on the left sidebar, and find your project to review its results.

Understand the scan process

Endor Labs performs the following steps when it scans Rust projects:

• Resolves dependencies for the package version
• Performs static analysis on your Rust code

Resolving Dependencies & Static Analysis for Rust

Endor Labs leverages the Cargo.toml file in Rust and uses this file to build the package version using cargo. Endor Labs uses the output from cargo metadata to resolve dependencies specified in Cargo.toml files and construct the dependency graph.

Perform Static Analysis on Your Rust Code

• After building the package, call graphs are created for your package. These are then combined with the call graphs of the dependencies in your dependency tree to form a comprehensive call graph for the entire project. Use the call graphs to understand if vulnerabilities in your Rust code are reachable through a function associated with the known vulnerability.
• Endor Labs performs an inside-out analysis of the software to determine the reachability of dependencies in your project.
• The static analysis time may vary depending on the number of dependencies in the package and the number of packages in the project.

Known Limitations

• To perform static analysis on Rust packages, the package version must be able to be successfully built.
• Performing Endor Labs scans on the Microsoft Windows operating system is currently unsupported.

Troubleshoot errors

• Host system check failure errors: These errors occur when Rust is not installed or not present in the path variable. Install Rust and try again.
• Call graph errors: These errors occur when the project is not buildable and the required dependencies cannot be located.

2 - Scanning strategies

As you deploy Endor Labs in your environment, it’s important for your team to understand key scanning strategies.

Set a default branch

The findings, metrics, and data shown on the dashboard and the project listing page are based on scanning the default branch, which is also known as the main context.

endorctl scan --as-default-branch

If you do not set the flag as-default-branch, the first branch you scan is automatically considered as the default branch. After a scan, if you switch the default branch to another using --as-default-branch, scans from the previous branches are erased, and their findings will no longer be available.

You do not need to set a default branch if you are using the Endor Labs GitHub App or not scanning multiple branches.

Testing and monitoring different versions of your code

Across the software engineering lifecycle its important that continuous testing is separated from what is monitored and reported on regularly. Often, engineering organizations want to test each and every change that enters a code base, but if security teams reported on each test they would quickly find themselves overwhelmed with noise. Endor Labs enables teams to separate what should be reported on relative to what should be tested but not reported on. Endor Labs allows teams to select reporting strategies for their software applications when integrated into CI/CD pipelines.

Here are the primary scanning and reporting strategies:

• Reporting on the default branch - All pull request commits are tested and all pushes or merges to the default branch are reported on and monitored by security and management teams.
• Reporting on the latest release - All reporting and monitoring is performed against tagged release versions. This requires each team have a mature release tagging strategy.

How to deploy a strategy for reporting

The endorctl scan command by default will continuously monitor a version of your code for new findings such as unmaintained, outdated or vulnerable dependencies in the bill of materials for a package. To test a version of your code without monitoring and reporting on it, use the flag --pr or environment variable ENDOR_SCAN_PR as part of your scan.

When adopting a strategy such as reporting on the default branch, you will want to run any push or merge event to the default branch without the --pr flag and run any pull_request or merged_request event with the --pr flag. This allows you to test changes before they have been approved and report what has been merged to the default branch as your closest proxy to what is in production.

Lets use the following GitHub actions workflow as an example! In this workflow any push event will be scanned without the --pr flag but any pull_request event is scanned as a point in time test of that specific version of your code.

name: Endor Labs Scan
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload sarif artifact to GitHub
      contents: read # Used to check out a private repository but actions/checkout.
      actions: read # Required for private repositories to upload sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication to Endor Labs
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Build Package
        run: mvn clean install
      - name: Endor Labs Scan Pull Request
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
          pr_baseline: $GITHUB_BASE_REF
      - name: Endor Labs Reporting Scan
        if: github.event_name == 'push'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: false
          sarif_file: 'findings.sarif'
      - name: Endor Labs Testing Scan
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
      - name: Upload findings to github
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'findings.sarif'

Scanning detached refs

In some CI/CD based environments, each time code is pushed to the default branch the exact commit SHA is checked out as a detached Git Reference. This is notably the case with Jenkins, CircleCI and GitLab Pipelines.

In these scenarios, on push or merge events Endor Labs must be told that the reference should be monitored as the default branch. You can do this with the --detached-ref-name flag or ENDOR_SCAN_DETACHED_REF_NAME environment variable. You should also couple this flag with the --as-default-branch flag or ENDOR_SCAN_AS_DEFAULT_BRANCH environment variable. This allows you to set this version of code as a version that should be monitored as well as define the name associated with the branch.

This strategy may be used for both a strategy reporting on the default branch on push events and a strategy reporting on tag creation event for that version of code.

You can see in the below GitLab Pipelines example defining the logic to manage a detached reference on GitLab.

    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi

You can find the full GitLab pipelines reference below:

Endor Labs Dependency Scan:
  stage: Scan
  image: node # Modify this image to align with the build tools nessesary to build your software packages
  dependencies: []
  variables:
    ENDOR_ENABLED: "true"
    ENDOR_ALLOW_FAILURE: "true"
    ENDOR_NAMESPACE: "demo"
    ENDOR_PROJECT_DIR: "."
    ENDOR_ARGS: |
      --path=${ENDOR_PROJECT_DIR}
      --show-progress=false
      --detached-ref-name=$CI_COMMIT_REF_NAME
      --output-type=summary
      --exit-on-policy-warning
      --dependencies --secrets --git-logs      
  before_script:
    - npm install yarn
  script:
    - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl;
    - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
      if [ $? -ne 0 ]; then
       echo "Integrity check failed";
       exit 1;
      fi
    - chmod +x ./endorctl
    - if [ "$DEBUG" == "true" ]; then
        export ENDOR_LOG_VERBOSE=true;
        export ENDOR_LOG_LEVEL=debug;
      fi
    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi
    - ./endorctl scan ${ENDOR_ARGS}
  rules:
  - if: $ENDOR_ENABLED != "true"
    when: never
  - if: $CI_COMMIT_TAG
    when: never
  - if: $CI_COMMIT_REF_NAME != $CI_DEFAULT_BRANCH && $ENDOR_FEATURE_BRANCH_ENABLED != "true"
    when: never
  - if: $ENDOR_ALLOW_FAILURE == "true"
    allow_failure: true
  - if: $ENDOR_ALLOW_FAILURE != "true"
    allow_failure: false

Implementing baseline scans

One of the common concerns software development teams have when adopting preventative controls is ownership of issues. Often, software has accrued significant technical debt, or new vulnerabilities arise that don’t directly impact their changes. Security teams want to have all known issues addressed while the development teams are focused on fixing issues or delivering core business value. They can’t be hindered each time a new issue impacts their entire code base.

To prevent new issues from entering the environment, security teams sometimes set policies that may break the build or return a non-zero exit code that can fail automated tests. This creates friction as there is no context around what changes a developer is responsible for versus what technical debt exists in a codebase on that day.

Establishing a baseline of what issues already exist in a software project and what issues may occur because of new updates is crucial to enabling preventative control adoption.

Accelerating preventative control adoption with CI baselines

The high-level steps to establish and measure policies against a baseline scan are as follows:

1. Establish a baseline scan of your default branch or any other branch that undergoes regular testing
2. Integrate baseline scans into your automated workflows
3. Evaluate policy violations within the context of the branches to which you routinely merge

Implementing baseline scan into your program

Development teams often have different delivery strategies. Some merge changes to a default branch. Others merge to a release branch that is then released to their environment. While these strategies differ across organizations, a baseline scan must exist to measure against attribute ownership.

To establish a baseline scan, your team must perform regular scans on the branch to which you merge. This often means that you scan each push of your default branch to monitor your environment and you test each pull request using the --pr and --pr-baseline flags.

The --pr flag is a user’s declaration that they are testing their code as they would in a CI pipeline. The --pr-baseline flag tells Endor Labs which Git reference to measure any changes.

For this example, we will use the default branch as a merging strategy. In this strategy, you’ll want to scan the default branch on each push event to re-establish your baseline. You’ll also want to establish your CI baseline as the default branch.

The following GitHub workflow illustrates this strategy:

• See the GitHub Actions documentation for GitHub default variables.
• See the GitLab CI/CD documentation for the GitLab default variables.

name: Endor Labs Scan
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload sarif artifact to GitHub
      contents: read # Used to check out a private repository but actions/checkout.
      actions: read # Required for private repositories to upload sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication to Endor Labs
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v3
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Build Package
        run: mvn clean install
      - name: Endor Labs Scan Pull Request
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
          pr_baseline: $GITHUB_BASE_REF
      - name: Endor Labs Reporting Scan
        if: github.event_name == 'push'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: false
          sarif_file: 'findings.sarif'
      - name: Endor Labs Testing Scan
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.1
        with:
          namespace: 'example'
          pr: true
          sarif_file: 'findings.sarif'
      - name: Upload findings to github
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'findings.sarif'

3 - Perform incremental scans

Perform incremental scans to scan the parts of a codebase that have changed since the last complete scan, rather than scanning the entire codebase every time. This process is efficient and faster, especially for large projects, because it focuses on new or modified code that might introduce new vulnerabilities or issues.

How does incremental scan work

To initiate incremental scans, first perform a complete scan to set a baseline. After establishing the baseline, you can start incremental scans to compare against it. The first scanned branch is considered the baseline, and this branch is continuously monitored. See set a default branch.

Ensure the baseline scan is successful and error-free. A successful complete scan will resolve dependencies, run analytics, and generate call graphs for supported languages.

During an incremental scan, Endor Labs first identifies the packages and their dependencies within a project. If a package and its dependencies remain unchanged, the rest of the scanning process is skipped, and a message No changes found in package is displayed along with the package name. If any changes are detected in the packages or dependencies, the scan proceeds with only the modified packages, and prints their results.

Incremental scanning is not performed if errors are found while resolving dependencies, if the project’s path has changed, or if there are failures within the project’s packages. In these cases, the system will automatically perform a complete scan.

Run an incremental scan

1. First, perform a complete scan.

endorctl scan

2. Use the --pr-incremental argument to perform an incremental scan, detecting and scanning dependencies that have changed since the baseline scan. This scan will generate call graphs only for the modified packages. See endorctl scan command arguments.

endorctl scan --pr --pr-baseline=main --pr-incremental

4 - Scoping scans

Exclude and include filters help your team to focus their attention on the open source packages that matter most and to improve scan performance. Use inclusion patterns when you have many packages that you want to scan separately and exclusion patterns when you want to filter out packages that are not important to you.

You can include or exclude packages using the following standard patterns:

1. Include or exclude specific packages.
2. Include or exclude specific directories.
3. Include or exclude with a Glob style expressions.
4. Use include and exclude patterns together to exclude specific directories such as a test directory from a scan.
5. Use multiple include and exclude patterns together to exclude or include specific directories or file paths.

To include or exclude a package based on its file name:

• Include path
• Exclude path

endorctl scan --include-path="path/to/your/manifest/file/package.json"

endorctl scan --exclude-path="path/to/your/manifest/file/package.json"

To include or exclude a package based on its directory

• Include Directory
• Exclude Directory

endorctl scan --include-path="directory/path/**"

endorctl scan --include-path="src/java/**"

endorctl scan --exclude-path="path/to/your/directory/**"

endorctl scan --exclude-path="src/ruby/**"

Examples

• Use --exclude-path="src/java/**" to exclude all files under src/java, including all its sub-directories.
• Use --exclude-path=src/java/* to only exclude the files under src/java, but not its sub-directories.
• Use --include-path and --exclude-path together to exclude specific directories such as test directories. For example,

endorctl scan --include-path="src/java/**" --exclude-path="src/java/test/**"

• Use multiple inclusion patterns together. For example,

endorctl scan --quick-scan --include-path="src/java/**" --include-path="src/dotnet/**"

• Use multiple exclusion patterns together. For example,

endorctl scan --include-path="src/java/**" --exclude-path="src/java/gradle/**" --exclude-path="src/java/maven/**"

Best practices

Here are a few best practices:

• Ensure that you enclose your exclude pattern in double quotes to avoid shell expansion issues. For example, do not use --exclude-path=src/test/**, instead, use --exclude-path="src/test/**".
• Inclusion patterns are not designed for documentation or example directories. You can not explicitly include documentation or example directories:
  • docs/
  • documentation/
  • groovydoc/
  • javadoc
  • man/
  • examples/
  • demos/
  • inst/doc/
  • samples/
• The specified paths must be relative to the root of the directory.
• If you are using Javascript workspaces, take special consideration when including and excluding the root package:
  • When using include or exclude patterns, it’s crucial to make sure you never exclude and always include the parent workspace package. Otherwise, all child packages won’t build properly.
  • You can always exclude child packages in the workspace if the root is included.
  • There is only one lock file for the workspace that exists in the workspace root directory. Make sure to include the lock file to perform a successful scan.

5 - Working with monorepos

Large monorepos are a reality for many organizations. Since monorepos can have anywhere from tens to even hundreds of packages scanning all packages in a monorepo can take significant periods of time. While the time requirements may vary based on your development team and pipeline times, in general, development teams need quick testing times to improve their productivity while security teams need full visibility across a monorepo. These two needs can conflict without performance engineering or an asynchronous scanning strategy. This documentation outlines some performance engineering and scanning strategies for large monorepos.

Asynchronous scanning strategies

When scanning a large monorepo, a common approach taken by security teams is to run an asynchronous cron job outside a CI/CD-based environment. This is often the point of least friction but is prohibitive. With this approach, inline blocking of critical issues is not generally possible. We would be remiss not to mention this as a scanning strategy for monorepos but this approach is NOT recommended beyond a step to get initial visibility into a large monorepo.

Performance Enhancements for inline scanning strategies

The following performance enhancements may be used with Endor Labs to enable the scanning of large monorepos:

Scoping scans based on changed files

For many CI/CD systems path filters are readily available. For example, with GitHub actions, dorny path filters is a readily accessible way to establish a set of filters by a path. This is generally the most effective path to handle monorepo deployments but does require the highest level of investment in terms of human time. The human time investment is made up for by the time saved by reducing the need to scan everything on each change.

Based on the paths that change you can scope scans based on the files that have actually changed. For example, you can scan only the packages in a monorepo that are housed under the ui/ directory when this path has changed by running a scan such as endorctl scan --include-path=ui/ when this path has been modified.

Using a path filtering approach each team working in a monorepo would need to be responsible for the packages that they maintain, but generally, each team may be associated with one to several pre-defined directory paths.

Parallelizing scans for many packages

When scanning a large monorepo organizations can choose to regularly scan the whole monorepo based on the packages or directories they’d like to scan. Different jobs may be created that scan each directory simultaneously.

Parallelizing with scoped scans

Using scoped scans for monorepos with multiple parallel include patterns is a common performance optimization for monorepos.

Below is an example parallel GitHub action scan that can be used as a reference. In this example, the directories ui/ and backend/ are both scanned simultaneously and the results are aggregated by Endor Labs.

This approach can improve the overall scan performance across a monorepo where each directory can be scanned independently.

name: Parallel Actions
on:
  push:
    branches: [main]
jobs:
  scan-ui:
    runs-on: ubuntu-latest
    steps:
      - name: UI Endor Labs Scan
        run: endorctl scan --include-path=ui/
  scan-backend:
    runs-on: ubuntu-latest
    steps:
      - name: Backend Endor Labs Scan
        run: endorctl scan --include-path=backend/

To include or exclude a package based on its directory.

endorctl scan --include-path="directory/path/"

See scoping scans for more information on approaches to scoping scans.

Parallelizing across languages

For teams that work out of smaller monorepos, it is often most reasonable to parallelize scanning based on the language that is being scanned and performance optimize for individual languages based on need.

Below is an example parallel GitHub action scan that can be used as a reference. In this example, JavaScript and Java are scanned at the same time and aggregated together by Endor Labs. This approach can improve the overall scan performance across a monorepo with multiple languages.

name: Parallel Actions
on:
  push:
    branches: [main]
jobs:
  scan-java:
    runs-on: ubuntu-latest
    steps:
      - name: Java Endor Labs Scan
        run: endorctl scan --languages=java
  scan-javascript:
    runs-on: ubuntu-latest
    steps:
      - name: Javascript Endor Labs Scan
        run: endorctl scan --languages=javascript,typescript

To scan a project for only packages written in typescript or javascript use the command:

endorctl scan --languages=javascript,typescript

To scan a project for only packages used for packages written in java use the command:

endorctl scan --languages=java

6 - Repository Security Posture Management

Secure critical components of your software supply chain, including code, open source libraries to ensure the security posture of your software development code and its repositories.

• Endor Labs comes with out-of-the box finding policies that help you detect misconfigurations, enforcing coding best practices, and staying compliant with industry standards such as CIS benchmarks for GitHub and more.

• Review the available finding RSPM policy templates.

• Endor Labs regularly updates its existing policies and also includes several new policies. Configure policy settings to ensure that you benefit from these regular updates.

• The policies provide up-to-date insights into critical risks, so you can manage security threats before your projects even start. They also include remediation advice that can help you fix and mitigate issues.

7 - Rescan projects

Endor Labs enables you to rescan your GitHub projects. When you make a code change or upgrade a dependency, rescanning your GitHub projects ensures the integrity and security of your software.

To enable periodic scanning of your GitHub projects, install the GitHub App from Endor Labs. For more information, see Install the GitHub App.

Endor Labs automatically triggers a rescan of your GitHub projects every 24 hours. However, you can manually trigger a rescan. Follow these steps:

1. Sign in to Endor Labs and click Manage Projects.

2. Select a project configured for automated scanning using the GitHub App.

3. Click Rescan Project to start rescanning.

   

8 - Perform scans in CI pipelines

endorctl is a command line utility designed to bring the functionality of Endor Labs into your software delivery workflows.endorctl has several command flags to help you facilitate operational and security risk monitoring. Developers can integrate Endor Labs into Continuous Integration Workflows using the endorctl scan.

• endorctl scan - You can use endorctl scan to monitor your projects using Endor Labs, and you can update the scan information each time to keep monitoring the project for new findings. The endorctl scan command will scan a specific version of your repository, such as the default branch, a tagged release version, or a commit SHA.
• endorctl scan --pr - You can use the endorctl scan --pr command to scan a specific version of your source code for security and operational risks as part of your continuous integration workflows or CI runs. The endorctl scan --pr command performs a one-time evaluation of your project, focusing on security and operational risks, rather than providing continuous monitoring. CI runs are shown in the Scan History section of each project and are stored for 30 days so that you can analyze and review them on the Endor Labs user interface.

Any continuous integration workflows generally run using the endorctl scan --pr command unless a scan is run on a created tag release, a push to the default or specific branch, or a commit SHA that will be deployed to production.

Authenticating in CI with Keyless Authentication

Keyless Authentication enhances security and minimizes the expenses associated with secret rotation. Keyless authentication is Endor Labs recommended path to scan your projects in the CI workflows. See Keyless Authentication for details.

Configuring GitHub Actions workflows with Endor Labs

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. See the Endor Labs Configuring GitHub Actions workflows. for details.

Configuring GitLab CI/CD pipelines with Endor Labs

GitLab CI/CD pipelines are a part of GitLab’s integrated continuous integration and deployment features. They allow you to define and automate the different stages and tasks in your software development workflow. See the Endor Labs Configuring GitLab CI/CD for details.

8.1 - Set up keyless authentication

At Endor Labs, we believe that the most secure secret is one that doesn’t exist. That’s why in CI/CD environments we recommend using keyless authentication for machine authentication. Keyless Authentication uses OAuth for API authentication and removes the need to maintain and rotate an API key to Endor Labs.

Keyless Authentication is more secure and reduces the cost of secret rotation.

Configure keyless authentication using:

• Google Cloud
• GitHub OIDC
• AWS Cloud

Enabling Keyless Authentication in Google Cloud

To enable Keyless Authentication in GCP you’ll first need permissions to create service accounts and assign these accounts roles to GCP.

The workflow to enable keyless authentication is:

1. Create a service account with no permissions for federation.
2. If you do not attach a service account to compute resources or use the default service account, we recommend creating a new service account for the compute resources. Create a service account to attach to compute resources and impersonate the federation service account.
3. Create an authorization policy to allow the federation service account to authenticate to Endor Labs.
4. Provision the compute resources with the appropriate permissions.
5. Test Keyless authentication.

Creating GCP Service Accounts and Authorization Policies

To create your service accounts, first export your GCP project name as an environment variable:

export PROJECT=<insert-gcp-project>

Once you’ve set the environment variable you’ll create a service account that will be used to federate access to Endor Labs and will be provided permission to access the Endor Labs APIs:

Step 1: Create a federation service account called endorlabs-federation:

gcloud iam service-accounts create endorlabs-federation --description="Endor Labs Keyless Federation Service Account" --display-name="Endor Labs Federation Service Account"

We’ll also create a second service account, that will have access to impersonate the endorlabs-federation account. We’ll call this endorlabs-compute-service.

This is needed if you don’t already have service accounts for your compute resources. If you do, you need to modify the existing permissions to allow the existing service account to create a federation token.

Step 2: Create a keyless authentication service account to assign to compute resources called endorlabs-compute-service:

gcloud iam service-accounts create endorlabs-compute-service --description="Endor Labs Service account for keyless authentication" --display-name="Endor Labs Compute Instance SA"

Finally, we’ll assign endorlabs-compute-service permissions to impersonate the endorlabs-federation account to authenticate to Endor Labs through OIDC.

Step 3: Assign the serviceAccountOpenIdTokenCreator role to the endorlabs-compute-service service account:

gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com" --role="roles/iam.serviceAccountOpenIdTokenCreator"

Once we’ve created the necessary account permissions we will create an authorization policy in Endor Labs to allow the account endorlabs-federation to your Endor Labs tenant:

Use the following command to create an authorization policy in Endor Labs.

Note: Make sure to replace <your-tenant> with your Endor Labs tenant name and <insert-your-project> with your GCP project name in the following command.**

endorctl api create -r AuthorizationPolicy -d '{
    "tenant_meta": { "namespace": "<your-tenant>" },
    "meta": {
        "name": "Keyless Auth",
        "kind": "AuthorizationPolicy",
        "tags": ["gcp"]
     },
     "spec": {
        "clause": ["email=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com", "gcp"],
        "target_namespaces": ["<your-tenant>"],
        "propagate": true,
        "permissions": {
            "rules": {},
            "roles": [
                "SYSTEM_ROLE_CODE_SCANNER"
            ]
        },
    }
}'

You’ve now set up the foundation of keyless authentication. You’ll now need to provision your compute resources with the appropriate GCP scopes and service account.

See Provisioning and Testing Keyless Authentication for GKE workloads for instructions on setting up GKE for keyless authentication.

See Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances for instructions on setting up a virtual machine instance for keyless authentication.

Provisioning and Testing Keyless Authentication for GKE workloads

Prerequisites

The following prerequisites are required to setup keyless authentication on GKE workloads:

• Workload identity is enabled on the target GKE cluster. See the GCP documentation on using workload identity for instructions on migrating existing cluster node pools or creating new clusters to use GCP workload identity.
• The gcloud auth plugin is installed and operational on your machine. See the GCP instructions on enabling the gcloud auth plugin for more details.
• The kubectl CLI is installed. See the Kubernetes documentation for instructions.

Procedure

1. (Optional) Create a GKE cluster with workload identity enabled if you do not already have a GKE cluster
2. Authenticate to the GKE cluster
3. (Optional) Create a namespace for Endor Labs scans
4. Create a Kubernetes service account to impersonate your GKE compute service account
5. Bind your Kubernetes service account to your GCP compute service account
6. Annotate your Kubernetes service account with your GCP service account to complete your binding
7. Test a scanning workload using keyless authentication

Setting Up and Testing Keyless authentication in GKE

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT
• The GKE cluster as CLUSTER_NAME

export PROJECT=<Insert_GCP_Project>
export CLUSTER_NAME=<GKE_CLUSTER>

Optional Step 1: To create a GKE cluster with workload identity enabled if you do not already have a GKE cluster with workload identity enabled run the following command:

gcloud container clusters create keyless-test --workload-pool=endor-github.svc.id.goog --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To authenticate to your GKE cluster run the following command:

gcloud container clusters get-credentials $CLUSTER_NAME

Optional Step 3: To create a namespace for Endor Labs scans run the following command:

kubectl create namespace endorlabs

Step 4: To create a Kubernetes service account to impersonate your GKE compute service account run the following command:

kubectl create serviceaccount endorlabs-compute-service -n endorlabs

Step 5: To bind your Kubernetes service account to your GCP compute service account run the following command:

Note: Make sure to replace <insert-your-project> in the following command with your GCP project name.

gcloud iam service-accounts add-iam-policy-binding endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:<insert-your-project>.svc.id.goog[endorlabs/endorlabs-compute-service]"

Step 6: To annotate your Kubernetes service account with your GCP service account to complete your binding run the following command:

Note: If you have created a different service account name replace endorlabs-compute-service with the appropriate service account name.

kubectl annotate serviceaccount endorlabs-compute-service -n endorlabs iam.gke.io/gcp-service-account=endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com

Step 7: Test scan your project with keyless authentication

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication.

Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances

The following high-level procedure describes the required steps to use keyless authentication with a GCP virtual machine instance:

Procedure:

1. Create a virtual machine instance with the appropriate scopes
2. Download and install endorctl on the virtual machine instance
3. Launch a test scan with keyless authentication

Setting up and Testing Keyless authentication on a GCP virtual machine instance

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT

export PROJECT=<Insert_GCP_Project>

To successfully test keyless authentication first you’ll need to provision a compute resource with the service account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com and the scope https://www.googleapis.com/auth/cloud-platform:

Step 1: To create a virtual machine instance with the appropriate scopes run the following command:

gcloud compute instances create test-keyless --service-account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To download and install endorctl on the virtual machine instance run the following series of commands:

First, SSH to the virtual machine instance you’ve created to test:

gcloud compute ssh --zone "us-west1-b" "test-keyless"  --project $PROJECT

Then download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com for federated access to Endor Labs such as in the below example:

endorctl api list --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com -r Project -n <insert-your-tenant> --count

If this scan runs successfully you’ve tested and scanned a project with keyless authentication to Endor Labs.

Enabling Keyless Authentication in GitHub

To enable Keyless Authentication for GitHub Actions, you’ll need to perform the following steps:

1. Ensure you are using the Endor Labs GitHub Action in your GitHub workflow.
2. Edit your GitHub Action workflow to add permission settings for the GitHub id-token
3. Create an authorization policy for GitHub Action OIDC
4. Test that you can successfully scan a project using Github Action OIDC

Add a GitHub Action OIDC authorization policy

To ensure that the GitHub action OIDC identity can successfully login to Endor Labs you’ll need to create an authorization policy in Endor Labs.

To create an authorization policy:

1. Under Manage go to Access Control
2. Navigate to the “Auth Policy” tab
3. Click on the “Add Auth Policy” button
4. Select “GitHub Action OIDC” as your identity provider
5. Select the permission for the GitHub Action. This permission should be “Code Scanner”
6. For the claim use the key user and put in a matching value that maps to the organization of your GitHub repository.

Configuring your GitHub Action workflow

To configure your GitHub Action workflow with GitHub Action OIDC you can use the following example as a baseline.

The important items in this workflow are:

1. The Usage of the Endor Labs GitHub action.
2. Setting Job level permissions to allow writing to the GitHub id-token

name: Example Scan of OWASP Java
on: workflow_dispatch
jobs:
  create_project_owasp:
    permissions:
      id-token: write # This is required for requesting the JWT
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3
        with:
          repository: OWASP-Benchmark/BenchmarkJava
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Compile Package
        run: mvn clean install
      - name: Scan with Endor Labs
        uses: endorlabs/github-action@main # This workflow uses the Endor Labs GitHub action to scan.
        with:
          namespace: 'demo'
          scan_summary_output_type: 'json'
          pr: false
          scan_secrets: true
          scan_dependencies: true

Now that you’ve successfully configured your GitHub action workflow file you can use this workflow file or one of your own designs to run a test scan using Keyless authentication for GitHub actions.

Enabling Keyless Authentication in AWS

To enable Keyless Authentication in AWS you’ll first need permissions to create or modify the following roles and an instance profile with the appropriate roles configured.

1. An instance access role - The instance access role is assigned to the compute resource, which needs to access Endor Labs. Your instance access role may already exist and you must ensure this role provides the permissions to allow the role to assume the role of a dedicated federation role.
2. A dedicated federation role - The dedicated federation role should have no permissions in AWS. Endor Labs will authorize requests that come from this role.

Procedure:

1. Create or modify an existing Instance Profile that may be used to assign a role to your EC2 instance.
2. Create or modify a role an instance access role, which enables services to assume a dedicated federation role.
3. Assign this role to the Instance Profile
4. Create a dedicated federation role to provide access to Endor Labs, which will be assumed by the instance access role
5. Create an authorization policy in Endor Labs
6. Test Keyless Authentication

Create or select an Instance Profile

An instance profile allows users to attach a single role to an EC2 instance. If you do not already have a pre-defined role or instance profile used by your EC2 instances you should create an instance profile for Endor Labs access.

To create an instance profile using the AWS CLI:

aws iam create-instance-profile --instance-profile-name EndorLabsAccessProfile

Create or modify an instance access role

To successfully authenticate to Endor Labs you will need to assign an instance access role to the instance profile you created above.

The instance access role must at a minimum allow the compute resources that require access to perform the action sts:AssumeRole. If you already have a role you intend to assign to your instance profile, ensure that it has permissions to allow your compute resources to perform this action.

If you do not have an existing role you intend to use, create the role named endorlabs-instance-access-role using the instructions below.

Add the following json to a file called endorlabs-instance-access-role.json

cat > endorlabs-instance-access-role.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Use this file to create your instance access role using the following command:

aws iam create-role --role-name endorlabs-instance-access-role --assume-role-policy-document file://endorlabs-instance-access-role.json

Next, ensure that the instance access role is assigned to the instance profile using the following command:

aws iam add-role-to-instance-profile --instance-profile-name EndorLabsAccessProfile --role-name endorlabs-instance-access-role

Finally, create your EC2 instance and ensure that your instance profile is assigned to it.

Create a dedicated federation role

A dedicated federation role is leveraged to provide a least privileged role that enables access to Endor Labs. This role is designed to be assumed only by specific other roles and does not provide access to AWS resources.

To create your federation role you will need the name and AWS account number of the instance access role, which should look similar to the following policy json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$ACCOUNT:role/$ROLE_NAME"
            },
            "Action": "sts:AssumeRole"
        }
    ]
  }

First, get the account number of the role:

export ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')

Then define the name of the instance access role. For the following example, we will assume it is endorlabs-instance-access-role.

export ROLE_NAME=endorlabs-instance-access-role

Next, create the IAM policy document.

cat > endorlabs-federation-aws-role.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::${ACCOUNT}:role/${ROLE_NAME}"
          },
          "Action": "sts:AssumeRole"
      }
  ]
}
EOF

Next, apply the policy document as a role:

aws iam create-role --role-name endorlabs-federation --assume-role-policy-document file://endorlabs-federation-aws-role.json

Finally, fetch the ARN of the IAM role you’ve created using the following command and create an authorization policy for it in Endor Labs.

To fetch the ARN of the Endor Labs federation role use the following command:

aws iam list-roles | jq -r '.Roles[] | select(.Arn|contains("endorlabs-federation"))'.Arn

To add the authorization policy to Endor Labs:

1. Login to Endor Labs as an administrator.
2. Under Manage, navigate to Access Control > Auth Policy
3. Click Add Auth Policy
4. Under Identity Provider Select AWS role
5. Provide the appropriate permissions for your authorization policy.
6. Under claims use the Key User and the value of the ARN that you fetched in the previous command.
7. Click Save Auth Policy to finalize your keyless authentication setup.

Testing Keyless Authentication with AWS

On the EC2 instance you’ve configured for keyless authentication, download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --aws-role-arn=<insert-your-arn> for federated access to Endor Labs such as in the below example:

endorctl --aws-role-arn=<insert-your-arn> api list -r Project -n <insert-your-endorlabs-tenant> --page-size=1

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication with AWS.

8.2 - Scanning with GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can use GitHub Actions to include Endor Labs into your CI pipeline seamlessly.

Using this pipeline, developers can view and detect:

• Policy violations in the source code
• Secrets inadvertently included in the source code

The Endor Labs verifications are conducted as automated checks and help you discover violations before pushing code to the repository. Information about the violations can be included as comments on the corresponding pull request (PR). This enables developers to easily identify issues and take remedial measures early in the development life cycle.

• For policy violations, the workflow is designed to either emit a warning or return an error based on your action policy configurations.
• For secrets discovered in the commits, developers can view the PR comments and take necessary remedial measures.

To start using Endor Labs with GitHub:

• Install Software Prerequisites
• Example GitHub Action Workflow
• Authenticate with Endor Labs
  • Authentication Without Keyless Authentication for GitHub
• Endor Labs GitHub Action Configuration Parameters
  • Common parameters
  • Scanning parameters
• Configure Endor Labs Action Policies
• PR Comments
  • Enable PR Comments
  • PR Comments Example
  • Customize GitHub PR comments notification templates
    • Data model

Install Software Prerequisites

To ensure the successful execution of the Endor Labs GitHub action, the following prerequisites must be met:

• The GitHub action must be able to authenticate with the Endor Labs API.
• You must have the value of the Endor Labs namespace handy for authentication.
• You must have access to the Endor Labs API.
• If you use keyless authentication, you must set an authorization policy in Endor Labs. See Authorization policies for details.

Example GitHub Action Workflow

Endor Labs scanning workflow using GitHub actions that accomplishes the following tasks in your CI environment:

• Tests PRs to the default branch and monitors the most recent push to the default branch.
• Builds a Java project and sets up the Java build tools. If your project is not on Java, then configure this workflow with your project-specific steps and build tools.
• Authenticates to Endor Labs with GitHub Actions keyless authentication.
• Scan with Endor Labs.
• Comments on PRs if any policy violations occur.
• Generates findings and uploads results to GitHub in SARIF format.

The following example workflow shows how to scan with Endor Labs for a Java application using the recommended keyless authentication for GitHub actions:

name: Endor Labs Dependency and Secrets Scan
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload Sarif artifact to GitHub
      contents: read # Used to check out a private repository
      actions: read # Required for private repositories to upload Sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication with Endor Labs
      pull-requests: write # Required to automatically comment on PRs for new policy violations
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: Endor Labs Scan Pull Request
      if: github.event_name == 'pull_request'
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: true
        enable_pr_comments: true # Required to automatically comment on PRs for new policy violations
        github_token: ${{ secrets.GITHUB_TOKEN }} # Required for PR comments on new policy violations

  scan-main:
    permissions:
      id-token: write
      repository-projects: read
      pull-requests: read
      contents: read
    name: endorctl-scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: 'Endor Labs Scan Push to main'
      if: ${{ github.event_name == 'push' }}
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: false
        scan_summary_output_type: 'table'
        sarif_file: 'findings.sarif'
    - name: Upload findings to github
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'findings.sarif'

Authenticate with Endor Labs

Endor Labs recommends using keyless authentication in CI environments. Keyless authentication is more secure and reduces the cost of secret rotation. To set up keyless authentication see Keyless Authentication.

If you choose not to use keyless authentication, you can configure an API key and secret in GitHub for authentication as outlined in Managing API keys.

Authentication Without Keyless Authentication for GitHub

If you are not using keyless authentication for GitHub Actions, you must not provide id-token: write permissions to your GitHub token unless specifically required by a step in this job. You must also set enable_github_action_token: false in your Endor Labs GitHub action configuration.

The following example configuration uses the Endor Labs API key for authentication:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          api_key: ${{ secrets.ENDOR_API_CREDENTIALS_KEY }}
          api_secret: ${{ secrets.ENDOR_API_CREDENTIALS_SECRET }}
          enable_github_action_token: false

The following example configuration uses a GCP service account for keyless authentication to Endor Labs:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          gcp_service_account: '<Insert_Your_Service_Account>@<Insert_Your_Project>.iam.gserviceaccount.com'
          enable_github_action_token: false

Endor Labs GitHub Action Configuration Parameters

The following input configuration parameters are supported for the Endor Labs GitHub Action:

Common parameters

The following input global parameters are supported for the Endor Labs GitHub action:

Scanning parameters

The following input parameters are also supported for the Endor Labs GitHub action when used for scanning:

Configure Endor Labs Action Policies

Configure an action policy in the Endor Labs user interface to perform an action when a rule is triggered. See Action Policies for details on action policies.

• Set the Policy Template to Detected Secrets and select the Template Parameters as desired.
• Choose Enforce Policy and
  • Select Warn as the recommended action.
  • Select Break the Build to fail the build CI pipeline.

PR Comments

You can configure GitHub Actions to comment on PRs if there are any policy violations. When you raise a PR, Endor Labs scans and detects policy violations. A comment is added in the PR with the details of violation. The CI pipeline warns you or fails the build based on your action policy configuration. The PR comments also include recommendations to help you take necessary remedial actions.

You can customize the template of the comment according to your requirement.

Enable PR Comments

Make sure that your GitHub action workflow includes the following configuration.

• The workflow must have a with clause including: enable_pr_comments to true to publish new findings as review comments and github_token: ${{ secrets.GITHUB_TOKEN }}. This token is automatically provisioned by GitHub when using GitHub actions. See GitHub configuration parameters for more information.
• To grant Endor Labs the ability to comment on PRs you must include the permission pull-requests: write.

The following example configuration comments on PRs if a policy violation is detected.

      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example' # Update with your Endor Labs namespace
          scan_summary_output_type: 'table'
          scan_dependencies: true
          scan_secrets: true
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

PR Comments Example

You can view a live example of a GitHub repository with a workflow that enables PR comments.

The main.yaml file contains the following configuration to enable PR comments.

name: Build Release
on:
  pull_request:
    branches: [main]
  workflow_dispatch:
  push:
    branches: [main]
  schedule:
    - cron: "23 23 * * 0"
jobs:
  build:
    permissions:
      pull-requests: write
      security-events: write
      contents: read
      id-token: write
      actions: read
    runs-on: ubuntu-latest
    env:
      ENDOR_NAMESPACE: "endorlabs-hearts-github"
    steps:
      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1
        with:
          namespace: ${{ env.ENDOR_NAMESPACE }}
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

The PR #10 introduced a reachable vulnerability. Since the workflow has enable_pr_comments set as true, a comment is added to the PR on the policy violation.

You can expand the comment to view more details about the violation and take steps to resolve the issue.

Customize GitHub PR comments notification templates

Endor Labs provides a default template with standard information that will be included in your pull requests as comments. You can use the default template, or you can choose to edit and customize this template to fit your organization’s specific requirements. You can also create custom templates using Go Templates.

1. Sign in to Endor Labs and navigate to Manage>Integrations
2. Look for GitHub PR comments under Notifications.
3. Click Edit Template.
4. Make the required changes and click Save Template.
5. Click Restore to Default to revert the changes.
6. Use the download icon in the top right corner to download this template.
7. Use the copy icon to copy the information in the template.

Data model

To create custom templates for PR comments, you must understand the data supplied to the template.

See the following protobuf specification for the GithubCommentData message that this template uses.

syntax = "proto3";

package internal.endor.ai.endor.v1;

import "google/protobuf/wrappers.proto";
import "spec/internal/endor/v1/finding.proto";
import "spec/internal/endor/v1/package_version.proto";

option go_package = "github.com/endorlabs/monorepo/src/golang/spec/internal.endor.ai/endor/v1";
option java_package = "ai.endor.internal.spec";

// The list of finding UUIDs.
message FindingUuids {
  repeated string uuids = 1;
}

// The map of dependency name to findings.
message DependencyToFindings {
  map<string, FindingUuids> dependency_to_findings = 1;
}

// The map of PackageVersion UUID to DependencyToFindings.
message PackageToDependencies {
  map<string, DependencyToFindings> package_to_dependencies = 1;
}

message GithubCommentData {
  // The header of the PR comment. Identifies the PR comment published by Endor Labs.
  // It should always be at top of the template.
  google.protobuf.StringValue comment_header = 1;

  // The footer of the PR comment.
  google.protobuf.StringValue comment_footer = 2;

  // The map of finding UUID to finding object.
  map<string, internal.endor.ai.endor.v1.Finding> findings_map = 3;

  // The map of policy UUID to policy name.
  // This will contain only the policies that are triggered or violated.
  map<string, string> policies_map = 4;

  // The map of policy UUID to the list of finding UUIDs.
  map<string, FindingUuids> policy_findings_map = 5;

  // The map of PackageVersion UUID to PackageVersion object.
  map<string, internal.endor.ai.endor.v1.PackageVersion> package_versions_map = 6;

  // The data needs to be grouped as follows:
  //
  // - Policy 1
  // 		- Package 1
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 2
  //			- Dependency Package 2
  //				- Finding 3
  //				- Finding 4
  // 		- Package 2
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 5
  // - Policy 2
  //		....
  //
  //		Map 0[PolicyUUID]/Map 1[PkgVerUUID]/Map 2 [Dep Names]/Finding UUID
  map<string, PackageToDependencies> data_map = 7;

  google.protobuf.StringValue api_endpoint = 8;
}

To understand Finding and PackageVersion definitions that are used in this protobuf specification, see:

• Finding resource kind
• PackageVersion resource kind

See the following specification to understand the additional functions that are also available. You can access these functions by using their corresponding keys.

// FuncMap contains the additional functions that are available to GithubCommentTemplate.
var FuncMap = template.FuncMap{
	"now": toTime, // 'now' gives the current time

	// 'enumToString' coverts the enums for finding level, finding category and finding tags to string
	"enumToString": enumToString,

	// 'getPackageVersionURL' returns the URL for a given PackageVersion
	"getPackageVersionURL": func(apiURL string, packageVersion *endorpb.PackageVersion) string {
		result, err := common.GetPackageVersionURL(apiURL, packageVersion)
		if err != nil {
			return ""
		}
		return result
	},

	// 'getFindingURL' returns the URL for a given Finding
	"getFindingURL": func(apiURL string, finding *endorpb.Finding) string {
		result, err := common.GetFindingURL(apiURL, finding)
		if err != nil {
			return ""
		}
		return result
	},

	// 'add' returns the sum of two integers
	"add": func(n int, incr int) int {
		return n + incr
	},

	// 'getOtherFindingsPackageMarker' returns the key for _findingsWithNoPackages for lookup in DataMap
	// Not all findings are associated with a PackageVersion, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsPackageMarker": func() string { return _findingsWithNoPackages },

	// 'getOtherFindingsDependencyMarker' returns the key for _findingsWithNoDeps for lookup in DataMap
	// Not all findings are associated with a dependency, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsDependencyMarker": func() string { return _findingsWithNoDeps },

	// 'getFindingsCountString' returns a string with number of findings, example - "5 findings"
	"getFindingsCountString": func(dataMap *endorpb.PackageToDependencies) string {
		count := 0

		for _, depMap := range dataMap.PackageToDependencies {
			for _, findingMap := range depMap.DependencyToFindings {
				count += len(findingMap.Uuids)
			}
		}

		findingsStr := "findings"
		if count == 1 {
			findingsStr = "finding"
		}

		return fmt.Sprintf("%d %s", count, findingsStr)
	},
}

8.3 - Scanning in Bitbucket Pipelines

Bitbucket Pipelines is a continuous integration and continuous delivery (CI/CD) service built into Bitbucket. It allows developers to automatically build, test, and deploy their code based on a configuration file bitbucket-pipelines.yml defined in the root of their repository.

To integrate Endor Labs into a Bitbucket pipeline:

1. Authenticate to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authenticate to Endor Labs

Configure an API key and secret in the bitbucket-pipelines.yml file for authentication. See managing API keys for more information on generating an API key for Endor Labs.

Configure your Bitbucket pipeline

To create a Bitbucket pipeline reference the following steps:

1. Create a bitbucket-pipelines.yml file in your repository if you do not already have one.
2. In your bitbucket-pipelines.yml file customize the job configuration based on your project’s requirements using the following example.
3. Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project.
4. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
5. Update your default branch from main if you do not use main as the default branch name.
6. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Example

Use the following example to get started. Make sure to customize this job with your specific build environment and build steps.

Bitbucket configuration

simage: maven:3.6.3-jdk-11

pipelines:
  branches:
    main:
      - step:
          name: "Build and Test"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
  pull-requests:
    '**':
      - step:
          name: "Build and Test on PR to Main"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs PR Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan --pr --pr-baseline=main --languages=java --output-type=json -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET | tee output.json
            #Optional - Comment on the PR
            # - apt-get update
            # - apt-get install -y python3 python3-pip
            # - pip3 install -r requirements.txt
            # - python3 add-bitbucket-pr-comments.py output.json

Once you’ve set up Endor Labs, you can test your CI implementation to ensure it is successful and then proceed with your scans.

View PR comments for policy violations

You can also use the Insights feature in Bitbucket Pipelines to indicate if the changes in your pull requests violated any policies set in Endor Labs.

import json
import os
import sys
import requests
import argparse

# Check for required environment variables
BITBUCKET_REPO_OWNER = os.getenv('BITBUCKET_REPO_OWNER')
BITBUCKET_REPO_SLUG = os.getenv('BITBUCKET_REPO_SLUG')
BITBUCKET_COMMIT = os.getenv('BITBUCKET_COMMIT')

if not all([BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT]):
     sys.exit("Error: One or more required environment variables (BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT) are not set.")

#This is an internal proxy running in the BitBucket environment to accept Code Insights
proxies = {"http": "http://localhost:29418"}

def load_json_with_unescaped_characters(file_path):
    """Load and return JSON data from a file, replacing unescaped characters if necessary."""
    try:
        with open(file_path, 'r', encoding='utf-8') as file:
            json_str = file.read().strip()
        return json.loads(json_str)
    except json.decoder.JSONDecodeError as e:
        print(f"Failed to parse JSON: {e}")
        return None
    except FileNotFoundError:
        print(f"File not found: {file_path}")
        sys.exit()

def construct_report_payload(endor_findings):
    """Construct and return the payload for creating a Bitbucket report."""
    warning_findings_count = len(endor_findings.get('warning_findings', []))
    blocking_findings_count = len(endor_findings.get('blocking_findings', []))
    total_violations = warning_findings_count + blocking_findings_count
    result = "PASSED" if total_violations == 0 else "FAILED"
    report_payload = {
        "title": "Endor Labs Policy Violations",
        "details": f"Endor Labs detected {total_violations} policy violations associated with this pull request.\n\n{endor_findings['warnings'][0]}",
        "report_type": "SECURITY",
        "reporter": "Endor Labs",
        "link": f"https://app.endorlabs.com/t/{namespace}/projects/{project_uuid}/pr-runs/{report_id}",
        "logo_url": "https://avatars.githubusercontent.com/u/92199924",
        "result": result,
        "data": [
            {"title": "Warning Findings", "type": "NUMBER", "value": warning_findings_count},
            {"title": "Blocking Findings", "type": "NUMBER", "value": blocking_findings_count}
        ]
    }
    return report_payload

def construct_annotation_payload(finding):
    """Construct and return the payload for creating an annotation in Bitbucket."""
    title = "Endor Labs Policy Violation"
    summary = finding['meta']['description']
    details =  f"{finding['spec']['summary']}\n\n{finding['spec']['remediation']}"
    severity = "CRITICAL" if finding['spec']['level'] == "FINDING_LEVEL_CRITICAL" else \
               "HIGH" if finding['spec']['level'] == "FINDING_LEVEL_HIGH" else \
               "MEDIUM" if finding['spec']['level'] == "FINDING_LEVEL_MEDIUM" else "LOW"
    affected_paths = finding['spec'].get('dependency_file_paths', [])
    path = affected_paths[0] if affected_paths else "Unknown file"
    annotation_payload = {
        "external_id": finding['uuid'],
        "title": title,
        "annotation_type": "VULNERABILITY",
        "summary": summary,
        "details": details,
        "severity": severity,
        "path": path
    }
    return annotation_payload

def send_report(report_payload):
    """Send the constructed report payload to the Bitbucket API."""
    report_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports/{report_id}"
    response = requests.put(report_url, json=report_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Report created or updated successfully")
    else:
        print(f"Failed to create or update report: {response.text}")

def send_annotation(annotation_payload):
    """Send the constructed annotation payload to the Bitbucket API."""
    annotation_url = f"{base_url}/{report_id}/annotations/{annotation_payload['external_id']}"
    response = requests.put(annotation_url, json=annotation_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Annotation added successfully")
    else:
        print(f"Failed to add annotation: {response.text}")

def process_findings(filename):
    """Load findings from JSON, create a report, and add annotations for each finding."""
    endor_findings = load_json_with_unescaped_characters(filename)
    if endor_findings is None:
        print("Failed to load findings. Exiting.")
        return

    global report_id, project_uuid, namespace

    # Define the order of keys to check
    finding_types = ['all_findings', 'warning_findings', 'blocking_findings']

    # Iterate over finding types and extract the first one found
    for finding_type in finding_types:
        if endor_findings.get(finding_type):
            first_finding = endor_findings[finding_type][0]
            report_id = first_finding['context']['id']
            project_uuid = first_finding['spec']['project_uuid']
            namespace = first_finding['tenant_meta']['namespace']
            break  # Stop after finding the first non-empty list

    if not report_id:
        print("No findings found.")
        sys.exit()

    # Prepare the base URL for Bitbucket API requests
    global base_url
    base_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports"

    # Create the report
    report_payload = construct_report_payload(endor_findings)
    send_report(report_payload)

    # Iterate over findings and create annotations
    for finding in endor_findings.get('blocking_findings', []) + endor_findings.get('warning_findings', []):
        annotation_payload = construct_annotation_payload(finding)
        send_annotation(annotation_payload)

def main():
    """Main function to parse arguments and process findings."""
    parser = argparse.ArgumentParser(description="Script to process findings and update Bitbucket via API.")
    parser.add_argument("filename", help="Filename containing the JSON findings.")
    args = parser.parse_args()

    process_findings(args.filename)

if __name__ == "__main__":
    main()