Upgrades and remediation

• 1: Upgrade impact analysis
• 2: Endor patches
• 2.1: Connect to the Endor Labs Patch Factory
• 2.2: Automatic patching with Endor Patches
• 2.3: Patch transparency
• 2.4: Configure JFrog Artifactory to use Endor patches
• 2.5: Configure Sonatype Nexus Repository to use Endor patches

1 - Upgrade impact analysis

To help developers and security teams make informed decisions, Endor Labs provides a prioritized list of recommended upgrades for each project and package. The recommendations are made after assessing the following criteria to determine the impact and complexity of an upgrade:

• Vulnerabilities associated with a dependency’s current version and those of its transitive dependencies
• Resolved vulnerabilities associated with a dependency’s later versions and those of its transitive dependencies
• Heuristic factors that influence the probability of breaking changes
• Program analysis to directly identify breaking changes

Endor Labs provides an assessment of upgrade options for each dependency, including the potential impact and risk of each option. These options include:

• The latest version of the software
• The latest vulnerable free version
• The most impactful update with moderate evidence of breaking changes
• The most impactful update with low evidence of breaking changes

Remediation risk

Endor Labs evaluates the remediation options for each recommended upgrade and assigns a remediation risk. There are three categories of remediation risk.

• High Remediation Risk: This risk level is assigned when Endor Labs has high confidence that a breaking change will occur.
• Medium Remediation Risk: This risk level is assigned when Endor Labs has identified a potential breaking change but has low to moderate confidence in its impact. It is also assigned in cases of major version conflicts that could be affected by the upgrade.
• Low Remediation Risk: This risk level is assigned when there is minimal or no evidence suggesting that a breaking change will occur. The absence of evidence does NOT guarantee that it will not break your application.

To assign remediation risk, Endor Labs looks for breaking changes associated with the upgrade and conflicts between dependency versions.

Breaking changes

Breaking changes may necessitate refactoring your code to complete an upgrade due to newly introduced incompatibilities. A breaking change may occur due to the following criteria:

• API Changes: When the public interface of a library changes, such as through renaming or removing functions, altering function signatures, or modifying expected input or output parameters.
• Behavioral Changes: When the underlying behavior of a function or method changes, even if the interface remains the same. This can lead to unexpected results or introduce issues.
• Dependency Updates: When a dependency of a dependency, that is a transitive dependency, introduces breaking changes, it can affect the higher-level dependency.
• Deprecations and Removals: When deprecated features are finally removed or altered significantly.
• Configuration Changes: When the configuration format or options for a library change.
• Changes in Supported Platforms: When a library drops support for certain platforms or versions of platforms, for example, an older version of Go.

Dependency conflicts

Dependency conflicts occur when different parts of a software project require different versions of the same dependency. These conflicts can cause various issues, such as build failures, runtime errors, or unexpected behavior. When there are major or minor version conflicts in your dependency graph, the impact can vary depending on the nature of the conflicts and the specific dependencies involved.

While conflicts do not necessarily guarantee that updating will impact your application, they increase the likelihood that changes may affect it.

View upgrade recommendations

To see Endor Labs upgrade recommendations:

1. Login to Endor Labs
2. Go to Projects and navigate to a project you would like to review.
3. Once you’re in the project click Remediations.
4. You’ll be presented with a list of dependencies that have fixed security vulnerabilities in your project and their associated packages. You can filter the attributes of these findings to identify the dependencies with the most impact on your organization.
5. Click on the dependency that most interests you to review upgrade options.
6. Each upgrade option will have information about the fixed findings and risks associated with that upgrade.

Review remediation risk

Once you have a specific remediation option of interest you can review the impact and evidence of remediation risk associated with each upgrade.

To review the impact and risk of an upgrade option:

1. Click on the drawer icon on the right side of an upgrade recommendation.
2. Under Overview, you can review a summary of the upgrade and the Fixed Findings associated with an upgrade.
3. Under Potential Conflicts, you can review any major or minor version conflicts in the package as well as which direct dependencies have transitives that are in conflict.
4. Under Breaking Changes, you can review any identified breaking changes, their confidence, and their call paths.

2 - Endor patches

Endor patches is a curated repository of software packages with backported vulnerability fixes for your security and convenience. Endor Labs identifies vulnerable functions and the commits that fixed each vulnerability in the open-source community. These fixes, along with necessary supporting commits, are applied to older software versions to create a minimum viable security patch for each library supported by Endor Labs. See Connect to the Endor Patch Factory to get started.

Endor patches are a result of extensive research. In security, trust is crucial. Therefore, the patch details are fully transparent. The builds are hermetic ensuring they are consistent, reproducable, and reliable. The exact code changes, along with builds, build steps, and logs, are auditable and available for review. See information about patch transparency and trust for more details

Customers can access Endor Patches patches through a hosted repository, where each software component has three types of versions:

• A version associated with a specific patch date for build reproducibility. For instance: v2.9.10.3-2024-07-11.
• A version with the latest patched version of a library, incorporating all current patches. This can be used by appending -endor-latest to a package version. For instance: v2.9.10.3-endor-latest.
• A version matching the upstream open-source version, allowing users to use the patched version without code changes. See auto patch versions for more information on how to automatically use an Endor Patch. For instance: v2.9.10.3.

By minimizing changes to fix known vulnerabilities and providing complete transparency, Endor Patches offer a comprehensive solution to help teams quickly address vulnerabilities, even when a fix is challenging.

2.1 - Connect to the Endor Labs Patch Factory

You can start using Endor patches with 3 simple steps:

1. Configure an API Key to connect to the Endor Labs Patch Factory
2. Configure your package manager to use Endor patches.
   • Configure Gradle
   • Configure Maven
3. Specify the Endor Patch you want to use.

Create an API key

To gain Rest API access to Endor Labs Patch Factory, you have to generate API credentials to authenticate to the repository.

1. From Manage, navigate to API Keys.
2. Select Generate API Key.
3. Enter a name to identify the API key, such as “Endor Patch Factory”.
4. Select the permissions to apply to the API Key, you’ll need at least Read Only.
5. Select the expiration date of the API key. This may be either 30, 60, or 90 days.

Using these credentials, you can configure Endor Labs your package manager or Artifact Repository proxy to authenticate to the Endor Patch Factory.

Configure Gradle

1. Open the build.gradle file of the package you’d like to configure to use patches.
2. Include a repositories section in the build.gradle file to establish a repository connection to the Endor Labs Patch Factory. Make sure to replace namespace with the name of your Endor Labs namespace.
3. Include a reference to the Endor Patch version in the build.gradle file.

Example repositories section:

repositories {
    mavenCentral()
    maven {
        url "https://factory.endorlabs.com/v1/namespaces/<namespace>/maven2"
        credentials {
            username "$ENDOR_API_CREDENTIALS_KEY"
            password "$ENDOR_API_CREDENTIALS_SECRET"
    }
}

Finally, include the Endor Labs patch version you’d like to use. For example, to use the latest patched version from Endor Labs add -endor-latest to the version of your dependency.

dependencies {
    implementation("com.fasterxml.jackson.core:jackson-databind:2.9.10.3-endor-latest")
}

Configure Maven

1. Open the pom.xml file of the package you’d like to configure to use patches.
2. If there is no section in the pom.xml, then create one.
3. Include a repositories section in the pom.xml file to establish a repository connection to the Endor Labs Patch Factory. Make sure to replace with the name of your Endor Labs namespace.

<repositories>
  <repository>
    <id>endorlabs</id>
    <url>https://factory.endorlabs.com/v1/namespaces/<namespace>/maven2</url>
  </repository>
</repositories>

4. Next, open the Maven settings.xml file located at $HOME/.m2/settings.xml and add a section to the settings file with your Endor Labs credentials.
   • The username value must be your API key.
   • The password must be your API key secret.
   • The id value must be same as the value provided in the pom.xml.

Example:

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                              http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>endorlabs</id>
            <username>${env.ENDOR_API_CREDENTIALS_KEY}</username>
            <password>${env.ENDOR_API_CREDENTIALS_SECRET}</password>
        </server>
    </servers>
</settings>

5. Finally, include the Endor Labs patch version you’d like to use in to your manifest. For example, to use the latest patched version from Endor Labs include -endor-latest to the version of your dependency.

<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-databind</artifactId>
   <version>2.9.10.3-endor-latest</version>
</dependency>

2.2 - Automatic patching with Endor Patches

Upgrading software can be challenging for development teams. Endor Automatic Patching allows you to seamlessly fix security vulnerabilities during each software build, minimizing the effort required to maintain a secure codebase.

By enabling automatic patching with Endor Labs for every build, you can automatically address vulnerabilities in both direct and transitive dependencies. This approach helps prevent a growing backlog of security issues.

Enable Automatic Patching

To start using Endor Lab’s automatic patching, follow these steps:

1. Configure Endor Labs Patch Factory

Set Endor Labs Patch Factory as the top priority package repository in your package manager or Artifactory virtual repository.

For detailed instructions, refer to the following documentation:

• Learn how to connect to the Endor Labs Patch Factory.
• Learn how to configure JFrog Artifactory.
• Learn how to configure a Nexus repository.

2. Enable Auto Patching in Endor Labs

To enable auto patching in Endor Labs:

1. Access Settings: Navigate to Manage > Settings in your Endor Labs tenant.
2. Activate Auto Patching: click Enable Auto Patching.
3. Save Configuration: click Save Patch Settings and acknowledge the warning regarding reproducible builds.

Note: Enabling or disabling auto patching may take up to ten minutes to take effect. During this period, changes to your patch settings might not be immediately applied.

Configure projects for automatic patching

After enabling automatic patching globally, you must activate it for individual projects to ensure findings are correctly updated.

Enable automatic patching on a project

To enable auto patching on one of your projects:

1. Select Project: Go to Projects and choose the project or projects you want to enable for auto patching.
2. Edit Project Tags: Click Edit Tags located on the top right side of the project list.
3. Assign Patching Tag: Add the tag use_streaming_patches=true to the project.
4. Save Tags:: Click Save Tags to apply the changes.
5. Rescan Project: Rescan the project to update the bill of materials and associate the findings with Endor Patches.

Note: If you do not set this tag, Endor Labs will continue to report vulnerabilities based on the upstream open-source packages without applying automatic patches.

Considerations for automatic patching

While automatic patching enhances security by promptly addressing vulnerabilities, it introduces some trade-offs:

Build Reproducibility: Automatically applied patches may alter the build process or the resulting binaries in unpredictable ways, potentially affecting build reproducibility.

Endor Labs strives to provide the minimal necessary security patches to ensure your software remains secure without introducing significant changes. With automatic patching enabled, new patches are applied automatically as they become available, reducing manual intervention and enhancing your security posture.

2.3 - Patch transparency

In security, trust is crucial. Therefore, the patch details of an Endor patch are fully transparent. You can audit the exact code changes, builds, build steps, and logs. The builds are reproducible and hermetic.

Review patch transparency information

To review patches, build, test and deploy proccess used to create an Endor patch, use the AssuredPackageVersion API.

The commands and logs used to test, deploy and build this package are stored for each version of a package as an attestation.

Review attestations

To see all information about the patch, build, test and deploy proccess for this Endor patch use the command:

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3"

Review security attestations

To see the exact changes used for a given security patch, Endor Labs provides a security attestation which shows:

1. Fixed vulnerabilities
2. Exact code changes for each package
3. Exact commits used and if they are upstream commits or commits applied by Endor Labs directly

To see a security attestation use the following command with the name of the package version you’d like to inspect. For this example we’ll use com.fasterxml.jackson.core:jackson-databind@2.9.10.3:

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3" --field-mask="spec.security_attestation"

Review build attestations

To see the build steps and build logs for an Endor patch, you can see that patch build attestation.

To see a build attestation use the following command with the name of the package version you’d like to inspect. For this example we’ll use com.fasterxml.jackson.core:jackson-databind@2.9.10.3

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3" --field-mask="spec.build_attestation"

Reviewing Test Attestations

To see the test steps and test logs for an Endor patch, you can see that patch test attestation.

To see a deployment attestation use the following command with the name of the package version you’d like to inspect. For this example we’ll use com.fasterxml.jackson.core:jackson-databind@2.9.10.3

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3" --field-mask="spec.test_attestation"

Review deploy attestations

To review the deployment steps and logs for an Endor patch, check the patch deployment attestation.

To see a deployment attestation, use the following command with the name of the package version you’d like to inspect. For this example, we’ll use com.fasterxml.jackson.core:jackson-databind@2.9.10.3.

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3" --field-mask="spec.deploy_attestation"

Reproducible Build

To download the reproducible build of the patched artifact, with the name of the package version you’d like to inspect. For this example, we’ll use com.fasterxml.jackson.core:jackson-databind@2.9.10.3.

endorctl api get -r AssuredPackageVersion -n oss --name="mvn://com.fasterxml.jackson.core:jackson-databind@2.9.10.3" --field-mask="spec.reproducible_build_source_code_url"

Use the URL to download the source code to reproduce the build. You can find instructions on building the artifact in the README of the downloaded tar.

Note: You will need Bazel and Docker installed on your host.

2.4 - Configure JFrog Artifactory to use Endor patches

Configure JFrog Artifactory to ensure that the patched dependencies from Endor Labs are fetched and used correctly. The following procedures use Maven as the repository type, you can select the repository type based on your requirements.

Create a remote repository for Endor Patching

Create a remote repository to fetch artifacts from the Endor Patch repository.

1. Log in to the JFrog Platform as an administrator.
2. In the Administration module, select Repositories.
3. Select Create a Repository and click Remote.
4. Select Maven from the list of repository types.
5. In Repository Key, enter a name such as endor-patch.
6. Create an API Key in Endor Labs to authenticate to the Endor Patch repository with “Read-Only” permissions. See creating an API key for more detail. Keep these details handy.
7. In URL, enter the Endor Patch repository URL, https://factory.endorlabs.com/v1/namespaces/$NAMESPACE/maven2. Replace $NAMESPACE with your Endor Labs tenant name.
8. Enter your Endor Labs API Key ID as the User Name and your Endor Labs API Key secret as the password for your new remote repository.
9. Click Test to ensure you are able to successfully connect to the remote repository.
10. Click Advanced and select Priority Resolution to ensure that the Endor patch repository is prioritized over other remote repositories.
11. Click Create Remote Repository.

Create a virtual repository for Endor Patching

Create a virtual repository to simply access to Endor patch repositories and other remote repositories.

1. Log in to the JFrog Platform.
2. In the Administration module, select Repositories.
3. Select Create a Repository and click Virtual.
4. Select Maven from the list of repository types.
5. In Repository Key, enter a name such as endor-patch.
6. Add the endor-patch remote repository to this virtual repository along with other required remote repositories.
7. Ensure endor-patch repository is at the top of the list to prioritize it if you are using auto patching. See the auto patching documentation for more details
8. Click Create Virtual Repository.

Edit an existing virtual repository

Edit an existing virtual repository to access the Endor Patch repositories and other remote repositories.

1. Log in to the JFrog Platform.
2. In the Administration module, select Repositories.
3. Select the Virtual tab and click into the existing virtual repository you’d like to edit.
4. Under Repositories move the endor-patch remote repository to the selected repositories.
5. Ensure endor-patch repository is at the top of the list of selected repositories to prioritize it if you are using auto patching. See the auto patching documentation for more information.
6. Click Save.

2.5 - Configure Sonatype Nexus Repository to use Endor patches

Configure Sonatype Nexus Repository Manager to ensure that the patched dependencies from Endor Labs are fetched and used correctly. The following procedures use Maven as the repository type, you can select the repository type based on your requirements.

Create a remote repository for Endor Patching

Create a remote repository to fetch artifacts from the Endor Patch repository.

1. Log in to the Nexus Repository Manager.
2. Go to Repositories and click Create repository.
3. Select maven2 (proxy) as the recipe.
4. Enter the repository name, such as endor-patch.
5. In Remote Storage, enter the Endor Patch repository URL, typically given by Endor Labs, like https://factory.endorlabs.com/v1/namespaces/<namespace>/maven2. Replace <namespace> with your Endor Labs tenant name.
6. Select Authentication, and enter your Endor Labs API Key ID as the User Name and your Endor Labs API Key secret as the password.
7. Click Create repository to save.

Prioritize Endor patch repository in Maven group

If you have a Maven group repository that combines multiple repositories, you need to prioritize the Endor patch repository.

1. Log in to the Nexus Repository Manager.
2. Select Browse and navigate to your Maven group repository that combines multiple repositories.
3. Edit the group repository and move the endor-patch repository to the top of the order in the members list. This ensures that Endor Patch is checked first before any other repository for patch dependencies.
4. Click Save to save the changes.

Set up routing rules in other repositories

You can set up routing rules in repositories, other than the Endor patch repositories, to exclude Endor patch repositories. This will prevent other repositories from overriding the Endor patch dependencies.

1. Log in to the Nexus Repository Manager.
2. Select Repository in the Administration menu.
3. Select Create Routing Rule.
4. Enter a name such as exclude-endor-patch.
5. Select Block as the mode.
6. Enter the regular expression to block Endor patches in Matchers. For example, com/endor/patch/.*.
7. Click Create Routing Rule to save the rule.
8. Select Browse and navigate to the proxy repository that you want to edit.
9. Click Edit and select the routing rule that you created as the Routing Rule.
10. Click Save.