Endor Labs allows you to determine whether the OS packages present in a container image are actually used by your application at runtime. Use container reachability to distinguish dependencies that are merely installed from those that are actively exercised during execution, helping security teams prioritize the most critical security issues for remediation.

Endor Labs supports the following two container reachability modes. Choose the mode that aligns with how your workload executes and what dependencies it requires at runtime.

• Basic Reachability: Endor Labs executes and profiles the container image in a local environment during the scan. Select this method when the application can run without relying on external services.
• Instrumented Reachability: Endor Labs integrates a sensor into the container image and deploys it in the environment where the application normally runs. The sensor captures runtime behavior and produces a profiling report, which is analyzed. Select this method for workloads that rely on external services or interactions that cannot be triggered in a brief local execution.

Prerequisites

To perform container reachability analysis, ensure you meet the following system requirements:

• The container must have sufficient CPU and memory resources to run successfully.

• The container must be runnable.

• The container must have network access if its startup process requires external communication.

• Docker daemon dockerd must be installed on the host, must be runnable and accessible to the current user without elevated privileges. For example, docker images should work without sudo.

• The negotiated Docker API version between the client and server must be 1.48 or higher.

• The scan must be run on either a Linux or macOS host machine. Container reachability is supported for both amd64 and arm64 architectures.

Determine container reachability

Endor Labs determines container reachability by extracting OS packages from the container image, profiling the container’s runtime behavior, and correlating the results to identify which dependencies are actually used during execution. The steps below describe how container reachability is determined.

1. Dependency extraction - Endor Labs extracts all packages and dependencies present in the container image by analyzing its file system to identify installed OS packages and their file path locations within the image.

2. Dynamic profiling - Endor Labs runs the container image in a controlled environment, monitoring which OS-level files and dependencies are accessed during execution. The profiling captures runtime behavior including system calls, process IDs, and file path access patterns. It also identifies the main process that starts the container as the entry point and uses it to determine which packages are reachable through their dependency relationships.

3. Path matching and reachability determination - Endor Labs correlates the results from both steps by comparing the extracted dependency file paths against the file access patterns captured during profiling, then assigns each dependency a reachability status based on whether it was accessed during execution.

Run the following command with the --os-reachability flag to include container reachability analysis in the scan.

endorctl container scan \
  --namespace=<your-namespace> \
  --image=<image_name:tag> \
  --project-name=<endor_project_name> \
  --os-reachability

You can also run container scans with OS reachability using GitHub Actions. See Scan containers with OS reachability for details.

Image qualification

Before dynamic profiling begins, Endor Labs performs a series of image qualification checks to determine if the container image is suitable for profiling. These checks include:

• Image size - Verifies that the uncompressed image size does not exceed the configured limit. The default limit is 10 GB. Use the --profiling-max-size flag to adjust this limit.

• Runnability - Runs the container to check that it starts without errors. If the container exits with an error, the error details are shown in the CLI output and surfaced in the Endor Labs user interface.

If the image fails any of the qualification checks, dynamic profiling is skipped and the scan proceeds without reachability analysis.

Container reachability options

You can run the endorctl container scan --os-reachability command with the following options.

Container reachability status

The container reachability status indicates whether a dependency was used during runtime profiling or whether its usage could not be determined.

• Reachable - The dependency is observed in runtime signals or confidently inferred through correlation analysis.

• Potentially reachable - The dependency has not been observed during profiling, and there is no correlation evidence of its usage. However, its usage cannot be definitively ruled out without additional analysis, such as extended runtime monitoring.

• Unreachable - The dependency was not observed during profiling and has no path from the container image’s entry point to it.

Recommended approach for prioritizing remediation

Use reachability information to prioritize vulnerability remediation effectively. The following table provides recommended actions based on the combination of vulnerability severity and reachability status.

Filter container findings by reachability

You can filter findings across all projects by their reachability status.

1. Select Findings from the left sidebar.
2. Select Attributes, and in the Reachable Dependency filter, select Yes, Potentially, or No to narrow down findings by reachability status.

Limitations

Keep the following limitations in mind when using container reachability analysis:

• Code coverage - The scan does not detect dependencies accessed after the profiling window ends.

• OS packages - Container reachability analysis applies only to OS-level packages. It identifies which OS packages are used at runtime but does not analyze specific vulnerable functions within those packages. Use Software Composition Analysis reachability to assess the runtime relevance of application dependencies.

• Windows not supported - Container scanning and reachability are not supported on Windows.

• Tar image paths - Dynamic profiling is not supported for container images referenced with a tar path.

Troubleshoot issues

Instrumented container reachability is an advanced OS package reachability mode that embeds a runtime sensor in your container image to record how the image is used in a real environment. Use it when your container relies on complex external services that cannot be exercised during a short local run, so that reachability results then reflect actual usage.

Use instrumented reachability when:

• The container has complex external dependencies such as databases, message queues, third‑party services that cannot be realistically exercised in a local ephemeral run.
• You want profiling to happen in a realistic environment, such as staging, that mirrors production traffic.
• You already have integration or end‑to‑end tests and want reachability to reflect those tests.

Prerequisites

To perform instrumented container reachability analysis, ensure that:

• Container scanning is enabled using the --os-reachability flag.
• endorctl is installed and authenticated.
• Docker daemon dockerd is installed on the host, runnable and accessible to the current user without elevated privileges. For example, docker images should work without sudo.
• The negotiated Docker API version between the client and server is 1.48 or higher.
• You must run the scan from either a Linux or a macOS host machine. Container reachability is supported for both amd64 and arm64 architectures.

To run the instrumented container images, you need either Docker or Kubernetes, based on your setup:

• Docker: Docker daemon is available and you can run the instrumented image locally.
• Kubernetes: kubectl is configured with access to your cluster so you can deploy and run the instrumented image in a pod.

Determine instrumented container reachability

Follow these steps to scan the original image, collect runtime profiling data, and determine reachability for containers.

1. Run endorctl container instrument to create a new image with a lightweight sensor injected into the filesystem. The resulting image will have -instrumented appended to the tag.

   endorctl container instrument \
     --image=<image_name-tag> \
     --app-stop-signal=QUIT \
     --load-instrumented-image=true
   

   • --image: Original container image to instrument.
   • --app-stop-signal: Signal used to stop the application. This is required so that the sensor can flush profiling data before the container exits.
   • --load-instrumented-image: Loads the instrumented image into your local Docker runtime so it can be referenced by Kubernetes.

2. Define how the instrumented image runs in a manifest. Create a manifest file such as demo-manifest-file.yaml to identify your workload. In the manifest, reference the instrumented image from step 1 and use a pod name and container name you can reuse later. You can also add env, volumes, and other options as needed for your application.

   apiVersion: v1
   kind: Pod
   metadata:
     name: <pod-name>
   spec:
     restartPolicy: OnFailure
     containers:
       - name: <container-name>
         image: <instrumented-image>
         ports:
           - containerPort: <container-port>
               hostPort: <host-port>
           securityContext:
             privileged: true
   

   • Set securityContext.privileged to true so the profiling sensor can run.
   • Set restartPolicy to OnFailure so that the pod does not restart automatically after you stop the app to generate the report.

3. Deploy the instrumented image to Kubernetes. The application runs normally while the sensor observes file access and process activity during execution.

   kubectl apply -f <manifest-file>
   kubectl get pods <pod-name>
   

   Replace <manifest-file>, <pod-name>, and <container-name> with the values from your manifest.

• If you need to access the application locally, run:

  kubectl port-forward pod/<pod-name> <host-port>:<container-port>
  

• You can also run your tests or interact with the application normally. The profiling sensor will capture runtime activity.

4. After you finish testing, send the --app-stop-signal, for example, QUIT, to stop the application gracefully. This signal triggers the profiling sensor to write the creport.json file and generate the profiling data.

   kubectl exec -it <pod-name> -c <container-name> -- sh -c "kill -QUIT 1"
   

5. The sensor writes a profiling report to a known artifacts directory inside the container.

• Verify that the creport.json file exists in the container:

  kubectl exec -it <pod-name> -c <container-name> -- sh -c "ls -ls /opt/_instrumented/artifacts"
  

6. Create a local directory and copy the report to it.

   # Create output directory
   mkdir -p collect_output
   # Copy the profiling data
   kubectl cp <pod-name>:/opt/_instrumented/artifacts/creport.json collect_output/creport.json -c <container-name>
   

• To verify that the file is copied, run:

  ls -la collect_output/
  

• Alternatively, you can use endorctl container collect to stop the running application and retrieve the profiling report from the instrumented container into a local directory. Skip steps 4, 5, and 6 if you are using this command.

  endorctl container collect \
    --dynamic-profiling-data=true \
    --output-dir=collect_output \
    --image=<instrumented-image>
  

• Set --dynamic-profiling-data to true to collect profiling data from the instrumented container.

• Set --output-dir to the local directory where the collected data is saved. A subdirectory is created under this path cluster/pod/container. Use that path for --profiling-data-dir in the next step.

7. Run endorctl container scan with the path to the directory that contains the collected profiling data, and OS reachability enabled. Endor Labs loads the report, maps runtime files to OS packages, and marks the corresponding packages as reachable.

   endorctl container scan \
     --image=<original-image:tag> \
     --profiling-data-dir=collect_output \
     --project-name=<project-name> \
     --os-reachability
   

8. Remove the pod after completing the analysis:

   kubectl delete pod <pod-name>
   

Instrumented reachability options

You can run the endorctl container instrument command with the following options.

You can run the endorctl container collect command with the following options.

Troubleshoot issues

A container registry is a centralized service that stores and distributes your container images. Endor Labs lets you scan images directly from your registry, giving you full visibility into the security posture of your containerized workloads at scale. You can discover images across repositories, control the scope of your scans, avoid redundant work by skipping images that are already scanned, and run consistent scans over time using saved scan plans.

A scan plan is a JSON file that defines the set of container images to scan, along with the registry and filters used to select them. It acts as a predefined template for selecting container images and can be verified and tested ahead of time before the actual registry scan runs. Once saved, the scan plan can be reused to scan the exact same set of images without querying the registry again, making recurring or batch scans consistent and easier to share across runs or environments.

With registry scanning, you can list all repositories and tags, or a filtered subset, in a registry without manually specifying each image. You can save an enumerated image list as a scan plan and reuse it later so the same set of images is scanned without re-querying the registry each time.

Endor Labs supports the following container registries:

• AWS ECR
• Azure ACR
• Docker Hub
• GitHub Container Registry (GHCR)
• JFrog Artifactory
• Quay

Use the endorctl container registry commands to list and scan images stored in your registry.

• List images from a registry: Use endorctl container registry list to preview which images match your filters before scanning. This lets you verify the scope and adjust filtering parameters such as --include, --exclude, --recent, and --limit. You can also save the results as a scan plan for the scan step.

• Scan images from a registry: Use endorctl container registry scan to enumerate and scan container images from a registry in a single step. You can also provide a saved scan plan from the list command instead of enumerating the registry again.

Use a scan plan when you want to review the list of images before scanning. The scan plans make it easier to reuse these pre-qualified combinations of scanned parameters and ensure consistent results.

List command

The list command connects to your registry, enumerates container images based on your configured filters, and prints a summary with a table of image paths. You can also save the results as a scan plan to reuse with the scan command.

endorctl container registry list --registry-type=<type> [options]

You can apply filters such as include, exclude, recent, and limit to narrow down the images returned. If you provide a namespace and API credentials, the saved plan automatically excludes already scanned images, so it is ready to scan only new or updated images.

Filters are applied in the following order:

1. include
2. exclude
3. recent
4. limit

You can use the endorctl container registry list command with the following flags.

Scan command

The scan command runs Endor Labs container scans on a set of images. You can pass a saved scan plan from the list command or enumerate the registry with the same filter flags as list. The command pulls each image if needed, runs the scan, and by default removes pulled images after scanning. The --namespace and API credentials are required. Images that are already scanned are automatically skipped.

• Scan using a saved scan plan:

  endorctl container registry scan --namespace=<namespace> --scan-plan=<path> [options]
  

• Scan using a registry type. When you do not use --scan-plan, pass --registry-type.

  endorctl container registry scan --namespace=<namespace> --registry-type=<type> [options]
  

You can use the endorctl container registry scan command with the following flags.

Supported container registries

The endorctl container registry list and endorctl container registry scan commands support the following container registries. Use the Registry_type value for --registry-type and the Registry_host value for --registry.

Output format

The list and scan commands both produce output that includes summary lines and, when there are image rows, a table. The scan command shows this when --show-scan-plan is enabled.

If any image rows remain after filters, the command prints a table with the following columns:

Scan plan output

The scan plan is a JSON file written by the endorctl container registry list command with --save-as-plan and read by the scan command with --scan-plan. When list is run with --namespace and API credentials, the saved plan excludes images that are already scanned so that it is ready to scan only new or unscanned images. The structure is:

parameters:
  registry_type: string          # required
  server: string                 # optional
  namespace: string              # optional
  account: string                # optional. Used only for Docker Hub and GHCR.
  repo_key: string               # optional. Used only for JFrog.
  architecture: string           # optional
  include: string                # optional
  exclude: string                # optional
  recent: string                 # optional
  limit: integer                 # optional
  include_untagged: boolean      # optional
  include_untagged_only: boolean # optional
  validate_tag_digest: boolean   # optional
  timeout_seconds: integer      # required

counts:
  repositories: integer
  tags: integer
  untagged_manifests: integer    # optional
  matching_repositories: integer # optional
  matching_tags: integer         # optional
  matching_untagged: integer     # optional
  ignored_repositories: integer  # optional
  ignored_tags: integer          # optional
  ignored_untagged: integer      # optional
  digest_validated_tags: integer # optional
  digest_mismatch_tags: integer  # optional
  digest_lookup_errors: integer  # optional

images:                          # array
  - path: string                 # full image reference, tag or digest
    created: string
    updated: string
    multi_arch: boolean          # optional
    arch: string                 # optional
    multi_arch_digest: string    # optional

Container registry scanning with AWS ECR

The following commands use AWS ECR to show how to list images, apply filters, save a scan plan, and run scans. Use the appropriate --registry-type, --registry, and --registry-namespace values for other registries. See supported container registries to learn more.

• List all images in an AWS ECR registry.

endorctl container registry list --registry-type aws.ecr

• Filter images updated in the last 7 days, include only tags matching latest, and exclude release candidate tags.

endorctl container registry list --registry-type aws.ecr --recent 7d --include '.*:latest' --exclude '.*:-rc.*'

• Save the generated image list to a JSON scan plan file for use with the container registry scan command.

endorctl container registry list --registry-type aws.ecr --save-as-plan registry-scan-plan.json

• List images including untagged manifests.

endorctl container registry list --registry-type aws.ecr --include-untagged

• List only images that match a preferred architecture such as arm64 when the repository contains multi-architecture images.

endorctl container registry list --registry-type aws.ecr --architecture arm64

• Scan images defined in a previously saved scan plan file.

endorctl container registry scan --namespace demo --registry-type=aws.ecr --reauth --scan-plan aws_ecr_scan_plan.json

Troubleshooting

endorctl container registry list --registry-type artifactory --registry jfrog-host --registry-namespace repo-key --save-as-plan registry-scan-plan.json

endorctl container registry list --registry-type=quay --registry-namespace myorg

endorctl container registry list --registry-type=quay --registry http://localhost:8080 --registry-namespace myorg

Endor Labs enhances software supply chain security by providing transparent mechanisms for signing and verifying software artifacts.

• Integrity of container images and build artifacts: Using a cryptographic signature ensures that container images and other build artifacts are genuine and crafted by the organization. This adds an extra layer of security to the software supply chain, making sure that only authorized and unaltered items are scheduled for execution.

• Traces across workflows: Beyond just verification, the framework offers thorough traceability. Users can trace the roots of container images and build artifacts, navigating through workflows and environments. Complete traceability ensures transparency, enabling organizations to validate the entire lifecycle of their software, from creation to deployment.

• Certificate validity: Endor Labs uses a short-lived certificate with a validity period of 5 minutes to ensure that the build artifact has been signed during this time frame. To further guarantee the signing occurred within the valid window, a timestamp is added alongside the certificate and signature, confirming the signing within the specified time frame.

You can sign artifacts using the following methods.

• Using GitHub Action
• Using endorctl CLI

Sign artifacts using GitHub Action

Use the Endor Labs GitHub Actions to sign artifacts.

1. Set up authentication to Endor Labs.
   • (Recommended) If you are using GitHub Action keyless authentication, set an authorization policy in Endor Labs to allow your organization or repository to authenticate. See Keyless Authentication for more information.
   • Alternatively, authenticate with a GCP service account setup for keyless authentication from GitHub Actions or an Endor Labs API key added as a repository secret.
2. Checkout your code.
3. Install your build toolchain.
4. Build your code.
5. Sign your artifacts with Endor Labs.

Endor Labs GitHub Action

Use the GitHub Action endorlabs/github-action/sign@version to sign your artifacts. Set the following input parameters.

See the following example workflows to sign an artifact.

- name: Sign artifacts with Endor Labs
  on: [push, workflow_dispatch]
  name: build
  jobs:
  ko-publish:
    name: Release ko artifact
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      packages: write
      contents: read
    steps:
      - uses: actions/setup-go@v4
        with:
          go-version: '1.20.x'
      - uses: actions/checkout@v3
      - uses: ko-build/setup-ko@v0.6
      - run: ko build
      - name: Login to the GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Publish
        run: KO_DOCKER_REPO=ghcr.io/endorlabs/hello-sign ko publish --bare github.com/endorlabs/hello-sign
      - name: Get Image Digest to Sign
        run: |
          IMAGE_SHA=$(docker inspect ghcr.io/endorlabs/hello-sign:latest | jq -r '.[].Id')
          SIGNING_TARGET="ghcr.io/endorlabs/hello-sign@$IMAGE_SHA"
          echo ARTIFACT="$SIGNING_TARGET" >> $GITHUB_ENV
      - name: Sign with Endor Labs
        uses: endorlabs/github-action/sign@version
        with:
          namespace: "example"
          artifact_name: ${{ env.ARTIFACT }}

Provenance information in signed artifacts

The signed artifacts contain provenance metadata that describe the origin, history, and ownership of an artifact throughout its lifecycle. Including this information in signed artifacts enhances transparency, trustworthiness, and accountability.

The following provenance information is included in the signed artifacts.

Sign artifacts using endorctl

Use the endorctl CLI to sign an artifact. Ensure you have downloaded the latest endorctl binary.

To sign an artifact, run the following command.

endorctl artifact sign --name string --source-repository-ref string --certificate-oidc-issuer string

Specify the following options with the endorctl artifact sign command to include provenance information in your signed artifacts.

View the signed artifacts

To view the signed artifacts:

1. Sign in to Endor Labs and select Inventory from the left sidebar.

2. Select Artifacts. The list shows signed artifacts with Name, Created, and Last Updated details.

   

3. Use the search bar to find artifacts by name, description, or tags. You can use the following filters:

   • Artifact Types: Filter by artifact type, for example container image.
   • Created: Filter by when the artifact was created.

4. Select an artifact to see its signed artifact digests in the list and provenance information. The list shows Artifact Digest, Reference, Created, and Last Updated for each digest.

5. Select an artifact digest to open Artifact Digest Details and view the metadata, signature and certificate details, build configuration, and source repository information.

   

Understand the signing process

When you run the endorctl artifact sign <image> command, Endor Labs initiates the following processes:

• Authentication: Initiates regular authentication and retrieves a token from the OIDC or workflow provider while using an authentication option such as --enable-github-action-token or API keys.
• Key Generation: Generates a public and private key using ECDSA-256.
• Certificate Request: Sends a certificate request to the private Certificate Authority to obtain a short-lived certificate.
• Provenance Inclusion: Incorporates provenance information from the token (if available) or provided with the CLI, adding it as a set of extensions to the certificate using ASN.1 encoding.
• Image Signing: Uses the private key to actively sign the image.
• Certificate Storage: Stores the certificate containing provenance information along with the signature in the database.
• Timestamp: Adds a timestamp of the signing event.

Verify the signed artifacts

To verify a signed artifact, use the following command:

endorctl artifact verify --name <artifact> --certificate-oidc-issuer <issuer>

Use the following command-line options with endorctl artifact verify:

Understand the verification process

When you run the endorctl artifact verify --name <artifact> --certificate-oidc-issuer string command, Endor Labs initiates the following verification processes:

• Authentication: Initiates regular authentication and retrieves a token from the OIDC or workflow provider while using an authentication option such as --enable-github-action-token or API keys.
• Signature Retrieval: Retrieves a signature entry from the database using the artifact name.
  • If the entry is not found, the verification process fails.
• Certificate Authority Check: Checks for a trusted Certificate Authority.
• Image Signature Validation: Validates the image signature using the public key from the certificate.
• Timestamp Validation: Validates that the timestamp in the signature entry is within the certificate’s validity.
• OIDC Issuer Verification: Checks whether the issuer provided matches the contents of the certificate.
• Provenance Verification: Ensures that any provenance information from the CLI matches the ones in the certificate.

Revoke the artifact signature

You can revoke a signature of a signed artifact for reasons such as a precautionary measure to safeguard against security risks, to maintain compliance, or to uphold trust and integrity.

To revoke a signature linked to an artifact and prevent its usage, use the following command:

endorctl artifact revoke-signature --name <image> --source-repository-ref "ref"

Specify the following command-line options for endorctl artifact revoke-signature:

Revoking the artifact signature invalidates the corresponding database entry and ensures that any attempts to verify the signature will fail.

Best practices

• While specifying the artifact name during the signing process, for the container images, adhere to the structure registry.example.com/repository/image@sha256:digest.
• The signing process does not support tags. Ensure that you specify a SHA256 digest with the artifact you are signing to represent a cryptographic hash of the image’s content. This ensures a unique digest is created for every minor alteration in the image.

With the release of the new endorctl container scan commands, the old endorctl scan container commands and their related flags will be removed after a three-month deprecation period.

Use the new dedicated command to ensure continued compatibility.

Mapping of deprecated and new container scan commands

Examples

• To scan a basic container image:

  • Old: endorctl scan --container nginx:latest --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --namespace my-namespace

• To scan a container tar file:

  • Old: endorctl scan --container-tar /path/to/image.tar --namespace my-namespace
  • New: endorctl container scan --image-tar /path/to/image.tar --namespace my-namespace

• To scan a container with a project name:

  • Old: endorctl scan --container nginx:latest --project-name my-nginx --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --project-name my-nginx --namespace my-namespace

• To scan a container in a reference context:

  • Old: endorctl scan --container nginx:latest --container-as-ref --namespace my-namespace
  • New: endorctl container scan --image nginx:latest --as-ref --namespace my-namespace