CI Scans

• 1: Set up keyless authentication
• 2: Scanning with GitHub Actions
• 3: Scanning with Jenkins
• 4: Scanning in GitLab Pipelines
• 5: Scanning in Azure Pipelines
• 6: Scanning in Bitbucket Pipelines
• 7: Scanning with CircleCI
• 8: Scanning with Google Cloud Build

1 - Set up keyless authentication

At Endor Labs, we believe that the most secure secret is one that doesn’t exist. That’s why in CI/CD environments we recommend using keyless authentication for machine authentication. Keyless Authentication uses OAuth for API authentication and removes the need to maintain and rotate an API key to Endor Labs.

Keyless Authentication is more secure and reduces the cost of secret rotation.

Configure keyless authentication using:

• Google Cloud
• GitHub OIDC
• AWS Cloud

Enabling Keyless Authentication in Google Cloud

To enable Keyless Authentication in GCP you’ll first need permissions to create service accounts and assign these accounts roles to GCP.

The workflow to enable keyless authentication is:

1. Create a service account with no permissions for federation.
2. If you do not attach a service account to compute resources or use the default service account, we recommend creating a new service account for the compute resources. Create a service account to attach to compute resources and impersonate the federation service account.
3. Create an authorization policy to allow the federation service account to authenticate to Endor Labs.
4. Provision the compute resources with the appropriate permissions.
5. Test Keyless authentication.

Creating GCP Service Accounts and Authorization Policies

To create your service accounts, first export your GCP project name as an environment variable:

export PROJECT=<insert-gcp-project>

Once you’ve set the environment variable you’ll create a service account that will be used to federate access to Endor Labs and will be provided permission to access the Endor Labs APIs:

Step 1: Create a federation service account called endorlabs-federation:

gcloud iam service-accounts create endorlabs-federation --description="Endor Labs Keyless Federation Service Account" --display-name="Endor Labs Federation Service Account"

We’ll also create a second service account, that will have access to impersonate the endorlabs-federation account. We’ll call this endorlabs-compute-service.

This is needed if you don’t already have service accounts for your compute resources. If you do, you need to modify the existing permissions to allow the existing service account to create a federation token.

Step 2: Create a keyless authentication service account to assign to compute resources called endorlabs-compute-service:

gcloud iam service-accounts create endorlabs-compute-service --description="Endor Labs Service account for keyless authentication" --display-name="Endor Labs Compute Instance SA"

Finally, we’ll assign endorlabs-compute-service permissions to impersonate the endorlabs-federation account to authenticate to Endor Labs through OIDC.

Step 3: Assign the serviceAccountOpenIdTokenCreator role to the endorlabs-compute-service service account:

gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com" --role="roles/iam.serviceAccountOpenIdTokenCreator"

Once we’ve created the necessary account permissions we will create an authorization policy in Endor Labs to allow the account endorlabs-federation to your Endor Labs tenant:

Use the following command to create an authorization policy in Endor Labs.

Note: Make sure to replace <your-tenant> with your Endor Labs tenant name and <insert-your-project> with your GCP project name in the following command.**

endorctl api create -r AuthorizationPolicy -d '{
    "tenant_meta": { "namespace": "<your-tenant>" },
    "meta": {
        "name": "Keyless Auth",
        "kind": "AuthorizationPolicy",
        "tags": ["gcp"]
     },
     "spec": {
        "clause": ["email=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com", "gcp"],
        "target_namespaces": ["<your-tenant>"],
        "propagate": true,
        "permissions": {
            "rules": {},
            "roles": [
                "SYSTEM_ROLE_CODE_SCANNER"
            ]
        },
    }
}'

You’ve now set up the foundation of keyless authentication. You’ll now need to provision your compute resources with the appropriate GCP scopes and service account.

See Provisioning and Testing Keyless Authentication for GKE workloads for instructions on setting up GKE for keyless authentication.

See Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances for instructions on setting up a virtual machine instance for keyless authentication.

Provisioning and Testing Keyless Authentication for GKE workloads

Prerequisites

The following prerequisites are required to setup keyless authentication on GKE workloads:

• Workload identity is enabled on the target GKE cluster. See the GCP documentation on using workload identity for instructions on migrating existing cluster node pools or creating new clusters to use GCP workload identity.
• The gcloud auth plugin is installed and operational on your machine. See the GCP instructions on enabling the gcloud auth plugin for more details.
• The kubectl CLI is installed. See the Kubernetes documentation for instructions.

Procedure

1. (Optional) Create a GKE cluster with workload identity enabled if you do not already have a GKE cluster
2. Authenticate to the GKE cluster
3. (Optional) Create a namespace for Endor Labs scans
4. Create a Kubernetes service account to impersonate your GKE compute service account
5. Bind your Kubernetes service account to your GCP compute service account
6. Annotate your Kubernetes service account with your GCP service account to complete your binding
7. Test a scanning workload using keyless authentication

Setting Up and Testing Keyless authentication in GKE

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT
• The GKE cluster as CLUSTER_NAME

export PROJECT=<Insert_GCP_Project>
export CLUSTER_NAME=<GKE_CLUSTER>

Optional Step 1: To create a GKE cluster with workload identity enabled if you do not already have a GKE cluster with workload identity enabled run the following command:

gcloud container clusters create keyless-test --workload-pool=endor-github.svc.id.goog --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To authenticate to your GKE cluster run the following command:

gcloud container clusters get-credentials $CLUSTER_NAME

Optional Step 3: To create a namespace for Endor Labs scans run the following command:

kubectl create namespace endorlabs

Step 4: To create a Kubernetes service account to impersonate your GKE compute service account run the following command:

kubectl create serviceaccount endorlabs-compute-service -n endorlabs

Step 5: To bind your Kubernetes service account to your GCP compute service account run the following command:

Note: Make sure to replace <insert-your-project> in the following command with your GCP project name.

gcloud iam service-accounts add-iam-policy-binding endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:<insert-your-project>.svc.id.goog[endorlabs/endorlabs-compute-service]"

Step 6: To annotate your Kubernetes service account with your GCP service account to complete your binding run the following command:

Note: If you have created a different service account name replace endorlabs-compute-service with the appropriate service account name.

kubectl annotate serviceaccount endorlabs-compute-service -n endorlabs iam.gke.io/gcp-service-account=endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com

Step 7: Test scan your project with keyless authentication

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication.

Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances

The following high-level procedure describes the required steps to use keyless authentication with a GCP virtual machine instance:

Procedure:

1. Create a virtual machine instance with the appropriate scopes
2. Download and install endorctl on the virtual machine instance
3. Launch a test scan with keyless authentication

Setting up and Testing Keyless authentication on a GCP virtual machine instance

The following instructions require you to export the following environment variables to appropriately run:

• The GCP Project as PROJECT

export PROJECT=<Insert_GCP_Project>

To successfully test keyless authentication first you’ll need to provision a compute resource with the service account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com and the scope https://www.googleapis.com/auth/cloud-platform:

Step 1: To create a virtual machine instance with the appropriate scopes run the following command:

gcloud compute instances create test-keyless --service-account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/cloud-platform

Step 2: To download and install endorctl on the virtual machine instance run the following series of commands:

First, SSH to the virtual machine instance you’ve created to test:

gcloud compute ssh --zone "us-west1-b" "test-keyless"  --project $PROJECT

Then download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com for federated access to Endor Labs such as in the below example:

endorctl api list --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com -r Project -n <insert-your-tenant> --count

If this scan runs successfully you’ve tested and scanned a project with keyless authentication to Endor Labs.

Enabling Keyless Authentication in GitHub

To enable Keyless Authentication for GitHub Actions, you’ll need to perform the following steps:

1. Ensure you are using the Endor Labs GitHub Action in your GitHub workflow.
2. Edit your GitHub Action workflow to add permission settings for the GitHub id-token
3. Create an authorization policy for GitHub Action OIDC
4. Test that you can successfully scan a project using Github Action OIDC

Add a GitHub Action OIDC authorization policy

To ensure that the GitHub action OIDC identity can successfully login to Endor Labs you’ll need to create an authorization policy in Endor Labs.

To create an authorization policy:

1. Under Manage go to Access Control
2. Navigate to the “Auth Policy” tab
3. Click on the “Add Auth Policy” button
4. Select “GitHub Action OIDC” as your identity provider
5. Select the permission for the GitHub Action. This permission should be “Code Scanner”
6. For the claim use the key user and put in a matching value that maps to the organization of your GitHub repository.

Configuring your GitHub Action workflow

To configure your GitHub Action workflow with GitHub Action OIDC you can use the following example as a baseline.

The important items in this workflow are:

1. The Usage of the Endor Labs GitHub action.
2. Setting Job level permissions to allow writing to the GitHub id-token

name: Example Scan of OWASP Java
on: workflow_dispatch
jobs:
  create_project_owasp:
    permissions:
      id-token: write # This is required for requesting the JWT
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3
        with:
          repository: OWASP-Benchmark/BenchmarkJava
      - name: Setup Java
        uses: actions/setup-java@v3
        with:
          distribution: 'microsoft'
          java-version: '17'
      - name: Compile Package
        run: mvn clean install
      - name: Scan with Endor Labs
        uses: endorlabs/github-action@main # This workflow uses the Endor Labs GitHub action to scan.
        with:
          namespace: 'demo'
          scan_summary_output_type: 'json'
          pr: false
          scan_secrets: true
          scan_dependencies: true

Now that you’ve successfully configured your GitHub action workflow file you can use this workflow file or one of your own designs to run a test scan using Keyless authentication for GitHub actions.

Enabling Keyless Authentication in AWS

To enable Keyless Authentication in AWS you’ll first need permissions to create or modify the following roles and an instance profile with the appropriate roles configured.

1. An instance access role - The instance access role is assigned to the compute resource, which needs to access Endor Labs. Your instance access role may already exist and you must ensure this role provides the permissions to allow the role to assume the role of a dedicated federation role.
2. A dedicated federation role - The dedicated federation role should have no permissions in AWS. Endor Labs will authorize requests that come from this role.

Procedure:

1. Create or modify an existing Instance Profile that may be used to assign a role to your EC2 instance.
2. Create or modify a role an instance access role, which enables services to assume a dedicated federation role.
3. Assign this role to the Instance Profile
4. Create a dedicated federation role to provide access to Endor Labs, which will be assumed by the instance access role
5. Create an authorization policy in Endor Labs
6. Test Keyless Authentication

Create or select an Instance Profile

An instance profile allows users to attach a single role to an EC2 instance. If you do not already have a pre-defined role or instance profile used by your EC2 instances you should create an instance profile for Endor Labs access.

To create an instance profile using the AWS CLI:

aws iam create-instance-profile --instance-profile-name EndorLabsAccessProfile

Create or modify an instance access role

To successfully authenticate to Endor Labs you will need to assign an instance access role to the instance profile you created above.

The instance access role must at a minimum allow the compute resources that require access to perform the action sts:AssumeRole. If you already have a role you intend to assign to your instance profile, ensure that it has permissions to allow your compute resources to perform this action.

If you do not have an existing role you intend to use, create the role named endorlabs-instance-access-role using the instructions below.

Add the following json to a file called endorlabs-instance-access-role.json

cat > endorlabs-instance-access-role.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Use this file to create your instance access role using the following command:

aws iam create-role --role-name endorlabs-instance-access-role --assume-role-policy-document file://endorlabs-instance-access-role.json

Next, ensure that the instance access role is assigned to the instance profile using the following command:

aws iam add-role-to-instance-profile --instance-profile-name EndorLabsAccessProfile --role-name endorlabs-instance-access-role

Finally, create your EC2 instance and ensure that your instance profile is assigned to it.

Create a dedicated federation role

A dedicated federation role is leveraged to provide a least privileged role that enables access to Endor Labs. This role is designed to be assumed only by specific other roles and does not provide access to AWS resources.

To create your federation role you will need the name and AWS account number of the instance access role, which should look similar to the following policy json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::$ACCOUNT:role/$ROLE_NAME"
            },
            "Action": "sts:AssumeRole"
        }
    ]
  }

First, get the account number of the role:

export ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')

Then define the name of the instance access role. For the following example, we will assume it is endorlabs-instance-access-role.

export ROLE_NAME=endorlabs-instance-access-role

Next, create the IAM policy document.

cat > endorlabs-federation-aws-role.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::${ACCOUNT}:role/${ROLE_NAME}"
          },
          "Action": "sts:AssumeRole"
      }
  ]
}
EOF

Next, apply the policy document as a role:

aws iam create-role --role-name endorlabs-federation --assume-role-policy-document file://endorlabs-federation-aws-role.json

Finally, fetch the ARN of the IAM role you’ve created using the following command and create an authorization policy for it in Endor Labs.

To fetch the ARN of the Endor Labs federation role use the following command:

aws iam list-roles | jq -r '.Roles[] | select(.Arn|contains("endorlabs-federation"))'.Arn

To add the authorization policy to Endor Labs:

1. Login to Endor Labs as an administrator.
2. Under Manage, navigate to Access Control > Auth Policy
3. Click Add Auth Policy
4. Under Identity Provider Select AWS role
5. Provide the appropriate permissions for your authorization policy.
6. Under claims use the Key User and the value of the ARN that you fetched in the previous command.
7. Click Save Auth Policy to finalize your keyless authentication setup.

Testing Keyless Authentication with AWS

On the EC2 instance you’ve configured for keyless authentication, download and install the latest version of endorctl. See our documentation for instructions on downloading the latest version

To scan with keyless authentication you must use the flag --aws-role-arn=<insert-your-arn> for federated access to Endor Labs such as in the below example:

endorctl --aws-role-arn=<insert-your-arn> api list -r Project -n <insert-your-endorlabs-tenant> --page-size=1

You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication with AWS.

2 - Scanning with GitHub Actions

GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can use GitHub Actions to include Endor Labs into your CI pipeline seamlessly.

Using this pipeline, developers can view and detect:

• Policy violations in the source code
• Secrets inadvertently included in the source code

The Endor Labs verifications are conducted as automated checks and help you discover violations before pushing code to the repository. Information about the violations can be included as comments on the corresponding pull request (PR). This enables developers to easily identify issues and take remedial measures early in the development life cycle.

• For policy violations, the workflow is designed to either emit a warning or return an error based on your action policy configurations.
• For secrets discovered in the commits, developers can view the PR comments and take necessary remedial measures.

Install Software Prerequisites

To ensure the successful execution of the Endor Labs GitHub action, the following prerequisites must be met:

• The GitHub action must be able to authenticate with the Endor Labs API.
• You must have the value of the Endor Labs namespace handy for authentication.
• You must have access to the Endor Labs API.
• If you use keyless authentication, you must set an authorization policy in Endor Labs. See Authorization policies for details.

Example GitHub Action Workflow

Endor Labs scanning workflow using GitHub actions that accomplishes the following tasks in your CI environment:

• Tests PRs to the default branch and monitors the most recent push to the default branch.
• Builds a Java project and sets up the Java build tools. If your project is not on Java, then configure this workflow with your project-specific steps and build tools.
• Authenticates to Endor Labs with GitHub Actions keyless authentication.
• Scan with Endor Labs.
• Comments on PRs if any policy violations occur.
• Generates findings and uploads results to GitHub in SARIF format.

The following example workflow shows how to scan with Endor Labs for a Java application using the recommended keyless authentication for GitHub actions:

name: Endor Labs Dependency and Secrets Scan
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
jobs:
  scan:
    permissions:
      security-events: write # Used to upload Sarif artifact to GitHub
      contents: read # Used to check out a private repository
      actions: read # Required for private repositories to upload Sarif files. GitHub Advanced Security licenses are required.
      id-token: write # Used for keyless authentication with Endor Labs
      pull-requests: write # Required to automatically comment on PRs for new policy violations
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: Endor Labs Scan Pull Request
      if: github.event_name == 'pull_request'
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: true
        enable_pr_comments: true # Required to automatically comment on PRs for new policy violations
        github_token: ${{ secrets.GITHUB_TOKEN }} # Required for PR comments on new policy violations

  scan-main:
    permissions:
      id-token: write
      repository-projects: read
      pull-requests: read
      contents: read
    name: endorctl-scan
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Repository
      uses: actions/checkout@v3
    - name: Setup Java
      uses: actions/setup-java@v3
      with:
        distribution: 'microsoft'
        java-version: '17'
    - name: Build Package
      run: mvn clean install
    - name: 'Endor Labs Scan Push to main'
      if: ${{ github.event_name == 'push' }}
      uses: endorlabs/github-action@v1.1.4
      with:
        namespace: 'example' # Replace with your Endor Labs tenant namespace
        scan_dependencies: true
        scan_secrets: true
        pr: false
        scan_summary_output_type: 'table'
        sarif_file: 'findings.sarif'
    - name: Upload findings to github
      uses: github/codeql-action/upload-sarif@v3
      with:
        sarif_file: 'findings.sarif'

Authenticate with Endor Labs

Endor Labs recommends using keyless authentication in CI environments. Keyless authentication is more secure and reduces the cost of secret rotation. To set up keyless authentication see Keyless Authentication.

If you choose not to use keyless authentication, you can configure an API key and secret in GitHub for authentication as outlined in Managing API keys.

Authentication Without Keyless Authentication for GitHub

If you are not using keyless authentication for GitHub Actions, you must not provide id-token: write permissions to your GitHub token unless specifically required by a step in this job. You must also set enable_github_action_token: false in your Endor Labs GitHub action configuration.

The following example configuration uses the Endor Labs API key for authentication:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          api_key: ${{ secrets.ENDOR_API_CREDENTIALS_KEY }}
          api_secret: ${{ secrets.ENDOR_API_CREDENTIALS_SECRET }}
          enable_github_action_token: false

The following example configuration uses a GCP service account for keyless authentication to Endor Labs:

      - name: Scan with Endor Labs
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example'
          gcp_service_account: '<Insert_Your_Service_Account>@<Insert_Your_Project>.iam.gserviceaccount.com'
          enable_github_action_token: false

Endor Labs GitHub Action Configuration Parameters

The following input configuration parameters are supported for the Endor Labs GitHub Action:

Common parameters

The following input global parameters are supported for the Endor Labs GitHub action:

Scanning parameters

The following input parameters are also supported for the Endor Labs GitHub action when used for scanning:

Configure Endor Labs Action Policies

Configure an action policy in the Endor Labs user interface to perform an action when a rule is triggered. See Action Policies for details on action policies.

• Set the Policy Template to Detected Secrets and select the Template Parameters as desired.
• Choose Enforce Policy and
  • Select Warn as the recommended action.
  • Select Break the Build to fail the build CI pipeline.

PR Comments

You can configure GitHub Actions to comment on PRs if there are any policy violations. When you raise a PR, Endor Labs scans and detects policy violations. A comment is added in the PR with the details of violation. The CI pipeline warns you or fails the build based on your action policy configuration. The PR comments also include recommendations to help you take necessary remedial actions.

You can customize the template of the comment according to your requirement.

Enable PR Comments

Make sure that your GitHub action workflow includes the following configuration.

• The workflow must have a with clause including: enable_pr_comments to true to publish new findings as review comments and github_token: ${{ secrets.GITHUB_TOKEN }}. This token is automatically provisioned by GitHub when using GitHub actions. See GitHub configuration parameters for more information.
• To grant Endor Labs the ability to comment on PRs you must include the permission pull-requests: write.

The following example configuration comments on PRs if a policy violation is detected.

      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1.1.4
        with:
          namespace: 'example' # Update with your Endor Labs namespace
          scan_summary_output_type: 'table'
          scan_dependencies: true
          scan_secrets: true
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

PR Comments Example

You can view a live example of a GitHub repository with a workflow that enables PR comments.

The main.yaml file contains the following configuration to enable PR comments.

name: Build Release
on:
  pull_request:
    branches: [main]
  workflow_dispatch:
  push:
    branches: [main]
  schedule:
    - cron: "23 23 * * 0"
jobs:
  build:
    permissions:
      pull-requests: write
      security-events: write
      contents: read
      id-token: write
      actions: read
    runs-on: ubuntu-latest
    env:
      ENDOR_NAMESPACE: "endorlabs-hearts-github"
    steps:
      - name: Endor Labs Scan PR to Default Branch
        if: github.event_name == 'pull_request'
        uses: endorlabs/github-action@v1
        with:
          namespace: ${{ env.ENDOR_NAMESPACE }}
          pr: true
          enable_pr_comments: true
          github_token: ${{ secrets.GITHUB_TOKEN }}

The PR #10 introduced a reachable vulnerability. Since the workflow has enable_pr_comments set as true, a comment is added to the PR on the policy violation.

You can expand the comment to view more details about the violation and take steps to resolve the issue.

Customize GitHub PR comments notification templates

Endor Labs provides a default template with standard information that will be included in your pull requests as comments. You can use the default template, or you can choose to edit and customize this template to fit your organization’s specific requirements. You can also create custom templates using Go Templates.

1. Sign in to Endor Labs and navigate to Manage>Integrations
2. Look for GitHub PR comments under Notifications.
3. Click Edit Template.
4. Make the required changes and click Save Template.
5. Click Restore to Default to revert the changes.
6. Use the download icon in the top right corner to download this template.
7. Use the copy icon to copy the information in the template.

Data model

To create custom templates for PR comments, you must understand the data supplied to the template.

See the following protobuf specification for the GithubCommentData message that this template uses.

syntax = "proto3";

package internal.endor.ai.endor.v1;

import "google/protobuf/wrappers.proto";
import "spec/internal/endor/v1/finding.proto";
import "spec/internal/endor/v1/package_version.proto";

option go_package = "github.com/endorlabs/monorepo/src/golang/spec/internal.endor.ai/endor/v1";
option java_package = "ai.endor.internal.spec";

// The list of finding UUIDs.
message FindingUuids {
  repeated string uuids = 1;
}

// The map of dependency name to findings.
message DependencyToFindings {
  map<string, FindingUuids> dependency_to_findings = 1;
}

// The map of PackageVersion UUID to DependencyToFindings.
message PackageToDependencies {
  map<string, DependencyToFindings> package_to_dependencies = 1;
}

message GithubCommentData {
  // The header of the PR comment. Identifies the PR comment published by Endor Labs.
  // It should always be at top of the template.
  google.protobuf.StringValue comment_header = 1;

  // The footer of the PR comment.
  google.protobuf.StringValue comment_footer = 2;

  // The map of finding UUID to finding object.
  map<string, internal.endor.ai.endor.v1.Finding> findings_map = 3;

  // The map of policy UUID to policy name.
  // This will contain only the policies that are triggered or violated.
  map<string, string> policies_map = 4;

  // The map of policy UUID to the list of finding UUIDs.
  map<string, FindingUuids> policy_findings_map = 5;

  // The map of PackageVersion UUID to PackageVersion object.
  map<string, internal.endor.ai.endor.v1.PackageVersion> package_versions_map = 6;

  // The data needs to be grouped as follows:
  //
  // - Policy 1
  // 		- Package 1
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 2
  //			- Dependency Package 2
  //				- Finding 3
  //				- Finding 4
  // 		- Package 2
  //			- Dependency Package 1
  //				- Finding 1
  //				- Finding 5
  // - Policy 2
  //		....
  //
  //		Map 0[PolicyUUID]/Map 1[PkgVerUUID]/Map 2 [Dep Names]/Finding UUID
  map<string, PackageToDependencies> data_map = 7;

  google.protobuf.StringValue api_endpoint = 8;
}

To understand Finding and PackageVersion definitions that are used in this protobuf specification, see:

• Finding resource kind
• PackageVersion resource kind

See the following specification to understand the additional functions that are also available. You can access these functions by using their corresponding keys.

// FuncMap contains the additional functions that are available to GithubCommentTemplate.
var FuncMap = template.FuncMap{
	"now": toTime, // 'now' gives the current time

	// 'enumToString' coverts the enums for finding level, finding category and finding tags to string
	"enumToString": enumToString,

	// 'getPackageVersionURL' returns the URL for a given PackageVersion
	"getPackageVersionURL": func(apiURL string, packageVersion *endorpb.PackageVersion) string {
		result, err := common.GetPackageVersionURL(apiURL, packageVersion)
		if err != nil {
			return ""
		}
		return result
	},

	// 'getFindingURL' returns the URL for a given Finding
	"getFindingURL": func(apiURL string, finding *endorpb.Finding) string {
		result, err := common.GetFindingURL(apiURL, finding)
		if err != nil {
			return ""
		}
		return result
	},

	// 'add' returns the sum of two integers
	"add": func(n int, incr int) int {
		return n + incr
	},

	// 'getOtherFindingsPackageMarker' returns the key for _findingsWithNoPackages for lookup in DataMap
	// Not all findings are associated with a PackageVersion, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsPackageMarker": func() string { return _findingsWithNoPackages },

	// 'getOtherFindingsDependencyMarker' returns the key for _findingsWithNoDeps for lookup in DataMap
	// Not all findings are associated with a dependency, such findings are grouped under this key
	// in the DataMap
	"getOtherFindingsDependencyMarker": func() string { return _findingsWithNoDeps },

	// 'getFindingsCountString' returns a string with number of findings, example - "5 findings"
	"getFindingsCountString": func(dataMap *endorpb.PackageToDependencies) string {
		count := 0

		for _, depMap := range dataMap.PackageToDependencies {
			for _, findingMap := range depMap.DependencyToFindings {
				count += len(findingMap.Uuids)
			}
		}

		findingsStr := "findings"
		if count == 1 {
			findingsStr = "finding"
		}

		return fmt.Sprintf("%d %s", count, findingsStr)
	},
}

3 - Scanning with Jenkins

Jenkins is an open-source automation server widely used for building, testing, and deploying software. Specifically in the context of CI/CD pipelines, Jenkins serves as a powerful tool to automate various stages of the software development lifecycle.

To integrate Endor Labs into your Jenkins CI/CD processes:

1. Authenticate to Endor Labs
2. Install NodeJS plugin in Jenkins.
3. Install your build toolchain
4. Build your code
5. Scan with Endor Labs

Authenticate to Endor Labs

To configure keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in Jenkins for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.

1. In your Jenkins environment, navigate to Manage Jenkins.
2. Enter a credential name for reference such as endorlabs or re-use an existing context.
3. Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
4. In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
5. Select Add Environment Variable.

Install NodeJS plugin in Jenkins

See Jenkins documentation to install nodeJS plugin in Jenkins. You must have the nodeJS plugin to use npm and download endorctl.

Configure your Jenkins pipeline

To create a Jenkins pipeline:

1. Create a configuration pipeline file in your repository if you do not already have one using the pipeline project.
2. In your configuration pipeline file customize the job configuration based on your project’s requirements using one of the examples, simple Jenkins configuration or Jenkins pipeline using curl.
3. Ensure that the context you created is part of the workflow if you are not using keyless authentication.
4. Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
5. Update your Endor Labs tenant namespace to the appropriate namespace for your project.
6. Update your default branch from main if you do not use main as the default branch name.
7. Modify any dependency or artifact caches to align with the languages and caches used by your project.

Examples

Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.

Simple Jenkins configuration using npm

pipeline {
    agent any
    tools {nodejs "NodeJS"}
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY_1')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET_1')
    }
    stages {
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                script {
                    // Define the Node.js installation name configured in Jenkins
                    NODEJS_HOME = tool name: 'NodeJS', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
                    PATH = "$NODEJS_HOME/bin:${env.PATH}"
                }

                // Download and install endorctl.
                sh 'npm install -g endorctl'
                // Check endorctl version and installation.
                sh 'endorctl --version'
                // Run the scan.
                sh('endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET')

            }
        }

        stage('Results') {
            steps {
                // Publish or process the vulnerability scan results
                // Publish reports, fail the build on vulnerabilities, etc.
                echo 'Publish results'
            }
        }
    }

}

Jenkins pipe line for curl to download endorctl binary

The following example includes curl to download the endorctl binary.

pipeline {
    agent any

    // Endorctl scan uses following environment variables to the trigger endorctl scan
    environment {
        ENDOR_API = credentials('ENDOR_API')
        ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
        ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY')
        ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET')
    }
    stages {
        // Not required if repository is allready cloned to trigger a endorctl scan
        stage('Checkout') {
            steps {
                // Checkout the Git repository
                checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
            }
        }

        stage('Build') {
            // Not required if project is already built
            steps {
                // Perform any build steps if required
                sh 'mvn clean install'
            }
        }

        stage('endorctl Scan') {
            steps {
                // Download and install endorctl.
                sh '''#!/bin/bash
                    echo "Downloading latest version of endorctl"
                    VERSION=$(curl $ENDOR_API/meta/version | jq -r '.Service.Version')
                    ENDORCTL_SHA=$(curl $ENDOR_API/meta/version | jq -r '.ClientChecksums.ARCH_TYPE_LINUX_AMD64')
                    curl $ENDOR_API/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
                    echo "$ENDORCTL_SHA  endorctl" | sha256sum -c
                    if [ $? -ne 0 ]; then
                        echo "Integrity check failed"
                        exit 1
                    fi
                    chmod +x ./endorctl
                    // Check endorctl version and installation.
                    ./endorctl --version
                    // Run the scan.
                    ./endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
                '''
            }
        }
    }
}

Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.

4 - Scanning in GitLab Pipelines

GitLab CI/CD pipelines are a part of GitLab’s integrated continuous integration and deployment features. They allow you to define and automate the different stages and tasks in your software development workflow.

This documentation page provides an overview and example job to integrate Endor Labs into your GitLab CI pipeline.

High Level Usage Steps

1. Setup authentication to Endor Labs
2. Install your build toolchain
3. Build your code
4. Scan with Endor Labs

Authentication to Endor Labs

Endor Labs recommends using keyless authentication in continuous integration environments.

Keyless Authentication is more secure and reduces the cost of secret rotation.

To setup keyless authentication see the keyless authentication documentation

If you choose not to use keyless authentication you can configure an API key and secret in GitLab for authentication using the following steps. See managing API keys for more information on getting an API key for Endor Labs authentication

1. In your GitLab environment, select the project you want to scan.
2. Go to Settings > CI/CD.
3. Click Expand in the Variables section.
4. Click the Add variable button at the bottom of the section.
5. In the Key field, enter ENDOR_API_CREDENTIALS_SECRET.
6. In the Value field, enter your Endor Labs API secret.
7. Under Flags, make sure you select Mask variable.
8. Repeat the previous steps to add your API key as the variable ENDOR_API_CREDENTIALS_KEY.

Configuring your GitLab CI pipeline

1. Create a .gitlab-ci.yml file in the root directory of your project if you do not already have one.
2. In your .gitlab-ci.yml file customize the job configuration based on your project’s requirements using the example below.
3. Modify the image field to align with the build tools necessary for building your software packages.
4. Update the before_script section to include any additional steps required before executing the scan, such as installing dependencies or building your project.
5. Save and commit the .gitlab-ci.yml file to your GitLab repository.
6. GitLab will automatically detect the .gitlab-ci.yml file and trigger the defined job whenever there are changes pushed to the repository.
7. Monitor the progress and results of the CI pipeline in the GitLab CI/CD interface.

Below is an example job to get you started. Please ensure to customize this job with your specific build environment and build steps as needed.

# You can copy and paste this template into a new `.gitlab-ci.yml` file.
# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword.
#
stages:
  - Scan
Endor Labs Dependency Scan:
  stage: test
  image: node # Modify this image to align with the build tools nessesary to build your software packages
  dependencies: []
  variables:
   ## Scan scoping section
   #
   ## Use the following environment variables for custom paths, inclusions and exclusions.
   # ENDOR_SCAN_PATH: "Insert a custom path to your git repository. Defaults to your pwd"
   # ENDOR_SCAN_EXCLUDE_PATH: "Insert a Glob style pattern of paths to exclude in the scan. Generally used for monorepos."
   # ENDORCTL_SCAN_INCLUDE_PATH: "Insert a Glob style pattern of paths to include in the scan. Generally used for monorepos."
   #
   ## Authentication to Endor Labs
   #
   ## Use the following environment variables for keyless authentication with your cloud provider. For more information visit: https://docs.endorlabs.com/continuous-integration/keyless-authentication/
   #
   # ENDOR_GCP_CREDENTIALS_SERVICE_ACCOUNT: "endorlabs@<yourproject.iam.gserviceaccount.com"
   # ENDOR_AWS_CREDENTIALS_ROLE_ARN: "arn:aws:iam::123456789012:role/my-role"
   #
   ## Follow the below instructions to use an API key and secret instead of keyless authentication.
   ## In your GitLab environment, select the project you want to scan.
   ## Go to `Settings` > `CI/CD`.
   ## Click `Expand` in the `Variables` section.
   ## Click the `Add variable` button at the bottom of the section.
   ## In the `Key` field, enter ENDOR_API_CREDENTIALS_SECRET.
   ## In the `Value` field, enter your Endor Labs API secret.
   ## Under `Flags` make sure you select `Mask variable`.
   ## Repeat to add your API key as the variable ENDOR_API_CREDENTIALS_KEY
   #
    ENDOR_ENABLED: "true"
    ENDOR_ALLOW_FAILURE: "false"
    ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    ENDOR_PROJECT_DIR: "."
    ENDOR_ARGS: |
      --path=${ENDOR_PROJECT_DIR}
      --detached-ref-name=$CI_COMMIT_REF_NAME
      --output-type=summary
      --exit-on-policy-warning
      --dependencies --secrets --git-logs      
  before_script:
    - npm install yarn # Replace with the build steps for your Endor Labs job.
  script:
    - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl;
    - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
      if [ $? -ne 0 ]; then
       echo "Integrity check failed";
       exit 1;
      fi
    - chmod +x ./endorctl
    - if [ "$DEBUG" == "true" ]; then
        export ENDOR_LOG_VERBOSE=true;
        export ENDOR_LOG_LEVEL=debug;
      fi
    - if [ "$CI_COMMIT_REF_NAME" == "$CI_DEFAULT_BRANCH" ]; then
        export ENDOR_SCAN_AS_DEFAULT_BRANCH=true;
        export ENDOR_SCAN_DETACHED_REF_NAME="$CI_COMMIT_REF_NAME";
      else
        export ENDOR_SCAN_PR=true;
      fi
    - ./endorctl scan ${ENDOR_ARGS}
  rules:
  - if: $ENDOR_ENABLED != "true"
    when: never
  - if: $ENDOR_ALLOW_FAILURE == "true"
    allow_failure: true
  - if: $ENDOR_ALLOW_FAILURE != "true"
    allow_failure: false

5 - Scanning in Azure Pipelines

Azure Pipelines is a continuous integration and continuous delivery (CI/CD) service available in Azure DevOps ecosystem. It facilitates continuous integration, continuous testing, and continuous deployment for seamless building, testing, and delivery of software.

Complete the Prerequisites

Ensure that you complete the following prerequisites before you proceed.

Set up an Endor Labs tenant

You must have an Endor Labs tenant set up for your organization. You can also set up namespaces according to your requirements. See Set up namespaces

Configure Endor Labs authentication

Configure an API key and secret for authentication. See managing API keys for more information on generating an API key for Endor Labs. Store API key and secret as environment variables, ENDOR_API_CREDENTIALS_KEY and ENDOR_API_CREDENTIALS_SECRET.

Enable Advanced Security in Azure

You need to enable Advanced Security in your Azure repository to view results in Azure.

1. Log in to Azure and open Project Settings.
2. Navigate to Repos > Repositories in the left navigation panel.
3. Select your repository.
4. Enable Advanced Security.

Configure Endor Labs variables in the pipeline

You can manage Endor Labs variables centrally by configuring them within your Azure project. You can assign these variables to various pipelines.

1. Log in to Azure and select Pipelines > Library.
2. Click +Variable Group to add a new variable group for Endor Labs.
3. Enter a name for the variable group, for example, tenant-variables, and click Add under Variables.
4. Add the following variables.
   • ENDOR_API_CREDENTIALS_KEY
   • ENDOR_API_CREDENTIALS_SECRET
   • NAMESPACE
5. Select the variable group that you created.
6. Click Pipeline Permissions.
7. Click + to add the pipelines in which you want to use the variable group.

Configure your Azure pipeline

1. Create azure-pipelines.yml file in your project, if it doesn’t exist.

2. In the azure-pipelines.yml file, customize the job configuration based on your project’s requirements.

3. Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project. For example, update the node pool settings based on your operating system.

```bash
pool:
name: Default
vmImage: "windows-latest"
```

</div>
<div class="tab-body tab-pane fade"
    id="tabs-00-01" role="tabpanel" aria-labelled-by="tabs-00-01-tab" tabindex="0">
    <pre><code>```bash

</code></pre>

    </div>
    <div class="tab-body tab-pane fade"
        id="tabs-00-02" role="tabpanel" aria-labelled-by="tabs-00-02-tab" tabindex="0">
        <pre><code>```bash
pool:
name: Default
vmImage: &quot;macOS-latest&quot;

</div>

 - bash: |
     echo "Downloading latest version of endorctl"
     VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
     curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
     echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe)  endorctl" | sha256sum -c
     if [ $? -ne 0 ]; then
       echo "Integrity check failed"
       exit 1
     fi     

 - bash: |
     echo "Downloading latest version of endorctl"
     VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
     curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
     echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
     if [ $? -ne 0 ]; then
       echo "Integrity check failed"
       exit 1
     fi     
 ```

 - bash: |
     echo "Downloading latest version of endorctl"
     VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
     curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
     echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64)  endorctl" | shasum -a 256 --check
     if [ $? -ne 0 ]; then
       echo "Integrity check failed"
       exit 1
     fi     
 ```

 - script: |
     .\endorctl.exe scan -n $(NAMESPACE) -s scanresults.sarif
     env:
       ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
       ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)     

 - script: |
     .\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
     env:
       ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
       ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)     

 - script: |
     .\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
     env:
       ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
       ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)     

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "windows-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
    - bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
       echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe)  endorctl" | sha256sum -c
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
    ```    
  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    .\endorctl.exe scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)\

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "ubuntu-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
    - bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
        ## Modify the permissions of the binary to ensure it is executable
        chmod +x ./endorctl
        ## Create an alias of the endorctl binary to ensure it is available in other directories
        alias endorctl="$PWD/endorctl"    

  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    ./endorctl scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)/

trigger:
- none

pool:
  name: Azure Pipelines
  vmImage: "macos-latest"

variables:
- group: tenant-variables

steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
        echo "Downloading latest version of endorctl"
        VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
        curl https://api.endorlabs.com/download/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64)  endorctl" | shasum -a 256 --check
        if [ $? -ne 0 ]; then
          echo "Integrity check failed"
          exit 1
        fi
        ## Modify the permissions of the binary to ensure it is executable
        chmod +x ./endorctl
        ## Create an alias of the endorctl binary to ensure it is available in other directories
        alias endorctl="$PWD/endorctl"        
  displayName: 'Downloading latest version of endorctl'
  continueOnError: false

- script: |
    ./endorctl scan -n $(NAMESPACE) -s scanresults.sarif    
  displayName: 'Run a scan against the repository using your API key & secret pair'
  env:
    ENDOR_API_CREDENTIALS_KEY: $(ENDOR_API_CREDENTIALS_KEY)
    ENDOR_API_CREDENTIALS_SECRET: $(ENDOR_API_CREDENTIALS_SECRET)

- task: AdvancedSecurity-Publish@1
  displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
  inputs:
   SarifsInputDirectory: $(Build.SourcesDirectory)/

simage: maven:3.6.3-jdk-11

pipelines:
  branches:
    main:
      - step:
          name: "Build and Test"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
  pull-requests:
    '**':
      - step:
          name: "Build and Test on PR to Main"
          script:
            - mvn install -DskipTests
            - echo "Running Endor Labs PR Scan"
            - curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            - echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
            - chmod +x ./endorctl
            - ./endorctl scan --pr --pr-baseline=main --languages=java --output-type=json -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET | tee output.json
            #Optional - Comment on the PR
            # - apt-get update
            # - apt-get install -y python3 python3-pip
            # - pip3 install -r requirements.txt
            # - python3 add-bitbucket-pr-comments.py output.json

import json
import os
import sys
import requests
import argparse

# Check for required environment variables
BITBUCKET_REPO_OWNER = os.getenv('BITBUCKET_REPO_OWNER')
BITBUCKET_REPO_SLUG = os.getenv('BITBUCKET_REPO_SLUG')
BITBUCKET_COMMIT = os.getenv('BITBUCKET_COMMIT')

if not all([BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT]):
     sys.exit("Error: One or more required environment variables (BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT) are not set.")

#This is an internal proxy running in the BitBucket environment to accept Code Insights
proxies = {"http": "http://localhost:29418"}

def load_json_with_unescaped_characters(file_path):
    """Load and return JSON data from a file, replacing unescaped characters if necessary."""
    try:
        with open(file_path, 'r', encoding='utf-8') as file:
            json_str = file.read().strip()
        return json.loads(json_str)
    except json.decoder.JSONDecodeError as e:
        print(f"Failed to parse JSON: {e}")
        return None
    except FileNotFoundError:
        print(f"File not found: {file_path}")
        sys.exit()

def construct_report_payload(endor_findings):
    """Construct and return the payload for creating a Bitbucket report."""
    warning_findings_count = len(endor_findings.get('warning_findings', []))
    blocking_findings_count = len(endor_findings.get('blocking_findings', []))
    total_violations = warning_findings_count + blocking_findings_count
    result = "PASSED" if total_violations == 0 else "FAILED"
    report_payload = {
        "title": "Endor Labs Policy Violations",
        "details": f"Endor Labs detected {total_violations} policy violations associated with this pull request.\n\n{endor_findings['warnings'][0]}",
        "report_type": "SECURITY",
        "reporter": "Endor Labs",
        "link": f"https://app.endorlabs.com/t/{namespace}/projects/{project_uuid}/pr-runs/{report_id}",
        "logo_url": "https://avatars.githubusercontent.com/u/92199924",
        "result": result,
        "data": [
            {"title": "Warning Findings", "type": "NUMBER", "value": warning_findings_count},
            {"title": "Blocking Findings", "type": "NUMBER", "value": blocking_findings_count}
        ]
    }
    return report_payload

def construct_annotation_payload(finding):
    """Construct and return the payload for creating an annotation in Bitbucket."""
    title = "Endor Labs Policy Violation"
    summary = finding['meta']['description']
    details =  f"{finding['spec']['summary']}\n\n{finding['spec']['remediation']}"
    severity = "CRITICAL" if finding['spec']['level'] == "FINDING_LEVEL_CRITICAL" else \
               "HIGH" if finding['spec']['level'] == "FINDING_LEVEL_HIGH" else \
               "MEDIUM" if finding['spec']['level'] == "FINDING_LEVEL_MEDIUM" else "LOW"
    affected_paths = finding['spec'].get('dependency_file_paths', [])
    path = affected_paths[0] if affected_paths else "Unknown file"
    annotation_payload = {
        "external_id": finding['uuid'],
        "title": title,
        "annotation_type": "VULNERABILITY",
        "summary": summary,
        "details": details,
        "severity": severity,
        "path": path
    }
    return annotation_payload

def send_report(report_payload):
    """Send the constructed report payload to the Bitbucket API."""
    report_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports/{report_id}"
    response = requests.put(report_url, json=report_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Report created or updated successfully")
    else:
        print(f"Failed to create or update report: {response.text}")

def send_annotation(annotation_payload):
    """Send the constructed annotation payload to the Bitbucket API."""
    annotation_url = f"{base_url}/{report_id}/annotations/{annotation_payload['external_id']}"
    response = requests.put(annotation_url, json=annotation_payload, proxies=proxies)
    if response.status_code in [200, 201]:
        print("Annotation added successfully")
    else:
        print(f"Failed to add annotation: {response.text}")

def process_findings(filename):
    """Load findings from JSON, create a report, and add annotations for each finding."""
    endor_findings = load_json_with_unescaped_characters(filename)
    if endor_findings is None:
        print("Failed to load findings. Exiting.")
        return

    global report_id, project_uuid, namespace

    # Define the order of keys to check
    finding_types = ['all_findings', 'warning_findings', 'blocking_findings']

    # Iterate over finding types and extract the first one found
    for finding_type in finding_types:
        if endor_findings.get(finding_type):
            first_finding = endor_findings[finding_type][0]
            report_id = first_finding['context']['id']
            project_uuid = first_finding['spec']['project_uuid']
            namespace = first_finding['tenant_meta']['namespace']
            break  # Stop after finding the first non-empty list

    if not report_id:
        print("No findings found.")
        sys.exit()

    # Prepare the base URL for Bitbucket API requests
    global base_url
    base_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports"

    # Create the report
    report_payload = construct_report_payload(endor_findings)
    send_report(report_payload)

    # Iterate over findings and create annotations
    for finding in endor_findings.get('blocking_findings', []) + endor_findings.get('warning_findings', []):
        annotation_payload = construct_annotation_payload(finding)
        send_annotation(annotation_payload)

def main():
    """Main function to parse arguments and process findings."""
    parser = argparse.ArgumentParser(description="Script to process findings and update Bitbucket via API.")
    parser.add_argument("filename", help="Filename containing the JSON findings.")
    args = parser.parse_args()

    process_findings(args.filename)

if __name__ == "__main__":
    main()

version: 2.1

jobs:
  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            
  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
    steps:
      - checkout
      - run:
          name: "Build"
          command: |
            mvn clean install -Dskiptests            
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - watch-endorlabs-scan:
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - test-endorlabs-scan:
          context:
            - endorlabs

# You can copy and paste portions of this `config.yml` file as an easy reference.
#
version: 2.1

jobs:
  build:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build steps
    steps:
      - checkout
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Build Your Project"
          command: |
            mvn clean install            
      - persist_to_workspace:
          root: .
          paths:
            - target/ # Persist artifact across job. Change this if you are creating your artifact in a location outside of the target directory.
      - save_cache:
          paths:
            - ~/.m2/repository
          key: maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}

  test-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example"
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Test"
          command: |
            ./endorctl scan --pr --pr-baseline=main --dependencies --secrets            

  watch-endorlabs-scan:
    docker:
      - image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
    environment:
      ENDORCTL_VERSION: "latest"
      ENDOR_NAMESPACE: "example" #Replace with your namespace in Endor Labs
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            # when lock file changes, use increasingly general patterns to restore cache
            - maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
            - maven-repo-v1-{{ .Branch }}-
            - maven-repo-v1-
      - run:
          name: "Install endorctl"
          command: |
            curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
            echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c;
            if [ $? -ne 0 ]; then
              echo "Integrity check failed";
              exit 1;
            fi
            chmod +x ./endorctl
            ./endorctl --version            
      - run:
          name: "Endor Labs Watch"
          command: |
            ./endorctl scan --dependencies --secrets            
workflows:
  build_and_endorlabs_watch:
    when:
      equal: [ main, << pipeline.git.branch >> ]
    jobs:
      - build
      - watch-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs
  build_and_endorlabs_test:
    jobs:
      - build
      - test-endorlabs-scan:
          requires:
            - build
          context:
            - endorlabs

steps:
  # Step 1: Fetch The Trigger Branch
  # This step addresses a known issue where Cloud Build renames the pulled branch to main.
  # If you are not encountering this issue with your build, you can skip this step.
  - name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        echo "Fetching all branches..."
        git fetch origin
        echo "Checking out branch: ${BRANCH_NAME}"
        git checkout ${BRANCH_NAME}
  # Step 2: Build With Maven
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'mvn'
    args: ['clean', 'install']
    id: 'Build'
  # Step 3: Install latest version of endorctl
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
        echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64)  endorctl" | sha256sum -c
        chmod +x ./endorctl
        ./endorctl --version
    id: 'Install latest version of endorctl'
  # Step 4: SCA Scan With EndorLabs
  - name: 'maven:3.8.6-openjdk-11'
    entrypoint: 'bash'
    args: ["-c", "./endorctl scan -n $$ENDOR_NAMESPACE --api-key=$$ENDOR_API_CREDENTIALS_KEY --api-secret=$$ENDOR_API_CREDENTIALS_SECRET --as-default-branch=true"]
    secretEnv: ['ENDOR_API_CREDENTIALS_KEY', 'ENDOR_API_CREDENTIALS_SECRET']
    env:
      - 'ENDOR_NAMESPACE=demo'
    id: 'SCA Scan With EndorLabs'

# Fetch Endor Labs API Token and Secret From Secrets Manager
availableSecrets:
  secretManager:
  - versionName: projects/{your-project-id}/secrets/endor-api-key/versions/1
    env: 'ENDOR_API_CREDENTIALS_KEY'
  - versionName: projects/{your-project-id}/secrets/endor-api-secret/versions/1
    env: 'ENDOR_API_CREDENTIALS_SECRET'

options:
  # Choose your log configuration
  logging: 'CLOUD_LOGGING_ONLY'
  # Select a private pool if the default runners do not meet the minimum requirements.
  pool:
    name: 'projects/{your-project-id}/locations/{your_location}/workerPools/{your_worker_pool_id}'