1 - Set up keyless authentication
Learn how to implement keyless authentication for CI environments.
At Endor Labs, we believe that the most secure secret is one that doesn’t exist. That’s why in CI/CD environments we recommend using keyless authentication for machine authentication. Keyless Authentication uses OAuth for API authentication and removes the need to maintain and rotate an API key to Endor Labs.
Keyless Authentication is more secure and reduces the cost of secret rotation.
Configure keyless authentication using:
Enabling Keyless Authentication in Google Cloud
To enable Keyless Authentication in GCP you’ll first need permissions to create service accounts and assign these accounts roles to GCP.
The workflow to enable keyless authentication is:
- Create a service account with no permissions for federation.
- If you do not attach a service account to compute resources or use the default service account, we recommend creating a new service account for the compute resources. Create a service account to attach to compute resources and impersonate the federation service account.
- Create an authorization policy to allow the federation service account to authenticate to Endor Labs.
- Provision the compute resources with the appropriate permissions.
- Test Keyless authentication.
Creating GCP Service Accounts and Authorization Policies
To create your service accounts, first export your GCP project name as an environment variable:
export PROJECT=<insert-gcp-project>
Once you’ve set the environment variable you’ll create a service account that will be used to federate access to Endor Labs and will be provided permission to access the Endor Labs APIs:
Step 1: Create a federation service account called endorlabs-federation
:
gcloud iam service-accounts create endorlabs-federation --description="Endor Labs Keyless Federation Service Account" --display-name="Endor Labs Federation Service Account"
We’ll also create a second service account, that will have access to impersonate the endorlabs-federation
account. We’ll call this endorlabs-compute-service
.
This is needed if you don’t already have service accounts for your compute resources. If you do, you need to modify the existing permissions to allow the existing service account to create a federation token.
Step 2: Create a keyless authentication service account to assign to compute resources called endorlabs-compute-service
:
gcloud iam service-accounts create endorlabs-compute-service --description="Endor Labs Service account for keyless authentication" --display-name="Endor Labs Compute Instance SA"
Finally, we’ll assign endorlabs-compute-service
permissions to impersonate the endorlabs-federation
account to authenticate to Endor Labs through OIDC.
Step 3: Assign the serviceAccountOpenIdTokenCreator
role to the endorlabs-compute-service
service account:
gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com" --role="roles/iam.serviceAccountOpenIdTokenCreator"
Once we’ve created the necessary account permissions we will create an authorization policy in Endor Labs to allow the account endorlabs-federation
to your Endor Labs tenant:
Use the following command to create an authorization policy in Endor Labs.
Note: Make sure to replace <your-tenant>
with your Endor Labs tenant name and <insert-your-project>
with your GCP project name in the following command.**
endorctl api create -r AuthorizationPolicy -d '{
"tenant_meta": { "namespace": "<your-tenant>" },
"meta": {
"name": "Keyless Auth",
"kind": "AuthorizationPolicy",
"tags": ["gcp"]
},
"spec": {
"clause": ["email=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com", "gcp"],
"target_namespaces": ["<your-tenant>"],
"propagate": true,
"permissions": {
"rules": {},
"roles": [
"SYSTEM_ROLE_CODE_SCANNER"
]
},
}
}'
You’ve now set up the foundation of keyless authentication. You’ll now need to provision your compute resources with the appropriate GCP scopes and service account.
See Provisioning and Testing Keyless Authentication for GKE workloads for instructions on setting up GKE for keyless authentication.
See Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances for instructions on setting up a virtual machine instance for keyless authentication.
Provisioning and Testing Keyless Authentication for GKE workloads
Prerequisites
The following prerequisites are required to setup keyless authentication on GKE workloads:
Procedure
- (Optional) Create a GKE cluster with workload identity enabled if you do not already have a GKE cluster
- Authenticate to the GKE cluster
- (Optional) Create a namespace for Endor Labs scans
- Create a Kubernetes service account to impersonate your GKE compute service account
- Bind your Kubernetes service account to your GCP compute service account
- Annotate your Kubernetes service account with your GCP service account to complete your binding
- Test a scanning workload using keyless authentication
Setting Up and Testing Keyless authentication in GKE
The following instructions require you to export the following environment variables to appropriately run:
- The GCP Project as PROJECT
- The GKE cluster as CLUSTER_NAME
export PROJECT=<Insert_GCP_Project>
export CLUSTER_NAME=<GKE_CLUSTER>
Optional Step 1: To create a GKE cluster with workload identity enabled if you do not already have a GKE cluster with workload identity enabled run the following command:
gcloud container clusters create keyless-test --workload-pool=endor-github.svc.id.goog --scopes https://www.googleapis.com/auth/cloud-platform
Step 2: To authenticate to your GKE cluster run the following command:
gcloud container clusters get-credentials $CLUSTER_NAME
Optional Step 3: To create a namespace for Endor Labs scans run the following command:
kubectl create namespace endorlabs
Step 4: To create a Kubernetes service account to impersonate your GKE compute service account run the following command:
kubectl create serviceaccount endorlabs-compute-service -n endorlabs
Step 5: To bind your Kubernetes service account to your GCP compute service account run the following command:
Note: Make sure to replace <insert-your-project>
in the following command with your GCP project name.
gcloud iam service-accounts add-iam-policy-binding endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:<insert-your-project>.svc.id.goog[endorlabs/endorlabs-compute-service]"
Step 6: To annotate your Kubernetes service account with your GCP service account to complete your binding run the following command:
Note: If you have created a different service account name replace endorlabs-compute-service with the appropriate service account name.
kubectl annotate serviceaccount endorlabs-compute-service -n endorlabs iam.gke.io/gcp-service-account=endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com
Step 7: Test scan your project with keyless authentication
You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication.
Provisioning and Testing Keyless Authentication for GCP Virtual Machine Instances
The following high-level procedure describes the required steps to use keyless authentication with a GCP virtual machine instance:
Procedure:
- Create a virtual machine instance with the appropriate scopes
- Download and install endorctl on the virtual machine instance
- Launch a test scan with keyless authentication
Setting up and Testing Keyless authentication on a GCP virtual machine instance
The following instructions require you to export the following environment variables to appropriately run:
- The GCP Project as PROJECT
export PROJECT=<Insert_GCP_Project>
To successfully test keyless authentication first you’ll need to provision a compute resource with the service account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com
and the scope https://www.googleapis.com/auth/cloud-platform
:
Step 1: To create a virtual machine instance with the appropriate scopes run the following command:
gcloud compute instances create test-keyless --service-account endorlabs-compute-service@$PROJECT.iam.gserviceaccount.com --scopes https://www.googleapis.com/auth/cloud-platform
Step 2: To download and install endorctl on the virtual machine instance run the following series of commands:
First, SSH to the virtual machine instance you’ve created to test:
gcloud compute ssh --zone "us-west1-b" "test-keyless" --project $PROJECT
Then download and install the latest version of endorctl
. See our documentation for instructions on downloading the latest version
To scan with keyless authentication you must use the flag --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com
for federated access to Endor Labs such as in the below example:
endorctl api list --gcp-service-account=endorlabs-federation@<insert-your-project>.iam.gserviceaccount.com -r Project -n <insert-your-tenant> --count
If this scan runs successfully you’ve tested and scanned a project with keyless authentication to Endor Labs.
Enabling Keyless Authentication in GitHub
To enable Keyless Authentication for GitHub Actions, you’ll need to perform the following steps:
- Ensure you are using the Endor Labs GitHub Action in your GitHub workflow.
- Edit your GitHub Action workflow to add permission settings for the GitHub
id-token
- Create an authorization policy for
GitHub Action OIDC
- Test that you can successfully scan a project using
Github Action OIDC
Add a GitHub Action OIDC authorization policy
To ensure that the GitHub action OIDC identity can successfully login to Endor Labs you’ll need to create an authorization policy in Endor Labs.
To create an authorization policy:
- Under Manage go to Access Control
- Navigate to the “Auth Policy” tab
- Click on the “Add Auth Policy” button
- Select “GitHub Action OIDC” as your identity provider
- Select the permission for the GitHub Action. This permission should be “Code Scanner”
- For the claim use the key
user
and put in a matching value that maps to the organization of your GitHub repository.
Configuring your GitHub Action workflow
To configure your GitHub Action workflow with GitHub Action OIDC you can use the following example as a baseline.
The important items in this workflow are:
- The Usage of the Endor Labs GitHub action.
- Setting Job level permissions to allow writing to the GitHub
id-token
name: Example Scan of OWASP Java
on: workflow_dispatch
jobs:
create_project_owasp:
permissions:
id-token: write # This is required for requesting the JWT
runs-on: ubuntu-latest
steps:
- name: Checkout Repo
uses: actions/checkout@v3
with:
repository: OWASP-Benchmark/BenchmarkJava
- name: Setup Java
uses: actions/setup-java@v3
with:
distribution: 'microsoft'
java-version: '17'
- name: Compile Package
run: mvn clean install
- name: Scan with Endor Labs
uses: endorlabs/github-action@main # This workflow uses the Endor Labs GitHub action to scan.
with:
namespace: 'demo'
scan_summary_output_type: 'json'
pr: false
scan_secrets: true
scan_dependencies: true
Now that you’ve successfully configured your GitHub action workflow file you can use this workflow file or one of your own designs to run a test scan using Keyless authentication for GitHub actions.
Enabling Keyless Authentication in AWS
To enable Keyless Authentication in AWS you’ll first need permissions to create or modify the following roles and an instance profile with the appropriate roles configured.
- An instance access role - The instance access role is assigned to the compute resource, which needs to access Endor Labs. Your instance access role may already exist and you must ensure this role provides the permissions to allow the role to assume the role of a dedicated federation role.
- A dedicated federation role - The dedicated federation role should have no permissions in AWS. Endor Labs will authorize requests that come from this role.
Procedure:
- Create or modify an existing Instance Profile that may be used to assign a role to your EC2 instance.
- Create or modify a role an instance access role, which enables services to assume a dedicated federation role.
- Assign this role to the Instance Profile
- Create a dedicated federation role to provide access to Endor Labs, which will be assumed by the instance access role
- Create an authorization policy in Endor Labs
- Test Keyless Authentication
To configure keyless authentication with EKS, you will need an existing IAM ODIC provider for your cluster and to configure a Kubernetes service account annotated with your instance access role. You won’t need to create or assign roles to instance profiles or create any instance profiles. See the
AWS Documentation for additional information.
Create or select an Instance Profile
An instance profile allows users to attach a single role to an EC2 instance. If you do not already have a pre-defined role or instance profile used by your EC2 instances you should create an instance profile for Endor Labs access.
To create an instance profile using the AWS CLI:
aws iam create-instance-profile --instance-profile-name EndorLabsAccessProfile
Create or modify an instance access role
To successfully authenticate to Endor Labs you will need to assign an instance access role to the instance profile you created above.
The instance access role must at a minimum allow the compute resources that require access to perform the action sts:AssumeRole
. If you already have a role you intend to assign to your instance profile, ensure that it has permissions to allow your compute resources to perform this action.
If you do not have an existing role you intend to use, create the role named endorlabs-instance-access-role using the instructions below.
Add the following json to a file called endorlabs-instance-access-role.json
cat > endorlabs-instance-access-role.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
EOF
Use this file to create your instance access role using the following command:
aws iam create-role --role-name endorlabs-instance-access-role --assume-role-policy-document file://endorlabs-instance-access-role.json
Next, ensure that the instance access role is assigned to the instance profile using the following command:
Your instance profile name and role name will need to be updated based on the names of these resources in your environment.
aws iam add-role-to-instance-profile --instance-profile-name EndorLabsAccessProfile --role-name endorlabs-instance-access-role
Finally, create your EC2 instance and ensure that your instance profile is assigned to it.
Create a dedicated federation role
A dedicated federation role is leveraged to provide a least privileged role that enables access to Endor Labs. This role is designed to be assumed only by specific other roles and does not provide access to AWS resources.
To create your federation role you will need the name and AWS account number of the instance access role, which should look similar to the following policy json:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::$ACCOUNT:role/$ROLE_NAME"
},
"Action": "sts:AssumeRole"
}
]
}
First, get the account number of the role:
export ACCOUNT=$(aws sts get-caller-identity | jq -r '.Account')
Then define the name of the instance access role. For the following example, we will assume it is endorlabs-instance-access-role.
export ROLE_NAME=endorlabs-instance-access-role
Next, create the IAM policy document.
cat > endorlabs-federation-aws-role.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::${ACCOUNT}:role/${ROLE_NAME}"
},
"Action": "sts:AssumeRole"
}
]
}
EOF
Next, apply the policy document as a role:
aws iam create-role --role-name endorlabs-federation --assume-role-policy-document file://endorlabs-federation-aws-role.json
Finally, fetch the ARN of the IAM role you’ve created using the following command and create an authorization policy for it in Endor Labs.
To fetch the ARN of the Endor Labs federation role use the following command:
aws iam list-roles | jq -r '.Roles[] | select(.Arn|contains("endorlabs-federation"))'.Arn
To add the authorization policy to Endor Labs:
- Login to Endor Labs as an administrator.
- Under Manage, navigate to Access Control > Auth Policy
- Click Add Auth Policy
- Under Identity Provider Select AWS role
- Provide the appropriate permissions for your authorization policy.
- Under claims use the Key User and the value of the ARN that you fetched in the previous command.
- Click Save Auth Policy to finalize your keyless authentication setup.
Testing Keyless Authentication with AWS
On the EC2 instance you’ve configured for keyless authentication, download and install the latest version of endorctl
. See our documentation for instructions on downloading the latest version
To scan with keyless authentication you must use the flag --aws-role-arn=<insert-your-arn>
for federated access to Endor Labs such as in the below example:
endorctl --aws-role-arn=<insert-your-arn> api list -r Project -n <insert-your-endorlabs-tenant> --page-size=1
You’ve set up and configured keyless authentication. Now you can run a test scan to ensure you can successfully scan projects using keyless authentication with AWS.
2 - Scanning with GitHub Actions
Learn how to implement Endor Labs in GitHub action workflows.
GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.
You can use GitHub Actions to include Endor Labs into your CI pipeline seamlessly.
Using this pipeline, developers can view and detect:
- Policy violations in the source code
- Secrets inadvertently included in the source code
The Endor Labs verifications are conducted as automated checks and help you discover violations before pushing code to the repository. Information about the violations can be included as comments on the corresponding pull request (PR). This enables developers to easily identify issues and take remedial measures early in the development life cycle.
- For policy violations, the workflow is designed to either emit a warning or return an error based on your action policy configurations.
- For secrets discovered in the commits, developers can view the PR comments and take necessary remedial measures.
To start using Endor Labs with GitHub:
Install Software Prerequisites
To ensure the successful execution of the Endor Labs GitHub action, the following prerequisites must be met:
- The GitHub action must be able to authenticate with the Endor Labs API.
- You must have the value of the Endor Labs namespace handy for authentication.
- You must have access to the Endor Labs API.
- If you use keyless authentication, you must set an authorization policy in Endor Labs. See Authorization policies for details.
Example GitHub Action Workflow
Endor Labs scanning workflow using GitHub actions that accomplishes the following tasks in your CI environment:
- Tests PRs to the default branch and monitors the most recent push to the default branch.
- Builds a Java project and sets up the Java build tools. If your project is not on Java, then configure this workflow with your project-specific steps and build tools.
- Authenticates to Endor Labs with GitHub Actions keyless authentication.
- Scan with Endor Labs.
- Comments on PRs if any policy violations occur.
- Generates findings and uploads results to GitHub in SARIF format.
The following example workflow shows how to scan with Endor Labs for a Java application using the recommended keyless authentication for GitHub actions:
name: Endor Labs Dependency and Secrets Scan
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
scan:
permissions:
security-events: write # Used to upload Sarif artifact to GitHub
contents: read # Used to check out a private repository
actions: read # Required for private repositories to upload Sarif files. GitHub Advanced Security licenses are required.
id-token: write # Used for keyless authentication with Endor Labs
pull-requests: write # Required to automatically comment on PRs for new policy violations
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Setup Java
uses: actions/setup-java@v3
with:
distribution: 'microsoft'
java-version: '17'
- name: Build Package
run: mvn clean install
- name: Endor Labs Scan Pull Request
if: github.event_name == 'pull_request'
uses: endorlabs/github-action@v1.1.4
with:
namespace: 'example' # Replace with your Endor Labs tenant namespace
scan_dependencies: true
scan_secrets: true
pr: true
enable_pr_comments: true # Required to automatically comment on PRs for new policy violations
github_token: ${{ secrets.GITHUB_TOKEN }} # Required for PR comments on new policy violations
scan-main:
permissions:
id-token: write
repository-projects: read
pull-requests: read
contents: read
name: endorctl-scan
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v3
- name: Setup Java
uses: actions/setup-java@v3
with:
distribution: 'microsoft'
java-version: '17'
- name: Build Package
run: mvn clean install
- name: 'Endor Labs Scan Push to main'
if: ${{ github.event_name == 'push' }}
uses: endorlabs/github-action@v1.1.4
with:
namespace: 'example' # Replace with your Endor Labs tenant namespace
scan_dependencies: true
scan_secrets: true
pr: false
scan_summary_output_type: 'table'
sarif_file: 'findings.sarif'
- name: Upload findings to github
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'findings.sarif'
Authenticate with Endor Labs
Endor Labs recommends using keyless authentication in CI environments. Keyless authentication is more secure and reduces the cost of secret rotation. To set up keyless authentication see Keyless Authentication.
If you choose not to use keyless authentication, you can configure an API key and secret in GitHub for authentication as outlined in Managing API keys.
Authentication Without Keyless Authentication for GitHub
If you are not using keyless authentication for GitHub Actions, you must not provide id-token: write
permissions to your GitHub token unless specifically required by a step in this job. You must also set enable_github_action_token: false
in your Endor Labs GitHub action configuration.
The following example configuration uses the Endor Labs API key for authentication:
- name: Scan with Endor Labs
uses: endorlabs/github-action@v1.1.4
with:
namespace: 'example'
api_key: ${{ secrets.ENDOR_API_CREDENTIALS_KEY }}
api_secret: ${{ secrets.ENDOR_API_CREDENTIALS_SECRET }}
enable_github_action_token: false
The following example configuration uses a GCP service account for keyless authentication to Endor Labs:
- name: Scan with Endor Labs
uses: endorlabs/github-action@v1.1.4
with:
namespace: 'example'
gcp_service_account: '<Insert_Your_Service_Account>@<Insert_Your_Project>.iam.gserviceaccount.com'
enable_github_action_token: false
Endor Labs GitHub Action Configuration Parameters
The following input configuration parameters are supported for the Endor Labs GitHub Action:
Common parameters
The following input global parameters are supported for the Endor Labs GitHub action:
Flags |
Description |
api_key |
Set the API key used to authenticate with Endor Labs. |
api_secret |
Set the secret corresponding to the API key used to authenticate with Endor Labs. |
enable_github_action_token |
Set to false if you prefer to use another form of authentication over GitHub action OIDC tokens. (Default: true ) |
endorctl_checksum |
Set to the checksum associated with a pinned version of endorctl. |
endorctl_version |
Set to a version of endorctl to pin this specific version for use. Defaults to the latest version. |
log_level |
Set the log level. (Default: info ) |
log_verbose |
Set to true to enable verbose logging. (Default: false ) |
namespace |
Set to the namespace of the project that you are working with. (Required) |
gcp_service_account |
Set the target service account for GCP based authentication. GCP authentication is only enabled if this flag is set. Cannot be used with api_key . |
Scanning parameters
The following input parameters are also supported for the Endor Labs GitHub action when used for scanning:
Flags |
Description |
additional_args |
Use additional_args with endorctl scan for advanced scenarios. However, no example use case currently exists as standard options suffice for typical needs. |
enable_pr_comments |
Set to true to publish new findings as review comments. Requires pr and github_token . Additionally, the pull-requests: write permissions must be set in the workflow. (Default: false ) |
export_scan_result_artifact |
Set to false to disable the json scan result artifact export. (Default: true ) |
github_token |
Set the token used to authenticate with GitHub. Mandatory if enable_pr_comments is set to true |
phantom_dependencies |
Set to true to enable phantom dependency analysis. (Default: false ) |
output_file |
Set a file to save the scan results. Use this instead of export_scan_result_artifact to save any scan results data to a file in the workspace for processing by others steps in the same job, instead of the workflow run log. |
pr |
Set to false to track this scan as a monitored version within Endor Labs, as opposed to a point-in-time policy and finding test for a PR. (Default: true ) |
pr_baseline |
Set to the Git reference that you are merging to, such as the default branch. Enables endorctl to compare findings, so developers are only alerted to issues in the current changeset. Example: pr_baseline: "main" . Note: Not needed if enable_pr_comments is set to true . |
run_stats |
Set to false to disable reporting of CPU/RAM/time scan statistics via time -v (sometimes required on Windows runners). (Default: true ) |
sarif_file |
Set to a path on your GitHub runner to store the analysis results in SARIF format. |
scan_dependencies |
Scan Git commits and generate findings for all dependencies. (Default: true ) |
scan_git_logs |
Perform a more complete and detailed scan of secrets in the repository history. Must be used together with scan_secrets . (Default: false ) |
scan_github_actions |
Scan source code repository for GitHub actions used in workflow files to analyze vulnerabilities and malware. (Default: false ) |
scan_tools |
Scan source code repository for CI/CD tools. (Default: false ) |
scan_package |
Scan a specified artifact or a package. Set the path to an artifact with scan_path . (Default: false ) |
scan_container |
Scan a specified container image. The image must be set with image and a project can be defined with project_name . (Default: false ) |
project_name |
Specify a project name for a container image scan or a package scan. |
image |
Specify a container image to scan. |
scan_path |
Set the path of the directory to scan. (Default: . ) |
scan_secrets |
Scan the source code repository and generate findings for secrets. See also scan_git_logs . (Default: false ) |
scan_summary_output_type |
Set the desired output format to table , json , yaml , or summary . (Default: json ) |
tags |
Specify a list of user-defined tags to add to this scan. You can use tags to search and filter scans later. |
use-bazel |
Enable the usage of Bazel for the scan. (Default: false ) |
bazel_exclude_targets |
Specify a list of Bazel targets to exclude from the scan. |
bazel_include_targets |
Specify a list of Bazel targets to scan. If bazel_targets_include is not set, the bazel_targets_query value is used to determine which Bazel targets to scan. |
bazel_targets_query |
Specify a Bazel query to determine which Bazel targets to scan. Ignored if bazel_targets_include is set. |
Configure an action policy in the Endor Labs user interface to perform an action when a rule is triggered.
See Action Policies for details on action policies.
- Set the Policy Template to Detected Secrets and select the Template Parameters as desired.
- Choose Enforce Policy and
- Select Warn as the recommended action.
- Select Break the Build to fail the build CI pipeline.
You can configure GitHub Actions to comment on PRs if there are any policy violations. When you raise a PR, Endor Labs scans and detects policy violations. A comment is added in the PR with the details of violation. The CI pipeline warns you or fails the build based on your action policy configuration. The PR comments also include recommendations to help you take necessary remedial actions.
You can customize the template of the comment according to your requirement.
Make sure that your GitHub action workflow includes the following configuration.
- The workflow must have a
with
clause including: enable_pr_comments
to true
to publish new findings as review comments and github_token: ${{ secrets.GITHUB_TOKEN }}
. This token is automatically provisioned by GitHub when using GitHub actions. See GitHub configuration parameters for more information.
- To grant Endor Labs the ability to comment on PRs you must include the permission
pull-requests: write
.
The following example configuration comments on PRs if a policy violation is detected.
- name: Endor Labs Scan PR to Default Branch
if: github.event_name == 'pull_request'
uses: endorlabs/github-action@v1.1.4
with:
namespace: 'example' # Update with your Endor Labs namespace
scan_summary_output_type: 'table'
scan_dependencies: true
scan_secrets: true
pr: true
enable_pr_comments: true
github_token: ${{ secrets.GITHUB_TOKEN }}
You can view a live example of a GitHub repository with a workflow that enables PR comments.
The main.yaml file contains the following configuration to enable PR comments.
name: Build Release
on:
pull_request:
branches: [main]
workflow_dispatch:
push:
branches: [main]
schedule:
- cron: "23 23 * * 0"
jobs:
build:
permissions:
pull-requests: write
security-events: write
contents: read
id-token: write
actions: read
runs-on: ubuntu-latest
env:
ENDOR_NAMESPACE: "endorlabs-hearts-github"
steps:
- name: Endor Labs Scan PR to Default Branch
if: github.event_name == 'pull_request'
uses: endorlabs/github-action@v1
with:
namespace: ${{ env.ENDOR_NAMESPACE }}
pr: true
enable_pr_comments: true
github_token: ${{ secrets.GITHUB_TOKEN }}
The PR #10 introduced a reachable vulnerability. Since the workflow has enable_pr_comments
set as true
, a comment is added to the PR on the policy violation.
![PR Comment](../../../images/prcomment.png)
You can expand the comment to view more details about the violation and take steps to resolve the issue.
![PR Comment Details](../../../images/prcommentdetail.png)
Endor Labs provides a default template with standard information that will be included in your pull requests as comments. You can use the default template, or you can choose to edit and customize this template to fit your organization’s specific requirements. You can also create custom templates using Go Templates.
- Sign in to Endor Labs and navigate to Manage>Integrations
- Look for GitHub PR comments under Notifications.
- Click Edit Template.
- Make the required changes and click Save Template.
- Click Restore to Default to revert the changes.
- Use the download icon in the top right corner to download this template.
- Use the copy icon to copy the information in the template.
Data model
To create custom templates for PR comments, you must understand the data supplied to the template.
See the following protobuf specification for the GithubCommentData
message that this template uses.
syntax = "proto3";
package internal.endor.ai.endor.v1;
import "google/protobuf/wrappers.proto";
import "spec/internal/endor/v1/finding.proto";
import "spec/internal/endor/v1/package_version.proto";
option go_package = "github.com/endorlabs/monorepo/src/golang/spec/internal.endor.ai/endor/v1";
option java_package = "ai.endor.internal.spec";
// The list of finding UUIDs.
message FindingUuids {
repeated string uuids = 1;
}
// The map of dependency name to findings.
message DependencyToFindings {
map<string, FindingUuids> dependency_to_findings = 1;
}
// The map of PackageVersion UUID to DependencyToFindings.
message PackageToDependencies {
map<string, DependencyToFindings> package_to_dependencies = 1;
}
message GithubCommentData {
// The header of the PR comment. Identifies the PR comment published by Endor Labs.
// It should always be at top of the template.
google.protobuf.StringValue comment_header = 1;
// The footer of the PR comment.
google.protobuf.StringValue comment_footer = 2;
// The map of finding UUID to finding object.
map<string, internal.endor.ai.endor.v1.Finding> findings_map = 3;
// The map of policy UUID to policy name.
// This will contain only the policies that are triggered or violated.
map<string, string> policies_map = 4;
// The map of policy UUID to the list of finding UUIDs.
map<string, FindingUuids> policy_findings_map = 5;
// The map of PackageVersion UUID to PackageVersion object.
map<string, internal.endor.ai.endor.v1.PackageVersion> package_versions_map = 6;
// The data needs to be grouped as follows:
//
// - Policy 1
// - Package 1
// - Dependency Package 1
// - Finding 1
// - Finding 2
// - Dependency Package 2
// - Finding 3
// - Finding 4
// - Package 2
// - Dependency Package 1
// - Finding 1
// - Finding 5
// - Policy 2
// ....
//
// Map 0[PolicyUUID]/Map 1[PkgVerUUID]/Map 2 [Dep Names]/Finding UUID
map<string, PackageToDependencies> data_map = 7;
google.protobuf.StringValue api_endpoint = 8;
}
To understand Finding and PackageVersion definitions that are used in this protobuf specification, see:
See the following specification to understand the additional functions that are also available. You can access these functions by using their corresponding keys.
// FuncMap contains the additional functions that are available to GithubCommentTemplate.
var FuncMap = template.FuncMap{
"now": toTime, // 'now' gives the current time
// 'enumToString' coverts the enums for finding level, finding category and finding tags to string
"enumToString": enumToString,
// 'getPackageVersionURL' returns the URL for a given PackageVersion
"getPackageVersionURL": func(apiURL string, packageVersion *endorpb.PackageVersion) string {
result, err := common.GetPackageVersionURL(apiURL, packageVersion)
if err != nil {
return ""
}
return result
},
// 'getFindingURL' returns the URL for a given Finding
"getFindingURL": func(apiURL string, finding *endorpb.Finding) string {
result, err := common.GetFindingURL(apiURL, finding)
if err != nil {
return ""
}
return result
},
// 'add' returns the sum of two integers
"add": func(n int, incr int) int {
return n + incr
},
// 'getOtherFindingsPackageMarker' returns the key for _findingsWithNoPackages for lookup in DataMap
// Not all findings are associated with a PackageVersion, such findings are grouped under this key
// in the DataMap
"getOtherFindingsPackageMarker": func() string { return _findingsWithNoPackages },
// 'getOtherFindingsDependencyMarker' returns the key for _findingsWithNoDeps for lookup in DataMap
// Not all findings are associated with a dependency, such findings are grouped under this key
// in the DataMap
"getOtherFindingsDependencyMarker": func() string { return _findingsWithNoDeps },
// 'getFindingsCountString' returns a string with number of findings, example - "5 findings"
"getFindingsCountString": func(dataMap *endorpb.PackageToDependencies) string {
count := 0
for _, depMap := range dataMap.PackageToDependencies {
for _, findingMap := range depMap.DependencyToFindings {
count += len(findingMap.Uuids)
}
}
findingsStr := "findings"
if count == 1 {
findingsStr = "finding"
}
return fmt.Sprintf("%d %s", count, findingsStr)
},
}
3 - Scanning in Bitbucket Pipelines
Learn how to implement Endor Labs in a Bitbucket pipeline.
Bitbucket Pipelines is a continuous integration and continuous delivery (CI/CD) service built into Bitbucket. It allows developers to automatically build, test, and deploy their code based on a configuration file bitbucket-pipelines.yml
defined in the root of their repository.
To integrate Endor Labs into a Bitbucket pipeline:
- Authenticate to Endor Labs
- Install your build toolchain
- Build your code
- Scan with Endor Labs
Authenticate to Endor Labs
Configure an API key and secret in the bitbucket-pipelines.yml
file for authentication. See managing API keys for more information on generating an API key for Endor Labs.
To create a Bitbucket pipeline reference the following steps:
- Create a
bitbucket-pipelines.yml
file in your repository if you do not already have one.
- In your
bitbucket-pipelines.yml
file customize the job configuration based on your project’s requirements using the following example.
- Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project.
- Update your Endor Labs tenant namespace to the appropriate namespace for your project.
- Update your default branch from main if you do not use main as the default branch name.
- Modify any dependency or artifact caches to align with the languages and caches used by your project.
Example
Use the following example to get started. Make sure to customize this job with your specific build environment and build steps.
Bitbucket configuration
simage: maven:3.6.3-jdk-11
pipelines:
branches:
main:
- step:
name: "Build and Test"
script:
- mvn install -DskipTests
- echo "Running Endor Labs Scan"
- curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
- echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c
- chmod +x ./endorctl
- ./endorctl scan -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
pull-requests:
'**':
- step:
name: "Build and Test on PR to Main"
script:
- mvn install -DskipTests
- echo "Running Endor Labs PR Scan"
- curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
- echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c
- chmod +x ./endorctl
- ./endorctl scan --pr --pr-baseline=main --languages=java --output-type=json -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET | tee output.json
#Optional - Comment on the PR
# - apt-get update
# - apt-get install -y python3 python3-pip
# - pip3 install -r requirements.txt
# - python3 add-bitbucket-pr-comments.py output.json
Once you’ve set up Endor Labs, you can test your CI implementation to ensure it is successful and then proceed with your scans.
You can also use the Insights feature in Bitbucket Pipelines to indicate if the changes in your pull requests violated any policies set in Endor Labs.
import json
import os
import sys
import requests
import argparse
# Check for required environment variables
BITBUCKET_REPO_OWNER = os.getenv('BITBUCKET_REPO_OWNER')
BITBUCKET_REPO_SLUG = os.getenv('BITBUCKET_REPO_SLUG')
BITBUCKET_COMMIT = os.getenv('BITBUCKET_COMMIT')
if not all([BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT]):
sys.exit("Error: One or more required environment variables (BITBUCKET_REPO_OWNER, BITBUCKET_REPO_SLUG, BITBUCKET_COMMIT) are not set.")
#This is an internal proxy running in the BitBucket environment to accept Code Insights
proxies = {"http": "http://localhost:29418"}
def load_json_with_unescaped_characters(file_path):
"""Load and return JSON data from a file, replacing unescaped characters if necessary."""
try:
with open(file_path, 'r', encoding='utf-8') as file:
json_str = file.read().strip()
return json.loads(json_str)
except json.decoder.JSONDecodeError as e:
print(f"Failed to parse JSON: {e}")
return None
except FileNotFoundError:
print(f"File not found: {file_path}")
sys.exit()
def construct_report_payload(endor_findings):
"""Construct and return the payload for creating a Bitbucket report."""
warning_findings_count = len(endor_findings.get('warning_findings', []))
blocking_findings_count = len(endor_findings.get('blocking_findings', []))
total_violations = warning_findings_count + blocking_findings_count
result = "PASSED" if total_violations == 0 else "FAILED"
report_payload = {
"title": "Endor Labs Policy Violations",
"details": f"Endor Labs detected {total_violations} policy violations associated with this pull request.\n\n{endor_findings['warnings'][0]}",
"report_type": "SECURITY",
"reporter": "Endor Labs",
"link": f"https://app.endorlabs.com/t/{namespace}/projects/{project_uuid}/pr-runs/{report_id}",
"logo_url": "https://avatars.githubusercontent.com/u/92199924",
"result": result,
"data": [
{"title": "Warning Findings", "type": "NUMBER", "value": warning_findings_count},
{"title": "Blocking Findings", "type": "NUMBER", "value": blocking_findings_count}
]
}
return report_payload
def construct_annotation_payload(finding):
"""Construct and return the payload for creating an annotation in Bitbucket."""
title = "Endor Labs Policy Violation"
summary = finding['meta']['description']
details = f"{finding['spec']['summary']}\n\n{finding['spec']['remediation']}"
severity = "CRITICAL" if finding['spec']['level'] == "FINDING_LEVEL_CRITICAL" else \
"HIGH" if finding['spec']['level'] == "FINDING_LEVEL_HIGH" else \
"MEDIUM" if finding['spec']['level'] == "FINDING_LEVEL_MEDIUM" else "LOW"
affected_paths = finding['spec'].get('dependency_file_paths', [])
path = affected_paths[0] if affected_paths else "Unknown file"
annotation_payload = {
"external_id": finding['uuid'],
"title": title,
"annotation_type": "VULNERABILITY",
"summary": summary,
"details": details,
"severity": severity,
"path": path
}
return annotation_payload
def send_report(report_payload):
"""Send the constructed report payload to the Bitbucket API."""
report_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports/{report_id}"
response = requests.put(report_url, json=report_payload, proxies=proxies)
if response.status_code in [200, 201]:
print("Report created or updated successfully")
else:
print(f"Failed to create or update report: {response.text}")
def send_annotation(annotation_payload):
"""Send the constructed annotation payload to the Bitbucket API."""
annotation_url = f"{base_url}/{report_id}/annotations/{annotation_payload['external_id']}"
response = requests.put(annotation_url, json=annotation_payload, proxies=proxies)
if response.status_code in [200, 201]:
print("Annotation added successfully")
else:
print(f"Failed to add annotation: {response.text}")
def process_findings(filename):
"""Load findings from JSON, create a report, and add annotations for each finding."""
endor_findings = load_json_with_unescaped_characters(filename)
if endor_findings is None:
print("Failed to load findings. Exiting.")
return
global report_id, project_uuid, namespace
# Define the order of keys to check
finding_types = ['all_findings', 'warning_findings', 'blocking_findings']
# Iterate over finding types and extract the first one found
for finding_type in finding_types:
if endor_findings.get(finding_type):
first_finding = endor_findings[finding_type][0]
report_id = first_finding['context']['id']
project_uuid = first_finding['spec']['project_uuid']
namespace = first_finding['tenant_meta']['namespace']
break # Stop after finding the first non-empty list
if not report_id:
print("No findings found.")
sys.exit()
# Prepare the base URL for Bitbucket API requests
global base_url
base_url = f"http://api.bitbucket.org/2.0/repositories/{BITBUCKET_REPO_OWNER}/{BITBUCKET_REPO_SLUG}/commit/{BITBUCKET_COMMIT}/reports"
# Create the report
report_payload = construct_report_payload(endor_findings)
send_report(report_payload)
# Iterate over findings and create annotations
for finding in endor_findings.get('blocking_findings', []) + endor_findings.get('warning_findings', []):
annotation_payload = construct_annotation_payload(finding)
send_annotation(annotation_payload)
def main():
"""Main function to parse arguments and process findings."""
parser = argparse.ArgumentParser(description="Script to process findings and update Bitbucket via API.")
parser.add_argument("filename", help="Filename containing the JSON findings.")
args = parser.parse_args()
process_findings(args.filename)
if __name__ == "__main__":
main()
5 - Scanning with CircleCI
Learn how to implement Endor Labs in a CircleCI pipeline.
CircleCI CI/CD pipelines allow you to configure your pipeline as code. Your entire CI/CD process is orchestrated through a single file called config.yml
. The config.yml
file is located in a folder called .circleci
at the root of your project which defines the entire pipeline.
To integrate Endor Labs into your CircleCI CI/CD processes:
- Authenticate to Endor Labs
- Install your build toolchain
- Build your code
- Scan with Endor Labs
Authenticate to Endor Labs
Endor Labs recommends using keyless authentication in continuous integration environments. Keyless Authentication is more secure and reduces the cost of secret rotation but is only available on self-hosted runners in CircleCI.
To configure keyless authentication see the keyless authentication documentation
If you choose not to use keyless authentication you can configure an API key and secret in CircleCI for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.
- In your CircleCI environment, navigate to Organizational Settings.
- From Contexts and select Create Context.
- Enter a context name for reference such as endorlabs or re-use an existing context.
- Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
- In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
- Select Add Environment Variable.
- Repeat the previous 3 steps to add your API key secret as the environment variable ENDOR_API_CREDENTIALS_SECRET. Have the name of the context handy to reference in the workflows later.
To create a CircleCI pipeline reference the following steps:
- Create a
.cirlceci/config.yml
file in your repository if you do not already have one.
- In your
config.yml
file customize the job configuration based on your project’s requirements using one of the examples, simple CircleCI configuration or advanced CircleCI configuration.
- Create two workflows called
build_and_watch_endorlabs
and build_and_test_endorlabs
.
- Ensure that the context you created is part of the workflow if you are not using keyless authentication.
- Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
- Update your Endor Labs tenant namespace to the appropriate namespace for your project.
- Update your default branch from main if you do not use main as the default branch name.
- Modify any dependency or artifact caches to align with the languages and caches used by your project.
Examples
Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.
Simple CircleCI configuration
version: 2.1
jobs:
test-endorlabs-scan:
docker:
- image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
environment:
ENDORCTL_VERSION: "latest"
ENDOR_NAMESPACE: "example"
steps:
- checkout
- run:
name: "Build"
command: |
mvn clean install -Dskiptests
- run:
name: "Install endorctl"
command: |
curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c;
if [ $? -ne 0 ]; then
echo "Integrity check failed";
exit 1;
fi
chmod +x ./endorctl
./endorctl --version
- run:
name: "Endor Labs Test"
command: |
./endorctl scan --pr --pr-baseline=main --dependencies --secrets
watch-endorlabs-scan:
docker:
- image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
environment:
ENDOR_NAMESPACE: "example" # Replace with your Endor Labs namespace
steps:
- checkout
- run:
name: "Build"
command: |
mvn clean install -Dskiptests
- run:
name: "Install endorctl"
command: |
curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c;
if [ $? -ne 0 ]; then
echo "Integrity check failed";
exit 1;
fi
chmod +x ./endorctl
./endorctl --version
- run:
name: "Endor Labs Watch"
command: |
./endorctl scan --dependencies --secrets
workflows:
build_and_endorlabs_watch:
when:
equal: [ main, << pipeline.git.branch >> ]
jobs:
- watch-endorlabs-scan:
context:
- endorlabs
build_and_endorlabs_test:
jobs:
- test-endorlabs-scan:
context:
- endorlabs
Advanced CircleCI configuration
The following example is an advanced implementation of Endor Labs in circleCI which includes several optional performance optimizations and job maintainability updates.
This includes:
- Caching and restoring caches of jobs and artifacts to improve performance. Caches should be modified to reflect the build artifacts and dependencies of your project.
- Segmenting jobs and scans.
# You can copy and paste portions of this `config.yml` file as an easy reference.
#
version: 2.1
jobs:
build:
docker:
- image: maven:3.6.3-jdk-11 # Modify this image as needed for your build steps
steps:
- checkout
- restore_cache:
keys:
# when lock file changes, use increasingly general patterns to restore cache
- maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
- maven-repo-v1-{{ .Branch }}-
- maven-repo-v1-
- run:
name: "Build Your Project"
command: |
mvn clean install
- persist_to_workspace:
root: .
paths:
- target/ # Persist artifact across job. Change this if you are creating your artifact in a location outside of the target directory.
- save_cache:
paths:
- ~/.m2/repository
key: maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
test-endorlabs-scan:
docker:
- image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
environment:
ENDORCTL_VERSION: "latest"
ENDOR_NAMESPACE: "example"
steps:
- checkout
- attach_workspace:
at: .
- restore_cache:
keys:
# when lock file changes, use increasingly general patterns to restore cache
- maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
- maven-repo-v1-{{ .Branch }}-
- maven-repo-v1-
- run:
name: "Install endorctl"
command: |
curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c;
if [ $? -ne 0 ]; then
echo "Integrity check failed";
exit 1;
fi
chmod +x ./endorctl
./endorctl --version
- run:
name: "Endor Labs Test"
command: |
./endorctl scan --pr --pr-baseline=main --dependencies --secrets
watch-endorlabs-scan:
docker:
- image: maven:3.6.3-jdk-11 # Modify this image as needed for your build tools
environment:
ENDORCTL_VERSION: "latest"
ENDOR_NAMESPACE: "example" #Replace with your namespace in Endor Labs
steps:
- checkout
- attach_workspace:
at: .
- restore_cache:
keys:
# when lock file changes, use increasingly general patterns to restore cache
- maven-repo-v1-{{ .Branch }}-{{ checksum "pom.xml" }}
- maven-repo-v1-{{ .Branch }}-
- maven-repo-v1-
- run:
name: "Install endorctl"
command: |
curl https://api.endorlabs.com/download/latest/endorctl_linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c;
if [ $? -ne 0 ]; then
echo "Integrity check failed";
exit 1;
fi
chmod +x ./endorctl
./endorctl --version
- run:
name: "Endor Labs Watch"
command: |
./endorctl scan --dependencies --secrets
workflows:
build_and_endorlabs_watch:
when:
equal: [ main, << pipeline.git.branch >> ]
jobs:
- build
- watch-endorlabs-scan:
requires:
- build
context:
- endorlabs
build_and_endorlabs_test:
jobs:
- build
- test-endorlabs-scan:
requires:
- build
context:
- endorlabs
Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.
7 - Scanning with Jenkins
Learn how to implement Endor Labs in a Jenkins pipeline.
Jenkins is an open-source automation server widely used for building, testing, and deploying software. Specifically in the context of CI/CD pipelines, Jenkins serves as a powerful tool to automate various stages of the software development lifecycle.
To integrate Endor Labs into your Jenkins CI/CD processes:
- Authenticate to Endor Labs
- Install nodeJS plugin in Jenkins.
- Install your build toolchain
- Build your code
- Scan with Endor Labs
Authenticate to Endor Labs
To configure keyless authentication see the keyless authentication documentation
If you choose not to use keyless authentication you can configure an API key and secret in Jenkins for authentication using the following steps. See managing API keys for more information on generating an API key for Endor Labs.
- In your Jenkins environment, navigate to Manage Jenkins.
- Enter a credential name for reference such as endorlabs or re-use an existing context.
- Click into your new or existing context. Add any project restrictions and select Add Environment Variable.
- In Environment Variable Name, enter ENDOR_API_CREDENTIALS_KEY and in Value, enter the Endor Labs API Key.
- Select Add Environment Variable.
Install nodeJS plugin in Jenkins
See Jenkins documentation to install nodeJS plugin in Jenkins. You must have the nodeJS plugin to use npm and download endorctl.
To create a Jenkins pipeline:
- Create a configuration pipeline file in your repository if you do not already have one using the pipeline project.
- In your configuration pipeline file customize the job configuration based on your project’s requirements using one of the examples, simple Jenkins configuration or Jenkins pipeline using curl.
- Ensure that the context you created is part of the workflow if you are not using keyless authentication.
- Adjust the image field to conform to the required build tools for constructing your software packages, and synchronize your build steps with those of your project.
- Update your Endor Labs tenant namespace to the appropriate namespace for your project.
- Update your default branch from main if you do not use main as the default branch name.
- Modify any dependency or artifact caches to align with the languages and caches used by your project.
Examples
Use the following examples to get started. Make sure to customize this job with your specific build environment and build steps.
Simple Jenkins configuration using npm
pipeline {
agent any
tools {nodejs "NodeJS"}
environment {
ENDOR_API = credentials('ENDOR_API')
ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY_1')
ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET_1')
}
stages {
stage('Checkout') {
steps {
// Checkout the Git repository
checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
}
}
stage('Build') {
steps {
// Perform any build steps if required
sh 'mvn clean install'
}
}
stage('endorctl Scan') {
steps {
script {
// Define the Node.js installation name configured in Jenkins
NODEJS_HOME = tool name: 'NodeJS', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
PATH = "$NODEJS_HOME/bin:${env.PATH}"
}
// Download and install endorctl.
sh 'npm install -g endorctl'
// Check endorctl version and installation.
sh 'endorctl --version'
// Run the scan.
sh('endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET')
}
}
stage('Results') {
steps {
// Publish or process the vulnerability scan results
// Publish reports, fail the build on vulnerabilities, etc.
echo 'Publish results'
}
}
}
}
Jenkins pipe line for curl to download endorctl binary
The following example includes curl to download the endorctl binary.
pipeline {
agent any
// Endorctl scan uses following environment variables to the trigger endorctl scan
environment {
ENDOR_API = credentials('ENDOR_API')
ENDOR_NAMESPACE = credentials('ENDOR_NAMESPACE')
ENDOR_API_CREDENTIALS_KEY = credentials('ENDOR_API_CREDENTIALS_KEY')
ENDOR_API_CREDENTIALS_SECRET = credentials('ENDOR_API_CREDENTIALS_SECRET')
}
stages {
// Not required if repository is allready cloned to trigger a endorctl scan
stage('Checkout') {
steps {
// Checkout the Git repository
checkout scmGit(branches: [[name: '*/main']], userRemoteConfigs: [[url: 'https://github.com/endorlabstest/app-java-demo.git']])
}
}
stage('Build') {
// Not required if project is already built
steps {
// Perform any build steps if required
sh 'mvn clean install'
}
}
stage('endorctl Scan') {
steps {
// Download and install endorctl.
sh '''#!/bin/bash
echo "Downloading latest version of endorctl"
VERSION=$(curl $ENDOR_API/meta/version | jq -r '.Service.Version')
ENDORCTL_SHA=$(curl $ENDOR_API/meta/version | jq -r '.ClientChecksums.ARCH_TYPE_LINUX_AMD64')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
echo "$ENDORCTL_SHA endorctl" | sha256sum -c
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
chmod +x ./endorctl
// Check endorctl version and installation.
./endorctl --version
// Run the scan.
./endorctl scan -a $ENDOR_API -n $ENDOR_NAMESPACE --api-key $ENDOR_API_CREDENTIALS_KEY --api-secret $ENDOR_API_CREDENTIALS_SECRET
'''
}
}
}
}
Once you’ve set up Endor Labs you can test your CI implementation is successful and begin scanning.
8 - Scanning in Azure Pipelines
Learn how to implement Endor Labs in an Azure Pipeline.
Azure Pipelines is a continuous integration and continuous delivery (CI/CD) service available in Azure DevOps ecosystem. It facilitates continuous integration, continuous testing, and continuous deployment for seamless building, testing, and delivery of software.
To integrate Endor Labs into an Azure pipeline:
Complete the Prerequisites
Ensure that you complete the following prerequisites before you proceed.
Set up an Endor Labs tenant
You must have an Endor Labs tenant set up for your organization. You can also set up namespaces according to your requirements. See Set up namespaces
Configure an API key and secret for authentication. See managing API keys for more information on generating an API key for Endor Labs. Store API key and secret as environment variables, ENDOR_API_CREDENTIALS_KEY
and ENDOR_API_CREDENTIALS_SECRET
.
Enable Advanced Security in Azure
You need to enable Advanced Security in your Azure repository to view results in Azure.
- Log in to Azure and open Project Settings.
- Navigate to Repos > Repositories in the left navigation panel.
- Select your repository.
- Enable Advanced Security.
![Enable Advanced Security](../../../images/EnableAdvancedSecurity.png)
You can manage Endor Labs variables centrally by configuring them within your Azure project. You can assign these variables to various pipelines.
- Log in to Azure and select Pipelines > Library.
- Click +Variable Group to add a new variable group for Endor Labs.
- Enter a name for the variable group, for example,
tenant-variables
, and click Add under Variables.
- Add the following variables.
ENDOR_API_CREDENTIALS_KEY
ENDOR_API_CREDENTIALS_SECRET
NAMESPACE
![Create Variables](../../../images/createvariables.png)
- Select the variable group that you created.
![Create Variables](../../../images/variableset.png)
- Click Pipeline Permissions.
- Click + to add the pipelines in which you want to use the variable group.
![Create Variables](../../../images/assignvariableset.png)
- Create
azure-pipelines.yml
file in your project, if it doesn’t exist.
- In the
azure-pipelines.yml
file, customize the job configuration based on your project’s requirements.
- Adjust the image field to use the necessary build tools for constructing your software packages, and align your build steps with those of your project.
For example, update the node pool settings based on your operating system.
pool:
name: Default
vmImage: "windows-latest"
pool:
name: Default
vmImage: "ubuntu-latest"
pool:
name: Default
vmImage: "macOS-latest"
- Update your default branch from main if you do not use main as the default branch name.
- Modify any dependency or artifact caches to align with the languages and caches used by your project.
- Enter the following steps in the
azure-pipelines.yml
file to download endorctl.
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe) endorctl" | sha256sum -c
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64) endorctl" | shasum -a 256 --check
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
-
Enter the steps to build your project if your project needs building and setup steps.
-
Enter the following step in the azure-pipelines.yml
file to run endorctl scan to generate the SARIF file.
You can run endorctl scan with options according to your requirement, but you must include the -s
option to generate the SARIF file.
For example, use the --secrets
flag to scan for secrets.
- script: |
.\endorctl.exe scan -n $(NAMESPACE) -s scanresults.sarif
- script: |
.\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
- script: |
.\endorctl scan -n $(NAMESPACE) -s scanresults.sarif
-
Enter the following task in the azure-pipelines.yml
to publish the scan results.
- task: AdvancedSecurity-Publish@1
displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
inputs:
SarifsInputDirectory: $(Build.SourcesDirectory)\
After a successful run of the pipeline, you can view the results in Azure.
Azure Pipeline Examples
trigger:
- none
pool:
name: Azure Pipelines
vmImage: "windows-latest"
variables:
- group: tenant-variables
steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_windows_amd64.exe -o endorctl.exe
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_windows_amd64.exe) endorctl" | sha256sum -c
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
```
displayName: 'Downloading latest version of endorctl'
continueOnError: false
- script: |
.\endorctl.exe scan --secrets -n $(NAMESPACE) -s scanresults.sarif
displayName: 'Run a scan against the repository using your API key & secret pair'
- task: AdvancedSecurity-Publish@1
displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
inputs:
SarifsInputDirectory: $(Build.SourcesDirectory)\
trigger:
- none
pool:
name: Azure Pipelines
vmImage: "ubuntu-latest"
variables:
- group: tenant-variables
steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"__linux_amd64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_linux_amd64) endorctl" | sha256sum -c
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
## Modify the permissions of the binary to ensure it is executable
chmod +x ./endorctl
## Create an alias of the endorctl binary to ensure it is available in other directories
alias endorctl="$PWD/endorctl"
displayName: 'Downloading latest version of endorctl'
continueOnError: false
- script: |
./endorctl scan --secrets -n $(NAMESPACE) -s scanresults.sarif
displayName: 'Run a scan against the repository using your API key & secret pair'
- task: AdvancedSecurity-Publish@1
displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
inputs:
SarifsInputDirectory: $(Build.SourcesDirectory)/
trigger:
- none
pool:
name: Azure Pipelines
vmImage: "macos-latest"
variables:
- group: tenant-variables
steps:
# All steps related to building of the project should be before this step.
# Implement and scan with Endor Labs after your build is complete.
- bash: |
echo "Downloading latest version of endorctl"
VERSION=$(curl https://api.endorlabs.com/meta/version | grep -o '"Version":"[^"]*"' | sed 's/.*"Version":"\([^"]*\)".*/\1/')
curl https://storage.googleapis.com/endorlabs/"$VERSION"/binaries/endorctl_"$VERSION"_macos_arm64 -o endorctl
echo "$(curl -s https://api.endorlabs.com/sha/latest/endorctl_macos_arm64) endorctl" | shasum -a 256 --check
if [ $? -ne 0 ]; then
echo "Integrity check failed"
exit 1
fi
## Modify the permissions of the binary to ensure it is executable
chmod +x ./endorctl
## Create an alias of the endorctl binary to ensure it is available in other directories
alias endorctl="$PWD/endorctl"
displayName: 'Downloading latest version of endorctl'
continueOnError: false
- script: |
./endorctl scan --secrets -n $(NAMESPACE) -s scanresults.sarif
displayName: 'Run a scan against the repository using your API key & secret pair'
- task: AdvancedSecurity-Publish@1
displayName: Publish '.\sarif\scanresults.sarif' to Advanced Security
inputs:
SarifsInputDirectory: $(Build.SourcesDirectory)/
View scan results in Azure
After the pipeline runs, you can view the scan results in Azure.
- Log in to Azure and navigate to your projects.
- Select Repos > Advanced Security to view the scan results.
![View Azure advanced security](../../../images/azure-advanced-security.png)
- Click an alert to view more details.
![View Azure alert](../../../images/azure-singleissue.png)
- If you ran endorctl with
--secrets
flag, you can view if there are any secret leaks.
Click the entry to view more details.
![View Azure secret leak expanded](../../../images/azuresecretexpanded.png)