Quickstart

This project uses the Hexagonal Architecture pattern and is designed to extract information from GitHub, GitLab, Bitbucket, and Azure DevOps. This information is stored, processed and analyzed automatically, allowing the calculation of important indicators for a project or team.

Project structure

The code is structured into three main packages:

• Application: This package contains all the application layer code which determines how the various components of the system interact with each other. It contains various services, controllers, and ports implementation to handle incoming data.
• Domain: The domain package contains business logic, models, and interfaces (Ports). Business rules and logic reside in this package, which allows these rules and logic to be technology-agnostic.
• Framework: The framework package includes the technology-specific implementation of the system. This is where the details about data sources, databases, APIs, etc., are implemented.

Data Flow

The process for data retrieval and handling is as follows:

• The system receives payload data from a webhook (GitHub webhook, GitLab webhook, Azure DevOps webhook, Bitbucket webhook).
• The system checks the payload for necessary data, if some data, for example, the stats from the commits, are missing in the payload, the system triggers a call to the API (GitHub API, GitLab API, Azure DevOps API, Bitbucket API) to fetch the missing data.
• The extracted data is then processed and used as needed in the client port.

Requirements

• Java 17
• Docker
• Intellij/Eclipse
• Mongodb Compass

General Settings

Environment variables that must be configured in the project.

• MONGODB Environment variable that stores the value of the connection url to the Database example: mongodb+srv://username:[email protected]/performance-persistence
• AUTH_BASIC_USERNAME Username for connection to the API
• AUTH_BASIC_PASSWORD Password for connection to the API
• PROFILES_DEFAULT Environment variable that stores the value of the profiles or platforms to process

  Possible profile values ​​are: github, gitlab, bitbucket, ado and jira. For local tests, the local profile is also used. Examples:

  • PROFILES_DEFAULT = local
  • PROFILES_DEFAULT = gitlab
  • PROFILES_DEFAULT = bitbucket
  • PROFILES_DEFAULT = github
  • PROFILES_DEFAULT = ado
  • PROFILES_DEFAULT = jira
  • PROFILES_DEFAULT = copilot
• LICENSE_API_APP_KEY Connection key with the license API

GitHub Settings

Environment variables that must be configured in the project for GitHub

• GITHUB_ORGANIZATION Environment variable that stores the value of the workspace to process

  The account owner of the repository. (Organization)

• GITHUB_ENTERPRISE Environment variable that stores the value of the GitHub enterprise to process

  If the enterprise and the organization have the same value, it is not necessary to define the environment variable GITHUB_ENTERPRISE, only the environment variable GITHUB_ORGANIZATION must be defined.

• GITHUB_WEBHOOK_SECRET Environment variable that stores the secret key value to process webhooks

  If provided, the secret will be used as the key to generate the HMAC hex digest value for delivery signature headers.

• GITHUB_TOKEN Environment variable that stores the value of the token for the connection to GRAPHQL and API

  How to obtain the GITHUB_TOKEN:

  1. Go to the GitHub Developer settings.
  2. Click on Personal access tokens and select Tokens (classic).
  3. Select Generate new token and select Generate new token (classic).
  4. Ensure you create a token (this often means selecting a scope such as Organization or User with the necessary permissions) with

  • repo (repo:status, repo_deployment, public_repo, repo:invite, security_events)
  • workflow
  • read:packages
  • read:org
  • write:repo_hook (read:repo_hook)
  • admin:org_hook
  • read:user
  • user:email
  • read:enterprise
  • read:project

  5. Click on Generate token.

  If you also want to add permissions for Copilot Metric, be sure to add manage_billing:copilot or read:enterprise to the token and it can be used in the COPILOT_TOKEN environment variable.

GitLab Settings

Environment variables that must be configured in the project for GitLab

• GITLAB_CLIENT_AUTHORIZATION Environment variable that stores the value of the token to connect to the API

Azure DevOps Settings

Environment variables that must be configured in the project for Azure DevOps

• AZURE_DEVOPS_TOKEN Environment variable that stores the value of the token to connect to the API

  How to obtain the AZURE_DEVOPS_TOKEN:

  1. Navigate to Azure DevOps (https://dev.azure.com/{your_organization}).
  2. Click on your profile picture in the upper-right corner and select User Setting.
  3. Click on Personal Access Tokens and the page is loading.
  4. Click blue New Token button.
  5. Ensure you create a token (this often means selecting a scope such as Organization or User with the necessary permissions) with

  • Build (Read)
  • Code (Read)
  • Team Dashboard (Read)
  • Entitlements (Read)
  • Environment (Read & manage)
  • Extension Data (Read)
  • Graph (Read)
  • Identity (Read)
  • Deployment Groups (Read & manage)
  • Member Entitlement Management (Read)
  • Pipeline Resources (Use)
  • User Profile (Read)
  • Project and Team (Read)
  • Security (Manage)
  • Task Groups (Read)
  • Pull Request Threads (Read & write)
  • Variable Groups (Read)
  • Notifications(Read)
  • Packaging (Read)
  • Release (Read)
  • Service Connections (Read)
  • Test Management (Read)

  6. Copy the token value and set it as the AZURE_DEVOPS_TOKEN environment variable.
  7. Note: For operations involving team management, the token must have elevated permissions or be a super token.
• AZURE_DEVOPS_ORGANIZATION Environment variable that stores the value of the workspace to process, and the token should be created by Team Project Administrator Role.

  How to obtain the AZURE_DEVOPS_ORGANIZATION:

  1. Go to your Azure DevOps organization URL (e.g., https://dev.azure.com/{your_organization}).
  2. The organization name is part of the URL and is typically found after dev.azure.com/. For example, in https://dev.azure.com/cleverit, the organization is cleverit.
  3. Set this value as the AZURE_DEVOPS_ORGANIZATION environment variable.

Bitbucket Settings

Environment variables that must be configured in the project for Bitbucket

• BITBUCKET_CLIENT_ID Environment variable that stores the value of the user's KEY token to connect to the API
• BITBUCKET_CLIENT_SECRET Environment variable that stores the value of the user's SECRET token to connect to the API

  How to obtain the BITBUCKET_CLIENT_ID and BITBUCKET_CLIENT_SECRET: To obtain the value of the BITBUCKET_CLIENT_ID and BITBUCKET_CLIENT_SECRET variables we can access it by following Settings->Workspace settings->OAuth consumers->Add OAuth consumer.

  1. Click Settings and then click Workspace settings.
  2. In the Workspace settings section, locate the OAuth consumers option in the menu and click on it.
  3. Click the Add consumer button and configure the properties for the connection to Bitbucket. You must provide a name (Name), any url (Callback URL), check the This is a private consumer option and check all read permissions.
  4. Save the changes by clicking the Save button. Copy the generated Key and Secret values.
• BITBUCKET_WORKSPACE Environment variable that stores the value of the workspace to process

GitHub Copilot Settings

Environment variables that must be configured in the project for obtain GitHub Copilot metrics usage

• COPILOT_ENTERPRISE Environment variable that stores the value of the GitHub Copilot enterprise to process

• COPILOT_ORGANIZATION Environment variable that stores the value of the GitHub Copilot organization to process

  If the enterprise and the organization have the same value, it is not necessary to define the environment variable COPILOT_ORGANIZATION, only the environment variable COPILOT_ENTERPRISE must be defined.

• COPILOT_TOKEN Environment variable that stores the value of the OAuth app tokens for the connection

  OAuth app tokens and personal access tokens (classic) need either the manage_billing:copilot or read:enterprise scopes to use this endpoint.

• COPILOT_METRICS_LEVEL Environment variable that indicates the load level of the metrics

  Its possible values ​​are organization | enterprise. By default, if this variable is not set, enterprise is assumed.

• To create a new OAuth app, go to the GitHub Developer settings.

Jira Settings

Environment variables that must be configured in the project for Jira

• JIRA_URL Environment variable with the connection URL to Jira
• JIRA_TOKEN Environment variable with the token to connect

  Create an API token for basic authentication (Admin user):

  1. Log in to your Atlassian account
  2. Go to https://id.atlassian.com/manage-profile/security/api-tokens
  3. Select Create API token
  4. Name the token and set an expiration date
  5. Select Create
  6. Select Copy to clipboard
• JIRA_USER Environment variable with the user's email associated with the token to connect