Cloud Vendor Aggregator

The Cloud Vendor Aggregator processor is designed to aggregate events from various cloud vendors into a standardized asset shape. This processor is particularly useful for organizations that need to manage and analyze assets across multiple cloud environments.

info

This processor is tightly coupled with the Cloud vendor-specific sources.

It's usage is recommended only if you are using the following sources:

• Google Cloud Asset Inventory API Pub/Sub
• AWS CloudTrail SQS
• Microsoft Azure Monitor Activity Log Event Hub

In such cases, the processor MUST be used as the very first processor in the pipeline to ensure that the events are aggregated correctly before being sent to the next pipeline processors and the sink.

Processor Overview

The Processor is designed to produce a standardized asset shape that has the following structure:

Field Name	Type	Description
name	string	The name of the asset, typically derived from the cloud vendor-specific asset name.
type	string	The type of the asset, derived from the cloud vendor-specific asset type.
provider	string	The name of the cloud vendor, such as gcp, aws, or azure.
location	string	The location of the asset, such as a region or zone.
tags	object	A map of tags associated with the asset, where keys are tag names and values are tag values.
relationships	array	An array of strings representing relationships to other assets (e.g. the organization/project the resource is coming from)
timestamp	string	The timestamp of the generated asset, the string follows the ISO format
rawData	string	The event raw data from the cloud vendor, containing all the fields that were present in the original event. Note: this value is a base64 representation of the original binary data

Configuration

When configuring the Cloud Vendor Aggregator processor, you need to provide the following parameters in your configuration file:

• type (string): The type of the processor, which should be set to cloud-vendor-aggregator.
• cloudVendorName (string): The name of the cloud vendor for which the events are being aggregated. This can be one of the following values:
  • gcp
  • aws
  • azure
• authOptions (object, optional): Authentication options for the cloud vendor; the fields are vendor-specific, check out the below table for details.

caution

The cloudVendorName parameter is mandatory and must be set to the name of the cloud vendor based on the source type you are using.

Authentication Options

Google Cloud Platform (GCP)

Field Name	Type	Description
credentialsJson	SecretSource	The content of the credentials.json provided by GCP, specified as a secret source.

AWS

Field Name	Type	Description
accessKeyId	string	Access Key ID from AWS.
secretAccessKey	SecretSource	Secret Access Key from AWS, specified as a secret source.
sessionToken	SecretSource	Session Token from AWS, specified as a secret source.
region	string (optional)	The AWS region where the assets are located.

Microsoft Azure

Field Name	Type	Description
tenantId	string	The Azure tenant ID.
clientId	SecretSource	The Azure client ID, specified as a secret source.
clientSecret	SecretSource	The Azure client secret, specified as a secret source.

Example

Google Cloud Platform (GCP)

{
    "type": "cloud-vendor-aggregator",
    "cloudVendorName": "gcp",
    "authOptions": {
        "credentialsJson": {
            "fromEnv": "GCP_CREDENTIALS_JSON"
        }
    }
}

AWS

{
    "type": "cloud-vendor-aggregator",
    "cloudVendorName": "aws",
    "authOptions": {
        "accessKeyId": "your-access-key",
        "secretAccessKey": {
            "fromEnv": "AWS_SECRET_ACCESS_KEY"
        },
        "sessionToken": {
            "fromEnv": "AWS_SESSION_TOKEN"
        }
    }
}

Microsoft Azure

{
  "type": "cloud-vendor-aggregator",
  "cloudVendorName": "azure",
  "authOptions": {
    "tenantId": "your-tenant-id",
    "clientId": {
      "fromEnv": "AZURE_CLIENT_ID"
    },
    "clientSecret": {
      "fromEnv": "AZURE_CLIENT_SECRET"
    }
  }
}