Introduction

Most vulnerability programs do not fail because they lack feeds. They fail because every new feed adds more work than the team can absorb.

NVD entries, vendor advisories, CISA Known Exploited Vulnerabilities (KEV), scanner findings, cloud posture alerts, container image results, software bill of materials (SBOM) data, and threat intelligence feeds all describe pieces of the same problem. But without a prioritization model, they become an expensive notification system: more tickets, more dashboards, more exceptions, and less confidence about what must be fixed first.

Vulnerability intelligence should answer one operational question:

Which vulnerabilities create the most plausible risk to the systems that matter right now?

That is a prioritization question, not a feed aggregation question. A useful vulnerability-intelligence program combines public sources such as NVD and CISA KEV with internal context: asset criticality, internet exposure, compensating controls, exploit maturity, ownership, business function, and remediation path. The result is fewer alerts, clearer service-level objectives, and better conversations between security, engineering, and operations.

Current Landscape Statistics

• 25,000+ CVEs are published annually in recent years, and the volume continues to grow across operating systems, applications, cloud services, and open-source dependencies.
• 1,000+ entries are listed in the CISA KEV catalog, representing vulnerabilities with confirmed exploitation in the wild.
• Single CVSS scores are insufficient for operational triage because they describe technical severity, not whether the vulnerable software is present, exposed, exploitable, or business-critical in your environment.
• Alert fatigue is a security risk: when every vulnerability looks urgent, teams either ignore the queue or create informal prioritization rules that are hard to audit.
• Context-rich programs reduce noise by suppressing non-applicable findings, escalating externally exposed critical assets, and routing work directly to accountable owners.

The Feed Trap: More Data Does Not Equal Better Decisions

Adding another vulnerability feed feels productive because it increases visibility. The problem is that visibility without decision logic increases cognitive load.

A typical security team may receive vulnerability signals from:

• NVD CVE records and CVSS metadata
• CISA KEV entries for known exploited vulnerabilities
• Vendor advisories and emergency bulletins
• Container and package scanners
• Endpoint detection and response tooling
• Cloud security posture management findings
• Web application scanners
• Asset inventory systems
• SBOM and dependency graphs
• Threat intelligence reports

Each source is useful. None of them is sufficient by itself.

NVD can tell you that a vulnerability exists and provide standardized scoring metadata. CISA KEV can tell you that a vulnerability has been exploited in the wild. A scanner can tell you that a package or configuration appears vulnerable. Your asset inventory can tell you where the affected technology exists. Your cloud telemetry can tell you whether the system is internet-facing. Your change-management data can tell you who owns it and whether a patch is deployable.

The mistake is treating all of those signals as separate queues. That creates duplicate work and competing definitions of urgency. A better model normalizes the signals into one decision pipeline.

Start with a Triage Contract

Before adding automation, define a triage contract that everyone can understand. The contract should describe how vulnerability intelligence becomes action.

At minimum, document:

1. Intake sources — Which sources are authoritative for CVE metadata, exploitation status, asset inventory, and ownership?
2. Applicability rules — How do you prove a finding is relevant to an asset, package, image, runtime, or exposed service?
3. Priority model — Which factors can raise or lower urgency?
4. Service-level objectives — How quickly must each priority level be remediated or mitigated?
5. Exception rules — Who can accept residual risk, for how long, and with what compensating controls?
6. Feedback loop — How are false positives, duplicate tickets, and stale ownership records corrected?

A triage contract prevents the vulnerability program from becoming a personality-driven escalation process. It also makes automation safer because the logic is explicit before it is encoded into workflows.

Use NVD for Baseline Metadata, Not Final Priority

The National Vulnerability Database is a foundational source for CVE enrichment. It provides standardized identifiers, descriptions, references, affected products through Common Platform Enumeration (CPE), and CVSS scoring. NVD is excellent for baseline metadata.

But NVD should not be the last word on urgency.

CVSS answers, “How severe could this vulnerability be under standardized assumptions?” Operational prioritization asks, “How likely is this vulnerability to affect our environment in a meaningful way?”

Those are different questions.

What NVD Is Good For

Use NVD to normalize and enrich:

• CVE identifiers and descriptions
• CVSS base metrics
• Affected product references
• Vendor advisory links
• Weakness mappings such as CWE
• Initial severity bands for routing and reporting

Where NVD Needs Context

NVD does not know whether:

• The vulnerable component is deployed in production
• The system is reachable from the internet
• The vulnerable feature is enabled
• The asset supports a critical business process
• Existing controls block exploit paths
• An exploit is reliable and actively used by attackers
• A patch would break a revenue-impacting workflow

That missing context is where your internal intelligence layer matters.

Use CISA KEV as a Fast Escalation Signal

CISA KEV is one of the most operationally useful public vulnerability sources because inclusion means the vulnerability is known to be exploited in the wild. That does not mean every KEV item is equally urgent for every organization, but it does mean the finding deserves faster evaluation.

A practical KEV rule looks like this:

If a KEV-listed vulnerability is present on an internet-facing or mission-critical asset, escalate it above routine scanner backlog and require mitigation, patching, or documented risk acceptance.

This works because KEV changes the conversation from theoretical severity to demonstrated attacker behavior.

KEV Triage Questions

For each KEV match, answer:

• Is the affected product or component present?
• Is the affected version actually running?
• Is the vulnerable interface reachable?
• Is the asset internet-facing, partner-facing, or internal-only?
• Is the asset tied to identity, remote access, perimeter security, build infrastructure, or sensitive data?
• Are compensating controls already reducing exploitability?
• Is there a vendor patch, configuration workaround, or isolation option?

The goal is not to panic every time KEV updates. The goal is to create a reliable fast lane for vulnerabilities that have crossed from “could be exploited” to “has been exploited.”

Build a Context-Weighted Priority Model

A good prioritization model separates signal from decision. Each source contributes evidence. The model turns evidence into action.

Consider scoring vulnerabilities across five dimensions:

This produces better outcomes than raw severity. A medium CVSS vulnerability on an exposed identity system with active exploitation may deserve faster action than a critical CVSS vulnerability in a disabled library on a non-production host.

Example Priority Bands

Use plain-language bands that map to operational behavior:

Priority 1: Active Risk

• KEV-listed or confirmed active exploitation
• Present on internet-facing, identity, build, security, or business-critical assets
• Requires immediate mitigation, patching, isolation, or formal risk acceptance

Priority 2: Likely Risk

• Public exploit code or strong exploit indicators
• Present on production assets or reachable internal services
• Requires near-term remediation with clear ownership

Priority 3: Managed Risk

• High technical severity but limited exposure or strong compensating controls
• Requires scheduled remediation and monitoring

Priority 4: Hygiene Backlog

• Low exploitability, non-production scope, or uncertain applicability
• Remediate through normal patch cycles or dependency refreshes

The exact labels matter less than the behavior they trigger. Every band should define ownership, SLA, escalation path, and closure evidence.

Prevent Alert Fatigue with Suppression and Routing Rules

Alert fatigue is not just a human problem. It is usually a systems-design problem.

A vulnerability-intelligence pipeline should reduce noise before it reaches engineers. That means suppressing findings that are not actionable, deduplicating repeated signals, and routing tickets to the right owner with the context needed to fix the issue.

Suppress Non-Actionable Findings

Suppress or downgrade findings when:

• The affected package is present only in a build cache or unused layer
• The vulnerable feature is disabled and cannot be reached
• The asset has been decommissioned or is outside the supported scope
• A compensating control blocks the known exploit path
• The finding duplicates an existing tracked ticket

Suppression must be auditable. Avoid silent drops. Record why the finding was suppressed, who approved the logic, and when the rule expires or must be revalidated.

Route with Remediation Context

A useful ticket should include:

• CVE identifier and summary
• Priority band and reasons for the priority
• Affected asset, package, image, or service
• Evidence of exposure or exploitability
• CISA KEV status if applicable
• Vendor advisory and patch reference
• Recommended mitigation path
• Asset owner or service team
• Due date based on the triage contract

If engineers have to open five dashboards to understand why a ticket exists, the pipeline is creating toil instead of reducing risk.

AWS-Centric Implementation Pattern

You can implement a context-driven vulnerability-intelligence workflow using managed AWS services without building a heavy platform on day one.

A practical architecture looks like this:

1. Ingest public vulnerability metadata from NVD, CISA KEV, vendor advisories, and scanner outputs.
2. Normalize records into a common schema keyed by CVE, package, product, image, and asset identifier.
3. Enrich with asset inventory, cloud tags, ownership, exposure, environment, and business criticality.
4. Score findings using your priority model.
5. Route actionable items to ticketing, chat, or security orchestration workflows.
6. Measure aging, exceptions, false positives, and remediation outcomes.

Reference Service Map

This pattern keeps the system modular. You can start with scheduled KEV and NVD enrichment, then add scanner outputs, SBOM data, and cloud exposure over time.

Example Priority Logic

The following pseudocode shows the shape of a context-weighted score. The specific weights should match your triage contract.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
def prioritize(finding):
    score = 0
    reasons = []

    if finding.get("cisa_kev") is True:
        score += 40
        reasons.append("listed in CISA KEV")

    if finding.get("internet_exposed") is True:
        score += 25
        reasons.append("internet-facing asset")

    if finding.get("asset_tier") in ["identity", "build", "security", "critical-production"]:
        score += 20
        reasons.append(f"critical asset tier: {finding['asset_tier']}")

    if finding.get("public_exploit") is True:
        score += 15
        reasons.append("public exploit available")

    if finding.get("compensating_control") is True:
        score -= 15
        reasons.append("compensating control documented")

    if finding.get("applicability") == "not-affected":
        return {
            "priority": "suppressed",
            "reasons": ["not applicable to deployed configuration"]
        }

    if score >= 70:
        priority = "P1 Active Risk"
    elif score >= 45:
        priority = "P2 Likely Risk"
    elif score >= 20:
        priority = "P3 Managed Risk"
    else:
        priority = "P4 Hygiene Backlog"

    return {"priority": priority, "score": score, "reasons": reasons}

The important part is explainability. A priority value without reasons will be challenged. A priority value with clear evidence becomes a decision record.

Where VulnDB-Style Intelligence Fits

Commercial vulnerability intelligence can add value when it improves timeliness, affected-version precision, exploit context, or remediation guidance. A source such as VulnDB is most useful when it fills gaps between public CVE publication, vendor disclosure, and operational scanner coverage.

The pattern still matters more than the product. Treat any intelligence source as evidence, not as an automatic ticket generator. If a feed cannot be mapped to affected assets, exploitability, exposure, and ownership, it will add noise faster than it adds value.

Implementation Roadmap

Phase 1: Stabilize the Basics (Weeks 1-2)

• Define authoritative sources for CVE metadata, KEV status, scanner findings, assets, and ownership
• Create a four-level priority model with clear remediation expectations
• Identify internet-facing, identity, build, and security-control assets
• Document exception and suppression requirements
• Establish reporting for vulnerability age by priority band

Phase 2: Add Context and Automation (Weeks 3-6)

• Ingest NVD and CISA KEV updates on a scheduled basis
• Normalize scanner findings by CVE, package, image, and asset
• Enrich findings with exposure, environment, business function, and owner
• Deduplicate repeated findings before ticket creation
• Route P1 and P2 findings with evidence and remediation guidance

Phase 3: Optimize the Feedback Loop (Weeks 7-12)

• Track false positives and convert recurring patterns into suppression rules
• Review exception aging and compensating-control evidence
• Measure mean time to remediate by priority, team, and asset tier
• Tune scoring weights based on incidents, exploit activity, and remediation outcomes
• Publish a monthly vulnerability-intelligence review for engineering leadership

Best Practices for Sustainable Vulnerability Intelligence

Make Priority Explainable

Every escalated finding should explain why it matters. “Critical CVSS” is not enough. “CISA KEV, exposed perimeter appliance, production identity dependency, patch available” is actionable.

Separate Discovery from Demand

Discovery systems should find as much as possible. Demand systems should ask humans to fix only what has been triaged into an actionable state. Mixing those functions creates noise.

Keep Suppression Rules Temporary

Permanent suppression is risky. Use expiration dates, revalidation checks, and owner review. A finding that is not exploitable today may become exploitable after an architecture change.

Measure Outcomes, Not Feed Volume

Do not celebrate ingesting more advisories. Measure reduced exposure windows, fewer duplicate tickets, faster remediation for exploited vulnerabilities, and better exception quality.

Preserve Human Judgment for Edge Cases

Automation should handle routing, enrichment, and obvious priority decisions. Humans still need to evaluate business impact, complex compensating controls, outage risk, and ambiguous exploitability.

Related Articles

• Event-Driven Security Automation: EventBridge + Step Functions with Open-Source Threat Intel
• SBOM + Supply Chain Security: Syft, Grype, and AWS Inspector
• Security Observability with OpenTelemetry and AWS
• AWS DevSecOps Pipeline Security Automation: Complete Implementation Guide

Additional Resources

Official Documentation

• National Vulnerability Database - CVE enrichment, CVSS data, references, and vulnerability metadata
• CISA Known Exploited Vulnerabilities Catalog - Public catalog of vulnerabilities known to be exploited in the wild
• FIRST CVSS - Common Vulnerability Scoring System documentation
• AWS Security Hub - Aggregation and prioritization for AWS security findings

Tools and Frameworks

• OpenVEX - Machine-readable vulnerability exploitability exchange format
• CycloneDX VEX - VEX support for communicating affected and not-affected status
• Grype - Open-source vulnerability scanner for container images and filesystems
• OSV - Vulnerability database focused on open-source ecosystems

Conclusion

Vulnerability intelligence is valuable when it changes decisions. NVD gives you baseline metadata. CISA KEV gives you a fast signal for known exploitation. Scanner outputs tell you where vulnerable components may exist. Internal context tells you whether the issue is reachable, important, owned, and fixable.

The winning pattern is not “more feeds.” It is a context-rich prioritization system that turns vulnerability data into focused remediation work without exhausting the people responsible for fixing it.

If you are building a vulnerability-intelligence program or refining how security context reaches engineering teams, connect through jonprice.io for practical cloud security and DevSecOps guidance.

Introduction

Cloud migration has fundamentally shifted where workloads run, but it has not eliminated network-layer threats. Attackers still rely on command-and-control channels, lateral movement across VPC subnets, data exfiltration over DNS, and cryptomining traffic that hides in plain sight. According to the IBM X-Force 2025 Threat Intelligence Index, network-based intrusion attempts remain a top initial access vector, and the intrusion detection and prevention systems market is projected to reach USD 9.11 billion by 2030.

AWS Network Firewall gives you a managed, scalable network inspection service sitting inline with your VPC traffic. What many teams do not realize is that under the hood, Network Firewall runs Suricata, the open-source network threat detection engine maintained by the Open Information Security Foundation (OISF). This means you can move far beyond the AWS-managed rule groups and tap into the massive open-source Suricata rule community — tens of thousands of detection signatures maintained by researchers worldwide, updated daily, and completely free.

This guide walks you through the architecture, shows you how to import community rules from the Emerging Threats (ET Open) ruleset, automates rule updates with Lambda, and demonstrates detection scenarios for the most common cloud threats. The approach follows our core philosophy: build it yourself with open source for maximum freedom and flexibility.

Current Landscape Statistics

• USD 9.11B projected IDPS market size by 2030, growing at 5.4% CAGR
• 60% faster threat detection for organizations using AI-augmented IDS systems
• 108 days sooner breach discovery with automated detection versus manual approaches
• 30,000 maximum Suricata rules supported per AWS Network Firewall policy
• Daily updates from the Emerging Threats Open ruleset covering active threat campaigns

AWS Network Firewall Architecture — It Is Suricata Under the Hood

AWS Network Firewall is a managed, stateful network inspection and filtering service. It sits in a dedicated firewall subnet within your VPC and inspects traffic as it flows between your workloads and the internet, between VPCs, or between subnets.

AWS Network Firewall architecture with custom Suricata rules and CloudWatch alerting

The Suricata Engine Inside

The stateful inspection engine in AWS Network Firewall is Suricata. As of November 2024, AWS upgraded from Suricata 6.0.9 to Suricata 7.0, bringing PCRE2 regex support and improved protocol detection. This is the same engine used by thousands of organizations running on-premises IDS/IPS deployments, which means:

• Full deep packet inspection (DPI) of traffic flows
• Application-layer protocol detection for HTTP, TLS, DNS, SSH, and more
• Stateful connection tracking with flow-aware rule evaluation
• Suricata-compatible rule syntax for writing custom detection logic

Key Architecture Components

Firewall Policy: The top-level container that defines rule evaluation order and default actions. Use strict order evaluation so rules execute in your defined sequence rather than Suricata’s default action-order processing.

Stateful Rule Groups: Collections of Suricata-compatible rules. You can have up to 20 rule groups (managed + custom combined) per firewall policy.

Firewall Endpoints: Gateway Load Balancer endpoints deployed in firewall subnets that route traffic through the inspection engine.

Logging Configuration: Flow logs and alert logs that ship to CloudWatch Logs, S3, or Kinesis Data Firehose.

VPC Architecture with Network Firewall Inspection

Here is a Terraform module that deploys the foundational VPC architecture with Network Firewall inline inspection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
# vpc.tf - VPC with Network Firewall inspection architecture
resource "aws_vpc" "protected" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true

  tags = {
    Name        = "protected-vpc"
    Customer    = "internal"
    Application = "network-firewall-demo"
    Environment = "production"
    Owner       = "security-team"
    Costcenter  = "security"
  }
}

# Firewall subnet - dedicated to Network Firewall endpoints
resource "aws_subnet" "firewall" {
  count             = 2
  vpc_id            = aws_vpc.protected.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "firewall-subnet-${count.index}"
  }
}

# Public subnet - internet-facing resources
resource "aws_subnet" "public" {
  count                   = 2
  vpc_id                  = aws_vpc.protected.id
  cidr_block              = cidrsubnet("10.0.0.0/16", 8, count.index + 10)
  availability_zone       = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "public-subnet-${count.index}"
  }
}

# Private subnet - workloads
resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.protected.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 8, count.index + 20)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "private-subnet-${count.index}"
  }
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.protected.id
}

# Route table for IGW - sends return traffic through firewall
resource "aws_route_table" "igw" {
  vpc_id = aws_vpc.protected.id

  route {
    destination_cidr_block = aws_subnet.public[0].cidr_block
    vpc_endpoint_id        = tolist(aws_networkfirewall_firewall.main.firewall_status[0].sync_states[*].attachment[0].endpoint_id)[0]
  }

  tags = {
    Name = "igw-route-table"
  }
}

resource "aws_route_table_association" "igw" {
  gateway_id     = aws_internet_gateway.main.id
  route_table_id = aws_route_table.igw.id
}

Managed Rule Groups — What AWS Provides Out of the Box

AWS offers several categories of managed rule groups through Network Firewall and via AWS Marketplace partners:

AWS Managed Threat Signatures

AWS provides managed stateful rule groups that cover common threat categories:

• Malware signatures — known malware communication patterns
• Botnet command-and-control — signatures for known C2 infrastructure
• Threat intelligence feeds — AWS-curated indicators of compromise

AWS Marketplace Partner Rules

Third-party providers like CrowdStrike, Fortinet, and Palo Alto Networks offer managed rule groups through AWS Marketplace with additional coverage.

What You Get with Managed Rules

Why Managed Rules Are Not Enough

Managed rule groups provide a baseline, but they leave significant gaps that matter for serious security operations:

No rule visibility: You cannot inspect the actual Suricata signatures in managed rule groups. You see category names and can override actions (alert vs. drop), but you cannot read, modify, or understand the detection logic. This is a black box.

No HOME_NET customization: Managed rule groups do not let you override Suricata’s $HOME_NET variable. If your VPC CIDR does not align with the default, directional rules may not match correctly.

Limited coverage for targeted threats: AWS managed rules cover broad categories but do not include signatures for many specific threat campaigns, emerging exploits, or industry-specific attack patterns that the open-source community tracks daily.

Update timing is opaque: You do not control when managed rules update, and AWS does not publish a changelog. The ET Open community publishes updates daily on weekdays with full transparency.

Rule capacity constraints: With a maximum of 20 combined rule groups per policy, every managed group you add reduces your capacity for custom detection logic.

No tuning for your environment: Every environment has unique traffic patterns. Custom rules let you write signatures specific to your protocols, internal services, and known-good traffic patterns to minimize false positives.

Suricata Rule Format Deep Dive

Understanding Suricata rule syntax is essential for writing effective custom detections. Every rule consists of three parts: action, header, and options.

Rule Structure

1
action protocol source_ip source_port -> destination_ip destination_port (options;)

Actions Supported by AWS Network Firewall

Example: Detecting DNS Exfiltration

1
alert dns $HOME_NET any -> any any (msg:"ET POLICY Possible DNS Tunneling - Long TXT Query"; dns.query; content:"."; offset:40; sid:9000001; rev:1; classtype:policy-violation;)

Breaking this down:

• alert — generate an alert, do not block
• dns — match DNS protocol traffic
• $HOME_NET any -> any any — from any internal host/port to any destination
• dns.query — inspect the DNS query field specifically
• content:"."; offset:40 — look for a dot character 40+ bytes into the query (unusually long subdomain, common in DNS tunneling)
• sid:9000001 — signature ID above 9,000,000 to avoid collisions with community rules
• classtype:policy-violation — categorize the alert

Key Rule Keywords for Cloud Detection

1
2
3
4
5
6
7
8
# TLS inspection - detect suspicious certificate properties
alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"Suspicious Self-Signed Certificate"; tls.cert_subject; content:"O="; nocase; content:"Let"; distance:0; tls.cert_issuer; content:"O="; nocase; content:"Let"; distance:0; sid:9000010; rev:1;)

# HTTP inspection - detect beacon behavior
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Possible C2 Beacon - Fixed Interval HTTP POST"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"/api/"; pcre:"/^\/api\/[a-f0-9]{8}$/"; threshold:type both, track by_src, count 5, seconds 60; sid:9000020; rev:1;)

# SSH anomaly - detect SSH over non-standard ports
alert ssh any any -> $EXTERNAL_NET !22 (msg:"SSH Traffic on Non-Standard Port"; flow:established; sid:9000030; rev:1; classtype:policy-violation;)

AWS Network Firewall Limitations

Not all Suricata features work in Network Firewall. These are not supported:

• Datasets (dataset and datarep keywords)
• ENIP/CIP protocol keywords
• File extraction
• FTP-data protocol detection
• IP reputation (iprep keyword)
• Lua scripting
• PCRE1 format (must use PCRE2 as of Suricata 7.0)

Always test your rules against these constraints before deploying.

Emerging Threats (ET Open) — The Community Rule Ecosystem

The Emerging Threats Open (ET Open) ruleset is the gold standard for community-maintained Suricata detection signatures. Maintained by Proofpoint’s Emerging Threats team, ET Open provides:

• Tens of thousands of signatures covering malware, exploits, policy violations, and protocol anomalies
• Daily updates on weekdays, with out-of-band updates for critical threats
• Categorized rule files organized by threat type (malware, exploit, policy, info, etc.)
• Free and open source under a permissive license
• Suricata 7.0 compatible rulesets available

ET Open Rule Categories

Downloading ET Open Rules

1
2
3
4
5
6
7
8
9
10
# Download the latest ET Open ruleset for Suricata 7.0
curl -LO https://rules.emergingthreats.net/open/suricata-7.0/emerging.rules.tar.gz

# Extract and inspect
tar xzf emerging.rules.tar.gz
ls rules/

# Count total rules
grep -c "^alert\|^drop\|^pass\|^reject" rules/*.rules
# Typical output: 30,000+ signatures

Importing Custom Suricata Rules into Network Firewall

Terraform Configuration for Custom Rule Groups

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
# network_firewall.tf - Network Firewall with custom Suricata rules

resource "aws_networkfirewall_firewall" "main" {
  name                = "suricata-nfw"
  firewall_policy_arn = aws_networkfirewall_firewall_policy.main.arn
  vpc_id              = aws_vpc.protected.id

  dynamic "subnet_mapping" {
    for_each = aws_subnet.firewall[*].id
    content {
      subnet_id = subnet_mapping.value
    }
  }

  tags = {
    Name        = "suricata-nfw"
    Customer    = "internal"
    Application = "network-security"
    Environment = "production"
    Owner       = "security-team"
    Costcenter  = "security"
  }
}

resource "aws_networkfirewall_firewall_policy" "main" {
  name = "suricata-policy"

  firewall_policy {
    stateless_default_actions          = ["aws:forward_to_sfe"]
    stateless_fragment_default_actions = ["aws:forward_to_sfe"]

    stateful_engine_options {
      rule_order = "STRICT_ORDER"
    }

    # Custom ET Open rules - highest priority
    stateful_rule_group_reference {
      priority     = 1
      resource_arn = aws_networkfirewall_rule_group.et_malware.arn
    }

    stateful_rule_group_reference {
      priority     = 2
      resource_arn = aws_networkfirewall_rule_group.et_trojan.arn
    }

    stateful_rule_group_reference {
      priority     = 3
      resource_arn = aws_networkfirewall_rule_group.et_exploit.arn
    }

    stateful_rule_group_reference {
      priority     = 4
      resource_arn = aws_networkfirewall_rule_group.custom_cloud_rules.arn
    }

    # AWS managed rules as supplementary coverage
    stateful_rule_group_reference {
      priority     = 10
      resource_arn = "arn:aws:network-firewall:us-east-1:aws-managed:stateful-rulegroup/ThreatSignaturesMalware"
    }
  }
}

# Custom rule group: ET Open Malware signatures
resource "aws_networkfirewall_rule_group" "et_malware" {
  capacity = 5000
  name     = "et-open-malware"
  type     = "STATEFUL"

  rule_group {
    rule_variables {
      ip_sets {
        key = "HOME_NET"
        ip_set {
          definition = ["10.0.0.0/16"]
        }
      }
      ip_sets {
        key = "EXTERNAL_NET"
        ip_set {
          definition = ["0.0.0.0/0"]
        }
      }
    }

    rules_source {
      rules_string = file("${path.module}/rules/emerging-malware-filtered.rules")
    }

    stateful_rule_options {
      capacity = 5000
    }
  }

  tags = {
    Name   = "et-open-malware"
    Source = "emerging-threats-open"
  }
}

# Custom rule group: Cloud-specific threat detections
resource "aws_networkfirewall_rule_group" "custom_cloud_rules" {
  capacity = 1000
  name     = "custom-cloud-threats"
  type     = "STATEFUL"

  rule_group {
    rule_variables {
      ip_sets {
        key = "HOME_NET"
        ip_set {
          definition = ["10.0.0.0/16"]
        }
      }
    }

    rules_source {
      rules_string = file("${path.module}/rules/custom-cloud-threats.rules")
    }
  }
}

Custom Cloud Threat Detection Rules

Save these as rules/custom-cloud-threats.rules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# =====================================================
# Custom Cloud Threat Detection Rules for AWS
# SID Range: 9000001 - 9000999
# =====================================================

# --- C2 Communication Detection ---
# Detect Cobalt Strike default HTTP beacon pattern
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"CLOUD C2 - Possible Cobalt Strike HTTP Beacon"; flow:established,to_server; http.method; content:"GET"; http.uri; pcre:"/^\/[a-zA-Z]{4}$/"; http.header; content:"Cookie:"; pcre:"/SESSIONID=[a-f0-9]{32}/"; threshold:type both, track by_src, count 3, seconds 300; sid:9000001; rev:2; classtype:trojan-activity;)

# Detect DNS-over-HTTPS to known DoH providers (potential C2 evasion)
alert tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"CLOUD POLICY - DNS-over-HTTPS to Public Resolver"; tls.sni; content:"dns.google"; sid:9000002; rev:1; classtype:policy-violation;)

alert tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"CLOUD POLICY - DNS-over-HTTPS to Cloudflare"; tls.sni; content:"cloudflare-dns.com"; sid:9000003; rev:1; classtype:policy-violation;)

# --- Data Exfiltration Detection ---
# Large DNS TXT responses (possible DNS exfiltration)
alert dns any any -> $HOME_NET any (msg:"CLOUD EXFIL - Large DNS TXT Response"; dns.opcode:0; content:"|00 10|"; offset:40; dsize:>512; threshold:type both, track by_dst, count 10, seconds 60; sid:9000010; rev:1; classtype:bad-unknown;)

# HTTP POST with large body to uncommon TLD
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"CLOUD EXFIL - Large HTTP POST to Uncommon TLD"; flow:established,to_server; http.method; content:"POST"; http.host; pcre:"/\.(xyz|top|club|work|buzz|tk|ml|ga|cf)\//i"; http.content_len; content:"1"; byte_test:6,>,100000,0,string; sid:9000011; rev:1; classtype:bad-unknown;)

# --- Cryptomining Detection ---
# Stratum mining protocol
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"CLOUD CRYPTO - Stratum Mining Protocol"; flow:established,to_server; content:"{"; content:"\"method\""; distance:0; content:"\"mining."; distance:0; sid:9000020; rev:1; classtype:policy-violation;)

# Known mining pool domains via TLS SNI
alert tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"CLOUD CRYPTO - Connection to Mining Pool"; tls.sni; pcre:"/(pool\.|mine\.|mining\.|xmr\.|monero|nicehash|nanopool|ethermine|f2pool)/i"; sid:9000021; rev:1; classtype:policy-violation;)

# --- Lateral Movement Detection ---
# SMB traffic between unexpected subnets
alert tcp $HOME_NET any -> $HOME_NET 445 (msg:"CLOUD LATERAL - SMB Traffic Between Subnets"; flow:established,to_server; threshold:type both, track by_src, count 5, seconds 10; sid:9000030; rev:1; classtype:bad-unknown;)

# RDP brute force detection
alert tcp $EXTERNAL_NET any -> $HOME_NET 3389 (msg:"CLOUD LATERAL - Possible RDP Brute Force"; flow:established,to_server; threshold:type both, track by_src, count 10, seconds 60; sid:9000031; rev:1; classtype:attempted-admin;)

# --- IMDS Abuse Detection ---
# Detect attempts to reach EC2 metadata service from unexpected sources
alert http any any -> 169.254.169.254 any (msg:"CLOUD SSRF - EC2 Instance Metadata Access Attempt"; flow:established,to_server; http.uri; content:"/latest/meta-data"; sid:9000040; rev:1; classtype:web-application-attack;)

# IMDSv1 token theft attempt
alert http any any -> 169.254.169.254 any (msg:"CLOUD SSRF - IMDSv1 Credential Access"; flow:established,to_server; http.uri; content:"/latest/meta-data/iam/security-credentials"; sid:9000041; rev:1; classtype:web-application-attack;)

AWS CLI for Rule Group Management

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Create a rule group from a local rules file
aws network-firewall create-rule-group \
  --rule-group-name "et-open-malware" \
  --type STATEFUL \
  --capacity 5000 \
  --rules "$(cat rules/emerging-malware-filtered.rules)" \
  --tags Key=Source,Value=emerging-threats-open \
  --region us-east-1

# Update an existing rule group with new rules
aws network-firewall update-rule-group \
  --rule-group-name "et-open-malware" \
  --type STATEFUL \
  --rules "$(cat rules/emerging-malware-filtered.rules)" \
  --update-token "$(aws network-firewall describe-rule-group \
    --rule-group-name et-open-malware \
    --type STATEFUL \
    --query 'UpdateToken' --output text)" \
  --region us-east-1

# List all rule groups
aws network-firewall list-rule-groups --type STATEFUL --region us-east-1

# Describe a specific rule group to check capacity usage
aws network-firewall describe-rule-group \
  --rule-group-name "et-open-malware" \
  --type STATEFUL \
  --query '{Name: RuleGroupResponse.RuleGroupName, Capacity: RuleGroupResponse.Capacity, NumberOfAssociations: RuleGroupResponse.NumberOfAssociations}' \
  --region us-east-1

Rule Management Automation — Lambda for ET Open Updates

Manual rule updates do not scale. Here is a Lambda function that automatically downloads the latest ET Open rules, filters them for AWS Network Firewall compatibility, and updates your rule groups daily.

Lambda Function: Automated ET Open Rule Updater

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
"""
ET Open Rule Updater for AWS Network Firewall
Automatically downloads, filters, and deploys Suricata rules from Emerging Threats.
"""
import boto3
import io
import json
import logging
import os
import re
import tarfile
import urllib.request

logger = logging.getLogger()
logger.setLevel(logging.INFO)

NFW_CLIENT = boto3.client("network-firewall")

# Configuration from environment variables
SURICATA_VERSION = os.environ.get("SURICATA_VERSION", "7.0")
RULE_GROUP_MAPPING = json.loads(os.environ.get("RULE_GROUP_MAPPING", "{}"))
HOME_NET_CIDR = os.environ.get("HOME_NET_CIDR", "10.0.0.0/16")
MAX_RULES_PER_GROUP = int(os.environ.get("MAX_RULES_PER_GROUP", "5000"))

# Keywords not supported by AWS Network Firewall
UNSUPPORTED_KEYWORDS = [
    "dataset:", "datarep:", "iprep", "lua:",
    "filestore", "fileext:", "filemagic:",
    "ftpdata_command:", "enip_command:",
]


def download_et_open() -> dict[str, str]:
    """Download and extract ET Open ruleset."""
    url = f"https://rules.emergingthreats.net/open/suricata-{SURICATA_VERSION}/emerging.rules.tar.gz"
    logger.info(f"Downloading ET Open from {url}")

    response = urllib.request.urlopen(url, timeout=30)
    tar_bytes = io.BytesIO(response.read())

    rules_by_file = {}
    with tarfile.open(fileobj=tar_bytes, mode="r:gz") as tar:
        for member in tar.getmembers():
            if member.name.endswith(".rules"):
                f = tar.extractfile(member)
                if f:
                    filename = os.path.basename(member.name)
                    rules_by_file[filename] = f.read().decode("utf-8")

    logger.info(f"Extracted {len(rules_by_file)} rule files")
    return rules_by_file


def filter_rules(rules_content: str) -> str:
    """Remove rules with unsupported keywords and comments."""
    filtered_lines = []
    total = 0
    removed = 0

    for line in rules_content.splitlines():
        stripped = line.strip()

        # Keep comments and empty lines
        if not stripped or stripped.startswith("#"):
            continue

        # Skip rules with unsupported keywords
        if any(kw in stripped for kw in UNSUPPORTED_KEYWORDS):
            removed += 1
            continue

        total += 1
        filtered_lines.append(stripped)

    logger.info(f"Filtered rules: {total} kept, {removed} removed (unsupported keywords)")
    return "\n".join(filtered_lines[:MAX_RULES_PER_GROUP])


def set_home_net(rules_content: str, cidr: str) -> str:
    """Replace $HOME_NET references with the actual CIDR if needed."""
    # Network Firewall handles HOME_NET via rule_variables, so we
    # leave $HOME_NET in the rules and set it in the rule group config.
    return rules_content


def update_rule_group(rule_group_name: str, rules_string: str) -> None:
    """Update an AWS Network Firewall rule group with new rules."""
    try:
        # Get current update token
        response = NFW_CLIENT.describe_rule_group(
            RuleGroupName=rule_group_name,
            Type="STATEFUL",
        )
        update_token = response["UpdateToken"]
        current_arn = response["RuleGroupResponse"]["RuleGroupArn"]

        # Build rule group configuration with HOME_NET
        rule_group = {
            "RuleVariables": {
                "IPSets": {
                    "HOME_NET": {
                        "Definition": [HOME_NET_CIDR],
                    },
                    "EXTERNAL_NET": {
                        "Definition": ["0.0.0.0/0"],
                    },
                },
            },
            "RulesSource": {
                "RulesString": rules_string,
            },
        }

        NFW_CLIENT.update_rule_group(
            UpdateToken=update_token,
            RuleGroupArn=current_arn,
            RuleGroup=rule_group,
            Type="STATEFUL",
        )
        logger.info(f"Successfully updated rule group: {rule_group_name}")

    except NFW_CLIENT.exceptions.ResourceNotFoundException:
        logger.error(f"Rule group not found: {rule_group_name}")
        raise
    except Exception as e:
        logger.error(f"Failed to update {rule_group_name}: {e}")
        raise


def handler(event, context):
    """Lambda handler - download ET Open rules and update Network Firewall."""
    logger.info("Starting ET Open rule update")

    # Download latest rules
    rules_by_file = download_et_open()

    # Update each mapped rule group
    # RULE_GROUP_MAPPING format: {"et-open-malware": ["emerging-malware.rules", "emerging-trojan.rules"]}
    results = {}
    for group_name, rule_files in RULE_GROUP_MAPPING.items():
        combined_rules = []
        for rule_file in rule_files:
            if rule_file in rules_by_file:
                filtered = filter_rules(rules_by_file[rule_file])
                combined_rules.append(filtered)
                logger.info(f"Added {rule_file} to {group_name}")
            else:
                logger.warning(f"Rule file not found in ET Open: {rule_file}")

        if combined_rules:
            rules_string = "\n".join(combined_rules)
            update_rule_group(group_name, rules_string)
            results[group_name] = "updated"
        else:
            results[group_name] = "no rules found"

    logger.info(f"Update complete: {json.dumps(results)}")
    return {"statusCode": 200, "body": results}

Terraform for the Lambda Deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
# lambda_rule_updater.tf

resource "aws_lambda_function" "et_rule_updater" {
  function_name = "et-open-rule-updater"
  runtime       = "python3.12"
  handler       = "et_rule_updater.handler"
  timeout       = 120
  memory_size   = 512

  filename         = data.archive_file.et_updater.output_path
  source_code_hash = data.archive_file.et_updater.output_base64sha256

  role = aws_iam_role.et_updater_role.arn

  environment {
    variables = {
      SURICATA_VERSION   = "7.0"
      HOME_NET_CIDR      = "10.0.0.0/16"
      MAX_RULES_PER_GROUP = "5000"
      RULE_GROUP_MAPPING = jsonencode({
        "et-open-malware" = ["emerging-malware.rules", "emerging-trojan.rules"]
        "et-open-exploit" = ["emerging-exploit.rules", "emerging-web_server.rules"]
        "et-open-policy"  = ["emerging-policy.rules", "emerging-dns.rules"]
      })
    }
  }

  tags = {
    Name        = "et-open-rule-updater"
    Customer    = "internal"
    Application = "network-security"
    Environment = "production"
    Owner       = "security-team"
    Costcenter  = "security"
  }
}

# Schedule daily updates at 6 AM UTC
resource "aws_cloudwatch_event_rule" "daily_rule_update" {
  name                = "et-open-daily-update"
  schedule_expression = "cron(0 6 ? * MON-FRI *)"
}

resource "aws_cloudwatch_event_target" "lambda_target" {
  rule      = aws_cloudwatch_event_rule.daily_rule_update.name
  target_id = "ETRuleUpdater"
  arn       = aws_lambda_function.et_rule_updater.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.et_rule_updater.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.daily_rule_update.arn
}

# IAM role for the Lambda function
resource "aws_iam_role" "et_updater_role" {
  name = "et-rule-updater-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

resource "aws_iam_role_policy" "et_updater_policy" {
  name = "et-rule-updater-policy"
  role = aws_iam_role.et_updater_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "network-firewall:DescribeRuleGroup",
          "network-firewall:UpdateRuleGroup",
        ]
        Resource = "arn:aws:network-firewall:*:*:stateful-rulegroup/*"
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
        ]
        Resource = "arn:aws:logs:*:*:*"
      },
    ]
  })
}

VPC Flow Logs Integration for Network Visibility

VPC Flow Logs complement Network Firewall by providing connection-level metadata for traffic that the firewall inspects. Together, they give you both deep packet inspection (Network Firewall) and flow-level visibility (Flow Logs).

Enable VPC Flow Logs to CloudWatch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
# flow_logs.tf
resource "aws_flow_log" "vpc_flow_log" {
  vpc_id               = aws_vpc.protected.id
  traffic_type         = "ALL"
  log_destination_type = "cloud-watch-logs"
  log_destination      = aws_cloudwatch_log_group.flow_logs.arn
  iam_role_arn         = aws_iam_role.flow_log_role.arn
  max_aggregation_interval = 60

  tags = {
    Name = "vpc-flow-logs"
  }
}

resource "aws_cloudwatch_log_group" "flow_logs" {
  name              = "/aws/vpc/flow-logs/protected-vpc"
  retention_in_days = 30
}

# Network Firewall alert logging
resource "aws_networkfirewall_logging_configuration" "main" {
  firewall_arn = aws_networkfirewall_firewall.main.arn

  logging_configuration {
    log_destination_config {
      log_destination = {
        logGroup = aws_cloudwatch_log_group.nfw_alerts.name
      }
      log_destination_type = "CloudWatchLogs"
      log_type             = "ALERT"
    }

    log_destination_config {
      log_destination = {
        logGroup = aws_cloudwatch_log_group.nfw_flow.name
      }
      log_destination_type = "CloudWatchLogs"
      log_type             = "FLOW"
    }
  }
}

resource "aws_cloudwatch_log_group" "nfw_alerts" {
  name              = "/aws/network-firewall/alerts"
  retention_in_days = 90
}

resource "aws_cloudwatch_log_group" "nfw_flow" {
  name              = "/aws/network-firewall/flow"
  retention_in_days = 30
}

Comparison: Managed Rules vs Custom Suricata Rules

Recommendation: Use both. AWS managed rules provide a baseline with zero effort. Layer custom ET Open rules on top for depth, transparency, and threat-specific detection that managed rules miss.

Detection Scenarios

Scenario 1: Command-and-Control Communication

Modern C2 frameworks like Cobalt Strike, Sliver, and Mythic use HTTPS with domain fronting, malleable C2 profiles, and encrypted channels. ET Open tracks hundreds of C2 framework signatures:

1
2
3
4
5
6
7
8
9
# ET Open rules detect known C2 patterns
# emerging-malware.rules contains signatures for:
# - Cobalt Strike beacon HTTP/HTTPS patterns
# - Sliver implant communication
# - Metasploit Meterpreter sessions
# - Brute Ratel C4 callbacks

# Custom rule: detect potential domain fronting
alert tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"CLOUD C2 - Possible Domain Fronting (SNI/Host Mismatch Indicator)"; flow:established,to_server; tls.sni; content:".cloudfront.net"; sid:9000050; rev:1; classtype:trojan-activity;)

Scenario 2: Data Exfiltration

1
2
3
4
5
# Detect large outbound transfers to cloud storage providers
alert tls $HOME_NET any -> $EXTERNAL_NET 443 (msg:"CLOUD EXFIL - Large Transfer to External Cloud Storage"; tls.sni; pcre:"/(dropbox\.com|drive\.google\.com|onedrive\.live\.com|mega\.nz|transfer\.sh)/i"; flow:established,to_server; threshold:type both, track by_src, count 20, seconds 300; sid:9000060; rev:1; classtype:policy-violation;)

# Detect base64-encoded data in DNS queries (DNS exfiltration)
alert dns $HOME_NET any -> any any (msg:"CLOUD EXFIL - Possible Base64 in DNS Query"; dns.query; pcre:"/^[a-zA-Z0-9+\/]{30,}\./"; threshold:type both, track by_src, count 5, seconds 60; sid:9000061; rev:1; classtype:bad-unknown;)

Scenario 3: Lateral Movement

1
2
3
4
5
# Detect port scanning behavior within VPC
alert tcp $HOME_NET any -> $HOME_NET any (msg:"CLOUD LATERAL - Internal Port Scan Detected"; flags:S,12; threshold:type both, track by_src, count 25, seconds 10; sid:9000070; rev:1; classtype:attempted-recon;)

# Detect WinRM/PSRemoting lateral movement
alert tcp $HOME_NET any -> $HOME_NET 5985 (msg:"CLOUD LATERAL - WinRM HTTP Connection"; flow:established,to_server; content:"POST"; content:"wsman"; distance:0; within:50; sid:9000071; rev:1; classtype:bad-unknown;)

Scenario 4: Cryptomining

1
2
3
4
5
6
# Stratum protocol variants
alert tcp $HOME_NET any -> any any (msg:"CLOUD CRYPTO - XMRig Mining Communication"; flow:established,to_server; content:"mining.submit"; sid:9000080; rev:1; classtype:policy-violation;)

# Detect connections to known mining pool IPs via ET Open
# emerging-policy.rules contains extensive mining pool coverage
# These rules are automatically imported by the Lambda updater

Performance Considerations and Rule Optimization

AWS Network Firewall supports up to 30,000 rules per policy, but loading every ET Open rule blindly wastes capacity and increases inspection latency. Optimize your deployment with these practices:

1. Filter Rules by Relevance

Not every ET Open category applies to your environment. If you run no Windows workloads, skip emerging-activex.rules and Windows-specific exploit rules. If you have no FTP services, skip FTP rules.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Rule filtering by environment profile
LINUX_CLOUD_PROFILE = [
    "emerging-malware.rules",
    "emerging-trojan.rules",
    "emerging-exploit.rules",
    "emerging-dns.rules",
    "emerging-policy.rules",
    "emerging-hunting.rules",
    "emerging-web_server.rules",
    "emerging-web_client.rules",
]

# Skip these for Linux-only cloud workloads
SKIP_FOR_LINUX = [
    "emerging-activex.rules",
    "emerging-netbios.rules",
    "emerging-shellcode.rules",  # x86-specific shellcode
]

2. Use Rule Priorities and Fast Pattern Matching

Place high-confidence, low-false-positive rules first in strict evaluation order. Suricata uses the fast_pattern keyword to optimize multi-pattern matching:

1
2
# Use fast_pattern on the most unique content match
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"Malware Callback"; flow:established,to_server; http.uri; content:"/gate.php"; fast_pattern; content:"bot_id="; sid:9000090; rev:1;)

3. Capacity Planning

4. Rule Testing and Validation

Before deploying rules to production, validate them with this script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
"""
Validate Suricata rules for AWS Network Firewall compatibility.
Checks for unsupported keywords and syntax issues.
"""
import re
import sys

UNSUPPORTED_KEYWORDS = [
    "dataset:", "datarep:", "iprep", "lua:",
    "filestore", "fileext:", "filemagic:",
    "ftpdata_command:", "enip_command:",
]

VALID_ACTIONS = {"alert", "pass", "drop", "reject"}

SID_PATTERN = re.compile(r"sid:\s*(\d+)")


def validate_rule(line: str, line_num: int) -> list[str]:
    """Validate a single Suricata rule line."""
    errors = []

    # Check action
    action = line.split()[0] if line.split() else ""
    if action not in VALID_ACTIONS:
        errors.append(f"Line {line_num}: Invalid action '{action}'")

    # Check for unsupported keywords
    for kw in UNSUPPORTED_KEYWORDS:
        if kw in line:
            errors.append(f"Line {line_num}: Unsupported keyword '{kw}'")

    # Check for SID
    if not SID_PATTERN.search(line):
        errors.append(f"Line {line_num}: Missing sid")

    # Check for balanced parentheses
    if line.count("(") != line.count(")"):
        errors.append(f"Line {line_num}: Unbalanced parentheses")

    return errors


def validate_file(filepath: str) -> tuple[int, int, list[str]]:
    """Validate all rules in a file."""
    all_errors = []
    total_rules = 0
    sids_seen = set()

    with open(filepath) as f:
        for line_num, line in enumerate(f, 1):
            stripped = line.strip()
            if not stripped or stripped.startswith("#"):
                continue

            total_rules += 1
            errors = validate_rule(stripped, line_num)
            all_errors.extend(errors)

            # Check for duplicate SIDs
            sid_match = SID_PATTERN.search(stripped)
            if sid_match:
                sid = sid_match.group(1)
                if sid in sids_seen:
                    all_errors.append(f"Line {line_num}: Duplicate SID {sid}")
                sids_seen.add(sid)

    return total_rules, len(all_errors), all_errors


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python validate_rules.py <rules_file>")
        sys.exit(1)

    filepath = sys.argv[1]
    total, error_count, errors = validate_file(filepath)

    print(f"Validated {total} rules from {filepath}")
    if errors:
        print(f"Found {error_count} errors:")
        for err in errors:
            print(f"  - {err}")
        sys.exit(1)
    else:
        print("All rules passed validation.")
        sys.exit(0)

Monitoring and Alerting on Network Firewall Findings

Network Firewall alert logs contain Suricata’s alert output including the rule message, SID, classification, and flow metadata. Set up CloudWatch alarms to surface high-priority detections.

CloudWatch Metric Filter and Alarm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# monitoring.tf - CloudWatch alerting for Network Firewall

# Metric filter for high-severity alerts
resource "aws_cloudwatch_log_metric_filter" "nfw_high_severity" {
  name           = "nfw-high-severity-alerts"
  pattern        = "{ $.event.alert.severity <= 2 }"
  log_group_name = aws_cloudwatch_log_group.nfw_alerts.name

  metric_transformation {
    name          = "HighSeverityAlerts"
    namespace     = "NetworkFirewall"
    value         = "1"
    default_value = "0"
  }
}

# Alarm for high-severity alerts
resource "aws_cloudwatch_metric_alarm" "nfw_critical" {
  alarm_name          = "network-firewall-critical-alert"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "HighSeverityAlerts"
  namespace           = "NetworkFirewall"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Network Firewall detected high-severity threat"

  alarm_actions = [
    "arn:aws:sns:us-east-1:181303648587:alert-critical"
  ]
}

# Metric filter for cryptomining detection
resource "aws_cloudwatch_log_metric_filter" "nfw_cryptomining" {
  name           = "nfw-cryptomining-alerts"
  pattern        = "{ $.event.alert.signature_id >= 9000020 && $.event.alert.signature_id <= 9000029 }"
  log_group_name = aws_cloudwatch_log_group.nfw_alerts.name

  metric_transformation {
    name          = "CryptominingAlerts"
    namespace     = "NetworkFirewall"
    value         = "1"
    default_value = "0"
  }
}

resource "aws_cloudwatch_metric_alarm" "nfw_cryptomining" {
  alarm_name          = "network-firewall-cryptomining-detected"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "CryptominingAlerts"
  namespace           = "NetworkFirewall"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Network Firewall detected cryptomining activity"

  alarm_actions = [
    "arn:aws:sns:us-east-1:181303648587:alert-critical"
  ]
}

# Metric filter for data exfiltration patterns
resource "aws_cloudwatch_log_metric_filter" "nfw_exfiltration" {
  name           = "nfw-exfiltration-alerts"
  pattern        = "{ $.event.alert.signature_id >= 9000010 && $.event.alert.signature_id <= 9000019 }"
  log_group_name = aws_cloudwatch_log_group.nfw_alerts.name

  metric_transformation {
    name          = "ExfiltrationAlerts"
    namespace     = "NetworkFirewall"
    value         = "1"
    default_value = "0"
  }
}

resource "aws_cloudwatch_metric_alarm" "nfw_exfiltration" {
  alarm_name          = "network-firewall-exfiltration-detected"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ExfiltrationAlerts"
  namespace           = "NetworkFirewall"
  period              = 300
  statistic           = "Sum"
  threshold           = 0
  alarm_description   = "Network Firewall detected possible data exfiltration"

  alarm_actions = [
    "arn:aws:sns:us-east-1:181303648587:alert-critical"
  ]
}

Querying Network Firewall Alerts

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Query recent high-severity alerts from CloudWatch Logs
aws logs filter-log-events \
  --log-group-name "/aws/network-firewall/alerts" \
  --start-time $(date -d '1 hour ago' +%s000) \
  --filter-pattern '{ $.event.alert.severity <= 2 }' \
  --query 'events[*].message' \
  --output text | jq -r '.event.alert | "\(.signature_id) | \(.signature) | \(.severity) | \(.src_ip) -> \(.dest_ip)"'

# Count alerts by signature over the last 24 hours
aws logs start-query \
  --log-group-name "/aws/network-firewall/alerts" \
  --start-time $(date -d '24 hours ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'stats count(*) as alert_count by event.alert.signature | sort alert_count desc | limit 20'

Implementation Roadmap

Phase 1: Foundation (Week 1-2)

• Deploy Network Firewall in inspection architecture with Terraform
• Enable alert and flow logging to CloudWatch Logs
• Import AWS managed threat signature rule groups
• Validate traffic routing through firewall endpoints

Phase 2: Custom Rules (Week 3-4)

• Download and filter ET Open ruleset for your environment
• Create custom rule groups for malware, exploit, and policy categories
• Write environment-specific rules (IMDS protection, cloud API abuse)
• Validate all rules with the compatibility checker script
• Deploy in alert-only mode (no drop actions)

Phase 3: Automation (Week 5-6)

• Deploy the Lambda rule updater with EventBridge schedule
• Set up CloudWatch metric filters and alarms for critical signatures
• Establish a baseline of normal alert volume
• Begin tuning false positives with threshold and suppression rules

Phase 4: Active Protection (Week 7-8)

• Promote high-confidence rules from alert to drop action
• Integrate alerts with your SIEM or Security Hub
• Document runbooks for each alert category
• Schedule quarterly rule review and optimization

Related Articles

• AWS CloudFront Geographic Access Control — Restrict content delivery by geography using CloudFront and WAF
• AWS WAF and CloudFront Application Protection — Layer 7 protection with managed and custom WAF rules
• AWS Cloud Security Best Practices Implementation Guide — Comprehensive security baseline for AWS environments

Conclusion

AWS Network Firewall gives you a managed Suricata engine at VPC scale. The managed rule groups provide a starting point, but the real power comes from the open-source ecosystem that Suricata was built on. The Emerging Threats Open ruleset delivers tens of thousands of community-maintained detection signatures — updated daily, fully transparent, and free. By combining managed rules for baseline coverage with custom ET Open rules for depth, and automating updates with a simple Lambda function, you build a network detection capability that rivals expensive commercial IDPS solutions.

The approach is straightforward: AWS provides the managed infrastructure, the open-source community provides the detection intelligence, and you maintain full control over what runs in your environment. That is the build-it-yourself advantage.

Want to discuss network security architecture for your AWS environment? Connect with me on LinkedIn to continue the conversation.

Introduction

Security teams have an observability problem. While DevOps engineers have spent years building sophisticated monitoring stacks for uptime and performance, security telemetry remains fragmented across disconnected tools, vendor-locked dashboards, and siloed alert streams. The result is a Mean Time to Detect (MTTD) that still hovers around 204 days for breach discovery and a Mean Time to Respond (MTTR) measured in weeks rather than minutes.

The observability gap in security is not a tooling problem – it is an architecture problem. Traditional security monitoring treats logs, metrics, and traces as separate concerns. A WAF logs blocked requests in one system. Authentication failures appear in another. API latency anomalies surface in a third. Correlating these signals to detect a coordinated attack requires manual investigation across multiple consoles, by which point the attacker has already moved laterally.

OpenTelemetry changes this equation. As a vendor-neutral, CNCF-graduated observability framework, it provides a single instrumentation layer that unifies traces, metrics, and logs into correlated telemetry. AWS recognized this shift so strongly that they built their own distribution – AWS Distro for OpenTelemetry (ADOT) – and are actively deprecating the legacy X-Ray SDKs in its favor.

This article demonstrates how to build a security observability pipeline using ADOT as the collection layer, with the freedom to export to CloudWatch for native integration or Grafana and Prometheus for vendor-neutral dashboards. The approach follows the red-team.sh philosophy: start with AWS-native, then layer open source for maximum flexibility and zero lock-in.

Current Landscape Statistics

• Breach Detection Time: Organizations take an average of 204 days to identify a breach and 73 days to contain it (IBM Cost of a Data Breach 2025)
• Observability Adoption: 85% of organizations now consider observability critical to their security posture, up from 62% in 2023
• OpenTelemetry Adoption: OTel is the second-most active CNCF project after Kubernetes, with contributions from AWS, Google, Microsoft, and 1,000+ organizations
• Cost Impact: Organizations with mature security observability reduce breach costs by an average of $1.76 million compared to those without
• ADOT Transition: AWS X-Ray SDKs entered maintenance mode on February 25, 2026, making ADOT the official path forward for distributed tracing on AWS

Security Observability vs Traditional Monitoring

Before diving into implementation, it is important to understand what separates security observability from the monitoring you already have in place. Traditional monitoring asks “is it up?” – security observability asks “is it compromised?”

Traditional Monitoring

Traditional monitoring focuses on availability and performance:

• Uptime checks: Is the service responding?
• Resource metrics: CPU, memory, disk, network utilization
• Error rates: HTTP 5xx counts, exception frequency
• Latency percentiles: p50, p95, p99 response times
• Throughput: Requests per second, transactions per minute

This data tells you whether your systems are healthy. It does not tell you whether an attacker is exfiltrating data through a series of seemingly normal API calls.

Security Observability

Security observability extends monitoring with threat-aware context:

• Authentication telemetry: Failed login patterns, credential stuffing velocity, impossible travel detection
• Authorization anomalies: Privilege escalation attempts, unusual role assumptions, cross-account access patterns
• API behavior analysis: Endpoint enumeration, parameter fuzzing, rate limit probing, unusual payload sizes
• Data flow tracing: Which identities accessed which data, through which services, at what volume
• Infrastructure mutation tracking: Security group changes, IAM policy modifications, encryption state changes

The key difference is correlation. A single failed login is noise. Fifty failed logins from rotating IPs against the same account, followed by a successful login from a new geography, followed by an API call to list all S3 buckets – that is a signal. Security observability makes this correlation possible by unifying all three telemetry pillars (traces, metrics, logs) under a common context.

AWS CloudWatch and X-Ray: Strong but AWS-Locked

AWS provides a comprehensive native observability stack. Before reaching for open source, it is worth understanding what CloudWatch and X-Ray deliver – and where they fall short.

What CloudWatch Does Well

• Zero-config integration with 70+ AWS services
• Logs Insights query language for ad-hoc investigation
• Contributor Insights for identifying top-N patterns (top IPs, top error codes)
• Anomaly Detection using machine learning on metric streams
• Cross-account observability through CloudWatch Organization sinks
• Metric Math for derived security metrics (error rate ratios, authentication failure velocity)

What X-Ray Does Well

• Distributed tracing across Lambda, ECS, EKS, and API Gateway
• Service maps that visualize request flow and identify bottlenecks
• Trace sampling that reduces cost while maintaining visibility
• Insights for automated anomaly detection on trace data

Where Native Falls Short

Vendor lock-in: CloudWatch metrics, logs, and X-Ray traces use proprietary formats. Moving to another cloud or a hybrid environment means rebuilding your entire observability pipeline.

Cost at scale: CloudWatch pricing compounds quickly. At $0.30 per custom metric per month, $0.50 per GB for log ingestion, and $0.01 per 1,000 GetMetricData API calls, a medium-sized security monitoring deployment can easily exceed $2,000/month in CloudWatch costs alone.

Limited correlation: While CloudWatch ServiceLens attempts to unify metrics, logs, and traces, the experience remains fragmented compared to a purpose-built observability platform like Grafana.

Dashboard limitations: CloudWatch dashboards lack the flexibility of Grafana for building complex security visualization panels with template variables, annotations, and cross-datasource correlation.

Retention costs: Long-term metric and log retention in CloudWatch is expensive. Security compliance often requires 1-7 years of retention, which can become a significant cost driver.

OpenTelemetry: Traces, Metrics, and Logs Unified

OpenTelemetry (OTel) is the vendor-neutral answer to the observability fragmentation problem. It provides a single set of APIs, SDKs, and protocols for generating, collecting, and exporting telemetry data.

The Three Pillars

Traces follow a request as it flows through distributed services. Each trace contains spans representing individual operations, with attributes that carry security-relevant context like user identity, source IP, and authorization decisions.

Metrics are numerical measurements collected at regular intervals. OTel supports counters, histograms, and gauges – the building blocks for security metrics like authentication failure rates, API abuse scores, and data transfer volumes.

Logs capture discrete events with structured attributes. OTel’s log data model standardizes log records across languages and frameworks, making it possible to correlate a log entry with the trace that generated it.

Why OTel Matters for Security

The real power of OpenTelemetry for security is context propagation. When a request enters your system, OTel generates a trace ID that follows it through every service, function, and database call. By enriching spans with security attributes (user ID, IP address, authentication method, authorization scope), you create a complete audit trail that can be queried, correlated, and alerted on.

This is fundamentally different from scraping CloudWatch logs after the fact. With OTel instrumentation, the security context is embedded in the telemetry at generation time, not reconstructed during investigation.

OTel Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌─────────────────────────────────────────────────────┐
│                  Application Code                    │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │  Tracer   │  │  Meter   │  │  Logger           │  │
│  │  Provider │  │  Provider│  │  Provider         │  │
│  └─────┬────┘  └─────┬────┘  └────────┬─────────┘  │
│        │             │                │             │
│  ┌─────▼─────────────▼────────────────▼──────────┐  │
│  │            OTel SDK (Auto + Manual)            │  │
│  └──────────────────┬────────────────────────────┘  │
│                     │  OTLP Export                   │
└─────────────────────┼───────────────────────────────┘
                      │
              ┌───────▼───────┐
              │ ADOT Collector │
              │  (Receivers,   │
              │  Processors,   │
              │  Exporters)    │
              └───┬───┬───┬───┘
                  │   │   │
          ┌───────┘   │   └────────┐
          ▼           ▼            ▼
    CloudWatch   Prometheus    Jaeger
     + X-Ray     + Grafana

ADOT: AWS’s Own OpenTelemetry Distribution

AWS Distro for OpenTelemetry (ADOT) is AWS’s supported, production-ready distribution of the OpenTelemetry Collector and SDKs. It is not a fork – it is upstream OTel with AWS-specific receivers, processors, and exporters baked in.

Why ADOT Over Upstream OTel

• AWS-specific exporters: Native exporters for CloudWatch EMF (Embedded Metric Format), X-Ray traces, and CloudWatch Logs
• AWS resource detection: Automatic enrichment with EC2 instance metadata, EKS cluster info, Lambda function context
• Managed add-ons: Available as an EKS add-on, Lambda layer, and ECS sidecar with minimal configuration
• AWS support: Covered under AWS support plans, unlike community OTel builds
• Security patches: AWS backports security fixes and maintains release cadence aligned with upstream

ADOT Collector Components

The ADOT Collector follows the standard OTel Collector architecture:

Receivers ingest telemetry from applications and infrastructure:

• otlp – Standard OTLP receiver for traces, metrics, and logs
• awsxray – Receives X-Ray formatted segments
• prometheus – Scrapes Prometheus-format metrics endpoints
• statsd – Receives StatsD metrics

Processors transform and enrich telemetry:

• batch – Batches data before export for efficiency
• resourcedetection – Auto-detects AWS resource attributes (region, account, instance)
• filter – Drops or keeps telemetry based on attribute conditions
• attributes – Adds, modifies, or removes span/metric attributes

Exporters send telemetry to backends:

• awsxray – Exports traces to AWS X-Ray
• awsemf – Exports metrics as CloudWatch Embedded Metric Format logs
• awscloudwatchlogs – Exports logs to CloudWatch Logs
• prometheusremotewrite – Exports to Prometheus-compatible backends (including Amazon Managed Prometheus)
• otlp – Exports to any OTLP-compatible backend (Grafana Cloud, Jaeger, etc.)

ADOT Collector routing telemetry to both AWS-native and OSS backends

Security-Specific Instrumentation

Generic observability instrumentation captures HTTP status codes and latency. Security observability requires intentional enrichment with threat-relevant attributes. Here is what to instrument and why.

Authentication Events

Every authentication attempt should generate a span with security attributes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
from opentelemetry import trace
from opentelemetry.trace import StatusCode
import hashlib

tracer = trace.get_tracer("auth-service")

def authenticate_user(request):
    with tracer.start_as_current_span("auth.authenticate") as span:
        # Security-relevant attributes
        span.set_attribute("auth.method", request.auth_method)  # "password", "mfa", "sso"
        span.set_attribute("auth.username", request.username)
        span.set_attribute("net.peer.ip", request.client_ip)
        span.set_attribute("http.user_agent", request.user_agent)
        span.set_attribute("auth.geo.country", geoip_lookup(request.client_ip))

        try:
            result = verify_credentials(request)
            span.set_attribute("auth.result", "success")
            span.set_attribute("auth.user_id", result.user_id)
            span.set_attribute("auth.roles", ",".join(result.roles))
            span.set_status(StatusCode.OK)
            return result

        except AuthenticationError as e:
            span.set_attribute("auth.result", "failure")
            span.set_attribute("auth.failure_reason", str(e))
            span.set_status(StatusCode.ERROR, str(e))

            # Increment security metric
            auth_failure_counter.add(1, {
                "auth.method": request.auth_method,
                "auth.failure_reason": str(e),
                "net.peer.ip": request.client_ip
            })
            raise

API Access Patterns

Track API access with enough detail to detect enumeration and abuse:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
from opentelemetry import trace, metrics

meter = metrics.get_meter("api-security")
api_request_counter = meter.create_counter(
    "security.api.requests",
    description="API requests with security context"
)
api_payload_histogram = meter.create_histogram(
    "security.api.payload_size",
    description="Request payload sizes for anomaly detection",
    unit="bytes"
)

def security_middleware(request, call_next):
    span = trace.get_current_span()

    # Enrich with security context
    span.set_attribute("security.api.endpoint", request.path)
    span.set_attribute("security.api.method", request.method)
    span.set_attribute("security.api.authenticated", bool(request.user))
    span.set_attribute("security.api.user_id", getattr(request.user, "id", "anonymous"))
    span.set_attribute("security.api.source_ip", request.client.host)
    span.set_attribute("security.api.request_size", len(request.body or b""))

    # Track payload sizes for data exfiltration detection
    api_payload_histogram.record(
        len(request.body or b""),
        {"endpoint": request.path, "method": request.method}
    )

    response = call_next(request)

    span.set_attribute("security.api.response_size", response.content_length or 0)
    span.set_attribute("security.api.status_code", response.status_code)

    # Record security metric
    api_request_counter.add(1, {
        "endpoint": request.path,
        "method": request.method,
        "status": str(response.status_code),
        "authenticated": str(bool(request.user))
    })

    return response

IAM and Authorization Events

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def check_authorization(user, resource, action):
    with tracer.start_as_current_span("authz.check") as span:
        span.set_attribute("authz.user_id", user.id)
        span.set_attribute("authz.resource", resource)
        span.set_attribute("authz.action", action)
        span.set_attribute("authz.roles", ",".join(user.roles))

        allowed = policy_engine.evaluate(user, resource, action)

        span.set_attribute("authz.decision", "allow" if allowed else "deny")

        if not allowed:
            authz_denial_counter.add(1, {
                "user_id": user.id,
                "resource": resource,
                "action": action
            })

        return allowed

Deploying the ADOT Collector on EKS and Lambda

ADOT Collector Configuration for Security Telemetry

The following ADOT Collector configuration receives OTLP telemetry, enriches it with AWS resource attributes, and dual-exports to both CloudWatch/X-Ray and Prometheus/Grafana:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
# adot-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  # Scrape Prometheus metrics from application security endpoints
  prometheus:
    config:
      scrape_configs:
        - job_name: "security-metrics"
          scrape_interval: 15s
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

  # Auto-detect AWS resource attributes
  resourcedetection:
    detectors: [eks, ec2, env]
    timeout: 5s
    override: false

  # Add security classification attributes
  attributes/security:
    actions:
      - key: telemetry.domain
        value: security
        action: upsert
      - key: environment
        from_attribute: ENVIRONMENT
        action: upsert

  # Filter to separate security telemetry from general telemetry
  filter/security_traces:
    traces:
      span:
        - 'attributes["security.api.endpoint"] != nil'
        - 'name == "auth.authenticate"'
        - 'name == "authz.check"'

  # Memory limiter to prevent OOM
  memory_limiter:
    check_interval: 5s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  # AWS X-Ray for distributed tracing
  awsxray:
    region: us-east-1
    indexed_attributes:
      - "auth.result"
      - "auth.username"
      - "security.api.endpoint"

  # CloudWatch EMF for metrics
  awsemf:
    region: us-east-1
    namespace: "SecurityObservability"
    log_group_name: "/security/otel/metrics"
    dimension_rollup_option: "NoDimensionRollup"
    metric_declarations:
      - dimensions:
          - ["auth.method", "auth.result"]
          - ["security.api.endpoint", "security.api.method"]
        metric_name_selectors:
          - "security.*"

  # Amazon Managed Prometheus for long-term metric storage
  prometheusremotewrite:
    endpoint: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-XXXX/api/v1/remote_write"
    auth:
      authenticator: sigv4auth
    resource_to_telemetry_conversion:
      enabled: true

  # OTLP export to Grafana for unified dashboards
  otlp/grafana:
    endpoint: "grafana-agent.monitoring.svc.cluster.local:4317"
    tls:
      insecure: false
      ca_file: /etc/ssl/certs/ca-certificates.crt

  # CloudWatch Logs for security event logs
  awscloudwatchlogs:
    region: us-east-1
    log_group_name: "/security/otel/logs"
    log_stream_name: "security-events"

extensions:
  sigv4auth:
    region: us-east-1
    service: aps

  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [sigv4auth, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, attributes/security, batch]
      exporters: [awsxray, otlp/grafana]

    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, resourcedetection, attributes/security, batch]
      exporters: [awsemf, prometheusremotewrite]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, attributes/security, batch]
      exporters: [awscloudwatchlogs, otlp/grafana]

Kubernetes Deployment for ADOT Collector

Deploy the ADOT Collector as a DaemonSet on EKS for node-level collection, or as a Deployment for centralized collection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
# adot-collector-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: adot-security-collector
  namespace: monitoring
  labels:
    app: adot-security-collector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: adot-security-collector
  template:
    metadata:
      labels:
        app: adot-security-collector
    spec:
      serviceAccountName: adot-collector-sa
      containers:
        - name: adot-collector
          image: public.ecr.aws/aws-observability/aws-otel-collector:v0.47.0
          ports:
            - containerPort: 4317  # OTLP gRPC
              name: otlp-grpc
            - containerPort: 4318  # OTLP HTTP
              name: otlp-http
            - containerPort: 13133 # Health check
              name: health
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 1000m
              memory: 1Gi
          volumeMounts:
            - name: config
              mountPath: /etc/otel
          env:
            - name: ENVIRONMENT
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /
              port: health
            initialDelaySeconds: 15
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /
              port: health
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: config
          configMap:
            name: adot-security-config
---
apiVersion: v1
kind: Service
metadata:
  name: adot-security-collector
  namespace: monitoring
spec:
  selector:
    app: adot-security-collector
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
  type: ClusterIP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: adot-collector-sa
  namespace: monitoring
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/ADOTCollectorRole

Lambda Layer Configuration for ADOT

For Lambda-based workloads, ADOT ships as a managed Lambda layer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# template.yaml (AWS SAM)
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Runtime: python3.12
    Timeout: 30
    MemorySize: 512
    Tracing: Active
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
        OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml
        OTEL_SERVICE_NAME: security-api
        OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production"
    Layers:
      # ADOT Lambda Layer for Python
      - !Sub "arn:aws:lambda:${AWS::Region}:901920570463:layer:aws-otel-python-amd64-ver-1-25-0:1"

Resources:
  SecurityApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.handler
      CodeUri: src/
      Description: Security API with ADOT instrumentation
      Policies:
        - AWSXRayDaemonWriteAccess
        - Statement:
            - Effect: Allow
              Action:
                - "aps:RemoteWrite"
              Resource: "*"
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /api/{proxy+}
            Method: ANY

Grafana for Security Dashboards

Grafana transforms raw security telemetry into actionable dashboards. Whether you run self-hosted Grafana on ECS or use Amazon Managed Grafana (AMG), the dashboard definitions are identical.

Terraform for Grafana on ECS Fargate

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
# grafana.tf
resource "aws_ecs_cluster" "monitoring" {
  name = "security-monitoring"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }

  tags = {
    Customer    = "red-team"
    Application = "security-observability"
    Environment = "production"
    Owner       = "jon"
    Costcenter  = "security"
  }
}

resource "aws_ecs_task_definition" "grafana" {
  family                   = "grafana-security"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "1024"
  memory                   = "2048"
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = aws_iam_role.grafana_task.arn

  container_definitions = jsonencode([
    {
      name  = "grafana"
      image = "grafana/grafana-oss:12.0.0"
      portMappings = [
        {
          containerPort = 3000
          protocol      = "tcp"
        }
      ]
      environment = [
        { name = "GF_SECURITY_ADMIN_PASSWORD", value = "CHANGE_ME" },
        { name = "GF_INSTALL_PLUGINS", value = "grafana-clock-panel,grafana-piechart-panel" },
        { name = "GF_AUTH_ANONYMOUS_ENABLED", value = "false" },
        { name = "GF_SECURITY_ALLOW_EMBEDDING", value = "false" },
        { name = "GF_SECURITY_COOKIE_SECURE", value = "true" },
        { name = "GF_SECURITY_STRICT_TRANSPORT_SECURITY", value = "true" }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/grafana-security"
          "awslogs-region"        = "us-east-1"
          "awslogs-stream-prefix" = "grafana"
        }
      }
      mountPoints = [
        {
          sourceVolume  = "grafana-data"
          containerPath = "/var/lib/grafana"
        }
      ]
    }
  ])

  volume {
    name = "grafana-data"
    efs_volume_configuration {
      file_system_id = aws_efs_file_system.grafana.id
    }
  }

  tags = {
    Customer    = "red-team"
    Application = "security-observability"
    Environment = "production"
    Owner       = "jon"
    Costcenter  = "security"
  }
}

resource "aws_ecs_service" "grafana" {
  name            = "grafana-security"
  cluster         = aws_ecs_cluster.monitoring.id
  task_definition = aws_ecs_task_definition.grafana.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.grafana.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.grafana.arn
    container_name   = "grafana"
    container_port   = 3000
  }
}

resource "aws_efs_file_system" "grafana" {
  creation_token = "grafana-security-data"
  encrypted      = true

  tags = {
    Customer    = "red-team"
    Application = "security-observability"
    Environment = "production"
    Owner       = "jon"
    Costcenter  = "security"
  }
}

Grafana Dashboard JSON for Security Metrics

The following dashboard provides a security operations overview with authentication monitoring, API abuse detection, and authorization anomaly panels:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
{
  "dashboard": {
    "title": "Security Observability - OTel",
    "tags": ["security", "opentelemetry", "adot"],
    "timezone": "browser",
    "panels": [
      {
        "title": "Authentication Failures (15m rolling)",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(security_auth_failures_total[15m])) by (auth_method, failure_reason)",
            "legendFormat": " - "
          }
        ],
        "fieldConfig": {
          "defaults": {
            "custom": {
              "drawStyle": "line",
              "fillOpacity": 20,
              "lineWidth": 2
            },
            "thresholds": {
              "steps": [
                { "color": "green", "value": null },
                { "color": "yellow", "value": 10 },
                { "color": "red", "value": 50 }
              ]
            }
          }
        }
      },
      {
        "title": "API Request Anomalies",
        "type": "timeseries",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "sum(rate(security_api_requests_total{status=~\"4..\"}[5m])) by (endpoint)",
            "legendFormat": "4xx: "
          },
          {
            "expr": "histogram_quantile(0.99, rate(security_api_payload_size_bytes_bucket[5m]))",
            "legendFormat": "p99 payload size"
          }
        ]
      },
      {
        "title": "Authorization Denials by Resource",
        "type": "barchart",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
        "targets": [
          {
            "expr": "topk(10, sum(increase(security_authz_denials_total[1h])) by (resource, action))",
            "legendFormat": " / "
          }
        ]
      },
      {
        "title": "Top Source IPs by Failed Auth",
        "type": "table",
        "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
        "targets": [
          {
            "expr": "topk(20, sum(increase(security_auth_failures_total[1h])) by (source_ip))",
            "format": "table",
            "instant": true
          }
        ]
      },
      {
        "title": "Security Event Rate (All Types)",
        "type": "stat",
        "gridPos": { "h": 4, "w": 6, "x": 0, "y": 16 },
        "targets": [
          {
            "expr": "sum(rate(security_events_total[5m]))",
            "legendFormat": "events/sec"
          }
        ]
      },
      {
        "title": "Unique Source IPs (1h)",
        "type": "stat",
        "gridPos": { "h": 4, "w": 6, "x": 6, "y": 16 },
        "targets": [
          {
            "expr": "count(count by (source_ip)(security_api_requests_total))",
            "legendFormat": "unique IPs"
          }
        ]
      }
    ],
    "templating": {
      "list": [
        {
          "name": "environment",
          "type": "query",
          "query": "label_values(security_api_requests_total, environment)",
          "current": { "text": "production", "value": "production" }
        },
        {
          "name": "service",
          "type": "query",
          "query": "label_values(security_api_requests_total, service_name)"
        }
      ]
    },
    "time": { "from": "now-6h", "to": "now" },
    "refresh": "30s"
  }
}

Prometheus for Security Metrics Collection

Prometheus serves as the metrics backbone for security observability. When paired with ADOT’s prometheusremotewrite exporter and Amazon Managed Prometheus (AMP) for storage, you get durable, queryable security metrics without managing Prometheus infrastructure.

Prometheus Alerting Rules for Security Events

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
# security-alerts.yaml
groups:
  - name: authentication_security
    interval: 30s
    rules:
      # Credential stuffing detection
      - alert: HighAuthFailureRate
        expr: |
          sum(rate(security_auth_failures_total[5m])) > 10
        for: 2m
        labels:
          severity: warning
          team: security
        annotations:
          summary: "High authentication failure rate detected"
          description: "Authentication failures exceeding 10/sec for 2 minutes. Possible credential stuffing attack."
          runbook_url: "https://wiki.internal/runbooks/credential-stuffing"

      # Brute force detection per IP
      - alert: BruteForceAttempt
        expr: |
          sum(rate(security_auth_failures_total[5m])) by (source_ip) > 5
        for: 1m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "Brute force attempt from "
          description: "Single IP  generating >5 auth failures/sec."

      # Impossible travel detection
      - alert: ImpossibleTravel
        expr: |
          count(
            count by (user_id, geo_country)(
              security_auth_success_total
            )
          ) by (user_id) > 2
          and
          sum(rate(security_auth_success_total[30m])) by (user_id) > 0
        for: 1m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "Impossible travel detected for user "
          description: "User authenticating from multiple countries within 30 minutes."

  - name: api_security
    interval: 30s
    rules:
      # API enumeration detection
      - alert: APIEnumeration
        expr: |
          count(
            count by (source_ip, endpoint)(
              rate(security_api_requests_total{status="404"}[5m]) > 0
            )
          ) by (source_ip) > 20
        for: 2m
        labels:
          severity: warning
          team: security
        annotations:
          summary: "API enumeration detected from "
          description: "IP hitting >20 unique 404 endpoints in 5 minutes."

      # Data exfiltration detection
      - alert: AnomalousDataTransfer
        expr: |
          histogram_quantile(0.99, rate(security_api_payload_size_bytes_bucket[5m]))
          > 10 * histogram_quantile(0.99, rate(security_api_payload_size_bytes_bucket[1h]))
        for: 5m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "Anomalous data transfer detected"
          description: "Response payload sizes 10x above normal p99 baseline."

  - name: authorization_security
    interval: 30s
    rules:
      # Privilege escalation attempts
      - alert: PrivilegeEscalationAttempt
        expr: |
          sum(rate(security_authz_denials_total{action=~"admin.*|delete.*|modify_policy.*"}[5m])) by (user_id) > 3
        for: 1m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "Privilege escalation attempts by "
          description: "User repeatedly denied access to privileged operations."

Comparison: CloudWatch vs OpenTelemetry + Grafana

When to Use Each

Choose CloudWatch + X-Ray when:

• Your workloads are 100% AWS and will stay that way
• Your team lacks the capacity to manage observability infrastructure
• You need FedRAMP/HIPAA compliance out of the box
• Your telemetry volume is small enough that CloudWatch costs are reasonable

Choose ADOT + Grafana when:

• You operate in multi-cloud or hybrid environments
• You want vendor-neutral telemetry that survives a cloud migration
• CloudWatch costs are growing faster than your budget
• You need advanced visualization, correlation, and alerting
• Your security team requires custom dashboards beyond what CloudWatch offers

The sweet spot: Use ADOT as the collection layer and export to both. Send traces to X-Ray for AWS console integration and simultaneously to Grafana for advanced analysis. Send metrics to both CloudWatch (for native alarms) and Prometheus (for Grafana dashboards). This dual-export approach gives you the best of both worlds at minimal additional cost.

Security Use Cases

API Abuse Detection

API abuse often manifests as patterns invisible to traditional monitoring. By instrumenting with OTel, you can detect:

Credential stuffing: High-velocity authentication failures from distributed IP ranges. The authentication span attributes (auth.result, net.peer.ip, auth.method) feed Prometheus metrics that trigger alerts when failure rates exceed baseline.

Endpoint enumeration: Attackers probe for undocumented API endpoints. Tracking 404 responses by source IP and correlating with the total unique endpoints hit reveals scanning behavior that per-endpoint monitoring misses.

Rate limit evasion: Sophisticated attackers rotate through multiple API keys or source IPs to stay under per-client rate limits. By aggregating request patterns across all identifiers at the OTel Collector level, you can detect distributed abuse that no single rate limiter would catch.

Authentication Anomaly Detection

Authentication telemetry collected through OTel enables several detection patterns:

Impossible travel: When a user authenticates from New York and then from Tokyo 30 minutes later, the geographic attributes on authentication spans make this trivially detectable. The Prometheus alert rule shown earlier implements this pattern.

Credential reuse across accounts: By hashing and comparing authentication attempt patterns (same password hash across different usernames from the same IP), you can identify credential reuse attacks that would otherwise appear as unrelated failed logins.

Session anomalies: Track session creation, renewal, and usage patterns. A session that suddenly starts accessing resources outside its historical pattern may indicate session hijacking.

Latency-Based Threat Detection

This is where security observability truly differentiates from traditional monitoring. Certain attacks create subtle latency signatures:

SQL injection probing: Injected SQL payloads that trigger error-path processing often exhibit different latency profiles than normal queries. By tracking database span duration distributions and alerting on shifts in the p99, you can detect injection attempts before the WAF catches the payload.

Timing side channels: Authentication endpoints that take longer to reject valid usernames than invalid ones leak information. OTel trace data makes these timing differences visible and alertable.

Cryptomining detection: Compromised containers running cryptominers exhibit characteristic CPU utilization patterns that diverge from their historical baseline. Prometheus metrics collected via OTel surface these anomalies.

Building Security Alerting Pipelines with OTel

The complete alerting pipeline connects OTel instrumentation to incident response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Application Instrumentation (OTel SDK)
    │
    ▼
ADOT Collector (filter, enrich, batch)
    │
    ├──► Amazon Managed Prometheus (metrics storage)
    │        │
    │        ▼
    │    Prometheus AlertManager / Grafana Alerting
    │        │
    │        ├──► SNS Topic (arn:aws:sns:us-east-1:...:alert-critical)
    │        ├──► PagerDuty / Opsgenie
    │        └──► Slack #security-alerts
    │
    ├──► CloudWatch Logs (log storage)
    │        │
    │        ▼
    │    CloudWatch Alarm → Lambda → Security Hub Finding
    │
    └──► Grafana (visualization + investigation)
             │
             ▼
         Security Analyst Investigation Workflow

Connecting Grafana Alerting to AWS SNS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# grafana-alerting-contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: security-critical
    receivers:
      - uid: sns-critical
        type: sns
        settings:
          topic: "arn:aws:sns:us-east-1:181303648587:alert-critical"
          authProvider: default
        disableResolveMessage: false
      - uid: slack-security
        type: slack
        settings:
          url: "${SLACK_WEBHOOK_URL}"
          recipient: "#security-alerts"
          title: |
            [] 
          text: |
            
            *Summary*: 
            *Description*: 
            *Severity*: 
            

Lambda Function for Security Hub Integration

When a Prometheus alert fires, you can create a Security Hub finding to centralize security events:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
# security_hub_finding.py
import boto3
import json
from datetime import datetime

securityhub = boto3.client("securityhub")

def handler(event, context):
    """Convert Prometheus/Grafana alert to Security Hub finding."""
    alert = json.loads(event["body"])

    severity_map = {
        "critical": {"Label": "CRITICAL", "Normalized": 90},
        "warning": {"Label": "HIGH", "Normalized": 70},
        "info": {"Label": "MEDIUM", "Normalized": 40},
    }

    for alert_item in alert.get("alerts", []):
        severity = severity_map.get(
            alert_item["labels"].get("severity", "info"),
            {"Label": "MEDIUM", "Normalized": 40}
        )

        finding = {
            "SchemaVersion": "2018-10-08",
            "Id": f"otel-security/{alert_item['labels']['alertname']}/{alert_item['startsAt']}",
            "ProductArn": f"arn:aws:securityhub:{context.invoked_function_arn.split(':')[3]}:{context.invoked_function_arn.split(':')[4]}:product/{context.invoked_function_arn.split(':')[4]}/default",
            "GeneratorId": "opentelemetry-security-observability",
            "AwsAccountId": context.invoked_function_arn.split(":")[4],
            "Types": ["Software and Configuration Checks/Vulnerabilities"],
            "CreatedAt": alert_item["startsAt"],
            "UpdatedAt": datetime.utcnow().isoformat() + "Z",
            "Severity": severity,
            "Title": alert_item["annotations"].get("summary", alert_item["labels"]["alertname"]),
            "Description": alert_item["annotations"].get("description", "Security alert from OTel pipeline"),
            "Resources": [
                {
                    "Type": "Other",
                    "Id": alert_item["labels"].get("service_name", "unknown"),
                    "Region": context.invoked_function_arn.split(":")[3]
                }
            ],
        }

        securityhub.batch_import_findings(Findings=[finding])

    return {"statusCode": 200, "body": "Findings imported"}

Implementation Roadmap

Phase 1: Foundation (Week 1-2)

• Deploy ADOT Collector on EKS as a centralized Deployment
• Configure dual-export to CloudWatch/X-Ray and Amazon Managed Prometheus
• Instrument authentication endpoints with security span attributes
• Verify telemetry flow end-to-end

Phase 2: Security Instrumentation (Week 3-4)

• Add OTel instrumentation to all API gateways and authentication services
• Implement authorization event tracking
• Deploy ADOT Lambda layer for serverless workloads
• Create baseline security metrics (auth failure rate, API error rate, payload sizes)

Phase 3: Dashboards and Alerting (Week 5-6)

• Deploy Grafana on ECS Fargate (or enable Amazon Managed Grafana)
• Import security dashboard JSON configurations
• Configure Prometheus alerting rules for authentication and API security
• Connect Grafana Alerting to SNS and Slack for incident notification

Phase 4: Advanced Detection (Week 7-8)

• Implement impossible travel detection rules
• Add data exfiltration monitoring via payload size anomaly detection
• Create Security Hub integration for centralized finding management
• Build investigation workflows in Grafana Explore for trace-to-log correlation

Related Articles

• Real-Time Intrusion Detection Using AWS GuardDuty – Pair GuardDuty findings with OTel traces for context-rich threat detection
• AWS Lambda Security: Automated Threat Detection – Extend Lambda security with ADOT instrumentation for deeper observability
• AWS Cloud Security Best Practices Implementation Guide – Foundation security controls that feed into your observability pipeline

Conclusion

Security observability is not optional – it is the difference between detecting a breach in hours versus months. OpenTelemetry provides the vendor-neutral foundation, ADOT gives you production-ready AWS integration, and Grafana delivers the visualization layer your security team actually wants to use.

The approach outlined in this article – ADOT as the universal collector, dual-exporting to both AWS-native services and open source backends – gives you the best of both worlds. You keep CloudWatch integration for native AWS alarms and compliance. You gain Grafana for the advanced dashboards, correlation, and alerting that security operations demand. And you maintain the freedom to move your telemetry pipeline to any cloud or on-prem environment without rewriting a single line of instrumentation.

Start with authentication instrumentation. It is the highest-signal security telemetry you can collect, and the ADOT Collector makes it trivial to route to multiple backends. From there, expand to API behavior tracking, authorization monitoring, and the advanced detection patterns covered in this guide.

The observability gap in security is closing. OpenTelemetry is the open standard making it happen, and AWS is all in with ADOT. The question is not whether to adopt it – it is how quickly you can get your security telemetry flowing.

For hands-on guidance implementing security observability with OpenTelemetry on AWS, connect with me on LinkedIn. I regularly share DevSecOps strategies, AWS security patterns, and open source tooling recommendations.

Introduction: Why Container Signing Matters

Every container image your cluster pulls is a trust decision. When your EKS node downloads myapp:v2.1.0 from ECR, how does it know that image was built by your CI pipeline and not injected by an attacker who compromised your registry credentials? Without cryptographic signatures, the answer is: it does not.

The container supply chain has become one of the most targeted attack surfaces in cloud-native environments. The container security market reached $3.05 billion in 2025 and is projected to hit $9.01 billion by 2030, driven by organizations realizing that unsigned images are an open door for supply chain attacks. Chainguard’s $356 million Series D at a $3.5 billion valuation underscores just how seriously the industry is taking this problem.

Recent incidents paint a stark picture:

• 78% of organizations rely on third-party container images with no verification of provenance
• Researchers demonstrated successful supply chain attacks against 92% of popular ML frameworks through poisoned container images
• The average cost of a supply chain compromise has reached $6.8 million per incident

The solution is cryptographic signing and verification: every image gets a signature proving who built it, when, and from what source. At deployment time, admission controllers reject anything unsigned. This is zero-trust applied to your artifact pipeline.

AWS provides its own signing service (AWS Signer with managed ECR signing), but the open-source Sigstore ecosystem, led by Cosign, has become the industry standard. Kubernetes itself uses Sigstore to sign its releases. The key advantage: keyless signing with Fulcio eliminates the entire problem of key management, while Rekor transparency logs provide a public, tamper-evident audit trail.

This guide walks through implementing container image signing on AWS using both approaches, with a clear recommendation: lead with Cosign for portability and keyless signing, use AWS Signer when you need managed simplicity within a pure-AWS environment.

Container signing is a core control in the AI supply chain security pillar, especially when ML training, inference, and batch jobs depend on container images that must be verified before deployment.

The Sigstore Ecosystem

Sigstore is an open-source project under the Linux Foundation that provides free, easy-to-use tools for signing, verifying, and protecting software artifacts. It consists of three core components that work together.

Cosign: The Signing Tool

Cosign is the command-line tool that signs container images and other OCI artifacts. It supports multiple signing modes:

• Keyless signing via Fulcio (recommended) – no keys to manage
• Key-pair signing with local or KMS-managed keys (AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault)
• Hardware key signing with PKCS#11 tokens and Yubikeys

Cosign stores signatures as OCI artifacts alongside the image in the same registry. This means ECR, Docker Hub, GitHub Container Registry, and any OCI-compliant registry work out of the box with no additional infrastructure.

Cosign v3, the latest major release, introduces the standardized bundle format (--new-bundle-format) that packages all signed material and verification data into a single artifact. The upcoming Cosign v4 will simplify the CLI further by removing legacy flags.

Fulcio: The Certificate Authority

Fulcio is a free, public certificate authority that issues short-lived code signing certificates (valid for 10 minutes) bound to an OpenID Connect (OIDC) identity. Instead of managing long-lived signing keys, you authenticate with an identity provider (GitHub, Google, Microsoft) and Fulcio issues a certificate that proves you (or your CI system) performed the signing.

This eliminates the single biggest operational burden in code signing: key lifecycle management. No key generation, no key rotation, no key revocation, no key storage – just identity.

Rekor: The Transparency Log

Rekor is an immutable, append-only transparency log that records every signing event. When Cosign signs an image, it uploads the signature and certificate to Rekor, creating a tamper-evident record. This serves three purposes:

1. Auditability – you can verify when any artifact was signed and by whom
2. Compromise detection – if Fulcio is compromised and issues rogue certificates, the transparency log reveals it
3. Offline verification – verifiers can check that a signature was logged at a time when the certificate was valid, even after the certificate expires

Together, these three components create a “keyless” signing workflow that is more secure than traditional key-based signing because it eliminates the key management attack surface entirely.