From Source VLAN to Evidence Bundle: Building a PCI DSS Segmentation Test Runner

Segmentation testing is easy to describe badly. “Scan the cardholder data environment from outside and prove it is blocked” sounds decisive, but it hides the work that makes a result defensible: confirming the source network, controlling the target list, recording the route, choosing probes, preserving raw output, and separating observation from judgment.

That is the problem I designed seg-test to address. The project is a Bash runner for repeatable, one-shot reachability testing from each approved source VLAN toward an explicitly supplied cardholder data environment (CDE) target list. This article describes the design captured in the immutable source revision, including what it can establish, what it cannot establish, and where I would harden it next.

The central claim is deliberately narrow: this tool collects structured network evidence for an authorized assessor. It does not determine PCI DSS compliance, certify segmentation, or replace a qualified tester's methodology and judgment.

Why evidence collection is the real engineering problem

PCI DSS is a baseline of technical and operational requirements for protecting payment account data. PCI SSC published PCI DSS v4.0.1 as a limited revision that clarified guidance without adding or deleting requirements (PCI SSC: “Just Published: PCI DSS v4.0.1”).

Where segmentation is used to isolate the CDE, Requirement 11.4.5 in the standard available through PCI SSC's PCI DSS document library expects segmentation penetration testing at least every 12 months and after changes to segmentation controls or methods. The work must cover the controls in use, follow the entity's methodology, confirm that isolation is effective, and be performed by a qualified resource with organizational independence. A scan is therefore an input to a broader assessment, not the assessment itself.

The boundary is also no longer synonymous with one firewall between two static VLANs. PCI SSC's modern network architecture guidance calls out hybrid and multi-cloud environments, zero-trust architecture, micro-segmentation, ephemeral services, asset inventories, and the complexity of modern configurations. It is supplemental guidance, not a replacement for the standard, but it makes the engineering lesson clear: a useful runner must preserve context about where a test originated and what boundary it exercised.

The design: one bounded run per source VLAN

I chose a portable kit rather than a central scanner. The script, the CDE list, a per-engagement configuration, and the resulting evidence stay together. An operator runs the kit from inside each source VLAN named in the rules of engagement (ROE). That makes the source position an explicit test variable instead of an assumption buried in a report.

The normal preparation flow is:

1. Obtain signed authorization and a precise ROE before sending a packet.
2. Scaffold a configuration with --init <client>.
3. Populate cde-all.txt with only approved CDE targets, one per line.
4. Define an optional traceroute anchor and any prior-finding hot spots.
5. Preview the run with --dry-run.
6. Enter the approved source VLAN and use --expect-cidr <approved-cidr> as a source-position guard.
7. Execute the run, review the raw output, and preserve the resulting evidence package.

A sanitized command sequence looks like this:

./seg-test.sh --init <client>
./seg-test.sh --client <client> <source-vlan> --dry-run
sudo ./seg-test.sh \
  --client <client> <source-vlan> \
  --expect-cidr <approved-cidr>

The dry run matters. It shows the selected configuration, target count, source guard, scan families, hot-spot count, and intended evidence path without sending packets or requiring root. It creates a human checkpoint before an intrusive action.

What happens during a run

1. Fail early on obvious setup errors

The runner validates the client and VLAN labels, requires a matching client configuration, checks that the target list exists and is non-empty, and verifies its dependencies. It can report missing tools or install the relevant packages on Debian/Kali through apt. Real SYN and UDP scans require root.

2. Create a timestamped evidence workspace

Each execution receives its own directory under evidence/, divided into baseline, Nmap, manual, packet-capture, and screenshot locations. The runner starts a transcript and records metadata such as the source label, host, operator, arguments, target count, anchor, expected CIDR, and selected ports.

The empty packet-capture and screenshot directories are intentional placeholders. They make room for human follow-up without pretending that the runner captured evidence it did not collect.

3. Record the source-side baseline

Before scanning, the script records time, interface addresses, routes, neighbor state, and—when configured—a short traceroute to a CDE anchor. It prints the detected IPv4 source addresses prominently.

The --expect-cidr guard is cheap insurance: if no local interface falls inside the approved source CIDR, the run aborts. There is a --force escape hatch, but using it transfers the burden to the operator and should be documented as an exception in the final report.

4. Exercise more than one reachability signal

The first network phase performs Nmap host discovery. The official Nmap host-discovery documentation explains why a single ping is insufficient: -sn uses discovery probes without a port scan, while -Pn skips discovery and treats targets as online for later scanning. Firewalls can suppress probes or replies, so “did not answer discovery” is not equivalent to “unreachable.”

The substantive port tests therefore use -Pn and attempt every approved target:

• A TCP SYN scan checks a fixed set of common and assessment-relevant ports.
• An optional full TCP scan covers all ports with a rate limit.
• A selected UDP scan checks a bounded list of infrastructure and management ports unless explicitly skipped.
• Configured TCP and HTTPS hot spots re-test paths documented in an earlier report.

The runner does not silently expand scope. Nmap consumes the supplied target file, and manual hot spots come from the engagement configuration. That design is useful only if both inputs have been independently reviewed against the ROE.

5. Preserve raw artifacts before summarizing

Nmap writes normal, XML, and grepable output. The script extracts open-port lines into a quick-look summary, but the summary is not the evidence of record; the raw files and transcript are. Finally, it archives the run and writes a SHA-256 digest beside the archive.

This produces a convenient integrity check: later, an assessor can detect a change if the separately trusted digest remains available. A bare hash stored beside the archive does not prove who produced the evidence or when, so the package is integrity-checkable—not tamper-evident or tamper-proof.

Exact non-goals

The source code defines four explicit boundaries, and they should remain visible in every use of the tool:

• It does not make pass/fail decisions; that is a human reporting task.
• It does not capture packet traces automatically; ambiguous results may justify a controlled, interactive capture.
• It does not brute-force, exploit, or run Nmap vulnerability scripts; those actions are outside this runner's role and must be separately authorized.
• It does not touch addresses outside cde-all.txt or the explicitly configured hot-spot lists.

I would add one umbrella non-goal for publication: it does not validate PCI DSS compliance. Compliance conclusions depend on scope, methodology, qualifications, independence, complete control coverage, corroborating evidence, and the applicable assessment process.

Interpreting the evidence without overclaiming

An open TCP port demonstrates that the tested source could complete enough of the protocol exchange for Nmap to classify the port at that time. That is strong evidence of a reachable path requiring investigation. It is not automatically a segmentation failure: the connection may be explicitly authorized, mediated, or required, and the assessor must compare it with the approved data flow and control design.

closed usually means the host was reachable but no service accepted the connection. That still demonstrates network reachability; it does not prove isolation.

filtered, silence, and many UDP results are ambiguous. A firewall may have dropped the request, the return path may differ, the host may be down, an intermediate control may rate-limit, or the service may simply not respond to the selected payload. Those observations need correlation with firewall logs, routing, control configuration, and—when authorized—a packet capture from an appropriate observation point.

The report should therefore distinguish facts from conclusions. A useful sentence is: “From approved source position A, target B returned a SYN/ACK on TCP port C at time D.” The assessor can then explain whether that path is expected, how it maps to segmentation policy, and what remediation or exception applies.

Devil's advocate: 14 reasons not to overclaim

1. The source guard is IPv4 and interface based. It does not prove the actual egress path, VRF, policy identity, tunnel selection, or IPv6 position used by every packet.
2. --force can bypass the strongest safety check. An exception can turn a correctly labeled run into evidence from the wrong network.
3. The target list, traceroute anchor, and hot-spot entries are trust boundaries. The runner does not cryptographically bind them to the signed ROE, enforce that the anchor is in the target list, or prove that every entry is authorized.
4. The client environment file is sourced as shell code. A modified configuration can execute with the runner's privileges; it is not merely inert data.
5. Root increases impact. A typo, compromised dependency, or malicious configuration executes in a highly privileged context.
6. -Pn -sn is not independent reachability proof. Because -Pn treats targets as online and -sn disables port scanning, that artifact must not be read as a successful response from every listed host.
7. A fixed common-port list is incomplete. Segmentation leakage on an unlisted port will be missed unless the optional full TCP scan is approved and completed.
8. Selected UDP coverage is inherently narrow and ambiguous. Silent UDP services and filtered responses can produce false reassurance.
9. The full-scan rate is not environment aware. A static maximum rate can be too aggressive for fragile controls or unnecessarily slow elsewhere.
10. No automatic packet capture exists. The evidence may be insufficient to distinguish a dropped probe, asymmetric return path, proxy response, or scanner-side problem.
11. A local timestamp is not trusted time. Clock drift or manipulation weakens chronology unless time synchronization is verified and recorded.
12. A sidecar SHA-256 file authenticates neither origin nor custody. Anyone able to replace the archive can also replace its digest.
13. Command failure can yield a partial-looking package. The script does not use fail-fast execution or a machine-readable completeness manifest for every scan stage.
14. VLAN reachability is only one architecture view. Cloud security groups, service meshes, identity-aware proxies, workloads that appear and disappear, and control-plane paths may not be exercised from a traditional source VLAN.

These are not reasons to discard the runner. They define where automation ends and assessment discipline begins.

Hardening roadmap

My next iteration would prioritize safety and evidence quality before adding more probes.

First, bind execution to authorization. Replace free-form targets with a validated manifest containing approved CIDRs, source positions, permitted protocols, change ticket, ROE identifier, and engagement window. Normalize every target before Nmap sees it, reject unexpected syntax, and require hot spots to fall inside the same allowlist. Store hashes of the manifest, configuration, target list, and exact script revision in run metadata.

Second, make completion explicit. Capture each command's start time, end time, exit code, and output files in a machine-readable run manifest. Mark interrupted or failed stages as incomplete and refuse to label their archive as a completed run. Correct the discovery phase so -sn is used for observed discovery and -Pn is reserved for scans that must test every target regardless of discovery response.

Third, strengthen provenance. Record Nmap and operating-system versions, scanner image or package versions, UTC time, and time-sync status. Sign the final manifest and archive with a detached signature whose public key and verification procedure are maintained outside the engagement kit. Store the signed digest in a separate evidence system so archive and checksum cannot be replaced together without detection.

Fourth, improve interpretation. Parse Nmap XML into a structured finding table while retaining all raw artifacts. Use explicit states such as observed-open, observed-closed, filtered, no-response, tool-error, and not-tested. Never collapse those states into a generated “pass.” Add a reviewer field and require a reason for every allowed reachable path.

Fifth, support targeted corroboration. Offer an opt-in packet-capture mode for specific approved targets and ports, with data-minimization limits, capture duration, storage warnings, and a separate authorization gate. Add collection hooks for firewall logs and policy snapshots rather than trying to infer the entire control path from scanner output.

Finally, model modern boundaries. Extend the test plan beyond VLANs to IPv6, cloud accounts and regions, security groups, Kubernetes network policies, service-mesh identities, remote-access paths, and ephemeral assets. The runner should consume an assessor-defined coverage matrix and report which source-to-destination control paths were actually exercised.

The useful product is disciplined repeatability

The value of seg-test is not that it turns Nmap into a compliance button. It turns a fragile sequence of manual steps into a reviewable workflow: confirm the source, constrain the targets, preview the action, collect baseline context, run bounded probes, retain raw output, summarize observations, and package evidence.

That is the right level of automation for this problem. The machine should make the work consistent and auditable. The assessor should still own authorization, scope, interpretation, exceptions, and the final conclusion.

Source: CyberKareem/seg-test · Immutable revision: db8d0cce2c3a06af20d3974a9cc59611f4415099