Unit test validator

.github/workflows/test.yaml

@@ -29,4 +29,28 @@ jobs:
           yarn redocly lint --max-problems 9999 --format markdown >> $GITHUB_STEP_SUMMARY || true
           yarn redocly lint --max-problems 9999 --format github-actions
       - name: Build
-        run: yarn build
+        run: yarn build
+      - name: Setup Go for validation
+        uses: actions/setup-go@v4
+        with:
+          go-version: '1.21'
+      - name: Run OpenAPI validation
+        run: |
+          cd validator
+
+          # Find the latest request capture file for 2025/08/26
+          CAPTURE_FILE=$(ls -1 request_capture.json 2>/dev/null | tail -1)
+          if [ -z "$CAPTURE_FILE" ]; then
+            echo "No request capture file found"
+            exit 1
+          fi
+
+          echo "Running OpenAPI validator..."
+
+          # Run validator and capture output (don't fail on validation errors)
+          go run main.go "$CAPTURE_FILE" ../dist/latest/openapi.yaml > validation_output.txt
+          cat validation_output.txt >> $GITHUB_STEP_SUMMARY
+      - name: Generate validation report
+        run: |
+          cd scripts
+          ./generate-validation-report.sh >> $GITHUB_STEP_SUMMARY

.gitignore

@@ -45,3 +45,7 @@ node_modules
 out
 tmp
 .tmp
+
+# validator
+validator/validator
+validator/validation_output.txt

README.md

@@ -64,6 +64,41 @@ First, this command runs the build script to ensure the latest specification is
 
 Runs the `bin/create-version.js` script, which is used to create a new versioned directory under `apis/cf/`. This is useful when a new version of the CAPI is released and you need to update the specification. It copies the current `apis/cf/latest/openapi.yaml` to a new versioned directory, and maintains the `redocly.yaml` file for the new version. After running this command and rebuilding, the new version will be available in scalar.
 
+**Validation Testing**
+
+`./scripts/validate-local.sh`
+
+Runs comprehensive validation testing against real captured API requests. This script:
+1. Builds the latest OpenAPI schema
+2. Finds the most recent request capture file from Cloud Foundry production traffic
+3. Validates all captured requests against the OpenAPI specification
+4. Generates a detailed validation report with error analysis and statistics
+
+The validation results include:
+- Total requests tested and success/failure rates
+- Categorized error types (missing response codes, schema validation errors, etc.)
+- Top error patterns and most problematic endpoints
+- Detailed output saved to `validator/validation_output.txt`
+
+**Validation Report Generation**
+
+`./scripts/generate-validation-report.sh`
+
+Generates a comprehensive validation report from existing validation output. This script outputs markdown-formatted reports to STDOUT and can be run independently to analyze validation results. The output is consistent whether run locally or in CI.
+
+**Prerequisites for validation:**
+- Go must be installed and available in your `PATH`
+- Request capture files must be present in the `validator/` directory
+
+**Automated Validation**
+
+The repository includes validation testing as part of the test workflow (`test.yaml`) that automatically runs on every push and pull request. The workflow:
+- Builds the OpenAPI schema
+- Runs validation against the latest request capture data
+- Generates detailed reports in the GitHub job summary
+- Uploads validation results as artifacts
+- Never fails the build - validation errors are reported but don't block PRs
+
 **Compliance Testing**
 
 `yarn test:compliance`

scripts/generate-validation-report.sh

@@ -0,0 +1,97 @@
+#!/bin/bash
+set -e
+
+# Configuration
+VALIDATION_OUTPUT_FILE="validator/validation_output.txt"
+
+# Function to get error count from validation output
+get_error_count() {
+    local pattern="$1"
+    if [ -f "$VALIDATION_OUTPUT_FILE" ]; then
+        grep -c "$pattern" "$VALIDATION_OUTPUT_FILE" 2>/dev/null || echo "0"
+    else
+        echo "0"
+    fi
+}
+
+# Function to extract value from validation summary
+extract_summary_value() {
+    local field="$1"
+    if [ -f "$VALIDATION_OUTPUT_FILE" ]; then
+        grep "$field:" "$VALIDATION_OUTPUT_FILE" | sed 's/.*'"$field"': \([0-9]*\).*/\1/' | head -1 || echo ""
+    else
+        echo ""
+    fi
+}
+
+# Check if validation output exists
+if [ ! -f "$VALIDATION_OUTPUT_FILE" ]; then
+    echo "Validation output file not found: $VALIDATION_OUTPUT_FILE"
+    echo "Run validation first with: ./scripts/validate-local.sh"
+    exit 1
+fi
+
+echo "## OpenAPI Request Spec Validation Report"
+echo ""
+echo ""
+
+# Extract summary statistics
+TOTAL_REQUESTS=$(extract_summary_value "Total requests")
+VALID_REQUESTS=$(extract_summary_value "Valid requests")
+INVALID_REQUESTS=$(extract_summary_value "Invalid requests")
+
+if [ -n "$TOTAL_REQUESTS" ] && [ "$TOTAL_REQUESTS" != "" ]; then
+    echo "### Summary Statistics"
+    echo "- **Total Requests:** $TOTAL_REQUESTS"
+    echo "- **Valid Requests:** $VALID_REQUESTS"
+    echo "- **Invalid Requests:** $INVALID_REQUESTS"
+
+    if [ "$TOTAL_REQUESTS" -gt 0 ]; then
+        SUCCESS_RATE=$(( VALID_REQUESTS * 100 / TOTAL_REQUESTS ))
+        echo "- **Success Rate:** ${SUCCESS_RATE}%"
+    fi
+else
+    echo "**No validation summary found**"
+fi
+
+echo ""
+
+# Error category analysis
+echo "### Error Categories"
+
+MISSING_RESPONSES=$(get_error_count "response code.*does not exist")
+MISSING_PATHS=$(get_error_count "Path.*not found")
+REQUEST_BODY_ERRORS=$(get_error_count "request body.*failed to validate")
+RESPONSE_BODY_ERRORS=$(get_error_count "response body.*failed to validate")
+QUERY_PARAM_ERRORS=$(get_error_count "Query.*parameter.*does not match")
+CONTENT_TYPE_ERRORS=$(get_error_count "content type.*does not exist")
+PATH_PARAM_ERRORS=$(get_error_count "Path parameter.*does not match\|Path parameter.*is missing")
+
+echo "- **Missing Response Codes:** $MISSING_RESPONSES"
+echo "- **Missing Paths:** $MISSING_PATHS"
+echo "- **Request Body Validation Errors:** $REQUEST_BODY_ERRORS"
+echo "- **Response Body Validation Errors:** $RESPONSE_BODY_ERRORS"
+echo "- **Query Parameter Errors:** $QUERY_PARAM_ERRORS"
+echo "- **Path Parameter Errors:** $PATH_PARAM_ERRORS"
+echo "- **Content Type Errors:** $CONTENT_TYPE_ERRORS"
+
+# Calculate total categorized errors
+TOTAL_CATEGORIZED=$((MISSING_RESPONSES + MISSING_PATHS + REQUEST_BODY_ERRORS + RESPONSE_BODY_ERRORS + QUERY_PARAM_ERRORS + PATH_PARAM_ERRORS + CONTENT_TYPE_ERRORS))
+
+# Top error patterns
+echo ""
+echo "### Top Error Patterns"
+
+if [ -f "$VALIDATION_OUTPUT_FILE" ]; then
+    echo ""
+    echo "**Most common error messages:**"
+    echo '```'
+    grep -o "Reason: [^,]*" "$VALIDATION_OUTPUT_FILE" | sort | uniq -c | sort -nr | head -10
+    echo '```'
+
+    echo ""
+    echo "**Most problematic endpoints:**"
+    echo '```'
+    grep -o "url [^?]*" "$VALIDATION_OUTPUT_FILE" | sort | uniq -c | sort -nr | head -10
+    echo '```'
+fi

validator/README.md

@@ -0,0 +1,123 @@
+# Cloud Foundry OpenAPI Unit test Validator
+
+Validates the http calls to Cloud Controller API based on data captured from running CAPI request tests.
+
+## Installation
+
+1. Install dependencies:
+   ```bash
+   go mod download
+   ```
+
+## Usage
+
+To capture the json file. Do the following:
+1. Pull https://github.com/cloudfoundry/cloud_controller_ng/tree/open_api_request_capturer
+2. Run the tests with capture enabled CAPTURE_REQUESTS=true bundle exec rspec 'spec/request'  
+3. `out/request_capture.json` is the input to this program.
+
+
+### Running the validator
+
+Run the validator with command-line arguments:
+
+```bash
+go run main.go <requests.json> <openapi.yaml>
+```
+
+Example:
+```bash
+go run main.go example_request_capture.json ../dist/latest/openapi.yaml
+```
+
+### Building and Running
+
+You can also build the program into an executable:
+
+```bash
+go build -o validator main.go
+./validator cf_requests.json openapi.yaml
+```
+
+The program will:
+1. Load the OpenAPI specification from the specified YAML file
+2. Read HTTP request/response data from the specified JSON file
+3. Validate each request and response against the OpenAPI schema
+4. Display validation results with detailed logging
+5. Show a summary of validation results
+
+### Command Line Arguments
+
+- `<requests.json>` - Path to the JSON file containing HTTP request/response data
+- `<openapi.yaml>` - Path to the OpenAPI specification file (YAML format)
+
+## Input File Format
+
+The `cf_requests.json` file should contain an array of objects with this structure:
+
+```json
+[
+  {
+    "timestamp": "2025-08-22T15:39:32-06:00",
+    "request": {
+      "method": "GET",
+      "path": "/v3/buildpacks",
+      "headers": {
+        "Authorization": "bearer ...",
+        "Content-Type": "application/json"
+      },
+      "body": null
+    },
+    "response": {
+      "status": 200,
+      "headers": {
+        "Content-Type": "application/json; charset=utf-8",
+        "X-VCAP-Request-ID": "..."
+      },
+      "body": {
+        "pagination": {...},
+        "resources": [...]
+      }
+    }
+  }
+]
+```
+
+## Output
+
+The program outputs:
+- Detailed logs for each request validation
+- Individual validation results (pass/fail)
+- Final summary with total counts of valid and invalid requests
+
+Example output:
+```bash
+$ go run main.go cf_requests.json openapi.yaml
+2025/08/22 15:44:30 Loaded 47 requests for validation
+2025/08/22 15:44:30 Validating request 1: GET /v3/buildpacks
+2025/08/22 15:44:30 Request 1 validation passed
+2025/08/22 15:44:30 Response 1 validation passed
+...
+2025/08/22 15:44:30 Validation Summary:
+2025/08/22 15:44:30 Total requests: 47
+2025/08/22 15:44:30 Valid requests: 47
+2025/08/22 15:44:30 Invalid requests: 0
+```
+
+### Error Handling
+
+If you run the program without the correct arguments, you'll see:
+```bash
+$ go run main.go
+Usage: main <requests.json> <openapi.yaml>
+Example: main cf_requests.json openapi.yaml
+```
+
+## Dependencies
+
+- [github.com/pb33f/libopenapi](https://github.com/pb33f/libopenapi) - OpenAPI document parsing
+- [github.com/pb33f/libopenapi-validator](https://github.com/pb33f/libopenapi-validator) - OpenAPI validation
+
+## License
+
+This project is provided as-is for demonstration purposes.

validator/go.mod

@@ -0,0 +1,18 @@
+module cc_openapi_validator
+
+go 1.23.0
+
+toolchain go1.24.6
+
+require (
+	github.com/bahlo/generic-list-go v0.2.0 // indirect
+	github.com/buger/jsonparser v1.1.1 // indirect
+	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/pb33f/libopenapi v0.22.2 // indirect
+	github.com/pb33f/libopenapi-validator v0.4.7 // indirect
+	github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 // indirect
+	github.com/speakeasy-api/jsonpath v0.6.2 // indirect
+	github.com/wk8/go-ordered-map/v2 v2.1.9-0.20240815153524-6ea36470d1bd // indirect
+	golang.org/x/text v0.25.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)

validator/go.sum

@@ -0,0 +1,22 @@
+github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
+github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
+github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
+github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
+github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
+github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
+github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
+github.com/pb33f/libopenapi v0.22.2 h1:ChXG911vrr24KE7wzIib3eL8Td73ANFCNSpWf1C9hy4=
+github.com/pb33f/libopenapi v0.22.2/go.mod h1:utT5sD2/mnN7YK68FfZT5yEPbI1wwRBpSS4Hi0oOrBU=
+github.com/pb33f/libopenapi-validator v0.4.7 h1:sS6RvphkhlgMdad4WutRVd/yzNu/7QE4RdUTjxp0dY4=
+github.com/pb33f/libopenapi-validator v0.4.7/go.mod h1:0G2+HeGK4Oc0ugTG+npGVHVCOPVlc60Bj4ZbVW7B+Dc=
+github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 h1:PKK9DyHxif4LZo+uQSgXNqs0jj5+xZwwfKHgph2lxBw=
+github.com/santhosh-tekuri/jsonschema/v6 v6.0.1/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
+github.com/speakeasy-api/jsonpath v0.6.2 h1:Mys71yd6u8kuowNCR0gCVPlVAHCmKtoGXYoAtcEbqXQ=
+github.com/speakeasy-api/jsonpath v0.6.2/go.mod h1:ymb2iSkyOycmzKwbEAYPJV/yi2rSmvBCLZJcyD+VVWw=
+github.com/wk8/go-ordered-map/v2 v2.1.9-0.20240815153524-6ea36470d1bd h1:dLuIF2kX9c+KknGJUdJi1Il1SDiTSK158/BB9kdgAew=
+github.com/wk8/go-ordered-map/v2 v2.1.9-0.20240815153524-6ea36470d1bd/go.mod h1:DbzwytT4g/odXquuOCqroKvtxxldI4nb3nuesHF/Exo=
+golang.org/x/text v0.25.0 h1:qVyWApTSYLk/drJRO5mDlNYskwQznZmkpV2c8q9zls4=
+golang.org/x/text v0.25.0/go.mod h1:WEdwpYrmk1qmdHvhkSTNPm3app7v4rsT8F2UD6+VHIA=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=