[{"slug":"ZV-2026-0220","server_name":"ia-qa.com","severity":"breaking","title":"ia-qa.com: Field top_n was renamed to k on rerank_evaluate.","summary":"[risky] Description of rerank_evaluate changed (78% word delta). [risky] Optional field api_key was added to rerank_evaluate; may shift model behaviour. [risky] Field has_ground_truth was added to rerank_evaluate output. [breaking] Field top_n was renamed to k on rerank_evaluate. [risky] Field latency_ms was added to rerank_evaluate output. [breaking] Field query was renamed to mode on rerank_evaluate. [risky] Field precision_at_k was added to rerank_evaluate output. [breaking] Field results was renamed to ranked on rerank_evaluate. [risky] Field recall_at_k was added to rerank_evaluate output. [risky] Field threshold was added to rerank_evaluate output. [risky] Field total_passages was added to rerank_evaluate output. [risky] Field verdict was added to rerank_evaluate output. [risky] Description of validate_agent_trajectory changed (53% word delta). [breaking] Field total was removed from validate_agent_trajectory output; consumers reading it will break. [breaking] Field failed was removed from validate_agent_trajectory output; consumers reading it will break. [breaking] Field passed was removed from validate_agent_trajectory output; consumers reading it will break. [breaking] Field assertions was renamed to results on validate_agent_trajectory. [risky] Field summary was added to validate_agent_trajectory output. [risky] Field warnings was added to validate_agent_trajectory output.","changes":[{"kind":"description_changed","tool":"rerank_evaluate","after":"Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels. Two modes. (1) BYO scores — give each passage the `score` your own reranker produced (Cohere, Jina, a self-hosted NIM, a cross-encoder): deterministic, offline, no API key, and it evaluates YOUR reranker rather than someone else's. (2) Live NVIDIA reranker — omit scores and supply `api_key` for an NVIDIA account with reranking entitlement. Note that NVIDIA retired its hosted reranking endpoints on 2026-05-18, so mode 1 is the path that works for everyone.","before":"Evaluate RAG retrieval quality using the NVIDIA neural reranker (nv-rerankqa-mistral-4b-v3). Ranks passages by semantic relevance to a query and computes Precision@k and Recall@k. Optionally accepts ground-truth relevance labels to produce a PASS/FAIL CI/CD verdict.","detail":"Description of `rerank_evaluate` changed (78% word delta).","severity":"risky","descriptionDelta":0.78},{"kind":"input_property_added","path":"inputSchema.properties.api_key","tool":"rerank_evaluate","after":{"type":"string","description":"Your NVIDIA API key (BYOK), used only when no passage carries a score. Transits RAM for the single call, never stored."},"detail":"Optional field `api_key` was added to `rerank_evaluate`; may shift model behaviour.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.has_ground_truth","tool":"rerank_evaluate","after":{"type":"boolean"},"detail":"Field `has_ground_truth` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_renamed","path":"outputSchema.properties.top_n","tool":"rerank_evaluate","after":"k","before":"top_n","detail":"Field `top_n` was renamed to `k` on `rerank_evaluate`.","severity":"breaking"},{"kind":"output_property_added","path":"outputSchema.properties.latency_ms","tool":"rerank_evaluate","after":{"type":"number"},"detail":"Field `latency_ms` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_renamed","path":"outputSchema.properties.query","tool":"rerank_evaluate","after":"mode","before":"query","detail":"Field `query` was renamed to `mode` on `rerank_evaluate`.","severity":"breaking"},{"kind":"output_property_added","path":"outputSchema.properties.precision_at_k","tool":"rerank_evaluate","after":{"type":"number"},"detail":"Field `precision_at_k` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_renamed","path":"outputSchema.properties.results","tool":"rerank_evaluate","after":"ranked","before":"results","detail":"Field `results` was renamed to `ranked` on `rerank_evaluate`.","severity":"breaking"},{"kind":"output_property_added","path":"outputSchema.properties.recall_at_k","tool":"rerank_evaluate","after":{"type":"number"},"detail":"Field `recall_at_k` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.threshold","tool":"rerank_evaluate","after":{"type":"number"},"detail":"Field `threshold` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.total_passages","tool":"rerank_evaluate","after":{"type":"number"},"detail":"Field `total_passages` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.verdict","tool":"rerank_evaluate","after":{"type":"string"},"detail":"Field `verdict` was added to `rerank_evaluate` output.","severity":"risky"},{"kind":"description_changed","tool":"validate_agent_trajectory","after":"Run declarative assertions on an agent trace (OpenAI tool-call messages, Anthropic tool_use/tool_result blocks, LangChain run trees, or plain text ReAct logs). No LLM call — deterministic. Assertion types: order (tool A before B), must_call, must_not_call, max_calls, min_calls, no_error, recovery (agent completes a successful step after its last error). A step counts as errored when the trace says so — is_error/isError, status/state in {error,failed,exception,…}, an error field, a JSON body with error/success:false — or when its text payload STARTS with an error marker (Error:, Traceback, TypeError:). Model prose is never scanned for keywords, and every errored step reports error_signal naming what flagged it. Returns per-assertion PASS/FAIL, parsed steps, warnings (a trace parsing to 0 steps is flagged — assertions passing on emptiness prove nothing), and an overall verdict. Use this to gate CI/CD on agent behavior correctness.","before":"Run declarative assertions on an agent trace (OpenAI tool-call messages, LangChain run trees, or plain text logs). No LLM call — deterministic. Assertion types: order (tool A before B), must_call, must_not_call, max_calls, min_calls, no_error, recovery (agent continues after error). Returns per-assertion PASS/FAIL, parsed steps, and an overall verdict. Use this to gate CI/CD on agent behavior correctness.","detail":"Description of `validate_agent_trajectory` changed (53% word delta).","severity":"risky","descriptionDelta":0.5277777777777778},{"kind":"output_property_removed","path":"outputSchema.properties.total","tool":"validate_agent_trajectory","before":{"type":"number"},"detail":"Field `total` was removed from `validate_agent_trajectory` output; consumers reading it will break.","severity":"breaking"},{"kind":"output_property_removed","path":"outputSchema.properties.failed","tool":"validate_agent_trajectory","before":{"type":"number"},"detail":"Field `failed` was removed from `validate_agent_trajectory` output; consumers reading it will break.","severity":"breaking"},{"kind":"output_property_removed","path":"outputSchema.properties.passed","tool":"validate_agent_trajectory","before":{"type":"number"},"detail":"Field `passed` was removed from `validate_agent_trajectory` output; consumers reading it will break.","severity":"breaking"},{"kind":"output_property_renamed","path":"outputSchema.properties.assertions","tool":"validate_agent_trajectory","after":"results","before":"assertions","detail":"Field `assertions` was renamed to `results` on `validate_agent_trajectory`.","severity":"breaking"},{"kind":"output_property_added","path":"outputSchema.properties.summary","tool":"validate_agent_trajectory","after":{"type":"object","additionalProperties":true},"detail":"Field `summary` was added to `validate_agent_trajectory` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.warnings","tool":"validate_agent_trajectory","after":{"type":"array","items":{"type":"string"}},"detail":"Field `warnings` was added to `validate_agent_trajectory` output.","severity":"risky"}],"published_at":"2026-08-20T08:46:38.958Z"},{"slug":"ZV-2026-0199","server_name":"ia-qa.com","severity":"breaking","title":"ia-qa.com: risk_level on cors_test narrowed to a closed enum (safe, low, medium, high, critical, unknown); previously valid values may now be rejected.","summary":"[risky] Description of cookie_security_audit changed (25% word delta). [risky] Description of cors_test changed (51% word delta). [risky] Field error was added to cors_test output. [risky] Field origins_conclusive was added to cors_test output. [risky] Field origins_inconclusive was added to cors_test output. [risky] Field origins_reachable was added to cors_test output. [risky] Field warning was added to cors_test output. [breaking] risk_level on cors_test narrowed to a closed enum (safe, low, medium, high, critical, unknown); previously valid values may now be rejected. [risky] Description of json_diff changed (59% word delta). [risky] Description of parse_csv changed (41% word delta). [risky] Description of regex_test changed (40% word delta). [risky] Field elapsed_ms was added to regex_test output. [risky] Field redos_detected was added to regex_test output. [risky] Field verdict was added to regex_test output. [risky] Description of toxicity_scan changed (71% word delta). [risky] Field method was added to toxicity_scan output.","changes":[{"kind":"description_changed","tool":"cookie_security_audit","after":"Audit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, missing Secure flag, SameSite=None without Secure, overly broad Domain, and excessive TTL. Returns per-cookie grades and an overall security score (0–100) — the score is the WEAKEST cookie, not the average, so one leaking session cookie cannot be averaged into a green result (average_score is reported separately).","before":"Audit the security attributes of cookies set by any URL. Fetches the URL and inspects all Set-Cookie headers for: HttpOnly, Secure, SameSite, Domain scope, Path scope, Max-Age/Expires, __Host-/__Secure- prefixes. Flags insecure patterns: missing HttpOnly on session cookies, missing Secure flag, SameSite=None without Secure, overly broad Domain, and excessive TTL. Returns per-cookie grades and an overall security score (0–100).","detail":"Description of `cookie_security_audit` changed (25% word delta).","severity":"risky","descriptionDelta":0.25373134328358204},{"kind":"description_changed","tool":"cors_test","after":"Test a URL for CORS misconfigurations. Sends preflight (OPTIONS) requests with various Origin headers to detect: wildcard origins with credentials, origin reflection (echoing any origin), null origin acceptance, subdomain wildcard bypass, and missing Vary headers. Returns risk level (safe/low/medium/high/critical) plus per-origin results. \"unknown\" means nothing was actually tested — every origin either failed to connect or answered 5xx, so the target returned no CORS decision; never read it as \"safe\". A 4xx preflight IS a real result (the server refused it and a browser would fail closed).","before":"Test a URL for CORS misconfigurations. Sends preflight (OPTIONS) and cross-origin requests with various Origin headers to detect: wildcard origins with credentials, origin reflection (echoing any origin), null origin acceptance, subdomain wildcard bypass, and missing Vary headers. Returns risk level (safe/low/medium/high/critical), per-test results, and fix recommendations. Essential for API security audits.","detail":"Description of `cors_test` changed (51% word delta).","severity":"risky","descriptionDelta":0.5125},{"kind":"output_property_added","path":"outputSchema.properties.error","tool":"cors_test","after":{"type":"string"},"detail":"Field `error` was added to `cors_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.origins_conclusive","tool":"cors_test","after":{"type":"number"},"detail":"Field `origins_conclusive` was added to `cors_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.origins_inconclusive","tool":"cors_test","after":{"type":"number"},"detail":"Field `origins_inconclusive` was added to `cors_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.origins_reachable","tool":"cors_test","after":{"type":"number"},"detail":"Field `origins_reachable` was added to `cors_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.warning","tool":"cors_test","after":{"type":"string"},"detail":"Field `warning` was added to `cors_test` output.","severity":"risky"},{"kind":"enum_narrowed","path":"outputSchema.properties.risk_level","tool":"cors_test","after":"enum[safe,low,medium,high,critical,unknown]","before":"open","detail":"`risk_level` on `cors_test` narrowed to a closed enum (safe, low, medium, high, critical, unknown); previously valid values may now be rejected.","severity":"breaking"},{"kind":"description_changed","tool":"json_diff","after":"Compute a deep structural diff between two JSON values. Returns added, removed, and changed keys with dot-notation paths. Like git diff but for JSON objects — perfect for API response regression testing. Arrays are not compared blindly by position: the same elements in a different order collapse to a single \"reordered\" change, and an array of records sharing a stable identity field (id, uuid, key, name…) is matched by that field, so paths read [id=42] and a moved record is not reported as N rewrites.","before":"Compute a deep structural diff between two JSON values. Returns added, removed, and changed keys with dot-notation paths. Like git diff but for JSON objects — perfect for API response regression testing.","detail":"Description of `json_diff` changed (59% word delta).","severity":"risky","descriptionDelta":0.5915492957746479},{"kind":"description_changed","tool":"parse_csv","after":"Parse a CSV string into a JSON array of objects (or raw arrays). Full RFC 4180: quoted fields may contain the delimiter, embedded newlines (the Excel/Sheets multi-line cell), and doubled quotes. Custom delimiters supported. An unterminated quote is rejected with its position rather than parsed into corrupted rows. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.","before":"Parse a CSV string into a JSON array of objects (or raw arrays). Handles RFC 4180 quoted fields, escaped quotes, and custom delimiters. Use when processing spreadsheet exports, data imports, or structured text pipelines where the source is CSV. Supports up to 200 KB.","detail":"Description of `parse_csv` changed (41% word delta).","severity":"risky","descriptionDelta":0.40909090909090906},{"kind":"description_changed","tool":"regex_test","after":"Test a regular expression pattern against an input string and return all matches with their index positions and named capture groups. Use for validating user inputs, extracting structured data from text, or debugging regex patterns. Supports flags g, i, m, s, u, y. The match runs in an isolated thread with a 500 ms budget: a pattern that blows up (catastrophic backtracking, e.g. \"(a+)+$\") comes back as redos_detected:true — a real ReDoS verdict on your pattern — instead of hanging.","before":"Test a regular expression pattern against an input string and return all matches with their index positions and named capture groups. Use for validating user inputs, extracting structured data from text, or debugging regex patterns. Supports flags g, i, m, s, u, y.","detail":"Description of `regex_test` changed (40% word delta).","severity":"risky","descriptionDelta":0.4},{"kind":"output_property_added","path":"outputSchema.properties.elapsed_ms","tool":"regex_test","after":{"type":"number"},"detail":"Field `elapsed_ms` was added to `regex_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.redos_detected","tool":"regex_test","after":{"type":"boolean"},"detail":"Field `redos_detected` was added to `regex_test` output.","severity":"risky"},{"kind":"output_property_added","path":"outputSchema.properties.verdict","tool":"regex_test","after":{},"detail":"Field `verdict` was added to `regex_test` output.","severity":"risky"},{"kind":"description_changed","tool":"toxicity_scan","after":"Scan text for toxic language, hate speech, bias/stereotype framing, violence, sexual and self-harm content. Lexical + structural pattern matching (identity term + predicate), not a semantic classifier — returns per-category risk plus the named rules that fired, so every finding can be checked. Useful for LLM safety guardrail testing and triage; signal-only, not a calibrated CI gate.","before":"Scan text for toxic language, bias indicators, profanity, and harmful content categories. Returns risk scores per category. Useful for LLM safety guardrail testing.","detail":"Description of `toxicity_scan` changed (71% word delta).","severity":"risky","descriptionDelta":0.711864406779661},{"kind":"output_property_added","path":"outputSchema.properties.method","tool":"toxicity_scan","after":{"type":"string"},"detail":"Field `method` was added to `toxicity_scan` output.","severity":"risky"}],"published_at":"2026-08-19T15:22:37.303Z"}]