FPGA RTL timing and verification audit

A complete find → fix → re-close loop, executed and measured on the auditor’s own open-source design so every number can be checked. A client audit follows this exact structure on your RTL.

Design: cxl-kv-forge-qos · Part: AMD Zynq UltraScale+ xczu7ev-ffvc1156-2-e · Target: 400 MHz (2.5 ns) · Vivado 2025.2

1. Executive summary

The audit asked one question of a design already closed at 350 MHz: what, exactly, stops 400 MHz, and what is the cheapest change that recovers it? Post-route analysis at the 400 MHz target identified a single routing-dominated cross-module net as the worst path (Finding A, §3). Registering that crossing (§4) recovered full closure: 400 MHz post-route with WNS +0.033 ns, and the new worst path is the registered crossing’s own output net — zero logic levels, 96% routing — a wire that now fits comfortably inside its own clock period. The strengthened verification required to sign off the fix also exposed a pre-existing functional bug in the dequeue path (Finding B, §4) that the original test suite could not see — found, fixed, and re-verified in the same loop.

What that buys, in capability terms: 350 to 400 MHz is a 14.3% clock-rate increase on the same part and the same RTL footprint. At the documented 256-bit datapath, that is the difference between 89.6 and 102.4 Gb/s sustained per stream (256 b × 350 vs 400 MHz) — 179.2 to 204.8 Gb/s aggregate — purchased with one register slice and one cycle of arbitration latency the budget already tolerates. The audit’s product is recovered headroom on silicon you already chose, quantified before you commit to it.

source: 256-bit datapath + throughput: results/PHASE2_1_FINAL.md (350 MHz — 89.6 / 179.2 Gb/s) · results/PHASE2_2_FINAL.md (400 MHz — 102.4 / 204.8 Gb/s); percentages are on-page arithmetic from those figures

source: results/PHASE2_1_FINAL.md (350 baseline) · results/impl/baseline_400mhz_prefix/post_route_timing.rpt (limiter, pre-fix) · results/impl/strategy_Performance_Explore/post_route_timing.rpt + phase2_strategy_sweep.md (re-close) in github.com/taitashaw/cxl-kv-forge-qos

2. Design under audit + methodology

cxl-kv-forge-qos is a multi-tenant KV-cache QoS controller: 15 SystemVerilog RTL modules plus a shared package (16 sources under rtl/), spanning request parsing, per-tenant token-bucket credit enforcement, a 16-entry-per-tenant queue manager, a pipelined tournament-tree deadline arbiter, a single-issue memory engine against a BRAM model, and an AXI4-Lite control plane with performance counters. The design clock is generated by an MMCM (clk_wiz_0) inside the Vivado block design; synthesis runs with global retiming (STEPS.SYNTH_DESIGN.ARGS.RETIMING true), and implementation is a four-strategy parallel sweep with post-route phys-opt, winner picked on WNS then TNS.

source: rtl/ (module inventory) · vivado/create_block_design.tcl (clk_wiz) · vivado/synth_impl_bitstream.tcl (retiming, sweep, winner selection) in github.com/taitashaw/cxl-kv-forge-qos

Verification baseline: a 13-test XSim regression (12 inherited + 1 added by this audit) driving AXIS traffic against a scoreboard, with concurrent SVA for AXIS stability, response-status integrity, and AXI4-Lite liveness bound into the bench. The audit flow is: read the post-route worst path, attribute it (logic vs route, levels, endpoints), change the minimum RTL, re-verify bit-exact, re-run the same P&R flow, and read the new report. Tools: Vivado 2025.2 and XSim on the same install.

source: sim/tb/tb_kvq_top.sv, sim/tb/kvq_assertions.sv (regression + SVA) · results/rtl_sim/phase1_xsim_summary.csv (13/13 PASS, committed; raw xsim.log is gitignored) in github.com/taitashaw/cxl-kv-forge-qos

Two further designs by the same author corroborate the methodology at the weaker evidence tier (README-documented): flashattn-softmax-engine (400 MHz, WNS +0.413 ns, WHS +0.017 ns, 16/16 tests, 550 LUTs / 882 FFs / 0 DSP / 0 BRAM) and kvcache-compress-engine (400 MHz, WNS +0.308 ns, 16/16 tests). Their numbers are documented in their READMEs; unlike the primary design, their timing reports are not committed, so this report cites them only as corroboration and never as proof.

source: README.md of github.com/taitashaw/flashattn-softmax-engine and github.com/taitashaw/kvcache-compress-engine

At the 400 MHz target the design fails timing by exactly one classifiable mechanism. The worst setup path runs from a credit-bucket register in the credit engine to a queue-memory write-enable in the queue manager — two different modules — and its delay is 2.499 ns against the 2.500 ns budget: 0.703 ns logic (28%) across 8 levels (CARRY8×2, LUT6×3, LUT4, LUT2, MUXF7), and 1.796 ns routing (72%). Sweep-wide, the target misses with WNS −0.192 ns, TNS −834.045 ns across 9,123 failing endpoints (best of four strategies, Performance_ExtraTimingOpt).

Root cause: this is not a logic-depth problem. Eight levels in 0.703 ns is healthy for this part; the path fails because the credit decision and the queue write sit on opposite sides of a long cross-module net that the placer cannot shorten — 72% of the delay is wire. Classifying the dominator correctly matters because the two failure modes have opposite fixes: logic depth wants restructuring or retiming, routing dominance wants a register that splits the net across two clock periods. Retiming was already on and could not move a register into this path, because no register existed on the crossing to move.

source: results/impl/baseline_400mhz_prefix/post_route_timing.rpt (path detail, WNS, TNS, endpoints) · results/impl/baseline_400mhz_prefix/phase2_strategy_sweep.md (pre-fix four-strategy table) in github.com/taitashaw/cxl-kv-forge-qos

4. The fix + measured recovery

The fix is a forward register slice on the credit→queue-manager beat: {valid, req, max_depth} is captured into registers and held until the queue manager accepts it, with ready passed through so the handshake semantics are unchanged. The captured beat lands one cycle later — a latency the QoS arbitration budget tolerates (§6) — and the long net now has a full clock period on each side. The complete RTL change:

assign ce_out_ready = !enqp_valid || qm_enq_ready;

always_ff @(posedge clk or negedge rst_n) begin
  if (!rst_n) begin
    enqp_valid     <= 1'b0;
    enqp_req       <= '0;
    enqp_max_depth <= '0;
  end else if (soft_reset) begin
    enqp_valid     <= 1'b0;
  end else if (ce_out_valid && ce_out_ready) begin
    enqp_valid     <= 1'b1;
    enqp_req       <= ce_out_req;
    enqp_max_depth <= lookup_contract.max_queue_depth;
  end else if (qm_enq_ready) begin
    enqp_valid     <= 1'b0;
  end
end

Re-verification, then re-measurement — in that order. The full XSim regression passes 13/13 bit-exact with all SVA clean, including a new backpressure-injection test written for this change: today’s queue manager never stalls (its ready is structurally constant-high), so the test forces a synthetic stall window on the crossing and proves the slice holds the captured beat unmutated and loses nothing — the one failure mode a registered handshake can hide. A new SVA pins that property permanently.

The re-run of the identical four-strategy P&R flow at the 400 MHz target measured: closed. WNS +0.033 ns on the 400 MHz clock group (design-level WNS +0.026 ns including recovery checks, also MET), TNS 0.000, zero failing endpoints of 77,540, hold WNS +0.010 ns. Three of the four strategies met timing independently (+0.033, +0.025, +0.002 ns); the fourth missed by 12 ps. The new worst path is the slice’s own output net into the queue memory: 2.168 ns at zero logic levels, 96.3% routing — the cross-module wire now owns a full period, which is exactly what the fix was for. One disclosure: 4 pulse-width (min-period) violations remain on the PS AXI port clocks (PS8 MAXIGP*ACLK, spec 333.33 MHz, slack −0.500 ns). They predate this work, are present identically in the pre-fix baseline, and sit outside the PL fabric; a deployment bridges the PS-facing AXI onto a ≤333 MHz domain. Fabric setup, hold, and pulse-width are clean.

source: before: results/impl/baseline_400mhz_prefix/post_route_timing.rpt · after: results/impl/strategy_Performance_Explore/post_route_timing.rpt (both committed) in github.com/taitashaw/cxl-kv-forge-qos

Independent re-run, for honesty about determinism: the committed winner reads WNS +0.033 ns. Re-running the same strategy from scratch in this audit session (2026-06-19) — a fresh place-and-route, different run, same RTL and constraints — reproduced WNS +0.033 ns, the same worst path (the slice’s own output net), and the same 2.168 ns data-path delay. The number did not move. Place-and-route is seed- and version-sensitive, so a re-run can legitimately land a few picoseconds either side; here it landed on the signoff number exactly, which is the stronger evidence that the +0.033 ns close is robust rather than a single-seed artifact.

source: committed: results/impl/strategy_Performance_Explore/post_route_timing.rpt (WNS +0.033 ns) · fresh re-run 2026-06-19: results/impl/rerun_Performance_Explore/post_route_timing.rpt (WNS +0.033 ns), Vivado 2025.2, xczu7ev-ffvc1156-2-e — generated by the auditor in this session

Signing off the slice meant strengthening the bench, and the stronger bench found a bug that predates this audit. The new injection test is the first in the suite to require every queued request to complete — the inherited tests spot-check one request per batch. It failed: with two requests queued, the pipelined tournament arbiter granted 4 times, issued the first request 4 times, and the second request never. Root cause: the tree samples the queue heads into a new wave every cycle, but the pop lands at the tree’s output stage cycles later — so one un-popped head rides several overlapping waves, and each stale grant pops a successor entry while presenting the old request’s data. Silent data loss plus silent duplication, invisible to any test that doesn’t demand exact-once completion.

Fix: a wave-admission interlock — a new wave enters the tree only when no wave is in flight and the head readout has refreshed after the last grant. The tree’s pipelined structure (the reason it closes timing) is untouched. After the fix the same counters read 2 grants for 2 requests, each issued exactly once, 13/13 tests pass, and the suite’s total observed-response count drops from 52 to 33 — the difference was duplicate responses the old arbiter had been emitting all along.

source: sim/tb/tb_kvq_top.sv (ENQ_CROSSING_BACKPRESSURE_INJECT, instrumented counters) · rtl/kvq_deadline_arbiter.sv (interlock) · results/rtl_sim/phase1_xsim_summary.csv (committed; raw xsim.log gitignored) in github.com/taitashaw/cxl-kv-forge-qos

5. CDC

Single clock domain — verified, not assumed. The one design clock is the MMCM output of clk_wiz_0; the constraints file contains no create_clock (only a commented example for a standalone flow), the block design derives every fabric clock from that single MMCM output, and the AXIS, AXI4-Lite, and debug interfaces all run on it. There are no cross-clock crossings, therefore no synchronizers to audit and no CDC findings to report. On a client design this section enumerates every crossing with its synchronizer structure and a max-skew check; here the content is the verification that none exist.

source: vivado/constraints.xdc (no create_clock) · vivado/create_block_design.tcl (single clk_wiz output) in github.com/taitashaw/cxl-kv-forge-qos

6. Cycle-level latency budget

The arbitration path — request in, through credit and queueing, to memory-engine issue — is documented at ~10–11 cycles (28.6–31.4 ns at the 350 MHz baseline). This audit’s changes move it to ~11–12 cycles — 27.5–30.0 ns at the re-closed 400 MHz (2.5 ns) — the register slice adds exactly one. The components verified directly against RTL structure in this audit: the slice (1 cycle), the queue manager’s registered head readout (1), the three tournament stages (3), and grant-to-pop (2). The remainder sits in the parser/lookup/issue path and is taken from the signoff document rather than re-derived. The interlock of Finding B serializes wave admission (~1 grant per 6 cycles per tenant), which the single-issue, multi-cycle memory engine downstream absorbs — it was never able to consume faster than that.

source: results/PHASE2_1_FINAL.md (10-11 cycle arbitration latency) · rtl/kvq_top.sv, kvq_per_tenant_queue_manager.sv, kvq_deadline_arbiter.sv (per-stage structure) in github.com/taitashaw/cxl-kv-forge-qos

The same budgeting discipline applied to a second design surfaces the mirror-image limiter — a logic-depth cliff rather than a routing one. It is worked below as a second finding, at the weaker README evidence tier: its numbers are documented but its timing reports are not committed, so it corroborates the method without ever standing as proof the way the primary design’s committed reports do.

From the corroborating design (README-documented): a hardened softmax engine for transformer attention, built around a five-stage pipelined exponential unit in Q8.24 fixed-point. It closes 400 MHz with WNS +0.413 ns, 16/16 tests bit-exact against a FlashAttention-2 golden model — but one stage caps the whole pipeline’s ceiling.

Stage 5 collapses two expensive operations into one combinational stage ahead of the output register: a 32-bit leading-zero detect built as a linear priority encoder, and a variable barrel shift that consumes its result. The derived shift-amount control fans out across all 32 bit positions with no register between encoder and shifter — the fanout cliff. Raise the target toward the ~480 MHz theoretical ceiling and this path fails first.

A 32-input linear priority encoder (~5 levels) and a 32-bit barrel shifter (~5 levels) collapse into one combinational stage ahead of the output register, with the shift-amount control fanning out across all 32 bit positions and no register between them.

Recommended fix: retime stage 5 into two stages (LZD, then shift) — one cycle of latency, throughput unchanged at one result per clock — and replace the linear encoder with a tree leading-zero detector, cutting encoder depth from ~5 LUT levels to ~2–3 and splitting the shift-control fanout into balanced sub-trees. Structurally this is the same prescription as Finding A — put a register where two expensive structures meet — applied to a logic-depth dominator instead of a routing dominator.

source: README.md of github.com/taitashaw/flashattn-softmax-engine (400 MHz, WNS +0.413 ns, WHS +0.017 ns, 16/16, 550 LUT / 882 FF / 0 DSP / 0 BRAM; logic-depth figures are estimates, stated as such — no timing report is committed for this design)

7. Prioritized mitigation roadmap

Ranked by measured Fmax recovery on this design — the ranking a client report would carry, with each row’s cost and payoff already demonstrated rather than estimated:

The pattern the ranking teaches: tool flags and bigger parts bought 6 MHz; structural fixes rooted in a measured worst path bought 4.4×. Remaining headroom past this audit’s result lives in deeper structural work — multi-outstanding memory issue and queue-manager BRAM packing (both already scoped in the repo’s Phase 2 plan) — not in further flag sweeps.

source: results/PHASE2_1_FINAL.md (builds, recoveries) · docs/phase1_known_limitations.md (Phase 2 scope) in github.com/taitashaw/cxl-kv-forge-qos

8. Re-closure plan — as executed

A client report’s re-closure plan is a commitment; this one is a record, because it was executed. The loop: baseline confirmed (350 MHz closed, WNS +0.004 ns) → limiter attributed at 400 MHz from the post-route report (§3) → minimum RTL change (§4) → bit-exact re-verification with strengthened coverage (13/13 + SVA) → identical four-strategy sweep re-run with retiming on and post-route phys-opt → result read from the new post-route report. Wall clock for the measured re-closure iteration: 25 minutes on a 24-thread workstation (project + 400 MHz synthesis with retiming + four parallel implementation strategies with post-route phys-opt + bitstream). One earlier build was discarded as invalid: a clock-wizard input-buffer config left the fabric clock underived, so the entire datapath timed unconstrained and the tool happily reported WNS +15.6 ns against a 2.5 ns budget — an impossible number. It was caught by reading the clock table before believing the slack, which is the first thing this audit does to any report it is handed. Constraints were not touched at any step — the 400 MHz target is set by the block design’s MMCM, so before/after reports answer the same question.

source: vivado/synth_impl_bitstream.tcl (flow) · results/impl/phase2_strategy_sweep.md (committed result) · wall-clock read from the auditor's local synth run log (results/synth/synth.log, gitignored — not clone-reproducible) in github.com/taitashaw/cxl-kv-forge-qos

9. Scope and limitations

What this design is not, stated as bluntly as the repo states it. These boundaries are the trust signal of the report: numbers above are claims about this RTL on this part, nothing wider.

source: docs/phase1_known_limitations.md (verbatim scope boundaries) in github.com/taitashaw/cxl-kv-forge-qos

10. Appendix — reproduction + raw report

Every result in this report regenerates from a clean clone with two commands. No GUI steps, no hidden state.

git clone https://github.com/taitashaw/cxl-kv-forge-qos
cd cxl-kv-forge-qos

# 13/13 XSim regression + SVA (writes results/rtl_sim/xsim.log)
bash scripts/run_xsim.sh

# 400 MHz synth + 4-strategy post-route sweep on xczu7ev (writes results/impl/)
VIVADO_PART=xczu7ev-ffvc1156-2-e bash scripts/run_vivado_synth.sh

Slack (MET) :             0.033ns  (required time - arrival time)
  Source:                 kvq_phase1_bd_i/kvq_top_0/inst/u_kvq_top/enqp_req_reg[kv_address][5]/C
                            (rising edge-triggered cell FDCE clocked by
                             clk_out1_kvq_phase1_bd_clk_wiz_0_0 {rise@0.000ns fall@1.250ns period=2.500ns})
  Destination:            kvq_phase1_bd_i/kvq_top_0/inst/u_kvq_top/u_qmgr/q_mem_reg[4][13][kv_address][5]/D
  Path Group:             clk_out1_kvq_phase1_bd_clk_wiz_0_0
  Path Type:              Setup (Max at Slow Process Corner)
  Requirement:            2.500ns
  Data Path Delay:        2.168ns  (logic 0.081ns (3.736%)  route 2.087ns (96.264%))
  Logic Levels:           0

The pre-fix limiter path quoted in §3 is preserved verbatim at results/impl/baseline_400mhz_prefix/post_route_timing.rpt; the regenerated post-fix sweep lives at results/impl/phase2_strategy_sweep.md with per-strategy reports under results/impl/strategy_*/. The independent 2026-06-19 re-run that reproduced WNS +0.033 ns (§4) is at results/impl/rerun_Performance_Explore/post_route_timing.rpt.