MR-Transformer on FPGA — Front End (Guide Architecture)

Roadmap · Blueprint · Folder guide · Results


1. What this package contains

MRT_FrontEnd_GuideArch\
├── 00_ROADMAP.html / .md        ← this document (open the .html in a browser)
├── 01_guide_notes\               ← the 3 pages of the guide's hand-drawn architecture
├── 02_results\                   ← function sheet + check sheets (Excel) + fold proof
├── S1_input_rom_raw_src\         ← Step 1   Input ROM (I, Q)
├── S2_s2p\                       ← Step 2   S2P1 / S2P2 shift registers
├── S4_patch_embed_src\           ← Step 3+4 ROM-W' + matrix-vector multiplier
├── S5_token_pos_src\             ← Step 5+6 Token ROM + MUX + RAM-E + ROM-P' + adder
└── S7_front_end_src\             ← Step 7   complete front end (all blocks, one top level)

Every S.. folder is one Vitis HLS component source: hls_config.cfg, src\ (hardware C++), tb\ (testbench), data\ (ROM contents + reference values) and a README.txt.


2. Guide's architecture (from 01_guide_notes)

Guide page What the guide drew
Page 1 – "MRT functions" Input ROM: I-part 1×128 → I_out, Q-part 1×128 → Q_out, per clock one output.
Learnable weight matrix ROM (W): 32 ROMs, each 1×128 (ROM1 → W1 … ROM32 → W32) = ROM-W.
"Similar structure for ROM-P (position vectors)".
Page 2 – "Architecture" Input ROM → S2P1 (I) and S2P2 (Q) → 16 + 16 values → Matrix-vector multiplier ← 32 weights from ROM-W → MUX ("mux needed") → RAM-E with TOKEN → RAM-E → MUX → adder (+) ← MUX ← ROM-P → write back into RAM-E.
Page 3 Token ROM: 1 ROM of size 1×128.
Embedding RAM (RAM_E): "32 RAMs each 1×128".
Serial-to-parallel register (S2P): x → R1 → R2 → R3 … → R16, outputs y1 … y16.
Guide page 1
Guide page 2
Guide page 3

Blueprint of the implemented design

Step 7 · mrt_front (dataflow) · S7_front_end_src ① Input ROMI, Q 1×128 rawStep 1 · S1_input_rom_raw_src ② S2P1 + S2P2R1…R16 → y1…y16Step 2 · S2_s2p ③ ROM-W′ (32 ROMs)W/std, 18-bit, 1×128 each ④ Matrix-vector mult32 DSP MAC / clockStep 4 · S4_patch_embed_src Token ROMCLS 1×128 ⑤ MUX → RAM-E9 × 128, 2-port BRAMSteps 5+6 · S5_token_pos_src ROM-P′9×128, P + b′ ⑥ adder (+)E + P′ 800-bit patch ⑦ write back → Encoder 0 input (9 × 128)

Data flow (order of operations)

 ① Input ROM ──I──► ② S2P1 (R1..R16) ──16──┐
   (1x128 each) ─Q──► ② S2P2 (R1..R16) ──16──┼──► ④ Matrix-vector multiplier ──► ⑤ MUX ──► RAM-E (9 x 128)
                                             │        (32 multiplies + sum)           ▲           │
                 ③ ROM-W' (32 ROMs) ───32────┘                                   TOKEN ROM        ▼
                                                                                    ⑥ adder (+) ◄── ROM-P'
                                                                                        │
                                                                    ⑦ write back ◄──────┘ ──► to Encoder 0

3. Guide steps → our steps (the blueprint)

Order Guide block Function (function sheet) Our step Folder HLS top
0 (not drawn) (raw − channel_mean) / channel_std removed – folded into ROM-W′ and ROM-P′ S1_input_rom_raw_src\rom_data –
① Input ROM (I, Q 1×128, 1 output per clock) input (raw signal 2×128) Step 1 S1_input_rom_raw_src input_rom
② S2P1 + S2P2 (R1…R16, y1…y16) extract_patches (2,128) → (8,32) Step 2 S2_s2p s2p
①→② wiring Input ROM → S2P – Step 3 (wiring only, done in Step 7) – –
③④ ROM-W (32 ROMs 1×128) + matrix-vector multiplier patch_embedding → linear (8,32) → (8,128) Step 4 S4_patch_embed_src patch_embed
⑤ Token ROM + MUX → RAM-E add_token_and_positions (part 1) Step 5 S5_token_pos_src token_pos
⑥⑦ ROM-P + adder + write back add_token_and_positions (8,128) → (9,128) Step 6 (same block as 5) S5_token_pos_src token_pos
all the complete drawing on page 2 functions 1–3 together Step 7 S7_front_end_src mrt_front

Differences from the hand-drawn sketch (and why)

Sketch Our design Reason
No normalization block (x − mean)/std removed, merged into ROM-W′ = W/std and ROM-P′ raw data in the ROM, no retraining, mathematically identical (proof: max error 9.1e-13)
No bias ROM patch bias b′ added into ROM-P′ rows 1–8 nn.Linear(32→128) has a bias; this keeps the drawing unchanged
RAM-E "32 RAMs × 1×128" RAM-E = 9 × 128 (token + 8 patches) the output of add_token_and_positions is (9,128)
ROM-P "similar structure" ROM-P′ = 9 × 128 one position vector per token
no control drawn HLS generates the counters, MUX selects and handshakes from the C++ loops
no hold register patch held for its 128 multiplier clocks (x[32] in Step 4) S2P fills every 16 clocks, multiplier needs 128 clocks per patch

4. The normalization removal (Step 0)

norm_stats.npz: mean I = 7.121e-05, mean Q = −6.219e-04, std I = 0.00581268, std Q = 0.00624442

normal :  x_n = (raw − mean)/std ;  E = W·x_n + b ;  out = [token ; E] + P
folded :  E′  = W′·raw            ;  out = [token ; E′] + P′
          W′[:,k] = W[:,k] / std        (k = 0..15 → std_I , k = 16..31 → std_Q)
          b′      = b − Σk W[:,k]·mean/std
          P′[0]   = P[0] ;  P′[t] = P[t] + b′  (t = 1..8)

Proof: S1_input_rom_raw_src\rom_data\fold_check.py → 02_results\fold_check_result.txt (normal path and folded path both match Token_position_output_calculated.csv with max error 9.1e-13).


5. Step by step: folder · purpose · output · results

Step 1 – Input ROM · folder S1_input_rom_raw_src · top input_rom

Step 2 – S2P1 / S2P2 · folder S2_s2p · top s2p

Step 3 – Input ROM → S2P (wiring only)

Step 4 – ROM-W′ + matrix-vector multiplier · folder S4_patch_embed_src · top patch_embed

Step 5 + 6 – Token ROM + MUX + RAM-E + ROM-P′ + adder + write-back · folder S5_token_pos_src · top token_pos

Step 7 – Complete front end · folder S7_front_end_src · top mrt_front


6. Summary of results

Step Block Error vs model Clocks BRAM DSP FF LUT Est. clock
1 Input ROM 2.98e-08 (dataset) 130 2 0 11 81 6.764 ns
2 S2P1 + S2P2 5.1e-06 (Patch_output) 130 0 0 1,562 105 5.024 ns
4 ROM-W′ + multiplier 1.65e-05 (Patch_embedding) 1,061 32 32 3,776 1,224 6.548 ns
5+6 Token / RAM-E / ROM-P′ 1.88e-05 (Token_position) 2,313 6 0 135 430 6.610 ns
7 Front end (dataflow) 1.88e-05 2,352 40 32 5,888 2,158 6.904 ns

XC7S50 use by the complete front end: BRAM 27 % (of 150), DSP 27 % (of 120), FF 9 % (of 65,200), LUT 7 % (of 32,600). Dataflow is 35 % faster than running the blocks one after another (3,634 clocks).

Full table with notes: 02_results\RML2016_Transformer_Functions_FPGA.xlsx → sheet FPGA Hardware Trace (the guide's function sheet, original sheets unchanged, plus "FPGA Hardware Trace" and "Step1 Input ROM per clock").


7. Setting up on another PC

  1. Copy the zip and extract it, e.g. to C:\Users\<you>\Documents\MRT_FrontEnd_GuideArch\ (check there is no double folder: S1_input_rom_raw_src etc. must be directly inside).
  2. Open Vitis 2026.1. For each step: File → New Component → HLS
  3. Run in the Flow panel: C SIMULATION → C SYNTHESIS → C/RTL COSIMULATION. Expected results are in each README.txt and in section 6.
  4. Waveform (no license needed): cmd → cd …\<component>\<top>\hls\sim\verilog → C:\AMDDesignTools\2026.1\Vivado\bin\xsim.bat --gui <top>.wdb → File → Simulation Waveform → Open Configuration → <top>.wcfg. Each README has a one-line Tcl command that adds the per-clock signals.
  5. All paths inside the configs and sources are relative, so nothing needs editing.

Common pitfalls seen so far: double folder after "Extract All"; component location set inside the source folder (workspace switch); component name equal to the folder name except for capital letters (Windows treats them as the same).


8. Next steps

  1. Vivado implementation of mrt_front (Flow → IMPLEMENTATION → Run): real timing (WNS) and resources for the report.
  2. Optimization: add ROM-P′ while writing RAM-E (one pass) → front end ≈ 1,200 clocks (≈ 2× faster).
  3. Encoder 0 (LayerNorm → Q/K/V → attention → FFN), starting from RAM-E (9,128) — ask the guide for an encoder sketch.