TransformerNoConv | patches=8 (size 16, stride 16) | pool=cls | D=128 heads=4 layers=6 FFN=512 | params=1,196,811
======================================================================
RML2016.10a — AMC training / evaluation (v2)
======================================================================
PyTorch: 2.11.0+cu128 | CUDA: True | device: cuda
GPU: Tesla T4 | 14.6 GB
Models: ['TransformerNoConv'] | run: experiments/4models
Seed: 42 | deterministic: False

Loading RML2016.10a from /kaggle/input/datasets/tiodomm/project-rml-2016-10a/RML2016.10a_dict_optimized.pkl
Samples: (220000, 2, 128) | Labels: (220000,) | SNR: (220000,)
  train     : per (class, SNR) cell count min=600 max=600 | per class min=12000 max=12000 | per SNR min=6600 max=6600
  validation: per (class, SNR) cell count min=200 max=200 | per class min=4000 max=4000 | per SNR min=2200 max=2200
  test      : per (class, SNR) cell count min=200 max=200 | per class min=4000 max=4000 | per SNR min=2200 max=2200
  exact duplicate samples  train&test: 0 | train&val: 0 | val&test: 0
  norm stats: stored std vs train-recomputed rel diff=2.52e-08, vs all-data=2.72e-04; mean diff (std units)=4.33e-09

====================================================================================================
TransformerNoConv | recipe=improved | params=1,196,811 | epochs=120 | bs=256 | lr=0.0005 | wd=0.05 | warmup=5 | ls=0.1 | augment=True | clip=1.0 | AMP=True
Selection metric: high_snr (high-SNR threshold +10 dB) | early-stopping patience: 25
Training samples: 132000 | validation samples: 44000 | test samples: NOT LOADED
====================================================================================================
  ep  tr_loss  tr_acc  va_loss  va_acc   va_hi  va_+18        lr    sec
   1   2.0809  25.67%   1.5782  39.08%  54.26%  54.00%  1.94e-07   11.4  *best
   2   1.7197  39.22%   1.4894  41.86%  59.01%  58.59%  1.00e-04   10.7  *best
   3   1.6683  42.35%   1.4359  45.06%  64.66%  64.64%  2.00e-04   10.7  *best
   4   1.5838  47.64%   1.3266  49.63%  70.78%  70.82%  3.00e-04   10.6  *best
   5   1.5389  49.27%   1.3007  50.40%  72.65%  72.00%  4.00e-04   10.7  *best
   6   1.5227  49.94%   1.2982  50.50%  72.75%  72.32%  5.00e-04   10.5  *best
   7   1.5127  50.29%   1.2948  51.02%  73.95%  74.73%  5.00e-04   10.6  *best
   8   1.5064  50.52%   1.2741  51.56%  73.34%  72.59%  5.00e-04   10.7
   9   1.4993  50.93%   1.2599  52.13%  75.15%  75.00%  4.99e-04   10.6  *best
  10   1.4954  51.25%   1.2698  51.53%  73.95%  73.50%  4.99e-04   10.6
  11   1.4872  51.80%   1.2747  52.06%  74.95%  74.14%  4.98e-04   10.5
  12   1.4818  52.47%   1.2557  53.04%  77.08%  76.50%  4.97e-04   10.5  *best
  13   1.4731  53.17%   1.2395  53.85%  78.26%  77.23%  4.95e-04   10.5  *best
  14   1.4639  53.92%   1.2314  54.03%  79.40%  78.64%  4.94e-04   10.5  *best
  15   1.4558  54.63%   1.2152  55.85%  81.71%  81.50%  4.93e-04   10.5  *best
  16   1.4492  55.14%   1.2147  55.83%  82.60%  83.09%  4.91e-04   10.6  *best
  17   1.4417  55.72%   1.2055  56.03%  82.70%  82.45%  4.89e-04   10.6  *best
  18   1.4347  56.33%   1.1801  57.57%  84.82%  84.59%  4.87e-04   10.6  *best
  19   1.4269  56.96%   1.1742  58.14%  86.50%  86.18%  4.85e-04   10.6  *best
  20   1.4192  57.46%   1.1786  58.01%  85.56%  85.41%  4.82e-04   10.7
  21   1.4140  57.80%   1.1650  58.76%  87.78%  87.55%  4.80e-04   10.5  *best
  22   1.4083  58.24%   1.1555  59.28%  88.53%  88.82%  4.77e-04   10.5  *best
  23   1.4047  58.66%   1.1477  59.56%  88.58%  89.05%  4.74e-04   10.6  *best
  24   1.3989  59.01%   1.1514  59.56%  88.75%  88.91%  4.71e-04   10.6  *best
  25   1.3950  59.14%   1.1510  59.61%  89.16%  89.09%  4.67e-04   10.6  *best
  26   1.3926  59.26%   1.1421  60.06%  89.15%  88.91%  4.64e-04   10.6
  27   1.3874  59.64%   1.1379  60.15%  89.39%  89.41%  4.60e-04   10.6  *best
  28   1.3852  59.70%   1.1357  60.38%  90.56%  91.00%  4.57e-04   10.6  *best
  29   1.3811  59.98%   1.1267  60.62%  90.24%  90.18%  4.53e-04   10.5
  30   1.3790  60.16%   1.1343  60.64%  90.85%  90.77%  4.49e-04   10.5  *best
  31   1.3755  60.29%   1.1304  60.52%  90.31%  90.55%  4.44e-04   10.6
  32   1.3751  60.35%   1.1338  60.56%  90.12%  90.23%  4.40e-04   11.0
  33   1.3719  60.48%   1.1285  60.78%  90.56%  90.73%  4.36e-04   11.2
  34   1.3709  60.56%   1.1160  61.21%  91.37%  91.27%  4.31e-04   11.0  *best
  35   1.3684  60.69%   1.1339  60.51%  90.42%  89.73%  4.26e-04   11.0
  36   1.3677  60.67%   1.1209  60.94%  91.09%  91.23%  4.21e-04   11.1
  37   1.3656  60.86%   1.1133  61.46%  91.51%  91.50%  4.16e-04   11.2  *best
  38   1.3624  61.13%   1.1164  61.31%  91.18%  90.86%  4.11e-04   11.1
  39   1.3607  61.10%   1.1174  61.44%  91.04%  90.95%  4.06e-04   10.6
  40   1.3594  61.21%   1.1079  61.80%  92.01%  92.00%  4.01e-04   10.6  *best
  41   1.3585  61.25%   1.1101  61.59%  91.90%  92.18%  3.95e-04   10.5
  42   1.3558  61.37%   1.1117  61.63%  91.94%  92.09%  3.90e-04   10.6
  43   1.3556  61.39%   1.1055  61.73%  92.08%  91.59%  3.84e-04   10.5  *best
  44   1.3529  61.48%   1.1079  61.70%  91.63%  91.50%  3.78e-04   10.6
  45   1.3519  61.55%   1.1069  61.73%  92.00%  92.09%  3.72e-04   10.5
  46   1.3486  61.80%   1.1125  61.53%  91.44%  91.36%  3.66e-04   10.6
  47   1.3510  61.69%   1.1084  61.75%  91.38%  91.14%  3.60e-04   10.6
  48   1.3478  61.79%   1.1074  61.83%  92.52%  92.45%  3.54e-04   10.8  *best
  49   1.3471  61.83%   1.1106  61.74%  91.42%  91.18%  3.48e-04   10.6
  50   1.3455  61.92%   1.1106  61.74%  92.45%  92.14%  3.42e-04   10.6
  51   1.3445  62.01%   1.0982  62.06%  91.88%  91.68%  3.35e-04   10.6
  52   1.3432  62.03%   1.1022  61.83%  92.05%  91.82%  3.29e-04   10.6
  53   1.3405  62.25%   1.1167  61.42%  91.59%  91.32%  3.23e-04   10.6
  54   1.3406  62.22%   1.1030  61.89%  92.39%  92.59%  3.16e-04   10.6
  55   1.3391  62.37%   1.0992  62.13%  92.42%  92.68%  3.09e-04   10.6
  56   1.3383  62.32%   1.1052  61.75%  91.95%  91.91%  3.03e-04   10.5
  57   1.3376  62.39%   1.1045  61.74%  91.78%  91.55%  2.96e-04   10.6
  58   1.3365  62.40%   1.0997  62.06%  92.21%  92.09%  2.90e-04   11.0
  59   1.3339  62.52%   1.1005  62.04%  92.32%  92.05%  2.83e-04   10.6
  60   1.3334  62.61%   1.1046  62.05%  92.17%  92.00%  2.76e-04   10.7
  61   1.3322  62.62%   1.1021  61.86%  92.02%  91.77%  2.69e-04   10.5
  62   1.3306  62.68%   1.1013  62.12%  92.55%  92.68%  2.63e-04   10.6  *best
  63   1.3295  62.71%   1.1033  62.00%  92.49%  92.45%  2.56e-04   10.6
  64   1.3289  62.73%   1.0977  62.11%  92.67%  92.86%  2.49e-04   10.6  *best
  65   1.3273  62.82%   1.0947  62.24%  92.65%  92.73%  2.42e-04   10.6
  66   1.3265  62.89%   1.0984  62.11%  92.55%  92.50%  2.36e-04   10.6
  67   1.3248  63.05%   1.0991  62.13%  92.31%  91.95%  2.29e-04   10.6
  68   1.3245  63.01%   1.0943  62.21%  92.62%  92.64%  2.22e-04   10.5
  69   1.3229  63.14%   1.0956  62.12%  92.54%  92.50%  2.15e-04   10.6
  70   1.3228  63.08%   1.0962  62.15%  92.48%  92.32%  2.09e-04   10.5
  71   1.3203  63.33%   1.0966  62.25%  92.57%  92.27%  2.02e-04   10.6
  72   1.3180  63.46%   1.1007  62.10%  92.38%  92.05%  1.96e-04   10.6
  73   1.3177  63.40%   1.0974  62.25%  92.54%  92.32%  1.89e-04   10.5
  74   1.3161  63.52%   1.0970  62.19%  92.55%  92.41%  1.82e-04   10.7
  75   1.3161  63.51%   1.0935  62.48%  92.79%  92.73%  1.76e-04   10.6  *best
  76   1.3147  63.52%   1.0979  62.14%  92.46%  92.36%  1.70e-04   10.6
  77   1.3129  63.66%   1.0975  62.37%  92.59%  92.55%  1.63e-04   10.6
  78   1.3120  63.63%   1.0927  62.40%  92.83%  92.77%  1.57e-04   10.6  *best
  79   1.3106  63.81%   1.0960  62.37%  92.79%  92.73%  1.51e-04   10.6
  80   1.3095  63.83%   1.0942  62.33%  92.58%  92.32%  1.45e-04   10.6
  81   1.3085  63.87%   1.0949  62.38%  92.55%  92.36%  1.39e-04   10.6
  82   1.3073  63.95%   1.0982  62.27%  92.17%  91.91%  1.33e-04   10.6
  83   1.3074  63.89%   1.0984  62.21%  92.25%  92.18%  1.27e-04   10.6
  84   1.3050  64.11%   1.0935  62.32%  92.58%  92.23%  1.21e-04   10.5
  85   1.3040  64.11%   1.1035  62.16%  92.31%  92.05%  1.15e-04   10.5
  86   1.3034  64.13%   1.0936  62.42%  92.55%  92.27%  1.10e-04   10.5
  87   1.3025  64.22%   1.0942  62.30%  92.89%  92.82%  1.04e-04   10.5  *best
  88   1.3022  64.28%   1.0921  62.43%  92.64%  92.32%  9.89e-05   10.6
  89   1.3002  64.33%   1.0997  62.27%  92.73%  92.59%  9.37e-05   10.6
  90   1.2993  64.32%   1.0939  62.48%  93.11%  92.73%  8.86e-05   10.5  *best
  91   1.2982  64.37%   1.0978  62.39%  92.46%  92.32%  8.36e-05   10.6
  92   1.2972  64.45%   1.0981  62.36%  92.55%  92.27%  7.87e-05   10.6
  93   1.2964  64.56%   1.1003  62.37%  92.81%  92.50%  7.39e-05   10.6
  94   1.2944  64.62%   1.0981  62.33%  92.64%  92.50%  6.93e-05   10.6
  95   1.2951  64.55%   1.0966  62.39%  92.76%  92.64%  6.48e-05   10.6
  96   1.2934  64.69%   1.0976  62.38%  92.63%  92.32%  6.05e-05   10.6
  97   1.2925  64.77%   1.0987  62.40%  92.62%  92.27%  5.63e-05   11.0
  98   1.2932  64.72%   1.0963  62.47%  92.89%  92.68%  5.23e-05   11.0
  99   1.2929  64.71%   1.0998  62.28%  92.71%  92.55%  4.84e-05   11.1
 100   1.2907  64.82%   1.0975  62.43%  92.65%  92.32%  4.46e-05   11.1
 101   1.2902  64.87%   1.1014  62.30%  92.38%  92.05%  4.10e-05   11.1
 102   1.2905  64.85%   1.0989  62.42%  92.73%  92.55%  3.76e-05   11.0
 103   1.2891  64.95%   1.0979  62.40%  92.92%  92.68%  3.43e-05   10.6
 104   1.2895  64.90%   1.1004  62.33%  92.45%  92.32%  3.12e-05   10.6
 105   1.2878  64.97%   1.0977  62.42%  92.67%  92.59%  2.83e-05   10.6
 106   1.2883  64.94%   1.0983  62.47%  92.66%  92.45%  2.55e-05   10.7
 107   1.2870  65.02%   1.1004  62.40%  92.55%  92.27%  2.29e-05   10.6
 108   1.2868  65.04%   1.1003  62.34%  92.58%  92.41%  2.04e-05   10.6
 109   1.2867  65.09%   1.0991  62.43%  92.73%  92.55%  1.82e-05   10.6
 110   1.2857  65.12%   1.0986  62.46%  92.65%  92.41%  1.61e-05   10.6
 111   1.2865  65.10%   1.1005  62.39%  92.65%  92.50%  1.42e-05   10.6
 112   1.2855  65.12%   1.0995  62.50%  92.65%  92.45%  1.24e-05   10.7
 113   1.2858  65.06%   1.0999  62.40%  92.59%  92.32%  1.09e-05   10.6
 114   1.2853  65.15%   1.1000  62.44%  92.69%  92.55%  9.51e-06   10.7
 115   1.2843  65.13%   1.1006  62.44%  92.68%  92.41%  8.32e-06   10.6
Early stopping: no improvement in 'high_snr' for 25 epochs.
Best epoch 90: val acc 62.48% | val high-SNR 93.11% | val +18 dB 92.73% | this session 20.6 min

Evaluating validation-selected TransformerNoConv checkpoint on the test set...
RML2016.10a — TransformerNoConv — final results
======================================================================
Recipe: improved | parameters: 1,196,811
Checkpoint selected on VALIDATION (high_snr), epoch 90
  validation accuracy (all SNR): 62.48%
  validation accuracy (SNR >= +10 dB): 93.11%
  validation accuracy (+18 dB): 92.73%

TEST (untouched until this evaluation):
  overall accuracy: 62.76%  (95% CI 62.31–63.21, n=44000)
  SNR >= +10 dB accuracy: 93.16%  (95% CI 92.68–93.62, n=11000)
  mean accuracy SNR >= 0 dB: 92.60%
  +18 dB accuracy: 93.36%  (2054/2200; 95% CI 92.25–94.33)
  Target (+18 dB >= 90%): MET (95% CI entirely above 90%)

Per-SNR test accuracy:
  -20 dB:   9.45%  (n=2200)
  -18 dB:   9.55%  (n=2200)
  -16 dB:  10.68%  (n=2200)
  -14 dB:  12.23%  (n=2200)
  -12 dB:  15.59%  (n=2200)
  -10 dB:  23.27%  (n=2200)
   -8 dB:  39.32%  (n=2200)
   -6 dB:  56.45%  (n=2200)
   -4 dB:  71.41%  (n=2200)
   -2 dB:  81.27%  (n=2200)
   +0 dB:  89.50%  (n=2200)
   +2 dB:  92.55%  (n=2200)
   +4 dB:  92.68%  (n=2200)
   +6 dB:  92.50%  (n=2200)
   +8 dB:  92.91%  (n=2200)
  +10 dB:  92.68%  (n=2200)
  +12 dB:  93.18%  (n=2200)
  +14 dB:  93.55%  (n=2200)
  +16 dB:  93.05%  (n=2200)
  +18 dB:  93.36%  (n=2200)

RML2016.10a — model comparison (test metrics from each model's validation-selected checkpoint)
======================================================================

            model   recipe  parameters  best_epoch  val_accuracy  val_accuracy_high_snr  val_accuracy_plus18  test_accuracy  test_accuracy_high_snr  test_mean_acc_snr_ge0  test_accuracy_plus18  test_accuracy_plus18_ci95_low  test_accuracy_plus18_ci95_high
              CNN   legacy      407243          19      0.571932               0.890182             0.880000       0.572636                0.888364               0.874682              0.892273                       0.878627                        0.904551
        MobileNet   legacy      534667           4      0.538636               0.829000             0.830455       0.539136                0.826636               0.808773              0.824545                       0.808089                        0.839870
           ResNet   legacy     3849995           7      0.558614               0.836273             0.836818       0.558523                0.838636               0.827818              0.838636                       0.822676                        0.853416
TransformerNoConv improved     1196811          90      0.624795               0.931091             0.927273       0.627591                0.931636               0.925955              0.933636                       0.922461                        0.943300


All outputs under: experiments/4models
