TransformerNoConv | patches=8 (size 16, stride 16) | pool=cls | D=128 heads=4 layers=3 FFN=512 | params=601,995
======================================================================
RML2016.10a — AMC training / evaluation (v2)
======================================================================
PyTorch: 2.11.0+cu130 | CUDA: True | device: cuda
GPU: Tesla T4 | 14.6 GB
Models: ['TransformerNoConv'] | run: /kaggle/working/experiments/enc3
Seed: 42 | deterministic: False

Loading RML2016.10a from /kaggle/input/datasets/tiodomm/project-rml-2016-10a/RML2016.10a_dict_optimized.pkl
Samples: (220000, 2, 128) | Labels: (220000,) | SNR: (220000,)
  train     : per (class, SNR) cell count min=600 max=600 | per class min=12000 max=12000 | per SNR min=6600 max=6600
  validation: per (class, SNR) cell count min=200 max=200 | per class min=4000 max=4000 | per SNR min=2200 max=2200
  test      : per (class, SNR) cell count min=200 max=200 | per class min=4000 max=4000 | per SNR min=2200 max=2200
  exact duplicate samples  train&test: 0 | train&val: 0 | val&test: 0
  norm stats: stored std vs train-recomputed rel diff=2.52e-08, vs all-data=2.72e-04; mean diff (std units)=4.33e-09

====================================================================================================
TransformerNoConv | recipe=improved | params=601,995 | epochs=120 | bs=256 | lr=0.0005 | wd=0.05 | warmup=5 | ls=0.1 | augment=True | clip=1.0 | AMP=True
Selection metric: high_snr (high-SNR threshold +10 dB) | early-stopping patience: 25
Training samples: 132000 | validation samples: 44000 | test samples: NOT LOADED
====================================================================================================
  ep  tr_loss  tr_acc  va_loss  va_acc   va_hi  va_+18        lr    sec
   1   2.1097  24.60%   1.6128  38.27%  53.11%  53.77%  1.94e-07    6.9  *best
   2   1.7473  38.14%   1.4941  40.39%  56.10%  54.55%  1.00e-04    6.3  *best
   3   1.6969  39.88%   1.4787  41.44%  57.64%  56.59%  2.00e-04    6.3  *best
   4   1.6265  45.45%   1.3376  49.16%  70.47%  70.50%  3.00e-04    6.4  *best
   5   1.5669  48.35%   1.3216  49.67%  71.55%  71.95%  4.00e-04    6.3  *best
   6   1.5444  49.08%   1.2888  50.59%  73.02%  72.77%  5.00e-04    6.3  *best
   7   1.5275  49.68%   1.2829  50.71%  72.85%  72.77%  5.00e-04    6.3
   8   1.5201  50.03%   1.2825  51.08%  72.83%  71.55%  5.00e-04    6.3
   9   1.5157  50.19%   1.2716  51.63%  74.41%  73.59%  4.99e-04    6.4  *best
  10   1.5079  50.58%   1.2742  51.50%  74.13%  73.50%  4.99e-04    6.4
  11   1.5036  50.93%   1.2732  51.36%  73.70%  73.05%  4.98e-04    6.3
  12   1.4999  51.20%   1.2659  51.90%  75.06%  74.64%  4.97e-04    6.4  *best
  13   1.4956  51.64%   1.2586  52.41%  75.15%  73.77%  4.95e-04    6.3  *best
  14   1.4896  52.38%   1.2462  53.53%  77.91%  77.82%  4.94e-04    6.3  *best
  15   1.4796  53.12%   1.2367  54.41%  79.20%  78.64%  4.93e-04    6.3  *best
  16   1.4719  53.61%   1.2378  54.15%  79.60%  79.09%  4.91e-04    6.4  *best
  17   1.4651  54.12%   1.2216  55.23%  81.84%  80.91%  4.89e-04    6.4  *best
  18   1.4605  54.30%   1.2100  55.45%  80.68%  80.45%  4.87e-04    6.4
  19   1.4559  54.87%   1.2050  56.03%  82.63%  81.14%  4.85e-04    6.3  *best
  20   1.4512  54.90%   1.2069  55.47%  81.70%  80.91%  4.82e-04    6.4
  21   1.4471  55.12%   1.1976  55.80%  82.05%  81.73%  4.80e-04    6.3
  22   1.4443  55.44%   1.1968  55.88%  83.15%  82.45%  4.77e-04    6.3  *best
  23   1.4427  55.47%   1.1895  56.51%  83.42%  83.55%  4.74e-04    6.3  *best
  24   1.4391  55.69%   1.1961  56.12%  83.12%  82.05%  4.71e-04    6.3
  25   1.4367  55.69%   1.1863  56.72%  83.55%  82.59%  4.67e-04    6.2  *best
  26   1.4356  55.88%   1.1840  56.61%  83.65%  82.27%  4.64e-04    6.3  *best
  27   1.4321  56.04%   1.1885  56.57%  83.49%  83.14%  4.60e-04    6.4
  28   1.4311  56.05%   1.1960  56.48%  84.55%  83.73%  4.57e-04    6.2  *best
  29   1.4289  56.33%   1.1819  57.16%  84.51%  83.64%  4.53e-04    6.3
  30   1.4273  56.42%   1.1860  56.96%  84.67%  84.18%  4.49e-04    6.3  *best
  31   1.4251  56.57%   1.1834  57.14%  85.04%  84.95%  4.44e-04    6.3  *best
  32   1.4222  56.76%   1.1739  57.45%  84.69%  84.55%  4.40e-04    6.3
  33   1.4217  56.82%   1.1831  57.52%  85.04%  84.23%  4.36e-04    6.3
  34   1.4193  57.16%   1.1697  57.79%  85.71%  84.95%  4.31e-04    6.5  *best
  35   1.4168  57.22%   1.1683  58.02%  86.30%  85.68%  4.26e-04    6.3  *best
  36   1.4148  57.58%   1.1731  57.86%  86.75%  86.45%  4.21e-04    6.3  *best
  37   1.4123  57.70%   1.1650  58.23%  86.54%  86.18%  4.16e-04    6.3
  38   1.4093  57.89%   1.1555  58.79%  86.79%  86.18%  4.11e-04    6.3  *best
  39   1.4077  58.12%   1.1520  59.13%  88.14%  87.86%  4.06e-04    6.4  *best
  40   1.4047  58.34%   1.1526  59.17%  88.36%  87.95%  4.01e-04    6.3  *best
  41   1.4023  58.49%   1.1475  59.58%  89.15%  89.18%  3.95e-04    6.3  *best
  42   1.4012  58.61%   1.1526  59.33%  88.70%  88.59%  3.90e-04    6.3
  43   1.3996  58.73%   1.1396  60.03%  89.39%  89.32%  3.84e-04    6.3  *best
  44   1.3978  58.82%   1.1427  59.58%  88.47%  87.59%  3.78e-04    6.3
  45   1.3956  58.95%   1.1367  60.12%  89.54%  89.32%  3.72e-04    6.3  *best
  46   1.3926  59.07%   1.1364  60.05%  89.35%  89.14%  3.66e-04    6.3
  47   1.3916  59.17%   1.1304  60.26%  89.75%  89.18%  3.60e-04    6.2  *best
  48   1.3900  59.27%   1.1414  60.02%  89.89%  90.09%  3.54e-04    6.3  *best
  49   1.3876  59.44%   1.1288  60.40%  90.26%  89.32%  3.48e-04    6.3  *best
  50   1.3852  59.60%   1.1345  60.47%  90.59%  90.68%  3.42e-04    6.2  *best
  51   1.3835  59.74%   1.1209  60.79%  90.27%  89.91%  3.35e-04    6.3
  52   1.3816  59.72%   1.1296  60.45%  90.40%  90.00%  3.29e-04    6.3
  53   1.3798  60.00%   1.1274  60.58%  90.10%  89.95%  3.23e-04    6.3
  54   1.3769  60.16%   1.1214  60.75%  90.65%  90.27%  3.16e-04    6.3  *best
  55   1.3764  60.17%   1.1230  60.83%  90.77%  90.32%  3.09e-04    6.3  *best
  56   1.3748  60.25%   1.1200  60.89%  90.25%  89.23%  3.03e-04    6.3
  57   1.3733  60.28%   1.1175  60.92%  90.34%  89.86%  2.96e-04    7.1
  58   1.3721  60.40%   1.1167  61.15%  90.83%  90.45%  2.90e-04    6.3  *best
  59   1.3710  60.42%   1.1153  61.22%  91.12%  91.05%  2.83e-04    6.3  *best
  60   1.3696  60.51%   1.1190  61.19%  91.20%  90.95%  2.76e-04    6.3  *best
  61   1.3682  60.62%   1.1141  61.14%  91.10%  91.14%  2.69e-04    6.3
  62   1.3662  60.69%   1.1161  61.10%  91.35%  91.32%  2.63e-04    6.4  *best
  63   1.3660  60.72%   1.1150  61.12%  91.02%  91.18%  2.56e-04    6.3
  64   1.3650  60.73%   1.1092  61.37%  91.37%  91.32%  2.49e-04    6.3  *best
  65   1.3637  60.87%   1.1092  61.50%  91.57%  91.05%  2.42e-04    6.5  *best
  66   1.3621  61.05%   1.1107  61.44%  91.43%  91.36%  2.36e-04    6.5
  67   1.3627  60.96%   1.1152  61.13%  91.25%  91.05%  2.29e-04    6.3
  68   1.3616  61.02%   1.1072  61.38%  91.69%  91.00%  2.22e-04    6.4  *best
  69   1.3595  61.24%   1.1026  61.57%  91.75%  91.55%  2.15e-04    6.3  *best
  70   1.3579  61.17%   1.1088  61.52%  91.88%  91.32%  2.09e-04    6.4  *best
  71   1.3564  61.34%   1.1058  61.62%  91.59%  91.41%  2.02e-04    6.3
  72   1.3563  61.33%   1.1064  61.58%  91.92%  91.55%  1.96e-04    6.3  *best
  73   1.3545  61.41%   1.1073  61.45%  91.29%  91.14%  1.89e-04    6.3
  74   1.3545  61.40%   1.1067  61.45%  91.89%  91.77%  1.82e-04    6.4
  75   1.3539  61.37%   1.0989  61.82%  92.06%  92.05%  1.76e-04    6.3  *best
  76   1.3530  61.46%   1.1002  61.69%  92.07%  91.91%  1.70e-04    6.3  *best
  77   1.3513  61.62%   1.0994  61.69%  92.04%  91.82%  1.63e-04    6.3
  78   1.3513  61.58%   1.1000  61.88%  92.15%  91.95%  1.57e-04    6.3  *best
  79   1.3483  61.83%   1.0998  61.83%  92.33%  91.86%  1.51e-04    6.4  *best
  80   1.3495  61.72%   1.0997  61.85%  92.16%  91.77%  1.45e-04    6.4
  81   1.3481  61.89%   1.0987  61.88%  92.05%  92.09%  1.39e-04    6.3
  82   1.3471  61.77%   1.0968  62.00%  91.85%  91.23%  1.33e-04    6.3
  83   1.3465  61.87%   1.0963  61.89%  91.85%  91.50%  1.27e-04    6.3
  84   1.3467  61.86%   1.0977  61.92%  92.15%  91.68%  1.21e-04    6.3
  85   1.3442  61.98%   1.1029  61.72%  91.83%  91.45%  1.15e-04    6.3
  86   1.3442  61.97%   1.0943  62.12%  92.23%  91.64%  1.10e-04    6.4
  87   1.3435  62.01%   1.0984  61.96%  92.34%  91.95%  1.04e-04    6.3  *best
  88   1.3425  62.08%   1.0929  62.16%  92.35%  91.73%  9.89e-05    6.3  *best
  89   1.3417  62.14%   1.0995  61.91%  92.36%  92.18%  9.37e-05    6.3  *best
  90   1.3422  62.06%   1.0924  62.23%  92.63%  92.27%  8.86e-05    6.4  *best
  91   1.3401  62.14%   1.0935  62.19%  92.15%  91.82%  8.36e-05    6.4
  92   1.3402  62.20%   1.0948  62.27%  92.40%  92.00%  7.87e-05    6.3
  93   1.3386  62.27%   1.0951  62.11%  92.48%  92.00%  7.39e-05    6.4
  94   1.3387  62.27%   1.0930  62.17%  92.31%  91.68%  6.93e-05    6.3
  95   1.3380  62.38%   1.0927  62.10%  92.35%  92.18%  6.48e-05    6.3
  96   1.3383  62.22%   1.0911  62.14%  92.25%  91.95%  6.05e-05    6.6
  97   1.3363  62.41%   1.0927  62.10%  92.39%  91.86%  5.63e-05    6.3
  98   1.3364  62.40%   1.0920  62.07%  92.52%  92.05%  5.23e-05    6.3
  99   1.3367  62.41%   1.0942  62.10%  92.39%  92.00%  4.84e-05    6.4
 100   1.3353  62.46%   1.0904  62.27%  92.37%  92.00%  4.46e-05    6.3
 101   1.3359  62.42%   1.0929  62.15%  92.34%  91.68%  4.10e-05    6.4
 102   1.3341  62.55%   1.0905  62.28%  92.38%  91.95%  3.76e-05    6.3
 103   1.3349  62.52%   1.0892  62.29%  92.53%  92.23%  3.43e-05    6.3
 104   1.3349  62.51%   1.0904  62.28%  92.43%  92.00%  3.12e-05    6.4
 105   1.3336  62.65%   1.0886  62.30%  92.58%  92.32%  2.83e-05    6.3
 106   1.3339  62.45%   1.0884  62.38%  92.68%  92.41%  2.55e-05    6.4  *best
 107   1.3340  62.48%   1.0903  62.30%  92.46%  92.14%  2.29e-05    6.4
 108   1.3327  62.60%   1.0893  62.27%  92.51%  92.18%  2.04e-05    6.4
 109   1.3322  62.74%   1.0891  62.26%  92.55%  92.14%  1.82e-05    6.3
 110   1.3317  62.66%   1.0887  62.32%  92.55%  92.18%  1.61e-05    6.3
 111   1.3313  62.62%   1.0897  62.31%  92.55%  92.14%  1.42e-05    6.4
 112   1.3316  62.67%   1.0890  62.26%  92.63%  92.27%  1.24e-05    6.3
 113   1.3315  62.65%   1.0893  62.32%  92.55%  91.95%  1.09e-05    6.3
 114   1.3325  62.64%   1.0890  62.33%  92.61%  92.23%  9.51e-06    6.4
 115   1.3317  62.65%   1.0886  62.31%  92.56%  92.18%  8.32e-06    6.4
 116   1.3313  62.64%   1.0884  62.37%  92.56%  92.09%  7.31e-06    6.4
 117   1.3308  62.69%   1.0887  62.32%  92.59%  92.23%  6.48e-06    6.3
 118   1.3305  62.74%   1.0889  62.32%  92.66%  92.32%  5.83e-06    6.3
 119   1.3311  62.73%   1.0890  62.31%  92.55%  92.23%  5.37e-06    6.4
 120   1.3315  62.72%   1.0885  62.30%  92.59%  92.23%  5.09e-06    6.3
Best epoch 106: val acc 62.38% | val high-SNR 92.68% | val +18 dB 92.41% | this session 12.8 min

Evaluating validation-selected TransformerNoConv checkpoint on the test set...
RML2016.10a — TransformerNoConv — final results
======================================================================
Recipe: improved | parameters: 601,995
Checkpoint selected on VALIDATION (high_snr), epoch 106
  validation accuracy (all SNR): 62.38%
  validation accuracy (SNR >= +10 dB): 92.68%
  validation accuracy (+18 dB): 92.41%

TEST (untouched until this evaluation):
  overall accuracy: 62.60%  (95% CI 62.15–63.05, n=44000)
  SNR >= +10 dB accuracy: 92.56%  (95% CI 92.06–93.04, n=11000)
  mean accuracy SNR >= 0 dB: 92.06%
  +18 dB accuracy: 92.59%  (2037/2200; 95% CI 91.42–93.61)
  Target (+18 dB >= 90%): MET (95% CI entirely above 90%)

Per-SNR test accuracy:
  -20 dB:   9.36%  (n=2200)
  -18 dB:   9.55%  (n=2200)
  -16 dB:   9.95%  (n=2200)
  -14 dB:  12.45%  (n=2200)
  -12 dB:  15.41%  (n=2200)
  -10 dB:  22.68%  (n=2200)
   -8 dB:  38.59%  (n=2200)
   -6 dB:  59.05%  (n=2200)
   -4 dB:  72.05%  (n=2200)
   -2 dB:  82.32%  (n=2200)
   +0 dB:  89.18%  (n=2200)
   +2 dB:  91.77%  (n=2200)
   +4 dB:  92.23%  (n=2200)
   +6 dB:  92.18%  (n=2200)
   +8 dB:  92.45%  (n=2200)
  +10 dB:  92.00%  (n=2200)
  +12 dB:  92.41%  (n=2200)
  +14 dB:  92.95%  (n=2200)
  +16 dB:  92.86%  (n=2200)
  +18 dB:  92.59%  (n=2200)

RML2016.10a — model comparison (test metrics from each model's validation-selected checkpoint)
======================================================================

            model   recipe  parameters  best_epoch  val_accuracy  val_accuracy_high_snr  val_accuracy_plus18  test_accuracy  test_accuracy_high_snr  test_mean_acc_snr_ge0  test_accuracy_plus18  test_accuracy_plus18_ci95_low  test_accuracy_plus18_ci95_high
TransformerNoConv improved      601995         106      0.623818               0.926818             0.924091       0.626023                0.925636               0.920636              0.925909                       0.914206                        0.936127


All outputs under: /kaggle/working/experiments/enc3
