Special Ensembles

Given some of the insights from (best strategies across different sizes), a natural- maybe kind of wacky extension- is to say: “What happens when you ensembles across existing parcellations?

In this section we do exactly that, ensembling across 6 special subsets of existing parcellations.

Stacked Ensemble

Plotted below is a comparison between just the base stacking ensembles across random parcellations (both from fixed and across sizes) in contrast with those from these 6 special combinations of parcellations:

X

In this case, we note that most combinations of special parcellation ensembles appear to fit right into the expected scaling (with some deviance), and that the ensemble over all 82 existing parcellations outperforms any other.

Voted Ensemble

Plotted below is a comparison between just the base voting ensembles across random parcellations (both from fixed and across sizes) in contrast with those from these 6 special combinations of parcellations:

X

Here we note that the special voting ensembles over combinations of existing parcellations appear to outperform, as adjusted for size, their randomly generated counterparts. Also relative to the stacking based, where we saw that the ensemble over all 82 parcellations was the new top performer, we did not see this behavior with the voting ensemble. This could be a result of the stacking ensemble able to filter through and not weight some of the poor performing results well in the ensemble, whereas the voting ensemble always weights everything equally.

Grid Ensemble

Plotted below is a comparison between just the base grid searches across random parcellations (both from fixed and across sizes) in contrast with those from these 6 special combinations of parcellations:

X

The grid search based results show that the strategy of searching over special ensembles is somewhat effective, and scales generally about as expected.

Highest Performing Comparison

In order to evaluate the ‘All’ ensemble with these special combinations of pooled existing parcellations, we will consider a modified versions of the plot / comparisons performed here in the what’s best section. The difference here is that the voted and stacked SVM and ‘All’ ensembles are included as valid comparisons (along with all of the same results as before).

special

It’s a bit hard from this plot to parse out exactly which specific new / from the special ensembles are most competitive, so it is likely worthwhile to consider this plot along with the sortable table of results below.

If curious though, results indicate that the very best single combination, is as perhaps expected, the stacking ‘All’ ensemble across all 82 parcellations (17,420 unique parcellations) and 3 pipelines.

Results Table

This table includes just results from the subsets as plotted above - i.e., the same subsets as the best of section, but with the added special ensembles included.

The mean relative rankings shown here are as computed only between these subsets (and as averaged across target variable). averaged across and computed across parcellation and ML pipeline. Mean R2 and ROC AUC are calculated only from their relevant subsets of 22 and 23 target variables respectively.

Table columns are sortable!

Parcellation Pipeline Mean Rank Size Mean R2 Mean ROC AUC
stacked existing All 39.4667 17420 0.1107 0.6392
voted random 500 10 0 All 40.0889 5000 0.0960 0.6416
stacked icosahedron All 42.1111 6716 0.1044 0.6392
voted icosahedron All 43.0889 6716 0.0974 0.6422
voted icosahedron SVM 44.9111 6716 0.0968 0.6405
voted random 500 8 0 All 46.7111 4000 0.0957 0.6410
stacked random 500 10 1 All 51.0667 5000 0.1016 0.6373
stacked random 400 10 1 All 52.0000 4000 0.1000 0.6380
stacked random 500 10 0 All 52.0667 5000 0.1021 0.6369
stacked icosahedron SVM 52.5778 6716 0.0982 0.6386
voted random 500 10 1 All 52.6667 5000 0.0957 0.6406
stacked random 400 8 1 All 53.6222 3200 0.0994 0.6380
stacked random 500 8 1 All 54.1556 4000 0.1008 0.6376
voted random 400 10 1 All 54.7556 4000 0.0945 0.6414
voted random 500 10 0 SVM 56.2000 5000 0.0936 0.6399
stacked random 500 8 0 All 56.2889 4000 0.1017 0.6363
stacked random 400 10 0 All 57.3111 4000 0.1002 0.6366
stacked random 500 5 0 All 57.3778 2500 0.1008 0.6366
voted random 500 8 1 All 58.0889 4000 0.0953 0.6400
voted random 500 5 0 All 59.9333 2500 0.0955 0.6394
voted random 500 10 1 SVM 60.3111 5000 0.0933 0.6396
stacked random 400 8 0 All 60.3333 3200 0.1001 0.6368
voted random 500 8 0 SVM 62.1333 4000 0.0933 0.6393
voted random 400 8 1 All 63.2444 3200 0.0940 0.6405
voted random 400 10 0 All 64.1778 4000 0.0945 0.6399
stacked existing SVM 64.3778 17420 0.1046 0.6352
stacked random 300 8 1 All 64.8222 2400 0.0984 0.6362
stacked random 300 10 1 All 66.2667 3000 0.0988 0.6361
stacked random 500 5 1 All 66.8889 2500 0.0994 0.6359
voted random 400 8 0 All 67.0000 3200 0.0944 0.6394
voted random 500 8 1 SVM 67.7111 4000 0.0927 0.6389
stacked random 500 3 0 All 69.1333 1500 0.0993 0.6347
stacked random 400 5 0 All 69.4222 2000 0.0989 0.6348
stacked random 400 5 1 All 69.5778 2000 0.0983 0.6366
voted random 300 10 1 All 69.6444 3000 0.0923 0.6406
voted random 400 10 1 SVM 69.6889 4000 0.0928 0.6392
stacked difumo All 69.8444 1853 0.1001 0.6347
voted random 400 10 0 SVM 70.2444 4000 0.0930 0.6389
voted existing All 71.1556 17420 0.0893 0.6422
voted schaefer All 71.7111 5498 0.0950 0.6389
voted random 300 8 1 All 71.9333 2400 0.0919 0.6402
voted random 300 10 0 All 72.4444 3000 0.0919 0.6396
voted random 500 5 1 All 72.5778 2500 0.0945 0.6390
voted random 400 8 0 SVM 72.9556 3200 0.0926 0.6388
stacked random 500 10 0 SVM 73.4000 5000 0.0951 0.6370
stacked random 200 10 1 All 74.2000 2000 0.0975 0.6355
stacked random 500 10 1 SVM 74.2222 5000 0.0947 0.6373
stacked random 500 8 0 SVM 74.8444 4000 0.0947 0.6371
stacked schaefer All 74.8444 5498 0.1009 0.6343
voted random 500 5 0 SVM 74.9111 2500 0.0927 0.6380
stacked random 300 10 0 All 75.3333 3000 0.0989 0.6333
voted random 400 5 0 All 75.6444 2000 0.0938 0.6383
stacked random 200 10 0 All 76.3778 2000 0.0965 0.6364
voted random 400 8 1 SVM 76.4222 3200 0.0922 0.6386
stacked random 300 8 0 All 76.7556 2400 0.0983 0.6340
voted random 500 5 1 SVM 79.1111 2500 0.0919 0.6381
stacked random 200 8 1 All 79.6222 1600 0.0967 0.6350
voted random 500 3 0 All 79.8222 1500 0.0948 0.6372
stacked random 500 8 1 SVM 79.8444 4000 0.0940 0.6369
voted random 400 5 1 All 80.0889 2000 0.0935 0.6383
voted random 300 8 0 All 80.6889 2400 0.0915 0.6387
stacked random 500 3 1 All 81.1556 1500 0.0967 0.6356
stacked random 400 10 0 SVM 81.3333 4000 0.0942 0.6369
stacked random 400 10 1 SVM 81.5111 4000 0.0940 0.6365
voted random 300 10 1 SVM 81.9111 3000 0.0896 0.6390
stacked random 300 10 1 SVM 82.2222 3000 0.0921 0.6376
voted random 200 10 1 All 82.2444 2000 0.0906 0.6397
stacked random 400 8 0 SVM 83.3111 3200 0.0938 0.6366
voted schaefer SVM 83.7778 5498 0.0930 0.6372
voted existing SVM 83.9111 17420 0.0882 0.6397
voted random 400 5 0 SVM 84.6889 2000 0.0914 0.6380
stacked random 300 5 1 All 85.4000 1500 0.0957 0.6350
stacked random 500 5 0 SVM 85.5111 2500 0.0938 0.6362
stacked random 500 5 1 SVM 85.7556 2500 0.0929 0.6368
stacked random 300 8 1 SVM 86.0000 2400 0.0914 0.6375
voted difumo All 86.1333 1853 0.0925 0.6375
stacked difumo SVM 86.5333 1853 0.0940 0.6359
stacked random 400 3 0 All 87.4000 1200 0.0965 0.6338
stacked random 400 8 1 SVM 87.9778 3200 0.0934 0.6358
voted random 200 8 1 All 88.0444 1600 0.0901 0.6391
voted random 300 10 0 SVM 88.3778 3000 0.0894 0.6380
voted random 200 10 0 All 88.6444 2000 0.0897 0.6390
voted random 300 8 1 SVM 88.8000 2400 0.0891 0.6383
voted random 400 5 1 SVM 88.8667 2000 0.0910 0.6379
stacked random 400 3 1 All 88.8889 1200 0.0958 0.6354
stacked random 400 5 0 SVM 89.1333 2000 0.0923 0.6365
stacked random 200 8 0 All 89.2444 1600 0.0961 0.6351
stacked random 400 5 1 SVM 89.5333 2000 0.0921 0.6366
stacked random 300 10 0 SVM 91.1556 3000 0.0918 0.6355
stacked random 300 5 0 All 91.4000 1500 0.0968 0.6330
voted random 300 5 1 All 92.0889 1500 0.0905 0.6383
stacked random 300 8 0 SVM 93.1778 2400 0.0911 0.6359
stacked schaefer SVM 93.5778 5498 0.0942 0.6345
voted random 300 5 0 All 93.9556 1500 0.0908 0.6375
voted random 300 8 0 SVM 93.9556 2400 0.0889 0.6376
voted random 200 8 0 All 95.2444 1600 0.0895 0.6381
voted random 500 3 1 All 95.8444 1500 0.0924 0.6372
icosahedron 1442 (dlab) Existing 97.6667 2637 0.0920 0.6367
stacked random 200 5 1 All 98.7778 1000 0.0948 0.6342
stacked random 300 3 1 All 100.6889 900 0.0932 0.6354
voted random 500 3 0 SVM 100.6889 1500 0.0912 0.6361
voted random 400 3 0 All 100.7556 1200 0.0926 0.6358
voted random 500 3 1 SVM 100.7778 1500 0.0896 0.6370
voted random 400 3 1 All 101.2444 1200 0.0916 0.6373
stacked random 300 3 0 All 101.3111 900 0.0939 0.6329
stacked random 200 5 0 All 102.4667 1000 0.0946 0.6331
voted random 200 10 1 SVM 103.9333 2000 0.0872 0.6374
stacked random 500 3 0 SVM 104.6889 1500 0.0919 0.6350
stacked random 200 10 1 SVM 105.4889 2000 0.0512 0.6359
icosahedron 1002 (dlab) Existing 105.7556 1838 0.0913 0.6358
voted random 200 10 0 SVM 106.4889 2000 0.0865 0.6370
voted random 400 3 0 SVM 106.6667 1200 0.0895 0.6364
voted random 300 5 0 SVM 106.7778 1500 0.0878 0.6368
stacked random 500 3 1 SVM 106.8889 1500 0.0903 0.6357
voted difumo SVM 106.9111 1853 0.0894 0.6358
stacked random 300 5 0 SVM 106.9333 1500 0.0896 0.6355
voted random 300 5 1 SVM 108.0889 1500 0.0871 0.6368
stacked random 300 5 1 SVM 108.5556 1500 0.0887 0.6365
voted random 200 8 1 SVM 109.0889 1600 0.0867 0.6369
stacked random 200 8 1 SVM 110.6222 1600 0.0522 0.6354
voted random 300 3 1 All 111.0000 900 0.0888 0.6369
voted random 200 5 1 All 111.1778 1000 0.0888 0.6365
stacked mist All 111.9111 1037 0.0952 0.6302
stacked basc All 112.4000 1037 0.0949 0.6308
stacked random 200 10 0 SVM 113.7333 2000 0.0681 0.6327
stacked random 200 8 0 SVM 114.0444 1600 0.0864 0.6346
voted random 200 8 0 SVM 114.0444 1600 0.0862 0.6362
icosahedron 642 (dlab) Existing 114.7778 1186 0.0895 0.6350
voted random 200 5 0 All 115.6000 1000 0.0888 0.6361
stacked random 400 3 0 SVM 116.2444 1200 0.0898 0.6337
stacked random 300 3 0 SVM 116.6889 900 0.0869 0.6355
voted random 400 3 1 SVM 116.8000 1200 0.0885 0.6363
stacked random 400 3 1 SVM 116.8889 1200 0.0896 0.6352
voted random 300 3 0 All 117.1556 900 0.0890 0.6355
stacked random 100 10 1 All 117.9778 1000 0.0927 0.6299
voted random 300 3 0 SVM 121.0889 900 0.0857 0.6357
difumo 1024 Existing 121.9333 949 0.0867 0.6352
stacked random 200 3 1 All 124.4000 600 0.0910 0.6327
stacked random 100 8 1 All 124.7778 800 0.0917 0.6289
stacked random 200 5 1 SVM 124.7778 1000 0.0577 0.6346
voted random 200 5 1 SVM 126.1556 1000 0.0848 0.6355
stacked random 200 3 0 All 128.3111 600 0.0922 0.6301
stacked random 100 10 0 All 128.8889 1000 0.0917 0.6295
stacked random 300 3 1 SVM 129.3111 900 0.0861 0.6345
voted random 300 3 1 SVM 129.7556 900 0.0850 0.6351
voted random 200 5 0 SVM 130.3111 1000 0.0848 0.6350
stacked random 200 5 0 SVM 130.7333 1000 0.0789 0.6334
voted random 100 10 1 All 130.9111 1000 0.0839 0.6336
voted random 100 10 0 All 133.8000 1000 0.0833 0.6339
stacked basc SVM 135.1333 1037 0.0888 0.6316
stacked mist SVM 135.5111 1037 0.0888 0.6315
voted random 200 3 1 All 136.7111 600 0.0861 0.6335
voted random 100 8 1 All 136.8000 800 0.0833 0.6329
voted random 200 3 0 All 138.0667 600 0.0875 0.6328
stacked random 100 8 0 All 139.5111 800 0.0916 0.6279
schaefer 1000 Existing 143.0667 998 0.0864 0.6336
voted random 100 8 0 All 143.8889 800 0.0832 0.6324
icosahedron 362 (dlab) Existing 144.6889 669 0.0863 0.6328
schaefer 900 Existing 145.0000 900 0.0860 0.6330
stacked random 100 5 1 All 145.0222 500 0.0898 0.6262
schaefer 800 Existing 145.2000 800 0.0859 0.6324
schaefer 700 Existing 145.8222 700 0.0853 0.6322
difumo 512 Existing 146.8444 478 0.0836 0.6323
sjh Existing 148.3778 507 0.0833 0.6336
voted random 200 3 1 SVM 150.1111 600 0.0812 0.6331
stacked random 100 5 0 All 150.6222 500 0.0901 0.6268
voted random 200 3 0 SVM 152.9111 600 0.0822 0.6321
voted random 100 5 1 All 153.1778 500 0.0825 0.6305
stacked random 100 10 0 SVM 154.1778 1000 0.0810 0.6288
voted random 100 10 0 SVM 156.1556 1000 0.0764 0.6311
voted random 100 10 1 SVM 156.6667 1000 0.0768 0.6314
voted random 100 5 0 All 157.0222 500 0.0826 0.6302
basc scale444 Existing 158.7556 374 0.0811 0.6320
mist 444 Existing 158.7778 374 0.0811 0.6320
stacked random 200 3 0 SVM 159.2222 600 0.0839 0.6290
voted mist All 160.6222 1037 0.0804 0.6300
schaefer 600 Existing 161.0889 600 0.0824 0.6318
voted random 100 8 1 SVM 161.1333 800 0.0759 0.6306
stacked random 100 8 1 SVM 161.7111 800 0.0798 0.6282
stacked random 100 10 1 SVM 162.0667 1000 0.0811 0.6270
voted basc All 162.2000 1037 0.0805 0.6293
stacked random 100 8 0 SVM 163.2444 800 0.0804 0.6279
icosahedron 162 (dlab) Existing 163.9333 305 0.0804 0.6303
stacked random 200 3 1 SVM 164.6667 600 0.0531 0.6272
voted random 100 8 0 SVM 165.8444 800 0.0762 0.6299
stacked random 100 3 1 All 167.3333 300 0.0864 0.6246
voted mist SVM 169.8444 1037 0.0771 0.6294
voted basc SVM 169.9111 1037 0.0771 0.6294
schaefer 400 Existing 170.5778 400 0.0804 0.6300
stacked random 100 3 0 All 171.6222 300 0.0863 0.6240
schaefer 500 Existing 172.2222 500 0.0822 0.6290
Slab1068 Existing 174.2667 699 0.0804 0.6289
voted random 100 5 1 SVM 175.2222 500 0.0741 0.6285
voted random 100 5 0 SVM 175.4667 500 0.0751 0.6284
voted random 100 3 1 All 176.0222 300 0.0804 0.6269
hcp mmp Existing 176.6889 360 0.0782 0.6294
stacked random 100 5 1 SVM 176.9333 500 0.0771 0.6264
glasser 2016 (dlab) Existing 177.1333 360 0.0782 0.6294
glasser (abox) Existing 177.1778 360 0.0782 0.6294
stacked random 100 5 0 SVM 178.9778 500 0.0783 0.6234
mist 325 Existing 179.0667 273 0.0776 0.6297
Slab907 Existing 179.1333 661 0.0783 0.6277
basc scale325 Existing 179.4000 273 0.0776 0.6297
voted random 100 3 0 All 179.7556 300 0.0800 0.6261
gordon (abox) Existing 179.8889 333 0.0781 0.6286
difumo 256 Existing 180.4000 240 0.0762 0.6280
CAPRSC Existing 183.1333 333 0.0762 0.6278
aicha Existing 185.4444 341 0.0777 0.6275
shen 368 Existing 186.9778 330 0.0757 0.6281
gordon Existing 189.7778 333 0.0754 0.6263
schaefer 300 Existing 190.5556 300 0.0766 0.6271
voted random 100 3 1 SVM 191.1111 300 0.0714 0.6259
basc scale197 Existing 191.2667 165 0.0737 0.6252
stacked random 100 3 1 SVM 191.5333 300 0.0731 0.6249
mist 197 Existing 191.6000 165 0.0737 0.6252
brainnetome Existing 195.9111 216 0.0742 0.6246
fan (abox) Existing 197.6889 210 0.0749 0.6239
destrieux (abox) Existing 198.1778 150 0.0725 0.6242
stacked random 100 3 0 SVM 198.8667 300 0.0731 0.6229
voted random 100 3 0 SVM 198.8667 300 0.0713 0.6239
shen (abox) Existing 201.3778 200 0.0724 0.6243
dextrieux (dlab) Existing 203.1111 149 0.0718 0.6236
smith rsn70 Existing 204.0889 70 0.0689 0.6196
shen 268 Existing 204.1111 207 0.0747 0.6237
CPAC200 Existing 206.9778 174 0.0691 0.6218
smith bm70 Existing 207.5778 70 0.0661 0.6198
freesurfer destr Existing 208.4667 150 0.0684 0.6215
allen Existing 208.8667 75 0.0706 0.6185
schaefer 200 Existing 213.1556 200 0.0697 0.6217
baldassano (abox) Existing 214.7333 171 0.0660 0.6181
mist 122 Existing 220.8444 102 0.0661 0.6190
basc scale122 Existing 221.0000 102 0.0661 0.6190
nspn500 Existing 222.1778 156 0.0628 0.6174
difumo 128 Existing 222.4000 124 0.0672 0.6179
MICCAI Existing 225.9111 110 0.0646 0.6150
harvard oxford cort 1mm Existing 225.9556 48 0.0646 0.6150
schaefer 100 Existing 226.6444 100 0.0629 0.6151
aal Existing 228.4667 84 0.0621 0.6136
power (abox) Existing 229.1778 130 0.0640 0.6161
mist 64 Existing 229.2222 55 0.0599 0.6121
basc scale064 Existing 229.4222 55 0.0599 0.6121
brodmann Existing 231.1556 41 0.0590 0.6114
harvard oxford cort maxprob thr25 1mm Existing 231.5778 48 0.0611 0.6117
harvard oxford cort maxprob thr0 1mm Existing 233.5111 48 0.0618 0.6099
icosahedron 42 (dlab) Existing 235.5556 81 0.0553 0.6114
aal (abox) Existing 237.3333 82 0.0579 0.6093
freesurfer desikan Existing 237.9111 68 0.0515 0.6073
vdg11b Existing 238.2000 104 0.0596 0.6083
harvard oxford cort maxprob thr50 1mm Existing 240.2000 47 0.0577 0.6091
yeo (abox) Existing 241.0222 96 0.0522 0.6088
economo Existing 242.0667 43 0.0584 0.6069
desikan (dlab) Existing 243.2000 69 0.0556 0.6072
desikan (abox) Existing 245.4444 70 0.0541 0.6071
difumo 64 Existing 245.5111 62 0.0511 0.6070
mist 36 Existing 246.0222 30 0.0529 0.6030
basc scale036 Existing 246.0889 30 0.0529 0.6030
Hammersmith Existing 247.0889 63 0.0501 0.6008
power2011 (dlab) Existing 248.2000 69 0.0476 0.6068
msdl Existing 249.7111 39 0.0475 0.6004
Juelich Existing 251.0222 92 0.0476 0.6019
basc scale020 Existing 252.1778 19 0.0450 0.5956
mist 20 Existing 252.3333 19 0.0450 0.5956
yeo 17networks Existing 253.8667 17 0.0422 0.5945
yeo 7networks Existing 257.9556 7 0.0352 0.5814
craddock tcorr mean Existing 262.5556 43 0.0282 0.5823
craddock tcorr 2level Existing 263.4000 43 0.0288 0.5841
mist 7 Existing 263.6889 7 0.0278 0.5725
basc scale007 Existing 263.9333 7 0.0278 0.5725
basc scale012 Existing 263.9778 12 0.0338 0.5809
mist 12 Existing 264.0889 12 0.0338 0.5809
craddock scorr 2level Existing 266.3778 43 0.0271 0.5761
Princetonvisual top Existing 266.8000 43 0.0233 0.5679
craddock scorr mean Existing 267.5778 43 0.0260 0.5767
oasis.chubs Existing 268.2444 8 0.0224 0.5578