Classifying blood cells with a Vision Transformer
Many blood tests still include a person sorting white cells by eye under a microscope. I fine-tuned a pretrained Vision Transformer to sort them into eight types from 46,232 microscope images, on a free Colab GPU in 40 minutes, then checked where it goes wrong, not just how often it is right.
Rows are the true cell type, columns what the model said. Everything on the diagonal is right. The orange cells are the mistakes, and they are not spread evenly: 17 neutrophils called plasma cells and 14 monocytes called lymphocytes are 31 of the 73 errors.
Monocytes and lymphocytes are both round white cells with a single nucleus, a pair people mix up too. Basophils are the rarest type here (152 test images) and still reach 0.971 F1.
| Cell type | Test images | Precision | Recall | F1 |
|---|---|---|---|---|
| Basophil | 152 | 0.950 | 0.993 | 0.971 |
| Eosinophil | 732 | 0.992 | 0.985 | 0.988 |
| Erythroblast | 155 | 0.981 | 1.000 | 0.990 |
| Lymphocyte | 793 | 0.978 | 0.995 | 0.986 |
| Monocyte | 531 | 0.990 | 0.966 | 0.978 |
| Neutrophil | 1,737 | 0.992 | 0.984 | 0.988 |
| Plasma cell | 289 | 0.936 | 0.965 | 0.951 |
| Platelet | 235 | 1.000 | 0.996 | 0.998 |
Plasma cells have the lowest precision (0.936): when the model says “plasma cell”, about 1 in 16 times it is something else, usually a neutrophil.
The honest next test is a split by source: train on some collections and test on one the model has never seen, which is also closer to what a new hospital's microscope would look like. That number will be lower, and it is the one that matters.
This is a learning project, not a diagnostic tool. Clinical use would need clinically validated data and regulatory review.
Data
46,232 images of 8 cell types (Hugging Face: ehottl/blood_dataset), split 80/10/10 with class proportions kept in every split, seed 42.
Model
google/vit-base-patch16-224, pretrained on ImageNet, with its 1,000-class head replaced by an 8-class one.
Training
Hugging Face Trainer on a free Colab T4: 4 epochs, batch 32, learning rate 2e-5 with 10% warm-up, mixed precision. Random crops, flips and small rotations for training images only.
Choosing the model
The checkpoint kept is the one with the best validation macro F1, not accuracy, so a model that ignores rare cell types cannot win. All settings live in one config file, so the run can be repeated.
Validation accuracy jumps from 95.2% to 98.1% between epochs 1 and 2, then gains only 0.4 points over the next two. More epochs would add little; better data (the split above) would matter more.
| Epoch | Train loss | Validation loss | Validation accuracy | Validation macro F1 |
|---|---|---|---|---|
| 1 | 0.2083 | 0.1307 | 95.2% | 95.9% |
| 2 | 0.1422 | 0.0505 | 98.1% | 98.2% |
| 3 | 0.1085 | 0.0480 | 98.2% | 98.2% |
| 4 | 0.1096 | 0.0370 | 98.5% | 98.5% |
- Accuracy hides the rare classes. With 1,737 neutrophils and 152 basophils in the test set, a model can look great while failing the rare cells. Macro F1 and a per-class table keep them visible.
- Read the confusion matrix, not just the score. The errors cluster in two pairs; that tells you what to fix, where a single number does not.
- Ask how the data was built. Reading the dataset card is what showed the pre-augmented images, and with them the reason to distrust a random split.