How We Achieved 92.45% BSL Recognition Accuracy on a 255KB Model

Our first BSL model reported 99.2% validation accuracy. We spent an afternoon feeling brilliant, and the following morning discovering that the number was worthless.

This post is the whole path: what the pipeline does, why we chose a GRU over the obvious alternatives, the contamination bug that inflated our early numbers, and what the honest figure — 92.45% on 93 signs, in a 255KB file — actually cost.

Why sign language is not a gesture-classification problem

The tempting first approach is to treat each sign as an image classification task: take a frame, predict a label. This fails immediately for British Sign Language, because BSL encodes meaning in movement.

Three specific properties break frame-wise classification:

1. Two-handed signs. Unlike ASL fingerspelling, most BSL lexical signs use both hands, often asymmetrically — one hand is the dominant articulator, the other a base. A single frame cannot tell you which is which.
2. Movement is phonemic. BSL distinguishes signs by handshape, location, orientation, and movement. Freeze the movement and you have thrown away a quarter of the phonology. AFTERNOON and NAME share a near-identical handshape and differ mainly in trajectory.
3. Non-manual features. Eyebrow position and head tilt change a statement into a question. Hands alone are not enough.

So the unit of prediction has to be a sequence, not a frame. That decision drives everything else.

The pipeline

Camera (30 fps)
   ↓
MediaPipe Holistic  →  hands + pose landmarks per frame
   ↓
Feature vector      →  258 floats per frame
   ↓
Sliding window      →  30 frames = 1 sequence
   ↓
GRU network         →  93-way softmax
   ↓
Confidence gate     →  label, or nothing

Step 1 — Landmarks, not pixels

We never feed raw pixels to the classifier. MediaPipe Holistic runs first and reduces each frame to a set of normalised landmark coordinates:

Source	Landmarks	Values per frame
Left hand	21	63 (x, y, z)
Right hand	21	63 (x, y, z)
Upper-body pose	33	132 (x, y, z, visibility)
Total	75	258

This is the single most important design decision in the whole system. Landmarks give us three things pixels do not:

• Invariance. Skin tone, lighting, sleeve colour, and background all disappear. A model trained on landmarks does not learn that signers wear dark jumpers.
• Compression. 258 floats replace a 640×480×3 frame. That is what makes a 255KB model plausible.
• A fair-by-construction input. A pixel model trained on a demographically narrow dataset encodes that narrowness. A landmark model largely cannot.

import mediapipe as mp
import numpy as np

holistic = mp.solutions.holistic.Holistic(
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5,
)

def extract_frame_features(frame_rgb):
    """Reduce one RGB frame to a 258-dim landmark vector."""
    results = holistic.process(frame_rgb)

    def hand(landmarks):
        if landmarks is None:
            return np.zeros(21 * 3)
        return np.array(
            [[p.x, p.y, p.z] for p in landmarks.landmark]
        ).flatten()

    def pose(landmarks):
        if landmarks is None:
            return np.zeros(33 * 4)
        return np.array(
            [[p.x, p.y, p.z, p.visibility] for p in landmarks.landmark]
        ).flatten()

    return np.concatenate([
        hand(results.left_hand_landmarks),    # 63
        hand(results.right_hand_landmarks),   # 63
        pose(results.pose_landmarks),         # 132
    ])                                        # -> 258

Note the zero-fill when a hand is missing. Roughly 8% of frames in our data have one hand outside the camera view or occluded. Dropping those frames would break the temporal structure; zero-filling preserves it and lets the network learn that absence is itself informative.

Step 2 — Windowing

We fix sequences at 30 frames, one second at 30fps. That number came from measurement, not intuition: we timed 400 sample signs from our recordings and found the 95th percentile of sign duration sat at 0.94 seconds. Thirty frames covers almost every sign without padding most of them into noise.

Signs shorter than 30 frames are centre-padded with the resting pose. Longer ones are uniformly subsampled rather than truncated, so the end of the movement — often the phonemically important part — survives.

Step 3 — The model

from tensorflow import keras
from tensorflow.keras import layers

def build_model(n_classes=93, timesteps=30, features=258):
    return keras.Sequential([
        layers.Input(shape=(timesteps, features)),
        layers.Masking(mask_value=0.0),

        layers.GRU(128, return_sequences=True),
        layers.Dropout(0.3),
        layers.GRU(64),
        layers.Dropout(0.3),

        layers.Dense(64, activation="relu"),
        layers.Dense(n_classes, activation="softmax"),
    ])

Three choices worth defending:

GRU over LSTM. We trained both. The LSTM variant scored 0.4 percentage points higher and was 38% larger. On a phone, 0.4 points is not worth 38% of the parameter budget. GRUs have three gates to the LSTM's four, which is most of the difference.

GRU over a Transformer. A small Transformer encoder reached comparable accuracy but needed roughly 2.4× the parameters to get there, and self-attention over 30 timesteps buys you very little — the useful dependencies in a one-second sign are local. Transformers win on long sequences. This is not a long sequence.

Two layers, not four. Depth past two layers gave us better training accuracy and worse validation accuracy on every run. With 9,400 sequences, four GRU layers is simply more capacity than the data supports.

The contamination bug

Here is the part we would rather not write about, and the reason this post exists.

Our first training run reported 99.2% validation accuracy. That should have been an alarm, not a celebration. State-of-the-art isolated sign recognition on comparable datasets sits in the high eighties to low nineties. We had apparently beaten it with a two-layer GRU and a laptop.

The bug was in how we split the data.

We recorded multiple takes of each sign from each contributor. A contributor signing HELLO five times produced five sequences. Our splitting code shuffled at the sequence level:

# WRONG — takes from the same recording session land on both sides
sequences, labels = load_all_sequences()
X_train, X_val, y_train, y_val = train_test_split(
    sequences, labels, test_size=0.2, stratify=labels, random_state=42
)

Consecutive takes from one person in one session, in one room, wearing the same clothes, are nearly identical in landmark space. Take 3 was in training and take 4 was in validation. The model was not recognising the sign. It was recognising that recording session.

The fix is to split on the group, never the sample:

from sklearn.model_selection import StratifiedGroupKFold

# RIGHT — every take from a contributor stays on one side of the split
splitter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_idx, val_idx = next(
    splitter.split(sequences, labels, groups=contributor_ids)
)

Validation accuracy fell from 99.2% to 87.1% overnight.

That drop was the most valuable result of the project. Everything since — the augmentation work, the confidence gate, the class rebalancing — was aimed at real generalisation rather than at a number we had accidentally invented.

Recovering the lost points

Getting from 87.1% to 92.45% took four changes, in order of how much each contributed:

1. Landmark-space augmentation (+2.8 pts). Small random rotations about the vertical axis (±12°), horizontal mirroring with a left/right hand swap, scale jitter (±8%), and temporal jitter (drop or duplicate one frame). Mirroring is only safe because we swap the hand blocks when we do it — otherwise you teach the model that dominant-hand signs are ambidextrous, which in BSL they are not.
2. Class rebalancing (+1.4 pts). Our raw distribution was badly skewed — the most common sign had 180 sequences, the rarest 41. Class-weighted loss plus targeted re-recording of the 12 thinnest classes.
3. Cleaning (+0.8 pts). We hand-reviewed sequences where MediaPipe detected no hand in more than 40% of frames and discarded 611 of them. The dataset went from 10,011 to 9,400 sequences and got better.
4. Longer training with cosine decay (+0.35 pts). 120 epochs, cosine learning-rate schedule, early stopping on validation loss with patience 15.

Final held-out test accuracy: 92.45% across 93 signs. Top-3 accuracy: 98.1%.

Getting to 255KB

The trained Keras model is 1.9MB. TFLite conversion with default optimisations does most of the work:

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS,
    tf.lite.OpsSet.SELECT_TF_OPS,   # GRU needs a few TF kernels
]
tflite_model = converter.convert()

with open("bsl_93.tflite", "wb") as f:
    f.write(tflite_model)

Dynamic-range quantisation takes the weights to int8 and leaves activations in float32. The result is 255KB and costs 0.11 percentage points of accuracy — 92.45% down to 92.34% on the same test set.

Full int8 quantisation would have got us under 100KB but cost 1.8 points. Not worth it. 255KB is already small enough to ship inside the app bundle rather than downloading it, which is the threshold that actually matters.

Measured on a Samsung Galaxy A54 — deliberately a mid-range device, not a flagship:

Stage	Time per inference
MediaPipe Holistic	22 ms
Feature assembly	1 ms
GRU inference (TFLite)	8 ms
Total	31 ms

That is comfortably inside a 33ms frame budget at 30fps, and none of it touches the network.

The confidence gate

Raw accuracy is the wrong metric for a deployed assistive tool. A tool that guesses confidently and wrongly is worse than one that says nothing, because a wrong sign in a GP appointment is a wrong answer to a clinical question.

So we do not surface a prediction unless it clears two conditions:

CONFIDENCE_THRESHOLD = 0.85
STABILITY_WINDOW = 5   # consecutive predictions that must agree

def gated_prediction(probabilities, history):
    label = int(np.argmax(probabilities))
    confidence = float(probabilities[label])

    if confidence < CONFIDENCE_THRESHOLD:
        return None

    history.append(label)
    if len(history) < STABILITY_WINDOW:
        return None

    recent = history[-STABILITY_WINDOW:]
    if len(set(recent)) != 1:
        return None   # still oscillating between candidates

    return label, confidence

Applying the gate takes recognition rate down to about 88% of attempted signs, and takes precision on the signs it does surface up to 97.3%. That trade is correct for this product. The user experience of "it did not catch that, sign again" is recoverable. The experience of confidently mistranslating someone is not.

What is still wrong

Being honest about the limits, since this is what a technical reviewer will ask:

• 93 signs is a vocabulary, not a language. Conversational BSL needs thousands. Our next dataset target is 500 signs by Q4 2025.
• We do not model non-manual features. No eyebrow or head-tilt tracking yet, which means we cannot currently distinguish a question from a statement.
• Signer diversity is too narrow. Our contributors skew younger and are predominantly right-hand-dominant. Left-dominant signers see measurably worse performance and that is a fairness defect, not a curiosity.
• Isolated signs only. Continuous signing requires segmentation — knowing where one sign ends and the next begins — which we have not solved.

Reproducing this

The architecture above is the whole model. Nothing in it is novel; the contributions are the dataset discipline, the group-aware split, and the deployment constraints. If you are building something similar, the one thing worth taking away is this:

Split your data on the group, not the sample, and be suspicious of any number that makes you feel clever.

Our 99.2% was not a model. It was a memory of a room.

---

Questions about the training pipeline, or interested in contributing BSL recordings? Get in touch at hello@devsandvisuals.com.