July 27, 2026
I've made good progress! Finally got my data scaling done which I'd been dreading for a while because it involves slogging through annotating match data. Now I feel confident that I have enough data to train, validate, and test models with deep learning architectures. I've been churning away at some models. I released v0.4.0, which is similar to the previous architecture of a bidirectional LSTM trained only on a "pointness" probability binary cross entropy objective, then postprocessed with hysteresis to yield point start, point end pairs. It's the exact same architecture, but trained with more data.
On the new and exciting stuff - something I've been thinking about is a way to train these models end to end - that is, be able to use the actual point start, point end ground truth in my training set as a direct training objective. After a bit of bouncing around on architectures, I decided on adding two additional heads: predicting the "startness" and "endness" of frames. The general intuition is that our original training loss head predicts how much a frame is in a point vs out of a point, which is very useful for the model to know. But now, we're more directly training the model on what the start and end of points look like - we might imagine that the model learns that a point typically follows a service motion, and typically ends around the time players turn away from the court and walk back. More specifically, for the startness and endness, we train the startness head by creating a gaussian centered at point start time with a tuned standard deviation of 0.5s. The "startness" objective for a given frame is just the value of the gaussian d seconds away from center (forgive loose terminology). In decode, we threshold a point detection on the pointness being above a certain threshold for a long enough duration, then we use soft-argmax over the discrete heatmap values to estimate the point start and end. This is also an area of exploration!
My second area of exploration is with architecture. Currently, I've been using a bidirectional LSTM to map my frame by frame pose inputs to pointness, startness, and endness outputs. However, I was running into issues with these. In my latest training block (and the first one with proper validation and test sets), my LSTM validation loss bottomed out after epoch 1 or 2, indicating serious overfitting. I attribute this to two related factors:
- General overparametrization (total parameter count too high)
- LSTMs are not the best architecture choice for this.
Starting with the first: our original 2-layer bidirectional LSTM has an input size of 362 and a hidden size of 128. That yields 899,329 total parameters in the model architecture used for v0.3.1. We trained with 3,310 20 second sequences, each overlapping 10 seconds and including flipped videos as an easy data augmentation to promote lefty/righty invariance. Even with these strategies to 4x our training sequences (though not adding new information), we still have 270 parameters per training sequence, and 2.7 parameters per frame of training data.
In v0.4.0, we scale data to 8,307 training sequences with overlap and without video flipping. That's 108 parameters per training sequence, and roughly 1 parameter per frame of training data. This data increase markedly improves performance! On my internal segmentation benchmark, good annotations increased from 14.3% to 30.7% (some caveats on this benchmark). I also tried a hidden size of 64 instead of 128 on the LSTM, yielding similar performance with only 318,593 parameters. This network also had validation loss decreasing until epochs 10-17, indicating that we weren't overfitting until much later.
This brings us to the second point. We observed similar performance and better training behavior from reducing our parameter count. What if we chose a more parameter efficient architecture?
The first architecture that comes to mind is a convolutional net. However, we need our network to be able to use context from anywhere in the input sequence, not just local frames. To enable long-range context on a normal temporal conv net, we need big kernels. However, those come at a parameter cost. Enter dilated time conv nets. The idea is that instead of needing kernels over many timesteps (blows up parameter count) or super deep networks (blows up parameter count), we skip exponentially more timesteps between each frame in our kernel - we start by convolving over every frame, then in the next layer we convolve over every 2 frames, then the next we do every 4 frames, and so on. This gets us a receptive field of 125, covering our entire sequence, in just 5 blocks. For a dilated TCN with 5 dilations each with 2 layers and 64 channels, that's 156,675 parameters, half the size of the 64-hidden-size bidirectional LSTM.
That's the biggest stuff I'm working on! I still have to tune the TCN more before I release it, then I'm going to focus (vibe-code) on the iOS app and also improving my input features. Also, I think the new updates will be in this half-update, half-technical-blog style since it's more fun for me to talk about developing these models and architecture!
- Ismael