PenGPT: A Generative Transformer for Handwriting and Sketches
A small transformer that writes and sketches from text. The effort was 80% data, 15% speed and 5% architecture: my first project that follows the Bitter Lesson.
Last year Zach Wimpee and I trained a tiny GPT to write cursive. It worked but it only knew one hand (mine) and it learned from 3,500 words that I wrote on a trackpad.
So here’s the obvious next question: can one small, ordinary transformer learn to hold a pen in lots of hands and draw as well as write? PenGPT is my attempt. There are two models. A 6M-parameter one writes print and cursive and picks up the handwriting of someone it has never seen from a few of their words. A 54M-parameter one draws from a text prompt.
Then I took the thing apart to find out why it works. The answer wasn’t what I expected.
How it works
That’s it for the modality-specific part: ten symbols and a merge count. The same tokenizer handles handwriting and drawings. For handwriting the model is told the text; for drawings it’s told a SigLIP 2 vector so a typed phrase can stand in for a picture.
The data
Almost nobody records pen strokes so I made the handwriting data from scans: 495,590 words by 1,255 writers, mostly from two research collections plus old letters and diaries from public archives. A vision-language model finds legible pages and transcribes them, the pages get cut into words, InkSight traces each word back into strokes and a set of checks throws out bad traces. Between a third and a half of the words survive.
The drawings are 477,670 sketches from ten public datasets. Quick, Draw! has 50 million and I kept the best 0.5%. I’m not releasing the datasets but the paper’s appendix explains how to build them.
Training a model
I compared PenGPT with older approaches then changed one thing at a time and retrained. Here are the rows I found most interesting; the paper has the rest.
| Handwriting | Score change | Style rank | Legible |
|---|---|---|---|
| Graves-style LSTM, same tokens | +0.029 | 2.2 | 49% |
| ScribeTokens BPE, 60x the vocabulary | −0.001 | 1.9 | 53% |
| PenGPT | 0.716 | 1.9 | 54% |
| A tenth of the data | +0.050 | 3.0 | 47% |
| No augmentation | +0.028 | 2.0 | 52% |
| One word per sequence | +0.009 | 6.0 | 52% |
| No pen position | +0.001 | 1.9 | 52% |
| No BPE merges | −0.010 | – | 55% |
| Twice the parameters | −0.004 | 1.9 | 55% |
| Real ink | 66% |
| Drawings | Score change | Matches label | Matches caption |
|---|---|---|---|
| sketch-rnn-style LSTM, same tokens | +0.049 | 40% | 20% |
| Trained on Quick, Draw! only | +0.002 | 74% | 37% |
| PenGPT | 0.591 | 76% | 43% |
| No augmentation | +0.013 | 4% | 6% |
| No pen position | +0.008 | 74% | 40% |
| Half the layers, half the width | +0.022 | 62% | 31% |
Data and how you show it. Those are the big rows. Cut the data and everything gets worse. Drop augmentation and the drawing model stops following its prompts entirely. Train on one word at a time and the model can no longer pick up a hand: style rank goes from 1.9 to 6.0.
The clever parts barely register. Telling the model where its pen is does nothing for handwriting. The BPE merges I was proud of make the score slightly worse; they buy speed (sequences are 3.2x shorter) and that’s all. A vocabulary sixty times larger changes nothing. Doubling the model changes nothing.
Even the transformer is optional, sort of. An LSTM on the same tokens is a bit behind on handwriting and still picks up new hands. On drawings, where sequences are several times longer, it falls well behind.
Samples
Enough tables. Here’s what the two models make.
The drawing model takes a few controls besides the prompt. One of them is how much ink to use.
It can also draw some things it was never shown as long as a blend of two familiar ideas will do.
Picking up a new hand
The trick is almost embarrassingly simple. Each training sequence holds up to four words by one writer. That teaches the model that later words look like earlier ones so at sampling time you hand it a few real words and it keeps going. No style encoder, no fine-tuning. That’s how the 21 hands at the top of this post were written.
It also works from a photo. There are no strokes in a photo so I trace a few of its words with InkSight and fit one new writer vector to the traces, with the rest of the model frozen.
Reading the letters
One head in layer 4 puts 81% of its attention on the letter being written. I looked for heads that specialize in cursive or print and found none; every head reads the same positions in both.
Do you need captions?
Captions are the expensive part of the drawing data. Since SigLIP puts pictures and text in the same space, maybe you could train on picture vectors alone and type a prompt at the end?
You can’t. The picture-only model has the best likelihood of the three and it draws scribbles. The reason is geometric: SigLIP’s vectors for sketches sit in a narrow cone, nearly at right angles to its vectors for text. A model that has only seen the cone is lost when you hand it a sentence. Labels are enough to draw labels (78%) but only captions get you phrases (43% against 10%).
What it can’t do
The drawing model blends but doesn’t bind. A teapot with a face is fine. Who wears the hat is not and “facing left” gives the same drawing as “facing right” because the single vector it’s told is close to a bag of words (those two captions have a cosine similarity of 0.999). Sampled handwriting is also less legible than the real thing, 54% against 66%. And since most of the handwriting was traced from scans, its stroke order is a model’s guess and not a record.
Closing thoughts
If you look over my blog, you’ll notice that many of my projects revolve around clever ideas. I fell in love with them because they were elegant. That isn’t a flaw in itself but I think Rich Sutton was wise to warn about it in The Bitter Lesson. It really is bitter and it has taken me more time than I’d like to admit to embrace it.
I think this post is my first step in the right direction. The results are interesting and the project was a pleasure to work on. And yet there was no clever architectural idea behind it. About 80% of my effort went into building the biggest, cleanest datasets possible and doing lots of quality control on them. Another 15% went into finding a decent 8xA100 node and getting my models to train fast and in parallel on it: 8 minutes for the handwriting model and about 35 for the sketch model. My previous attempt, the Cursive Transformer, took upwards of 10 hours on an A100 and only ever learned one hand. The tokenizer I used here is much simpler and far less elegant but it just works.
The bitter lesson kills or sidelines a lot of beautiful research ideas. It’s like gravity: it constrains you and most of the time it’s a very bad idea to fight back. But as I accept it and begin to work with it, I find things to admire. As a physicist I’m fascinated by the emerging science of scaling laws. And as a lover of biology I like the idea of putting a ton of work into the data and then literally growing a model on it, as you would an organism. Then you get to interact with it, learn what it’s capable of and study how it works. Seen this way, the bitter lesson has its own charm.