🎵 DeepMusic-OCR: How AI Learns to Read Sheet Music We adapted DeepSeek-OCR a model built for reading text and taught it to read the 2D language of music notation. Here’s what the paper is really about 👇 Thread 🧵

1/ Unlike normal text, music is two-dimensional: • Vertical = chords / simultaneity • Horizontal = rhythm / time Traditional OMR systems try to segment symbols. DeepMusic-OCR doesn’t. It reads the entire score at once.

2/ 🔍 The Encoder DeepMusic-OCR uses a vision encoder redesigned for music: • 8×8 fine-patch resolution for tiny details • 2D positional encoding aligned with staff lines • Dual attention: local (notes) + global (layout) • Pretrained on millions of synthetic sheets This lets the model capture both symbols and structure.

3/ 🎼 The Decoder Instead of outputting words, the decoder outputs musical events, like: <note:F#5-quarter> <clef:G> <key:D-major> It also handles: • Polyphony • Chords • Multiple voices …thanks to a Mixture-of-Experts architecture.

4/ 🧠 Musical Grammar Built In DeepMusic-OCR isn’t allowed to output impossible music. A “musical grammar loss” penalizes: • Broken measures • Impossible rhythms • Invalid symbols This gives the model a sense of musical correctness.

5/ 🖼️ Training Data Since real OMR data is limited, we generated millions of training examples from: • MusicXML • MuseScore • IMSLP Each score is rendered in multiple engraving styles, with distortions to simulate scanned pages. Synthetic data = the breakthrough.

6/ ⚡ Results With ~200 tokens per page, DeepMusic-OCR achieves: • High symbol accuracy • Consistent measures • Strong transfer to handwritten music And it does so at a fraction of the compute cost of traditional OMR systems.

7/ 🌍 Why This Matters DeepMusic-OCR enables: • Digitization of classical archives • Large-scale symbolic music analysis • Conditioning generative models with real scores • Education tools for musicians This isn’t just OCR it’s visual-symbolic music understanding.