---
license: mit
library_name: coreml
tags:
  - audio
  - music-source-separation
  - stem-separation
  - demucs
  - htdemucs
  - coreml
  - apple-silicon
  - swift
  - macos
  - ios
pipeline_tag: audio-to-audio
---

# HTDemucs → Core ML

Convert Meta's [Hybrid Transformer Demucs](https://github.com/facebookresearch/demucs) (HTDemucs) into a Core ML `.mlpackage` you can drop into a macOS or iOS app and run with `MLModel`.

The hard part of converting HTDemucs to Core ML is not the network itself — it is the STFT/ISTFT and the multi-head attention around it. This repo contains a **single-file converter** (`convert.py`, ~600 LoC) that solves the three blockers you hit otherwise:

1. Core ML doesn't support `complex64` → real-valued STFT/ISTFT.
2. `coremltools` can't trace `nn.MultiheadAttention` → manual decomposition.
3. Core ML's 1D `scatter_add` is fragile → pre-computed OLA index buffer.

The result is a **stand-alone `.mlpackage`** that takes raw stereo audio and outputs four stems (vocals, drums, bass, other) at 44.1 kHz.

## Why another conversion?

There is one prior public Core ML conversion of HTDemucs by [john-rocky/CoreML-Models](https://github.com/john-rocky/CoreML-Models) at 7.8 s segments / 80 MB. This repo offers:

- **Longer segments (10 s by default)** → fewer overlap-add boundaries on long files.
- **CLI flags** for segment length, FP16 quantization, compute-unit selection.
- **Source order** reordered to `[vocals, drums, bass, other]` (DJ/UI convention).
- **Documented workarounds** so you can reproduce or adapt the pipeline for other audio models (Spleeter, OpenUnmix, MDX-Net).

## Quick start

```bash
git clone https://github.com/dexxdean/htdemucs-coreml
cd htdemucs-coreml
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# default: 10 s segments, FP32, ~400 MB
python convert.py

# half size, ~200 MB, slight numerical drift but inaudible in practice
python convert.py --fp16

# shorter segments if you want lower latency / smaller buffers
python convert.py --segment 7
```

The output is `HTDemucs_CoreML.mlpackage` (or `HTDemucs_CoreML_FP16.mlpackage`).

## Usage in Swift

```swift
import CoreML
import AVFoundation

// 1. Load the model.
let url = Bundle.main.url(forResource: "HTDemucs_CoreML", withExtension: "mlpackage")!
let config = MLModelConfiguration()
config.computeUnits = .cpuAndGPU   // see "Compute units" below
let model = try MLModel(contentsOf: url, configuration: config)

// 2. Feed a (1, 2, 441000) Float32 MLMultiArray named "audio".
//    Output is a (1, 4, 2, 441000) Float32 array named "sources",
//    in the order [vocals, drums, bass, other].
```

A more complete example with chunking, overlap-add, and `AVAudioEngine` playback is in [`examples/swift/StemSeparator.swift`](examples/swift/StemSeparator.swift).

## Model I/O

| | |
|---|---|
| Input name | `audio` |
| Input shape | `(1, 2, segment_samples)` Float32 |
| Output name | `sources` |
| Output shape | `(1, 4, 2, segment_samples)` Float32 |
| Output order | `vocals, drums, bass, other` |
| Sample rate | 44 100 Hz, stereo |
| Default segment | 441 000 samples (10 s) |
| Min. deployment | macOS 14 / iOS 17 |

## Compute units

HTDemucs is **not stable on the Apple Neural Engine**. Use `.cpuAndGPU` (the default baked into the model). Forcing `.all` or `.cpuAndNeuralEngine` may produce silent garbage on some shapes — the validation step in `convert.py` will warn if numerical drift is large.

## File sizes

| Variant | Size | Notes |
|---|---|---|
| FP32, 10 s | ~400 MB | Default, full reference quality. |
| FP16, 10 s | ~200 MB | Inaudible quality difference for music separation. |
| FP32, 7.8 s | ~310 MB | Closer to john-rocky's segment length. |

## How it works

See [`CONVERSION_NOTES.md`](CONVERSION_NOTES.md) for the technical deep-dive on the three workarounds (real STFT, manual MHA, OLA scatter).

## License & attribution

This repo is **MIT-licensed** — see [`LICENSE`](LICENSE).

The converted model derives from [facebookresearch/demucs](https://github.com/facebookresearch/demucs) © Meta Platforms, Inc., MIT-licensed. The pre-trained HTDemucs weights are downloaded by the `demucs` Python package at conversion time from Meta's official release. **You must comply with Demucs' MIT license when redistributing the resulting `.mlpackage`** — keep the attribution in `ATTRIBUTION.md` alongside the model and in your app's about/legal screen.

This project is **not affiliated with Apple, Meta, or Demucs**. The package name `HTDemucs_CoreML.mlpackage` was chosen to avoid any confusion with Apple-internal model names (e.g., `MusicSourceSeparation`).

## Citation

If you use this in academic work, please cite the original Demucs papers:

```bibtex
@inproceedings{rouard2023hybrid,
  title={Hybrid Transformers for Music Source Separation},
  author={Rouard, Simon and Massa, Francisco and D{\'e}fossez, Alexandre},
  booktitle={ICASSP 2023},
  year={2023}
}
```