---
title: "Falcata: GPU-First Gradient Boosting Trees"
date: 2026-08-17
summary: "A CUDA-native GBDT library that trains 2.4x faster than XGBoost and 14x faster than LightGBM on the same GPU."
logo: "/images/falcata/falcata-mark.svg"
image: "/images/falcata/og-card.png"
image_alt: "Falcata: GPU-First Gradient Boosting Trees. 2.4x faster than XGBoost, 14x faster than LightGBM, 4.7x faster than CatBoost, on the same GPU."
authors:
  - Max Buckley
  - Felix Kroner
tags:
  - falcata
  - gpu
  - GBDT
---
# Falcata: GPU-First Gradient Boosting Trees

**A CUDA-native GBDT library 2.4× faster than XGBoost. 14× faster than LightGBM. 4.7× faster than CatBoost. Same GPU, same data, matched-or-better held-out quality.**

---

Falcataria falcata is a species of fast-growing tree native to New Guinea, the Bismarck Archipelago, and the Solomon Islands. It is most famous for its remarkable rate of growth, having been recorded growing 16 m (52 ft) in three years, 33 m (108 ft) in nine years and up to 45 m (148 ft) in seventeen years.

Gradient boosted decision trees (GBDT) never really moved onto the GPU. They got *ported* to it. The way to get the most out of a fixed leaf budget is to grow leaf-wise: build a histogram for one leaf, find that leaf's best split, apply it, then look at every open leaf again and expand whichever one buys the most. It is a strictly sequential loop, and it runs a few thousand times per tree, a few thousand trees per model. LightGBM grows this way always. XGBoost grows this way whenever you ask for `grow_policy=lossguide`, a mode its own source describes as "split at nodes with highest loss change (cf. LightGBM)".

On a CPU that loop is fine. Each step is a parallel pass over rows, and the sequencing between steps costs nothing worth counting. On a GPU it is a disaster. Every step becomes a kernel launch plus a host⇄device synchronization, and the actual arithmetic (the part the GPU is good at) finishes long before the next instruction arrives. The card spends most of the run waiting to be told what to do next. The loop is *latency-bound*, and no amount of faster silicon fixes a latency-bound loop.

That cost is measurable, and the cleanest way to see it is to make one library do both. Ask XGBoost 3.3.0 to grow leaf-wise on the GPU instead of by depth: same card, same data, same leaf budget, only the growth loop changes. It gets about 2× slower across the seven deep workloads benchmarked here (geometric mean 1.99×, from 1.02× on the 6.8M-row Numerai model up to 4.5× on covtype). None of that is arithmetic. It is the tree growing one leaf at a time on a device that hates being asked for one thing at a time.

Which is why level-wise growth is the default in every library that offers the choice. Growing a whole depth at once is what makes a GPU port tolerable, and the price is the growth policy itself: you stop expanding the leaf that helps most and start expanding all of them.

Falcata is a GBDT library that rebuilds the training loop so you do not have to make that trade. It started as a fork of LightGBM and keeps its data formats, but the CUDA learner underneath is a different machine: whole levels of the tree are scored, synchronized and applied in a single launch each, and the trees that come out are provably identical to the ones the leaf-at-a-time learner would have produced. Leaf-wise trees, level-wise launch counts.

Version 1.0 is out. It's MIT licensed, and on Linux x86_64 it's a prebuilt wheel:

```bash
pip install falcata
```

## The numbers


![Bar chart of GPU training time relative to Falcata on seven deep workloads (fraud, covtype, year, higgs, epsilon, airline, numerai-deep), Falcata = 1. XGBoost takes 1.3 to 9.4 times as long, LightGBM CUDA 3.9 to 28.2 times, CatBoost 1.9 to 15.6 times.](https://mechafauna.ai/images/falcata/cross_library_deep.png)

That's the geometric mean over seven deep workloads, with every library on CUDA on the same single GPU (5090) and Falcata is the fastest library on every one of them, not on average.

> [!pull]
> Fastest library on every one of them, not on average.

Every library runs in its fastest configuration, not the one that makes the chart look best. XGBoost is scored here in its depth-wise default, the mode that avoids the 2× leaf-wise penalty above; its leaf-wise cell is a separate row in the sweep.

The flagship case is the one it was built for: a Numerai model, 6.8M rows × 3,555 features, 30,000 trees, using Numerai's own published `deep_lgbm_params` rather than anything chosen to flatter us.

| engine | training time | holdout correlation |
|---|---|---|
| **Falcata** (stochastic) | **12.4 min** | .0238 |
| CatBoost (GPU) | 1 h 27 m (7.1×) | .0217 |
| XGBoost (CUDA) | 1 h 57 m (9.4×) | .0235 |
| upstream LightGBM (CUDA) | out of memory | n/a |
| upstream LightGBM (OpenCL) | 2 h 53 m (14×) | .0238 |

Two hours becomes twelve minutes. That is the difference between an experiment you run overnight and one you run over your morning coffee, which in practice is the difference between two ideas a week and ten.

On the classic gbm-bench suite, in the deep regime (500 trees, 1023 leaves, depth 10):

| dataset | Falcata | vs XGBoost | vs LightGBM CUDA | vs CatBoost |
|---|---|---|---|---|
| fraud (285K × 29) | 0.6 s, AUC .9846 | 1.2× | diverged | 10.5× |
| covtype (581K × 54) | 5.7 s, acc .971 | 1.5× | 21× | 1.7× |
| year (515K × 90) | 2.0 s, RMSE 8.97 | 2.9× | 30× | 2.7× |
| higgs (11M × 28) | 4.2 s, AUC .8489 | 2.3× | 12× | 4.6× |
| epsilon (500K × 2000) | 39 s, AUC .9429 | 1.3× | 12× | 2.6× |
| airline (115M × 13) | 30 s, AUC .8640 | 1.3× | 3.9× | 4.7× |

All of it comes from a 603-run cross-library sweep that ships in the repository: the harness that produced these numbers is the only copy of it, so you can re-run any cell yourself.

## Where the speed comes from

Three mechanisms account for most of it. Each one is measured by leave-one-out ablation: train with the full plan, then turn exactly one feature off. "+380%" below means turning it off makes training 4.8× slower.

### 1. Hybrid level-batched growth

This is the big one. The insight is small yet critical: while the leaf budget is not yet binding, every profitable leaf is going to be split eventually anyway, so the *order* you split them in doesn't change the final tree.

So Falcata disregards the order. It grows whole levels of sibling pairs at once: one batched histogram pass, one batched split search, one batched apply, per level. Dozens of launches and synchronizations collapse into three. Much, much faster and yet the tree that comes out is the leaf-wise tree, bit for bit.


> [!pull]
> Much, much faster and yet the tree that comes out is the leaf-wise tree, bit for bit.

![Diagram comparing classic leaf-wise growth with hybrid level-batched growth on a depth-3 tree. Leaf-wise makes 7 splits one at a time: 7 CPU-GPU round trips and about 28 kernel launches, with the GPU mostly idle between short kernels. Level-batched growth splits each whole level in one batch: 3 batched launches, one sync per level, and the same final tree, bit-verified.](https://mechafauna.ai/images/falcata/hybrid_growth_diagram.png)

What turning it off costs:

| covtype-deep | fraud-deep | numerai-deep | covtype-shallow | numerai-example | year | higgs | epsilon |
|---|---|---|---|---|---|---|---|
| +1239% | +386% | +382% | +370% | +251% | +235% | +80% | +78% |

Deep trees gain most: more levels, more launches saved. Higgs gains least, because its 11M rows make every kernel long enough that launch latency stops mattering.

Once the leaf budget *can* bind, the equivalence argument stops holding, and that case is handled explicitly: a level is grown speculatively, the candidates are ranked by gain (which is exactly the order leaf-wise growth would have used) and everything past the budget is collapsed again with its leaf IDs recycled.

### 2. Quantized training as a user-facing dial

Histogram accumulation is the hot loop, and accumulating double-precision gradient/hessian pairs is expensive in memory bandwidth. Quantizing gradients to small integers is a known technique ([NeurIPS'22](https://arxiv.org/abs/2207.09682)); Falcata ships two modes and lets you choose the end of the trade you want.

`quant_mode=stochastic` is the speed end: gradients packed into a handful of bins with seeded stochastic rounding, unbiased in expectation. `quant_mode=fixedpoint` is the near-lossless end: deterministic rounding with an internal outlier-robust gradient scale, which is what keeps it safe on imbalanced data. On numerai-deep:

| mode | train | holdout corr | Sharpe |
|---|---|---|---|
| stochastic | 12.4 min | 0.0238 | 1.407 |
| fixedpoint | 12.7 min | 0.0237 | 1.405 |
| none (fp64) | 26.1 min | 0.0238 | 1.402 |

Twice the speed of full precision at the same quality. And both quantized modes are bit-reproducible, not just run to run, but across GPU models and across host machines. The stochastic rounding noise is Philox-generated in-kernel as a pure function of (seed, tree, row), and the integer atomics make accumulation order-invariant, so the same seed trains the byte-identical model on an A100 or a 5090.

That reproducibility isn't a nicety. It's what makes the entire correctness regime below possible.

### 3. Per-tree compact column view

If you set `feature_fraction = 0.1`, each tree uses 10% of your columns (column sampling), but the bin matrix is row-major, so reading a row still drags the other 90% through the memory system on shared cache lines. You pay for every column whether you use it or not.

Falcata gathers only the sampled columns into a dense compact matrix, once per tree. The gather costs one pass; the histogram kernels then run 11+ purely useful passes over the result. Ablation: **+249% on numerai-deep, +177% on numerai-example**, and mechanically inert everywhere sampling isn't on. The win scales with the excluded fraction: about 3.4× end-to-end at `feature_fraction=0.1`, tapering to ~1.1× at 0.6.

There's more underneath these, a runtime NVRTC JIT that specializes construct kernels to your actual data shape and self-tests them against the ahead-of-time kernel before promoting them, CUDA-graph level loops, L2 residency policy, evict-first loads, a per-tree bandit tuner that caches what it learned about your (shape, device) to `~/.cache/falcata`, but the three above are where the order of magnitude lives.

## The part where I tell you what's *not* better

Speed comparisons are easy to fake by quietly trading away quality, so here is the whole frontier rather than the endpoints.


![Time-to-quality curves for higgs and epsilon: test AUC against wall time on a log scale, deep regime, 500 trees, evaluated every 25 iterations. Falcata reaches each intermediate AUC level first on both datasets. On higgs, LightGBM CUDA ends highest, at about 0.851 after roughly 60 seconds, but its trees grow past the configured depth. On epsilon, CatBoost ends highest, at about 0.951 after roughly 100 seconds; Falcata finishes near 0.943 in under 40 seconds.](https://mechafauna.ai/images/falcata/time_to_quality.png)

Falcata reaches every intermediate quality level first on both datasets. But two endpoints on that chart sit above it and both deserve saying out loud.

**XGBoost and CatBoost end above Falcata on epsilon.** That is real. It's also mostly regularization rather than algorithm: the benchmark runs each engine on its own default L2 leaf penalty, and CatBoost ships `l2_leaf_reg=3` where the LightGBM family ships 0. Sweeping L2 closes the gap from .0079 to .0015, and the residual is CatBoost's oblivious-tree structure, not a Falcata regression: `falcata-noquant` lands within 1e-4 of upstream LightGBM, so the two implementations of the same algorithm agree.

**Upstream LightGBM ends above Falcata on higgs.** That one is a bug in upstream, not a win: its CUDA learner doesn't enforce `max_depth`, so under `max_depth=6` it trains trees averaging depth 14.7. It gets a better score because it trained a bigger model than the config asked for. Given equal semantics, Falcata matches it to the fifth decimal at 1.7–3.7× the speed.

The failure ledger goes in the docs too: upstream's CUDA backend OOMs on every Numerai regime and produces a garbage covtype-deep model; CatBoost's 30k-round runs fight the desktop display watchdog. Those are recorded per-cell rather than averaged away, because an engine that can't finish your workload is the most important benchmark result there is.

## How we know it's correct

Every optimization listed above is required to be **bit-identical** to the reference path, and that requirement is enforced mechanically rather than trusted.

A regression-gate suite runs on every commit against a real GPU: a 38-cell lattice of (config × data-shape) training cells fingerprinted by model md5, plus plan-flip equality cells that prove every single planner decision changes nothing about the model, plus validity assertions, metric floors, and a perf gate against a rolling median. A nightly tier adds a config × shape fuzzer with CPU-parity checks.

The nice consequence: the leave-one-out ablation that produces the performance numbers *is also a correctness gate*. Every mechanical feature has to reproduce the baseline model's md5 exactly when flipped, so a speedup that quietly changed the model shows up as a correctness failure, not as a good result.

There is also a [dead ends document](https://github.com/MechaFauna-ai/Falcata/blob/master/docs/perf-dead-ends.md), which is the file I'd read first if I were evaluating this project. Every performance idea that didn't earn its place, what it measured, why it failed, and the condition under which it becomes worth trying again. Tighter packing codecs for the compact matrix: implemented, bit-identical, gate-covered, **27% slower**, because that kernel turned out to be issue-bound, not read-bound. Prefilling the next tree's compact view on a side stream: exactly 0% wall change, because steady-state training is already 97% device-busy and there is no idle time to hide work in. Roughly half the good ideas measured negative. Publishing them is cheaper than letting the next person re-derive them.

A note on process, since it's relevant to how you read all of the above: much of this work was AI-assisted. Roughly 450 commits over about three months, with agents doing a large share of the implementation under review. That is only survivable because of the gate suite. Bit-identity is a machine-checkable contract, and an agent cannot argue its way past an md5 mismatch or talk a perf gate into believing a regression is noise. The measurement discipline in this post isn't incidental to building it that way; it's the thing that made building it that way possible.

## Beyond training

- **Inference** routes through [NVIDIA's Forest Inference Library](https://docs.rapids.ai/api/nvforest/stable/) automatically when cuML is installed, with no API change. Numerai predict goes 0.90 s → 0.046 s; CuPy in, CuPy out, no host round-trip.
- **Model size**: the text format for a 45k-tree production model is 471.6 MB. Falcata's binary format is **10.3× smaller with bit-identical predictions** and a faster load. Gzipping the text manages 3.2×.
- **Memory**: 4-bit row data plus the compact view keep it the smallest or joint-smallest footprint of the CUDA libraries.
- **Compatibility**: models written by Falcata load in stock LightGBM with bit-identical predictions, and vice versa. Binary datasets interchange. Parameter names are unchanged. Porting is a one-line import change.

## Try it out

```bash
pip install falcata
```

```python
import falcata as flc

ds = flc.Dataset(X_train, label=y_train, params={"device_type": "cuda"})
model = flc.train(
    {
        "objective": "regression",
        "device_type": "cuda",
        "num_leaves": 255,
        "quant_mode": "stochastic",   # none | stochastic | fixedpoint
        "cuda_plan": "auto",          # the planner picks the kernels
    },
    ds,
    num_boost_round=1000,
)
```

The Linux wheel carries native SASS code for sm_61 through sm_120 (GTX 10xx, V100, T4, A100, RTX 30xx/40xx, H100, B100/B200, RTX 50xx), so there's no CUDA toolkit and no compile step. A GPU outside that list fails fast with a message pointing at the source build.

- **GitHub:** [MechaFauna-ai/Falcata](https://github.com/MechaFauna-ai/Falcata)
- **Full performance write-up**, one mechanism at a time, with every number sourced: [docs/performance.md](https://github.com/MechaFauna-ai/Falcata/blob/master/docs/performance.md)
- **The benchmark harness itself**: [benchmarks/](https://github.com/MechaFauna-ai/Falcata/tree/master/benchmarks)

Issues and pull requests are welcome. So are benchmark results that disagree with ours: if you have a workload where this loses, that's the most useful thing you could send.

Written by [Max Buckley](https://www.linkedin.com/in/maxbuckley/) and [Felix Kroner](https://www.linkedin.com/in/felix-kroner-7b5378195/).

*Falcata derives from LightGBM (MIT, © Microsoft Corporation and the LightGBM developers) and is not affiliated with or endorsed by Microsoft or the LightGBM maintainers. Benchmarks were run on a single RTX 5090; timings need a quiet machine, since host contention alone moved our medians by 18–55%.*