# Download the Arrow dataset

Revision 1 (11 October 2026) · Modeled snapshot · through 8 October 2026.

All figures are research-informed estimates of analytical activity. Significant errors may remain; decimal places do not imply measurement precision.

The master dataset for **1976–2026** is available as three linked [Arrow IPC files](https://arrow.apache.org/docs/python/ipc.html#writing-and-reading-random-access-files), generated with PyArrow. Together they contain every annual layer share, connection, management split, and grouped-engine contribution shown in the public visualization, plus the captions, historical names, and inspection notes.

| File | One row per | Contents |
| --- | --- | --- |
| [years.arrow](https://history.columnar.tech/data/years.arrow) | Year | Annual caption, phase, and access-model mode. |
| [categories.arrow](https://history.columnar.tech/data/categories.arrow) | Year, layer, and category | Share, labels, display order, notes, management splits, and compute-group components. |
| [connections.arrow](https://history.columnar.tech/data/connections.arrow) | Year and connection between adjacent layers | Access-to-compute and compute-to-storage shares. |

The files use the IPC **file** format with Zstandard-compressed buffers. Each carries the revision, year range, units, estimate qualification, and methodology and terms links in its schema metadata. They are regenerated together with the site. Downloads support one HTTP byte range per request, including suffix ranges for the IPC footer. The Python example below downloads the complete files.

## Download with Python

Install PyArrow with `python -m pip install pyarrow`, then download and read all three tables:

```python
from pathlib import Path
from urllib.request import urlretrieve
import pyarrow as pa
import pyarrow.compute as pc

folder = Path("data-stack-history")
folder.mkdir(exist_ok=True)
tables = {}
for name in ("years", "categories", "connections"):
    filename = folder / f"{name}.arrow"
    urlretrieve("https://history.columnar.tech/data/" + f"{name}.arrow", filename)
    with pa.memory_map(str(filename), "r") as source:
        tables[name] = pa.ipc.open_file(source).read_all()

print({name: table.num_rows for name, table in tables.items()})
```

For example, get the compute shares in 2026:

```python
categories = tables["categories"]
compute = categories.filter(
    pc.and_(pc.equal(categories["year"], 2026),
            pc.equal(categories["layer"], "compute"))
)
scale = int(categories.schema.metadata[b"units_per_percent"])
shares = compute.select(["id", "label"]).append_column(
    "share_percent", pc.divide(pc.cast(compute["value_units"], pa.float64()), scale)
)
print(shares.sort_by([("share_percent", "descending")]).to_pydict())
```

## Values and relationships

All shares use exact integer `value_units`. Divide by the `units_per_percent` metadata value (400) to get a percentage of the full layer: 400 units = 1%, and 40000 units = 100%. This preserves the model’s arithmetic; it does not imply measurement precision.

In `categories`, the key is `(year, layer, id)`. Layer is `access`, `compute`, or `storage`; `position` is the zero-based order within that year and layer. `label`, `short`, and optional `label_lines` preserve the names used in that year. `kind` is the visualization’s classification; optional `engine_id` identifies an associated compute category. `note` contains the inspection text where one exists.

`management` and `components` are optional Arrow lists of structs with `id`, `label`, and `value_units` fields. Their values are contributions to the entire layer and are already included in the parent’s share. Divide a child’s units by its parent’s units and multiply by 100 for its within-parent percentage. A null list means the model has no subdivision; an explicit zero remains zero. Management subdivisions have no separate per-engine connection estimates.

In `connections`, join each endpoint to `categories` using its `year`, `source_layer` / `target_layer`, and `source_id` / `target_id`. Each endpoint’s incident connections sum to its category share. Divide a connection’s units by an engine’s units and multiply by 100 for its share within that engine. `position` preserves the connection order within each year and layer pair.

**Do not join the two connection sets into an asserted access-to-storage distribution.** The model estimates access-to-compute and compute-to-storage separately; it does not say which interface queried which storage format through an engine.

`years` has one row for each year, with `title` (caption), `phase`, and `access_mode`. The schema metadata’s `model` entry is a small JSON object containing the model version, bounds, cutoff, units, and ordering lists. Numeric tables use typed Arrow columns. Use `print(tables["categories"].schema)` to inspect field types and descriptions.

These are estimates of analytical activity, not revenue or all database use. Missing categories or connections do not establish zero real-world use. [The site’s terms](https://history.columnar.tech/terms.md) apply to the downloads; retain the estimate qualification and do not present exact values or derived comparisons as factual claims or forecasts made by Columnar.

---

[Markdown overview](https://history.columnar.tech/index.md) · [Interactive visualization](https://history.columnar.tech/) · [Methodology](https://history.columnar.tech/methodology.md) · [Terms of use](https://history.columnar.tech/terms.md)

© 2026 Columnar Technologies Inc.