# Caution Ahead: Numerical Reasoning and Look-Ahead Bias in AI Models - Accompanying Code

Code for replicating the analyses in *Caution Ahead: Numerical Reasoning and Look-Ahead Bias in AI Models* (Levy, 2026).

## Usage

### Setup

Install the pinned dependency versions from the lockfile:

```bash
uv sync --frozen
```

Copy the example environment file and edit it for your system:

```bash
cp .env.example .env
```

In particular, set `CAUTION_AHEAD_PATHS__DIR_DATA` to the root of the
external data and artifact directory. The code expects the following layout
beneath that directory:

```text
caution_ahead_data/
├── compustat/
├── anonymized_fin_statements_md/
├── commercial_llms/
├── gbdt/
├── lab_free_llms/
├── lab_free_predictions/
└── lab_free_representations/
```

The `.env.example` file also documents the WRDS username, OpenAI API key, and
runtime controls.

### Data and Artifacts

Proprietary and/or licensed data are intentionally excluded from this package in order to satisfy licensing requirements. For example, the author is not licensed to redistribute Compustat thus only the code used to process raw Compustat into the relevant samples, rendered financial statements, etc. is provided.

#### Rendered Financial Statements

Constructing the financial statements rendered into Markdown requires two files, `comp_ann_balance_sheet.csv` and `comp_ann_income_statement.csv`, which are derived from the "Balancing Models" for Compustat North America. At the time of publication, these models were available from WRDS [here](https://wrds-www.wharton.upenn.edu/documents/900/Balancing_Models_-_North_America_Company_Data_X1BZb7h.xls).

#### Input Manifest

SHA-256 checksums for the upstream inputs and cached artifacts used in the logged run, including the rendered financial statements, model checkpoints, and intermediate analysis caches are enumerated in `input-manifest.sha256`. While the underlying proprietary data and large artifacts are not distributed with this package, this manifest provides a commitment to the exact files used. All paths in the manifest are relative to `CAUTION_AHEAD_PATHS__DIR_DATA`. The SHA-256 checksum of the manifest itself is:

```text
b0d14740d35fb124d3da907b04e2f1bf160d782fc6705eea77bac5b83e807367  input-manifest.sha256
```

### Reproduction

By default, the code will use cached artifacts for GPU-intensive local LLM forwards, API-based LLM calls, GBDT training, etc. Assuming the ~50GB worth of cached artifacts are available, the following command should fully reproduce all tables in the paper in a few minutes.

```bash
uv run python main.py
```

### Replication

If none of the cached artifacts are present, then several things are needed to reproduce the analyses:

- A WRDS account with access to Compustat in order to build the paper's sample. **Note**: this sample was built in Dec 2024. If the copy of Compustat that WRDS is serving has changed since then, so will the sample creation.
- A local GPU with 24GB+ of VRAM to support forwarding the look-ahead bias free models
- A valid OpenAI API key. **Note**: The code makes thousands of API calls which lead to significant billing (thousands of dollars) and the volume of calls is such that making them all serially will likely take days to complete. Ideally the account associated with the API key will have Tier 5 rate limits to facilitate making the calls in parallel.

#### Numerical Reasoning and Tool Use

At the time the tool use code was originally run, OpenAI recommended the Assistants API. This API has since been deprecated and replaced by the Responses API. This package pins the `openai` version to that which was used at the time of research, and therefore predates the replacement Responses API. On a cache miss these cells are not regenerated — the code logs a warning and leaves them empty. To replicate them today, re-issue the (already-constructed) requests via the OpenAI Responses API code_interpreter tool. In untabulated/uncached results this yielded perfect performance on the tasks.

## Log from Producing Final Output for Paper

Per the Journal of Accounting Research's Data Policy, a full log file from running the code is available in `logs/pipeline_20260421-114311.log`.

## Citation

```tex
@article{Levy2026CautionAhead,
author = {Bradford Levy},
title = {Caution Ahead: Numerical Reasoning and Look-Ahead Bias in AI Models},
journal = {Journal of Accounting Research},
volume = {64},
number = {3},
pages = {1139-1188},
keywords = {AI, large language models, look-ahead bias, memorization, numerical reasoning, accounting, finance, forecasting},
doi = {https://doi.org/10.1111/1475-679x.70058},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/1475-679x.70058},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/1475-679x.70058},
abstract = {Recent work within accounting and finance has highlighted that modern AI systems exhibit superhuman performance on a variety of foundational activities within these fields. However, the literature often does not provide economic rationale for why AI models seem to outperform, largely because these models are a black box. Through a series of experiments, I set out to open the black box and provide direct evidence on how and why AI models appear to perform so well on accounting and finance-related tasks. I show that much of the superior performance of AI models can be attributed to artifacts of the modeling itself, rather than to mechanisms grounded in economics. Focusing on two key components of AI models, which may bias inferences in papers that rely on them, I first show that Large Language Model (LLMs) exhibit extremely poor numerical reasoning and thus application in these settings should proceed with caution. Second, I highlight that commercial LLMs suffer from significant look-ahead bias, which may explain a large portion of their predictive ability in various settings.},
year = {2026}
}
```

## License

The original source code and accompanying documentation in this repository are
licensed under the MIT License. See `LICENSE`.

This license does not apply to Compustat/WRDS data, third-party model
checkpoints, cached API responses, or other third-party materials. Those
materials remain subject to their respective licenses and terms of use.
