Wall Street sign peeled back to reveal digital grid

Can AI Map Wall Street’s Genome?

The same architectures used to process language could shed new light on financial markets.

The AI boom began unassumingly, with a 2017 research paper titled “Attention Is All You Need” about how to translate text between languages. At the time, the findings did not look like the start of a trillion-dollar revolution. They looked more like a technical advance toward solving an engineering problem in natural language processing, then just one of several developing branches of artificial intelligence. It seems evident that the authors, all Google employees, had little inkling of the paper’s future fame in the way they simply went with a randomized order for their names rather than jockeying for first position.

Much has stemmed from that paper. It largely created the AI architecture that underpins large language models, which in turn run chatbots. It also gave it a name: the transformer. And although transformers grew out of natural language processing, they haven’t stayed in that domain. Netflix has trained its own version on viewing histories to improve recommendations; Stripe trained one on payment data to spot fraud; Google DeepMind trained one on weather patterns to improve forecasts; and the list goes on.

Now transformer models are being trained on the “language” of financial data, bringing their prediction-making capabilities to the world’s capital markets, whose worth is approaching $300 trillion. Sophisticated trading firms are already using them in the latest evolution of market sparring.

Markets may be the ultimate test for transformers, which are being used to illuminate unseen, dynamic, ever-changing relationships in stock returns, high-frequency trading data, and more. One group of researchers that includes Chicago Booth’s Ralph S. J. Koijen, collaborating on what they call the Market Genome Project, has turned a transformer model on investor portfolios to find out what moves prices. A stock’s momentum, a sector’s rotation, the aftershocks of an earnings surprise—these often aren’t random but reflect patterns that unfolded months earlier and expectations about future events. They’re traces of how investors think and act. The question is whether transformers can learn to read those traces.

How we got here

Let’s start with a simple lesson about transformers. “Paris → France as Berlin → __________.” If you were presented with this analogy, you would know that the missing word is Germany. The solution does not involve step-by-step reasoning but rather recognizing a relationship and applying it to find the answer. Paris is the capital of France, a country. Berlin is also a capital, and therefore the missing information must be the country for which Berlin is the capital.

Before the 2017 paper came out, machine learning generally processed language word by word and struggled to capture contextual relationships spread out across text. The models did not have a good mechanism for saying, essentially, “This thing here depends on something that appeared earlier.”

Transformers changed that. They process whole sentences at once, solving the problem that earlier models had with understanding these relationships. More precisely, they learn which pieces of information pertain to which other pieces, regardless of position. That single shift turned context from a liability into an asset. It made it possible to pretrain models on vast, messy data and extract useful structures. The architecture proved able to capture long-range dependencies—that is, connect distant words that matter to each other.

The best place to look for what moves prices is in investor portfolios, the researchers argued.

Markets have their own sequences of long-range data, which would seem ready-made for transformers. Much of investing is front-loaded with information. Investors examine cash flows, compare business models, and think through how the economy might evolve over the next year or two—all before investing capital.

These analyses resolve into decisions about what to own, how much of that asset to hold, and what to pair it with. The choices compress all of the prior work into a single transaction. This bottom-up approach, focused on individual companies or industries, has traditionally framed how asset pricing is understood. Fundamental analysis drives investor decisions, and prices and portfolio allocations are the outcomes.

But this framing misses that investors’ actual portfolio choices contain information of their own. In 2015, Koijen and Princeton’s Motohiro Yogo began developing what they called a demand system approach to asset pricing. Since asset prices ultimately reflect a supply of assets and investor demand for them, the best place to look for what moves prices is in investor portfolios, the researchers argued.

Starting with institutional portfolios, Koijen and Yogo reviewed quarterly US equity holdings from Form 13F filings to the Securities and Exchange Commission between 1980 and 2017. These filings reported the stock positions of institutional investors managing more than $100 million. The data included in the sample covered roughly 68 percent of the US stock market during that time period.

Their findings upended conventional wisdom about prices. Supply dynamics such as changes in the institutional investors’ share counts (the number of shares outstanding) explained about 2 percent of the variation in returns. Changes in firm characteristics explained about 10 percent. Dividend yield explained less than 1 percent. Put together, all supply-side factors accounted for only about 12 percent of the variation.

Koijen and Yogo find that demand mattered more, but not in the obvious ways. Growth or shrinkage in investors’ assets under management explained just over 2 percent of variation, while changes in how investors weight firm characteristics explained about 5 percent.

The big driver was something harder to see: shifts in latent demand not captured by traditional characteristics such as market capitalization, profitability, and value. Changes in which stocks institutions chose to own explained about 23 percent of the variation. Changes in how much they owned explained nearly 58 percent. These demand shifts accounted for more than 80 percent of the differences in stock returns.

Thus portfolios were compressed records of how investors processed information over time. The way investors valued the stocks was there, embedded in the underlying holdings. Being that thousands of institutional investors make frequent portfolio decisions, these holdings contained a regularly updated picture of how sophisticated money saw the market. The question for the researchers became whether a model could extract all of that information.

What’s behind stock returns?

Yes, supply and demand factors affect stock prices. But research suggests that most differences in returns reflect choices that are unobservable but can be inferred from studying investors’ portfolios.

Transformers to the rescue

To answer that, Koijen and Yogo worked with Harvard’s Xavier Gabaix and New York University’s Robert J. Richmond. They used stock holdings from data company FactSet covering mutual funds, exchange-traded funds, closed-end funds, variable annuity funds, and hedge funds sampled quarterly from 2005 through 2022. By the end of the sample, the dataset included roughly 2,000 investable firms, more than 8,000 institutional investors, and more than 1 million individual stock holdings per quarter.

The researchers adapted BERT, Google’s transformer architecture, to work on portfolios. The training process for their adapted model directly mirrored how language models such as BERT learn. In language, BERT sees a sentence with a word blanked out—such as “The Fed decided to the target rate.”—and uses context to predict what’s missing. The researchers’ model saw a portfolio with a stock masked out and predicted what belonged there based on everything else the investor owned. As a test, Gabaix, Koijen, Richmond, and Yogo showed their model the top-three holdings for the ARK Innovation ETF in July 2023 and then asked it to predict the fourth.

During training, they masked 15 percent of stocks in each portfolio for the model. They also tested simpler approaches—recommender systems based on principal component analysis (which simplifies large, high-dimensional datasets), and word2vec (the shallower neural network that preceded transformers in language research).

The output of their model was what the researchers call asset embeddings—vector representations that capture how investors see individual stocks, learned entirely from the pattern of where each is held. The model also produced investor embeddings that characterize each institution’s strategy.

The researchers used three benchmarks to evaluate their model and the two other approaches. The first benchmark was relative valuation. The researchers compared four-dimensional asset embeddings against four firm characteristics that decades of academic research have identified as the best predictors of price: asset growth, the ratio of dividends to assets, market beta, and profitability. The firm characteristics explained about 15 percent of the cross-sectional variation (as in, the variation observed at a single point in time). But the embeddings explained over 50 percent. And when scaled up to 128 dimensions, they explained over 70 percent. Adding firm characteristics on top barely moved the needle, suggesting the embeddings already captured most of the information those characteristics contain.

Return comovement, another benchmark, refers to the tendency of asset prices to move in sync. Six characteristics from the Fama-French five-factor model (developed by Booth’s Eugene F. Fama and Dartmouth’s Kenneth R. French) plus momentum explained about 5 percent of variation in monthly stock returns—a high bar refined over decades. The best asset embeddings reached about 7 percent. And combining the two always outperformed characteristics alone, evidence that the embeddings contain independent information about how returns move together.

The model seemed to be picking up on a demand pattern that went beyond standard industry classifications.

The most dramatic result came on the managed-portfolio benchmark, which tests whether a model can predict what a fund actually holds. The researchers’ model climbed from 35 percent predictive power at four dimensions to over 60 percent at higher dimensions. The recommender systems plateaued around 15 percent. The transformer’s advantage reflected its ability to produce contextualized embeddings—representations of a stock that shift depending on what else is in the portfolio. Apple, when in a portfolio alongside Dell and Intel, looks like a computer hardware company. But Apple alongside Netflix and Spotify looks like a digital content provider. The simpler models assigned each stock a single fixed vector regardless of context, and they missed this.

The researchers also performed a test to show that asset embeddings are distinct from text-based embeddings. They did this by comparing their asset-based approach to models, from AI companies Cohere and OpenAI, that use text-based embeddings, which represent firms on the basis of descriptions of their businesses—and which performed poorly.

Text captured semantic similarity. When OpenAI’s model was asked to find the firms most similar to Apple, its top result was the cloud computing company Appian—a company that shares the first three letters but little else beyond that it’s also a tech company. For Walmart, the model returned Walgreens.

The asset-embeddings model told a different story. In late 2022, according to the researchers’ model, the nine companies considered closest to Apple were Adobe, Alphabet, Amazon, Costco, Disney, Microsoft, Nike, NVIDIA, and Tesla. For Citigroup, the closest peers included AIG, Altria Group, ExxonMobil, General Motors, Goldman Sachs, and Wells Fargo. The model seemed to be picking up on a demand pattern that went beyond standard industry classifications—which the researchers note is only one of many characteristics investors weigh in portfolio decisions, alongside profitability, riskiness, and exposure to environmental, social, and governance concerns.

Just as the model learns a stock’s meaning from who holds it, it also learns which investors are following similar investment strategies. When the researchers ran their model on the holdings of AQR Arbitrage (an affiliate of AQR Capital Management), the company’s closest peers were BCK Capital Management, Water Island Capital, and the managers of several merger-arbitrage funds. The fund names don’t reveal a connection, but when the researchers checked SEC filings, the strategies matched: BCK followed what was described as a “special situations” strategy, and Water Island specialized in event-driven investing, in which decisions are linked to mergers, spin-offs, bankruptcies, and the like. The model identified the similarity purely from the pattern of holdings.

With text data (from earnings-call transcripts, say) and their model, the researchers can figure out why some assets are clustered together. For example, investors might see in a set of companies some common growth opportunities, risk exposures, or corporate strategies. The researchers show this live on mktgen.ai, the website for their Market Genome Project, where AI also generates narratives about what it finds.

The language of the market

Investor portfolios are just one kind of financial sequence. The same logic applies to stock returns, limit order books, and high-frequency-trading data.

Across these domains, early results from other research suggest the same lesson: Each dataset has its own grammar. A model trained on millisecond-level order-book data won’t be able to predict daily stock returns or bond portfolios. The architecture is general. The training has to be specific.

University of Manchester’s Eghbal Rahimikia, University College London’s Hao Ni, and Shanghai University’s Weiguan Wang tested this directly. They find that off-the-shelf AI models, which are pretrained on generic data such as weather patterns or electricity usage, perform poorly when applied directly to the stock market. This is particularly true for time series foundation models. When the researchers pretrained the same architectures from scratch using only financial data—2 billion daily observations of stock returns spanning 94 countries and 34 years—performance jumped.

“Scaling model size and expanding data coverage offer a promising path toward improving predictive performance in asset-return forecasting,” says Rahimikia. “However, progress remains constrained by computational limitations and the scarcity of large-scale, high-quality financial data.”

Other researchers are going faster, down to the millisecond level. Tampere University’s Eljas Linna, Kestutis Baltakys, Alexandros Iosifidis, and Juho Kanniainen trained a transformer-based model on limit-order-book messages with superior predictive results.

“The models themselves don’t really care if the sequences are about words or limit-order-book messages, as long as there are patterns,” says Kanniainen, who presented the research findings at a NeurIPS workshop. “When it comes to these messages and the whole stream of data, patterns emerge because trading and market-making algorithms react to past events, continuously creating chains of reactions.”

Kanniainen and his coresearchers used 470 million messages from the Nasdaq TotalView-ITCH feed covering four major stocks: Apple, Facebook (now Meta), Intel, and Microsoft. The data spanned from May to September 2015 and were divided into roughly 919,000 sequences for training and testing. The researchers adapted BERT’s architecture to handle limit-order-book data by creating a one-token-per-message scheme. This allowed their model—which they call LOBERT—to process a unified representation of discrete trade types (such as “new order” or “execution”) alongside continuous values for price, volume, and time. The model underwent “masked message modeling”—it learned the language of the market by predicting missing pieces of a message sequence—before being fine-tuned for specific tasks including price forecasting.

The computing power and data requirements are enormous. . . . These are not resources most researchers or even most trading firms can access.

In predicting the next full market message, LOBERT achieved nearly 28 percent accuracy, compared with only about 6 percent for previous leading models and about 0.3 percent for a random model. (There are 293 message classes.) This represents a massive leap in a model’s ability to understand the complex, high-dimensional sequence of market events. By consolidating entire messages into single tokens, LOBERT processes approximately 20 times fewer tokens per sequence than previous methods.

The model also demonstrates strong calibration when filtering for confidence. Midprice prediction—a technique that forecasts the direction of an asset’s midprice—is essentially a classification problem: Will the price go up, stay the same, or go down? When the confidence threshold is raised above 0.9, the F1 score (a metric that measures accuracy) increases from 0.51–0.55 to 0.82–0.88.

“The most surprising thing is the performance gap between this new model and older ones” that were previously state-of-the-art, Kanniainen says. “Those older models used snapshots of the limit order book as input but did not handle the message data itself. LOBERT uses a sequence of messages combined with snapshots, which clearly improves predictions.”

His model extracts information from unfilled orders, not just filled ones. “When I talk to market makers, they look at the message flow, because trading algorithms react to transactions, cancellations, and other messages that just took place. Beyond LOBERT, we are currently testing the next generation of our message-based transformer models, which have proven not only more accurate but also faster in tests on recent equity and crypto order book message data.”

Transformers in training

One of the largest experiments of training transformers on financial data is happening inside Hudson River Trading, a proprietary trading firm and market maker that accounts for roughly 10 percent of US equity trading volume. HRT is building foundation-style models trained on decades of global market data by applying techniques similar to those used in frontier language models for automated trading.

Where this leads

These results come with important caveats. The most straightforward: Traditional quantitative methods still hold an edge in many settings. In Rahimikia, Ni, and Wang’s study, gradient-boosted trees—ensembles of decision trees that quantitative traders have refined for years—returned about 47 percent, annualized, compared with about 37 percent for the best transformer model trained from scratch on financial data in the baseline comparison. The gap has narrowed substantially from where off-the-shelf models started, but it hasn’t closed. Pretraining these AI models on larger financial datasets, augmenting them with synthetic data, and carefully optimizing the models’ hyerparameters further improved their performance. Gradient-boosted trees are faster to train, cheaper to run, and better understood. Whether continued scaling will ultimately erase that advantage remains an open question.

In Gabaix, Koijen, Richmond, and Yogo’s study of asset embeddings, the picture is also mixed, although in a different way. The transformer outperformed the alternatives at predicting managed portfolio holdings—but on relative valuations, a simpler recommender system actually beat it. The researchers note that different models win on different benchmarks, and that the transformer may benefit from fine-tuning that it hasn’t yet received. Performance gains were real but uneven.

The computing power and data requirements are enormous. Rahimikia, Ni, and Wang logged roughly 50,000 GPU hours to train their models across 34 years of global stock data. These are not resources most researchers or even most trading firms can access.

Interpretability is another concern. Asset embeddings, by design, are latent vectors—lists of numbers without direct economic meaning. Gabaix, Koijen, Richmond, and Yogo acknowledge this and propose using LLMs to interpret what the embeddings capture, by feeding earnings-call transcripts through a model and asking it to identify shared risks among the companies that the embeddings grouped together. For regulators or risk managers who need to explain why a trade was made or why a model flagged a position, that may not be enough. (To learn more about interpretability, read “Can We Break Open AI’s Black Box?”)

A separate concern cuts across all the models: Financial regimes change. A model trained on two decades’ worth of data has learned the patterns of a specific era, like the one we’re in, marked by declining interest rates, central-bank interventions, and the growth of passive investing. Whether those patterns hold in a rising-rate environment, or during a market-structure shift, is untested.

Gabaix, Koijen, Richmond, and Yogo train their model quarterly, essentially reducing each era to three months. The LOBERT model addresses the challenge partly by making predictions only when confidence is high, filtering out cases where the model’s softmax probability doesn’t clear a 90 percent threshold. (Like a teacher grades on a curve to normalize class results, softmax takes the model’s scores and pushes the strongest patterns toward 1 while lowering weaker results toward 0 so a clear winner stands out.) But selective prediction is a partial solution. It tells you when a model is unsure; it doesn’t tell you when a model is confidently wrong because the regime has shifted in a way the model doesn’t recognize.

All that said, evidence is accumulating across time scales and data types that the same architectural breakthrough that revolutionized language processing appears to transfer to financial markets when models are trained on domain-specific data. In a separate study, Gabaix, Koijen, Richmond, and Yogo used an embeddings-informed model to extract risk signals from investors’ bond holdings. They find that it did a better job than metrics typically used by credit rating agencies of flagging investment-grade companies likely to fall to junk status. (For more, read “AI Identifies Early Signs of Ratings Downgrades.”)

At the daily level, transformers trained on global stock returns generate meaningful trading signals. At the millisecond level, they predict order-book behavior with accuracy that dwarfs prior approaches. At the industry frontier, Hudson River Trading’s scaling results suggest that more data and larger models continue to improve performance in ways that mirror the dynamics of LLMs, even if the traditional quant benchmark still edges out transformers in some head-to-head tests.

The question that will define the next chapter of this research is whether continued scaling will close that gap or whether transformers will plateau against the adversarial, low-signal-to-noise environment of financial markets. If scaling works, AI, with bigger models and more computing power, will uncover more of the hidden relationships in markets. If it does not, the transformer revolution in finance will remain a promising but incomplete translation of an architecture that found its first language in text.

Works Cited

Eugene F. Fama and Kenneth R. French, “A Five-Factor Asset Pricing Model,” Journal of Financial Economics, April 2015.
Xavier Gabaix, Ralph S. J. Koijen, Robert J. Richmond, and Motohiro Yogo, “Asset Embeddings,” Working paper, April 2025.
——— “Upgrading Credit Pricing and Risk Assessment Through Embeddings,” Working paper, February 2025.
Marc Khoury, “Foundation Models for Automated Trading,” International Conference on Machine Learning presentation, July 2025.
Ralph S. J. Koijen and Motohiro Yogo, “A Demand System Approach to Asset Pricing,” Journal of Political Economy, August 2019.
Eljas Linna, Kestutis Baltakys, Alexandros Iosifidis, and Juho Kanniainen, “LOBERT: Generative AI Foundation Model for Limit Order Book Messages,” NeurIPS Workshop, December 2025.
Eghbal Rahimikia, Hao Ni, and Weiguan Wang, “Re(Visiting) Time Series Foundation Models in Finance,” Working paper, November 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention Is All You Need,” Preprint, arXiv, June 2017, arXiv: 1706.03762.