# How Stock Market Participants Use Generative Artificial Intelligence: Evidence from User-Platform Interaction Data

## Overview

This document describes the replication procedures for the paper. As the raw data processing code is not publicly released, this document describes the workflow for each step so that researchers can reproduce the analysis.

## Replication Workflow

---

### Step 1: Sample Construction

#### 1.1 Keyword-Based Company Filtering 

Filter queries mentioning A-share listed companies.

1. Load a keyword list containing official firm names, common name variants, and stock ticker abbreviations for all A-share listed companies.
2. Build an Aho-Corasick automaton from the keyword list for efficient multi-pattern string matching.
3. For each query, check whether the query contains any keyword match. If matched, label the row and record the matched keyword. If not, repeat on the answer.
4. Processing is parallelized across input files using Python `asyncio` with `ProcessPoolExecutor`.

#### 1.2 Stock-Market Relevance and Query Type Classification 

Fine-tune `bert-base-chinese` (https://huggingface.co/google-bert/bert-base-chinese) for two separate binary classification tasks: (a) distinguishing stock-market-related queries from unrelated ones, and (b) classifying queries as firm-specific or general. Each task uses a separately labeled training set and a separately fine-tuned model.

**Training:** 70/15/15 train/validation/test split (seed=42), `BertTokenizer` with `max_length=128`, `BertForSequenceClassification`, AdamW optimizer (lr=2e-5), `CrossEntropyLoss`, batch size 16, and early stopping (patience=3) on validation loss. Save the best model checkpoint.

**Batch prediction:** Load the saved checkpoint onto GPU. Iterate over all CSV files in the keyword-filtered directory. For each file, tokenize the query using `BertTokenizer` (`max_length=128`). Run inference in batches (batch size 500) with `torch.no_grad()`. Record predicted class via `argmax` on logits. Retain only rows predicted as stock-market-related.

#### 1.3 LLM-Based Classification of Query Topics and Tasks

**LDA topic modeling:** As the first step toward identifying query topics, run unsupervised LDA on query and answer texts.

1. Tokenize texts using `jieba`, removing stopwords (merged from cn_stopwords, hit_stopwords, and baidu_stopwords; https://github.com/goto456/stopwords) and single-character tokens.
2. Build a dictionary and bag-of-words corpus via `gensim.corpora.Dictionary`.
3. Train a `gensim.models.LdaModel` with a fixed random seed (seed=42) for reproducibility, evaluating multiple specifications by varying the number of topics. Model selection uses both perplexity scores and the OpTop metric (Lewis and Grossetti, 2022).

**Topic classification:** Based on manual review of the most prevalent LDA topics and their distinctive keywords, define 25 stable and interpretable topic categories (Appendix B, an alternative fully LLM-based topic classification is conducted as a robustness check (see Online Appendix OA.5)). Then apply an LLM to classify all query–answer pairs into these categories. Topics are not mutually exclusive.

**Task classification:** Manually review a random sample of 20,000 queries to identify task types that are frequently observed (see Appendix C for the resulting task list and representative examples). Then apply an LLM to classify all queries into these task categories, grouped into information awareness, acquisition, and integration stages following Blankespoor, deHaan, and Marinovic (2020), plus generative and educational tasks. Tasks are not mutually exclusive.

See Online Appendix OA.3 and OA.4 for workflow charts and prompt templates.

---

### Step 2: Variable Construction

#### 2.1 Named Entity Recognition — *Specificity*

Run Stanford Stanza (https://stanfordnlp.github.io/stanza/) Chinese NER (`lang='zh'`, processors `tokenize,ner`) on queries and answers separately, following Hope, Hu, and Lu (2016).

- **NER tokens:** all tokens with a non-`'O'` NER tag.
- **Clean tokens:** all tokens after removing stopwords (merged from cn_stopwords, hit_stopwords, and baidu_stopwords; https://github.com/goto456/stopwords), punctuation, and pure digits. Digits are excluded to avoid noise from reference markers and page numbers in corporate filings.
- ***Specificity*:** Let N denote the set of NER tokens (tokens with a non-'O' tag) and C denote the set of clean tokens (after removing stopwords, punctuation, and pure digits). Digits are excluded from clean tokens to avoid noise from meaningless numbers (e.g., reference markers and page numbers in corporate filings), but digits that appear as part of named entities are retained in N. Define the unique token set as U = N ∪ C. Then Specificity = total character length of tokens in N / total character length of tokens in U.

Texts are processed individually through the Stanza pipeline in configurable batches (e.g., 2,560 rows), with results appended incrementally to CSV. Run separately for questions and answers.

#### 2.2 Sentence-Level BERT Classifiers — *Forward Looking*, *Subjectivity*, *Sentiment* 

Fine-tune `hfl/chinese-roberta-wwm-ext` (https://huggingface.co/hfl/chinese-roberta-wwm-ext) for three sentence-level attributes. Each attribute uses a separately labeled training set and a separately fine-tuned model. We use `hfl/chinese-roberta-wwm-ext` for the more demanding sentence-level attribute classification tasks, as its whole-word masking pre-training yields stronger performance on fine-grained Chinese NLP tasks than `bert-base-chinese`.

- ***Forward Looking*** (binary): classifies each sentence as forward-looking (1) or not (0).
- ***Subjectivity*** (binary): classifies each sentence as subjective (1) or objective (0).
- ***Sentiment*** (3-class): classifies each sentence as positive (2), neutral (1), or negative (0).

**Training:** 80/20 stratified train/validation split (seed=42), `AutoTokenizer` with `max_length=128`, `AutoModelForSequenceClassification`, 10 epochs, batch size 8, best-model selection on validation accuracy (`load_best_model_at_end=True`) via the Hugging Face `Trainer` API.

**Batch prediction:** Split each text into sentences using Chinese punctuation delimiters. Discard sentences with fewer than 3 Chinese characters. Construct a custom PyTorch `Dataset` and `DataLoader` (batch size 128). For each batch, compute softmax probabilities and record the predicted class per sentence.

**Aggregation to question level:**
- *Forward Looking* ratio = count of forward-looking sentences / total sentences.
- *Subjectivity* ratio = count of subjective sentences / total sentences.
- *Sentiment* ratio = (count of positive sentences − count of negative sentences) / total sentences.

Computed separately for questions and answers.

#### 2.3 Dataset Assembly 

**Input:** query-level data with LLM classification labels, NER outputs, and BERT predictions.

**Output:** three datasets: firm-day-level (`firm_date.dta`), user–query-level (`user_interaction.dta`), and user-level (`user_dataset.dta`).

##### Query-Level Variable Construction (used across all tables)

**Company symbol refinement.** The LLM identifies company names and stock codes mentioned in queries and answers. Since LLM-extracted company names tend to be more reliable than the corresponding stock codes, we cross-validate and refine the codes against a crosswalk table of stock codes, official names, and common abbreviations (with market-specific prefixes and suffixes stripped before matching). For firm-specific queries, we match the LLM-extracted short name to the crosswalk table; if matched, the crosswalk symbol takes precedence, otherwise we fall back to the LLM-provided code only if it appears in the valid A-share symbol set. For general queries, we apply the same logic but additionally attempt full company name matching before falling back.

**Signal Align.** Classify each answer's directional trading signal (positive / neutral / negative) using an LLM. Compute 7-day forward simple returns from CSMAR daily adjusted closing prices (`Adjprcwd`, closing price adjusted for cash dividend reinvestment), forward-filled up to 5 days for missing values. Rank stocks within industry-date terciles and define alignment = 1 if the mapped signal ({积极, 略微积极} → +1, 中立 → 0, {消极, 略微消极} → −1) equals the mapped return rank ({1,2,3} → {−1,0,+1}). For multi-stock queries, average across stocks.

**Topic Diversity.** Share of additional topics in the answer (beyond those in the question) out of all topics not mentioned in the query, across 25 categories.

**Text quality measures.** *Fog*: Chinese Gunning-Fog readability index. *#References*: count of bracketed numeric references (e.g., \[1\]–\[40\]) in the answer via regex. *#Words*: count of Chinese characters in both questions and answers.

**User engagement indicators.** *Positive FB*: 1 if the user thumbs-up, shared, or copied the answer. *Thumbs Down*: 1 if the user clicked thumbs-down. *#Future Queries*: number of stock-market-related queries submitted by the same user after the current answer. *Post-question counts*: number of subsequent queries by the same user after 1 hour and 24 hours; total queries in all subsequent sessions.

**NER ratios and BERT attribute ratios.** Merge *Specificity* from Step 2.1 and *Forward Looking*, *Subjectivity*, *Sentiment* ratios from Step 2.2 into the query-level dataset.

##### Financial Term Extraction and User Segmentation (Table 2)

**Financial term extraction for general queries.** For queries classified as general, supplement LLM-extracted financial terms with Aho-Corasick matching against the top 500 most frequently extracted terms. Create binary mention indicators (*Any FM*), term counts, and cumulative first-3-query flags for specific metrics (ROE, Solvency, Accounts Receivable).

**User segmentation.**
- *High AI usage*: users at or above the 90th percentile of total query count within their month-of-entry cohort.
- *High metrics usage*: users at or above the 90th percentile of average financial metric mentions per query.

##### Export → `user_interaction.dta`

##### User-Level Dataset (Table 5) → `user_dataset.dta`

Filter to users with ≥ 4 queries. Aggregate early behavior (first 3 queries: cumulative financial metric flags, mean mention rates, mean *Specificity*) and post-onboarding behavior (queries 4+: metric mention sums/rates, mean BERT attribute ratios, mean *Specificity*, mean query length).

##### Firm-Day Dataset (Tables 3, 6–7) → `firm_date.dta`

Aggregate query-level data to stock symbol × calendar day across six component tables:
1. Task counts (awareness, acquisition, integration subtasks).
2. Topic counts and firm-specific query counts.
3. Average *Sentiment* across all answers.
4. Engagement-stratified *Sentiment* (*Avg Sentiment PosFB* vs. *Avg Sentiment NonPosFB*).
5. Engagement-stratified query counts (by positive feedback flag).
6. User-segment-stratified query counts (by `month90ge` and `avg_90_qge`).

Merge all components by (firm, date) via sequential outer joins. Create a balanced panel covering all firm × day combinations from January 1 to June 30, 2024.

---

### Step 3: Descriptive Analysis and Figures 

- **Figure 1 (Intraday query density by task type):** Restrict to trading days (via `akshare`); bin queries into half-hour intervals; compute the average density of Awareness, Acquisition, and Integration tasks per interval.
- **Figure 2 (Per-user query distribution):** Identify users who first queried in January 2024 (to mitigate truncation bias); compute and plot the frequency distribution of total query counts.
- **Figure 3 (Task composition by query order):** Assign sequential query order within each user (capped at 20); compute mean likelihood of each task type by query position.
- Figures are produced in R using `ggplot2` (see `ELLW2026_figure.html`).
- **Appendix B (Topic keywords):** Tokenize questions by topic using `jieba` with a custom economics dictionary (CNEconDict ([https://github.com/sijichun/CNEconDict](https://github.com/sijichun/CNEconDict)) and Jiang, Lee, Martin, and Zhou (2019)). Compute relative TF-IDF scores—defined as the mean TF-IDF within a topic scaled by its share of the total TF-IDF across all topics—to upweight topic-distinctive terms, and extract the top keywords per topic.
- **Appendix C (Task frequencies):** Compute proportions of queries tagged with each task subcategory.

---

### Step 4: Regression Analyses

All regressions are estimated in Stata. Execute the do-files in `Stata_code_log_folder` in numerical order:

| Do-file                            | Description                                                                                                              |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `01_process_csmar_daily.do`        | Process CSMAR daily-frequency data (returns, spreads, volume, VPIN, etc.)                                                |
| `02_process_csmar_yearly.do`       | Process CSMAR annual data (firm characteristics)                                                                         |
| `03_other_dataset_construction.do` | Construct firm-year dataset for online appendix and supplementary datasets                                               |
| `04_dataset_cleaning.do`           | Merge all data sources and prepare analysis datasets: `firm_day_dataset.dta`, `user_interaction.dta`, `user_dataset.dta` |
| `05_main_tables.do`                | Produce Tables 1–7 in the main paper                                                                                     |

---

## Contact

For questions about the code or data, please contact the authors.
