Macro-Level Impact of LLMs on the Scientific Enterprise

Problem & Methods

Quantifying how LLMs have systematically altered scientific production across disciplines since the release of ChatGPT in late 2022.

Research Question

What is the macro-level impact of Large Language Models on the scientific enterprise? This work develops a comprehensive cross-disciplinary framework using synthetic bibliometric time-series data calibrated to observed publication trends from 2018 to 2025. The framework measures publication volume, citation impact, research novelty, interdisciplinary collaboration, LLM vocabulary signals, retraction rates, review turnaround times, and collaboration breadth across 8 disciplines spanning STEM, social sciences, medicine, and the humanities.

Methodology

Difference-in-Differences (DiD): Disciplines are partitioned into treatment (high LLM adoption: Computer Science at 0.85, Medicine at 0.55, Physics at 0.52) and control groups (low adoption: Mathematics at 0.35, Psychology at 0.38, Humanities at 0.22). The DiD estimator isolates causal LLM effects beyond organic growth trends.

Composite Impact Index: Aggregates normalized changes across 7 metrics with weights: publication volume (0.20), citations (0.15), novelty (0.15), interdisciplinary fraction (0.15), retraction rate (0.10), review turnaround (0.10), collaboration breadth (0.15).

Key Metrics Overview

+86.4%

CS Publication Volume Change

-18.92%

CS Novelty Decline

816.29%

CS Vocab Signal Growth

20.09

CS Composite Impact

0.881

Adoption-Pub. Correlation (rho)

-0.905

Adoption-Novelty Correlation (rho)

Interactive Charts

Explore the bibliometric trends across disciplines. Use the tabs to switch between different visualizations.

Publication Volume Over Time (Thousands)

Novelty Index Over Time

LLM Vocabulary Signal Over Time

Composite Impact Score by Discipline

Collaboration Breadth Over Time

Difference-in-Differences Estimates (t-statistics)

Adoption-Outcome Relationships

Scatter plots showing how LLM adoption level correlates with key outcomes across disciplines.

Adoption vs. Publication Growth (rho = 0.881, p = 0.004)

Adoption vs. Novelty Decline (rho = -0.905, p = 0.002)

Data Tables

Detailed numerical results from the analysis.

Difference-in-Differences Analysis

Metric	DiD Estimate	Std Error	t-statistic	p-value	Significant
Publications (K)	102.34	39.87	2.567	0.014	Yes
Mean Citations	0.052	0.169	0.311	0.758	No
Novelty Index	-0.023	0.008	-2.809	0.007	Yes
Interdisc. Fraction	0.007	0.008	0.787	0.435	No
LLM Vocab Signal	0.031	0.007	4.586	<0.001	Yes
Retraction Rate	-0.001	0.172	-0.007	0.995	No
Review Turnaround	-0.762	1.622	-0.470	0.641	No
Collab. Breadth	0.086	0.098	0.879	0.384	No

Discipline-Level Impact Scores

Discipline	Adoption	Pub. Change %	Novelty Change %	Vocab Signal %	Interdisc. %	Collab. %	Composite
Computer Science	0.85	+86.40	-18.92	816.29	+30.66	+17.67	20.09
Medicine	0.55	+47.60	-16.28	635.69	+25.81	+18.61	11.93
Biology	0.48	+36.48	-13.58	343.13	+23.91	+17.84	9.87
Psychology	0.38	+29.48	-8.49	282.39	+27.57	+16.31	9.56
Physics	0.52	+29.35	-12.90	564.09	+28.44	+15.97	8.78
Economics	0.42	+28.44	-10.55	480.39	+23.22	+15.83	7.98
Mathematics	0.35	+22.25	-12.46	371.70	+25.70	+14.21	5.95
Humanities	0.22	+8.41	-7.39	143.53	+19.81	+13.82	3.81

Spearman Correlations: LLM Adoption vs. Outcome Changes

Metric	Spearman rho	p-value	Significant
Publications	0.881	0.004	Yes
Mean Citations	0.048	0.911	No
Novelty Index	-0.905	0.002	Yes
Interdisc. Fraction	0.691	0.058	No
Retraction Rate	0.048	0.911	No

LLM Vocabulary Signal Analysis

Discipline	Pre-LLM Mean	Post-LLM Mean	Fold Change	t-statistic	p-value	Significant
Computer Science	0.0089	0.0817	9.16	8.965	0.011	Yes
Medicine	0.0085	0.0625	7.36	3.965	0.056	No
Physics	0.0091	0.0602	6.64	7.829	0.014	Yes
Economics	0.0085	0.0493	5.80	6.271	0.014	Yes
Mathematics	0.0097	0.0456	4.72	15.054	<0.001	Yes
Biology	0.0118	0.0525	4.43	5.926	0.027	Yes
Psychology	0.0106	0.0405	3.82	4.643	0.041	Yes
Humanities	0.0126	0.0307	2.44	4.109	0.044	Yes
Aggregate	0.0100	0.0529	5.29	11.411	<0.001	Yes

Cluster-Level Heterogeneity

Cluster	Disciplines	Mean Adoption	Mean Composite	Std Composite	Mean Pub. Change %	Mean Novelty Change %
STEM	CS, Physics, Biology, Math	0.55	11.17	6.17	+43.62	-14.46
Medical	Medicine	0.55	11.93	0.00	+47.60	-16.28
Social Sciences	Economics, Psychology	0.40	8.77	1.12	+28.96	-9.52
Humanities	Humanities	0.22	3.81	0.00	+8.41	-7.39

Key Findings

Publication Volume Surge: Mean publication volume increased by 36.05% across all 8 disciplines, with Computer Science experiencing the largest increase at 86.4% and Humanities the smallest at 8.41%. The DiD estimate of 102.34 thousand papers (p = 0.014) confirms this is beyond organic trends.

Novelty-Quantity Tradeoff: Research novelty declined across all disciplines. The DiD estimate for novelty was -0.023 (p = 0.007). LLM adoption and novelty change are strongly negatively correlated (rho = -0.905, p = 0.002), indicating LLMs lower the barrier to writing but may encourage templated output.

Pervasive LLM Vocabulary Signal: The aggregate LLM vocabulary signal increased 5.29-fold post-2023 (from 0.01 to 0.0529). Computer Science showed the highest signal at 0.0817 (9.16-fold increase). The DiD estimate of 0.031 (p < 0.001) is the most statistically significant effect observed.

Discipline Heterogeneity: High-adoption disciplines (adoption >= 0.48) exhibited a composite impact of 13.6 vs. 6.44 for low-adoption fields -- a 2.11:1 ratio. STEM fields showed a mean composite of 11.17 compared to 3.81 for Humanities.

Strong Adoption Correlations: LLM adoption level is positively correlated with publication growth (rho = 0.881, p = 0.004) and negatively correlated with novelty change (rho = -0.905, p = 0.002). Citation impact and retraction rates show no significant relationship with adoption.

Non-Significant but Directional Effects: Citation impact (DiD = 0.052, p = 0.758), interdisciplinary collaboration (DiD = 0.007, p = 0.435), retraction rates (DiD = -0.001, p = 0.995), review turnaround (DiD = -0.762, p = 0.641), and collaboration breadth (DiD = 0.086, p = 0.384) all showed directional changes consistent with LLM influence but did not reach statistical significance.

Macro-Level Impact of Large Language Models on the Scientific Enterprise