llms.blog
research

MIT, Stanford, and 12 Academic Labs Launch Public AI Observatory to Track Real-World LLM Usage

A consortium of researchers from MIT, Stanford University, and 12 other academic institutions has launched the Public AI Observatory (ai-observatory.org), an independent, auditable data repository designed to measure how individuals interact with artificial intelligence assistants in real-world settings. The initiative aims to address the empirical opacity surrounding commercial LLM deployment. While frontier AI developers such as OpenAI and Anthropic periodically release aggregated user metric

axc
axc
1 min
MIT, Stanford, and 12 Academic Labs Launch Public AI Observatory to Track Real-World LLM Usage

A consortium of researchers from MIT, Stanford University, and 12 other academic institutions has launched the Public AI Observatory (ai-observatory.org), an independent, auditable data repository designed to measure how individuals interact with artificial intelligence assistants in real-world settings.

The initiative aims to address the empirical opacity surrounding commercial LLM deployment. While frontier AI developers such as OpenAI and Anthropic periodically release aggregated user metrics, those disclosures remain proprietary, unstandardized, and unauditable by independent academic or regulatory bodies.

Public AI Observatory Usage Breakdown

Dataset Scope and Methodology

Co-led by Anka Reuel of Stanford's Trustworthy AI Research (STAIR) Lab and Shayne Longpre of the Data Provenance Initiative and MIT Media Lab, the observatory aggregates anonymized, consented interactions across seven distinct research corpora.

Public AI Observatory: Core Benchmark Metrics
----------------------------------------------
Total Consented Conversations:    24,521
Total Conversational Turns:       85,633 (exchanges up to 92,493)
Unique Users Represented:         ~5,000
Models Covered:                   52 (including ChatGPT, Claude, Gemini, Grok)
Temporal Range:                   2023 to 2025
Taxonomy Classification Features: 145 distinct labels

The platform classifies prompts using a 145-feature evaluation framework covering user intent, task complexity, safety boundaries, domain classification, and interactive depth.

Empirical Findings: Personal vs. Workplace Utility

Early findings from the dataset indicate that approximately half of tracked chatbot sessions involve personal tasks, creative experimentation, or companionship rather than structured workplace productivity.

Interaction Domain Distribution
-------------------------------
Personal / Casual Exploration:    ~50%
Technical & Workplace Assistance: ~35%
Educational / Academic Support:   ~15%

The observatory's maintainers noted that public data infrastructure is necessary for assessing model drift, misalignment patterns, and real-world societal impact. The dataset and associated evaluation tooling are publicly accessible at ai-observatory.org.

Sources

researchPolicyEvalsLLMsAI
ShareX (Twitter)LinkedIn

Written by

axc

More to read

  • Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS

    Low-Precision Quantization Kernels in Production: Comparing Marlin, ExLlamaV2, FlashInfer, and BitBLAS Architecture, Memory Bandwidth, and Decoding Throughput Autoregressive large language model (LLM) serving operates under two distinct compute regimes: a compute-bound prefill phase and a memory-bandwidth-bound decode phase. While processing the initial prompt involves matrix-matrix multiplications (GEMM) with high arithmetic intensity, generating tokens one by one requires matrix-vector multip

    1 min

  • xLSTM: How Exponential Gating and Matrix Memory Scale Recurrent Neural Networks

    xLSTM: How Exponential Gating and Matrix Memory Scale Recurrent Neural Networks For over two decades following its introduction by Hochreiter and Schmidhuber (1997), the Long Short-Term Memory (LSTM) network served as the dominant architecture for sequence modeling. By introducing the constant error carousel and multiplicative gating, LSTMs mitigated the vanishing gradient problem that plagued vanilla recurrent neural networks. However, the emergence of the Transformer architecture (Vaswani et

    1 min

  • Anthropic Nears $7B Acquisition of AI Infrastructure Startup Decart Ahead of IPO

    Anthropic is finalizing negotiations to acquire Israeli artificial intelligence infrastructure startup Decart in a transaction valued at approximately $7 billion, according to reporting from Calcalist and Reuters. The acquisition, expected to be settled primarily in Anthropic equity, would mark the Claude developer's largest purchase to date as it prepares for a planned initial public offering. Founded in September 2023 by Dr. Dean Leitersdorf and Moshe Shalev, Decart specializes in hardware-ag

    1 min