Incompressible Knowledge Probes

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

· 1 min · 337 words

topic/technologycomputer science/machine learning

Summary

This paper introduces Incompressible Knowledge Probes (IKPs), a method to estimate the parameter counts of closed-source LLMs by measuring their factual capacity. By using a benchmark of 1,400 obscure, incompressible facts and calibrating it against 89 open-weight models, the authors establish a log-linear mapping between knowledge and model size. They find that factual capacity continues to scale linearly with parameters—challenging the notion of scaling saturation—and that total parameters (rather than active parameters in MoE models) are the primary driver of factual knowledge.

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

High-Level Summary

The Problem:
Frontier labs (e.g., OpenAI, Anthropic) do not disclose the parameter counts of their closed-source models. Current estimation methods relying on “inference economics” (latency/cost) are noisy because they are influenced by external factors like hardware, batching, and serving stacks.

The Core Insight:
The authors identify a tighter intrinsic bound: storing a specific number of unique, incompressible facts requires a minimum number of weights. Therefore, measuring a model’s “factual capacity” can provide a lower bound for its parameter count.

The Methodology: Incompressible Knowledge Probes (IKPs)

  • The Benchmark: 1,400 factual questions across 7 tiers of obscurity. These are “incompressible,” meaning they cannot be derived through reasoning or architectural shortcuts; they must be explicitly stored in the model’s weights.
  • Calibration: A log-linear mapping was created using 89 open-weight models (135M to 1,600B parameters) from 19 different vendors.
  • Application: This mapping is used to estimate the size of black-box models based on their accuracy on the IKP benchmark.

Key Findings

  • Mixture-of-Experts (MoE): Total parameter count is a much better predictor of factual knowledge than the number of active parameters.
  • Safety Tuning: For models with heavy safety alignment, the estimates serve as lower bounds because refusal policies may hide “known but refused” capacity.
  • Scaling Trends: Contrary to some theories regarding the saturation of reasoning (the “Densing Law”), factual capacity continues to scale log-linearly with parameters across different generations and vendors.

Link: arXiv:2604.24827