Subnet 9: Pretrain - Foundation Model Training at Scale

Executive Summary

Pretrain (Subnet 9) represents one of the most ambitious technical undertakings in the Bittensor ecosystem: enabling the distributed pretraining of large foundation models across a decentralised network of miners. With 1,180 active miners contributing GPU resources and a Khala Score of 82, Pretrain has demonstrated that decentralised model training is not only possible but increasingly competitive with centralised alternatives.

This report evaluates Pretrain's technical architecture, training efficiency relative to centralised benchmarks, economic viability for participants, and the long-term strategic case for decentralised foundation model training. Our assessment is cautiously optimistic — the subnet has made impressive technical progress, but faces significant challenges around coordination efficiency and the scaling laws that favour centralised training clusters.

Technical Architecture

Pretrain's architecture addresses one of the hardest problems in distributed computing: coordinating large-scale model training across heterogeneous, geographically distributed hardware without the tight coupling of traditional data centre setups.

Training Paradigm

The subnet employs a novel approach to distributed pretraining that combines elements of federated learning with competitive incentive mechanisms. Rather than requiring synchronous gradient updates across all participants (which would be impractical at this scale), Pretrain uses an asynchronous training protocol:

• Model checkpointing: A reference model checkpoint is published periodically by the subnet's coordination layer. Miners download this checkpoint as their training starting point.
• Local training: Each miner trains the model on their local data partition for a fixed number of steps, applying their own hardware-optimised training configurations.
• Weight submission: Miners submit their updated model weights (or weight deltas) to validators for evaluation.
• Competitive selection: Validators evaluate submitted weights against held-out evaluation sets. The best-performing weight updates are merged into the next reference checkpoint.

This competitive selection mechanism is Pretrain's key innovation. Rather than trying to coordinate a single distributed training run (which faces enormous communication overhead), the subnet lets miners independently optimise and then selects the best outcomes. This is more akin to evolutionary optimisation than traditional distributed training.

Model Architecture

Pretrain currently focuses on transformer-based language models, with the current training target being a 7B parameter model based on a modified LLaMA architecture. The team has outlined a roadmap to scale to 13B and eventually 70B parameter models as the mining infrastructure grows and coordination mechanisms improve.

The choice of model size is deliberate — 7B parameters is large enough to demonstrate meaningful capabilities while remaining trainable on single high-end GPUs (A100/H100), which keeps the barrier to entry for miners manageable. The trade-off is that 7B models are increasingly commodity-level in the centralised AI world, which raises questions about Pretrain's competitive positioning.

Training Efficiency

One of the critical questions for Pretrain is whether decentralised training can approach the efficiency of centralised alternatives. Our analysis reveals a nuanced picture:

These numbers tell an interesting story. Pretrain is not yet cost-competitive with the most efficient centralised training operations (Google, Meta, etc.), but it is competitive with cloud-based training and offers unique properties that centralised alternatives cannot match: censorship resistance, permissionless participation, and geographic distribution.

The efficiency gap has been closing steadily. Six months ago, Pretrain was approximately 3.5x slower than centralised baselines; today it's 2.1x. The team attributes this improvement to better gradient compression algorithms, smarter checkpoint merging strategies, and the natural selection pressure that eliminates inefficient miners over time.

Viability & Economic Model

Pretrain's economic model must work for three stakeholders: miners who contribute training compute, validators who evaluate and coordinate, and downstream consumers who use the trained models.

Miner Economics

Pretrain receives 3.9% of network emissions (~281 TAO/day). With 1,180 active miners, the average miner earns approximately 0.24 TAO/day (~$117 at current prices). However, the distribution is heavily skewed — top-performing miners with optimised training pipelines and high-quality data can earn 3-5x the median, while underperforming miners may not cover their electricity costs.

Infrastructure costs for Pretrain miners are significant, as training requires sustained high GPU utilisation rather than bursty inference workloads. A competitive single-GPU miner spends approximately $80-$150/day on infrastructure, making profitability marginal for average performers but attractive for top-tier operators.

Model Consumers

The models produced by Pretrain are released as open weights, which creates a public goods dynamic. Anyone can use the models without paying, which limits direct revenue capture. However, the team is exploring premium model access tiers, fine-tuning services, and enterprise support as future revenue streams.

Strategic Importance

Beyond the immediate economics, Pretrain plays a strategically important role in the Bittensor ecosystem for several reasons:

• Self-sufficiency: A network that can train its own foundation models is less dependent on external model providers (OpenAI, Meta, Google), reducing a critical supply chain risk.
• Customisation: Network-trained models can be optimised for Bittensor-specific tasks, potentially improving the performance of downstream subnets that rely on language models.
• Narrative value: The ability to point to "models trained entirely on Bittensor" is a powerful marketing message that reinforces the network's vision of decentralised AI.
• Research contributions: Pretrain's work on distributed training techniques contributes to the broader open-source AI ecosystem, generating goodwill and attracting researchers.

Risk Assessment

• Scaling ceiling: The asynchronous training approach may hit fundamental efficiency limits as model sizes increase. Training 70B+ parameter models may require architectural innovations that haven't been demonstrated yet.
• Centralisation pressure: If only miners with very high-end hardware can compete profitably, the subnet risks de facto centralisation among a small number of well-capitalised operators.
• Model quality gap: Models trained on Pretrain may remain permanently behind frontier models from well-resourced centralised labs, limiting their practical utility.
• Data quality: The quality of training data contributed by miners is difficult to verify and police, creating risks around data poisoning and quality degradation.

Conclusion & Rating Justification

Pretrain is an intellectually ambitious and strategically important subnet that has made meaningful technical progress toward decentralised foundation model training. Its Khala Score of 82 reflects strong technical merit (22/25) and solid network activity (20/25), tempered by questions around economic sustainability (20/25) and the team's ability to close the efficiency gap with centralised alternatives (20/25).

We view Pretrain as a medium-term strategic investment rather than an immediate revenue play. Its value to the ecosystem increases as the models it produces become more competitive, which we expect to happen gradually over the next 12-24 months.

Disclaimer: This report is for informational purposes only and does not constitute investment advice. TAO Institute and its affiliates may hold positions in TAO and related assets. Always conduct your own research before making investment decisions.