# Cosmos / H3 training pilot — 2026-09-09

## Decision
Accept a bounded, slow-network pipeline-parallel LoRA research track on the existing Murakumo fleet. The owner explicitly requested recording and publishing the completed pilot. This extends the prior 32 GiB single-node research policy for this experiment only; it does not certify pooled fleet RAM as single-device memory or production model availability. Per-node memory, startup loading and activation size remain independent constraints. External MLX/PyTorch harnesses are research oracles, not a Kotoba-native runtime implementation.

## Observed on 2026-09-09
Cosmos-Reason2-2B locally converted 4-bit affine/group64, revision 9ce19a195e423419c349abfc86fd07178b230561: 28 language layers split 14/14 between two M4 16 GiB Macs. Frozen visual features from one 224x224 synthetic image; 145 tokens, 15 supervised tokens; q/v rank-4 LoRA, scale 2, Adam 1e-4, dropout 0. Three updates over Tailscale TCP produced losses 3.7668786049, 3.3222055435, 2.6303226948. Single-node same-conversion reference losses were 3.7668757439, 3.3221907616, 2.6303086281. Initial gradient relative L2 differences were 3.9158127e-5 and 2.3070275e-5. Final adapter max absolute differences against that reference were 0.0004998241 and 0.0004064792; not bitwise parity.

A clean restart after two updates and an injected transport disconnect after the second forward were exercised. Recovery from the last common saved step produced adapters AND Adam tensors with zero maximum absolute difference on both stages against the no-fault distributed trajectory. The unfinished second checkpoint was retained separately. This covers the injected fault, not arbitrary crash atomicity.

For this 145-token input, steady update time was about 0.71-0.75 seconds, activation plus returned gradient about 2.38 MB/update. MLX allocator peak about 1.96 GB per distributed rank, single-node reference about 2.37 GB; process RSS reached about 2.32 GB. Decimal GB, not a whole-system peak. Both ranks still load the complete 2B checkpoint at startup. These numbers do not qualify 27B/33B startup, long-context or video workloads.

MiniMax H3 revision 42ed227ee7df40d41602854ae760620d6eb651fe: block 0 from FL2VA/transformer/model-00001-of-00013.safetensors was retrieved by HTTP Range (1,291,143,680 data bytes). On Radeon 8060S / ROCm PyTorch 2.9.1, diffusers 0.40.0, the 645,571,840-parameter BF16 block plus 100,352 trainable rank-4 q/v LoRA parameters completed two forward/backward/update probes on 32 synthetic hidden-state tokens. GPU allocated peak 1,456,381,440 bytes. Input and adapter gradients were finite and nonzero. Synthetic loss increased from 1,662,965.375 to 1,664,710.875; no quality or learning improvement is claimed. Strict converted tensor keys are checked, but upstream numerical parity is not qualified. H3 is open-weight MiniMax H3, not Hailuo 2.3. This was not distributed H3 or a full video pipeline.

Cosmos-Predict2.5-2B base post-trained weight access with the existing authenticated account returned GatedRepoError HTTP 403. Access remains a separate prerequisite; Reason2 results do not validate Predict2.5 or Cosmos3 Edge.

## Consequences and next qualification gates
1. Load only assigned layers from startup; verify model, tokenizer, config, batch and optimizer identity.
2. Coordinate the common checkpoint before resume. Current mutual ACK is not a globally atomic commit; fsync/power-loss, other fault points and automatic reconciliation remain unqualified.
3. Evaluate multiple training images and a fixed untouched holdout; measure answer accuracy and regressions, not only training loss.
4. Establish H3 upstream block parity, then two-block stage-gradient parity and actual AMD/Mac operator and quantized backward support before distributing the full 50-block transformer. Encoders, VAEs, AdaLN and input/output heads require separately measured stages.
5. Use real cached video/text latents before claiming video tuning. No always-on trainer, deployment of a trained model, coding-quality, video-quality, robotics result or end-to-end 16/32 GiB H3 profile is established.

## Evidence and publication
Public article: https://murakumo.cloud/blog/2026-cosmos-h3-distributed-training-pilot
Public evidence: https://murakumo.cloud/research/cosmos-h3-2026-09-09/README.md
 SHA256SUMS identifies the public evidence bytes; it is not an IPFS CID. Raw experiment records remain in the operator's Cosmos/H3 training pilot workspace and the isolated per-node pilot directories. Base model weights are not republished. Publication is verified separately from accepting this ADR.