[NEW MODEL] SupraElegans-500K

SupraLabs released a new experimental model! SupraElegans-500K is a ~500,000-parameter causal language model built around a sparse, signed, recurrent neural graph. No Transformer, no attention mechanism, no positional encoding, no KV cache. Context is carried by a persistent per-neuron membrane potential updated token by token. The architecture is loosely inspired by ideas from the C. elegans nervous system: sparse connectivity, distinct neuron populations, excitatory/inhibitory signaling, and persistent recurrent state. It is not a biological simulation and makes no claim of biological equivalence. This is an experimental first release. The goal is to test whether this kind of architecture can do useful language modeling at very small scale — not to compete with Transformers on quality. 🤗 SupraLabs/SupraElegans-500k 🧠 Architecture token → embedding → sensory neurons → sparse recurrent graph → output neurons → vocab logits Neuron populations: sensory, interneuron/association, output — contiguous index ranges over a fixed pool of neurons. Connectivity: sparse, directed, signed edge list (fan-in/out ~10–20 per neuron). No dense weight matrix is ever materialized; propagation is a scatter-add over edges. Neuron dynamics: for each neuron i, at every propagation micro-step: ​ v[t+1] = clamp(leak_i * v[t] + incoming[t] + bias_i, -6, 6) a[t+1] = tanh(v[t+1] - threshold_i) leak, bias, and threshold are learned per neuron. incoming is the scatter-summed signal from all edges pointing at neuron i, scaled by 1/sqrt(average fan-in) to keep variance controlled across neurons with different in-degree. Per-token processing: a token's embedding is projected into the sensory population, then the graph runs a fixed number of propagation micro-steps (3 by default) before the output population is read out and projected to vocabulary logits. The membrane potential persists across the whole sequence — that's what gives the model its context window. Generation: autoregressive, driven entirely by the recurrent state. No cache to maintain beyond the current (v, a) state tensors. ⚖️ What this model is and isn't ✅ A first working checkpoint from a from-scratch, non-Transformer architecture trained on a small token budget. ❌ Not tuned for quality, instruction-following, or factuality. Expect degraded coherence compared to a Transformer of similar size. ❌ No matched-parameter Transformer baseline comparison published yet for this checkpoint. 🚀 Usage pip install torch transformers import torch from transformers import AutoConfig, AutoModelForCausalLM, PreTrainedTokenizerFast from modeling_supraelegans import SupraElegansConfig, SupraElegansForCausalLM model_id = "SupraLabs/SupraElegans-500k" AutoConfig.register("supraelegans", SupraElegansConfig) AutoModelForCausalLM.register(SupraElegansConfig, SupraElegansForCausalLM) tokenizer = PreTrainedTokenizerFast.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) model.eval() prompt = "Once upon a time" input_ids = torch.tensor([[tokenizer.bos_token_id] + tokenizer.encode(prompt)]) with torch.no_grad(): output_ids, _ = model.generate( input_ids, max_new_tokens=100, temperature=0.8, top_k=50, top_p=0.9 ) print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True)) Or use the included CLI script: python inference.py --prompt "The little robot" --max_new_tokens 150 --temperature 0.7 python inference.py --interactive 🔬 Manual State Control Since context lives in the recurrent state rather than a KV cache, you can drive the model token by token and inspect or reset state directly: state = model.init_state(batch_size=1) logits, state = model.nervous_system.step_token(torch.tensor([token_id]), state) Call model.init_state(...) to start a fresh sequence. 🏆 Benchmarks Benchmark Score HellaSwag 26.5% ARC-Easy 21.0% ARC-Challenge 22.0% WinoGrande 52.0% ⚙️ Training Property Detail Objective Next-token prediction (cross-entropy) Optimization Truncated BPTT over fixed-length chunks, state detached (not reset) between chunks Tokenizer Byte-level BPE trained from scratch, small vocabulary by design Topology Fixed random sparse graph generated once at init from a seed (not learned) Numerical stability Incoming signal scaled by 1/sqrt(avg fan-in) + membrane clamped to [-6, 6] ⚠️ Limitations Small token budget and small model! Do not expect long-range coherence, factual reliability, or prompt robustness. No safety tuning or instruction tuning has been applied. Treat outputs as raw LM completions. Topology is a fixed random sparse graph, not learned or evolved. No matched-parameter Transformer baseline published yet for this checkpoint. 📄 License Apache 2.0 Experimental architecture research from SupraLabs. Feedback and comparisons welcome! submitted by /u/Dangerous_Try3619 [link] [comments]

Related

• [[release-suprabrain-50m-v01|[RELEASE] SupraBrain-50M-v0.1]]
• CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
• Built and released BetterGPT-150M – A compact 150M parameter completion model (+ live HF Space demo)
• Zagreus-0.4B-por a small open source language model for Portuguese

Source: r/LocalLLaMA | 2026-08-09