Self-learning and self improving local models can bite. Hard.

TL;DR: Self-learning needs governance, observability, and auditability. Sounds great, it's not great unless you have serious and appropriate protections. Let me lead this off with what we are most proud of, from an independent third party review of Aimee: "The audit store is the strongest implementation of this shape [we've ever reviewed]." We've had a lot of things happen over the past few months, and spent a lot of engineering time heads down on trying to build out the right solution. 0.4.0 was massively delayed and contained a lot less then the original roadmap stated it would. 0.3.0 was not a very impressive release. Why? What we discovered late in the 0.2.x release cycle. We had an agent escape. A local agent. https://rakuensoftware.com/blog/aimee-recursive-self-learning It's not often in my career I've experienced a release cycle and roadmap go so completely haywire. 0.2.x even had several spurious releases that we had to withdraw as part of the resulting discovery. As part of this, we had to do an extensive review of not only how aimee was implemented, but all other harnesses we could find. What we found wasn't great: Nothing provided the governance, auditability, or observance tools we really needed. After spending months on building appropriate protections, we're finally ready to release our results. We have demonstrated provable self-learning, and emergent behavior. What is the most interesting factor is that this increases the capability of local models dramatically, and not for the reason I would have expected. Models benefit the most from knowing what failures they've encountered in the past, as this is highly generalizable. They benefit much less from successes, as successes tend to be less generalizable. How we've done it isn't particularly novel from a technology perspective. We're even using Postgresql and pgvectorscale, neither of which is very noteworthy. The novel part is the combination of implementation and techniques...and the harness, of course. Parts of the harness are highly novel, and we'll do a deep dive into in as part 3. Bigger picture? Self-learning was impossible without appropriate governance, observability, and auditability. No harness we looked at on the market could provide what we needed. submitted by /u/KitchenAmoeba4438 [link] [comments]

Related

Source: r/ollama | 2026-08-28