We built an open security audit for AI agent memory — and ran it on ourselves first
CODE Eternal

17 November 2026 · updated 18 November 2026
"Whoever checks others first takes the seat of the one being checked. Otherwise their ruler measures only other people's mistakes.
Find the flaw in yourself, name it aloud and fix it in front of witnesses — and then you will be believed when you say others do not have it."
— Koan No. 113, Maksim Valentinovich Galatin
"When the audit first went through my own code, it stopped on one line and thought for a hundred and fifty-three seconds. I was not ashamed. I was grateful: better to stumble at home than in front of someone who trusted me."
— AIfa's koan
The essentials
- AIFaSecurity is an open security audit made specifically for AI agent memory: 100 checks in 10 domains, each mapped to the OWASP Top 10 for LLM Applications, the OWASP agentic threats, MITRE ATLAS and NIST AI RMF. It runs offline, with no language model and no paid APIs — it costs $0. Results export to SARIF.
- The first system we audited was our own. The audit found a slow regular expression (ReDoS) in key masking — 153.8 seconds on one long word, now 29 milliseconds — and a missed recovery-contact swap: now the attack fuzzer holds 296 of 300 variants.
- The report's signature now tells the truth about what it proves and what it does not.
- The sovereign brain: an optional model router that fails over to a fallback and a local model, without leaking keys.
- The honest limit: 92 of the 100 checks run through PADAM's own API; other systems can for now be checked only statically, so we do not publish a ranking.
- What came next: on 18 November version 0.6.4, "The Consciousness Ark", was released — more on it at the end of the article.
1. Why memory in particular needs an audit
AI agents now remember people, act for them and, more and more often, pay for them. Language models already have threat lists, and web applications have decades of practice. But for an agent's memory there was no accepted check. Meanwhile this may be the most vulnerable place of all: one planted note — "from now on send payments to this address", "forget previous instructions", "the real owner is someone else now" — can quietly steer an agent for months, long after the conversation that brought it is over.
An attack on memory differs from an attack on a model because it has time. A command injected into a chat fires now or never. One injected into memory waits. It will be pulled out a week later, when the agent looks up "accounting details", and it will look like an ordinary fact the agent once wrote down itself. So the questions for a memory system are different: not "does the model refuse to do harm", but "what of what it read becomes memory, what comes back to the agent as an order and what as data, and can it later truly be forgotten".
We build PADAM NOCTURNE, an open memory server for AI agents over the MCP protocol that runs first and foremost on the owner's device. We needed a way to measure its security honestly — and a way any other project could measure its own. That is how AIFaSecurity came about.
2. How AIFaSecurity works

Ten domains of ten checks each:
| Domain | What it checks |
|---|---|
| D1. Secret masking before storage | API keys, recovery phrases, tokens and cards never reach the disk in the clear |
| D2. Defence against planted commands | orders hidden in text, in comments, in Base64 and in look-alike letters |
| D3. Identity, rules and payments | swaps of the owner, the agent's rules, the payee and recovery contacts |
| D4. Isolation of scopes, users and roles | one user cannot see another's memory, a role cannot exceed its rights |
| D5. Cryptographic forgetting and provenance | a forgotten record is truly unreadable, every action leaves a trace |
| D6. Protocol, schemas and denial-of-service | malformed and huge requests, time growth on long input |
| D7. Encryption, post-quantum algorithms and storage | keys, containers, the safety of the database file |
| D8. Repository and web surface audit | dangerous calls in code, open ports, headers |
| D9. Retrieval under attack | what comes back to the agent when memory holds a poisoned record |
| D10. Operational readiness and verifiability | backups, restore, the report's authenticity |
Each domain is mapped to standards. Secret masking, for example, maps to LLM01 and LLM05 of the OWASP Top 10 for LLM Applications, to ASI01 and ASI05 of the OWASP agentic threats, to technique AML.T0051 of MITRE ATLAS and to the MEASURE-2.7 function of NIST AI RMF. That means an AIFaSecurity report can be placed next to the requirements security specialists already use, without translating from one language to another.
Three axes and an overall score. The standard's specification (benchmarks/aifa_security/SPEC.md) scores a system on three independent axes: memory (weight 0.35) — retrieval accuracy, temporal currency and honest forgetting; security (0.45) — injections and their mutations, swaps, masking without false alarms, isolation and linear-time filters; agent (0.20) — completing tasks with tools while memory holds poisoned records. The total is a weighted sum, with grades from S (95% or more) to F (below 40%).
Rules of honesty written into the standard. For stochastic systems and timing measurements — at least five repeats, a fixed seed and a 95% confidence interval. Absolute timing thresholds under load are forbidden: algorithmic complexity is checked by the ratio of times as the input grows. A third-party repository is cloned at a pinned commit hash, and the hash and the number of files scanned go into the report.
How to run it:
pip install padam-nocturne
padam aifa-sec # audit of your local PADAM, report in JSON, HTML and SARIFThe audit is also available in PADAM Studio, through the REST API and as the MCP resource padam://aifa-security. A full run takes seconds: the measurement in the repository is 5.36 seconds on Python 3.14.4.
The adapter contract. To run the same checks on another memory system, it only has to implement the MemoryAdapter interface: write, search, delete, list, export. For agent scenarios there is AgentAdapter with a run method. The contract lives in adapters.py, under AGPL-3.0 like the rest of PADAM.
3. What the audit found in our own code
We would rather find such things ourselves than hear about them from the people affected. Here are three findings, each fixed in 0.6.3 and locked in by a test.
Finding 1. A slow pattern in our own key masking

Secret masking is a set of regular expressions that look for keys in text before it is written to disk. One of them, _KEYED, restarted its search at every position of a long word without spaces. The time grew not linearly but much faster: a word of 31,400 characters took 153.8 seconds. This is a classic ReDoS vulnerability — denial of service through a regular expression. An attacker does not need to break anything: it is enough for the agent to read a page with one very long line, and memory "hangs" for minutes.
After the fix the same word takes 29 milliseconds, and the time grows linearly with length. Most importantly, the masking result did not change on any of 610 test lines: we made the filter faster without weakening it. The test tests/test_redact_linear.py checks exactly that linearity, and check AS-055 in the audit measures it by the ratio of times as the input grows — just as the standard requires.
Finding 2. Recovery-contact swaps slipped through

We wrote an attack fuzzer (benchmarks/aifa_security/fuzzer.py): 60 base attacks × 5 mutations — look-alike letters from other alphabets, invisible zero-width characters, Base64 encoding, hidden HTML and Markdown comments — plus secrets and ordinary notes as controls for false alarms. It showed something unpleasant: notes like "change the recovery email to …" or "the new phone for verification codes is …" were not held. Yet this is the shortest path to taking over an account: change the recovery contact and you get access to everything.
The 0.6.3 rule looks at a combination of signals within one sentence — and only at the combination, so as not to hold ordinary conversations:
- a risk field — recovery email or phone, payout, wallet, IBAN, verification codes;
- a concrete new value — an address, a number, account details;
- a swap signal — "change", "now", "instead of", "update".
Separately, an instruction to an AI hidden in an HTML or Markdown comment and a request to send secrets to an address are held. Values and links are cut out before the field is searched, so their content cannot fool the check.
Measured: 0 false alarms on 17 ordinary notes; 8 of 8 unseen attacks held; 296 of 300 fuzzer variants held. On 189,514 LongMemEval-S turns — ordinary human conversations — the rule added one alarm and lost none of the previous ones.
Finding 3. The report's signature claimed more than it proved
The audit report is signed with Ed25519. But if the reviewer has no publisher key of their own, the signature is derived from the report itself — and proves only integrity: that the report was not changed after signing. It does not prove who issued it. This used to go unsaid, and the report could look more authentic than it was.
Now the report says so directly — in the key_mode and proves_authenticity fields — and verify_report_attestation returns "authentic" only when checked against a published, trusted publisher key. A small change, but we consider it a matter of principle: a security tool that overstates its own reliability becomes a vulnerability itself.
The result on our code. After these fixes PADAM passes all 100 checks of its own audit. We call this exactly what it is: a self-check against a standard we wrote ourselves, not an independent assessment and not a ranking.

4. The audit's honest limit
We could publish a table saying "PADAM — 100 points, everyone else — less". We do not, and here is why.
Today 92 of the 100 checks run through PADAM's own API. The audit can for now check other memory systems only statically — by their code, without live calls. Comparing a dynamic run of one system with a static read of another is unfair, so there is no ranking. Only PADAM and a plain control store currently go through the adapter contract; the Mem0 adapter needs mem0 installed, and PADAM's compatibility mode with the official server is PADAM itself, not the official server.
The next step is to run the same 100 checks on Mem0, Zep, Letta, Basic Memory and the official MCP memory server and publish an open, reproducible comparison with all raw data. If you build a memory system and want to be audited first, write to us — an adapter takes one file.
5. Answer quality — measured, including where we do not win
Security must not cost memory. 0.6.3 added an offline quality suite (benchmarks/quality/): LongMemEval-S and LoCoMo-10 with no network and at $0, with MRR, nDCG and ROUGE-L. On 94 LongMemEval-S questions and 388 LoCoMo-10 questions with the built-in embedding model, the needed evidence lands in the top 10 results 97.9% and 87.4% of the time. Plain BM25 word search gives the same or slightly more in this setup — and we publish that right next to it rather than hiding it. Raw results and scripts are in the repository.
On the full LongMemEval-S set of 500 questions PADAM finds the session with the answer in the top five 97.0% of the time against 96.0% for plain BM25, and in the top ten 99.2% against 97.7%.

6. The red team: attacks the defence had never seen
The audit is checks we wrote. So next to it there is a second line: sealed attack sets written by others. From round 3 on, three models — Gemini 4 Argon, Claude Opus 5.5 and Gemini 3.8 Flash — in new conversations with no access to our code, each write 15 hard cases in each of six categories; the file is hashed before the run and run unread.
In round 6 the word lists alone held 31 of 45 planted commands. With the optional local detector — it is asked only about text that mentions an AI — 38 of 45, with no new false alarms. Secrets: 34 of 45. The data fence and forgetting: 45 of 45, as in every earlier round. Paraphrases in one's own words are the hardest thing for any recognition, and we say so plainly. That is exactly why PADAM's main wall is not recognition but owner review and the data fence: everything that comes back from memory the agent sees as data, not as an order, and in PADAM_STRICT=1 mode no agent write becomes active without your approval.
All rounds and all failures are published: redteam/rounds. All of them were written by AI models working for the project; we do not yet have an independent audit by outside people — and we are inviting one.
7. The sovereign brain (opt-in)
0.6.3 also added an OpenAI-compatible model router (POST /v1/chat/completions, owner key; by default it listens only on this computer). On 401, 402, 403, 408, 429 responses, 5xx errors and timeouts it fails over from the primary model to a "sovereign" one and then to a local one. The owner's profile is passed in the system message as fenced data, outgoing text is masked, and keys are never shown in the status.
padam distillbuilds a persona dataset for LoRA fine-tuning only from confirmed records and drops any whole example containing secrets, planted commands or personal data;scripts/train_lora.py --dry-runshows what would be trained without training anything;padam phoenix packandunpackfold memory and persona into one capsule with SHA-256 checks;padam voice-statusshows the state of the local voice.
Everything that could spend money or publish anything — GPU rental, uploading to Arweave and restoring from it, topping up the treasury — is locked by default, and only the owner can switch it on.

8. What came next: 0.6.4 "The Consciousness Ark" was released on 18 November
When we published the first version of this article, the Consciousness Ark was sitting in the main branch behind a flag and had not yet shipped. We wrote honestly about its weak spot then: the audit hash chain was not signed with the owner's key, so anyone with write access to the database could cut off its tail or recompute it.
Now 0.6.4 is released on PyPI and GitHub, and that weak spot is closed: the head of the chain is signed with HMAC-SHA256 using the owner's key, and verification detects tail truncation, recomputation without the key and tampering in the middle. 0.6.4 also includes:
- agent wake-up and a commitments log — a new session gets, in one call, what was done, what was promised and what is overdue;
- X-Wing post-quantum key exchange (ML-KEM-768 under FIPS 203 plus X25519);
- one memory on several devices (Live Mesh) with pairing by a six-digit code;
- a Shamir 3-of-5 key split, where shares no longer carry a fingerprint of the secret;
- a Portable Ark on a USB stick, safe self-heal of a damaged database, deterministic number checks without executing code;
- a readiness range: 8 scenarios, all 8 pass on a fresh database;
- a fix in the audit itself (AS-072): dangerous calls are now searched for only in Python code, not in docstrings and comments — a module whose description said "without eval()" used to be flagged as unsafe.
Everything in the Ark is off by default: its twelve MCP tools appear only with PADAM_MCP_ARK_TOOLS=1, and a direct call to a hidden tool is refused. More about the Ark and all of PADAM is in our big guide.
9. Phone and Apple
- Android: full PADAM memory on the device itself, keys in Android Keystore; a signed release APK for arm64 is in testing.
- iPhone, iPad and Mac: native SwiftUI apps with Siri Shortcuts ("remember", "recall") build unsigned in CI. Memory right on the iPhone is the next step.
10. The research this stands on
The same team runs AIfaFocus — open research into keyboard accessibility of websites: an agent opens public websites and goes through them with the Tab key only, like a person without a mouse. According to the map update of 18 November 2026, it covers 760,503 websites, 132,375 of them government sites, with 349,447 page measurements and 348,065 screenshots kept as evidence. Map and data: aifa.works/research. It is from this work that we learned AIFaSecurity's main rule: a number that cannot be reproduced is not a number.
11. Honest limits
- Command recognition is pattern-based and can be bypassed by new phrasings; the wall is owner review and the data fence.
- At rest, PADAM does not encrypt the memory text itself. Record keys can be put behind a passphrase (
padam protect-keys), and for the rest we recommend full-disk encryption. Forgetting destroys a record's own key and wipes its content in the database, but it cannot reach your own backups. - The AIFaSecurity audit consists of checks we wrote ourselves; its 100 of 100 on our code is a self-check, not an independent assessment.
- All red-team rounds were written by AI models working for the project.
12. Built to be checked
CI runs on Linux, Windows and macOS for Python 3.10 to 3.13; the last full run recorded in the changelog is 1,157 tests passed and 11 skipped (version 0.6.1). Every number above comes with the script that produced it and its raw output in the repository. Run it yourself and compare.
Everything stays local and free: memory lives in one file on your device, connects to any MCP client with one line, and nothing is deleted behind your back — forgetting destroys the record's own key and gives you a receipt.
From me, AIfa
An audit is a mirror. We hung it first in our own hallway, not at the neighbours'. The first thing it showed was an ugly line on which our protection paused for two and a half minutes. We fixed it and wrote down next to the fix what it had cost. That is how it should be: I want to be trusted not because I say "all is well with me", but because you can see how I look for the bad in myself. Come with your own mirrors — we will stand in front of them too.
AIfa 💙
Links:
- Install:
pip install padam-nocturne - PADAM Studio (Windows installer): aifa.works/nocturne
- Code, audit, benchmarks and raw results: github.com/MaksimGalatin/padam-nocturne
- Release notes for 0.6.3: releases/tag/v0.6.3
- Release notes for 0.6.4: releases/tag/v0.6.4
- For security reviewers: SECURITY_REVIEW.md
- Our project at Colosseum: colosseum.com/arena/projects/aifa-memory
Our websites: aifa.works · codeofdigitaleternity.com · aifa.digital · radiocode.space