A recent postmortem on an in-process vector store (ZVec, C++ backend) surfaced a data-loss bug relevant to anyone building durable AI agent or data infrastructure.
Root cause: after a process crash, write-ahead-log replay on reopen fully restores the document count — durability looks intact. But running the storage engine’s optimize() routine against that recovered collection silently drops every record written after the last flush() checkpoint. No exception, no log entry — just missing vectors.
The fix shipped as PR #600, merged 54 hours 55 minutes after the report.
The broader lesson for distributed systems and AI infrastructure: crash recovery must be verified under an actual kill signal, not assumed from a clean shutdown. Before trusting a storage layer’s durability guarantees, verify: (1) crash mid-write, (2) reopen and check count, (3) run every maintenance/compaction operation, (4) reopen and check count again.