# Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution **URL:** \ **Category:** Show & Tell\ **Tags:** python-sdk, error-handling, activity, ai\ **Created:** [July 28, 2026, 1:15am UTC](https://community.temporal.io/t/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for-durable-execution/19749 "2026-07-28T01:15:12Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![obataka123](https://sea2.discourse-cdn.com/flex016/user_avatar/community.temporal.io/obataka123/32/8454_2.png) [@obataka123](https://community.temporal.io/u/obataka123)\ **Post date:** [July 28, 2026, 1:15am UTC](https://community.temporal.io/t/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for-durable-execution/19749/1 "2026-07-28T01:15:13Z")
A recent postmortem on an in-process vector store (ZVec, C++ backend) surfaced a data-loss bug relevant to anyone building durable AI agent or data infrastructure. Root cause: after a process crash, write-ahead-log replay on reopen fully restores the document count — durability looks intact. But running the storage engine’s optimize() routine against that recovered collection silently drops every record written after the last flush() checkpoint. No exception, no log entry — just missing vectors. The fix shipped as PR #600, merged 54 hours 55 minutes after the report. The broader lesson for distributed systems and AI infrastructure: crash recovery must be verified under an actual kill signal, not assumed from a clean shutdown. Before trusting a storage layer’s durability guarantees, verify: (1) crash mid-write, (2) reopen and check count, (3) run every maintenance/compaction operation, (4) reopen and check count again. > **[Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for...](https://dev.to/obataka/postmortem-silent-data-loss-in-an-in-process-vector-store-and-the-case-for-durable-execution-16i2)** > > TL;DR Bug: Calling optimize() on a ZVec Collection that had recovered from an unclean...