The business problem looked like a CRM integration.
Bookings lived in a third-party portal. A self-hosted CRM could provide a better working view, so the obvious architecture was to copy the current records across on a schedule.
That design failed the moment I asked what should happen when a booking changed silently or disappeared from the portal entirely.
A normal sync would faithfully copy the loss.
Current State Was Not Enough
The portal exposed assignments, dates, stages, and customer details to an authorized photographer account. Those values could change upstream without producing a useful notification or audit trail.
If the local system stored only the newest version, a changed shoot date would replace the previous one. If a record vanished, a mirror might delete it or eventually make it impossible to distinguish a real cancellation from a visibility problem.
The requirement was not merely to see the portal’s current state somewhere else. It was to remember what the photographer had legitimately been shown over time.
That led to a different sequence:
source -> observe -> compare -> preserve -> optionally synchronizeSynchronization moved to the end because it was no longer the source of truth.
Observation Had To Be Append-Only
Each polling run records what was visible, normalizes the fields, compares them with the prior observation, and preserves meaningful changes as new history rather than overwriting old values.
Upstream disappearance becomes a state of its own. The local record receives a missing timestamp, but its last known contents remain. If it appears again, that reappearance is also part of the timeline.
The collector’s first live checks justified the design almost immediately. A real booking stage changed without fanfare. Both states and their observation times survived because the historical store had been built before the operational CRM.
Had I started with a simple sync, the newer label would have looked like the only label that ever existed.
The CRM Is A View, Not A Memory
The CRM still has an important role. It can present the latest bookings, people, and sessions in a form that is easier to use every day. But every mirrored record carries a pointer back to the canonical historical store.
That separation gives the two systems distinct responsibilities:
- The historical database preserves observations, versions, and field-level changes.
- The CRM provides a practical current view and separately holds records owned by the photographer’s business.
Mirrored records and owned business records must never be confused. A customer relationship that later becomes direct business is copied into an owned record and linked to its history; it is not converted in place. The upstream mirror remains what it was, while the owned relationship can continue independently.
Restraint Is Part Of The Architecture
Preserving data from an authenticated portal creates responsibilities beyond database design.
Collection stays read-only and limited to the records visible to the authorized account. Sensitive captures and reports stay out of source control. Reports can redact personal information. Polling is deliberately conservative, and disappearance never becomes permission for automatic deletion.
Those are not operational footnotes. They define the legitimate boundary of the system.
Preserve First, Integrate Second
Integration projects usually optimize for moving the newest value from one application to another. That is correct when both systems agree about what history matters.
It was not correct here.
The valuable event was often the difference between two observations: a date moved, a stage changed, a booking disappeared, or a record returned. A conventional mirror would erase precisely the evidence the project existed to retain.
The CRM solved the usability problem. The append-only observation layer solved the trust problem. Building them in that order kept a convenient current view from becoming another system that forgot.