← Journal

From raw photos to a multichannel inventory: designing a robust ingestion pipeline

An architecture retrospective on a pipeline that identifies objects from photos, prepares their listings and keeps their lifecycle in sync across several marketplaces.

ArchitectureArtificial intelligenceMarketplacesIngestionEvent-driven

From raw photos to a multichannel inventory

Creating a listing from a handful of photos looks simple: recognise the object, generate a title, estimate a price, then call a marketplace API. That picture works for a demo. It falls short as soon as you process a whole crate, the photos arrive out of order, or the same object has to be distributed on several channels.

The real problem is then no longer content generation. It is controlling the lifecycle of a single physical object, from imperfect inputs to its sale, without mix-ups, duplicates or stock collisions.

This article first describes a working pipeline that goes from a folder of photos to draft listings. It then proposes its evolution towards a multichannel architecture in which an internal database remains the source of truth, while adapters keep eBay, Leboncoin or other distributors in sync.

Guiding idea: AI may produce observations and proposals. Identity, stock and irreversible transitions must stay under the control of explicit application rules.

A photo, an object and a listing are three different identities

A crate of books illustrates the problem well. The same book can be represented by its cover, its back, its spine, its title page and several inside pages. Conversely, two similar volumes may share an author, a series, a binding or a near-identical look.

At minimum, then, we must tell apart:

  • the media file, identified by its content and its hash;
  • the physical object, here a unique copy that may have several photos;
  • the canonical record, which gathers title, condition, attributes and price;
  • the remote listing, specific to a provider and its constraints;
  • the sale, which consumes or reserves the stock unit.

Confusing these identities produces the most expensive bugs: photos attached to the wrong book, duplicate listings, a product edited from marketplace-specific data, or a double sale of an object available in a single copy.

The initial pipeline: turning a crate into verifiable books

The first version of the system handles a book correctly when its photos are already grouped. Extending it to a crate therefore adds a preliminary step: finding the sets of photos that belong to each object, without relying on folder order, file names or artificial separators.

1. Inventory untrusted inputs

Every image is first treated as untrusted input:

  • the real format is verified by decoding, not just by extension;
  • limits apply to the number of files, their size and their decompressed pixels;
  • EXIF orientation is corrected locally without touching the original;
  • a hash is computed for identity, resumption and traceability;
  • the EXIF capture timestamp is read when it exists;
  • originals are kept until the result is validated.

File-system dates are not used as a business signal: they change too easily during a copy, an export or a trip through the cloud.

2. Describe before grouping

An inexpensive multimodal model describes the images in batches, following a closed schema. It chooses no path and moves no file. For each photo it extracts, among other things:

  • the type of view: cover, back, spine, title page, interior;
  • the visible bibliographic elements: title, author, publisher, ISBN;
  • distinctive marks: binding, defects, patterns, annotations;
  • whether a book is actually present, and the confidence of the observation.

These structured observations then feed a sparse graph of candidates. Two images can become candidates through an identical ISBN, overlapping text, compatible visual cues or temporal proximity.

The timestamp is deliberately a soft cue. Two photos taken a few minutes apart are likely to show the same object, but two different books may also have been photographed one after the other. Temporal proximity therefore opens a verification; it never triggers a merge on its own.

3. Prefer one split too many over a bad merge

Every candidate pair or group goes through a targeted visual check. A merge is accepted only on positive evidence. A strong conflict (two different ISBNs, for instance) blocks the match. An uncertain decision creates no relation.

This choice optimises precision before recall: a lone photo in the unassigned queue costs one human check; two books mixed into one listing can produce a wrong description, an inconsistent price and a disputed sale.

The result is a complete, traceable partition:

  • groups considered safe;
  • groups flagged for review;
  • unassigned photos;
  • ignored images, because no usable object was detected in them.

4. Keep a human at the irreversible boundary

The interface mirrors this input–output sequence directly: import, group, check, then create. Safe groups may be preselected; ambiguous cases never are. Unassigned and ignored photos stay visible, but they are not silently pushed into the next pipeline.

Once confirmed, each group becomes the input of the unit pipeline:

  1. detailed analysis of the object and its condition;
  2. construction of category attributes;
  3. search for comparables and price estimation;
  4. faithful preparation of the photos;
  5. media upload;
  6. creation of a draft, with no automatic publication.

The “metadata and price” and “photo preparation” branches can run in parallel, since they only depend on each other when the draft is composed.

MarketplaceUnit pipelineMultimodal modelSorterJob managerReview interfaceMarketplaceUnit pipelineMultimodal modelSorterJob managerReview interfaceloop[For each useful match]par[Metadata and price][Media preparation]loop[For each confirmed object]OperatorSelects a folder of photos1Creates a persistent job2Inventories and validates the originals3Hash, EXIF, dimensions, thumbnails4Describes the images in batches5Structured observations6Builds the candidate graph7Checks two images or groups8same object / different / uncertain9Applies conflicts and thresholds10Groups, review, unassigned, ignored11Result and progress12Explicitly confirms the groups13Starts the unit ingestion14Detailed analysis15Structured record16Rotation and faithful enhancement17Uploads the photos18Creates the record and the draft19Remote identifiers20Success or traceable error21Operator

What this first architecture already solves

Four properties matter more than the precise choice of model:

  1. Caution: no weak resemblance is enough to merge two objects.
  2. Resumption: costly observations and decisions are cached by hash, model and prompt version.
  3. Traceability: every media file appears exactly once in the final manifest, with its placement and the related decisions.
  4. Separation of responsibilities: the model observes, the domain core decides, the interface asks for confirmation, then an adapter talks to the provider.

This architecture is sufficient as long as a single marketplace remains the only distribution channel and its drafts can serve as a remote inventory. It hits a structural limit, however, as soon as a unique object is published in several places.

From a publishing pipeline to an inventory system

In a multichannel system, eBay, Leboncoin or any other distributor must not become the source of truth for the object. Each platform has its own categories, states and identifiers. None has a reliable view of the others.

The internal database must hold the canonical state: object identity, stock unit, media, reference price, lifecycle, remote listings and sale. The application reads this database; providers receive projections of it tailored to their contracts.

The target architecture combines three classic patterns:

  • ports and adapters to isolate the peculiarities of each marketplace;
  • transactional outbox/inbox to synchronise the database and remote calls without losing or dangerously replaying an intent;
  • periodic reconciliation to correct the inevitable drift between systems.
Photos and importsIngestion pipelineCanonical databaseOutboxSync workerseBay adapterLeboncoin adapterOther distributorseBayLeboncoinOther channelsEvent inbox and pollingSchedulerReconciliationApplication anddashboardAlerts

A data model centred on the physical object

For objects sold by the unit, the minimal model can stay simple:

EntityResponsibility
media_assetOriginal, hash, EXIF metadata and derivatives
itemCanonical record, independent of marketplaces
inventory_unitPhysical copy and actual availability
listingProjection of the object for a given provider
provider_bindingRemote identifiers, version and last known state
reservationTemporary hold on stock during a transaction
saleAccepted sale, originating provider and financial data
inbox_eventDeduplicated incoming event
outbox_eventSynchronisation intent to execute
sync_attemptAttempt, latency, normalised response and error

Commercial content may vary per channel, but identity and quantity must not be duplicated in each listing. For a unique book, inventory_unit.available_quantity is zero or one: that constraint must be guaranteed transactionally by the database.

Publishing without a fragile double write

A naive implementation performs two successive operations: update the database, then call the marketplace. If the process dies in between, the database and the provider diverge. Reversing the order solves nothing: the listing may be created while the local transaction fails.

The transactional outbox avoids this trap:

  1. a local transaction modifies the object and inserts an intent into outbox_event;
  2. a worker reads this intent and calls the relevant adapter;
  3. the adapter uses an idempotency key or a stable business identifier;
  4. the remote response updates provider_binding and the listing state;
  5. a retry replays the same intent, not a new logical creation.

We are not after a hypothetical distributed “exactly once”. We accept at least once delivery, then make every processing step idempotent.

The adapter exposes a shared vocabulary, for instance:

  • upsert_draft(item);
  • publish(listing);
  • update_price(listing, price);
  • reserve_or_pause(listing);
  • end_listing(listing, reason);
  • fetch_status(binding);
  • fetch_recent_sales(cursor).

Each provider then translates this contract into its API (or into whatever synchronisation mode the platform allows) without leaking its details into the business domain.

Manage a lifecycle rather than booleans

A published = true field is not enough. An explicit state machine makes transitions observable and prevents impossible combinations.

import validatedinsufficient confidencehuman correctioncanonical recordcompleteintents in the outboxat least one active listingprovider errorretry or correctiontemporary commitmentreservation expiredpayment or saleconfirmedremote sale confirmedwithdrawing the otherlistingschannels reconciledincomplete withdrawalIngestedNeedsReviewReadyPublishingAvailableSyncErrorReservedSoldClosingChannelsArchived

Listings have their own state (draft, publishing, active, pausing, ended, error) but they remain projections of the stock unit. An active listing never makes an object available if inventory_unit already says sold.

A sale on one channel must close the others

When a marketplace reports a sale, the incoming processing also goes through an idempotent inbox. The raw event is kept, deduplicated by its provider identifier, then applied in a local transaction.

AlertsMarketplace BWorkerOutboxCanonical databaseInboxWebhook or pollerMarketplace AAlertsMarketplace BWorkerOutboxCanonical databaseInboxWebhook or pollerMarketplace Aalt[Unit still available][Unit already reserved or sold]Sale confirmed1Records the remote event2Stock consumption transaction3Moves available to sold4Creates the sale5Adds END_OTHER_LISTINGS6Event accepted7Dispatches the intent8Ends or deactivates the listing9Remote confirmation10Marks the channel reconciled11Rejects the second consumption12Creates a collision alert13Event kept for resolution14

The available → sold transition must use a lock, a version comparison or a conditional update. Two concurrent events then cannot consume the same unit in the database.

This does not make the physical risk entirely nil: two buyers can confirm almost simultaneously on two external platforms before the withdrawal propagates. The architecture sharply narrows that window, detects the collision and provides a resolution process. The maximum guarantee then depends on the webhooks, reservation mechanisms and delays each provider allows.

Webhooks do not replace reconciliation

A webhook can be lost, delivered late or delivered several times. Some platforms do not offer one for every event. A poller and a reconciliation task therefore remain necessary, even in an event-driven architecture.

The cron should not contain business logic. It triggers idempotent jobs, protected against concurrent runs and observable in the same system as the other processing.

A reasonable starting cadence might be:

Indicative cadenceCheckExpected result
Every 1 to 5 minutesSales and reservations without webhookClose the other channels quickly
Every 10 to 15 minutesStates and quantities of active listingsDetect stock divergences
HourlyStuck jobs, retries and expired reservationsRepair or alert
NightlyExhaustive reconciliationFind orphaned listings and missed sales
Every morningOperational summaryGive the team its review priorities

These frequencies must respect each platform’s limits and terms of use.

An anomaly-oriented dashboard

A good dashboard does not merely display the number of listings. It must quickly answer four questions: what do we own, where is it published, what is drifting, and which human action is required?

The most useful metrics are:

  • objects available, reserved, sold and under review;
  • active listings per provider;
  • remote listings that are orphaned or lack a canonical object;
  • price, quantity or status divergences;
  • latency between a sale and the withdrawal of the other channels;
  • age of the oldest unprocessed inbox/outbox event;
  • failure rate and number of retries per adapter;
  • rate of unassigned photos and groups sent to review;
  • average cost of AI calls per ingested object;
  • sales volume and margin per channel.

Alerts can be ranked by impact:

  • critical: sale collision, sold object still active elsewhere;
  • high: unprocessed sale event, failed remote withdrawal;
  • medium: out-of-sync listing, stuck job, expired reservation;
  • informational: rising review rate or drifting ingestion cost.

Every alert must point to the object, the listings concerned, the last successful transition and the recommended action. An alert without context only shifts the diagnostic work onto the operator.

Rolling out this evolution without rewriting the pipeline

The migration can stay incremental:

  1. Introduce the canonical model. Persist objects and stock units before creating the drafts of the provider already supported.
  2. Encapsulate the first provider. Turn the existing integration into an adapter and systematically store the remote identifiers.
  3. Add outbox, inbox and reconciliation jobs. Make retries idempotent before increasing the number of channels.
  4. Plug in a second distributor. Start with drafts, then publication, and finally sale feedback.
  5. Automate multichannel closing. Measure latency, test collisions and keep a human resolution mode.
  6. Build alerts and dashboard. Feed them from the same events and states, rather than from parallel counters.

The image pipeline does not need to know about eBay or Leboncoin. It produces a canonical object and validated media. Adapters must not know how the photos were grouped. That boundary lets the AI models, the interface and the providers evolve independently.

Conclusion

Going from a few photos to multichannel distribution is not a matter of adding a loop over a list of APIs. It forces decisions about where the truth lives, who owns the stock, and how every transition can be replayed, audited or compensated.

The robust pipeline is therefore built in two stages:

  1. convert uncertain visual inputs into verified canonical objects;
  2. project those objects onto several channels without surrendering control of the lifecycle to them.

AI speeds up identification, description and pricing. The canonical database, transactional transitions, idempotent adapters and reconciliation protect operations. It is this combination (not the model alone) that turns a convincing automation into a real inventory system.