Marketplace Data Pipelines

Every marketplace seller who operates at scale eventually needs a data pipeline. The data starts in Amazon Seller Central, eBay, Shopify or another marketplace, and it needs to end up in a system where it can be joined with cost data, analysed, and acted on. The pipeline is the infrastructure that makes this happen: extraction from each marketplace API, transformation into a common schema, loading into a data store, and orchestration that keeps it running without manual intervention. This article covers the architecture of marketplace data pipelines, the extraction patterns for Amazon SP-API and other platforms, and what it costs to build a pipeline that handles real seller data volumes.

TL;DR

• Marketplace data pipelines follow the classic ETL pattern: extract from marketplace APIs, transform into a common schema, load into a data store.
• Amazon SP-API extraction uses Reports API (batch) and Notifications (real-time). eBay and Shopify use REST polling and webhooks respectively.
• The transform layer is where business value lives: joining sales data with COGS, calculating true profit, and flagging anomalies.
• Build cost starts at $4,000 for a single-marketplace pipeline and $9,500 for multi-marketplace.

Why marketplace sellers need data pipelines

Marketplace platforms give sellers dashboards, but those dashboards show platform data in isolation. Amazon shows sales and sessions. eBay shows orders and feedback. Shopify shows revenue and conversion. None of them show profit, because profit requires data that lives outside the marketplace: cost of goods, shipping costs, return processing costs, and advertising spend. A data pipeline extracts the marketplace data, joins it with these external sources, and loads the result into a system where the business can see what is actually happening. Without the pipeline, someone on the team is doing this manually in a spreadsheet every morning.

Extraction patterns by marketplace

Amazon SP-API has two extraction paths. The Reports API is batch-oriented: you request a report (orders, inventory, settlements, business metrics), wait for Amazon to generate it, download the file, and parse it. The Notifications API is event-driven: you subscribe to notification types (ORDER_CHANGE, ANY_OFFER_CHANGED) and Amazon pushes events to your SQS queue as they happen. A well-designed pipeline uses both: Notifications for real-time data and Reports for daily reconciliation. The Amazon SP-API developer guide covers the API details.

eBay uses REST API polling. The Fulfillment API's getOrders endpoint supports date-range filters, so the pipeline calls it on a schedule and processes new orders. There are no push notifications for order events. Shopify provides webhooks for real-time events (orders/create, inventory_levels/update) supplemented by REST API polling for data that webhooks do not cover. The Shopify API app development guide covers the webhook patterns.

Schema normalisation across marketplaces

The transform layer converts marketplace-specific data into a common schema that downstream systems can query without knowing which marketplace the data came from. An order from Amazon has different field names, different status values, and different fulfillment structures than an order from eBay or Shopify. The pipeline normalises these into a common Order record with standardised fields: order_id, marketplace, order_date, line_items (each with SKU, quantity, unit_price), shipping_address, fulfillment_status, and payment_status.

The normalisation layer also handles marketplace-specific quirks. Amazon orders include FBA and FBM indicators. eBay orders include buyer feedback and dispute status. Shopify orders include draft status and refund details. These marketplace-specific fields go into an 'extensions' object on the normalised record so they are available but do not pollute the common schema. The data pipeline engineering service builds these normalisation layers.

Enrichment and the profit calculation

Raw marketplace data tells you what sold. Enriched data tells you whether it was profitable. The enrichment pipeline joins normalised marketplace records with cost of goods from the ERP (using SKU as the join key), shipping costs from the 3PL or carrier invoice, advertising attribution from the Amazon Advertising API or Meta Ads API, and return and refund data from each marketplace's returns surface. The output is a record that shows revenue, COGS, shipping cost, ad cost, return cost, and net margin per unit per order per day.

This join is where pipelines break. COGS data updates weekly in most ERPs. Shipping cost data arrives after the carrier invoices (days to weeks after shipment). Ad attribution is approximate and has different lookback windows by platform. The pipeline needs to handle late-arriving data by re-computing enriched records when a source updates, and the dashboard needs to show the data confidence level (is this profit number based on actual COGS or estimated COGS?). The custom ecommerce tools team builds these enrichment layers with explicit confidence tracking.

Storage and query patterns

The data store choice depends on query patterns. For dashboards with daily aggregations and standard reporting, PostgreSQL is sufficient and the team already knows it. For sub-minute query performance on large datasets (millions of rows), ClickHouse or TimescaleDB is the right choice. For ad-hoc analysis by a data team, BigQuery or Snowflake provide the SQL interface with the scale. The pipeline should write to the operational store (Postgres or ClickHouse) for dashboards and to the analytical store (BigQuery) for ad-hoc work.

Orchestration and monitoring

The pipeline needs an orchestrator that schedules extraction jobs, manages dependencies between steps, retries failures, and alerts when something breaks. Apache Airflow is the standard for Python-based pipelines. Temporal is the alternative for Go or TypeScript teams. For simpler pipelines, a cron job with error handling and a Slack alert is adequate. The key is that the pipeline runs without manual intervention and someone is notified when it fails. The marketplace connectors infrastructure includes pipeline orchestration as part of every multi-channel data integration.

Monitoring has three layers. Infrastructure monitoring tracks whether the pipeline ran, how long each step took, and whether any step failed. Data quality monitoring checks for anomalies in the output: sudden drops in order count (extraction may have failed silently), negative profit margins (enrichment join may have used stale COGS), or missing data for a marketplace (the extractor may have hit a rate limit). Business monitoring alerts on metrics that matter to the seller: ACOS exceeding a threshold, inventory approaching stockout, or return rate spiking on a specific ASIN. All three layers should exist from day one rather than being added after the first production incident.

Build cost and timeline

A single-marketplace pipeline (Amazon SP-API extraction, basic enrichment with COGS, PostgreSQL storage, Metabase dashboard) starts at $4,000 and ships in 4 to 6 weeks. A multi-marketplace pipeline (Amazon plus eBay plus Shopify, normalisation layer, full enrichment with COGS, shipping and ad spend, ClickHouse storage, custom dashboard) starts at $9,500 and ships in 8 to 12 weeks. An enterprise data platform with real-time Notifications, BigQuery warehouse and a data team self-service layer starts at $21,000. See the Amazon SP-API development page for Amazon-specific pricing and the custom software development page for broader build scope.

Frequently Asked Questions

What is a marketplace data pipeline?

Infrastructure that extracts data from marketplace APIs (Amazon, eBay, Shopify), transforms it into a common schema, enriches it with external data (COGS, shipping, ads), and loads it into a queryable data store.

Why do I need one?

Marketplace dashboards show platform data in isolation. A pipeline joins it with cost data so you can see profit, not just revenue.

What extraction method does Amazon use?

Two paths: Reports API for batch data (daily) and Notifications for real-time events (ORDER_CHANGE, ANY_OFFER_CHANGED via SQS).

How do you handle multiple marketplaces?

A normalisation layer converts marketplace-specific data into a common schema. Each marketplace has its own extractor but downstream systems query a single unified model.

What does it cost?

Single-marketplace from $4,000. Multi-marketplace from $9,500. Enterprise data platform from $21,000.

How long to build?

Single: 4 to 6 weeks. Multi: 8 to 12 weeks. Enterprise: 14 to 20 weeks.

What data store should I use?

PostgreSQL for standard reporting. ClickHouse or TimescaleDB for sub-minute queries on large datasets. BigQuery for ad-hoc analysis by a data team.

How do you handle late-arriving data?

Re-compute enriched records when a source updates. Show data confidence level on dashboards (actual COGS vs estimated).