Data Pipeline Engineering. ELT, streaming, lakehouse, and reverse ETL that the business actually trusts.

ELT (Extract Load Transform) is the modern default. Load raw data into the warehouse first, transform there with dbt. Cheaper, more debuggable, easier to evolve. ETL (Extract Transform Load) still has a place for sensitive PII that cannot land in the warehouse raw, for streaming use cases, and for very high-volume sources where in-flight transformation reduces warehouse cost. We use both, picked per source.

dbt has become the default for SQL-based transformation because it adds three things SQL does not have natively: version control, tests, and documentation. Every model in source control. Every model tested for not_null, unique, relationships. Every model documented and lineage-visualised. The transformation layer is no longer a black box. Cost: SaaS dbt Cloud or self-hosted dbt Core. We use both depending on team preference and budget.

Most orgs do not. Batch every 15 to 60 minutes is fine for nearly all analytics. Streaming earns its complexity when you have customer-facing real-time features, real-time fraud detection, real-time pricing, real-time recommendations, or operational dashboards that must reflect the last 30 seconds. We push back on streaming requests when batch is sufficient because streaming infrastructure (Kafka, Kinesis, Pub/Sub) carries 5 to 10x the operational burden of batch.

Reverse ETL pushes warehouse data back into operational tools (Salesforce, HubSpot, Marketo, Slack, Zendesk). Why: the warehouse has the cleanest, most enriched view of the customer, but sales and marketing live in their CRM and marketing automation tools, not in the dashboard. Reverse ETL puts the warehouse intelligence into those tools. Hightouch and Census are the leading platforms. We build reverse ETL in nearly every modern data engagement because it is what makes the warehouse investment ROI-positive for non-technical teams.

PII handling depends on scope. For GDPR and CCPA we tag PII columns in dbt, apply masking policies in Snowflake or BigQuery, and route DSR (data subject request) workflows. For HIPAA we keep PHI out of analytics warehouses or use BAA-covered warehouses (Snowflake, BigQuery, and Databricks all support BAA tiers). For SOC 2 we implement access controls, audit logging, and change management via PR. Per-tenant isolation supported for multi-tenant SaaS warehouses.

Yes. Feature stores via Tecton, Feast, Hopsworks, or custom built on the warehouse. Feature engineering in dbt or Spark depending on scale. Online feature serving via Redis or DynamoDB. Offline feature retrieval for training via warehouse query. Feature monitoring and drift detection. Tightly coupled with the data warehouse so the offline and online features match.

Data observability is monitoring the health of your data the same way you monitor application health. Freshness (when did this table last update). Volume (is row count today within expected range). Schema (did a column change). Lineage (what depends on what). Tools: Monte Carlo, Bigeye, Datafold, or open source via Elementary and Re-data. We set this up so data issues are caught before stakeholders email asking why the dashboard is wrong.