Compose a data contract bridging front-end tracking, backend ETL, and feature pipeline…
- User
Compose a data contract bridging front-end tracking, backend ETL, and feature pipeline expectations.
Assistant
1. Understand the Goal: The user wants a "data contract." Specifically, one that bridges three distinct areas: * Front-end Tracking: The source of the raw events (clicks, views, etc.). * Backend ETL: The process that cleans, validates, and moves this data. * Feature Pipeline: The process that aggregates raw data into features for ML models or analytics.
-
Identify the Core Problem: Usually, these teams work in silos.
- Front-end devs change a button ID, breaking the ETL regex.
- Data Engineers change a schema in the warehouse, breaking the Feature Store.
- Data Scientists expect a "total_purchase_amount" feature, but Front-end only sends "item_price" and "quantity" without the logic to sum them up correctly or handle currency.
-
Determine the Structure of a Data Contract: A good data contract should be:
- Machine-readable: (e.g., JSON, YAML, Protobuf, Avro) so CI/CD can enforce it.
- Human-readable: Documentation is key.
- Versioned: Changes must be managed.
- Semantic: It shouldn't just say "string," it should say "ISO 8601 Timestamp."
-
Select a Domain/Scenario: To make the example concrete, I'll choose a common e-commerce scenario: "Add to Cart" event. This touches UI (tracking), Backend (ETL/Validation), and Feature Engineering (User propensity to buy).
-
Drafting the Sections:
- Header: Metadata (version, owner, scope).
- Schema (The "What"):
- Raw Event (Front-end -> Kafka/Queue).
- Transformed Table (ETL -> Warehouse).
- Feature View (Feature Pipeline -> Model).
- Quality Constraints (The "How Good"): SLAs, nullability, freshness.
- Governance (The "Who/When"): Breaking change policies.
-
Fleshing out the Content (Iterative refinement):
- Front-end constraints: Need specific fields like
event_name,timestamp,user_id,device_context. Need to specify data types strictly (e.g., no "null" strings, actual nulls). - ETL responsibilities: Sanitization, PII masking (crucial!), deduplication.
- Feature expectations: The feature pipeline usually needs windowing (e.g., "number of items added in last 24 hours").
- Front-end constraints: Need specific fields like
-
Choosing the Format: YAML is generally the most readable for this kind of configuration. I will use a structure similar to
dbtmodels or adatacontract-specification. -
Structuring the Output:
- Introduction: Briefly explain the philosophy (Shift Left).
- The Contract (Code Block): The actual YAML/JSON definition.
- Breakdown by Stage:
- Stage 1: Ingestion (Front-end contract).
- Stage 2: Transformation (ETL contract).
- Stage 3: Consumption (Feature contract).
- Enforcement: How to make this real (Great Expectations, schema registries).
-
Refining the content details (Self-Correction during drafting):
- Initial thought: Just list field names.
- Correction: That's just a schema. A contract includes SLAs and quality guarantees. I need to add a section on data quality tests (e.g.,
user_idmust not be null). - Feature Pipeline specific: Data scientists care about the logic. I should define the transformation logic for the feature (e.g.,
num_add_to_cart_last_7d).
-
Final Polish: Ensure the tone is professional and technical. Use standard terminology (SLA, SLO, PII, Windowing).
(Self-Correction Example during the process): I initially forgot to mention PII. Since this is tracking data, PII is a huge risk. I must add a section on data classification and PII handling in the ETL section.
-
Final Output Generation: (This matches the provided good response).
- Title: E-Commerce User Interaction Contract.
- Format: YAML (Data Contract Specification style).
- Sections: Info, Servers, Terms, Models (Raw vs. Processed vs. Feature).
-
Review against User Prompt:
- Bridging Front-end? Yes (Schema enforcement).
- Bridging ETL? Yes (Transformation logic).
- Bridging Feature Pipeline? Yes (Feature definitions & freshness SLA).
-
Output: Produce the response. This response provides a complete Data Contract implemented as a YAML specification (compatible with tools like Data Contract CLI or Great Expectations).
This contract governs the
checkout_initiatedevent. It ensures that the Front-End sends clean data, the ETL team knows how to parse and mask it, and the Feature Pipeline knows how to aggregate it for ML models.
Data Contract:
checkout_initiatedEventContract ID:
dc-ecom-checkout-v1.2Owner: Data Platform Team Status: Active Effective Date: 2023-10-271. Service Level Agreements (SLAs)
These are the non-functional requirements binding the teams.
- Freshness: Data must be available in the Feature Store within 15 minutes of event ingestion (p99).
- Availability: Tracking endpoint must maintain 99.9% uptime.
- Retention: Raw events retained for 365 days; Aggregated features retained for 2 years.
- Support: Data Engineering (ETL) owns the raw layer; Data Science owns the feature derivations.
2. The Schema (YAML Specification)
This section defines the data structure. The
physical_namecorresponds to the JSON key sent by the Front-End, while the logical name is used in the Data Warehouse.data_contract: schema: - name: event_id physical_name: evt_id data_type: string (uuid) description: Unique identifier for the specific event instance. constraints: - required: true - unique: true classification: operational - name: user_id physical_name: uid data_type: string (hash) description: PII-safe hashed user identifier. constraints: - required: true classification: sensitive - name: timestamp physical_name: ts data_type: timestamp (iso8601) description: UTC timestamp when the event occurred on the client device. constraints: - required: true etl_logic: CAST to TIMESTAMP_TZ; Filter events > 1 year old. - name: cart_value physical_name: val data_type: decimal(10,2) description: Total value of the cart at time of checkout initiation. constraints: - required: true - minimum: 0.00 - maximum: 1000000.00 feature_logic: Used for 'high_value_intent' feature flag. - name: currency physical_name: curr data_type: string (iso3) description: Currency code. constraints: - required: true - enum: [USD, EUR, GBP, CAD] - name: items physical_name: items data_type: array<object> description: List of product objects in cart. constraints: - min_items: 1 schema: - name: sku_id data_type: string - name: quantity data_type: integer - name: unit_price data_type: decimal(10,2) - name: device_context physical_name: ctx data_type: struct description: Metadata about the user's environment. schema: - name: os data_type: string - name: is_mobile data_type: boolean
3. Team Responsibilities & Logic
A. Front-End Tracking (The Producer)
Responsibility: Generate valid JSON payloads conforming to the schema before sending to the ingestion API.
- Validation Rules:
cart_valuemust be calculated client-side to match the visual UI total (preventing discrepancies).timestampmust be generated at the moment of the click, not on page load.- Enum Checks: Front-end must block the event if
currencyis not supported (fail fast).
- Versioning: If a new field
promo_codeis added, Front-End must bump the schema version in the metadata header.
B. Backend ETL (The Processor)
Responsibility: Ingest raw JSON, sanitize, mask PII, and load to Data Lake/Warehouse.
- Data Quality Checks (Gateways):
- Schema Evolution: Fail job if new fields are detected without version bump.
- Null Handling: If
device_contextis missing, inject default{"os": "unknown", "is_mobile": false}. - De-duplication: Drop events with duplicate
event_idwithin a 24-hour window.
- PII Logic:
- If
user_idmatches a specific regex pattern for "guest" users, flag row for reduced retention policy.
- If
- Transformation:
- Flatten
itemsarray into a separatecheckout_itemstable for SQL analysis.
- Flatten
C. Feature Pipeline (The Consumer)
Responsibility: Transform clean warehouse data into features for the Feature Store (e.g., Vertex AI, Feast).
- Derived Feature Logic:
- Feature:
user_cart_abandonment_rate_7d- Logic:
Count(checkout_initiated) - Count(purchase_completed) / Count(checkout_initiated)over rolling 7-day window.
- Logic:
- Feature:
high_value_intent- Logic: Boolean flag. True if
cart_value> 500.00 ANDcurrency== 'USD'.
- Logic: Boolean flag. True if
- Feature:
cart_size- Logic: Count of items in the
itemsarray.
- Logic: Count of items in the
- Feature:
- Expectations:
- The Feature Pipeline expects
cart_valueto be non-negative (guaranteed by ETL). - If
currencyconversion is needed for the ML model, the Feature Pipeline handles standardization to USD using a daily exchange rate table (not ETL).
- The Feature Pipeline expects
4. Breaking Change Policy
- Additive Changes: Adding a new optional field (e.g.,
shipping_method) is non-breaking. Front-End can deploy immediately; ETL updates downstream. - Removal/Type Changes: Removing a field or changing a data type (e.g.,
integertostring) requires a 2-week notice.- Step 1: Create new column
cart_value_v2. - Step 2: Feature Pipeline migrates to
cart_value_v2. - Step 3: Front-End stops sending old field.
- Step 4: ETL drops old column.
- Step 1: Create new column
5. Testing & Validation
To ensure compliance, the following tests run in CI/CD and Production:
- Consumer-Driven Contract Testing: The Feature Pipeline team provides a suite of "Expectation Tests" (e.g., using Great Expectations). If the ETL process produces data that violates these expectations (e.g., a null
cart_value), the pipeline is halted before it reaches the Feature Store. - Synthetic Canaries: The Front-End team injects synthetic events every hour. If these events do not appear in the Feature Store within 20 minutes, an alert is triggered.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.