The Client
The retail analytics function inside a large Indian consumer goods group — a central team serving five brands across electronics and home categories, all selling on Flipkart. Around 1,200 SKUs under coverage, feeding pricing, supply, and category teams across the group.
Client details are anonymised. Figures are representative of the engagement.
The Problem: A Scraper Everyone Depended On and Nobody Owned
The scraper had been built two years earlier by a data engineer who no longer worked at the company.
It had started, as these things do, as a small internal project. One engineer, roughly six weeks, a Python service and a cron job. It worked. It was celebrated. It went into production, informally, in the way that internal tools go into production — which is to say, nobody ever decided it was production, and so nobody ever gave it an owner.
Two years later, five brands were making pricing decisions on its output.
The engineer who built it had left. The scraper had been inherited, reluctantly, by a data engineer who described it in our first conversation as "about fifteen percent of my job and one hundred percent of my anxiety."
What Was Actually Wrong
The team's initial framing was that the scraper "kept breaking." That turned out to be the least of it.
It broke, and it broke at the worst possible times. Eleven parser failures over eighteen months. Two of them during Big Billion Days — which is not a coincidence but a structural consequence: sale events introduce page elements the parser has never seen, and they do it precisely when the data matters most.
It failed silently, which was far worse. Three months before we were engaged, a Flipkart front-end change had caused the parser to begin returning the Plus member price in the price field for a subset of records. The pipeline did not break. It kept running. It returned numbers. The numbers were plausible.
For roughly five weeks, one brand's pricing team believed a competitor had cut prices across a range of SKUs. They responded. They cut their own prices to match a competitor move that had not happened.
Nobody caught it. There was no null-rate monitoring, no delta gate, no range check. The parser asked for a price, got a number, and wrote a number. The number was for the wrong customer tier.
It captured the wrong things. One row per product page. No seller array — so the brand-protection team, which had asked for unauthorised-seller data twice, was told each time that it "would be a big change." No variant matrix — so in_stock: true on a listing whose main configuration had been unavailable for a fortnight. Single pincode — so the entire tier-2 and tier-3 picture, which was where the group's growth was, was simply absent.
It had no capacity headroom. During the last sale event it had saturated on day one, silently dropped roughly 40 percent of its target records, and reported success.
Nobody knew what it was supposed to capture. This was, in the end, the finding that settled the internal argument. There was no expected-record count. So there was no way to know what had been missed. The pipeline could not distinguish "captured everything" from "captured 60 percent," and it reported both identically.
The Honest Cost
The team's CFO had approved the original build on the basis of a simple comparison: six weeks of one engineer, versus a vendor contract. The build won easily.
That comparison was not wrong. It was incomplete, in a way that only becomes visible in retrospect.
| Cost line |
Budgeted at build time? |
Actual over 18 months |
| Initial build |
✅ Yes |
~6 engineer-weeks |
| Infrastructure and proxies |
✅ Yes |
As modelled |
| Parser maintenance |
❌ No |
~11 incidents, ~3–5 days each |
| Sale-day firefighting and on-call |
❌ No |
2 events, senior engineer, weekends |
| Feature requests never delivered |
❌ No |
Seller array, variants, pincodes — all deferred indefinitely |
| Data quality gates |
❌ No |
Never built |
| Decisions made on wrong data |
❌ No |
One five-week episode of unnecessary price cuts |
Add the maintenance lines and the in-house scraper had consumed substantially more engineering time in maintenance than it had in construction — while delivering a fraction of the fields the business had asked for.
And the last line is the one that ended the debate. The unnecessary price cuts, sustained across a range of SKUs for five weeks, cost more in foregone margin than several years of a vendor contract.
The Solution
Product Data Scrape replaced the in-house pipeline over four weeks, running both in parallel for two of them.
- A schema that models the platform — all price tiers preserved separately, nested seller arrays with F-Assured and default-seller flags and stable seller IDs, full variant matrices with structured attributes, structured bank offers with cap-aware computation, pincode resolution across a 60-pincode panel, sale-event deal flags.
- Validation gates between capture and delivery — range checks, event-aware delta checks, cross-field consistency, schema conformance, and, critically, per-field null-rate monitoring alerting on change rather than level. This is the gate that would have caught the Plus-price incident within a day.
- Expected-record counts. Every run knows what it intended to capture. A shortfall is an alert, not a silence.
- Capacity provisioned for the peak, with degradation rules defined in advance that protect hero-SKU capture under load, and a T-7 rehearsal before every sale event.
- The maintenance burden moved off the client entirely.
Sample Data: The Gate That Would Have Caught It
The null-rate monitor, illustrated on the exact failure mode that had cost them five weeks.
{
"monitor": "field_null_rate",
"field": "plus_exclusive_price",
"window": "24h",
"null_rate_today": 0.03,
"null_rate_trailing_7d": 0.61,
"delta": -0.58,
"threshold": 0.15,
"status": "ALERT",
"interpretation": "Field populated on 97% of records today vs 39% on trailing average. Likely parser drift: plus_exclusive_price may be capturing a value from a different tier.",
"records_quarantined": 4118,
"delivered_to_client": false
}
The records were quarantined, not delivered. That is the point of a gate.
The previous pipeline had no equivalent. It would have delivered all 4,118 records, confidently, into a dashboard, into a pricing review, and into a decision.
The Results
| Metric |
In-House Scraper |
Product Data Scrape |
| Fields captured |
9 |
60+ |
| Seller array |
❌ |
✅ Full, with stable seller IDs |
| Variant matrix |
❌ |
✅ Full |
| Pincode coverage |
1 |
60 |
| Data quality gates |
0 |
6 gate types |
| Silent-failure incidents (12 months) |
≥1 known, unknown true count |
0 delivered; 3 caught and quarantined |
| Sale-day record completeness |
~60% (undetected) |
99%+, with completeness reported |
| Engineer time on pipeline |
~15% of one FTE, ongoing |
~0 |
| Brand-protection team served |
❌ Deferred 2 years |
✅ Enforcement queue live in week 3 |
Figures are representative of the engagement outcome.
The reassigned engineer's time went to a demand-forecasting project that had been backlogged for a year.
And the brand-protection request that had been deferred for two years — because adding the seller array to the in-house scraper was "a big change" — was live in the third week, because it had never been a big change. It had simply been in a schema nobody had the capacity to extend.
The Lesson
The build-versus-buy conversation is almost always held on the wrong axis.
The question is not can we build this? Of course you can. A competent engineer will have a working Flipkart scraper in a few weeks, and it will feel like a win.
The right questions are:
- Who owns it in eighteen months? Not who builds it. Who is on call when it breaks during a sale, after the person who built it has moved on.
- How will you know when it is wrong? Not broken — wrong. Silently returning a plausible number from the wrong price tier. If you cannot answer this, you do not have a data pipeline. You have a source of confident errors, and the confidence is the dangerous part.
- What will it cost when a decision is made on bad data? Put a number on it. It is almost always larger than the entire engineering cost of the alternative, and it is the only line in the comparison that nobody ever includes.
The engineer who inherited this scraper had the most accurate summary of the whole engagement, and he offered it in the first meeting: "It's not that it doesn't work. It's that I have no way of knowing when it doesn't."
Work With Product Data Scrape
Product Data Scrape replaces in-house Flipkart scrapers with a maintained, validated, schema-complete data feed: all price tiers, nested seller arrays with stable identifiers, full variant matrices, structured offers, pincode panels, sale-event capacity — and validation gates that catch the silently wrong values before they reach a decision.
If you have an in-house scraper and no null-rate monitoring, we can tell you in one parallel run whether it is currently lying to you.
Product Data Scrape — turning marketplace complexity into decision-ready data.
Singapore
EMAIL
PHONE
.webp "thumb")
.webp "thumb")
.webp)
