What This Article Is Not
Let us be direct about this before anything else, because it shapes everything that follows.
This is not a guide to evading anti-bot systems. We do not build that, we do not sell it, and we would advise you against it for three reasons — in ascending order of how much they should matter to you.
The weakest reason is that it does not work for long. Adversarial approaches invite an adversarial response, and you end up in an arms race you did not want and cannot win economically.
The stronger reason is that it is fragile in exactly the wrong way. A pipeline built on evasion is a pipeline that fails hardest under the most scrutiny — which means it fails during sale events, which is when your data is worth the most.
The strongest reason is commercial, and it is the one that decides deals. No brand's legal or procurement team will sign a data vendor whose approach is built on circumventing another company's technical controls. If your ICP is a global FMCG brand, an electronics manufacturer, or a beauty conglomerate — the sort of organisation that has a general counsel and a vendor risk questionnaire — then a vendor that talks about defeating bot detection is a vendor that does not get past the first review. We have watched it happen to competitors, and it is a self-inflicted wound.
So this article is about something more useful and considerably harder: collecting at scale without triggering protections in the first place, degrading gracefully when platform-side controls do engage, and — the part almost nobody talks about — knowing when your data is wrong.
The Three Things That Actually Break Pipelines
Ask an engineer why their scraper failed and they will usually say "we got blocked." In our experience, that is the third most common cause, and by some distance the least expensive.
1. Capacity
The volume maths surprises people, so let us do it.
A modest brand programme: 500 SKUs × 100 pincodes = 50,000 product-location pairs.
Now expand each record. Average 6 variants and 4 sellers per SKU, and each pair becomes a nested object with roughly 24 sub-records. Capture 4 times daily and you are generating on the order of 4.8 million sub-records a day — in an ordinary week.
During a sale event, hero SKU frequency rises from 4 times daily to every 15 minutes. That is a 24× increase on the SKUs that matter, layered on top of a platform that is itself under peak load.
A pipeline provisioned for the average does not degrade gracefully at the peak. It queues, then times out, then silently drops the records it cannot serve — and the records it drops are, by construction, the hero SKUs it was queueing hardest on.
2. Parser drift
This is the expensive one, and it is expensive precisely because it does not look like a failure.
Flipkart's front end changes. Deal banners appear during sale events. Offer formats shift. A field moves. Any of these does one of two things to your parser:
- It breaks. Good. You get paged, you fix it, you lose a day.
- It keeps working and returns the wrong value. Catastrophic. The dashboard populates. The chart is smooth. No alert fires — from the pipeline's point of view, nothing is wrong. It asked for a price, it got a number, it wrote a number.
The number is the Plus price where you wanted the standard price. Or it is the base variant where you wanted the configuration you actually sell. Or plus_exclusive_price got coerced from null to zero, and your effective-price model just concluded that a competitor is giving the product away.
A pipeline that fails loudly is a nuisance. A pipeline that fails silently is a liability, and the cost is not measured in downtime. It is measured in the decision somebody made on the wrong number.
3. No degradation strategy
Under load, a well-designed pipeline sheds work deliberately: it drops long-tail SKU frequency to protect hero-SKU capture, and it tells you it has done so.
A badly designed one attempts everything at full frequency, saturates, and fails at everything — including the fourteen SKUs that actually mattered.
Request Budgeting: Do the Maths First
Before architecture, arithmetic.
daily_requests =
n_skus
× n_pincodes
× captures_per_day
× (1 + retry_rate)
peak_requests_per_minute =
hero_skus
× hero_pincodes
× (60 / capture_interval_minutes)
Compute the peak, not the average. Then provision for the peak, and then apply a concurrency ceiling below what your infrastructure could technically sustain.
That last step is the one that gets skipped, and it is the most important. Your concurrency ceiling should be set by what is respectful, not by what is possible. A pipeline that requests as fast as its infrastructure allows is a pipeline that will eventually get itself rate-limited, and it will deserve it.
Collecting Respectfully
The single best strategy for handling rate limits is not hitting them.
Set a concurrency ceiling and stay under it. Deliberately. Not as a fallback — as the design.
Exponential backoff with jitter. When a request fails or a rate limit is signalled, back off — and back off increasingly. Add jitter so that a fleet of workers does not synchronise into a thundering herd the moment the backoff expires.
Back off; do not escalate. This is the fork in the road. When platform-side protections engage, there are two responses. One is to slow down, spread out, and reduce load. The other is to try harder. The first is engineering. The second is an arms race that you will lose, that will make your pipeline less reliable, and that will cost you the enterprise deal when it comes up in the vendor questionnaire.
Schedule off-peak where you can. The long tail does not need to be captured at 20:00. Move it to 04:00 and leave headroom at peak for the SKUs that need it.
Cache aggressively. A long-tail SKU that has not moved in six weeks does not need four captures a day. Adaptive frequency — driven by observed volatility per SKU — typically cuts request volume substantially with no loss of signal, because most of the catalogue genuinely is not moving.
Respect the platform's stated terms and technical signals. This is not merely an ethical position, though it is that. It is the position that keeps your programme alive and your customers' legal teams comfortable.
Graceful Degradation: Decide Now, Not at 02:00
Tier your catalogue before the sale, and define what gets dropped before you are under load.
| Tier |
Normal frequency |
Under load |
| Hero SKUs |
Every 15 min (sale) |
Protected — never reduced |
| Competitor hero SKUs |
Every 15 min (sale) |
Protected |
| Core SKUs |
Hourly |
Reduce to every 3 hours |
| Long tail |
4× daily |
Reduce to daily |
| Non-critical pincodes |
Daily |
Suspend |
And then, crucially: surface the degradation. A dashboard that silently reports stale long-tail data as if it were fresh is a dashboard that lies. Every record should carry its own captured_at, and every view should show the freshness of what it is displaying.
Data Quality Gates: The Part Everyone Skips
If you take one thing from this article, take this section. Uptime is not the metric. Correctness is the metric, and it is far less commonly measured.
Validation gates that belong between capture and delivery:
Range checks. A price of 0, a price above MRP, a negative discount, a delivery ETA of 400 days. Cheap to implement, and they catch a surprising share of parser drift.
Delta checks. A price that moved more than a threshold since the last capture is flagged, not written. During a sale, deep moves are legitimate — so the threshold has to be event-aware, or your gate will fire on every real deal and get switched off by an exhausted engineer, which is the worst possible outcome.
Null-rate monitoring. This is the highest-value gate and the one almost nobody builds. If plus_exclusive_price was populated on 61 percent of records yesterday and 3 percent today, your parser has drifted — and every downstream number is now wrong in a way no range check will catch. Monitor the null rate of every field as a time series, and alert on the change, not the level.
Schema conformance. Types, enums, required fields, no silent coercion.
Never coerce null to zero. plus_exclusive_price: 0 looks like a bargain and is a bug. It will propagate into an effective-price model and out into a pricing decision, and by the time anyone traces it back, the price has already been cut.
Cross-field consistency. If in_stock: false, stock_signal should not read few_left. If deliverable: false, there should not be a price.
Sale-Day Capacity Planning
Provision for the peak, not the average. The peak is roughly 20–30× the average on hero SKUs during a major sale. Infrastructure sized for a typical Tuesday will not survive day one.
Rehearse at T-7. Run the full sale configuration for 24 hours. Every engagement we have rehearsed has surfaced at least one problem — an alert routing to an unmonitored channel, a competitor SKU mapped to the wrong internal reference, a degradation rule that silently dropped a hero SKU. All of these are cheap to fix a week out and expensive to discover at 02:00 on day one.
Pre-warm. Do not scale up when the sale opens. Scale up before it does.
Have a named on-call. Not a rota. A person, who has agreed to it.
Observability: Monitor the Monitor
| Metric |
Alert on |
| Records captured vs expected |
Any shortfall — a silent shortfall is the most dangerous state |
| Capture latency (p50, p95, p99) |
Rising p99 is your earliest warning of saturation |
| Null rate per field |
Change, not level |
| Validation gate rejections |
Any spike |
| Data freshness per SKU tier |
Staleness beyond the tier's SLA |
| Backoff events |
Trend, not individual events |
The first row deserves emphasis. You must know what you expected to capture, or you cannot know what you missed. A pipeline that captures 60 percent of its target and reports success is the default failure mode of every unmonitored scraper on earth.
Compliance: Why Your Buyer's Lawyer Cares
If your data is going to inform pricing decisions, brand-protection enforcement, or supply planning at a large brand, it will pass through a legal review. What that review asks:
- Is this publicly available information? It should be. Product listings, prices, and availability are public.
- Is any personal data collected? It should not be. None. Ever.
- Is any authentication circumvented? It must not be.
- Are technical protection measures circumvented? They must not be.
- Is collection rate-limited and respectful? It should be, and you should be able to describe how.
- Can the evidence chain be defended? For enforcement use, timestamped, continuous, reproducible capture with a documented methodology.
Every one of these is a reason to build the pipeline the way described above rather than the adversarial way. The compliant architecture and the reliable architecture are the same architecture. That is not a coincidence — respectful collection is stable collection, and stable collection is what survives a sale day.
Frequently Asked Questions
How do you handle rate limits?
Concurrency ceilings set below capacity, exponential backoff with jitter, adaptive frequency driven by observed SKU volatility, and off-peak scheduling for the long tail. We back off. We do not escalate.
What happens during Big Billion Days?
Capacity is provisioned in advance for the peak, hero-SKU capture is protected under load by design, degradation rules are defined before the event, and the configuration is rehearsed at T-7.
How do you catch silently wrong data?
Range checks, event-aware delta checks, per-field null-rate monitoring, schema conformance, and cross-field consistency gates — between capture and delivery, not after.
Do you circumvent anti-bot measures?
No. We collect publicly available product information within respectful rate limits. It is the right thing to do, it is what your legal team requires, and it is what keeps the pipeline standing on the day it matters.
Build for the Day It Breaks
Any competent engineer can build a Flipkart scraper that works on a Tuesday.
The question that decides whether a data programme is worth having is a different one: does it work on the first morning of Big Billion Days, at 02:00, under peak load, when a competitor has just cut a price and your hero SKU is about to go out of stock?
And a second question, harder and more important: when it returns 24,999, do you know that number is right?
Product Data Scrape operates Flipkart scraping at scale on the architecture above — respectful, rate-limited collection with no circumvention; capacity provisioned for the peak rather than the average; degradation rules that protect hero-SKU capture by design; and validation gates that catch the silently wrong values before they reach your dashboard.
We take the maintenance burden. Your engineer gets their evenings back, and your legal team gets an answer they can sign.
Product Data Scrape — turning marketplace complexity into decision-ready data.
Singapore
EMAIL
PHONE
.webp)
