Control cost, quality, retries, and responsible use

Success means more than HTTP 200. A page can load while required fields are missing, stale, regionally wrong, or semantically incompatible.

Understand the cost levers

Decodo’s current Web Scraping API documentation says request cost depends mainly on:

• proxy pool: Standard for simpler targets; Premium for more protected targets;
• JavaScript rendering: useful for dynamic pages, but consumes more browser resources;
• target template and plan: rates and allowances vary.

Start with Standard and rendering disabled for a static page. Escalate only when an acceptance check fails for a reason rendering or Premium access can fix.

Put a budget in every job

Maximum Decodo calls: 6
Normal calls: 3
Retries: at most 1 per failed target
Escalation: Standard to Premium only after a recorded access failure
Rendering: off by default; enable only when a required field is absent
Stop: if more than 20% of required fields are missing

Define acceptance before collection

Example product record:

{
  "source_url": "required URL",
  "observed_at": "required timestamp",
  "market": "required string",
  "product_id": "required string",
  "title": "required string",
  "price": "number or null",
  "currency": "ISO code or null",
  "stock_text": "string or null",
  "seller": "string or null"
}

Validation rules:

1. Required fields cannot be empty.
2. Currency must not be inferred from language alone.
3. Product ID must match the requested listing.
4. price: null is valid only with a reason such as unavailable or not shown.
5. Observation timestamp and source URL are mandatory.

Retry by cause, not hope

Failure	One useful response
401 Unauthorized	Check Basic token and header; do not retry unchanged
Empty dynamic section	Enable rendering once
Wrong region	Fix domain, geo, postal code, and locale alignment
Selector no longer matches	Inspect current public page; update stable selector
Rate limit	Respect backoff and plan limit; do not add parallel retries
Target-specific parser misses field	Preserve raw evidence; use another documented response format or report missing
Login/private content	Stop

Monitor accepted output

Track:

• requests attempted and billed;
• successful HTTP responses;
• records passing validation;
• missing-field rate by field;
• retries by cause;
• rendering/Premium escalation rate;
• latency;
• cost per accepted record;
• target and parser changes.

High HTTP success with low accepted-record success is not a healthy workflow.

Responsible-use gate

Before scheduling:

• Target is public or you have explicit authorization.
• Job respects applicable website terms and policies.
• No post-login or private content is collected.
• Personal data is minimized and has a legitimate purpose.
• Retention period is defined.
• Access to collected data is limited.
• People can review consequential outputs.
• A stop switch exists.

Final runbook

1. State target, deliverable, freshness, and request budget.
2. Choose the narrowest Decodo tool or API template.
3. Run one harmless smoke test.
4. Validate source, geography, timestamp, and required fields.
5. Add rendering, Premium access, actions, or retries only for a named failure.
6. Preserve raw evidence and normalized records.
7. Review claims before publishing or acting.
8. Monitor cost per accepted record.

You now have a complete path from a one-request MCP scrape to a bounded async workflow. Keep the human review gate after collection, not beside it.

Official references

• Web Scraping API introduction and pricing factors
• Usage statistics
• Web Scraping API response codes
• Decodo MCP server