A need to verify and monitor real-world data, product availability and authenticity across manufacturer or retailer portals, and company applications and financial documentation inside a portal, where a scraping approach had already failed or was actively hostile. Scraping a defended retailer is a legal and reliability risk, and a portal treated as a website with forms will fail to enforce ownership, access, and audit on financial documents.
The stated ask is scrape it, or build the portal. The real problem is compliant data integration: using an authorized interface where one exists and a compliant intermediary where it does not. And separately, treating a portal as a workflow platform, a chain from users to applications to assessments to documents to reviews to decisions to notifications, with role-based access, auditability, and secure document storage as first-class parts of the architecture rather than features added later.
The reflex, and the fix.
Scrape the defended portal
It needs no permission, no integration agreement, and no waiting.
It is a legal exposure and a reliability one. A defended portal changes to break you, and the failure arrives without warning on somebody else's schedule.
The authorized path, with an intermediary where none exists
Authorized integration where an interface exists, a compliant intermediary where it does not, and a slower start.
A data path that does not break when the source defends itself, and one that survives a compliance conversation.
Take the authorized path, because the hostile one is a liability that compounds with success.
Compliant integration rather than scraping
Authorized read-only product and availability integration where an interface exists, and an intermediary service where it does not, with monitoring across both.
The portal as a workflow platform
Authentication, authorization, application ownership, document access, and administrative actions separated, so role-based access, auditability, secure document storage, and controlled admin workflows have an architectural foundation rather than a retrofit.
Audit as a first-class concern
On financial documents, who saw what and when is part of the product. A system that has to reconstruct that after the fact has already failed the question.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
A scraper that works until the source defends itself, and a portal that treats documents as file uploads with no answer to who accessed what.
A compliant intermediary and monitoring architecture chosen after a scraping approach failed, with awareness of the specific failure modes and vendor-lock traps, and a workflow-platform portal with role-based access, document access, and audit as first-class concerns.
What it owns, and what it hands to a person.
Outcomes are qualitative. The authorized path is slower to stand up than scraping and depends on what each source actually exposes, which is the trade being made deliberately rather than a limitation being conceded.
When the naive path is hostile, the architecture question is which authorized path exists and what an intermediary has to do where none does. And a portal handling documents somebody is accountable for is a workflow platform, not a form, from the first design decision onward.