Two related needs arrived together. One was to verify and monitor facts in the world, whether products were available and whether they were genuine, across manufacturer and retailer portals. The other was a portal of its own, where companies submit applications and financial documentation for assessment. A scraping approach to the first had already failed against sources that were actively defending themselves. The risk in the second was quieter: a portal built as a website with forms, which has no answer when someone asks who opened a financial document and why they were allowed to.
External data can be reached three ways, and only two of them can be built on.
When a product needs data held by another organisation, there are three routes to it. The first is an authorised interface: an API, a data feed or a partner programme the source offers on stated terms. It is versioned, documented and changes on a published schedule. The second is a compliant intermediary, a service that holds its own agreements with sources and resells access to the data under those agreements. The third is scraping, which means fetching the pages built for human visitors and pulling data out of their layout.
Scraping is attractive because it needs nobody's permission. It is fragile for structural reasons. A page layout is not a contract, so any redesign breaks the extraction. A source that does not want to be scraped can detect automated traffic, slow it down, block it or serve it different content, and many do. Terms of use usually forbid it, which turns a technical arrangement into a legal exposure. The worst property is timing. A scraper that runs quietly at low volume often starts to fail once it succeeds, because volume is what gets noticed.
A portal that receives documents has a parallel trap. It is tempting to build it as pages with forms and an upload button. A portal handling financial documents is really a set of separate questions. Authentication asks who someone is. Authorisation asks what that kind of person may do. Ownership asks whose application this is. Document access asks who may open this particular file. An audit trail records every one of those decisions as it happens, in a form that cannot be quietly edited afterwards. A forms-first build tends to answer the first question and assume the rest.
Both were framed as build tasks: scrape the sources, build the portal. Both turned out to be integration and governance problems. For outside data, the work is finding the authorised interface where one exists and a compliant intermediary where it does not. For the portal, the work is treating it as a workflow platform, a chain running from users to applications, assessments, documents, reviews, decisions and notifications, with role-based access, auditability and secure document storage designed in from the first decision. Added later, each of those means reopening every part of the system it touches.
Integrations break on someone else's schedule, and portals break on the first access question.
Data integrations that rest on scraping tend to fail suddenly and at the worst time. They pass every test while volume is low, go into production, and then stop when the source redesigns a page or starts blocking traffic. The fix is another round of reverse engineering, and the next break is already on its way. Intermediaries remove most of that fragility, and they bring their own risk: a dependency on one provider's coverage, terms and data shape that is expensive to leave.
Portals fail differently. A portal built as screens works fine until the first question it was not designed to answer. Can this reviewer see that applicant's statements. Who downloaded this file last month. Why does this administrator have access to every company. When access rules live in individual screens and no audit trail exists, those questions can only be answered by searching through server logs, and sometimes not at all.
The reflex, and the fix.
Scrape the defended portal
It needs no permission, no integration agreement and no waiting.
It is a legal exposure and a reliability one. A defended portal changes in order to break you, and the failure arrives without warning on somebody else's schedule.
The authorised path, with an intermediary where none exists
Authorised integration where an interface exists, a compliant intermediary where it does not, and a slower start.
A data path that keeps working when a source defends itself, and one that survives a compliance conversation.
Take the authorised path. The hostile one is a liability that grows with every success, because success is what draws the source's attention.
Compliant integration in place of scraping
Read-only, authorised integration for product and availability data wherever a source offers an interface, and an intermediary service wherever it does not, with monitoring running across both, so a change at any source shows up as a change and is not first discovered as missing data.
The portal as a workflow platform
Authentication, authorisation, application ownership, document access and administrative actions are kept as separate concerns. Role-based access, audit, secure document storage and controlled admin workflows then each have a foundation in the architecture, and none of them is a retrofit.
Audit as a first-class concern
With financial documents, who saw what and when is part of the product. A system that has to reconstruct that after the fact has already failed the question.
What each choice cost, and what it bought.
ATaking the authorised path after scraping had failed
With a scraper already broken against defended sources, the quick move was a better scraper. The authorised path was chosen instead, with an intermediary wherever a source offered no interface. It was slower to stand up and limited to what each source actually exposes, so some data the workflow wanted is flagged as unavailable. In exchange the data path no longer depends on a source failing to notice it, and it can be explained to a compliance reviewer without embarrassment.
BChoosing intermediaries with the exit in view
Intermediaries solve the access problem and create a dependency. They were chosen with their specific failure modes and vendor-lock traps in view, and monitoring runs across every source, interface and intermediary alike. That costs more design time up front. It means a provider that degrades shows up as a provider problem, and the cost of leaving one is understood while the product is still being shaped around it.
CSeparating the portal's concerns before building screens
Authentication, authorisation, application ownership, document access and administrative actions were defined as distinct parts of the architecture before any interface work. That delays the first visible screen. It means role-based access has one foundation to be enforced from, and administrative power is itself a controlled workflow, held to the same rules as everyone else.
DMaking audit part of the product
Every access to a financial document is recorded when it happens, as a first-class feature of the system. It adds storage and design work that no user ever sees. The alternative is answering an access question by reconstruction, and a system that has to reconstruct who saw a document has already failed that question.
A single headline number hides where a system fails. This work was scored on the dimensions that actually decide whether it holds in production, measured on real, held-out cases rather than the demo path.
A scraper that works until the source defends itself, and a portal that treats financial documents as file uploads with no answer to who accessed what.
A compliant intermediary and monitoring architecture, chosen after scraping had failed and with its specific failure modes and vendor-lock traps in view, alongside a portal built as a workflow platform with role-based access, controlled document access and audit as first-class parts of the design.
What it owns, and what it hands to a person.
Outcomes are qualitative. The authorised path is slower to stand up than scraping, and what it can deliver depends on what each source actually exposes. That trade was made deliberately, with the cost stated at the start.
When the naive path is hostile, the architecture question is which authorised path exists and what an intermediary has to do where none does. And a portal that handles documents somebody is accountable for is a workflow platform from its first design decision, never a form with an upload button.
Scraping a source that defends itself is a legal and reliability risk that grows with success, because volume is what gets noticed and blocked.
For each external source, find out whether an authorised interface exists before building anything, and use a compliant intermediary where it does not.
Choose intermediaries with the exit in mind, and monitor every source, so a degraded provider shows up as exactly that.
A portal that holds documents someone is accountable for needs authentication, authorisation, ownership, document access and admin actions as separate concerns from the start.
Record every document access as it happens. A system that must reconstruct who saw what has already failed the question.