Opportunity Intelligence Engine

Problem statement

Procurement, freelance and supplier opportunities are scattered across feeds and websites. Manual monitoring produces stale spreadsheets, missed deadlines and duplicated rows with no audit trail.

Engineering objective

1. Mapped source connectors, normalisation rules and scoring dimensions required for explainable prioritisation.
2. Defined idempotent ingestion so repeated runs do not duplicate records.
3. Separated ambiguous duplicate handling from automatic merges.
4. Scoped reporting, API access and operator notifications around the core pipeline.

Solution

A deterministic Python automation platform that collects opportunities from multiple sources, preserves raw payloads, validates and normalises fields, deduplicates with explicit ambiguous states, scores each record with persisted reasons, and exports reports through API and dashboard surfaces.

• Multi-source collection with per-source failure isolation.
• Raw and normalised persistence with change detection on re-ingestion.
• Explainable scoring via configurable weights and deadline urgency classification.
• FastAPI API, Jinja dashboard, Prefect wrapper, and SQLite or PostgreSQL deployment options.

How it works

Python 3.12, FastAPI, Pydantic, SQLAlchemy 2, Alembic.

PostgreSQL or SQLite, httpx, BeautifulSoup, optional Playwright.

Prefect 3, openpyxl, Jinja2, pytest, Ruff, mypy, Docker.

REST API for opportunities, exports, run history and source health.

Example use case

How fragmented opportunity monitoring becomes a prioritised pipeline.

Controls and constraints

• Responsible HTTP client with timeouts, retries, rate limits and robots.txt checks.
• Raw records retained for provenance without auto-merging uncertain duplicates.
• Scraping limited to public, permitted collection in demo configuration.
• SMTP and webhook notifications optional; console notifications always available.

Key capabilities

• Collection, validation, normalisation and deduplication pipeline.
• Scoring engine with persisted reasons and deadline classification.
• Excel, CSV and JSON export paths.
• FastAPI service with dashboard and OpenAPI documentation.
• CLI, demo dataset and test suite.

Product and system views

Automating fragmented opportunity discovery into a traceable, scored data pipeline.

What the system enables

• Demonstrates end-to-end automation design without relying on an LLM for core logic.
• Shows how to make scoring and duplicate handling auditable for operations teams.
• Provides a reproducible case-study demo with synthetic bundled data.

Validation and testing

• pytest unit, integration and end-to-end coverage.
• Idempotent demo runs with documented count expectations.
• CI mocks external HTTP behaviour.

Limitations

• Demo HTML, RSS and JSON connectors read local fixtures rather than live portals.
• Playwright is optional and unused in CI.
• SQLite is for local demo, not concurrent production load.