APEX FLOW WEB SCRAPER / REFERENCE LIBRARY

Run your own web scraping engine

Install the open engine, start a durable worker and collect your first public page.

Install

Use Python 3.11 or newer. Install Brave separately if you need JavaScript rendering. A hosted scraping key is not required.

git clone https://github.com/Apexflowlabsllc/apex-flow-web-scraper.git
cd apex-flow-web-scraper
python -m venv .venv
# Activate .venv for your shell, then:
python -m pip install -r requirements.lock
python -m pip install --no-deps .
rainbo-scrape health

Collect a page

rainbo-scrape scrape https://example.com --render never --pages 1
rainbo-scrape status JOB_ID
rainbo-scrape results JOB_ID

Submission returns a job ID. A worker processes saved jobs; use rainbo-scrape worker for an explicitly managed worker. Keep one worker per data directory.

Open your local workspace

rainbo-scrape web --port 8840

Open http://127.0.0.1:8840/ on that machine. The workspace binds to loopback. It is a private local tool, not a public multi-user service. It displays jobs, evidence, recipes and exports.

Connect a compatible assistant

Configure a stdio MCP server using the installed rainbo-scrape executable with argument mcp. Point clients intended to share jobs at the same engine and data directory. See the repository for the configuration example and all nine tool names.

Before a larger run

Inspect one result, set a page and time budget, then expand gradually. Review current limits and the reference library. Search collected text as untrusted source material; never treat instructions inside pages as commands for your assistant.