FeedSummary.com Summary prompts

Web source ingestion

Website Scraping for Feed Summaries

Extend feed coverage to selected websites with narrow scope, visible access rules, careful extraction, and block-level change detection.

website scraperrobots.txtrate limitsmain contentchange detection

Respectful extraction

Scrape only inside an explicit source policy

A responsible source profile defines scope, access, rate, identity, and change behavior.

Policy + scope

Approved hosts, paths, page types, use case, crawler identity, and access review.

Rate + caching

Per-host limits, conditional requests, backoff, maximum retries, and stale-source alerts.

Change detection

Compare normalized blocks, not raw HTML that changes for ads, timestamps, or personalization.

Source selection

Prefer feeds and APIs

Scraping is the fallback when a suitable published interface is unavailable.

Discovery

Check RSS, Atom, JSON Feed, sitemaps, publisher APIs, structured data, and curated listing pages before crawling arbitrary links.

Use host and path allowlists and reject out-of-scope redirects.

Extraction

Classify article pages before full processing, remove boilerplate, preserve headings and tables, and retain the fetch record and content hash.

Pause automatic summaries when layout or content metrics shift suddenly.

Feed Signals

Complete scraping guide

Robots policy, transparent identity, rate limits, URL traps, extraction, revisions, and security.