1 The Problem
We want a scraper that downloads a web page and pulls out specific pieces — titles, links, prices. It teaches fetching HTML and extracting data from it, plus the responsibility that comes with scraping (respecting sites and their rules).
Where this shows up: price monitoring, news aggregation, research data collection, search engines, market analysis. When a site offers no API, scraping is how you get its data — carefully and respectfully.
2 How to Think About It
Think about fetch-then-extract, before any code:
The plan — in plain English
1. Download the page’s HTML. → 2. Parse it into a structure you can search. → 3. Find the specific elements you want (all the links, say). → 4. Extract their text or attributes. → Always: check the site allows it and do not hammer it with requests.
3 The Build — explained part by part
Here is the complete scraper. It uses Python’s built-in html.parser so nothing needs installing. Each part is explained below.
Pythonscraper.py
import urllib.request
from html.parser import HTMLParser
class LinkExtractor(HTMLParser):
def __init__(self):
super().__init__()
self.links = []
def handle_starttag(self, tag, attrs):
# Called for every opening tag. We want <a href="...">.
if tag == "a":
for name, value in attrs:
if name == "href":
self.links.append(value)
def fetch(url):
# A polite scraper identifies itself with a User-Agent.
request = urllib.request.Request(url, headers={"User-Agent": "CodexBot/1.0"})
with urllib.request.urlopen(request) as response:
return response.read().decode("utf-8")
def extract_links(html):
parser = LinkExtractor()
parser.feed(html)
return parser.links
if __name__ == "__main__":
html = fetch("https://example.com")
for link in extract_links(html):
print(link)
What each part does — in plain words
class LinkExtractor(HTMLParser): — we build on Python’s built-in HTML parser. As it reads the page, it calls our methods for each tag it meets.handle_starttag(self, tag, attrs) — the parser calls this for every opening tag. We check for
<a> tags and grab their href (the link). This is how you target specific elements.headers={"User-Agent": "CodexBot/1.0"} — a polite scraper identifies itself. Many sites block anonymous requests, and identifying yourself is good manners.
response.read().decode("utf-8") — read the raw page and turn the bytes into text.
parser.feed(html) — hand the HTML to the parser; it walks through, calling
handle_starttag for each tag, filling our links list.
Common mistakes — and how to avoid them
✗ Scraping without a User-Agent — many sites block anonymous requests.
✓ Identify your scraper with a
User-Agent header.✗ Hammering a site with rapid requests — rude, and gets you blocked.
✓ Add delays between requests and respect the site's rules.
✗ Ignoring robots.txt and terms of service — some sites forbid scraping.
✓ Always check what a site permits before scraping it.
4 Test & Prove Each Part
We test the extraction logic on a fixed HTML string — no network needed, fully predictable.
Links are extracted from anchor tags
A page with no links returns an empty list
Multiple links are all collected
Pythontest_scraper.py
from html.parser import HTMLParser
class LinkExtractor(HTMLParser):
def __init__(self):
super().__init__()
self.links = []
def handle_starttag(self, tag, attrs):
if tag == "a":
for name, value in attrs:
if name == "href":
self.links.append(value)
def extract_links(html):
parser = LinkExtractor()
parser.feed(html)
return parser.links
def test_extracts_link():
"""A single anchor's href is extracted."""
assert extract_links('<a href="/page">Link</a>') == ["/page"]
def test_no_links():
"""A page with no anchors returns empty."""
assert extract_links("<p>No links here</p>") == []
def test_multiple_links():
"""All anchor hrefs are collected."""
html = '<a href="/a">A</a><a href="/b">B</a>'
assert extract_links(html) == ["/a", "/b"]
Run with pytest -v. We test extraction on fixed HTML strings, never the live web. This makes tests instant and reliable — and means they pass offline.
5 The Interface
INPUTa URLthe page to scrape
What it expects
fetch("https://example.com")OUTPUTextracted linksevery href on the page
What it returns
https://www.iana.org/domains/example6 Run It & Automate It
Run it locally
python3 scraper.pyFetches example.com and lists its links. Scrape responsibly: check a site's robots.txt and terms, and do not send rapid repeated requests.
Jenkins runs the tests automatically — each line explained below.
What you should see when it works
Terminala real run
https://www.iana.org/domains/exampleIf it breaks — how to fix it
🚨
HTTPError 403 Forbidden.The site blocked you. Add a User-Agent header, or the site may not allow scraping.
🚨 No links found on a real page.
The page may load content with JavaScript, which this simple parser does not run. That needs a heavier tool.
GroovyJenkinsfile
// Jenkinsfile — automated tests on every change.
pipeline {
agent any
stages {
stage('Get the code') { steps { checkout scm } } // download the code
stage('Install') { steps { sh 'python3 -m venv venv && . venv/bin/activate && pip install pytest' } } // test tool
stage('Test') { steps { sh '. venv/bin/activate && pytest -v' } } // run every test
}
post {
success { echo 'All tests passed.' }
failure { echo 'A test failed — look above.' }
}
}
🎯 Try this next — make it yours
- Extract other things. Grab all headings or image sources. (Teaches: targeting more tags.)
- Use BeautifulSoup. Install the popular parser for easier extraction. (Teaches: third-party tools.)
- Follow links. Scrape linked pages too, politely. (Teaches: crawling.)
What you learned
You learned to fetch a web page and extract data from its HTML with a parser, to identify your scraper politely, and the responsibility of scraping respectfully. Pulling structured data from the messy web is a powerful, widely-used skill. Related: HTTP, String Methods.