Mastering Web Scraping with Python and Crawl4AI: A Step-by-Step Guide

Web scraping is an essential skill for data analysts, researchers, and developers. Whether you’re gathering real estate listings, tracking e-commerce prices, or extracting news articles, Python offers powerful tools to automate data collection.

In this article, we’ll explore Crawl4AI, a modern Python library that simplifies web scraping with CSS selectors, headless browsing, and AI-powered extraction. We’ll build a structured scraper step by step, separating concerns for maintainability and efficiency.

Why Crawl4AI?

Crawl4AI stands out because:
✔ Handles JavaScript-heavy websites (like React/Angular apps)
✔ Supports both CSS and AI-based extraction
✔ Runs asynchronously for faster scraping
✔ Built-in caching and proxy support

Let’s dive in!

Step 1: Setting Up the Project

create python project and Installation

pip install crawl4ai pydantic

Project Structure

scraper/  
│── main.py          # Runs the crawler  
│── extractors.py    # Defines data extraction logic  
│── schemas.py       # (Optional) Pydantic models for validation  

Step 2: Defining the Scraper Logic (extractors.py)

Option A: CSS-Based Extraction (Best for Structured Data)

from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

def get_css_extractor():
    schema = {
        "name": "Properties",
        "baseSelector": "div.property-card",  # Parent element
        "fields": [
            {"name": "title", "selector": "h2.title", "type": "text"},
            {"name": "price", "selector": "span.price", "type": "text"},
            {"name": "link", "selector": "a.property-link", "type": "attribute", "attribute": "href"},
        ]
    }
    return JsonCssExtractionStrategy(schema)

Option B: AI-Powered Extraction (Best for Unstructured Data)

from crawl4ai import LLMExtractionStrategy, LLMConfig
from schemas import PropertyModel  # Pydantic model (optional)

def get_llm_extractor():
    llm_config = LLMConfig(
        provider="groq/llama3-70b",  # Or "openai/gpt-4"
        api_key="your_api_key",
    )
    
    return LLMExtractionStrategy(
        llm_config=llm_config,
        schema=PropertyModel.model_json_schema(),
        instruction="Extract property title, price, and link.",
    )

Step 3: Data Validation (Optional, schemas.py)

If using AI extraction, Pydantic ensures clean data:

from pydantic import BaseModel

class PropertyModel(BaseModel):
    title: str
    price: str
    link: str

Step 4: Running the Crawler (main.py)

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
from extractors import get_css_extractor  # or get_llm_extractor

async def scrape_website(url: str):
    browser_config = BrowserConfig(
        headless=True,  # No GUI (faster)
        viewport_width=1280,
    )

    run_config = CrawlerRunConfig(
        extraction_strategy=get_css_extractor(),  # or get_llm_extractor()
        cache_mode="bypass",  # Disable caching
    )

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url, config=run_config)
        
        if result.success:
            print("✅ Data extracted successfully!")
            print(result.extracted_content)  # Save to JSON/DB
        else:
            print("❌ Failed:", result.error_message)

if __name__ == "__main__":
    target_url = "https://example.com/properties"
    asyncio.run(scrape_website(target_url))

Step 5: Advanced Techniques

1. Rate Limiting & Proxies

browser_config = BrowserConfig(
    proxy="http://user:pass@proxy_ip:port",
    request_delay=2.0,  # 2-second delay between requests
)

2. Saving Data to JSON

import json

if result.success:
    with open("output.json", "w") as f:
        json.dump(result.extracted_content, f)

3. Handling Pagination

Loop through multiple pages:

base_url = "https://example.com/properties?page="
for page in range(1, 6):  # Scrape pages 1-5
    url = base_url + str(page)
    asyncio.run(scrape_website(url))

Final Thoughts

✅ For structured sites: Use CSS extraction (faster).
✅ For unstructured text: Use LLM extraction (more flexible).
✅ Always check robots.txt before scraping.
✅ Use delays/proxies to avoid IP bans.

Crawl4AI makes web scraping easier, faster, and more reliable. Try it for your next data project!

Nisal Gunawardhana.

Email — contact@nisalgunawardhana.com | web — Nisal Gunawardhana

LinkedIn | Instagram | Facebook | YouTube | X(Twitter) | GitHub