Python Snippets

Concurrent URL Status Checker with asyncio and aiohttp

This snippet demonstrates how to efficiently check the status of multiple URLs concurrently using Python’s asyncio and aiohttp. This is useful for monitoring website availability, performing bulk health checks, or validating links in a scalable way.

import asyncio
import aiohttp
from typing import List, Dict

async def check_url(session: aiohttp.ClientSession, url: str) -> Dict[str, str]:
    try:
        async with session.get(url, timeout=5) as response:
            return {"url": url, "status": response.status, "error": None}
    except Exception as e:
        return {"url": url, "status": None, "error": str(e)}

async def check_urls(urls: List[str]) -> List[Dict[str, str]]:
    async with aiohttp.ClientSession() as session:
        tasks = [check_url(session, url) for url in urls]
        return await asyncio.gather(*tasks)

if __name__ == "__main__":
    urls = [
        "https://google.com",
        "https://github.com",
        "https://nonexistent.example.com",
    ]
    
    results = asyncio.run(check_urls(urls))
    for result in results:
        print(f"URL: {result['url']}, Status: {result['status']}, Error: {result['error']}")

Explanation

1. Concurrency with asyncio and aiohttp:
   • Instead of checking URLs sequentially (slow for many URLs), this uses asynchronous I/O for parallel HTTP requests.
2. Error Handling:
   • Catches exceptions like timeouts, DNS failures, or invalid SSL certificates, ensuring the script doesn’t crash.
3. Output:
   • Returns a list of dictionaries with each URL’s status (HTTP code) or error message if the request failed.

How to Run

1. Install Dependencies:

   pip install aiohttp
   

2. Save the Script:
   Copy the code into a file (e.g., url_checker.py).
3. Modify URLs:
   Replace the urls list with your target URLs.
4. Execute:
   Run with Python 3.7+:

   python url_checker.py
   

Use Cases

• Website Monitoring: Check if a list of endpoints is online.
• SEO/Link Validation: Identify broken links in bulk.
• Load Testing Prep: Verify URLs before running performance tests.

This snippet avoids disk/file operations, GUIs, or niche libraries, focusing on a scalable networking task with modern Python.