See? It’s not complicated at all.

This lesson introduces some new concepts, so let’s discuss them one line at a time.

First, we import two modules:

import urllib.request

import re

The ‘urllib’ module includes methods that open, read, and parse the desired URLs. We will also use the ‘re’ module, short for ‘Regular Expressions’ or ‘RegEx’, to create a search pattern that will filter the outbound links.

The function below runs when we start the application and tries to get the HTML content from Wikipedia’s homepage.

def fetch_html(url):

    try:

        with urllib.request.urlopen(url) as response:

            return response.read().decode('utf-8')

    except Exception as e:       

print(f"Error fetching HTML: {e}")

        return None

If it succeeds, it returns it decoded according to the UTF-8 standard, which can be understood by Python; otherwise, it displays an error message and returns None.

But let’s assume that everything went okay; Wikipedia was online, and our Internet connection worked great. In this case, we’ve got an HTML data chunk we can play with.

# Main program

if __name__ == "__main__":

    html = fetch_html("https://www.wikipedia.org/")

The following lines of code create a regular expression that extracts the actual URLs from HTML links. We have added a dedicated RegEx tutorial to the "Python Data Structures" course, so you can find a detailed explanation there.

    if html:

        link_pattern = r'href=["\'](https?://[^"\']+)["\']'

        links = re.findall(link_pattern, html)

Finally, we print the links one per line.

    for link in links:

        print(link)

Here’s how the output looks: