How to properly handle excepetions when webscraping with python and selenium?

Question

I'm doing some Webscraping and looking for the best way to handle exceptions (AttributeError and TypeError) when creating variables in a loop:

The code looks like:

# code to open webdriver, get url, search, open csv file...
while True:
    try:
        # code to get containers 
        for container in containers:
            title = container.find('div', {'class': ("mb-2 tipo")}).get_text()
            f.write(title + ';')
            rooms = container.find('div', {'class': ("quarto")}).get_text()
            f.write(rooms + ';')
            # and so on with lots of variables 
            

I do have a lot of variables to capture and then write on a csv file. But when scraping, there will be times some of the variables will not exist.

I'm looking for the best pythonic way to handle exceptions when this occurs while registering that the variable was not found (and this could be as simple as writing "Nan" in csv file).

Should I write excepetions individually for each variable? or maybe use if/else statements? What's best way?

I would like something like this to be done:

    for container in containers:
        #var 1
        title = container.find('div', {'class': ("mb-2 tipo")}).get_text()
            f.write(title + ';')
        #var 2 
        try:
             rooms = container.find('div', {'class': ("quarto")}).get_text()
            f.write(rooms)
        except Exception as e:
            print(e)
            f.write('Nan' + ';')
        #var 3 --> should I do the same as var 2 and so on?

My doubt is if this is the properly way to handle exceptions for each of a long list of variables

Answer 1

You could write a function like so that would handle the errors. This is assuming you are trying to find similar elements like the example you provided:

def write_element(file, container, class_name):
    try: 
        element = container.find('div', {'class': (class_name)}).get_text()
        file.write(element + ';')
    except Exception as e:
        print(e)
        file.write('Nan' + ';')

And then call it:

for container in containers:
    write_element(f, container, 'title')