'How to convert a dropdown selection menu element in HTML/URL to Pandas Dataframe?

While creating datasets for matching and extracting IDs and SubIDs with their names I have the following code in HTML after getting the file from requests module -

    <div class="feature">
            <h5>Network</h5>
            <div>
                <div class="row">
                    <ul class="tree network-tree">
                    
                        
    <li class="classification class-C  ">
        <span>
        <input type="checkbox" >
        <a href="/network/nt06410+N01032+N01031" target="_blank">nt06410</a> Calcium signaling

        </span>
        <ul>
    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01032" data-entry="N01032" target="_blank">N01032</a>
        
        Mutation-inactivated PRKN to mGluR1 signaling pathway

        </span>
    </li>

    <li class="entry class-D network ">
        <span>
        <input type="checkbox" >
        <a href="/entry/N01031" data-entry="N01031" target="_blank">N01031</a>
        
        Mutation-caused aberrant SNCA to VGCC-Ca2+ -apoptotic pathway

        </span>
    </li>

        </ul>
    </li>

What I want to do is get this particular dropdown selection menu that highlights particular linkages into a pandas dataframe - enter image description here

ID Name Subname SubID Network
nt06410 Calcium signaling Mutation-inactivated PRKN to mGluR1 signaling pathway N01032 nt06410+N01032+N01031

My code so far has been -

data = soup.find_all("ul", {"class": "tree network-tree"})

# get all list elements
lis = data[0].find_all('li')

# add a helper lambda, just for readability
find_ul = lambda x: x.find_all('ul')
uls = [find_ul(elem) for elem in lis if find_ul(elem) != []]

# use a nested list comprehension to iterate over the <ul> tags
# and extract text from each <li> into sublists
text = [[li.text.encode('utf-8') for li in ul[0].find_all('li')] for ul in uls]

print(text[0][1])


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source