'In Python, how do you append a list to a Dataframe?
I'm learning Python's Selenium, and I've run into a roadblock trying to scrape a "sales" person's data into a data frame.
When I run this code below, it iterates through pages 1 and page 2 which contain 10 salespeople each, which is great, but the problem is that I only get 10 results.
THE GOAL: To run this code which creates a data frame that brings 20 rows of names.
iter=1
while iter<3:
the_link="https://www.linkedin.com/search/results/people/?keywords=sales&origin=SWITCH_SEARCH_VERTICAL&page="
the_link = the_link+str(iter).strip()
browser.get(the_link)
desc = browser.find_elements_by_class_name("entity-result__title-text")
x=0
name = []
number_of_connections=[]
link = []
#Get Name and NUMBER of connections
while x < len(desc):
name_and_deg_aray = desc[x].text.split("\n")
name.append(name_and_deg_aray[0]) #Don't change 0, it grabs name and puts into array
number_of_connections.append(name_and_deg_aray[3])
link.append(browser.find_element_by_partial_link_text(name_and_deg_aray[0]).get_attribute("href"))
x=x+1
# Get JOB Title of sales person
title=[]
x=0
title_aray = browser.find_elements_by_class_name('entity-result__primary-subtitle')
while x < len(title_aray):
title.append(title_aray[x].text)
x=x+1
iter=iter+1
#create DataFrame
df=pd.DataFrame({'name': name,'title':title, 'number_of_connections':number_of_connections, 'link':link})
Current OUTPUT
['name1',
'name2',
'name3',
'name4',
'name5',
'name6',
'name7',
'name8',
'name9',
'name10']
IDEAL OUTPUT
['name1',
'name2',
'name3',
'name4',
'name5',
'name6',
'name7',
'name8',
...,
...,
...,
'name20']
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
