Scrap Hotel facilities of TripAdvisor with Selenium

Question

I want to scrape hotel facilities data on TripAdvisor using Selenium. To have all the facilities of an hotel I need to click on show more. However, I can't do that if i use the click function on the element show more. And if I scrap the information like that (without clicking show more) :

Property_Amenities= driver.find_elements_by_xpath(".//div[@class='bUmsU f ME H3 _c']")
for o in range(len(Property_Amenities)):
    Property_Amenities[o]=Property_Amenities[o].text
    
print(Property_Amenities)
    

It only retrieve 8 facilities and not everything.

All the data is stored in a data-ssrev-handlers above. How can I retrieve information like facilities name easily ?

<div class="ssr-init-26f" data-ssrev-handlers="{&quot;load&quot;:[3,&quot;R5ISL7&quot;,&quot;default&quot;,{&quot;amenities&quot;:{&quot;highlightedAmenities&quot;:{&quot;roomFeatures&quot;:[{&quot;tagId&quot;:18905,&quot;amenityNameLocalized&quot;:&quot;Blackout curtains&quot;,&quot;amenityCategoryName&quot;:&quot;Comfort&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:19085,&quot;amenityNameLocalized&quot;:&quot;Soundproof rooms&quot;,&quot;amenityCategoryName&quot;:&quot;Comfort&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:9658,&quot;amenityNameLocalized&quot;:&quot;Air conditioning&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:12120,&quot;amenityNameLocalized&quot;:&quot;Housekeeping&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:10589,&quot;amenityNameLocalized&quot;:&quot;Safe&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:9666,&quot;amenityNameLocalized&quot;:&quot;Minibar&quot;,&quot;amenityCategoryName&quot;:&quot;KitchenAmenities&quot;,&quot;amenityIcon&quot;:&quot;restaurants&quot;},{&quot;tagId&quot;:11269,&quot;amenityNameLocalized&quot;:&quot;Flatscreen TV&quot;,&quot;amenityCategoryName&quot;:&quot;Entertainment&quot;,&quot;amenityIcon&quot;:&quot;play&quot;},{&quot;tagId&quot;:18897,&quot;amenityNameLocalized&quot;:&quot;Bath / shower&quot;,&quot;amenityCategoryName&quot;:&quot;Bath&quot;,&quot;amenityIcon&quot;:&quot;bathroom-fill&quot;}],&quot;roomTypes&quot;:[{&quot;tagId&quot;:9184,&quot;amenityNameLocalized&quot;:&quot;Non-smoking rooms&quot;,&quot;amenityCategoryName&quot;:&quot;RoomTypes&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:9160,&quot;amenityNameLocalized&quot;:&quot;Suites&quot;,&quot;amenityCategoryName&quot;:&quot;RoomTypes&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:9659,&quot;amenityNameLocalized&quot;:&quot;Family rooms&quot;,&quot;amenityCategoryName&quot;:&quot;RoomTypes&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;}],&quot;propertyAmenities&quot;:[{&quot;tagId&quot;:9176,&quot;amenityNameLocalized&quot;:&quot;Free High Speed Internet (WiFi)&quot;,&quot;amenityCategoryName&quot;:&quot;Internet&quot;,&quot;amenityIcon&quot;:&quot;wifi&quot;},{&quot;tagId&quot;:12111,&quot;amenityNameLocalized&quot;:&quot;Breakfast available&quot;,&quot;amenityCategoryName&quot;:&quot;FoodAndDrink&quot;,&quot;amenityIcon&quot;:&quot;food-drink&quot;},{&quot;tagId&quot;:19103,&quot;amenityNameLocalized&quot;:&quot;Taxi service&quot;,&quot;amenityCategoryName&quot;:&quot;Transportation&quot;,&quot;amenityIcon&quot;:&quot;taxi&quot;},{&quot;tagId&quot;:19063,&quot;amenityNameLocalized&quot;:&quot;24-hour security&quot;,&quot;amenityCategoryName&quot;:&quot;General&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:18896,&quot;amenityNameLocalized&quot;:&quot;Baggage storage&quot;,&quot;amenityCategoryName&quot;:&quot;General&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:18924,&quot;amenityNameLocalized&quot;:&quot;24-hour check-in&quot;,&quot;amenityCategoryName&quot;:&quot;ReceptionServices&quot;,&quot;amenityIcon&quot;:&quot;suitcase-fill&quot;},{&quot;tagId&quot;:18977,&quot;amenityNameLocalized&quot;:&quot;24-hour front desk&quot;,&quot;amenityCategoryName&quot;:&quot;ReceptionServices&quot;,&quot;amenityIcon&quot;:&quot;suitcase-fill&quot;},{&quot;tagId&quot;:9180,&quot;amenityNameLocalized&quot;:&quot;Dry cleaning&quot;,&quot;amenityCategoryName&quot;:&quot;CleaningServices&quot;,&quot;amenityIcon&quot;:&quot;dry-cleaning&quot;}]},&quot;nonHighlightedAmenities&quot;:{&quot;roomFeatures&quot;:[{&quot;tagId&quot;:19104,&quot;amenityNameLocalized&quot;:&quot;Telephone&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:18911,&quot;amenityNameLocalized&quot;:&quot;Bottled water&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:19117,&quot;amenityNameLocalized&quot;:&quot;Wake-up service / alarm clock&quot;,&quot;amenityCategoryName&quot;:&quot;RoomAmenities&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:18949,&quot;amenityNameLocalized&quot;:&quot;Electric kettle&quot;,&quot;amenityCategoryName&quot;:&quot;KitchenAmenities&quot;,&quot;amenityIcon&quot;:&quot;restaurants&quot;},{&quot;tagId&quot;:18972,&quot;amenityNameLocalized&quot;:&quot;Complimentary toiletries&quot;,&quot;amenityCategoryName&quot;:&quot;Bath&quot;,&quot;amenityIcon&quot;:&quot;bathroom-fill&quot;},{&quot;tagId&quot;:18984,&quot;amenityNameLocalized&quot;:&quot;Hair dryer&quot;,&quot;amenityCategoryName&quot;:&quot;Bath&quot;,&quot;amenityIcon&quot;:&quot;bathroom-fill&quot;}],&quot;roomTypes&quot;:[],&quot;propertyAmenities&quot;:[{&quot;tagId&quot;:9173,&quot;amenityNameLocalized&quot;:&quot;Wifi&quot;,&quot;amenityCategoryName&quot;:&quot;Internet&quot;,&quot;amenityIcon&quot;:&quot;wifi&quot;},{&quot;tagId&quot;:18971,&quot;amenityNameLocalized&quot;:&quot;Complimentary Instant Coffee&quot;,&quot;amenityCategoryName&quot;:&quot;FoodAndDrink&quot;,&quot;amenityIcon&quot;:&quot;food-drink&quot;},{&quot;tagId&quot;:18933,&quot;amenityNameLocalized&quot;:&quot;Complimentary tea&quot;,&quot;amenityCategoryName&quot;:&quot;FoodAndDrink&quot;,&quot;amenityIcon&quot;:&quot;food-drink&quot;},{&quot;tagId&quot;:19080,&quot;amenityNameLocalized&quot;:&quot;Snack bar&quot;,&quot;amenityCategoryName&quot;:&quot;FoodAndDrink&quot;,&quot;amenityIcon&quot;:&quot;food-drink&quot;},{&quot;tagId&quot;:19087,&quot;amenityNameLocalized&quot;:&quot;Special diet menus&quot;,&quot;amenityCategoryName&quot;:&quot;FoodAndDrink&quot;,&quot;amenityIcon&quot;:&quot;food-drink&quot;},{&quot;tagId&quot;:9656,&quot;amenityNameLocalized&quot;:&quot;Concierge&quot;,&quot;amenityCategoryName&quot;:&quot;General&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:9670,&quot;amenityNameLocalized&quot;:&quot;Non-smoking hotel&quot;,&quot;amenityCategoryName&quot;:&quot;General&quot;,&quot;amenityIcon&quot;:&quot;hotels&quot;},{&quot;tagId&quot;:18925,&quot;amenityNameLocalized&quot;:&quot;Express check-in / check-out&quot;,&quot;amenityCategoryName&quot;:&quot;ReceptionServices&quot;,&quot;amenityIcon&quot;:&quot;suitcase-fill&quot;},{&quot;tagId&quot;:9623,&quot;amenityNameLocalized&quot;:&quot;Laundry service&quot;,&quot;amenityCategoryName&quot;:&quot;CleaningServices&quot;,&quot;amenityIcon&quot;:&quot;dry-cleaning&quot;},{&quot;tagId&quot;:19003,&quot;amenityNameLocalized&quot;:&quot;Ironing service&quot;,&quot;amenityCategoryName&quot;:&quot;CleaningServices&quot;,&quot;amenityIcon&quot;:&quot;dry-cleaning&quot;}]},&quot;languagesSpoken&quot;:[{&quot;tagId&quot;:18950,&quot;amenityNameLocalized&quot;:&quot;English&quot;},{&quot;tagId&quot;:18976,&quot;amenityNameLocalized&quot;:&quot;French&quot;},{&quot;tagId&quot;:19086,&quot;amenityNameLocalized&quot;:&quot;Spanish&quot;},{&quot;tagId&quot;:18890,&quot;amenityNameLocalized&quot;:&quot;Arabic&quot;},{&quot;tagId&quot;:18928,&quot;amenityNameLocalized&quot;:&quot;Chinese&quot;}]},&quot;loading&quot;:false,&quot;viewport&quot;:&quot;DESKTOP&quot;}]}"><div class="ccdzg S5 b Pf ME">Property amenities</div><div class="exmBD K"><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon wifi eqNRg Nd"></span>Free High Speed Internet (WiFi)</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon coffee-tea-cafe eqNRg Nd"></span>Breakfast available</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon taxi eqNRg Nd"></span>Taxi service</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>24-hour security</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Baggage storage</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon suitcase-fill eqNRg Nd"></span>24-hour check-in</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon suitcase-fill eqNRg Nd"></span>24-hour front desk</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon dry-cleaning eqNRg Nd"></span>Dry cleaning</div><div class="fphCZ K w fwzKF"><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon wifi eqNRg Nd"></span>Wifi</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon food-drink eqNRg Nd"></span>Complimentary Instant Coffee</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon food-drink eqNRg Nd"></span>Complimentary tea</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon food-drink eqNRg Nd"></span>Snack bar</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon food-drink eqNRg Nd"></span>Special diet menus</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Concierge</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon smoking-prohibited eqNRg Nd"></span>Non-smoking hotel</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon suitcase-fill eqNRg Nd"></span>Express check-in / check-out</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon dry-cleaning eqNRg Nd"></span>Laundry service</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon dry-cleaning eqNRg Nd"></span>Ironing service</div></div></div><div><div><span class="fqQue S4 _S Nc">Show more</span><span class="ui_icon caret-down bRRbx"></span></div></div><div class="ccdzg S5 b Pf ME">Room features</div><div class="exmBD K"><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Blackout curtains</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Soundproof rooms</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon air-conditioning eqNRg Nd"></span>Air conditioning</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Housekeeping</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Safe</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon restaurants eqNRg Nd"></span>Minibar</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon play eqNRg Nd"></span>Flatscreen TV</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon bathroom-fill eqNRg Nd"></span>Bath / shower</div><div class="fphCZ K w fwzKF"><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Telephone</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Bottled water</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Wake-up service / alarm clock</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon restaurants eqNRg Nd"></span>Electric kettle</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon bathroom-fill eqNRg Nd"></span>Complimentary toiletries</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon bathroom-fill eqNRg Nd"></span>Hair dryer</div></div></div><div><div><span class="fqQue S4 _S Nc">Show more</span><span class="ui_icon caret-down bRRbx"></span></div></div><div class="ccdzg S5 b Pf ME">Room types</div><div class="exmBD K"><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon smoking-prohibited eqNRg Nd"></span>Non-smoking rooms</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Suites</div><div class="bUmsU f ME H3 _c" data-test-target="amenity_text"><span class="ui_icon hotels eqNRg Nd"></span>Family rooms</div><div class="fphCZ K w fwzKF"></div></div><div></div></div>

Answer 1

to click on show more button, you can use the below XPath:

//div[text()='Show more']

I'd recommend you to use explicit wait to click on it:

Code:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='Show more']"))).click()

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answer 2

To solve the problem :

First, scroll to the link using Action class

element=driver.find_element_by_xpath(".//div[contains(@class, 'dPTxH S4 b _S')]")
    actions = ActionChains(driver)
    actions.move_to_element(element).perform()

Then click on the "show more" using :

ActionChains(driver, 20).move_to_element(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='Show more']")))).click().perform()

After that you can retrieve all the informations you want !