'Scrapy CrawlSpider parsing denied XPATHs
I am wondering why my CrawlSpider
is parsing pages which I restricted.
I was trying several approaches.
I tried to use the restrict_xpath
parameter like this
restrict_xpaths=(['//aside[@id="aside-st"]', # navigation bar
'//ul[@class="categoryLinks"]', # level 3 links
'//a[@rel="next"]',
'//li[@class="level01 group opened current" and not(contains(text(), "Themen"))]']),
and also the deny
like this
LinkExtractor(deny=['//li[@class="level01 group opened current" and not(contains(text(), "Themen"))]'],..
I was wondering if there'd be an error in my XPATH
, but I also tried several approaches here. Neither worked.
The HTML contruct looks like this:
<li class="level01 group opened current">
<a class="selected" href="https://shop.apotal.de/themen-welten" data-btattached="true"> Themen-Welten </a> <ul class="menu01"> <li class="level02 link"> <a class="" href="https://shop.apotal.de/schluss-mit-akne" data-btattached="true"> Schluss mit Akne </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/alzheimer-demenz" data-btattached="true"> Alzheimer - Demenz </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-allergie" data-btattached="true"> Allergie </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/bauch-beine-po" data-btattached="true"> Bauch, Beine, Po </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/calcium-sandoz" data-btattached="true"> Calcium Sandoz </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-darmgesundheit" data-btattached="true"> Darmgesundheit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-diabetes" data-btattached="true"> Diabetes </a> </li> <li class="level02 group"> <a class="" href="https://shop.apotal.de/onkologie" data-btattached="true"> Onkologie </a> </li> <li class="level02 group"> <a class="" href="https://shop.apotal.de/einfach-sparen-1a-pharma" data-btattached="true"> Einfach sparen - 1A Pharma </a> </li> <li class="level02 group"> <a class="" href="https://shop.apotal.de/frauengesundheit" data-btattached="true"> Frauengesundheit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/gingium-bei-tinnitus" data-btattached="true"> Gingium bei Tinnitus </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/grillzeit" data-btattached="true"> Grillzeit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-haarausfall" data-btattached="true"> Haarausfall </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten/hausapotheke" data-btattached="true"> Hausapotheke </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/hautsache" data-btattached="true"> Hautsache </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-kinderwunsch" data-btattached="true"> Kinderwunsch </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/kuckst-du" data-btattached="true"> Kuckst Du </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-listerine" data-btattached="true"> Listerine </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/maennergrippe" data-btattached="true"> Männergrippe </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/mineralstoff-versorgung" data-btattached="true"> Mineralstoff-Versorgung </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/naturapotheke" data-btattached="true"> Naturapotheke </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/omep" data-btattached="true"> Omep </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-pflanzliche-arzneimittel" data-btattached="true"> Pflanzliche Arzneimittel </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/reiseapotheke" data-btattached="true"> Reiseapotheke </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/themen-welten-reisezeit" data-btattached="true"> Reisezeit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/safer-sex" data-btattached="true"> Safer Sex </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/veroval-selbstdiagnostik" data-btattached="true"> Veroval Selbstdiagnostik </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/sport-u-freizeit" data-btattached="true"> Sport u. Freizeit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/thermacare" data-btattached="true"> Thermacare </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/vismed" data-btattached="true"> Vismed </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/weisse-beisser" data-btattached="true"> Weiße Beisser </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/winterblues" data-btattached="true"> Winterblues </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/phytolastil-anti-dehnungsstreifen-und-schwangerschaft-und-stillzeit" data-btattached="true"> Phytolastil - Anti-Dehnungsstreifen & Schwangerschaft & Stillzeit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/eucerin-anti-age" data-btattached="true"> Eucerin Anti-Age </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/eucerin-trockenehaut" data-btattached="true"> Eucerin Trockene Haut </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/hexoral" data-btattached="true"> Hexoral </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/ziele-erreichen" data-btattached="true"> Ziele erreichen </a> </li> <li class="level02 group"> <a class="" href="https://shop.apotal.de/cellavent-healthcare" data-btattached="true"> Cellavent Healthcare </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/arkeskin-pflege-in-den-wechseljahren-menopause-und-anti-age" data-btattached="true"> Arkeskin - Pflege in den Wechseljahren/Menopause & Anti-Age </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/abanta-hautcreme" data-btattached="true"> Abanta - Hautcreme </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/magenschmerzen-iberogast" data-btattached="true"> Magenschmerzen - Iberogast </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/schmerzen-aspirin" data-btattached="true"> Schmerzen - Aspirin </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/axicur" data-btattached="true"> axicur </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/exeltis-frauengesundheit" data-btattached="true"> Exeltis Frauengesundheit </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/-eisenmangel" data-btattached="true"> Eisenmangel </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/kadefungin-vaginalflora" data-btattached="true"> Kadefungin - Vaginalflora </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/physiogel-geroetete-haut" data-btattached="true"> Physiogel - gerötete Haut </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/clearblue-schwangerschaftstest" data-btattached="true"> Clearblue Schwangerschaftstest </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/clearblue-ovulationstest" data-btattached="true"> Clearblue Ovulationstest </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/beckenbodentraining-lubexx" data-btattached="true"> Beckenbodentraining-Lubexx </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/darmflora-symbiopharm" data-btattached="true"> Darmflora - Symbiopharm </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/dr-kappl" data-btattached="true"> Dr. Kappl </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/nhc-naturheilkunde" data-btattached="true"> NHC Naturheilkunde </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/wiedemann" data-btattached="true"> Wiedemann </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/heuschnupfenmittel-dhu" data-btattached="true"> Heuschnupfenmittel DHU </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/klimaktoplant-n" data-btattached="true"> Klimaktoplant N </a> </li> <li class="level02 link"> <a class="" href="https://shop.apotal.de/ankubero-osteoporose" data-btattached="true"> Ankubero Osteoporose </a> </li> <li class="level02 group"> <a class="" href="https://shop.apotal.de/ankubero-wechseljahre" data-btattached="true"> Ankubero Wechseljahre </a> </li> </ul></li>
So I basically want to follow the class level01 group opened current
but not if it's the one with Themen-Welt
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|