色婷婷狠狠18禁久久YY,CHINESE性内射高清国产,国产女人18毛片水真多1,国产AV在线观看

flex框中項目的Scrapy xpath或CSS

CSS
老白2年前8瀏覽0評論

完成Scrapy noob，在成功完成教程后，我嘗試在網(wǎng)站上抓取一個頁面，我正在收集數(shù)據(jù)以供進(jìn)一步分析，但我嘗試使用的xpath不斷返回任何內(nèi)容。我能告訴你的唯一不同是文本/URL在頁面上的flex框里面。我已經(jīng)嘗試了元素搜索欄中所有有效的迭代，但一無所獲。我是否缺少一個功能來允許訪問flexbox中的這些線路？

我試圖抓取的網(wǎng)頁的網(wǎng)址:https://partsmasterusa.com/product-category/crown/page/2/

我在scrapy shell中嘗試過的xpaths示例沒有任何結(jié)果:

response.xpath('//div[@id="main"]/div/div/div/div/main/div[@class="archive-products"]//div[@class="product-content"]/a/text()').get()

response.xpath('//div[@id="main"]/div/div/div/div/main/div[@class="archive-products"]//div[@class="product-content"]/a/text()').extract_first()

response.xpath('//li[contains(@class, "product-col")]//a[@href]/text()').extract_first()
etc etc

一個xpath響應(yīng)的例子，它返回了我所期望的結(jié)果:

response.xpath('//div//a/span/text()').extract_first()

TIA

###這個網(wǎng)站有點棘手。這是一個wordpress網(wǎng)站，所以它不是完全動態(tài)的，但是每個頁面上呈現(xiàn)的產(chǎn)品都是通過ajax調(diào)用加載的。因此，當(dāng)你試圖從網(wǎng)站中提取內(nèi)容時，它可能并不存在。

你可以做的是復(fù)制頁面觸發(fā)的POST請求，以便通過查看瀏覽器開發(fā)工具的network選項卡將所有產(chǎn)品加載到頁面上。

調(diào)查后，您會發(fā)現(xiàn)它向https://parts master USA . com/product-category/crown/page/{ page _ number _ here }/發(fā)出請求。計數(shù)=36。通過復(fù)制這個url、請求標(biāo)題、請求正文中發(fā)送的字段和值，您可以通過這種方式獲得所有產(chǎn)品及其信息。

例如:

import scrapy
import json

class PartSpider(scrapy.Spider):
    name = "partsmaster"

    def start_requests(self):
        url = "https://partsmasterusa.com/product-category/crown/page/{}/?count=36"
        body = {"portoajax": True, "load_posts_only": True}
        headers = {"X-Requested-With": "XMLHttpRequest"}
        for i in range(1,542):
            yield scrapy.Request(url.format(i), method="POST", body=json.dumps(body), headers=headers)

    def parse(self, response):
        for title in response.xpath("//a[@class='product-loop-title']/h3/text()"):
            yield {"title": title.get()}

部分輸出:

2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Insulator (093603)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'KEY RING  (107763)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Key Switch  (146289)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Key Switch Assembly (146286)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Kit Label EEC (126271-(1))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL (69395)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Contactor Component Map(869419-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Control Component Map(869421-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL KIT  (126270)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Power Component Map(869420-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Pump Motor Map(869422-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Drive Module Map(69444)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Drive Module Map(869444-(1))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Label Traction Motor Map(869423-(2))'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-CONTACTOR CONTROL MAP (69419)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-CONTROL COMPONENT MAP (69421)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-POWER COMPONENT MAP (69420)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-PUMP MOTOR MAP (69422)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LABEL-TRACTION MOTOR MAP (69423)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Load Wheel  (077086-201)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Load Wheel Assembly Includes Bearings  (093656-201)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Locknut (060043–008)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LOCKWASHER  (060005-003)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Lockwasher Without Quick Coast  (060005-045)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'LT SILVER GRADE ANTI-SEIZE  (065005-003)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Manual Coast Selector Warning  (069100)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MODULE AC4820 FIN RR5200 (142885-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MODULE VCM RR5200 AC SERVICE  (129325-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Module Warning Label(69376)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Momentum RR Decal (069372-001.)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'MOTOR – HYDRAULIC  (121659)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Motor Nameplate (021062-008)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Mount Wheel  (084009)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Mount Wheel  (115388)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Moving Contact  (114435)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/4/?count=36>
{'title': 'Moving Vehicle Warning  (069004)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '#4 REG SPLIT LCW  (060005-049)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '1/4 INT LCW  (060005-022)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': '350 BLUE HSNG  (078723-006)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'A.C.-TRACTION DRIVE MODULE  (130056)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'AC RR52 MOTOR 36V TRACTION (21067)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'AC RR52 MOTOR 36V TRACTION (21187)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 1 Display  (146688)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 1 MODULE NEW- (140131)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 2 MODULE NEW- (142517-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 3 Label(69375)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 4 MODULE NEW- (141779-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS 5 MODULE NEW- (143911-001-0S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ACCESS MODULE 2 (121611-00S)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ArmInner Primary (12.2924-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'ASM HEATER RR5000 24V  (129132-001)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Axle  (080191)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Axle  (116804)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130528)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130529)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130530)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130531)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BAR BUS  (130532)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'Bearing (.065081-045)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING – BALL SENSOR  (130692)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING SLEEVE  (130701)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BEARING- (065081-043)'}
2023-06-09 15:41:55 [scrapy.core.scraper] DEBUG: Scraped from <200 https://partsmasterusa.com/product-category/crown/page/1/?count=36>
{'title': 'BLOCK – TERMINAL (21053)'}