可以試試看自動化測試軟體 selenium 和 PhantomJS 來模擬瀏覽器瀏覽,取得網頁呈現的真實模樣。
- 先安裝 python selenium 套件
- 下載並解壓縮 PhantomJS 軟體,檔案路徑等下python程式碼中會用上
sudo pip install selenium
使用selenium 和 PhantomJS的程式碼範例:
from selenium import webdriver driver = webdriver.PhantomJS(executable_path='/您的PhantomJS目錄/bin/phantomjs') # 以PChome購物搜尋 macbook 為例 driver.get('http://ecshweb.pchome.com.tw/search/v3.3/?q=macbook') pageSource = driver.page_source print(pageSource) driver.close()
接下來整合 BeautifulSoup 的程式碼範例:
import requests from bs4 import BeautifulSoup from selenium import webdriver url = 'http://ecshweb.pchome.com.tw/search/v3.3/?q=macbook' driver = webdriver.PhantomJS(executable_path='/您的PhantomJS目錄/bin/phantomjs') driver.get(url) pageSource = driver.page_source #print(pageSource) soup = BeautifulSoup(pageSource, "lxml") item_count = 1 for item in soup.select('img'): #print(item) print('['+str(item_count)+']') print(item['title']) print(item['src']) item_count += 1 driver.close()
執行結果:
[1] MacBook Air 13 吋:256GB(Z0TB0001U) http://a.ecimg.tw/pic/v1/data/item/201606/D/G/A/X/5/Y/sDGAX5Y-A9007812U000_5752453b19aff.jpg [2] MacBook Air 13 吋:128GB(MMGF2TA/A) http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/0/7/sDGAX07-A90077H7W000_5733ebfc0a883.jpg [3] MacBook Pro 13 吋:2.7GHz 256GB (MF840TA/A) http://a.ecimg.tw/pic/v1/data/item/201511/D/G/A/X/9/6/sDGAX96-A9006MBQR000_563c649e0b12b.jpg [4] MacBook Pro 13 吋:2.7GHz 128GB (MF839TA/A)-0160830 http://a.ecimg.tw/pic/v1/data/item/201608/D/G/A/X/7/1/sDGAX71-A9007H2EZ000_57c53b72dab2e.jpg [5] MacBook Pro 13 吋:2.7GHz 128GB (MF839TA/A) http://a.ecimg.tw/pic/v1/data/item/201601/D/G/A/X/7/H/sDGAX7H-19006S5WQ000_5695c64a6dfdf.jpg [6] MacBook Air 13 吋:256GB(MMGG2TA/A) http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/0/7/sDGAX07-A90077H97000_5733ec30bc6f2.jpg [7] MacBook Air 13 吋:128GB(MMGF2TA/A) http://a.ecimg.tw/pic/v1/data/item/201609/D/G/A/X/0/7/sDGAX07-19007IBWT000_57d8cf53b058b.jpg [8] MacBook Air 13 吋:256GB(MMGG2TA/A) http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/0/7/sDGAX07-A900799NW000_5742d25e860a6.jpg [9] MacBook 12 吋 256GB 玫瑰金 (MMGL2TA/A) http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/4/X/sDGAX4X-A9007904G000_573d338a41048.jpg [10] MacBook Pro 13 吋:2.7GHz 128GB (MF839TA/A) http://a.ecimg.tw/pic/v1/data/item/201609/D/G/A/X/7/H/sDGAX7H-19007HCAE000_57c9111d721f8.jpg [11] MacBook Air 13 吋:128GB(MJVE2TA/A)-0160505 http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/7/1/sDGAX71-A90078DVN000_5731b259475be.jpg [12] MacBook Air 13 吋:256GB (Z0TB0001U) http://a.ecimg.tw/pic/v1/data/item/201606/D/G/A/X/5/Y/sDGAX5Y-A9007AD55000_5757bc01e00b8.jpg [13] MacBook Pro 13 吋:2.7GHz 256GB (MF840TA/A) http://a.ecimg.tw/pic/v1/data/item/201609/D/G/A/X/7/H/sDGAX7H-19007HCA4000_57c911687f3a0.jpg [14] MacBook Pro 13 吋:2.5 GHz 500GB (MD101TA/A) http://a.ecimg.tw/pic/v1/data/item/201602/D/G/A/X/3/N/sDGAX3N-19006UUHB000_56cd1e246ef7f.jpg [15] MacBook 12 吋 512GB 太空灰(MJY42TA/A)-0160830 http://a.ecimg.tw/pic/v1/data/item/201608/D/G/A/X/7/1/sDGAX71-A9007H2M9000_57c541fe92913.jpg [16] MacBook Air 13 吋:128GB(Z0TA0001B)-01600830 http://a.ecimg.tw/pic/v1/data/item/201608/D/G/A/X/7/1/sDGAX71-A9007H1FX000_57c5056a09d5c.jpg [17] MacBook Air 13 吋:256GB(Z0TB0001U) http://a.ecimg.tw/pic/v1/data/item/201606/D/G/A/X/5/Y/sDGAX5Y-A90078141000_5752441944474.jpg [18] MacBook Pro 13 吋:2.5 GHz 500GB (MD101TA/A) http://a.ecimg.tw/pic/v1/data/item/201409/D/G/A/X/3/N/sDGAX3N-19005FWJW000_541a90b4ee17a.jpg [19] MacBook Pro 15 吋: 2.8GHz 512GB 配備Retina顯示器(P/N.Z0RG0010V) http://a.ecimg.tw/pic/v1/data/item/201605/D/G/A/X/4/Y/sDGAX4Y-A90077V5U000_572869a2c5313.jpg [20] MacBook Air 11 吋:128GB(MJVM2TA/A) http://a.ecimg.tw/pic/v1/data/item/201504/D/G/A/X/0/7/sDGAX07-A90060KEB000_552b96e29b665.jpg
延伸閱讀:
#