Custom Recipes & Batch Multi-URL Deep Scraping
Advanced two-level scraping guide: collect URLs from list pages and automatically extract in-depth specifications, descriptions, and contact info.
Often, search listing pages only display brief summaries, while full specifications, SKU variants, contact details, and rich-text descriptions reside on dedicated secondary product detail pages. BeeCai provides powerful two-level deep scraping and bulk URL concurrency.
Two-Level Deep Scraping Concept
In two-level scraping, BeeCai first extracts the "Detail Page Link (URL)" from each row of the list, then navigates to each individual detail page in the background to capture deep fields, consolidating everything into a single comprehensive spreadsheet.
Detail Page Extraction Steps
- When selecting fields on the list page, make sure to include the "Detail Link URL" column.
- Click "Configure Detail Page Scraping" in the BeeCai control panel.
- BeeCai opens a sample detail page for you to point and click desired attributes (e.g.
Technical Specs,Seller Email/Phone,High-Res Gallery). - Save the rule and click "Start Batch Deep Extraction" to process all list URLs automatically.
Bulk URL List Batch Mode
If you already possess a list of URLs (such as 100 e-commerce category pages or competitor store links):
- Select "Batch URL Scraping" mode in the BeeCai panel.
- Paste your URL list (one per line) or upload a CSV/Excel file containing URLs.
- Assign a scraping recipe and BeeCai will open and extract data from each link sequentially.
Rate Limits & Anti-Block Best Practices
To ensure smooth and uninterrupted batch runs, follow these recommended settings:
- Enable "Random Request Delay (2~4 seconds recommended)" to simulate human browsing habits.
- For members-only data requiring authentication, log in before launching the scraping task.
Batch Scraping FAQ
Q1: Will scraping thousands of URLs consume high memory?
BeeCai uses streaming garbage collection, releasing memory as pages finish. Even extracting tens of thousands of rows keeps your browser snappy.
Q2: How do I resume if a batch job is interrupted?
BeeCai logs progress automatically. When restarting, select "Resume from Breakpoint" to skip already completed URLs.
Was this article helpful?
您的反馈将帮助我们持续改进文档质量