Crawling outside the domain

Get Help
1

Screenshot 2025-02-28 at 9.37.48 AM
Screenshot 2025-02-28 at 9.37.48 AM583×447 29.2 KB

Even though I made sure the node is only run for same domain,
the Run log showed it is crawling other URLs, and it was taking too long even for a single depth.

3

Hey @shinobi - Can you share the run link from the https://www.gumloop.com/history page please? Please also set the share access to ‘anyone with the link can view’ under the share button.

image
image3840×1984 272 KB

4

As for speed, the Website Crawler is more in depth hence slower but the Web Agent Scraper with the action Get all URLs is faster. It outputs the file URLs separated by a comma so you can use a Split Text node to get a list of all the URLs.

Here’s an eg: https://www.gumloop.com/pipeline?workbook_id=qwHFcSusCrk7QMZNgotwND&run_id=QwoDMWNhefEKzhJmQGEkbj

5

Thanks Wasay, actually I got confused by the “Use only Same domain” flag.
It was actually turned off. I turned it on and it was working as expected.

6

https://www.gumloop.com/pipeline?workbook_id=2F8UxyKQyMnLAbkmHNgRjp

since you asked, here is my workbook link anyway.

7

Awesome, glad you were able to solve it!

8

Would recommend looking into subflows + error shield to make this flow more robust: https://docs.gumloop.com/core-concepts/subflows