Scraping pagination links

Get Help
1

I’m scraping a website where the pagination links are time based and you won’t know the link till you scrape the site. (a while loop)

So I’m starting with the first link, extracting the next page link on the home page, and then I have a subflow that takes loads the next page link, scrapes it, gets the next link, checks if the date in the link is newer than 7 days (I only want to scrape back 7 days), and returns a boolean if it’s newer and the next url.

But how can I loop on this? Gummie help told me to use an if-else and loop the subflow back on itself.

But then when it hits the if-else it fails. I get " Automation Failed!
Could not find next node to run. Please double check the node connections."

I’ve connected the if back to the subflow, and added the if pagination_url to the subflows url input. (it has two). But nothings seems to be working.

Here’s my shared workbook. https://www.gumloop.com/pipeline?workbook_id=xir9wbPRz5rqMgJwe8sn3S&run_id=j3FzhvqD7ze4xNjWrEgLV5

Thanks!

2

Hey @OrionSeven! If you’re reporting an issue with a flow or an error in a run, please include the run link and make sure it’s shareable so we can take a look.

  1. Find your run link on the history page. Format: https://www.gumloop.com/pipeline?run_id={{your_run_id}}&workbook_id={{workbook_id}}

  2. Make it shareable by clicking “Share” → ‘Anyone with the link can view’ in the top-left corner of the flow screen.
    GIF guide

  3. Provide details about the issue—more context helps us troubleshoot faster.

You can find your run history here: https://www.gumloop.com/history

3

Hey @OrionSeven – Just to clarify, what you’re trying to set up would be a circular reference, where it loops through the same subflow – is that right? From what I can see, you’re looking to create a setup that references itself. At the moment, circular references aren’t possible in Gumloop. The main reason is that a circular reference could potentially run infinitely, which makes it difficult to control or stop, and it can get pretty confusing quickly.

Since the Hacker News website has a specific pagination URL format, each time you click the next page button, it basically adds results to the start page, and there are some initial links as well. What you can do is copy this format and have Gummie or AI write up a script that can output a list of paginated URLs for Hacker News, you can use this through the Run Code node. For example, if you want 2 paginated URLs or 3 paginated URLs, it outputs a list that you can simply loop over with your website scraper node. Would something like that work for you?

4

It’s thehackernews.com not the YC site. And their links are not built out in the same manner thus the need for the loop.

I’ll try something else. Thanks for the response.

5

No problem! I was actually referring to thehackernews.com as well. If you click on Next Page it starts to add query parameters in the URL, example: https://thehackernews.com/search?updated-max=2025-06-26T14:15:00%2B05:30&max-results=12&start=48&by-date=false

The idea was to use this format in a Run Code node and have it output a List of paginated URLs that you can loop through the Website Scraper node. Essentially, only max-results=12&start=48 are changing in the URL along with the date.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.