# Help Needed with Automating Pagination in Custom Node **URL:** \ **Category:** Get Help\ **Tags:** Website-Scraper\ **Created:** [February 22, 2025, 5:42pm UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364 "2025-02-22T17:42:18Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Manishdwivedi](https://avatars.discourse-cdn.com/v4/letter/m/779978/32.png) [@Manishdwivedi](https://forum.gumloop.com/u/Manishdwivedi)\ **Post date:** [February 22, 2025, 5:42pm UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364/1 "2025-02-22T17:42:18Z")
**Hello Gumloop Community,** I’m working on an **automation flow to scrape doctor profiles from a paginated listing page** on Gumloop, but I’m running into some issues with pagination in my **custom “Page Navigator” node**. **🔹 My Setup:** • I have a working **automation flow** that: 1. **Extracts all profile links** from a listing page. 2. **Scrapes each profile individually** and extracts details (name, specialty, address, phone, email). 3. **Saves the extracted data into Google Sheets.** • The problem is **the listing page URL does not change when navigating pages** (AJAX-based pagination). • I added a **“Page Navigator” custom node** to **click the “Next Page” button and navigate through all pages.** **🔍 Issues I’m Facing:** 1️⃣ **Pagination is not working correctly.** • My Page Navigator executes, but it **doesn’t move beyond Page 1** or **keeps extracting the same page.** 2️⃣ **Gumloop expects a list but receives a single URL input.** • The input node provides a **single listing page URL** , but the Page Navigator **expects a list**. 3️⃣ **Potential bot detection issues (reCAPTCHA appearing randomly).** • Sometimes, instead of loading new profiles, the page **returns a Google reCAPTCHA** in the extracted content. **🚀 What I’ve Tried So Far:** ✅ **Converting the input URL into a list** using a List Operations node. ✅ **Modifying the Page Navigator script** to: • Click **“Next Page”** only if visible. • Wait for new content to load before proceeding. • Stop pagination when no more pages exist. ✅ **Checking the output logs** to verify if new data loads after clicking “Next”. ✅ **Trying to detect AJAX-loaded content before moving to the next step.** **🛠 What I Need Help With** 1️⃣ **How can I ensure that my Page Navigator correctly moves through pages and loads new results?** 2️⃣ **What’s the best way to detect if a new page has loaded (instead of extracting the same data repeatedly)?** 3️⃣ **How can I prevent reCAPTCHA from blocking my automation?** 4️⃣ **Should I handle AJAX-based pagination differently in Gumloop?** Any insights, suggestions, or best practices would be **greatly appreciated!** 🚀 Thanks in advance for your help. Link to my workbook: [https://www.gumloop.com/pipeline?workbook\_id=daLprHVz33GRxZjzM67HJR&run\_id=9GLm4gh5eTR7NY4AYNJghJ](https://www.gumloop.com/pipeline?workbook_id=daLprHVz33GRxZjzM67HJR&run_id=9GLm4gh5eTR7NY4AYNJghJ) ---
**Author:** ![Donna](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.gumloop.com/donna/32/83_2.png) [@Donna](https://forum.gumloop.com/u/Donna)\ **Post date:** [February 23, 2025, 6:32am UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364/2 "2025-02-23T06:32:08Z")
This will do the job (I think) ![Split text Node](https://canada1.discourse-cdn.com/flex027/uploads/gumloop/original/1X/fc340b0775e9d12bea5a7394c6deefa0eaf535e3.jpeg) ---
**Author:** ![Manishdwivedi](https://avatars.discourse-cdn.com/v4/letter/m/779978/32.png) [@Manishdwivedi](https://forum.gumloop.com/u/Manishdwivedi)\ **Post date:** [February 24, 2025, 11:43am UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364/3 "2025-02-24T11:43:45Z")
> [@Donna](#): > > This will do the job (I think) The website i am trying to extract data from through pagination automation does not change URLs on new pages (it uses AJAX for pagination) unfortunately. Hence the issue. 😭 ---
**Author:** ![Wasay-Gumloop](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.gumloop.com/wasay-gumloop/32/69_2.png) [@Wasay-Gumloop](https://forum.gumloop.com/u/Wasay-Gumloop)\ **Post date:** [February 25, 2025, 12:11am UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364/4 "2025-02-25T00:11:28Z")
Hey @Manishdwivedi - This is going to be tricky. The custom node in your flow labelled “Page Navigator” - what is that doing exactly? If automating this end to end is not imperative you can try the Gumloop Chrome Extension: **Doc:** [https://docs.gumloop.com/nodes/browser\_extension/browser\_extension\_input](https://docs.gumloop.com/nodes/browser_extension/browser_extension_input) **Tutorial:** [https://www.loom.com/share/6b343be195ba4a55a66ce26894b303f9](https://www.loom.com/share/6b343be195ba4a55a66ce26894b303f9) The extension takes the screen you’re on as input and performs the set action on it (eg. scraping or screenshot), hence you can navigate through the pages and run the flow via the extension on each page. ---
**Author:** ![system](https://canada1.discourse-cdn.com/flex027/uploads/gumloop/original/1X/d02ff8efcde6ac5d3ac8aa0376137d5f804da549.png) [@system](https://forum.gumloop.com/u/system)\ **Post date:** [March 1, 2025, 12:11pm UTC](https://forum.gumloop.com/t/help-needed-with-automating-pagination-in-custom-node/1364/5 "2025-03-01T12:11:31Z")
This topic was automatically closed 4 days after the last reply. New replies are no longer allowed.