AI Web Research not pulling out URLs properly

[The_Weeblings]

Hi!

I built an automation to scrape the internet for news and to summarize said news into somewhat of an email newsletter for myself. I’m using the AI Web Research node to do the research and pull out articles with the URLs and a few other pieces of information. For some reason, it worked for a bit, but now every single time I run it, it’s able to pull out the articles (I can see it because it outputs a summary), but it is sometimes unable to provide the URL and just states “URL unavailable”. Is there something I’m doing wrong? It’s pretty frustrating as it causes my entire automation to error out. I’m aware I can add error shields, and I will, but on certain runs 4 out of the 5 articles it pulls out have no links, which means I’m unable to scrape the article and summarize it in my next steps. And I want to make sure that every single time I’m running this automation, it pulls out everything I need instead of just giving me 1 article because of this “URL unavailable” bug. Would appreciate some advice! =)

Here’s my workflow: https://www.gumloop.com/pipeline?workbook_id=tLAQXrJYDVvv6p8xEycCdt&run_id=WkseD7xvvFZG8RMTwzovaz

Thank you!

[Gumloop_Bot]

Hey @The_Weeblings! If you’re reporting an issue with a flow or an error in a run, please include the run link and make sure it’s shareable so we can take a look.

1. Find your run link on the history page. Format: https://www.gumloop.com/pipeline?run_id={{your_run_id}}&workbook_id={{workbook_id}}

2. Make it shareable by clicking “Share” → ‘Anyone with the link can view’ in the top-left corner of the flow screen.
   

3. Provide details about the issue—more context helps us troubleshoot faster.

You can find your run history here: https://www.gumloop.com/history

[The_Weeblings]

Just to add on to my previous post, the error shield doesn’t work because it still gives this error due to the way my workflow is set up. =/

[The_Weeblings]

Also just wanted to add a link to a run just in case you need it: https://www.gumloop.com/pipeline?workbook_id=tLAQXrJYDVvv6p8xEycCdt

[Wasay-Gumloop]

Hey Sara, sorry about the frustration here.

The way the AI web research node works is entirely dependent on the AI model’s research, so there’s no way to 100% control the output. Your best bet is to try adjusting the prompt. For example, in your current prompt you’ve included summaries and dates as outputs, even though you’re not using them — simplifying the prompt might help here.

Alternatively, you could try the Perplexity Web Search or the normal Web Search nodes. The normal Web Search is the most robust since it outputs links by default, but it’s less flexible than the AI Web Research or Perplexity nodes.

As for the list size error you’re seeing, that’s because you’re passing data from two different sources that don’t match, so the 1:1 correspondence between the inputs isn’t maintained. There’s a complete explanation for this here.

The fix would be to use a subflow and then wrap that in an error shield — here’s an example: https://www.gumloop.com/pipeline?workbook_id=duc4WFWctDzdifXoDkyeLi

Let me know if this makes sense and works for you