I am new and while an engineer at heart, I am NOT a coder so please be kind if the answers to these questions are obvious to everyone but me…lol. I have watched the videos and even built a simple flow but I am stuck on a couple of essential tasks.
BACKGROUND: I’m building a web searching and scraping flow to identify specific events/event details where particular keynote speakers have presented in the past. I have a google sheet with the speaker name and their website. I use a formula to create a specific google search query (excluding certain sites, for example). The goal is to have Gumloop use that search query to find event websites, scrape them, categorize them, extract data from them and put that data back into my Google Sheet. Here is more detailed flow explanation.
Where I am stuck:
-
How do I reuse/access outputs from previous steps in the flow? For example, after I categorize the websites, I want to take ONLY those in certain categories and continue to process them. I have done a filter but I need the website content to keep processing. Same thing for Speaker Name and URL - these are all known/used/outputed from previous steps but not the most recent step. Feel like I am missing something obvious here.
-
How do I get it to process more than 10 websites? I want it to ultimately run through 100s of websites for each speaker/query. How do we make that kind of magic happen?
Thanks for any assistance or guidance!
For reference, here is my workflow now. (I haven’t connected the Google Sheet at the start as I am still trying to get a single flow working).
https://www.gumloop.com/pipeline?workbook_id=7KD1wv6GsrUkAJsLA6h3kL&run_id=nWoXWHf6oR6G3wu9zjukXt