Tip: Retrieving all Google search results in n8n

To retrieve all of the Google search results for a given query string, you need to set up a do/while loop within n8n. This approach expands my workflow’s Google Search to return more than 10 results per call. 🙂

Below is JSON that I run as a sub-workflow, it does six things:

1. Initialize with Input code node creates the necessary values for the workflow to run
   • CX string required by the Google Custom Search API.
   • The query to run – I pass this into the sub-workflow, but I set up a default value for testing/debugging purposes. In a non-personal environment, I wouldn’t do this. 😉
   • And it initializes various ‘loop variables’ (start, allResults, hasMoreResults, totalResults, currentPage, and inputData).
2. Google Search Request is an httpRequest node that executes the query using the following query parameters:
   • key passes your Google API key
     ToDo – I would like to store this in the n8n credentials, but I can’t figure out how yet. 🙁
   • The other required search parameters, including start, which tells Google which page of results to return.
3. Process Results code node examines the output from the httpRequest and updates all of the relevant ‘loop variables’ and concatenates the items returned from httpRequest to the allResults and updates relevant variables for the next run (if one is needed).
4. Should Continue? is an if node that checks hasMoreResults from the prior node to determine if we need to retrieve more results, or whether we are done.
5. If we are not done, and Should Continue? returns true, we hit a pause node (Rate Limit Wait) to not overwhelm the API, and then we go to the Google Search Request node to request more results using the updated variables set in the Process Results node.
6. Once we are done, Prepare Output formats the JSON that we will return back to the calling workflow

In the parent workflow, this is what it looks like to call this workflow:

1. Set the search parameters - This is optional; I do this as a separate step because I use the same search string for multiple site searches.
2. I call the Google Search subworkflow and pass the domain name in the siteSearch input field.
3. Process the search results, extracting relevant summary values that I use later. This node is optional.
4. Split out the single result set into a collection of results that I can use in merge/combine nodes later

This approach works for me because I tightly scope my search string and the domain. As you experiment with it, consider capping the total number of results or pages that you retrieve from Google.

I hope that this helps you do more with your n8n workflows. If you have questions on this approach or suggestions on how to do it better, let me know!

Side Note: This is the first post to use a WordPress plugin I built to display n8n workflow JSON code visually using the n8n-demo-component project - I hope you enjoy it!