The lil’bots platform comes with Serp API access built in. You can use the Serp API to build bots that can scrape search engine results pages (SERPs) like Google and extract structured data from them. Perfect for letting your bots search the web for information.

​
Using the Serp API

The main endpoint for the Serp API is /search. You can make requests to this endpoint to scrape search engine results pages (SERPs) like Google. The following parameters are supported:

  • q (required) - the search query to scrape the SERP for. You can use anything that you would use in a regular Google search. e.g. inurl:, site:, intitle:. We also support advanced search query parameters such as as_dt and as_eq. See the full list of supported advanced search query parameters.
  • location (optional) - Parameter defines from where you want the search to originate. If several locations match the location requested, we’ll pick the most popular one. If location is omitted, the search may take on the location of the proxy.
  • google_domain - The Google domain to use for the search. e.g. google.com, google.co.uk, google.fr, etc.
  • device - Parameter defines the device to use to get the results. It can be set to desktop (default) to use a regular browser, tablet to use a tablet browser (currently using iPads), or mobile to use a mobile browser (currently using iPhones).
  • output - Parameter defines the final output you want. It can be set to json (default) to get a structured JSON of the results, or html to get the raw html retrieved.

This is not an exhaustive list of parameters. For a full list of supported parameters, see the Serp API documentation.

Here’s an example of the JSON returned from the API:

{
    "search_metadata": {
        "id": "66ea5a927e1d1fa1136136ac",
        "status": "Success",
        "json_endpoint": "https://serpapi.com/searches/8c3bc959c61aebb2/66ea5a927e1d1fa1136136ac.json",
        "created_at": "2024-09-18 04:44:02 UTC",
        "processed_at": "2024-09-18 04:44:02 UTC",
        "google_url": "https://www.google.com/search?q=Lil%27bots&oq=Lil%27bots&sourceid=chrome&ie=UTF-8",
        "raw_html_file": "https://serpapi.com/searches/8c3bc959c61aebb2/66ea5a927e1d1fa1136136ac.html",
        "total_time_taken": 2.25
    },
    "search_parameters": {
        "engine": "google",
        "q": "Lil'bots",
        "google_domain": "google.com",
        "device": "desktop"
    },
    "search_information": {
        "query_displayed": "Lil'bots",
        "total_results": 95300000,
        "time_taken_displayed": 0.34,
        "organic_results_state": "Results for exact spelling"
    },
    "organic_results": [
        {
            "position": 1,
            "title": "lil'bots",
            "link": "https://www.lilbots.io/",
            "redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.lilbots.io/&ved=2ahUKEwjLq-Sk18uIAxWlsoQIHXl6JG0QFnoECAoQAQ",
            "displayed_link": "https://www.lilbots.io",
            "favicon": "https://serpapi.com/searches/66ea5a927e1d1fa1136136ac/images/34d2c3078f353470fdc9ca9fcc5432f3352846c60b152babd7aa6586175abe2b.png",
            "snippet": "Your AI Pair Programmer. Why do all the hard work yourself? Lil'bots AI buddy can fix bugs, write functions, autocomplete your code or even craft entire bots ...",
            "snippet_highlighted_words": [
                "Lil",
                "bots"
            ],
            "source": "lil'bots"
        },
        {
            "position": 2,
            "title": "Software Development Collection",
            "link": "https://www.lilbots.io/collection/software-development",
            "redirect_link": "https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.lilbots.io/collection/software-development&ved=2ahUKEwjLq-Sk18uIAxWlsoQIHXl6JG0QFnoECA4QAQ",
            "displayed_link": "https://www.lilbots.io › collection › software-development",
            "favicon": "https://serpapi.com/searches/66ea5a927e1d1fa1136136ac/images/34d2c3078f353470fdc9ca9fcc5432f3d6be4bcfc1ccbe4d1153b2f1cfa5e8fa.png",
            "date": "Jan 16, 2024",
            "snippet": "These AI bots are build to help developers and software engineers in building new software. Use this bots to accelerate your coding and ...",
            "snippet_highlighted_words": [
                "bots",
                "bots"
            ],
            "source": "lil'bots"
        }
    ]
}

Here’s an example of a bot that uses the Serp API to scrape Google search results, using fetch to make the request:

export async function main(inputs, params) {
    const { query } = inputs;

    const response = await fetch(`https://serpapi.com/search?q=${query}`);
    const data = await response.json();

    return data.organic_results.map(result => ({
        title: result.title,
        markdown: `**${result.title}**\n\n${result.snippet}`
    }));
}
Firecrawl APIUser Accounts