Firecrawl API

The lil’bots platform comes with Firecrawl access built in. You can use the Firecrawl API to build bots that can generate human-like speech from text.

Firecrawl is the easiest way to extract structured data and text from web pages. If you need to extract text from any webpage to feed into an AI model, Firecrawl is the perfect tool for the job.

​

Using the Firecrawl API

You can use the following endpoints in the Firecrawl API:

• /v0/scrape - scrape a webpage and extract text docs
• /v0/search - searches the web based on a given keyword docs

To read the full Firecrawl API documentation, visit the Firecrawl API Reference.

​

Using the Firecrawl API - JavaScript Runtime

Here’s a simple example of a bot that uses the Firecrawl API to scrape a webpage and extract text:

export async function main(inputs, params, accounts) {
	const { url } = inputs;

	let res = await fetch('https://api.firecrawl.dev/v0/scrape', {
		method: 'POST',
		headers: {
			'Content-Type': 'application/json',
		},
		body: JSON.stringify({
			url,
			"pageOptions": {
				"onlyMainContent": true,
				"includeHtml": false
			},
			"timeout": 30000
		})
	})
	res = await res.json()
	const markdown = res.data.markdown

	return {
		title: 'Content',
		markdown
	}
}