Web research (search, read and crawl)

Description

Five nodes so a workflow, or an AI agent, can research the internet:

Node	What it does
webSearch — Web search	Searches Google and other engines through an API and returns, for each result, title, link, domain and snippet.
webReadPage — Read web page	Reads a public page and returns its title, description, text and visible contact details (emails, phones, social links and contact pages).
mapsSearch — Google Maps: search businesses	Searches Google Maps businesses through an API and returns name, address, phone, website, rating, category and coordinates.
webMetadata — Page metadata	Reads a page’s JSON-LD, Open Graph and microdata and returns a normalised product card (price, brand, SKU, GTIN, availability…). Built for online stores.
webCrawl — Crawl website (spider)	Takes a site URL and returns the internal URLs it finds, so you can analyse them afterwards with Read web page or an agent.

Place them anywhere in the flow or connect them to an AI Agent node as tools (tool type node): the agent decides what to search and which pages to open. The “AI agent prospecting” templates in the Library use them.

What they do not do

They do not scrape Google or LinkedIn. Search goes through a provider’s API and LinkedIn, Facebook, Instagram, X and TikTok are not read (they require a login or forbid automated access). To search LinkedIn, use the search engine with site:linkedin.com/company, which only works with the public snippet the engine shows.

Credential

Create a Web search credential and choose a provider:

Provider	What it returns	Notes
Serper.dev	Google results	Web and news. Limited free tier.
SerpApi	Google results	Web and news.
Brave Search	Brave’s own index	Web and news.
Tavily	Own index, built for AI	Web and news.
Google Programmable Search	Google, over the sites you configure	Also needs the search engine ID (cx); at most 10 results per call; does not tell news apart.

For the Google Maps node, Google Maps (Places API) also works: a key from your Google Cloud project with Places API (New) enabled (it only works for that node, not for web search). Serper and SerpApi work for both.

The Test button runs a one-result search and uses one query of your plan. The key is never shown in logs.

webSearch

Parameter	Required	Description
credentials_id	Yes	Web search credential.
query	Yes	What you would type in Google. Supports "exact phrase" and -word.
site	No	One or more comma-separated domains; adds site: to the query (several are joined with OR).
num	No	Results, 1 to 20 (default 10).
country / language	No	2-letter codes (es, mx, ar, cl, us…). country biases results to a country. One per search: for several countries (for example Latin America) run one search per country.
search_type	No	web or news.
time_range	No	Maximum age: day, week, month or year.
exclude_domains	No	Comma-separated domains to drop.
page	No	Results page, 1 to 10.
output_key	No	Where the result is stored (default search).

Output in data.search, keeping the rest of data:

{
  "search": {
    "provider": "serper",
    "query": "consulting Madrid site:linkedin.com/company",
    "count": 2,
    "results": [
      { "position": 1, "title": "Alfa Consulting | LinkedIn", "url": "https://es.linkedin.com/company/alfa", "domain": "linkedin.com", "snippet": "Consulting · Madrid · 51-200 employees", "date": null }
    ]
  }
}

It removes duplicates and results from excluded domains. On a provider error it retries up to twice (rate limit, 5xx errors, network).

webReadPage

Parameter	Required	Description
url	Yes	Public http or https page.
max_chars	No	Maximum text characters, 500 to 30000 (default 6000). Less text uses fewer AI tokens.
extract_contacts	No	Extracts emails, phones, social links and contact pages (on).
respect_robots	No	Respects robots.txt (on, recommended).
blocked_domains	No	Domains that are not read. By default LinkedIn, Facebook, Instagram, X and TikTok.
timeout_ms	No	Request timeout.
output_key	No	Where the result is stored (default page).

Output in data.page: title, description, site_name, lang, text, truncated, final_url, domain and contacts (emails, phones, social, contact_pages).

Security

Since the destination may be decided by an AI agent, the node protects itself:

• It only reads http and https on normal ports; it does not read localhost, private IPs or cloud metadata, and checks every redirect and every connection.
• At most 2 MB per page and 4 redirects. Web pages only (no PDFs or files).
• It identifies itself as FloogosBot and respects robots.txt.

mapsSearch

Searches Google Maps businesses through an API (it never scrapes Maps) and returns the data already normalised, whichever the provider.

Parameter	Required	Description
credentials_id	Yes	Web search credential with provider Google Maps (Places API), Serper or SerpApi.
query	Yes	What to search: dental clinic, hardware store, vegan restaurant…
location	No	Where: in Valencia, Spain. Added to the search.
country / language	No	2-letter codes (es, ar, cl, mx…). One country per search: for several countries, one search per country.
lat / lng / radius_m	No	Limits to an area with a point and a radius (100 to 50000 m).
max_results	No	1 to 60 (default 20). Every 20 is one more call to your quota.
detail	No	contacto (phone, website and rating; with Google Places it uses its highest tier) or basico (name, address and location).
min_rating / min_reviews	No	Filters by rating and minimum number of reviews.
require_website	No	Only businesses with a website.
exclude_closed	No	Excludes permanently closed ones (on).
output_key	No	Where the result is stored (default maps).

Output in data.maps, keeping the rest of data:

{
  "maps": {
    "provider": "google_places", "query": "dental clinic in Valencia", "count": 2, "pages": 1,
    "filtered": { "valoracion": 0, "opiniones": 0, "sin_web": 1, "cerrados": 0, "duplicados": 0 },
    "places": [
      { "position": 1, "place_id": "ChIJ…", "name": "Sonrisa Clinic", "address": "Calle Mayor 1, Valencia", "phone": "963 123 456",
        "website": "https://www.sonrisa.es", "domain": "sonrisa.es", "rating": 4.7, "reviews": 210, "category": "Dental clinic",
        "categories": ["dentist"], "lat": 39.47, "lng": -0.37, "maps_url": "https://maps.google.com/?cid=…", "status": "OPERATIONAL", "source": "google_places" }
    ]
  }
}

It does not return review texts. With domain you can chain Read web page or Page metadata directly to enrich each business.

Google Maps terms

Google Maps content has usage restrictions: the Google Maps Platform terms limit storing and reusing names, addresses or reviews outside the service (only the place identifier, place_id, can be kept without limit). Use it to look things up on the spot, or store only the place_id and query again when you need the data. Building a business list for marketing with it may conflict with those terms: read them first. If the result includes data of sole traders or professionals, the data-protection and commercial-communications rules of your country apply.

webMetadata

Reads the metadata a page publishes for machines —schema.org JSON-LD, Open Graph, Twitter Cards and microdata— and returns it in order. It is what Google or WhatsApp read to show a product, so it is usually the most reliable and lightest way to get data from an online store. It does not run JavaScript: if the store paints the data in the browser, it will not appear.

Parameter	Required	Description
url	Yes	A product page, a category or any public page.
urls	No	List of several pages (up to 30): one per line, comma-separated or a JSON list. If filled in, url is ignored. Read one by one, with a pause, and a failure does not stop the batch.
delay_ms / max_seconds	No	Only with urls: pause between pages, 500 to 10000 ms (1000), and batch time limit, 5 to 240 s (120).
include_json_ld	No	Also returns the raw JSON-LD (on; off in a batch).
max_json_ld_chars	No	Cap on the raw JSON-LD, 1000 to 100000 (20000). The normalised card does not depend on it.
respect_robots / blocked_domains / timeout_ms	No	As in Read web page.
output_key	No	Where the result is stored (default metadata).

Output in data.metadata: title, description, canonical, lang, platform (Shopify, WooCommerce, PrestaShop, Magento, VTEX, Tiendanube, BigCommerce, Wix…), types (the @types found), is_product_page, breadcrumbs, organization, open_graph, twitter, hreflang, json_ld and, the most useful part, product:

{
  "metadata": {
    "platform": "shopify", "is_product_page": true, "breadcrumbs": ["Home", "Sneakers"],
    "product": {
      "name": "Air Sneakers", "brand": "Brand X", "sku": "ZA-001", "gtin": "8412345678905", "description": "Light and comfortable",
      "image": ["https://cdn.shopify.com/a.jpg"], "price": 79.9, "currency": "EUR", "availability": "InStock", "in_stock": true,
      "rating": 4.6, "rating_count": 128, "variants": null, "source": "json-ld",
      "offers": [ { "price": 89.9, "currency": "EUR", "availability": "InStock", "in_stock": true } ]
    }
  }
}

With a URL list the result is data.metadata = { count, ok, failed, with_product, items: [...] }: each item carries its metadata (with ok: true) or ok: false and an error with type and message. It is the basis of the price tracking template.

product.source tells where the data comes from (json-ld, microdata or open-graph, in that order of preference). The price is the lowest of the offers and in_stock is true if any offer is in stock. For many products, combine it with Crawl website (sitemap) and an iterator, with a pause between pages.

webCrawl

Given a site URL, it returns the internal URLs (same site) so you can decide which ones to analyse. It is a polite, limited crawler: public pages only, one request at a time, with a pause.

Parameter	Required	Description
url	Yes	Home page or section to start from.
mode	No	auto (default: uses the sitemap if there is one, otherwise follows links), sitemap or links.
max_pages	No	Maximum URLs returned, 1 to 500 (default 50).
max_depth	No	Clicks from the start page, 0 to 5 (default 2). Only when following links.
include_subdomains	No	Also includes blog.company.com besides company.com (off). www counts as the same site.
exclude_paths / include_paths	No	Texts paths must (or must not) contain. By default login, cart, account, internal search and feeds are avoided.
keep_query	No	Keeps ?a=b parameters (off: they are removed, which avoids duplicates and personal data in URLs). Tracking ones (utm_, fbclid…) are always removed.
fetch_titles	No	With a sitemap, downloads each URL to get its title and status (off).
delay_ms	No	Pause between requests, 500 to 10000 ms (1000). If the site asks for a longer Crawl-delay, that is used.
max_seconds	No	Maximum time, 5 to 240 s (60). When reached, it returns what it found.
respect_robots	No	Respects robots.txt (on, recommended).
blocked_domains	No	Domains that are not crawled (by default, social networks).
output_key	No	Where the result is stored (default crawl).

Output in data.crawl, keeping the rest of data:

{
  "crawl": {
    "host": "company.com", "mode": "auto", "used_sitemap": true, "count": 48, "pages_fetched": 0,
    "stopped_reason": "completado", "truncated": false, "pending": 0,
    "sections": { "products": 30, "blog": 12, "(inicio)": 1 },
    "urls": [ { "url": "https://company.com/products/a", "depth": null, "status": null, "title": null, "source": "sitemap", "lastmod": "2026-10-01" } ],
    "documents": [ "https://company.com/catalog.pdf" ],
    "external_domains": { "other.com": 2 },
    "skipped": { "robots": 1, "externas": 3, "excluidas": 2, "duplicadas": 5, "errores": 0, "sin_seguir": 0 }
  }
}

stopped_reason tells why it ended: completado (finished), limite_de_paginas (page limit), tiempo (time limit), limite_del_sitio (the site answered 429) or acceso_denegado (403 at the start). If it is not completado, the result is partial.

How it behaves (and why)

• Respects robots.txt (disallowed paths and Crawl-delay), rel="nofollow", <meta name="robots" content="nofollow"> and X-Robots-Tag. If the site forbids crawling the start path, it does not crawl.
• Never leaves the site or follows private destinations; it does not log in, fill in forms or solve captchas.
• Does not crawl LinkedIn, Facebook, Instagram, X or TikTok, nor any domain Floogos has blocked at its owners’ request.
• Does not store page content: only URLs and titles.
• It identifies itself as FloogosBot.

Examples in the Library

• Weekly audit of your website (sitio-mapa-y-auditoria-de-urls-con-informe): crawls your site, checks the status and title of each URL and emails you a report with the broken, untitled and duplicate-title pages.
• Watch a competitor’s website for news (sitio-novedades-de-la-web-de-un-competidor-con-ia): crawls its sitemap, detects the new pages and an AI agent reads them, classifies them and scores their relevance.
• Daily price tracking of your suppliers (sitio-seguimiento-de-precios-de-productos): reads the price and stock of up to 30 products with Page metadata (URL list), stores the history and alerts on drops, rises, target price and back-in-stock.
• AI agent prospecting (six templates): an agent searches companies with Web search and reads them with Read web page.

A typical pattern

1. webCrawl on a company website (mode: sitemap).
2. A node that keeps the URLs under /blog or /products (include_paths).
3. webReadPage on each one (with an iterator) and an AI node that analyses them.

Use it responsibly

A page being public does not mean every use is free. Before crawling someone else’s site, check its terms of use, do not do it against its wishes (robots.txt, blocks, notices) and do not mass-reuse its content or databases. If the result includes people’s data, you need a legal basis and must comply with the data-protection rules of your country. If you own a site and do not want it crawled, you can block FloogosBot in your robots.txt or write to us.

Errors

The node does not continue and returns error.type: auth_error (wrong key), quota_exceeded, rate_limited, timeout, validation, domain_blocked, robots_disallowed, blocked_destination, not_found, forbidden, unsupported_type, host_not_found or connection_error. Inside an agent, the error is returned to it as the tool result and it can try something else.

As an agent tool

In the Agent tool node choose type node, set webSearch or webReadPage as Node type, the credential in Node credential and, in Base config, whatever you want fixed (for example {"site":"linkedin.com/company","country":"es","num":8}). What the agent decides (query, url) is added to that configuration. See the Library templates for complete examples.

Privacy and responsible use

Queries leave Floogos towards the search provider you choose and pages are read from the Floogos server. Use only professional, public company data, review the results before reaching out and follow the data-protection and commercial-communications rules of your country.