\n \n \n\n\n\n"
+},
+"executeOnce": true,
+"typeVersion": 1.2
+},
+{
+"id": "12bd72f5-3028-4572-b59e-1cc143e44a86",
+"name": "Everyday @ 9am",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+-720,
+460
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"triggerAtHour": 8
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "ca62c507-bce5-4a63-be0e-e60591408668",
+"name": "Everyday @ 8.30am",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+-720,
+-220
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"triggerAtHour": 8,
+"triggerAtMinute": 30
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "032bec7e-5aff-4103-b81e-e5bc4a88ddde",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-540,
+-420
+],
+"parameters": {
+"color": 7,
+"width": 700,
+"height": 480,
+"content": "## 1. Fetch Latest AI Grants, Ignore Those Already Seen\n[Learn more about the Remove Duplicates node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.removeduplicates/)\n\nA cool feature of n8n's remove duplicates node is that it works across executions. What this means for this template is that the node will help us keep track of grant IDs to know if we've already processed them and if so, filter them out so we won't have duplicate alerts."
+},
+"typeVersion": 1
+},
+{
+"id": "07147665-3571-4512-adce-2727dcb95240",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+180,
+-520
+],
+"parameters": {
+"color": 7,
+"width": 1000,
+"height": 720,
+"content": "## 2. Quickly Determine Eligibility Using AI\n[Learn more about the Information Extractor node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.information-extractor/)\n\nQualifying Leads requires a lot of contextual reasoning taking into account many factors such as commercials, location and eligibility criteria. Whilst it's not guaranteed AI can or will solve this for your particular requirements, it can however get you a good distance of the way there!\n\nAI in this template intends to reduce time (and therefore cost) for a team member needs to spend per grant listing or increase their coverage of grants which they would otherwise miss due to capacity."
+},
+"typeVersion": 1
+},
+{
+"id": "f4758b4d-727a-4ce8-b071-3388eb16b219",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1200,
+-280
+],
+"parameters": {
+"color": 7,
+"width": 520,
+"height": 480,
+"content": "## 3. Save Results to Grant Tracker\n[Learn more about the Airtable Node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.airtable/)\n\nIn n8n, it's easy to send your data anywhere to manage yourself, share with your team or reuse with other workflows. Here for demonstration purposes, we'll just store each grant as a row in our Airtable database.\n\nCheck out the sample Airtable here: https://airtable.com/appiNoPRvhJxz9crl/shrRdP6zstgsxjDKL"
+},
+"typeVersion": 1
+},
+{
+"id": "a7861a21-021f-4629-b863-2163c7436d13",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-540,
+240
+],
+"parameters": {
+"color": 7,
+"width": 620,
+"height": 500,
+"content": "## 4. Generate Latest AI Grants Alert Email\n[Learn more about the HTML Template node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.html/)\n\nUsing our freshly collected AI grants, it would be nice if we can share them with our team members via email. A nicely formatted email digest can be generated using the HTML template node, with added links for greater impact.\n\nHere in this demonstration, we will loop through all eligible new grants and compile them into a newsletter format using the HTML node.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "4d09af53-92cb-4288-86d7-dcf695bfb358",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+100,
+240
+],
+"parameters": {
+"color": 7,
+"width": 640,
+"height": 500,
+"content": "## 5. Send to a list of Subscribers\n[Learn more about the Gmail node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.gmail/)\n\nFinally, we can source a list of subscribers to send our generated email newsletter.\n\nHere, our subscriber list is another table alongside our grants table that we can import that list using the Airtable node. You can use any email provider that supports HTML but for this demonstration, we're using Gmail for simplicity sake."
+},
+"typeVersion": 1
+},
+{
+"id": "784d59f3-5b1f-4404-bc04-4bd58cf03585",
+"name": "Get Subscribers",
+"type": "n8n-nodes-base.airtable",
+"position": [
+240,
+500
+],
+"parameters": {
+"base": {
+"__rl": true,
+"mode": "list",
+"value": "appiNoPRvhJxz9crl",
+"cachedResultUrl": "https://airtable.com/appiNoPRvhJxz9crl",
+"cachedResultName": "US Grants.gov Tracker"
+},
+"table": {
+"__rl": true,
+"mode": "list",
+"value": "tblaS91hyhguntfaC",
+"cachedResultUrl": "https://airtable.com/appiNoPRvhJxz9crl/tblaS91hyhguntfaC",
+"cachedResultName": "Subscribers"
+},
+"options": {},
+"operation": "search",
+"filterByFormula": "AND({Status} = 'Active')"
+},
+"credentials": {
+"airtableTokenApi": {
+"id": "Und0frCQ6SNVX3VV",
+"name": "Airtable Personal Access Token account"
+}
+},
+"executeOnce": true,
+"typeVersion": 2.1
+},
+{
+"id": "3be0788b-90ef-4648-aa25-1170208a685d",
+"name": "Send Subscriber Email",
+"type": "n8n-nodes-base.gmail",
+"position": [
+480,
+500
+],
+"webhookId": "37eeec7a-1982-4137-8473-313bfb6c5b42",
+"parameters": {
+"sendTo": "={{ $json.Email }}",
+"message": "={{ $('Generate Email').first().json.html }}",
+"options": {},
+"subject": "Daily Newletter for Intersting US Grants"
+},
+"credentials": {
+"gmailOAuth2": {
+"id": "Sf5Gfl9NiFTNXFWb",
+"name": "Gmail account"
+}
+},
+"typeVersion": 2.1
+},
+{
+"id": "14a65482-b314-4a2f-9ce3-87e3aae126f9",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1280,
+300
+],
+"parameters": {
+"color": 7,
+"width": 460,
+"height": 200,
+"content": "## Scheduled Triggers\n[Learn more about Scheduled Triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.scheduletrigger)\n\nScheduled triggers are a great way to run this template automatically in the morning ready for your team before they start their working day.\n\nFeel free to adjust the interval to a time which suits you!"
+},
+"typeVersion": 1
+},
+{
+"id": "b172eb7a-58bc-4d4a-be22-796d34a59897",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1280,
+-620
+],
+"parameters": {
+"width": 460,
+"height": 900,
+"content": "## Try It Out!\n\n### This n8n templates demonstrates how to automatically ingest a source of leads at regular intervals and take advantage of n8n's remove duplicates node to simplify duplicate detection.\nAdditionally after the leads are captured, a simple alerts notification can be generated and shared with team members.\n\n### How it works\n* A scheduled trigger is set to fetch a list of AI grants listed on the grants.gov website in the past day.\n* A Remove Duplicates node is used to track Grant IDs to filter out those already processed by the workflow.\n* New grants are summarized and analysed by AI nodes to determine eligibility and interest which is then saved to an Airtable database.\n* Another scheduled trigger starts a little later than the first to collect and summarize the new grants\n* The results are then compiled into an email template using the HTML node, in the form of a newsletter designed to alert and brief team members of new AI grants.\n* This email is then sent to a list of subscribers using the gmail node.\n\n## How to use\n* Make a copy of sample Airtable here: https://airtable.com/appiNoPRvhJxz9crl/shrRdP6zstgsxjDKL\n* The filters for fetching the grants is currently set to the \"AI\" category. Feel free to change this to include more categories.\n* Not interested in grants, this template can works for other sources of leads just change the endpoint and how you're defining the item ID to track.\n\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "f9849413-4dad-44dc-92ec-8879d123bfd3",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+720,
+40
+],
+"parameters": {
+"width": 320,
+"height": 120,
+"content": "### Add your company details here!\nCompany details are added in the system prompt to help the AI determine eligibility. The more details the better!"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Merge": {
+"main": [
+[
+{
+"node": "Save to Tracker",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Everyday @ 9am": {
+"main": [
+[
+{
+"node": "Get New Eligible Grants Today",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate Email": {
+"main": [
+[
+{
+"node": "Get Subscribers",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Grants to List": {
+"main": [
+[
+{
+"node": "Only New Grants",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Subscribers": {
+"main": [
+[
+{
+"node": "Send Subscriber Email",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Only New Grants": {
+"main": [
+[
+{
+"node": "Get Grant Details",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Save to Tracker": {
+"main": [
+[]
+]
+},
+"Everyday @ 8.30am": {
+"main": [
+[
+{
+"node": "AI Grants since Yesterday",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Grant Details": {
+"main": [
+[
+{
+"node": "Summarize Synopsis",
+"type": "main",
+"index": 0
+},
+{
+"node": "Eligibility Factors",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Summarize Synopsis",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Eligibility Factors",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Summarize Synopsis": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Eligibility Factors": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"AI Grants since Yesterday": {
+"main": [
+[
+{
+"node": "Grants to List",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get New Eligible Grants Today": {
+"main": [
+[
+{
+"node": "Generate Email",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.json b/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.json
deleted file mode 100644
index e00ed5a..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.json
+++ /dev/null
@@ -1,960 +0,0 @@
-{
- "meta": {
- "instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
- },
- "nodes": [
- {
- "id": "192d3e4f-6bb0-4b87-a1fa-e32c9efb49cc",
- "name": "When clicking \"Test workflow\"",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 336,
- 34
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "32a7a772-76a6-4614-a6ab-d2b152a5811f",
- "name": "OpenAI Chat Model1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1220,
- 180
- ],
- "parameters": {
- "model": "gpt-4o",
- "options": {
- "temperature": 0
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "8c444314-ed7d-4ca0-b0fa-b6d1e964c698",
- "name": "Get Applicable Rows",
- "type": "n8n-nodes-base.airtable",
- "position": [
- 516,
- 34
- ],
- "parameters": {
- "base": {
- "__rl": true,
- "mode": "list",
- "value": "appbgxPBurOmQK3E7",
- "cachedResultUrl": "https://airtable.com/appbgxPBurOmQK3E7",
- "cachedResultName": "Building Inventory Survey Example"
- },
- "table": {
- "__rl": true,
- "mode": "id",
- "value": "tblEHkoTvKpa4Aa0Q"
- },
- "options": {},
- "operation": "search",
- "returnAll": false,
- "filterByFormula": "AND(Image!=\"\", AI_status=FALSE())"
- },
- "credentials": {
- "airtableTokenApi": {
- "id": "Und0frCQ6SNVX3VV",
- "name": "Airtable Personal Access Token account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "f90578fa-b886-4653-8ff7-0c91884dc517",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- 1257,
- 733
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "8f5959eb-45bd-4185-a959-10268827e41d",
- "name": "Edit Fields",
- "type": "n8n-nodes-base.set",
- "position": [
- 1417,
- 733
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "7263764b-8409-4cea-8db3-3278dd7ef9d8",
- "name": "=route",
- "type": "string",
- "value": "={{ $json.route }}"
- },
- {
- "id": "55c3b207-2e98-4137-8413-f72cbff17986",
- "name": "query",
- "type": "string",
- "value": "={{ $json.query }}"
- },
- {
- "id": "6eb873de-3c3a-4135-9dc0-1d441c63647c",
- "name": "",
- "type": "string",
- "value": ""
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "2c7f7274-12e9-4dd3-8ee4-679b408d5430",
- "name": "Fallback Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 1580,
- 875
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "{\n \"response\": {\n \"ok\": false,\n \"error\": \"The requested tool was not found or the service may be unavailable. Do not retry.\"\n }\n}\n"
- },
- "typeVersion": 3.3
- },
- {
- "id": "09f36f4d-eb88-4d93-a8b3-e9ba66b46b54",
- "name": "SERP Google Reverse Image API",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1860,
- 549
- ],
- "parameters": {
- "url": "https://serpapi.com/search.json",
- "options": {},
- "sendQuery": true,
- "authentication": "predefinedCredentialType",
- "queryParameters": {
- "parameters": [
- {
- "name": "engine",
- "value": "google_reverse_image"
- },
- {
- "name": "image_url",
- "value": "={{ $json.query }}"
- }
- ]
- },
- "nodeCredentialType": "serpApi"
- },
- "credentials": {
- "serpApi": {
- "id": "aJCKjxx6U3K7ydDe",
- "name": "SerpAPI account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "8e3a0f38-8663-4f5c-837f-4b9aa21f14fb",
- "name": "Reverse Image Search Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 2037,
- 547
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "de99a504-713f-4c78-8679-08139b2def31",
- "name": "response",
- "type": "string",
- "value": "={{ JSON.stringify($json.image_results.map(x => ({ position: x.position, title: x.title, link: x.link, description: x.snippet }))) }}"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "0cd2269a-5b1f-4f10-b180-7f9cff9b1102",
- "name": "Reverse Image Search Tool",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 1300,
- 340
- ],
- "parameters": {
- "name": "reverse_image_search",
- "fields": {
- "values": [
- {
- "name": "route",
- "stringValue": "serp.google_reverse_image"
- }
- ]
- },
- "workflowId": "={{ $workflow.id }}",
- "description": "Call this tool to perform a reverse image search. Reverse image searches return urls where similar looking products exists. Fetch the returned urls to gather more information. This tool requires the following object request body.\n```\n{\n \"type\": \"object\",\n \"properties\": {\n \"image_url\": { \"type\": \"string\" },\n }\n}\n```\nimage_url should be an absolute URL to the image."
- },
- "typeVersion": 1.1
- },
- {
- "id": "9825651e-b382-4e0a-97ef-37764cb5be9e",
- "name": "Firecrawl Scrape API",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1860,
- 889
- ],
- "parameters": {
- "url": "https://api.firecrawl.dev/v0/scrape",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "url",
- "value": "={{ $json.query }}"
- }
- ]
- },
- "genericAuthType": "httpHeaderAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "Content-Type",
- "value": "application/json"
- }
- ]
- }
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "OUOnyTkL9vHZNorB",
- "name": "Firecrawl API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "7f61d60b-b052-4b7c-abfd-9eb8e05a45a2",
- "name": "Scrape Success?",
- "type": "n8n-nodes-base.if",
- "position": [
- 2020,
- 889
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "a15a164f-d0c5-478f-8b27-f3d51746c214",
- "operator": {
- "type": "boolean",
- "operation": "true",
- "singleValue": true
- },
- "leftValue": "={{ $json.success }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2
- },
- {
- "id": "29c65ef4-6350-490a-b8e3-a5c869e656b2",
- "name": "Firecrawl Scrape Success Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 2180,
- 889
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "7db5c81f-de90-40e1-8086-3f13d40451c7",
- "name": "response",
- "type": "string",
- "value": "={{ $json.data.markdown.substring(0, 3000) }}"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "229b4008-d8a8-4609-854a-fc244a4ed630",
- "name": "Firecrawl scrape Error Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 2180,
- 1049
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "e691d86a-d366-44a2-baa6-3dba42527f6e",
- "name": "response",
- "type": "string",
- "value": "{ error: \"Unable to scrape website due to unknown error. Do not retry.\" }"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "f080069b-e849-45e0-88cf-03707d22c704",
- "name": "Firecrawl Web Scaper Tool",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 1440,
- 340
- ],
- "parameters": {
- "name": "webpage_url_scraper_tool",
- "fields": {
- "values": [
- {
- "name": "route",
- "stringValue": "firecrawl.scrape"
- }
- ]
- },
- "workflowId": "={{ $workflow.id }}",
- "description": "Call this tool to retrieve page contents of a url.\n```\n{\n \"type\": \"object\",\n \"properties\": {\n \"url\": { \"type\": \"string\" },\n }\n}\n```\nurl should be an absolute URL."
- },
- "typeVersion": 1.1
- },
- {
- "id": "4eff88bb-bd5e-4d6a-b5e1-8521632c461f",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 1500,
- 180
- ],
- "parameters": {
- "jsonSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"title\": { \"type\": \"string\" },\n \"description\": { \"type\": \"string\" },\n \"model\": { \"type\": \"string\" },\n \"material\": { \"type\": \"string\" },\n \"color\": { \"type\": \"string\" },\n \"condition\": { \"type\": \"string\" }\n }\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "328d106b-a473-4f54-82fd-55c30d813da9",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 280,
- -260
- ],
- "parameters": {
- "color": 7,
- "width": 402.5984702109446,
- "height": 495.4071184783251,
- "content": "## 1. Use Airtable to Capture Survey Photos\n[Read more about AirTable](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.airtable)\n\nTo enable this workflow, we need a database where we can retreive the title and photo to analyse and write the generate values back to. Airtable is perfect for this since it has a robust API we can work with.\n\nFor this demo, we'll manually trigger but this can be changed for forms or other triggers."
- },
- "typeVersion": 1
- },
- {
- "id": "e358775d-ff83-411d-9364-b43c87d98134",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 716.3106363781314,
- -160
- ],
- "parameters": {
- "color": 7,
- "width": 359.40869874940336,
- "height": 428.4787925736586,
- "content": "## 2. Use AI Vision Model to Analyse the Photo.\n[Read more about OpenAI Vision](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-langchain.openai)\n\nWe'll use OpenAi vision model to create a detailed description of the product in the photo. We split this step from the agent because it uses an image model rather than the usual text-based one."
- },
- "typeVersion": 1
- },
- {
- "id": "51b4a70c-9583-4e8a-8e8d-896a80ad53c3",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1111.3914848823072,
- -293.9250474768817
- ],
- "parameters": {
- "color": 7,
- "width": 593.0683948010671,
- "height": 803.956942672397,
- "content": "## 3. Build an AI Agent who Searches the Internet\n[Read more about OpenAI Agents](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-langchain.openai)\n\nThis AI Agent has the ability to perform reverse image searches using our captured photos as well visit external webpages in order to obtain accurate product names and attributes. The Agent along with the tools might mimic what the average human user would carry out the same task.\n\n* For reverse image search, we're using SERP API service however we won't use the built-in SERP node as we need to specify custom parameters. \n* For scraping, we'll use [Firecrawl](https://www.firecrawl.dev/) as this service also helps to parse and return the page as markdown which is more efficient."
- },
- "typeVersion": 1
- },
- {
- "id": "adfb519b-a5c7-432c-be32-5acfcc388b49",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1740,
- -149.28190375244515
- ],
- "parameters": {
- "color": 7,
- "width": 373.3601237414979,
- "height": 397.7168664109706,
- "content": "## 4. Overwrite our Rows with Enriched Results\n\nAnd Viola! Our AI agent has potentially saved hours of manual data entry work for our surveyor. This technique can be used for many other usecases."
- },
- "typeVersion": 1
- },
- {
- "id": "6444e217-b944-450e-892a-5822d4d390ce",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1200,
- 549
- ],
- "parameters": {
- "color": 7,
- "width": 554.6092633638649,
- "height": 490.7010880746526,
- "content": "## 5. Using the Custom Workflow Tool\n[Read more about Workflow Tools](https://docs.n8n.io/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.toolworkflow)\n\nAI Agents rely on Tools to make decisions and become exponentially more powerful the more tools they have. A common pattern to manage multiple tools is to create a routing system for tools using the API pattern."
- },
- "typeVersion": 1
- },
- {
- "id": "bf2459cf-a931-4232-9504-b36b15721194",
- "name": "Enrich Product Rows",
- "type": "n8n-nodes-base.airtable",
- "position": [
- 1880,
- 60
- ],
- "parameters": {
- "base": {
- "__rl": true,
- "mode": "list",
- "value": "appbgxPBurOmQK3E7",
- "cachedResultUrl": "https://airtable.com/appbgxPBurOmQK3E7",
- "cachedResultName": "Building Inventory Survey Example"
- },
- "table": {
- "__rl": true,
- "mode": "id",
- "value": "tblEHkoTvKpa4Aa0Q"
- },
- "columns": {
- "value": {
- "id": "={{ $('Get Applicable Rows').item.json.id }}",
- "Color": "={{ $json.output.output.color }}",
- "Model": "={{ $json.output.output.model }}",
- "Title": "={{ $json.output.output.title }}",
- "Material": "={{ $json.output.output.material }}",
- "AI_status": true,
- "Condition": "={{ $json.output.output.condition }}",
- "Description": "={{ $json.output.output.description }}"
- },
- "schema": [
- {
- "id": "id",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": true,
- "required": false,
- "displayName": "id",
- "defaultMatch": true
- },
- {
- "id": "Title",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Title",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Image",
- "type": "array",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Image",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Description",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Description",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Model",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Model",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Material",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Material",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Color",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Color",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Condition",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "Condition",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "AI_status",
- "type": "boolean",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "AI_status",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [
- "id"
- ]
- },
- "options": {},
- "operation": "update"
- },
- "credentials": {
- "airtableTokenApi": {
- "id": "Und0frCQ6SNVX3VV",
- "name": "Airtable Personal Access Token account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "19d736bf-c29d-46a2-93bc-b536ff28c4b5",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -100,
- -260
- ],
- "parameters": {
- "width": 359.6648027457353,
- "height": 381.0536322713287,
- "content": "## Try It Out!\n### This workflow does the following:\n* Scans an Airtable spreadsheet for rows with product photo images.\n* Uses an AI vision model to attempt to identify the product.\n* Uses an AI Agent to research the product on the internet to enrich the product data.\n* Overwrites our Airtable spreadsheet with the enriched data.\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
- },
- "typeVersion": 1
- },
- {
- "id": "25f15c48-16bf-4f92-942d-c224ed88d208",
- "name": "Analyse Image",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 840,
- 80
- ],
- "parameters": {
- "text": "=Focus on the {{ $json.Title }} in the image - we'll refer to this as the \"object\". Identify the following attributes of the object. If you cannot determine confidently, then leave blank and move to next attribute.\n* Decription of the object.\n* The model/make of the object.\n* The material(s) used in the construction of the object.\n* The color(s) of the object\n* The condition of the object. Use one of poor, good, excellent.\n",
- "options": {},
- "resource": "image",
- "imageUrls": "={{ $json.Image[0].thumbnails.large.url }}",
- "operation": "analyze"
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.3
- },
- {
- "id": "e6c99f71-ccc9-426e-b916-cc38864e3224",
- "name": "Object Identifier Agent",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 1260,
- 20
- ],
- "parameters": {
- "text": "=system: Your role is to help an building surveyor perform a object classification and data collection task whereby the surveyor will take photos of various objects and your job is to try and identify accurately certain product attributes of the objects as detailed below.\n\nThe surveyor has given you the following:\n1) photo url ```{{ $('Get Applicable Rows').item.json.Image[0].thumbnails.large.url }}```.\n2) photo description ```{{ $json.content }}```.\n\nFor each product attribute the surveyor is unable to determine, you may:\n1) use the reverse image search tool to search the product on the internet via the provided image url.\n2) use the web scraper tool to read webpages on the internet which may be relevant to the product.\n3) If after using these tools, you are still unable to determine the required product attributes then leave the data blank.\n\nUse all the information provided and gathered, to extract the following product attributes: title, description, model, material, color and condition.",
- "agent": "openAiFunctionsAgent",
- "options": {},
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.5
- },
- {
- "id": "661b14bd-6511-4f20-981c-2e68a7c34ec5",
- "name": "Actions Router",
- "type": "n8n-nodes-base.switch",
- "position": [
- 1577,
- 733
- ],
- "parameters": {
- "rules": {
- "values": [
- {
- "outputKey": "serp.google_reverse_image",
- "conditions": {
- "options": {
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "operator": {
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.route }}",
- "rightValue": "serp.google_reverse_image"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "firecrawl.scrape",
- "conditions": {
- "options": {
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "0a1f54ae-39f1-468d-ba6e-1376d13e4ee8",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.route }}",
- "rightValue": "firecrawl.scrape"
- }
- ]
- },
- "renameOutput": true
- }
- ]
- },
- "options": {
- "fallbackOutput": "extra"
- }
- },
- "typeVersion": 3
- },
- {
- "id": "c5078221-9239-4ec0-b25e-7cd880b58216",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 480,
- 20
- ],
- "parameters": {
- "width": 181.2788838920522,
- "height": 297.0159375852115,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n๐จ**Required**\n* Set Airtable Base and Table IDs here."
- },
- "typeVersion": 1
- },
- {
- "id": "c58c0db4-9b99-4a77-90ae-66fa3981b684",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1840,
- 40
- ],
- "parameters": {
- "width": 181.2788838920522,
- "height": 297.0159375852115,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n๐จ**Required**\n* Set Airtable Base and Table IDs here."
- },
- "typeVersion": 1
- },
- {
- "id": "e3a666d7-d7a5-43f5-8f04-7972332f8916",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1780,
- 440
- ],
- "parameters": {
- "color": 7,
- "width": 460.3301604548244,
- "height": 298.81538450684064,
- "content": "## 5.1 Google Reverse Image Tool\nThis tool uses Google's reverse image API to return websites where similar images are found."
- },
- "typeVersion": 1
- },
- {
- "id": "d7407cdb-16bb-4bd9-a28e-7a72a5289354",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1780,
- 769.9385328672522
- ],
- "parameters": {
- "color": 7,
- "width": 575.3216480295998,
- "height": 463.34699288922565,
- "content": "## 5.2 Webscraper Tool\nThis tool uses Firecrawl.dev API to crawl webpages and returns those pages in markdown format."
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Edit Fields": {
- "main": [
- [
- {
- "node": "Actions Router",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Analyse Image": {
- "main": [
- [
- {
- "node": "Object Identifier Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Actions Router": {
- "main": [
- [
- {
- "node": "SERP Google Reverse Image API",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Firecrawl Scrape API",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Fallback Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Scrape Success?": {
- "main": [
- [
- {
- "node": "Firecrawl Scrape Success Response",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Firecrawl scrape Error Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model1": {
- "ai_languageModel": [
- [
- {
- "node": "Object Identifier Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Get Applicable Rows": {
- "main": [
- [
- {
- "node": "Analyse Image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Firecrawl Scrape API": {
- "main": [
- [
- {
- "node": "Scrape Success?",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Object Identifier Agent": {
- "main": [
- [
- {
- "node": "Enrich Product Rows",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Edit Fields",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Object Identifier Agent",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Firecrawl Web Scaper Tool": {
- "ai_tool": [
- [
- {
- "node": "Object Identifier Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Reverse Image Search Tool": {
- "ai_tool": [
- [
- {
- "node": "Object Identifier Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "SERP Google Reverse Image API": {
- "main": [
- [
- {
- "node": "Reverse Image Search Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking \"Test workflow\"": {
- "main": [
- [
- {
- "node": "Get Applicable Rows",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.txt b/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.txt
new file mode 100644
index 0000000..720c260
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Enrich Property Inventory Survey with Image Recognition and AI Agent.txt
@@ -0,0 +1,960 @@
+{
+"meta": {
+"instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
+},
+"nodes": [
+{
+"id": "192d3e4f-6bb0-4b87-a1fa-e32c9efb49cc",
+"name": "When clicking \"Test workflow\"",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+336,
+34
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "32a7a772-76a6-4614-a6ab-d2b152a5811f",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+1220,
+180
+],
+"parameters": {
+"model": "gpt-4o",
+"options": {
+"temperature": 0
+}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "8c444314-ed7d-4ca0-b0fa-b6d1e964c698",
+"name": "Get Applicable Rows",
+"type": "n8n-nodes-base.airtable",
+"position": [
+516,
+34
+],
+"parameters": {
+"base": {
+"__rl": true,
+"mode": "list",
+"value": "appbgxPBurOmQK3E7",
+"cachedResultUrl": "https://airtable.com/appbgxPBurOmQK3E7",
+"cachedResultName": "Building Inventory Survey Example"
+},
+"table": {
+"__rl": true,
+"mode": "id",
+"value": "tblEHkoTvKpa4Aa0Q"
+},
+"options": {},
+"operation": "search",
+"returnAll": false,
+"filterByFormula": "AND(Image!=\"\", AI_status=FALSE())"
+},
+"credentials": {
+"airtableTokenApi": {
+"id": "Und0frCQ6SNVX3VV",
+"name": "Airtable Personal Access Token account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "f90578fa-b886-4653-8ff7-0c91884dc517",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+1257,
+733
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "8f5959eb-45bd-4185-a959-10268827e41d",
+"name": "Edit Fields",
+"type": "n8n-nodes-base.set",
+"position": [
+1417,
+733
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "7263764b-8409-4cea-8db3-3278dd7ef9d8",
+"name": "=route",
+"type": "string",
+"value": "={{ $json.route }}"
+},
+{
+"id": "55c3b207-2e98-4137-8413-f72cbff17986",
+"name": "query",
+"type": "string",
+"value": "={{ $json.query }}"
+},
+{
+"id": "6eb873de-3c3a-4135-9dc0-1d441c63647c",
+"name": "",
+"type": "string",
+"value": ""
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "2c7f7274-12e9-4dd3-8ee4-679b408d5430",
+"name": "Fallback Response",
+"type": "n8n-nodes-base.set",
+"position": [
+1580,
+875
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "{\n \"response\": {\n \"ok\": false,\n \"error\": \"The requested tool was not found or the service may be unavailable. Do not retry.\"\n }\n}\n"
+},
+"typeVersion": 3.3
+},
+{
+"id": "09f36f4d-eb88-4d93-a8b3-e9ba66b46b54",
+"name": "SERP Google Reverse Image API",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1860,
+549
+],
+"parameters": {
+"url": "https://serpapi.com/search.json",
+"options": {},
+"sendQuery": true,
+"authentication": "predefinedCredentialType",
+"queryParameters": {
+"parameters": [
+{
+"name": "engine",
+"value": "google_reverse_image"
+},
+{
+"name": "image_url",
+"value": "={{ $json.query }}"
+}
+]
+},
+"nodeCredentialType": "serpApi"
+},
+"credentials": {
+"serpApi": {
+"id": "aJCKjxx6U3K7ydDe",
+"name": "SerpAPI account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8e3a0f38-8663-4f5c-837f-4b9aa21f14fb",
+"name": "Reverse Image Search Response",
+"type": "n8n-nodes-base.set",
+"position": [
+2037,
+547
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "de99a504-713f-4c78-8679-08139b2def31",
+"name": "response",
+"type": "string",
+"value": "={{ JSON.stringify($json.image_results.map(x => ({ position: x.position, title: x.title, link: x.link, description: x.snippet }))) }}"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "0cd2269a-5b1f-4f10-b180-7f9cff9b1102",
+"name": "Reverse Image Search Tool",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+1300,
+340
+],
+"parameters": {
+"name": "reverse_image_search",
+"fields": {
+"values": [
+{
+"name": "route",
+"stringValue": "serp.google_reverse_image"
+}
+]
+},
+"workflowId": "={{ $workflow.id }}",
+"description": "Call this tool to perform a reverse image search. Reverse image searches return urls where similar looking products exists. Fetch the returned urls to gather more information. This tool requires the following object request body.\n```\n{\n \"type\": \"object\",\n \"properties\": {\n \"image_url\": { \"type\": \"string\" },\n }\n}\n```\nimage_url should be an absolute URL to the image."
+},
+"typeVersion": 1.1
+},
+{
+"id": "9825651e-b382-4e0a-97ef-37764cb5be9e",
+"name": "Firecrawl Scrape API",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1860,
+889
+],
+"parameters": {
+"url": "https://api.firecrawl.dev/v0/scrape",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "url",
+"value": "={{ $json.query }}"
+}
+]
+},
+"genericAuthType": "httpHeaderAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "Content-Type",
+"value": "application/json"
+}
+]
+}
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "OUOnyTkL9vHZNorB",
+"name": "Firecrawl API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "7f61d60b-b052-4b7c-abfd-9eb8e05a45a2",
+"name": "Scrape Success?",
+"type": "n8n-nodes-base.if",
+"position": [
+2020,
+889
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "a15a164f-d0c5-478f-8b27-f3d51746c214",
+"operator": {
+"type": "boolean",
+"operation": "true",
+"singleValue": true
+},
+"leftValue": "={{ $json.success }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2
+},
+{
+"id": "29c65ef4-6350-490a-b8e3-a5c869e656b2",
+"name": "Firecrawl Scrape Success Response",
+"type": "n8n-nodes-base.set",
+"position": [
+2180,
+889
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "7db5c81f-de90-40e1-8086-3f13d40451c7",
+"name": "response",
+"type": "string",
+"value": "={{ $json.data.markdown.substring(0, 3000) }}"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "229b4008-d8a8-4609-854a-fc244a4ed630",
+"name": "Firecrawl scrape Error Response",
+"type": "n8n-nodes-base.set",
+"position": [
+2180,
+1049
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "e691d86a-d366-44a2-baa6-3dba42527f6e",
+"name": "response",
+"type": "string",
+"value": "{ error: \"Unable to scrape website due to unknown error. Do not retry.\" }"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "f080069b-e849-45e0-88cf-03707d22c704",
+"name": "Firecrawl Web Scaper Tool",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+1440,
+340
+],
+"parameters": {
+"name": "webpage_url_scraper_tool",
+"fields": {
+"values": [
+{
+"name": "route",
+"stringValue": "firecrawl.scrape"
+}
+]
+},
+"workflowId": "={{ $workflow.id }}",
+"description": "Call this tool to retrieve page contents of a url.\n```\n{\n \"type\": \"object\",\n \"properties\": {\n \"url\": { \"type\": \"string\" },\n }\n}\n```\nurl should be an absolute URL."
+},
+"typeVersion": 1.1
+},
+{
+"id": "4eff88bb-bd5e-4d6a-b5e1-8521632c461f",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1500,
+180
+],
+"parameters": {
+"jsonSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"title\": { \"type\": \"string\" },\n \"description\": { \"type\": \"string\" },\n \"model\": { \"type\": \"string\" },\n \"material\": { \"type\": \"string\" },\n \"color\": { \"type\": \"string\" },\n \"condition\": { \"type\": \"string\" }\n }\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "328d106b-a473-4f54-82fd-55c30d813da9",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+280,
+-260
+],
+"parameters": {
+"color": 7,
+"width": 402.5984702109446,
+"height": 495.4071184783251,
+"content": "## 1. Use Airtable to Capture Survey Photos\n[Read more about AirTable](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.airtable)\n\nTo enable this workflow, we need a database where we can retreive the title and photo to analyse and write the generate values back to. Airtable is perfect for this since it has a robust API we can work with.\n\nFor this demo, we'll manually trigger but this can be changed for forms or other triggers."
+},
+"typeVersion": 1
+},
+{
+"id": "e358775d-ff83-411d-9364-b43c87d98134",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+716.3106363781314,
+-160
+],
+"parameters": {
+"color": 7,
+"width": 359.40869874940336,
+"height": 428.4787925736586,
+"content": "## 2. Use AI Vision Model to Analyse the Photo.\n[Read more about OpenAI Vision](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-langchain.openai)\n\nWe'll use OpenAi vision model to create a detailed description of the product in the photo. We split this step from the agent because it uses an image model rather than the usual text-based one."
+},
+"typeVersion": 1
+},
+{
+"id": "51b4a70c-9583-4e8a-8e8d-896a80ad53c3",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1111.3914848823072,
+-293.9250474768817
+],
+"parameters": {
+"color": 7,
+"width": 593.0683948010671,
+"height": 803.956942672397,
+"content": "## 3. Build an AI Agent who Searches the Internet\n[Read more about OpenAI Agents](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-langchain.openai)\n\nThis AI Agent has the ability to perform reverse image searches using our captured photos as well visit external webpages in order to obtain accurate product names and attributes. The Agent along with the tools might mimic what the average human user would carry out the same task.\n\n* For reverse image search, we're using SERP API service however we won't use the built-in SERP node as we need to specify custom parameters. \n* For scraping, we'll use [Firecrawl](https://www.firecrawl.dev/) as this service also helps to parse and return the page as markdown which is more efficient."
+},
+"typeVersion": 1
+},
+{
+"id": "adfb519b-a5c7-432c-be32-5acfcc388b49",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1740,
+-149.28190375244515
+],
+"parameters": {
+"color": 7,
+"width": 373.3601237414979,
+"height": 397.7168664109706,
+"content": "## 4. Overwrite our Rows with Enriched Results\n\nAnd Viola! Our AI agent has potentially saved hours of manual data entry work for our surveyor. This technique can be used for many other usecases."
+},
+"typeVersion": 1
+},
+{
+"id": "6444e217-b944-450e-892a-5822d4d390ce",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1200,
+549
+],
+"parameters": {
+"color": 7,
+"width": 554.6092633638649,
+"height": 490.7010880746526,
+"content": "## 5. Using the Custom Workflow Tool\n[Read more about Workflow Tools](https://docs.n8n.io/integrations/builtin/cluster-nodes/sub-nodes/n8n-nodes-langchain.toolworkflow)\n\nAI Agents rely on Tools to make decisions and become exponentially more powerful the more tools they have. A common pattern to manage multiple tools is to create a routing system for tools using the API pattern."
+},
+"typeVersion": 1
+},
+{
+"id": "bf2459cf-a931-4232-9504-b36b15721194",
+"name": "Enrich Product Rows",
+"type": "n8n-nodes-base.airtable",
+"position": [
+1880,
+60
+],
+"parameters": {
+"base": {
+"__rl": true,
+"mode": "list",
+"value": "appbgxPBurOmQK3E7",
+"cachedResultUrl": "https://airtable.com/appbgxPBurOmQK3E7",
+"cachedResultName": "Building Inventory Survey Example"
+},
+"table": {
+"__rl": true,
+"mode": "id",
+"value": "tblEHkoTvKpa4Aa0Q"
+},
+"columns": {
+"value": {
+"id": "={{ $('Get Applicable Rows').item.json.id }}",
+"Color": "={{ $json.output.output.color }}",
+"Model": "={{ $json.output.output.model }}",
+"Title": "={{ $json.output.output.title }}",
+"Material": "={{ $json.output.output.material }}",
+"AI_status": true,
+"Condition": "={{ $json.output.output.condition }}",
+"Description": "={{ $json.output.output.description }}"
+},
+"schema": [
+{
+"id": "id",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": true,
+"required": false,
+"displayName": "id",
+"defaultMatch": true
+},
+{
+"id": "Title",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Title",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Image",
+"type": "array",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Image",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Description",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Description",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Model",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Model",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Material",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Material",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Color",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Color",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Condition",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "Condition",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "AI_status",
+"type": "boolean",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "AI_status",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [
+"id"
+]
+},
+"options": {},
+"operation": "update"
+},
+"credentials": {
+"airtableTokenApi": {
+"id": "Und0frCQ6SNVX3VV",
+"name": "Airtable Personal Access Token account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "19d736bf-c29d-46a2-93bc-b536ff28c4b5",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-100,
+-260
+],
+"parameters": {
+"width": 359.6648027457353,
+"height": 381.0536322713287,
+"content": "## Try It Out!\n### This workflow does the following:\n* Scans an Airtable spreadsheet for rows with product photo images.\n* Uses an AI vision model to attempt to identify the product.\n* Uses an AI Agent to research the product on the internet to enrich the product data.\n* Overwrites our Airtable spreadsheet with the enriched data.\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "25f15c48-16bf-4f92-942d-c224ed88d208",
+"name": "Analyse Image",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+840,
+80
+],
+"parameters": {
+"text": "=Focus on the {{ $json.Title }} in the image - we'll refer to this as the \"object\". Identify the following attributes of the object. If you cannot determine confidently, then leave blank and move to next attribute.\n* Decription of the object.\n* The model/make of the object.\n* The material(s) used in the construction of the object.\n* The color(s) of the object\n* The condition of the object. Use one of poor, good, excellent.\n",
+"options": {},
+"resource": "image",
+"imageUrls": "={{ $json.Image[0].thumbnails.large.url }}",
+"operation": "analyze"
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.3
+},
+{
+"id": "e6c99f71-ccc9-426e-b916-cc38864e3224",
+"name": "Object Identifier Agent",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+1260,
+20
+],
+"parameters": {
+"text": "=system: Your role is to help an building surveyor perform a object classification and data collection task whereby the surveyor will take photos of various objects and your job is to try and identify accurately certain product attributes of the objects as detailed below.\n\nThe surveyor has given you the following:\n1) photo url ```{{ $('Get Applicable Rows').item.json.Image[0].thumbnails.large.url }}```.\n2) photo description ```{{ $json.content }}```.\n\nFor each product attribute the surveyor is unable to determine, you may:\n1) use the reverse image search tool to search the product on the internet via the provided image url.\n2) use the web scraper tool to read webpages on the internet which may be relevant to the product.\n3) If after using these tools, you are still unable to determine the required product attributes then leave the data blank.\n\nUse all the information provided and gathered, to extract the following product attributes: title, description, model, material, color and condition.",
+"agent": "openAiFunctionsAgent",
+"options": {},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "661b14bd-6511-4f20-981c-2e68a7c34ec5",
+"name": "Actions Router",
+"type": "n8n-nodes-base.switch",
+"position": [
+1577,
+733
+],
+"parameters": {
+"rules": {
+"values": [
+{
+"outputKey": "serp.google_reverse_image",
+"conditions": {
+"options": {
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"operator": {
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.route }}",
+"rightValue": "serp.google_reverse_image"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "firecrawl.scrape",
+"conditions": {
+"options": {
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "0a1f54ae-39f1-468d-ba6e-1376d13e4ee8",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.route }}",
+"rightValue": "firecrawl.scrape"
+}
+]
+},
+"renameOutput": true
+}
+]
+},
+"options": {
+"fallbackOutput": "extra"
+}
+},
+"typeVersion": 3
+},
+{
+"id": "c5078221-9239-4ec0-b25e-7cd880b58216",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+480,
+20
+],
+"parameters": {
+"width": 181.2788838920522,
+"height": 297.0159375852115,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n๐จ**Required**\n* Set Airtable Base and Table IDs here."
+},
+"typeVersion": 1
+},
+{
+"id": "c58c0db4-9b99-4a77-90ae-66fa3981b684",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1840,
+40
+],
+"parameters": {
+"width": 181.2788838920522,
+"height": 297.0159375852115,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n๐จ**Required**\n* Set Airtable Base and Table IDs here."
+},
+"typeVersion": 1
+},
+{
+"id": "e3a666d7-d7a5-43f5-8f04-7972332f8916",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1780,
+440
+],
+"parameters": {
+"color": 7,
+"width": 460.3301604548244,
+"height": 298.81538450684064,
+"content": "## 5.1 Google Reverse Image Tool\nThis tool uses Google's reverse image API to return websites where similar images are found."
+},
+"typeVersion": 1
+},
+{
+"id": "d7407cdb-16bb-4bd9-a28e-7a72a5289354",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1780,
+769.9385328672522
+],
+"parameters": {
+"color": 7,
+"width": 575.3216480295998,
+"height": 463.34699288922565,
+"content": "## 5.2 Webscraper Tool\nThis tool uses Firecrawl.dev API to crawl webpages and returns those pages in markdown format."
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Edit Fields": {
+"main": [
+[
+{
+"node": "Actions Router",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Analyse Image": {
+"main": [
+[
+{
+"node": "Object Identifier Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Actions Router": {
+"main": [
+[
+{
+"node": "SERP Google Reverse Image API",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Firecrawl Scrape API",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Fallback Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Scrape Success?": {
+"main": [
+[
+{
+"node": "Firecrawl Scrape Success Response",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Firecrawl scrape Error Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Object Identifier Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Get Applicable Rows": {
+"main": [
+[
+{
+"node": "Analyse Image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Firecrawl Scrape API": {
+"main": [
+[
+{
+"node": "Scrape Success?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Object Identifier Agent": {
+"main": [
+[
+{
+"node": "Enrich Product Rows",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Edit Fields",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Object Identifier Agent",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Firecrawl Web Scaper Tool": {
+"ai_tool": [
+[
+{
+"node": "Object Identifier Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Reverse Image Search Tool": {
+"ai_tool": [
+[
+{
+"node": "Object Identifier Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"SERP Google Reverse Image API": {
+"main": [
+[
+{
+"node": "Reverse Image Search Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking \"Test workflow\"": {
+"main": [
+[
+{
+"node": "Get Applicable Rows",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.json b/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.json
deleted file mode 100644
index 6d18645..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.json
+++ /dev/null
@@ -1,1194 +0,0 @@
-{
- "meta": {
- "instanceId": "6a2a7715680b8313f7cb4676321c5baa46680adfb913072f089f2766f42e43bd"
- },
- "nodes": [
- {
- "id": "f4b3833b-cf25-4bbc-927c-080586c5713c",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 700,
- 760
- ],
- "parameters": {
- "color": 7,
- "width": 330.5152611046425,
- "height": 239.5888196628349,
- "content": "### ... or watch set up video [13 min]\n[](https://youtu.be/6RmLZS8Yl4E)\n"
- },
- "typeVersion": 1
- },
- {
- "id": "64d96c53-b3e2-4aea-9a29-9b9e5c729f4f",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 400,
- 240
- ],
- "parameters": {
- "color": 7,
- "width": 636.2128494576581,
- "height": 497.1532689930921,
- "content": "\n## AI Agent To Chat With Youtube\n**Made by [Mark Shcherbakov](https://www.linkedin.com/in/marklowcoding/) from community [5minAI](https://www.skool.com/5minai)**\n\nNavigating the content generation and optimization process can be complex, especially without significant audience insight. This workflow automates insights extraction from YouTube videos and comments, empowering users to create more engaging and relevant content effectively.\n\nThe workflow integrates various APIs to gather insights from YouTube videos, enabling automated commentary analysis, video transcription, and thumbnail evaluation. The main functionalities include:\n- Extracting user preferences from comments.\n- Transcribing video content for enhanced understanding.\n- Analyzing thumbnails via AI for maximum viewer engagement insights.\n\n"
- },
- "typeVersion": 1
- },
- {
- "id": "57d2ede9-1bf9-4449-9dc9-af1ccee763b6",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 400,
- 760
- ],
- "parameters": {
- "color": 7,
- "width": 280.2462120317618,
- "height": 545.9087885077763,
- "content": "### Set up steps\n\n1. **API Setup**:\n - Create a [Google Cloud](https://console.cloud.google.com/apis/dashboard) project and enable the YouTube Data API.\n - Generate an API key for [Apify](https://www.apify.com?fpr=ujogj).\n - Generate API key for [OpenAI](https://platform.openai.com)\n - Create all credentials in N8N - OpenAI, Apify, Google Cloud.\n\n2. **YouTube Creator and Video Selection**:\n - Start by defining a request to identify top creators based on their video views.\n - Capture the YouTube video IDs for further analysis of comments and other video metrics.\n\n3. **Comment Analysis**:\n - Gather comments associated with the selected videos and analyze them for user insights.\n - Implement pagination to handle the maximum comment retrieval limits in API requests.\n\n4. **Video Transcription**:\n - Request transcriptions for videos of interest, ensuring to manage potential costs associated with longer video processing.\n - Utilize the insights from transcriptions to formulate content plans.\n\n5. **Thumbnail Analysis**:\n - Evaluate your video thumbnails by submitting the URL through the OpenAI API to gain insights into their effectiveness.\n\n6. **Data Management**:\n - Incorporate a database agent to organize video data and metrics, allowing efficient record management and future content planning."
- },
- "typeVersion": 1
- },
- {
- "id": "ca0fd549-88a7-44fd-ab81-7fd5ca140dae",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1540,
- 820
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "zJhr5piyEwVnWtaI",
- "name": "OpenAi club"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "7f2cf209-2e9d-4d6a-bc9e-d1bfd6df7266",
- "name": "get_channel_details",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 1900,
- 820
- ],
- "parameters": {
- "name": "get_channel_details",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "=get_channel_details"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Get channel_id, title and description by handle/username.\nChannel_id is required to find videos and details about this channel.\nIf Youtube link to channel provided - parse handle from there or return channel_id. (e.g. https://www.youtube.com/@example_handle - example_handle)\n\n\nExample Input:\nexample_handle\n\nExample Output:\nid:UCOgz_YflAsYnGbdvzXuKNCA\ntitle:Daniel Simmons\ndescription:Digital Diary ๐ค\\n\\n\\nWeekly videos around fashion...",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"handle\": {\n \"type\": \"string\",\n \"description\": \"Handle/username of channel\"\n }},\n \"required\": [\"handle\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "c02f5c19-6e50-4a06-95b9-eceb3eec1012",
- "name": "get_video_description",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2020,
- 820
- ],
- "parameters": {
- "name": "get_video_description",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "video_details"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Fetch video details - the full description, title, and publish date of a video using its video_id.\n\nExample input:\nvideo_id:dQw4w9WgXcQ\n\nExample Output:\ntitle:Never Gonna Give You Up\ndescription: \"The official video for โNever Gonna Give You Upโ by Rick Astley.\nduration:4 min\nviewCount:154\nlikeCount:6\nthumbnails: urls",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the video to fetch details for\"\n }\n },\n \"required\": [\"video_id\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "2d61160b-3a65-4766-ace6-947a7c5de6e5",
- "name": "get_list_of_videos",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2140,
- 820
- ],
- "parameters": {
- "name": "get_list_of_videos",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "videos"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Retrieve a list of videos from a channel using channel_id. Supports sorting by date, relevance, or view count.\n\nExample Input:\nchannel_id\": \"UCxxxxxxxxxxxxxxxx\"\nnumber_of_videos\": 5\norder: \"date\"\npublishedAfter: \"timestamp\"\n\nExample Output:\nvideo_id:abc123\ntitle:Latest Video\nshort cut description:Latest Video\npublished_at:2023-12-05T10:00:00Z",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"channel_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the channel to fetch videos from\"\n },\n \"number_of_videos\": {\n \"type\": \"integer\",\n \"description\": \"The maximum number of videos to retrieve (max 50)\"\n },\n \"order\": {\n \"type\": \"string\",\n \"enum\": [\"date\", \"relevance\", \"viewCount\"],\n \"description\": \"Order in which to fetch videos\"\n },\n \"publishedAfter\": {\n \"type\": \"string\",\n \"description\": \"Timestamp for filtering like 2023-11-03T15:28:05Z.\"\n }\n },\n \"required\": [\"channel_id\", \"number_of_videos\", \"order\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "c5aa2f7c-7748-4f88-abb6-fd274ad1295a",
- "name": "get_list_of_comments",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2260,
- 820
- ],
- "parameters": {
- "name": "get_list_of_comments",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "comments"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Retrieve a list of comments from a video using video_id.\n\nInput:\n \"video_id\": \"dQw4w9WgXcQ\"\n\nOutput:\n \"author\": \"John Doe\",\n \"comment\": \"This is an amazing video!\",\n \"published_at\": \"2023-12-04T12:00:00Z\"",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the video to fetch comments from\"\n }\n },\n \"required\": [\"video_id\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "c68cad77-1d71-45a3-b94b-8f7c701f56fb",
- "name": "search",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2380,
- 820
- ],
- "parameters": {
- "name": "search",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "search"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Search for videos or channels using a query. Supports filtering by type (video or channel) and sorting (date, viewCount, relevance). Use | for OR and - to exclude terms in the query.\n\nInput:\ntype: video or channel\nquery: search query\nsorting: date, viewCount, relevance\npublishedAfter: timestamp\n\nOutput:\n- id, title, short cut description, and published_at.",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\"video\", \"channel\"],\n \"description\": \"Type of results to retrieve: video or channel\"\n },\n \"query\": {\n \"type\": \"string\",\n \"description\": \"Search query. Supports | for OR and - to exclude terms\"\n },\n \"sorting\": {\n \"type\": \"string\",\n \"enum\": [\"date\", \"viewCount\", \"relevance\"],\n \"description\": \"Sorting criteria for search results\"\n },\n \"publishedAfter\": {\n \"type\": \"string\",\n \"description\": \"Timestamp for filtering like 2023-11-03T15:28:05Z\"\n }\n },\n \"required\": [\"type\", \"query\", \"sorting\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "c87d5392-8a5c-4999-9e58-89a5e0700c40",
- "name": "analyze_thumbnail",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2500,
- 820
- ],
- "parameters": {
- "name": "analyze_thumbnail",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "analyze_thumbnail"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Analyze a thumbnail image based on a given prompt. The prompt can be customized for specific analysis needs, such as design critique, color scheme evaluation, or content assessment.\nUse link of maxRes thumbnail. \n\nInput:\n- url: URL of the thumbnail image.\n- prompt: Customizable instruction for the analysis.\n\nOutput:\n- Results of the analysis based on the given prompt.",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"url\": {\n \"type\": \"string\",\n \"description\": \"URL of the thumbnail image to analyze\"\n },\n \"prompt\": {\n \"type\": \"string\",\n \"description\": \"Customizable instruction to guide the image analysis\"\n }\n },\n \"required\": [\"url\", \"prompt\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "1be2fa35-9091-4db8-a8eb-50f822d618d3",
- "name": "video_transcription",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2620,
- 820
- ],
- "parameters": {
- "name": "video_transcription",
- "fields": {
- "values": [
- {
- "name": "command",
- "stringValue": "video_transcription"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": {
- "__rl": true,
- "mode": "list",
- "value": "FgknOUpOBkpY85NX",
- "cachedResultName": "Youtube parser - tools"
- },
- "description": "Transcribe a video and retrieve its text transcription. Useful for analyzing video content or repurposing it for other formats.\n\nInput:\n- video_url: URL of the video to transcribe.\n\nOutput:\n- The text transcription of the video.",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_url\": {\n \"type\": \"string\",\n \"description\": \"URL of the video to transcribe\"\n }\n },\n \"required\": [\"video_url\"]\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.2
- },
- {
- "id": "fbfcd82f-e247-4a21-be12-339df7afe681",
- "name": "Postgres Chat Memory",
- "type": "@n8n/n8n-nodes-langchain.memoryPostgresChat",
- "position": [
- 1700,
- 820
- ],
- "parameters": {
- "sessionKey": "={{ $('When chat message received').item.json.sessionId }}",
- "sessionIdType": "customKey"
- },
- "credentials": {
- "postgres": {
- "id": "AO9cER6p8uX7V07T",
- "name": "Postgres 5minai"
- }
- },
- "typeVersion": 1.3
- },
- {
- "id": "6a4bbad9-27ab-448b-9222-2c8843fe241a",
- "name": "AI Agent",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 1760,
- 560
- ],
- "parameters": {
- "text": "={{ $('When chat message received').item.json.chatInput }}",
- "agent": "openAiFunctionsAgent",
- "options": {
- "systemMessage": "You are Youtube assistant. \nYou need to process user's requests and run relevant tools for that. \n\nPlan and execute in right order runs of tools to get data for user's request.\n\nIMPORTANT Search query and list of videos for channel tools returns all videos including shorts - use Get Video description tool to identify shorts (less than minute) and filter them out if needed.\n\nFeel free to ask questions before do actions - especially if you noticed some inconcistency in user requests that might be error/misspelling. "
- },
- "promptType": "define"
- },
- "typeVersion": 1.6
- },
- {
- "id": "739cc12a-27d1-48e9-b124-7f83fb372514",
- "name": "When chat message received",
- "type": "@n8n/n8n-nodes-langchain.chatTrigger",
- "position": [
- 1460,
- 600
- ],
- "webhookId": "6e95bc27-99a6-417c-8bf7-2831d7f7a4be",
- "parameters": {
- "options": {}
- },
- "typeVersion": 1.1
- },
- {
- "id": "613af9f2-77fa-42c4-86d3-87e20f2c0c89",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1380,
- 500
- ],
- "parameters": {
- "width": 1430.34590072234,
- "height": 588.1344471094899,
- "content": "## Scenario 1: AI agent"
- },
- "typeVersion": 1
- },
- {
- "id": "54116346-bc73-4a6a-8bca-f2a6e6699374",
- "name": "Get Comments",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 1598
- ],
- "parameters": {
- "url": "=https://www.googleapis.com/youtube/v3/commentThreads",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "part",
- "value": "id,snippet,replies"
- },
- {
- "name": "videoId",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.video_id }}"
- },
- {
- "name": "maxResults",
- "value": "100"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "1DXeuNaLSixqGPaU",
- "name": "Query Auth account Youtube"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "faabf71a-69f2-4113-802e-124a09fa9a0a",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- 1444,
- 1598
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "4b3ec3aa-7c69-4a72-a989-02f97acdf612",
- "name": "Get Channel Details",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 1278
- ],
- "parameters": {
- "url": "=https://www.googleapis.com/youtube/v3/channels",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "part",
- "value": "snippet"
- },
- {
- "name": "forHandle",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.handle }}"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "1DXeuNaLSixqGPaU",
- "name": "Query Auth account Youtube"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "ed8dec73-8c50-4eb9-8efe-68ee72c4d5e6",
- "name": "Get Video Description",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 1438
- ],
- "parameters": {
- "url": "=https://www.googleapis.com/youtube/v3/videos",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "part",
- "value": "snippet,contentDetails,statistics"
- },
- {
- "name": "id",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.video_id }}"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "1DXeuNaLSixqGPaU",
- "name": "Query Auth account Youtube"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "c1ff3837-8d7e-49ad-a333-c177833fcd05",
- "name": "Edit Fields",
- "type": "n8n-nodes-base.set",
- "position": [
- 2224,
- 1598
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "469d89ba-23fc-482a-b4ae-ce5d3bc13579",
- "name": "response",
- "type": "string",
- "value": "={{ JSON.stringify(` Comments: ${$json.items.map(item => { const topLevelComment = `${item.snippet.topLevelComment.snippet.authorDisplayName}: ${item.snippet.topLevelComment.snippet.textOriginal}`; const replies = item.replies?.comments.map(reply => `${reply.snippet.authorDisplayName}: ${reply.snippet.textOriginal}` ).join('\\n') || ''; return [topLevelComment, replies].filter(Boolean).join('\\n'); }).join('\\n\\n')} `) }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "5f0c44fe-2523-4170-a27d-0ccd1bef24a7",
- "name": "Run Query",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 1758
- ],
- "parameters": {
- "url": "=https://www.googleapis.com/youtube/v3/search",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "part",
- "value": "snippet"
- },
- {
- "name": "q",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.query }}"
- },
- {
- "name": "order",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.order }}"
- },
- {
- "name": "type",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.type }}"
- },
- {
- "name": "maxResults",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.number_of_videos }}"
- },
- {
- "name": "publishedAfter",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.publishedAfter }}"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "1DXeuNaLSixqGPaU",
- "name": "Query Auth account Youtube"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "3e192718-6710-4143-ac6e-15df79ee5363",
- "name": "Get Videos by Channel",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 1918
- ],
- "parameters": {
- "url": "=https://www.googleapis.com/youtube/v3/search",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "part",
- "value": "snippet"
- },
- {
- "name": "channelId",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.channel_id }}"
- },
- {
- "name": "order",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.order }}"
- },
- {
- "name": "maxResults",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.number_of_videos }}"
- },
- {
- "name": "type",
- "value": "video"
- },
- {
- "name": "publishedAfter",
- "value": "={{ $('Execute Workflow Trigger').item.json.query.publishedAfter }}"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "1DXeuNaLSixqGPaU",
- "name": "Query Auth account Youtube"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "8bcb50a4-0cd1-4311-ac6a-2ee8653cfb71",
- "name": "Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 2564,
- 1598
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "cfdbe2f5-921e-496d-87bd-9c57fdc22a7a",
- "name": "response",
- "type": "object",
- "value": "={{$json}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "7f5a36d3-6710-4e69-8459-7c8c748ee7d9",
- "name": "Switch",
- "type": "n8n-nodes-base.switch",
- "position": [
- 1624,
- 1578
- ],
- "parameters": {
- "rules": {
- "values": [
- {
- "outputKey": "get_channel_details",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "operator": {
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "get_channel_details"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "video_details",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "26a3ffe8-c8a6-4564-8d18-5494a8059372",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "video_details"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "comments",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "0f51cc26-2e42-42e1-a5c2-cb1d2e384962",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "comments"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "search",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "51031140-5ceb-48aa-9f33-d314131a9653",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "search"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "videos",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "f160bf0a-423f-448d-ab80-50a0b6a177ca",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "videos"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "analyze_thumbnail",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "29542ac4-7b9d-413f-aabb-a1cdabed2fa7",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "analyze_thumbnail"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "video_transcription",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "35fc39b8-6cf1-4ea6-9609-4a195c5526f8",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
- "rightValue": "video_transcription"
- }
- ]
- },
- "renameOutput": true
- }
- ]
- },
- "options": {}
- },
- "typeVersion": 3.2
- },
- {
- "id": "df432d53-33bf-4e91-9ead-7f4b36bd788a",
- "name": "Get Video Transcription",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2064,
- 2238
- ],
- "parameters": {
- "url": "=https://api.apify.com/v2/acts/dB9f4B02ocpTICIEY/run-sync-get-dataset-items",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"startUrls\": [\n \"{{ $('Execute Workflow Trigger').item.json.query.video_url }}\"\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth"
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "XDavOaI9qH5Zi3QC",
- "name": "Apify"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "8079e5c9-4a52-45ce-ac41-7fc707177a5a",
- "name": "OpenAI",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 2064,
- 2078
- ],
- "parameters": {
- "text": "={{ $('Execute Workflow Trigger').item.json.query.prompt }}",
- "modelId": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o",
- "cachedResultName": "GPT-4O"
- },
- "options": {},
- "resource": "image",
- "imageUrls": "={{ $('Execute Workflow Trigger').item.json.query.url }}",
- "operation": "analyze"
- },
- "credentials": {
- "openAiApi": {
- "id": "SphXAX7rlwRLkiox",
- "name": "Test club key"
- }
- },
- "typeVersion": 1.7
- },
- {
- "id": "7847e82a-fe82-498c-8c14-4c1c718d632c",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1380,
- 1140
- ],
- "parameters": {
- "width": 1427.3810326521016,
- "height": 1313.2689194736308,
- "content": "## Scenario 2: Agent tools"
- },
- "typeVersion": 1
- },
- {
- "id": "3a0fbbb0-4c0e-41f1-abb3-c87e955ad1b3",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1540,
- 960
- ],
- "parameters": {
- "color": 4,
- "width": 266.7375650720483,
- "height": 80,
- "content": "### Replace credentials"
- },
- "typeVersion": 1
- },
- {
- "id": "363eaca0-aaa5-4551-845f-528f19bba57a",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2004,
- 1178
- ],
- "parameters": {
- "color": 4,
- "width": 266.7375650720483,
- "height": 80,
- "content": "### Replace credentials in all nodes - Apify, OpenAI, Google"
- },
- "typeVersion": 1
- }
- ],
- "pinData": {
- "Execute Workflow Trigger": [
- {
- "query": {
- "type": "video",
- "query": "Web scraping data with n8n and Puppeteer",
- "sorting": "relevance"
- },
- "command": "search"
- }
- ]
- },
- "connections": {
- "OpenAI": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Switch": {
- "main": [
- [
- {
- "node": "Get Channel Details",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Get Video Description",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Get Comments",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Run Query",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Get Videos by Channel",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "OpenAI",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Get Video Transcription",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "search": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Run Query": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Edit Fields": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Comments": {
- "main": [
- [
- {
- "node": "Edit Fields",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "AI Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "analyze_thumbnail": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "get_list_of_videos": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Get Channel Details": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "get_channel_details": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "video_transcription": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Postgres Chat Memory": {
- "ai_memory": [
- [
- {
- "node": "AI Agent",
- "type": "ai_memory",
- "index": 0
- }
- ]
- ]
- },
- "get_list_of_comments": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Get Video Description": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Videos by Channel": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "get_video_description": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Get Video Transcription": {
- "main": [
- [
- {
- "node": "Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Switch",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When chat message received": {
- "main": [
- [
- {
- "node": "AI Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.txt b/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.txt
new file mode 100644
index 0000000..c0a38de
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Extract insights & analyse YouTube comments via AI Agent chat.txt
@@ -0,0 +1,1194 @@
+{
+"meta": {
+"instanceId": "6a2a7715680b8313f7cb4676321c5baa46680adfb913072f089f2766f42e43bd"
+},
+"nodes": [
+{
+"id": "f4b3833b-cf25-4bbc-927c-080586c5713c",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+700,
+760
+],
+"parameters": {
+"color": 7,
+"width": 330.5152611046425,
+"height": 239.5888196628349,
+"content": "### ... or watch set up video [13 min]\n[](https://youtu.be/6RmLZS8Yl4E)\n"
+},
+"typeVersion": 1
+},
+{
+"id": "64d96c53-b3e2-4aea-9a29-9b9e5c729f4f",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+400,
+240
+],
+"parameters": {
+"color": 7,
+"width": 636.2128494576581,
+"height": 497.1532689930921,
+"content": "\n## AI Agent To Chat With Youtube\n**Made by [Mark Shcherbakov](https://www.linkedin.com/in/marklowcoding/) from community [5minAI](https://www.skool.com/5minai)**\n\nNavigating the content generation and optimization process can be complex, especially without significant audience insight. This workflow automates insights extraction from YouTube videos and comments, empowering users to create more engaging and relevant content effectively.\n\nThe workflow integrates various APIs to gather insights from YouTube videos, enabling automated commentary analysis, video transcription, and thumbnail evaluation. The main functionalities include:\n- Extracting user preferences from comments.\n- Transcribing video content for enhanced understanding.\n- Analyzing thumbnails via AI for maximum viewer engagement insights.\n\n"
+},
+"typeVersion": 1
+},
+{
+"id": "57d2ede9-1bf9-4449-9dc9-af1ccee763b6",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+400,
+760
+],
+"parameters": {
+"color": 7,
+"width": 280.2462120317618,
+"height": 545.9087885077763,
+"content": "### Set up steps\n\n1. **API Setup**:\n - Create a [Google Cloud](https://console.cloud.google.com/apis/dashboard) project and enable the YouTube Data API.\n - Generate an API key for [Apify](https://www.apify.com?fpr=ujogj).\n - Generate API key for [OpenAI](https://platform.openai.com)\n - Create all credentials in N8N - OpenAI, Apify, Google Cloud.\n\n2. **YouTube Creator and Video Selection**:\n - Start by defining a request to identify top creators based on their video views.\n - Capture the YouTube video IDs for further analysis of comments and other video metrics.\n\n3. **Comment Analysis**:\n - Gather comments associated with the selected videos and analyze them for user insights.\n - Implement pagination to handle the maximum comment retrieval limits in API requests.\n\n4. **Video Transcription**:\n - Request transcriptions for videos of interest, ensuring to manage potential costs associated with longer video processing.\n - Utilize the insights from transcriptions to formulate content plans.\n\n5. **Thumbnail Analysis**:\n - Evaluate your video thumbnails by submitting the URL through the OpenAI API to gain insights into their effectiveness.\n\n6. **Data Management**:\n - Incorporate a database agent to organize video data and metrics, allowing efficient record management and future content planning."
+},
+"typeVersion": 1
+},
+{
+"id": "ca0fd549-88a7-44fd-ab81-7fd5ca140dae",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+1540,
+820
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "zJhr5piyEwVnWtaI",
+"name": "OpenAi club"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "7f2cf209-2e9d-4d6a-bc9e-d1bfd6df7266",
+"name": "get_channel_details",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+1900,
+820
+],
+"parameters": {
+"name": "get_channel_details",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "=get_channel_details"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Get channel_id, title and description by handle/username.\nChannel_id is required to find videos and details about this channel.\nIf Youtube link to channel provided - parse handle from there or return channel_id. (e.g. https://www.youtube.com/@example_handle - example_handle)\n\n\nExample Input:\nexample_handle\n\nExample Output:\nid:UCOgz_YflAsYnGbdvzXuKNCA\ntitle:Daniel Simmons\ndescription:Digital Diary ๐ค\\n\\n\\nWeekly videos around fashion...",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"handle\": {\n \"type\": \"string\",\n \"description\": \"Handle/username of channel\"\n }},\n \"required\": [\"handle\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "c02f5c19-6e50-4a06-95b9-eceb3eec1012",
+"name": "get_video_description",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2020,
+820
+],
+"parameters": {
+"name": "get_video_description",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "video_details"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Fetch video details - the full description, title, and publish date of a video using its video_id.\n\nExample input:\nvideo_id:dQw4w9WgXcQ\n\nExample Output:\ntitle:Never Gonna Give You Up\ndescription: \"The official video for โNever Gonna Give You Upโ by Rick Astley.\nduration:4 min\nviewCount:154\nlikeCount:6\nthumbnails: urls",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the video to fetch details for\"\n }\n },\n \"required\": [\"video_id\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "2d61160b-3a65-4766-ace6-947a7c5de6e5",
+"name": "get_list_of_videos",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2140,
+820
+],
+"parameters": {
+"name": "get_list_of_videos",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "videos"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Retrieve a list of videos from a channel using channel_id. Supports sorting by date, relevance, or view count.\n\nExample Input:\nchannel_id\": \"UCxxxxxxxxxxxxxxxx\"\nnumber_of_videos\": 5\norder: \"date\"\npublishedAfter: \"timestamp\"\n\nExample Output:\nvideo_id:abc123\ntitle:Latest Video\nshort cut description:Latest Video\npublished_at:2023-12-05T10:00:00Z",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"channel_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the channel to fetch videos from\"\n },\n \"number_of_videos\": {\n \"type\": \"integer\",\n \"description\": \"The maximum number of videos to retrieve (max 50)\"\n },\n \"order\": {\n \"type\": \"string\",\n \"enum\": [\"date\", \"relevance\", \"viewCount\"],\n \"description\": \"Order in which to fetch videos\"\n },\n \"publishedAfter\": {\n \"type\": \"string\",\n \"description\": \"Timestamp for filtering like 2023-11-03T15:28:05Z.\"\n }\n },\n \"required\": [\"channel_id\", \"number_of_videos\", \"order\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "c5aa2f7c-7748-4f88-abb6-fd274ad1295a",
+"name": "get_list_of_comments",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2260,
+820
+],
+"parameters": {
+"name": "get_list_of_comments",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "comments"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Retrieve a list of comments from a video using video_id.\n\nInput:\n \"video_id\": \"dQw4w9WgXcQ\"\n\nOutput:\n \"author\": \"John Doe\",\n \"comment\": \"This is an amazing video!\",\n \"published_at\": \"2023-12-04T12:00:00Z\"",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_id\": {\n \"type\": \"string\",\n \"description\": \"The ID of the video to fetch comments from\"\n }\n },\n \"required\": [\"video_id\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "c68cad77-1d71-45a3-b94b-8f7c701f56fb",
+"name": "search",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2380,
+820
+],
+"parameters": {
+"name": "search",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "search"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Search for videos or channels using a query. Supports filtering by type (video or channel) and sorting (date, viewCount, relevance). Use | for OR and - to exclude terms in the query.\n\nInput:\ntype: video or channel\nquery: search query\nsorting: date, viewCount, relevance\npublishedAfter: timestamp\n\nOutput:\n- id, title, short cut description, and published_at.",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\"video\", \"channel\"],\n \"description\": \"Type of results to retrieve: video or channel\"\n },\n \"query\": {\n \"type\": \"string\",\n \"description\": \"Search query. Supports | for OR and - to exclude terms\"\n },\n \"sorting\": {\n \"type\": \"string\",\n \"enum\": [\"date\", \"viewCount\", \"relevance\"],\n \"description\": \"Sorting criteria for search results\"\n },\n \"publishedAfter\": {\n \"type\": \"string\",\n \"description\": \"Timestamp for filtering like 2023-11-03T15:28:05Z\"\n }\n },\n \"required\": [\"type\", \"query\", \"sorting\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "c87d5392-8a5c-4999-9e58-89a5e0700c40",
+"name": "analyze_thumbnail",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2500,
+820
+],
+"parameters": {
+"name": "analyze_thumbnail",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "analyze_thumbnail"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Analyze a thumbnail image based on a given prompt. The prompt can be customized for specific analysis needs, such as design critique, color scheme evaluation, or content assessment.\nUse link of maxRes thumbnail. \n\nInput:\n- url: URL of the thumbnail image.\n- prompt: Customizable instruction for the analysis.\n\nOutput:\n- Results of the analysis based on the given prompt.",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"url\": {\n \"type\": \"string\",\n \"description\": \"URL of the thumbnail image to analyze\"\n },\n \"prompt\": {\n \"type\": \"string\",\n \"description\": \"Customizable instruction to guide the image analysis\"\n }\n },\n \"required\": [\"url\", \"prompt\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "1be2fa35-9091-4db8-a8eb-50f822d618d3",
+"name": "video_transcription",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2620,
+820
+],
+"parameters": {
+"name": "video_transcription",
+"fields": {
+"values": [
+{
+"name": "command",
+"stringValue": "video_transcription"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": {
+"__rl": true,
+"mode": "list",
+"value": "FgknOUpOBkpY85NX",
+"cachedResultName": "Youtube parser - tools"
+},
+"description": "Transcribe a video and retrieve its text transcription. Useful for analyzing video content or repurposing it for other formats.\n\nInput:\n- video_url: URL of the video to transcribe.\n\nOutput:\n- The text transcription of the video.",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"video_url\": {\n \"type\": \"string\",\n \"description\": \"URL of the video to transcribe\"\n }\n },\n \"required\": [\"video_url\"]\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.2
+},
+{
+"id": "fbfcd82f-e247-4a21-be12-339df7afe681",
+"name": "Postgres Chat Memory",
+"type": "@n8n/n8n-nodes-langchain.memoryPostgresChat",
+"position": [
+1700,
+820
+],
+"parameters": {
+"sessionKey": "={{ $('When chat message received').item.json.sessionId }}",
+"sessionIdType": "customKey"
+},
+"credentials": {
+"postgres": {
+"id": "AO9cER6p8uX7V07T",
+"name": "Postgres 5minai"
+}
+},
+"typeVersion": 1.3
+},
+{
+"id": "6a4bbad9-27ab-448b-9222-2c8843fe241a",
+"name": "AI Agent",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+1760,
+560
+],
+"parameters": {
+"text": "={{ $('When chat message received').item.json.chatInput }}",
+"agent": "openAiFunctionsAgent",
+"options": {
+"systemMessage": "You are Youtube assistant. \nYou need to process user's requests and run relevant tools for that. \n\nPlan and execute in right order runs of tools to get data for user's request.\n\nIMPORTANT Search query and list of videos for channel tools returns all videos including shorts - use Get Video description tool to identify shorts (less than minute) and filter them out if needed.\n\nFeel free to ask questions before do actions - especially if you noticed some inconcistency in user requests that might be error/misspelling. "
+},
+"promptType": "define"
+},
+"typeVersion": 1.6
+},
+{
+"id": "739cc12a-27d1-48e9-b124-7f83fb372514",
+"name": "When chat message received",
+"type": "@n8n/n8n-nodes-langchain.chatTrigger",
+"position": [
+1460,
+600
+],
+"webhookId": "6e95bc27-99a6-417c-8bf7-2831d7f7a4be",
+"parameters": {
+"options": {}
+},
+"typeVersion": 1.1
+},
+{
+"id": "613af9f2-77fa-42c4-86d3-87e20f2c0c89",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1380,
+500
+],
+"parameters": {
+"width": 1430.34590072234,
+"height": 588.1344471094899,
+"content": "## Scenario 1: AI agent"
+},
+"typeVersion": 1
+},
+{
+"id": "54116346-bc73-4a6a-8bca-f2a6e6699374",
+"name": "Get Comments",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+1598
+],
+"parameters": {
+"url": "=https://www.googleapis.com/youtube/v3/commentThreads",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "part",
+"value": "id,snippet,replies"
+},
+{
+"name": "videoId",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.video_id }}"
+},
+{
+"name": "maxResults",
+"value": "100"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "1DXeuNaLSixqGPaU",
+"name": "Query Auth account Youtube"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "faabf71a-69f2-4113-802e-124a09fa9a0a",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+1444,
+1598
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "4b3ec3aa-7c69-4a72-a989-02f97acdf612",
+"name": "Get Channel Details",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+1278
+],
+"parameters": {
+"url": "=https://www.googleapis.com/youtube/v3/channels",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "part",
+"value": "snippet"
+},
+{
+"name": "forHandle",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.handle }}"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "1DXeuNaLSixqGPaU",
+"name": "Query Auth account Youtube"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "ed8dec73-8c50-4eb9-8efe-68ee72c4d5e6",
+"name": "Get Video Description",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+1438
+],
+"parameters": {
+"url": "=https://www.googleapis.com/youtube/v3/videos",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "part",
+"value": "snippet,contentDetails,statistics"
+},
+{
+"name": "id",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.video_id }}"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "1DXeuNaLSixqGPaU",
+"name": "Query Auth account Youtube"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "c1ff3837-8d7e-49ad-a333-c177833fcd05",
+"name": "Edit Fields",
+"type": "n8n-nodes-base.set",
+"position": [
+2224,
+1598
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "469d89ba-23fc-482a-b4ae-ce5d3bc13579",
+"name": "response",
+"type": "string",
+"value": "={{ JSON.stringify(` Comments: ${$json.items.map(item => { const topLevelComment = `${item.snippet.topLevelComment.snippet.authorDisplayName}: ${item.snippet.topLevelComment.snippet.textOriginal}`; const replies = item.replies?.comments.map(reply => `${reply.snippet.authorDisplayName}: ${reply.snippet.textOriginal}` ).join('\\n') || ''; return [topLevelComment, replies].filter(Boolean).join('\\n'); }).join('\\n\\n')} `) }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "5f0c44fe-2523-4170-a27d-0ccd1bef24a7",
+"name": "Run Query",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+1758
+],
+"parameters": {
+"url": "=https://www.googleapis.com/youtube/v3/search",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "part",
+"value": "snippet"
+},
+{
+"name": "q",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.query }}"
+},
+{
+"name": "order",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.order }}"
+},
+{
+"name": "type",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.type }}"
+},
+{
+"name": "maxResults",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.number_of_videos }}"
+},
+{
+"name": "publishedAfter",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.publishedAfter }}"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "1DXeuNaLSixqGPaU",
+"name": "Query Auth account Youtube"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "3e192718-6710-4143-ac6e-15df79ee5363",
+"name": "Get Videos by Channel",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+1918
+],
+"parameters": {
+"url": "=https://www.googleapis.com/youtube/v3/search",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "part",
+"value": "snippet"
+},
+{
+"name": "channelId",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.channel_id }}"
+},
+{
+"name": "order",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.order }}"
+},
+{
+"name": "maxResults",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.number_of_videos }}"
+},
+{
+"name": "type",
+"value": "video"
+},
+{
+"name": "publishedAfter",
+"value": "={{ $('Execute Workflow Trigger').item.json.query.publishedAfter }}"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "1DXeuNaLSixqGPaU",
+"name": "Query Auth account Youtube"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8bcb50a4-0cd1-4311-ac6a-2ee8653cfb71",
+"name": "Response",
+"type": "n8n-nodes-base.set",
+"position": [
+2564,
+1598
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "cfdbe2f5-921e-496d-87bd-9c57fdc22a7a",
+"name": "response",
+"type": "object",
+"value": "={{$json}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "7f5a36d3-6710-4e69-8459-7c8c748ee7d9",
+"name": "Switch",
+"type": "n8n-nodes-base.switch",
+"position": [
+1624,
+1578
+],
+"parameters": {
+"rules": {
+"values": [
+{
+"outputKey": "get_channel_details",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"operator": {
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "get_channel_details"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "video_details",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "26a3ffe8-c8a6-4564-8d18-5494a8059372",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "video_details"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "comments",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "0f51cc26-2e42-42e1-a5c2-cb1d2e384962",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "comments"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "search",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "51031140-5ceb-48aa-9f33-d314131a9653",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "search"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "videos",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f160bf0a-423f-448d-ab80-50a0b6a177ca",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "videos"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "analyze_thumbnail",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "29542ac4-7b9d-413f-aabb-a1cdabed2fa7",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "analyze_thumbnail"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "video_transcription",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "35fc39b8-6cf1-4ea6-9609-4a195c5526f8",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $('Execute Workflow Trigger').item.json.command }}",
+"rightValue": "video_transcription"
+}
+]
+},
+"renameOutput": true
+}
+]
+},
+"options": {}
+},
+"typeVersion": 3.2
+},
+{
+"id": "df432d53-33bf-4e91-9ead-7f4b36bd788a",
+"name": "Get Video Transcription",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2064,
+2238
+],
+"parameters": {
+"url": "=https://api.apify.com/v2/acts/dB9f4B02ocpTICIEY/run-sync-get-dataset-items",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"startUrls\": [\n \"{{ $('Execute Workflow Trigger').item.json.query.video_url }}\"\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth"
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "XDavOaI9qH5Zi3QC",
+"name": "Apify"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8079e5c9-4a52-45ce-ac41-7fc707177a5a",
+"name": "OpenAI",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+2064,
+2078
+],
+"parameters": {
+"text": "={{ $('Execute Workflow Trigger').item.json.query.prompt }}",
+"modelId": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o",
+"cachedResultName": "GPT-4O"
+},
+"options": {},
+"resource": "image",
+"imageUrls": "={{ $('Execute Workflow Trigger').item.json.query.url }}",
+"operation": "analyze"
+},
+"credentials": {
+"openAiApi": {
+"id": "SphXAX7rlwRLkiox",
+"name": "Test club key"
+}
+},
+"typeVersion": 1.7
+},
+{
+"id": "7847e82a-fe82-498c-8c14-4c1c718d632c",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1380,
+1140
+],
+"parameters": {
+"width": 1427.3810326521016,
+"height": 1313.2689194736308,
+"content": "## Scenario 2: Agent tools"
+},
+"typeVersion": 1
+},
+{
+"id": "3a0fbbb0-4c0e-41f1-abb3-c87e955ad1b3",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1540,
+960
+],
+"parameters": {
+"color": 4,
+"width": 266.7375650720483,
+"height": 80,
+"content": "### Replace credentials"
+},
+"typeVersion": 1
+},
+{
+"id": "363eaca0-aaa5-4551-845f-528f19bba57a",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2004,
+1178
+],
+"parameters": {
+"color": 4,
+"width": 266.7375650720483,
+"height": 80,
+"content": "### Replace credentials in all nodes - Apify, OpenAI, Google"
+},
+"typeVersion": 1
+}
+],
+"pinData": {
+"Execute Workflow Trigger": [
+{
+"query": {
+"type": "video",
+"query": "Web scraping data with n8n and Puppeteer",
+"sorting": "relevance"
+},
+"command": "search"
+}
+]
+},
+"connections": {
+"OpenAI": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Switch": {
+"main": [
+[
+{
+"node": "Get Channel Details",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Video Description",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Comments",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Run Query",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Videos by Channel",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "OpenAI",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Video Transcription",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"search": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Run Query": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Edit Fields": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Comments": {
+"main": [
+[
+{
+"node": "Edit Fields",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "AI Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"analyze_thumbnail": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"get_list_of_videos": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Get Channel Details": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"get_channel_details": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"video_transcription": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Postgres Chat Memory": {
+"ai_memory": [
+[
+{
+"node": "AI Agent",
+"type": "ai_memory",
+"index": 0
+}
+]
+]
+},
+"get_list_of_comments": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Get Video Description": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Videos by Channel": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"get_video_description": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Get Video Transcription": {
+"main": [
+[
+{
+"node": "Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Switch",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When chat message received": {
+"main": [
+[
+{
+"node": "AI Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.json b/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.json
deleted file mode 100644
index cbd100e..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.json
+++ /dev/null
@@ -1,338 +0,0 @@
-{
- "meta": {
- "instanceId": "257476b1ef58bf3cb6a46e65fac7ee34a53a5e1a8492d5c6e4da5f87c9b82833",
- "templateId": "2473"
- },
- "nodes": [
- {
- "id": "1205b121-8aaa-4e41-874b-4e81aad6374e",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 120,
- 600
- ],
- "parameters": {
- "color": 4,
- "width": 462.4041757955455,
- "height": 315.6388466176832,
- "content": "## Generate SEO Seed Keywords Using AI\n\nThis flow uses an AI node to generate Seed Keywords to focus SEO efforts on based on your ideal customer profile\n\n**Outputs:** \n- List of 20 Seed Keywords\n\n\n**Pre-requisites / Dependencies:**\n- You know your ideal customer profile (ICP)\n- An AI API account (either OpenAI or Anthropic recommended)"
- },
- "typeVersion": 1
- },
- {
- "id": "d2654d75-2b64-4ec3-b583-57d2b6b7b195",
- "name": "Sticky Note13",
- "type": "n8n-nodes-base.stickyNote",
- "disabled": true,
- "position": [
- 640,
- 920
- ],
- "parameters": {
- "color": 7,
- "width": 287.0816455493243,
- "height": 330.47923074942287,
- "content": "**Generate draft seed KW based on ICP**\n\n"
- },
- "typeVersion": 1
- },
- {
- "id": "d248a58e-3705-4b6f-99cb-e9187e56781c",
- "name": "Anthropic Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatAnthropic",
- "position": [
- 680,
- 1120
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1.2
- },
- {
- "id": "71517d83-59f5-441a-8a75-c35f4e06a8a2",
- "name": "Split Out",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 980,
- 980
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "output.answer"
- },
- "typeVersion": 1
- },
- {
- "id": "1c68eff5-6478-4eba-9abe-3ccea2a17a5c",
- "name": "Sticky Note12",
- "type": "n8n-nodes-base.stickyNote",
- "disabled": true,
- "position": [
- 120,
- 920
- ],
- "parameters": {
- "color": 7,
- "width": 492.16246201447336,
- "height": 213.62075341687063,
- "content": "**Get data from airtable and format** "
- },
- "typeVersion": 1
- },
- {
- "id": "53dcc524-ef3d-40b8-b79d-976517dce4e7",
- "name": "Sticky Note17",
- "type": "n8n-nodes-base.stickyNote",
- "disabled": true,
- "position": [
- 960,
- 920
- ],
- "parameters": {
- "color": 7,
- "width": 348.42891651921957,
- "height": 213.62075341687063,
- "content": "**Add data to database**"
- },
- "typeVersion": 1
- },
- {
- "id": "570495fe-3f1d-44ae-bea0-9fa4b2ce15ef",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 640,
- 820
- ],
- "parameters": {
- "color": 6,
- "width": 393.46745700785266,
- "height": 80,
- "content": "**Costs to run**\nApprox. $0.02-0.05 for a run using Claude Sonnet 3.5"
- },
- "typeVersion": 1
- },
- {
- "id": "6e5e84c5-409f-4f37-931a-21a44aff7c5e",
- "name": "Set Ideal Customer Profile (ICP)",
- "type": "n8n-nodes-base.set",
- "position": [
- 160,
- 980
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "973e949e-1afd-4378-8482-d2168532eff6",
- "name": "product",
- "type": "string",
- "value": "=**Replace this with a string detailing your intended product (if you have one)**"
- },
- {
- "id": "ce9c0a8f-6157-4b46-8b77-133545dc71bd",
- "name": "pain points",
- "type": "string",
- "value": "=**Replace this with a string list of customer pain points**"
- },
- {
- "id": "5abc858a-c412-4acf-acb9-488e4d992d2f",
- "name": "goals",
- "type": "string",
- "value": "=**Replace this with a string list of your customers key goals/objectives**"
- },
- {
- "id": "fbdd1ef7-c1b9-48eb-b73e-a383f12b5ba1",
- "name": "current solutions",
- "type": "string",
- "value": "=**Replace this with a string detailing how your ideal customer currently solves their pain ppoints**"
- },
- {
- "id": "2e5c8f48-266e-486c-956f-51f1449f6288",
- "name": "expertise level",
- "type": "string",
- "value": "=**Replace this with a string detailing customer level of expertise**"
- }
- ]
- }
- },
- "notesInFlow": true,
- "typeVersion": 3.4
- },
- {
- "id": "bd5781f4-6f35-45d3-8182-12ea6712eddf",
- "name": "Aggregate for AI node",
- "type": "n8n-nodes-base.aggregate",
- "position": [
- 380,
- 980
- ],
- "parameters": {
- "options": {},
- "aggregate": "aggregateAllItemData"
- },
- "notesInFlow": true,
- "typeVersion": 1
- },
- {
- "id": "244943bf-e4dd-40fc-9a43-7a5cd0da1c5b",
- "name": "Sticky Note14",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 640,
- 1260
- ],
- "parameters": {
- "color": 3,
- "width": 284.87764467541297,
- "height": 80,
- "content": "**REQUIRED**\nConnect to your own AI API above"
- },
- "typeVersion": 1
- },
- {
- "id": "73c8f47a-4fdb-40c8-9062-890ef1265ab0",
- "name": "Sticky Note16",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 120,
- 1140
- ],
- "parameters": {
- "color": 3,
- "width": 284.87764467541297,
- "height": 80,
- "content": "**REQUIRED**\nSet your Ideal Customer Profile before proceeding"
- },
- "typeVersion": 1
- },
- {
- "id": "a5b93e6d-44ab-4b6f-b86a-25dc621b52b0",
- "name": "AI Agent",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 660,
- 980
- ],
- "parameters": {
- "text": "=User:\nHere are some important rules for you to follow:\n\n1. Analyze the ICP information carefully.\n2. Generate 15-20 seed keywords that are relevant to the ICP's needs, challenges, goals, and search behavior.\n3. Ensure the keywords are broad enough to be considered \"\"head\"\" terms, but specific enough to target the ICP effectively.\n4. Consider various aspects of the ICP's journey, including awareness, consideration, and decision stages.\n5. Include a mix of product-related, problem-related, and solution-related terms.\n6. Think beyond just the product itself - consider industry trends, related technologies, and broader business concepts that would interest the ICP.\n7. Avoid overly generic terms that might attract irrelevant traffic.\n8. Aim for a mix of keyword difficulties, including both competitive and less competitive terms.\n9. Include keywords that cover different search intents: informational, navigational, commercial, and transactional.\n10. Consider related tools or platforms that the ICP might use, and include relevant integration-related keywords.\n11. If applicable, include some location-specific keywords based on the ICP's geographic information.\n12. Incorporate industry-specific terminology or jargon that the ICP would likely use in their searches.\n13. Consider emerging trends or pain points in the ICP's industry that they might be searching for solutions to.\n13. Format the keywords in lowercase, without punctuation. Trim any leading or trailing white space.\n\n\nYour output should be an array of strings, each representing a seed keyword:\n\n['b2b lead generation', 'startup marketing strategies', 'saas sales funnel', ...]\n\n\nHere is the Ideal Customer Profile (ICP) information:\n\n{{ $json.data[0].product }}\n\n\nNow:\nBased on the provided ICP, generate an array of 15-20 seed keywords that will form the foundation of a comprehensive SEO strategy for this B2B SaaS company. These keywords should reflect a deep understanding of the ICP's needs, challenges, and search behavior, while also considering broader industry trends and related concepts.\n\nFirst, write out your ideas in {thoughts: } JSON as part of your analysis, then answer inside the {answer: } key in the JSON. ",
- "agent": "conversationalAgent",
- "options": {
- "systemMessage": "=System: You are an expert SEO strategist tasked with generating 15-20 key head search terms (seed keywords) for a B2B SaaS company. Your goal is to create a comprehensive list of keywords that will attract and engage the ideal customer profile (ICP) described."
- },
- "promptType": "define"
- },
- "typeVersion": 1.6
- },
- {
- "id": "ca3c0bd5-7ef0-4e2b-9b5e-071773c32c85",
- "name": "Connect to your own database",
- "type": "n8n-nodes-base.noOp",
- "position": [
- 1140,
- 980
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "94639a81-5e46-482a-851a-5443bfe9863c",
- "name": "Sticky Note15",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1120,
- 1140
- ],
- "parameters": {
- "color": 3,
- "width": 284.87764467541297,
- "height": 80,
- "content": "**REQUIRED**\nConnect to your own database / GSheet / Airtable base to output these"
- },
- "typeVersion": 1
- },
- {
- "id": "16498e92-c0d5-44f4-b993-c9c8930955bc",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -60,
- 980
- ],
- "parameters": {},
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "AI Agent": {
- "main": [
- [
- {
- "node": "Split Out",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Out": {
- "main": [
- [
- {
- "node": "Connect to your own database",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Anthropic Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "AI Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Aggregate for AI node": {
- "main": [
- [
- {
- "node": "AI Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Ideal Customer Profile (ICP)": {
- "main": [
- [
- {
- "node": "Aggregate for AI node",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Set Ideal Customer Profile (ICP)",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.txt b/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.txt
new file mode 100644
index 0000000..e9005c2
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Generate SEO Seed Keywords Using AI.txt
@@ -0,0 +1,338 @@
+{
+"meta": {
+"instanceId": "257476b1ef58bf3cb6a46e65fac7ee34a53a5e1a8492d5c6e4da5f87c9b82833",
+"templateId": "2473"
+},
+"nodes": [
+{
+"id": "1205b121-8aaa-4e41-874b-4e81aad6374e",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+120,
+600
+],
+"parameters": {
+"color": 4,
+"width": 462.4041757955455,
+"height": 315.6388466176832,
+"content": "## Generate SEO Seed Keywords Using AI\n\nThis flow uses an AI node to generate Seed Keywords to focus SEO efforts on based on your ideal customer profile\n\n**Outputs:** \n- List of 20 Seed Keywords\n\n\n**Pre-requisites / Dependencies:**\n- You know your ideal customer profile (ICP)\n- An AI API account (either OpenAI or Anthropic recommended)"
+},
+"typeVersion": 1
+},
+{
+"id": "d2654d75-2b64-4ec3-b583-57d2b6b7b195",
+"name": "Sticky Note13",
+"type": "n8n-nodes-base.stickyNote",
+"disabled": true,
+"position": [
+640,
+920
+],
+"parameters": {
+"color": 7,
+"width": 287.0816455493243,
+"height": 330.47923074942287,
+"content": "**Generate draft seed KW based on ICP**\n\n"
+},
+"typeVersion": 1
+},
+{
+"id": "d248a58e-3705-4b6f-99cb-e9187e56781c",
+"name": "Anthropic Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatAnthropic",
+"position": [
+680,
+1120
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1.2
+},
+{
+"id": "71517d83-59f5-441a-8a75-c35f4e06a8a2",
+"name": "Split Out",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+980,
+980
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output.answer"
+},
+"typeVersion": 1
+},
+{
+"id": "1c68eff5-6478-4eba-9abe-3ccea2a17a5c",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"disabled": true,
+"position": [
+120,
+920
+],
+"parameters": {
+"color": 7,
+"width": 492.16246201447336,
+"height": 213.62075341687063,
+"content": "**Get data from airtable and format** "
+},
+"typeVersion": 1
+},
+{
+"id": "53dcc524-ef3d-40b8-b79d-976517dce4e7",
+"name": "Sticky Note17",
+"type": "n8n-nodes-base.stickyNote",
+"disabled": true,
+"position": [
+960,
+920
+],
+"parameters": {
+"color": 7,
+"width": 348.42891651921957,
+"height": 213.62075341687063,
+"content": "**Add data to database**"
+},
+"typeVersion": 1
+},
+{
+"id": "570495fe-3f1d-44ae-bea0-9fa4b2ce15ef",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+640,
+820
+],
+"parameters": {
+"color": 6,
+"width": 393.46745700785266,
+"height": 80,
+"content": "**Costs to run**\nApprox. $0.02-0.05 for a run using Claude Sonnet 3.5"
+},
+"typeVersion": 1
+},
+{
+"id": "6e5e84c5-409f-4f37-931a-21a44aff7c5e",
+"name": "Set Ideal Customer Profile (ICP)",
+"type": "n8n-nodes-base.set",
+"position": [
+160,
+980
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "973e949e-1afd-4378-8482-d2168532eff6",
+"name": "product",
+"type": "string",
+"value": "=**Replace this with a string detailing your intended product (if you have one)**"
+},
+{
+"id": "ce9c0a8f-6157-4b46-8b77-133545dc71bd",
+"name": "pain points",
+"type": "string",
+"value": "=**Replace this with a string list of customer pain points**"
+},
+{
+"id": "5abc858a-c412-4acf-acb9-488e4d992d2f",
+"name": "goals",
+"type": "string",
+"value": "=**Replace this with a string list of your customers key goals/objectives**"
+},
+{
+"id": "fbdd1ef7-c1b9-48eb-b73e-a383f12b5ba1",
+"name": "current solutions",
+"type": "string",
+"value": "=**Replace this with a string detailing how your ideal customer currently solves their pain ppoints**"
+},
+{
+"id": "2e5c8f48-266e-486c-956f-51f1449f6288",
+"name": "expertise level",
+"type": "string",
+"value": "=**Replace this with a string detailing customer level of expertise**"
+}
+]
+}
+},
+"notesInFlow": true,
+"typeVersion": 3.4
+},
+{
+"id": "bd5781f4-6f35-45d3-8182-12ea6712eddf",
+"name": "Aggregate for AI node",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+380,
+980
+],
+"parameters": {
+"options": {},
+"aggregate": "aggregateAllItemData"
+},
+"notesInFlow": true,
+"typeVersion": 1
+},
+{
+"id": "244943bf-e4dd-40fc-9a43-7a5cd0da1c5b",
+"name": "Sticky Note14",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+640,
+1260
+],
+"parameters": {
+"color": 3,
+"width": 284.87764467541297,
+"height": 80,
+"content": "**REQUIRED**\nConnect to your own AI API above"
+},
+"typeVersion": 1
+},
+{
+"id": "73c8f47a-4fdb-40c8-9062-890ef1265ab0",
+"name": "Sticky Note16",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+120,
+1140
+],
+"parameters": {
+"color": 3,
+"width": 284.87764467541297,
+"height": 80,
+"content": "**REQUIRED**\nSet your Ideal Customer Profile before proceeding"
+},
+"typeVersion": 1
+},
+{
+"id": "a5b93e6d-44ab-4b6f-b86a-25dc621b52b0",
+"name": "AI Agent",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+660,
+980
+],
+"parameters": {
+"text": "=User:\nHere are some important rules for you to follow:\n\n1. Analyze the ICP information carefully.\n2. Generate 15-20 seed keywords that are relevant to the ICP's needs, challenges, goals, and search behavior.\n3. Ensure the keywords are broad enough to be considered \"\"head\"\" terms, but specific enough to target the ICP effectively.\n4. Consider various aspects of the ICP's journey, including awareness, consideration, and decision stages.\n5. Include a mix of product-related, problem-related, and solution-related terms.\n6. Think beyond just the product itself - consider industry trends, related technologies, and broader business concepts that would interest the ICP.\n7. Avoid overly generic terms that might attract irrelevant traffic.\n8. Aim for a mix of keyword difficulties, including both competitive and less competitive terms.\n9. Include keywords that cover different search intents: informational, navigational, commercial, and transactional.\n10. Consider related tools or platforms that the ICP might use, and include relevant integration-related keywords.\n11. If applicable, include some location-specific keywords based on the ICP's geographic information.\n12. Incorporate industry-specific terminology or jargon that the ICP would likely use in their searches.\n13. Consider emerging trends or pain points in the ICP's industry that they might be searching for solutions to.\n13. Format the keywords in lowercase, without punctuation. Trim any leading or trailing white space.\n\n\nYour output should be an array of strings, each representing a seed keyword:\n\n['b2b lead generation', 'startup marketing strategies', 'saas sales funnel', ...]\n\n\nHere is the Ideal Customer Profile (ICP) information:\n\n{{ $json.data[0].product }}\n\n\nNow:\nBased on the provided ICP, generate an array of 15-20 seed keywords that will form the foundation of a comprehensive SEO strategy for this B2B SaaS company. These keywords should reflect a deep understanding of the ICP's needs, challenges, and search behavior, while also considering broader industry trends and related concepts.\n\nFirst, write out your ideas in {thoughts: } JSON as part of your analysis, then answer inside the {answer: } key in the JSON. ",
+"agent": "conversationalAgent",
+"options": {
+"systemMessage": "=System: You are an expert SEO strategist tasked with generating 15-20 key head search terms (seed keywords) for a B2B SaaS company. Your goal is to create a comprehensive list of keywords that will attract and engage the ideal customer profile (ICP) described."
+},
+"promptType": "define"
+},
+"typeVersion": 1.6
+},
+{
+"id": "ca3c0bd5-7ef0-4e2b-9b5e-071773c32c85",
+"name": "Connect to your own database",
+"type": "n8n-nodes-base.noOp",
+"position": [
+1140,
+980
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "94639a81-5e46-482a-851a-5443bfe9863c",
+"name": "Sticky Note15",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1120,
+1140
+],
+"parameters": {
+"color": 3,
+"width": 284.87764467541297,
+"height": 80,
+"content": "**REQUIRED**\nConnect to your own database / GSheet / Airtable base to output these"
+},
+"typeVersion": 1
+},
+{
+"id": "16498e92-c0d5-44f4-b993-c9c8930955bc",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-60,
+980
+],
+"parameters": {},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"AI Agent": {
+"main": [
+[
+{
+"node": "Split Out",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Out": {
+"main": [
+[
+{
+"node": "Connect to your own database",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Anthropic Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "AI Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Aggregate for AI node": {
+"main": [
+[
+{
+"node": "AI Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Ideal Customer Profile (ICP)": {
+"main": [
+[
+{
+"node": "Aggregate for AI node",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Set Ideal Customer Profile (ICP)",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.json b/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.json
deleted file mode 100644
index 756cbe5..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.json
+++ /dev/null
@@ -1,745 +0,0 @@
-{
- "id": "0JsHmmyeHw5Ffz5m",
- "meta": {
- "instanceId": "d4d7965840e96e50a3e02959a8487c692901dfa8d5cc294134442c67ce1622d3",
- "templateCredsSetupCompleted": true
- },
- "name": "HN Who is Hiring Scrape",
- "tags": [],
- "nodes": [
- {
- "id": "f7cdb3ee-9bb0-4006-829a-d4ce797191d5",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -20,
- -220
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "0475e25d-9bf4-450d-abd3-a04608a438a4",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 60,
- -620
- ],
- "parameters": {
- "width": 460,
- "height": 340,
- "content": "## Go to https://hn.algolia.com\n- filter by \"Ask HN: Who is hiring?\" (important with quotes for full match)\n- sort by date\n- Chrome Network Tab > find API call > click \"Copy as cURL\"\n- n8n HTTP node -> import cURL and paste \n- I've set the API key as Header Auth so you will have to do the above yourself to make this work"
- },
- "typeVersion": 1
- },
- {
- "id": "a686852b-ff84-430b-92bb-ce02a6808e19",
- "name": "Split Out",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 400,
- -220
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "hits"
- },
- "typeVersion": 1
- },
- {
- "id": "cdaaa738-d561-4fa0-b2c7-8ea9e6778eb1",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1260,
- -620
- ],
- "parameters": {
- "width": 500,
- "height": 340,
- "content": "## Go to HN API \nhttps://github.com/HackerNews/API\n\nWe'll need following endpoints: \n- For example, a story: https://hacker-news.firebaseio.com/v0/item/8863.json?print=pretty\n- comment: https://hacker-news.firebaseio.com/v0/item/2921983.json?print=pretty\n\n"
- },
- "typeVersion": 1
- },
- {
- "id": "4f353598-9e32-4be4-9e7b-c89cc05305fd",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2680,
- -20
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "Fbb2ueT0XP5xMRme",
- "name": "OpenAi account 2"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "5bd0d7cc-497a-497c-aa4c-589d9ceeca14",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 2840,
- -20
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"company\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Name of the hiring company\"\n },\n \"title\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Job title/role being advertised\"\n },\n \"location\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Work location including remote/hybrid status\"\n },\n \"type\": {\n \"type\": [\n \"string\",\n null\n ],\n \"enum\": [\n \"FULL_TIME\",\n \"PART_TIME\",\n \"CONTRACT\",\n \"INTERNSHIP\",\n \"FREELANCE\",\n null\n ],\n \"description\": \"Employment type (Full-time, Contract, etc)\"\n },\n \"work_location\": {\n \"type\": [\n \"string\",\n null\n ],\n \"enum\": [\n \"REMOTE\",\n \"HYBRID\",\n \"ON_SITE\",\n null\n ],\n \"description\": \"Work arrangement type\"\n },\n \"salary\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Compensation details if provided\"\n },\n \"description\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Main job description text including requirements and team info\"\n },\n \"apply_url\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Direct application/job posting URL\"\n },\n \"company_url\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Company website or careers page\"\n }\n }\n}\n"
- },
- "typeVersion": 1.2
- },
- {
- "id": "b84ca004-6f3b-4577-8910-61b8584b161d",
- "name": "Search for Who is hiring posts",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 200,
- -220
- ],
- "parameters": {
- "url": "https://uj5wyc0l7x-dsn.algolia.net/1/indexes/Item_dev_sort_date/query",
- "method": "POST",
- "options": {},
- "jsonBody": "{\n \"query\": \"\\\"Ask HN: Who is hiring\\\"\",\n \"analyticsTags\": [\n \"web\"\n ],\n \"page\": 0,\n \"hitsPerPage\": 30,\n \"minWordSizefor1Typo\": 4,\n \"minWordSizefor2Typos\": 8,\n \"advancedSyntax\": true,\n \"ignorePlurals\": false,\n \"clickAnalytics\": true,\n \"minProximity\": 7,\n \"numericFilters\": [],\n \"tagFilters\": [\n [\n \"story\"\n ],\n []\n ],\n \"typoTolerance\": \"min\",\n \"queryType\": \"prefixNone\",\n \"restrictSearchableAttributes\": [\n \"title\",\n \"comment_text\",\n \"url\",\n \"story_text\",\n \"author\"\n ],\n \"getRankingInfo\": true\n}",
- "sendBody": true,
- "sendQuery": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "x-algolia-agent",
- "value": "Algolia for JavaScript (4.13.1); Browser (lite)"
- },
- {
- "name": "x-algolia-application-id",
- "value": "UJ5WYC0L7X"
- }
- ]
- },
- "headerParameters": {
- "parameters": [
- {
- "name": "Accept",
- "value": "*/*"
- },
- {
- "name": "Accept-Language",
- "value": "en-GB,en-US;q=0.9,en;q=0.8"
- },
- {
- "name": "Connection",
- "value": "keep-alive"
- },
- {
- "name": "DNT",
- "value": "1"
- },
- {
- "name": "Origin",
- "value": "https://hn.algolia.com"
- },
- {
- "name": "Referer",
- "value": "https://hn.algolia.com/"
- },
- {
- "name": "Sec-Fetch-Dest",
- "value": "empty"
- },
- {
- "name": "Sec-Fetch-Mode",
- "value": "cors"
- },
- {
- "name": "Sec-Fetch-Site",
- "value": "cross-site"
- },
- {
- "name": "User-Agent",
- "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
- },
- {
- "name": "sec-ch-ua",
- "value": "\"Chromium\";v=\"133\", \"Not(A:Brand\";v=\"99\""
- },
- {
- "name": "sec-ch-ua-mobile",
- "value": "?0"
- },
- {
- "name": "sec-ch-ua-platform",
- "value": "\"macOS\""
- }
- ]
- }
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "oVEXp2ZbYCXypMVz",
- "name": "Algolia Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "205e66f6-cd6b-4cfd-a6ec-2226c35ddaac",
- "name": "Get relevant data",
- "type": "n8n-nodes-base.set",
- "position": [
- 700,
- -220
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "73dd2325-faa7-4650-bd78-5fc97cc202de",
- "name": "title",
- "type": "string",
- "value": "={{ $json.title }}"
- },
- {
- "id": "44918eac-4510-440e-9ac0-bf14d2b2f3af",
- "name": "createdAt",
- "type": "string",
- "value": "={{ $json.created_at }}"
- },
- {
- "id": "00eb6f09-2c22-411c-949c-886b2d95b6eb",
- "name": "updatedAt",
- "type": "string",
- "value": "={{ $json.updated_at }}"
- },
- {
- "id": "2b4f9da6-f60e-46e0-ba9d-3242fa955a55",
- "name": "storyId",
- "type": "string",
- "value": "={{ $json.story_id }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "16bc5628-8a29-4eac-8be9-b4e9da802e1e",
- "name": "Get latest post",
- "type": "n8n-nodes-base.filter",
- "position": [
- 900,
- -220
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "d7dd7175-2a50-45aa-bd3e-4c248c9193c4",
- "operator": {
- "type": "dateTime",
- "operation": "after"
- },
- "leftValue": "={{ $json.createdAt }}",
- "rightValue": "={{$now.minus({days: 30})}} "
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "92e1ef74-5ae1-4195-840b-115184db464f",
- "name": "Split out children (jobs)",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 1460,
- -220
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "kids"
- },
- "typeVersion": 1
- },
- {
- "id": "d0836aae-b98a-497f-a6f7-0ad563c262a0",
- "name": "Trun into structured data",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 2600,
- -220
- ],
- "parameters": {
- "text": "={{ $json.cleaned_text }}",
- "messages": {
- "messageValues": [
- {
- "message": "Extract the JSON data"
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.5
- },
- {
- "id": "fd818a93-627c-435d-91ba-5d759d5a9004",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2600,
- -620
- ],
- "parameters": {
- "width": 840,
- "height": 340,
- "content": "## Data Structure\n\nWe use Openai GPT-4o-mini to transform the raw data in a unified data structure. Feel free to change this.\n\n```json\n{\n \"company\": \"Name of the hiring company\",\n \"title\": \"Job title/role being advertised\",\n \"location\": \"Work location including remote/hybrid status\",\n \"type\": \"Employment type (Full-time, Contract, etc)\",\n \"salary\": \"Compensation details if provided\",\n \"description\": \"Main job description text including requirements and team info\",\n \"apply_url\": \"Direct application/job posting URL\",\n \"company_url\": \"Company website or careers page\"\n}\n```"
- },
- "typeVersion": 1
- },
- {
- "id": "b70c5578-5b81-467a-8ac2-65374e4e52f3",
- "name": "Extract text",
- "type": "n8n-nodes-base.set",
- "position": [
- 1860,
- -220
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "6affa370-56ce-4ad8-8534-8f753fdf07fc",
- "name": "text",
- "type": "string",
- "value": "={{ $json.text }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "acb68d88-9417-42e9-9bcc-7c2fa95c4afd",
- "name": "Clean text",
- "type": "n8n-nodes-base.code",
- "position": [
- 2060,
- -220
- ],
- "parameters": {
- "jsCode": "// In a Function node in n8n\nconst inputData = $input.all();\n\nfunction cleanAllPosts(data) {\n return data.map(item => {\n try {\n // Check if item exists and has the expected structure\n if (!item || typeof item !== 'object') {\n return { cleaned_text: '', error: 'Invalid item structure' };\n }\n\n // Get the text, with multiple fallbacks\n let text = '';\n if (typeof item === 'string') {\n text = item;\n } else if (item.json && item.json.text) {\n text = item.json.text;\n } else if (typeof item.json === 'string') {\n text = item.json;\n } else {\n text = JSON.stringify(item);\n }\n\n // Make sure text is a string\n text = String(text);\n \n // Perform the cleaning operations\n try {\n text = text.replace(///g, '/');\n text = text.replace(/'/g, \"'\");\n text = text.replace(/&\\w+;/g, ' ');\n text = text.replace(/<[^>]*>/g, '');\n text = text.replace(/\\|\\s*/g, '| ');\n text = text.replace(/\\s+/g, ' ');\n text = text.replace(/\\s*(https?:\\/\\/[^\\s]+)\\s*/g, '\\n$1\\n');\n text = text.replace(/\\n{3,}/g, '\\n\\n');\n text = text.trim();\n } catch (cleaningError) {\n console.log('Error during text cleaning:', cleaningError);\n // Return original text if cleaning fails\n return { cleaned_text: text, warning: 'Partial cleaning applied' };\n }\n\n return { cleaned_text: text };\n \n } catch (error) {\n console.log('Error processing item:', error);\n return { \n cleaned_text: '', \n error: `Processing error: ${error.message}`,\n original: item\n };\n }\n }).filter(result => result.cleaned_text || result.error); \n}\n\ntry {\n return cleanAllPosts(inputData);\n} catch (error) {\n console.log('Fatal error:', error);\n return [{ \n cleaned_text: '', \n error: `Fatal error: ${error.message}`,\n input: inputData \n }];\n}\n"
- },
- "typeVersion": 2
- },
- {
- "id": "a0727b55-565d-47c0-9ab5-0f001f4b9941",
- "name": "Limit for testing (optional)",
- "type": "n8n-nodes-base.limit",
- "position": [
- 2280,
- -220
- ],
- "parameters": {
- "maxItems": 5
- },
- "typeVersion": 1
- },
- {
- "id": "650baf5e-c2ac-443d-8a2b-6df89717186f",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 580,
- -620
- ],
- "parameters": {
- "width": 540,
- "height": 340,
- "content": "## Clean the result \n\n```json\n{\n\"title\": \"Ask HN: Who is hiring? (February 2025)\",\n\"createdAt\": \"2025-02-03T16:00:43Z\",\n\"updatedAt\": \"2025-02-17T08:35:44Z\",\n\"storyId\": \"42919502\"\n},\n{\n\"title\": \"Ask HN: Who is hiring? (January 2025)\",\n\"createdAt\": \"2025-01-02T16:00:09Z\",\n\"updatedAt\": \"2025-02-13T00:03:24Z\",\n\"storyId\": \"42575537\"\n},\n```"
- },
- "typeVersion": 1
- },
- {
- "id": "1ca5c39f-f21d-455a-b63a-702e7e3ba02b",
- "name": "Write results to airtable",
- "type": "n8n-nodes-base.airtable",
- "position": [
- 3040,
- -220
- ],
- "parameters": {
- "base": {
- "__rl": true,
- "mode": "list",
- "value": "appM2JWvA5AstsGdn",
- "cachedResultUrl": "https://airtable.com/appM2JWvA5AstsGdn",
- "cachedResultName": "HN Who is hiring?"
- },
- "table": {
- "__rl": true,
- "mode": "list",
- "value": "tblGvcOjqbliwM7AS",
- "cachedResultUrl": "https://airtable.com/appM2JWvA5AstsGdn/tblGvcOjqbliwM7AS",
- "cachedResultName": "Table 1"
- },
- "columns": {
- "value": {
- "type": "={{ $json.output.type }}",
- "title": "={{ $json.output.title }}",
- "salary": "={{ $json.output.salary }}",
- "company": "={{ $json.output.company }}",
- "location": "={{ $json.output.location }}",
- "apply_url": "={{ $json.output.apply_url }}",
- "company_url": "={{ $json.output.company_url }}",
- "description": "={{ $json.output.description }}"
- },
- "schema": [
- {
- "id": "title",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "title",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "company",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "company",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "location",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "location",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "type",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "type",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "salary",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "salary",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "description",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "description",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "apply_url",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "apply_url",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "company_url",
- "type": "string",
- "display": true,
- "removed": false,
- "readOnly": false,
- "required": false,
- "displayName": "company_url",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "posted_date",
- "type": "string",
- "display": true,
- "removed": true,
- "readOnly": false,
- "required": false,
- "displayName": "posted_date",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [],
- "attemptToConvertTypes": false,
- "convertFieldsToString": false
- },
- "options": {},
- "operation": "create"
- },
- "credentials": {
- "airtableTokenApi": {
- "id": "IudXLNj7CDuc5M5a",
- "name": "Airtable Personal Access Token account"
- }
- },
- "typeVersion": 2.1
- },
- {
- "id": "d71fa024-86a0-4f74-b033-1f755574080c",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- -300
- ],
- "parameters": {
- "width": 380,
- "height": 500,
- "content": "## Hacker News - Who is Hiring Scrape\n\nIn this template we setup a scraper for the monthly HN Who is Hiring post. This way we can scrape the data and transform it to a common data strcutre.\n\nFirst we use the [Algolia Search](https://hn.algolia.com/) provided by hackernews to drill down the results.\n\nWe can use the official [Hacker News API](https://github.com/HackerNews/API\n) to get the post data and also all the replies!\n\nThis will obviously work for any kind of post on hacker news! Get creative ๐\n\nAll you need is an Openai Account to structure the text data and an Airtable Account (or similar) to write the results to a list.\n\nCopy my base https://airtable.com/appM2JWvA5AstsGdn/shrAuo78cJt5C2laR"
- },
- "typeVersion": 1
- },
- {
- "id": "7466fb0c-9f0c-4adf-a6de-b2cf09032719",
- "name": "HI API: Get the individual job post",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1660,
- -220
- ],
- "parameters": {
- "url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.kids }}.json?print=pretty",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "184abccf-5838-49bf-9922-e0300c6b145e",
- "name": "HN API: Get Main Post",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1260,
- -220
- ],
- "parameters": {
- "url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.storyId }}.json?print=pretty",
- "options": {}
- },
- "typeVersion": 4.2
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "387f7084-58fa-4643-9351-73c870d3f028",
- "connections": {
- "Split Out": {
- "main": [
- [
- {
- "node": "Get relevant data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Clean text": {
- "main": [
- [
- {
- "node": "Limit for testing (optional)",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract text": {
- "main": [
- [
- {
- "node": "Clean text",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get latest post": {
- "main": [
- [
- {
- "node": "HN API: Get Main Post",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get relevant data": {
- "main": [
- [
- {
- "node": "Get latest post",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Trun into structured data",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "HN API: Get Main Post": {
- "main": [
- [
- {
- "node": "Split out children (jobs)",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Trun into structured data",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Split out children (jobs)": {
- "main": [
- [
- {
- "node": "HI API: Get the individual job post",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Trun into structured data": {
- "main": [
- [
- {
- "node": "Write results to airtable",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Limit for testing (optional)": {
- "main": [
- [
- {
- "node": "Trun into structured data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Search for Who is hiring posts": {
- "main": [
- [
- {
- "node": "Split Out",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Search for Who is hiring posts",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "HI API: Get the individual job post": {
- "main": [
- [
- {
- "node": "Extract text",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.txt b/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.txt
new file mode 100644
index 0000000..a183678
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Hacker News Job Listing Scraper and Parser.txt
@@ -0,0 +1,745 @@
+{
+"id": "0JsHmmyeHw5Ffz5m",
+"meta": {
+"instanceId": "d4d7965840e96e50a3e02959a8487c692901dfa8d5cc294134442c67ce1622d3",
+"templateCredsSetupCompleted": true
+},
+"name": "HN Who is Hiring Scrape",
+"tags": [],
+"nodes": [
+{
+"id": "f7cdb3ee-9bb0-4006-829a-d4ce797191d5",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-20,
+-220
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "0475e25d-9bf4-450d-abd3-a04608a438a4",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+60,
+-620
+],
+"parameters": {
+"width": 460,
+"height": 340,
+"content": "## Go to https://hn.algolia.com\n- filter by \"Ask HN: Who is hiring?\" (important with quotes for full match)\n- sort by date\n- Chrome Network Tab > find API call > click \"Copy as cURL\"\n- n8n HTTP node -> import cURL and paste \n- I've set the API key as Header Auth so you will have to do the above yourself to make this work"
+},
+"typeVersion": 1
+},
+{
+"id": "a686852b-ff84-430b-92bb-ce02a6808e19",
+"name": "Split Out",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+400,
+-220
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "hits"
+},
+"typeVersion": 1
+},
+{
+"id": "cdaaa738-d561-4fa0-b2c7-8ea9e6778eb1",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1260,
+-620
+],
+"parameters": {
+"width": 500,
+"height": 340,
+"content": "## Go to HN API \nhttps://github.com/HackerNews/API\n\nWe'll need following endpoints: \n- For example, a story: https://hacker-news.firebaseio.com/v0/item/8863.json?print=pretty\n- comment: https://hacker-news.firebaseio.com/v0/item/2921983.json?print=pretty\n\n"
+},
+"typeVersion": 1
+},
+{
+"id": "4f353598-9e32-4be4-9e7b-c89cc05305fd",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+2680,
+-20
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "Fbb2ueT0XP5xMRme",
+"name": "OpenAi account 2"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "5bd0d7cc-497a-497c-aa4c-589d9ceeca14",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+2840,
+-20
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"company\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Name of the hiring company\"\n },\n \"title\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Job title/role being advertised\"\n },\n \"location\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Work location including remote/hybrid status\"\n },\n \"type\": {\n \"type\": [\n \"string\",\n null\n ],\n \"enum\": [\n \"FULL_TIME\",\n \"PART_TIME\",\n \"CONTRACT\",\n \"INTERNSHIP\",\n \"FREELANCE\",\n null\n ],\n \"description\": \"Employment type (Full-time, Contract, etc)\"\n },\n \"work_location\": {\n \"type\": [\n \"string\",\n null\n ],\n \"enum\": [\n \"REMOTE\",\n \"HYBRID\",\n \"ON_SITE\",\n null\n ],\n \"description\": \"Work arrangement type\"\n },\n \"salary\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Compensation details if provided\"\n },\n \"description\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Main job description text including requirements and team info\"\n },\n \"apply_url\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Direct application/job posting URL\"\n },\n \"company_url\": {\n \"type\": [\n \"string\",\n null\n ],\n \"description\": \"Company website or careers page\"\n }\n }\n}\n"
+},
+"typeVersion": 1.2
+},
+{
+"id": "b84ca004-6f3b-4577-8910-61b8584b161d",
+"name": "Search for Who is hiring posts",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+200,
+-220
+],
+"parameters": {
+"url": "https://uj5wyc0l7x-dsn.algolia.net/1/indexes/Item_dev_sort_date/query",
+"method": "POST",
+"options": {},
+"jsonBody": "{\n \"query\": \"\\\"Ask HN: Who is hiring\\\"\",\n \"analyticsTags\": [\n \"web\"\n ],\n \"page\": 0,\n \"hitsPerPage\": 30,\n \"minWordSizefor1Typo\": 4,\n \"minWordSizefor2Typos\": 8,\n \"advancedSyntax\": true,\n \"ignorePlurals\": false,\n \"clickAnalytics\": true,\n \"minProximity\": 7,\n \"numericFilters\": [],\n \"tagFilters\": [\n [\n \"story\"\n ],\n []\n ],\n \"typoTolerance\": \"min\",\n \"queryType\": \"prefixNone\",\n \"restrictSearchableAttributes\": [\n \"title\",\n \"comment_text\",\n \"url\",\n \"story_text\",\n \"author\"\n ],\n \"getRankingInfo\": true\n}",
+"sendBody": true,
+"sendQuery": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "x-algolia-agent",
+"value": "Algolia for JavaScript (4.13.1); Browser (lite)"
+},
+{
+"name": "x-algolia-application-id",
+"value": "UJ5WYC0L7X"
+}
+]
+},
+"headerParameters": {
+"parameters": [
+{
+"name": "Accept",
+"value": "*/*"
+},
+{
+"name": "Accept-Language",
+"value": "en-GB,en-US;q=0.9,en;q=0.8"
+},
+{
+"name": "Connection",
+"value": "keep-alive"
+},
+{
+"name": "DNT",
+"value": "1"
+},
+{
+"name": "Origin",
+"value": "https://hn.algolia.com"
+},
+{
+"name": "Referer",
+"value": "https://hn.algolia.com/"
+},
+{
+"name": "Sec-Fetch-Dest",
+"value": "empty"
+},
+{
+"name": "Sec-Fetch-Mode",
+"value": "cors"
+},
+{
+"name": "Sec-Fetch-Site",
+"value": "cross-site"
+},
+{
+"name": "User-Agent",
+"value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
+},
+{
+"name": "sec-ch-ua",
+"value": "\"Chromium\";v=\"133\", \"Not(A:Brand\";v=\"99\""
+},
+{
+"name": "sec-ch-ua-mobile",
+"value": "?0"
+},
+{
+"name": "sec-ch-ua-platform",
+"value": "\"macOS\""
+}
+]
+}
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "oVEXp2ZbYCXypMVz",
+"name": "Algolia Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "205e66f6-cd6b-4cfd-a6ec-2226c35ddaac",
+"name": "Get relevant data",
+"type": "n8n-nodes-base.set",
+"position": [
+700,
+-220
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "73dd2325-faa7-4650-bd78-5fc97cc202de",
+"name": "title",
+"type": "string",
+"value": "={{ $json.title }}"
+},
+{
+"id": "44918eac-4510-440e-9ac0-bf14d2b2f3af",
+"name": "createdAt",
+"type": "string",
+"value": "={{ $json.created_at }}"
+},
+{
+"id": "00eb6f09-2c22-411c-949c-886b2d95b6eb",
+"name": "updatedAt",
+"type": "string",
+"value": "={{ $json.updated_at }}"
+},
+{
+"id": "2b4f9da6-f60e-46e0-ba9d-3242fa955a55",
+"name": "storyId",
+"type": "string",
+"value": "={{ $json.story_id }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "16bc5628-8a29-4eac-8be9-b4e9da802e1e",
+"name": "Get latest post",
+"type": "n8n-nodes-base.filter",
+"position": [
+900,
+-220
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "d7dd7175-2a50-45aa-bd3e-4c248c9193c4",
+"operator": {
+"type": "dateTime",
+"operation": "after"
+},
+"leftValue": "={{ $json.createdAt }}",
+"rightValue": "={{$now.minus({days: 30})}} "
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "92e1ef74-5ae1-4195-840b-115184db464f",
+"name": "Split out children (jobs)",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+1460,
+-220
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "kids"
+},
+"typeVersion": 1
+},
+{
+"id": "d0836aae-b98a-497f-a6f7-0ad563c262a0",
+"name": "Trun into structured data",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+2600,
+-220
+],
+"parameters": {
+"text": "={{ $json.cleaned_text }}",
+"messages": {
+"messageValues": [
+{
+"message": "Extract the JSON data"
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "fd818a93-627c-435d-91ba-5d759d5a9004",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2600,
+-620
+],
+"parameters": {
+"width": 840,
+"height": 340,
+"content": "## Data Structure\n\nWe use Openai GPT-4o-mini to transform the raw data in a unified data structure. Feel free to change this.\n\n```json\n{\n \"company\": \"Name of the hiring company\",\n \"title\": \"Job title/role being advertised\",\n \"location\": \"Work location including remote/hybrid status\",\n \"type\": \"Employment type (Full-time, Contract, etc)\",\n \"salary\": \"Compensation details if provided\",\n \"description\": \"Main job description text including requirements and team info\",\n \"apply_url\": \"Direct application/job posting URL\",\n \"company_url\": \"Company website or careers page\"\n}\n```"
+},
+"typeVersion": 1
+},
+{
+"id": "b70c5578-5b81-467a-8ac2-65374e4e52f3",
+"name": "Extract text",
+"type": "n8n-nodes-base.set",
+"position": [
+1860,
+-220
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "6affa370-56ce-4ad8-8534-8f753fdf07fc",
+"name": "text",
+"type": "string",
+"value": "={{ $json.text }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "acb68d88-9417-42e9-9bcc-7c2fa95c4afd",
+"name": "Clean text",
+"type": "n8n-nodes-base.code",
+"position": [
+2060,
+-220
+],
+"parameters": {
+"jsCode": "// In a Function node in n8n\nconst inputData = $input.all();\n\nfunction cleanAllPosts(data) {\n return data.map(item => {\n try {\n // Check if item exists and has the expected structure\n if (!item || typeof item !== 'object') {\n return { cleaned_text: '', error: 'Invalid item structure' };\n }\n\n // Get the text, with multiple fallbacks\n let text = '';\n if (typeof item === 'string') {\n text = item;\n } else if (item.json && item.json.text) {\n text = item.json.text;\n } else if (typeof item.json === 'string') {\n text = item.json;\n } else {\n text = JSON.stringify(item);\n }\n\n // Make sure text is a string\n text = String(text);\n \n // Perform the cleaning operations\n try {\n text = text.replace(///g, '/');\n text = text.replace(/'/g, \"'\");\n text = text.replace(/&\\w+;/g, ' ');\n text = text.replace(/<[^>]*>/g, '');\n text = text.replace(/\\|\\s*/g, '| ');\n text = text.replace(/\\s+/g, ' ');\n text = text.replace(/\\s*(https?:\\/\\/[^\\s]+)\\s*/g, '\\n$1\\n');\n text = text.replace(/\\n{3,}/g, '\\n\\n');\n text = text.trim();\n } catch (cleaningError) {\n console.log('Error during text cleaning:', cleaningError);\n // Return original text if cleaning fails\n return { cleaned_text: text, warning: 'Partial cleaning applied' };\n }\n\n return { cleaned_text: text };\n \n } catch (error) {\n console.log('Error processing item:', error);\n return { \n cleaned_text: '', \n error: `Processing error: ${error.message}`,\n original: item\n };\n }\n }).filter(result => result.cleaned_text || result.error); \n}\n\ntry {\n return cleanAllPosts(inputData);\n} catch (error) {\n console.log('Fatal error:', error);\n return [{ \n cleaned_text: '', \n error: `Fatal error: ${error.message}`,\n input: inputData \n }];\n}\n"
+},
+"typeVersion": 2
+},
+{
+"id": "a0727b55-565d-47c0-9ab5-0f001f4b9941",
+"name": "Limit for testing (optional)",
+"type": "n8n-nodes-base.limit",
+"position": [
+2280,
+-220
+],
+"parameters": {
+"maxItems": 5
+},
+"typeVersion": 1
+},
+{
+"id": "650baf5e-c2ac-443d-8a2b-6df89717186f",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+580,
+-620
+],
+"parameters": {
+"width": 540,
+"height": 340,
+"content": "## Clean the result \n\n```json\n{\n\"title\": \"Ask HN: Who is hiring? (February 2025)\",\n\"createdAt\": \"2025-02-03T16:00:43Z\",\n\"updatedAt\": \"2025-02-17T08:35:44Z\",\n\"storyId\": \"42919502\"\n},\n{\n\"title\": \"Ask HN: Who is hiring? (January 2025)\",\n\"createdAt\": \"2025-01-02T16:00:09Z\",\n\"updatedAt\": \"2025-02-13T00:03:24Z\",\n\"storyId\": \"42575537\"\n},\n```"
+},
+"typeVersion": 1
+},
+{
+"id": "1ca5c39f-f21d-455a-b63a-702e7e3ba02b",
+"name": "Write results to airtable",
+"type": "n8n-nodes-base.airtable",
+"position": [
+3040,
+-220
+],
+"parameters": {
+"base": {
+"__rl": true,
+"mode": "list",
+"value": "appM2JWvA5AstsGdn",
+"cachedResultUrl": "https://airtable.com/appM2JWvA5AstsGdn",
+"cachedResultName": "HN Who is hiring?"
+},
+"table": {
+"__rl": true,
+"mode": "list",
+"value": "tblGvcOjqbliwM7AS",
+"cachedResultUrl": "https://airtable.com/appM2JWvA5AstsGdn/tblGvcOjqbliwM7AS",
+"cachedResultName": "Table 1"
+},
+"columns": {
+"value": {
+"type": "={{ $json.output.type }}",
+"title": "={{ $json.output.title }}",
+"salary": "={{ $json.output.salary }}",
+"company": "={{ $json.output.company }}",
+"location": "={{ $json.output.location }}",
+"apply_url": "={{ $json.output.apply_url }}",
+"company_url": "={{ $json.output.company_url }}",
+"description": "={{ $json.output.description }}"
+},
+"schema": [
+{
+"id": "title",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "title",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "company",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "company",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "location",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "location",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "type",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "type",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "salary",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "salary",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "description",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "description",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "apply_url",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "apply_url",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "company_url",
+"type": "string",
+"display": true,
+"removed": false,
+"readOnly": false,
+"required": false,
+"displayName": "company_url",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "posted_date",
+"type": "string",
+"display": true,
+"removed": true,
+"readOnly": false,
+"required": false,
+"displayName": "posted_date",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": false
+},
+"options": {},
+"operation": "create"
+},
+"credentials": {
+"airtableTokenApi": {
+"id": "IudXLNj7CDuc5M5a",
+"name": "Airtable Personal Access Token account"
+}
+},
+"typeVersion": 2.1
+},
+{
+"id": "d71fa024-86a0-4f74-b033-1f755574080c",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+-300
+],
+"parameters": {
+"width": 380,
+"height": 500,
+"content": "## Hacker News - Who is Hiring Scrape\n\nIn this template we setup a scraper for the monthly HN Who is Hiring post. This way we can scrape the data and transform it to a common data strcutre.\n\nFirst we use the [Algolia Search](https://hn.algolia.com/) provided by hackernews to drill down the results.\n\nWe can use the official [Hacker News API](https://github.com/HackerNews/API\n) to get the post data and also all the replies!\n\nThis will obviously work for any kind of post on hacker news! Get creative ๐\n\nAll you need is an Openai Account to structure the text data and an Airtable Account (or similar) to write the results to a list.\n\nCopy my base https://airtable.com/appM2JWvA5AstsGdn/shrAuo78cJt5C2laR"
+},
+"typeVersion": 1
+},
+{
+"id": "7466fb0c-9f0c-4adf-a6de-b2cf09032719",
+"name": "HI API: Get the individual job post",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1660,
+-220
+],
+"parameters": {
+"url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.kids }}.json?print=pretty",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "184abccf-5838-49bf-9922-e0300c6b145e",
+"name": "HN API: Get Main Post",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1260,
+-220
+],
+"parameters": {
+"url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.storyId }}.json?print=pretty",
+"options": {}
+},
+"typeVersion": 4.2
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "387f7084-58fa-4643-9351-73c870d3f028",
+"connections": {
+"Split Out": {
+"main": [
+[
+{
+"node": "Get relevant data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Clean text": {
+"main": [
+[
+{
+"node": "Limit for testing (optional)",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract text": {
+"main": [
+[
+{
+"node": "Clean text",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get latest post": {
+"main": [
+[
+{
+"node": "HN API: Get Main Post",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get relevant data": {
+"main": [
+[
+{
+"node": "Get latest post",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Trun into structured data",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"HN API: Get Main Post": {
+"main": [
+[
+{
+"node": "Split out children (jobs)",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Trun into structured data",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Split out children (jobs)": {
+"main": [
+[
+{
+"node": "HI API: Get the individual job post",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Trun into structured data": {
+"main": [
+[
+{
+"node": "Write results to airtable",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Limit for testing (optional)": {
+"main": [
+[
+{
+"node": "Trun into structured data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Search for Who is hiring posts": {
+"main": [
+[
+{
+"node": "Split Out",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Search for Who is hiring posts",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"HI API: Get the individual job post": {
+"main": [
+[
+{
+"node": "Extract text",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.json b/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.json
deleted file mode 100644
index ef34384..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.json
+++ /dev/null
@@ -1,1498 +0,0 @@
-{
- "id": "744G7emgZe0pXaPB",
- "meta": {
- "instanceId": "d868e3d040e7bda892c81b17cf446053ea25d2556fcef89cbe19dd61a3e876e9"
- },
- "name": "Hacker News to Video Template - AlexK1919",
- "tags": [
- {
- "id": "04PL2irdWYmF2Dg3",
- "name": "RunwayML",
- "createdAt": "2024-11-15T05:55:30.783Z",
- "updatedAt": "2024-11-15T05:55:30.783Z"
- },
- {
- "id": "yrY6updwSCXMsT0z",
- "name": "Video",
- "createdAt": "2024-11-15T05:55:34.333Z",
- "updatedAt": "2024-11-15T05:55:34.333Z"
- },
- {
- "id": "QsH2EXuw2e7YCv0K",
- "name": "OpenAI",
- "createdAt": "2024-11-15T04:05:20.872Z",
- "updatedAt": "2024-11-15T04:05:20.872Z"
- },
- {
- "id": "lvPj9rYRsKOHCi4J",
- "name": "Creatomate",
- "createdAt": "2024-11-19T15:59:16.134Z",
- "updatedAt": "2024-11-19T15:59:16.134Z"
- },
- {
- "id": "9LXACqpQLNtrM6or",
- "name": "Leonardo",
- "createdAt": "2024-11-19T15:59:21.368Z",
- "updatedAt": "2024-11-19T15:59:21.368Z"
- }
- ],
- "nodes": [
- {
- "id": "c777c41b-842d-4504-a1a0-ccbb034a0fdd",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -320,
- 300
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "74fafd7c-55a4-46ec-b4a8-33d46f2b5b54",
- "name": "Hacker News",
- "type": "n8n-nodes-base.hackerNews",
- "position": [
- -20,
- 300
- ],
- "parameters": {
- "resource": "all",
- "additionalFields": {}
- },
- "typeVersion": 1
- },
- {
- "id": "9cd87fd2-6a38-463a-a22e-e0c34910818f",
- "name": "Loop Over Items",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- 440,
- 300
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "611b24cd-558b-4025-a0a8-ea355ba61988",
- "name": "OpenAI Chat Model3",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 720,
- 580
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "f814682c-cf6f-49a8-8ea0-48fbc64a3ebe",
- "name": "HTTP Request1",
- "type": "@n8n/n8n-nodes-langchain.toolHttpRequest",
- "position": [
- 900,
- 580
- ],
- "parameters": {
- "url": "={{ $json.url }}",
- "toolDescription": "grab the article for the ai agent to use"
- },
- "typeVersion": 1.1
- },
- {
- "id": "2a4bcf69-23f0-440d-a3b0-c8261e153c62",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 1080,
- 580
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n\t\"type\": \"object\",\n\t\"properties\": {\n\t\t\"summary\": {\n\t\t\t\"type\": \"string\"\n\t\t},\n\t\t\"related\": {\n\t\t\t\"type\": \"string\"\n\t\t},\n \"image urls\": {\n\t\t\t\"type\": \"string\"\n }\n\t}\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "83c3b8f0-8d67-48a2-a5ce-b777ea1d7b32",
- "name": "Upload to Minio",
- "type": "n8n-nodes-base.s3",
- "position": [
- 4240,
- 1080
- ],
- "parameters": {
- "operation": "upload",
- "bucketName": "=",
- "additionalFields": {
- "grantRead": true,
- "parentFolderKey": "="
- }
- },
- "typeVersion": 1
- },
- {
- "id": "05b972ff-ccab-415b-8787-aafabb3b7292",
- "name": "News1",
- "type": "n8n-nodes-base.set",
- "position": [
- 2180,
- 320
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "ec8013d5-84b5-43c8-abcb-6986ef15939d",
- "name": "property_name",
- "type": "string",
- "value": "={{ $json.message.content['Article Title'] }}"
- },
- {
- "id": "4d91c4fc-12a2-4fe2-a58e-02284314e1de",
- "name": "property_text",
- "type": "string",
- "value": "={{ $json.message.content['Article Blurb'] }}"
- },
- {
- "id": "cad2b795-8b71-415f-a100-700d9ec62bbd",
- "name": "property_image_url",
- "type": "string",
- "value": "={{ $('If Topic').item.json.output['image urls'] }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "d175d366-e672-4452-b78e-a06336ef242b",
- "name": "Leo - Improve Prompt",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2720,
- 100
- ],
- "parameters": {
- "url": "https://cloud.leonardo.ai/api/rest/v1/prompt/improve",
- "method": "POST",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "jsonBody": "={\n \"prompt\": \"{{ $('Article Prep').item.json.message.content['Image Prompt 1'] }}\"\n}",
- "sendBody": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "accept",
- "value": "application/json"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "d8da7879-1a67-4da1-86db-f70e50b4e9da",
- "name": "Leo - Get imageId",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3320,
- 100
- ],
- "parameters": {
- "url": "=https://cloud.leonardo.ai/api/rest/v1/generations/{{ $json.body.sdGenerationJob.generationId }}",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "content-type",
- "value": "application/json"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "faf80246-3b1a-49c6-a277-0152428e46e1",
- "name": "Runway - Create Video",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2520,
- 300
- ],
- "parameters": {
- "url": "https://api.dev.runwayml.com/v1/image_to_video",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "promptImage",
- "value": "={{ $json.body.generations_by_pk.generated_images[0].url }}"
- },
- {
- "name": "promptText",
- "value": "string"
- },
- {
- "name": "model",
- "value": "gen3a_turbo"
- }
- ]
- },
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "X-Runway-Version",
- "value": "2024-11-06"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "e91c1f01-7870-4063-9557-24a6ba1d3db3",
- "name": "Runway - Get Video",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 300
- ],
- "parameters": {
- "url": "=https://api.dev.runwayml.com/v1/tasks/{{ $json.id }}",
- "options": {},
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "X-Runway-Version",
- "value": "2024-11-06"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "41ee2665-e1aa-4d48-ade6-e37af568f211",
- "name": "Wait2",
- "type": "n8n-nodes-base.wait",
- "position": [
- 2720,
- 300
- ],
- "webhookId": "ddca5833-a40b-404a-9140-686cd4fa26cb",
- "parameters": {
- "unit": "minutes",
- "amount": 3
- },
- "typeVersion": 1.1
- },
- {
- "id": "091e9e07-89ba-4fe3-9fc5-278fc333dbff",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -160,
- -40
- ],
- "parameters": {
- "color": 5,
- "width": 341,
- "height": 951,
- "content": "# Choose your data source \n## This can be swapped for any other data source of your choosing."
- },
- "typeVersion": 1
- },
- {
- "id": "9660a593-9966-4ebe-bfd7-f884dc185d56",
- "name": "If Topic",
- "type": "n8n-nodes-base.if",
- "position": [
- 1100,
- 320
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "56219de5-244d-4b7f-a511-f3061572cf93",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.output.related }}",
- "rightValue": "yes"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "e47140ac-20cc-417b-a6cd-30f780dc8289",
- "name": "Get Image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1500,
- 320
- ],
- "parameters": {
- "url": "={{ $('Article Analysis').first().json.output['image urls'] }}",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "26f80f71-2c3a-46fe-a960-21cdbc18ce34",
- "name": "Prompt Settings1",
- "type": "n8n-nodes-base.set",
- "position": [
- 2520,
- 100
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "56c8f20d-d9d9-4be7-ac2a-38df6ffdd722",
- "name": "model",
- "type": "string",
- "value": "6b645e3a-d64f-4341-a6d8-7a3690fbf042"
- }
- ]
- },
- "includeOtherFields": true
- },
- "typeVersion": 3.4
- },
- {
- "id": "ce697f6f-f8fc-4ba7-b776-17bbc2e870b7",
- "name": "Leo - Generate Image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 100
- ],
- "parameters": {
- "url": "https://cloud.leonardo.ai/api/rest/v1/generations",
- "method": "POST",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "jsonBody": "={\n \"alchemy\": true,\n \"width\": 1024,\n \"height\": 768,\n \"modelId\": \"6b645e3a-d64f-4341-a6d8-7a3690fbf042\",\n \"num_images\": 1,\n \"presetStyle\": \"MONOCHROME\",\n \"prompt\": \"{{ $json.body.promptGeneration.prompt }}; Use the rule of thirds, leading lines, & balance. DO NOT INCLUDE ANY WORDS OR LABELS.\",\n \"guidance_scale\": 7,\n \"highResolution\": true,\n \"promptMagic\": false,\n \"promptMagicStrength\": 0.5,\n \"promptMagicVersion\": \"v3\",\n \"public\": false,\n \"ultra\": false,\n \"photoReal\": false,\n \"negative_prompt\": \"\"\n} ",
- "sendBody": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "accept",
- "value": "application/json"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "e2067fe5-3fae-4f97-97c0-879967efd9b8",
- "name": "Wait1",
- "type": "n8n-nodes-base.wait",
- "position": [
- 3120,
- 100
- ],
- "webhookId": "256c3814-6a52-4eb1-969a-30f9f3b8e04e",
- "parameters": {
- "amount": 30
- },
- "typeVersion": 1.1
- },
- {
- "id": "f0ba57a5-1d27-4c75-a422-4bc0e2cead9d",
- "name": "Limit",
- "type": "n8n-nodes-base.limit",
- "position": [
- 240,
- 300
- ],
- "parameters": {
- "keep": "lastItems",
- "maxItems": 50
- },
- "typeVersion": 1
- },
- {
- "id": "e01152aa-961b-4e33-a1e3-186d47d81c55",
- "name": "Image Analysis",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 1300,
- 320
- ],
- "parameters": {
- "modelId": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o-mini",
- "cachedResultName": "GPT-4O-MINI"
- },
- "options": {
- "detail": "auto"
- },
- "resource": "image",
- "imageUrls": "={{ $json.output['image urls'] }}",
- "operation": "analyze"
- },
- "credentials": {
- "openAiApi": {
- "id": "ysxujEYFiY5ozRTS",
- "name": "AlexK OpenAi Key"
- }
- },
- "typeVersion": 1.6
- },
- {
- "id": "ab346129-c3d5-4f51-af5e-5d63cd154981",
- "name": "Wait3",
- "type": "n8n-nodes-base.wait",
- "disabled": true,
- "position": [
- 3080,
- 1020
- ],
- "webhookId": "6e4a0b8d-6c31-4a98-8ec3-2509aa2087e8",
- "parameters": {
- "unit": "minutes"
- },
- "typeVersion": 1.1
- },
- {
- "id": "872c35a3-bdd5-4eec-9bac-0959f3ff78e7",
- "name": "Article Analysis",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "onError": "continueErrorOutput",
- "position": [
- 740,
- 300
- ],
- "parameters": {
- "text": "=Can you tell me if the article at {{ $json.url }} is related to automation or ai? \n\nthen, create a 250 word summary of the article\n\nAlso, list any image url's related to the article content from the url. Limit to 1 image url.",
- "options": {
- "systemMessage": "You are a helpful assistant in summarizing and identifying articles related to automation and ai. \nOutput the results as:\nsummary: \nrelated: yes or no\nimage urls: "
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.7
- },
- {
- "id": "31c3a90e-10ee-4217-9b08-ff57bf17ea10",
- "name": "Dropbox",
- "type": "n8n-nodes-base.dropbox",
- "position": [
- 3640,
- 1080
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "22ccd0a0-f7f6-40ca-bd09-40ed4a7fcde1",
- "name": "Google Drive",
- "type": "n8n-nodes-base.googleDrive",
- "position": [
- 3840,
- 1080
- ],
- "parameters": {
- "fileId": {
- "__rl": true,
- "mode": "list",
- "value": ""
- },
- "options": {},
- "operation": "update"
- },
- "credentials": {
- "googleDriveOAuth2Api": {
- "id": "m8K1mbAUn7yuiEwl",
- "name": "AlexK1919 Google Drive account"
- }
- },
- "typeVersion": 3
- },
- {
- "id": "ea75931d-c1ee-4139-9bdc-7901056ba016",
- "name": "Microsoft OneDrive",
- "type": "n8n-nodes-base.microsoftOneDrive",
- "position": [
- 4040,
- 1080
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "38888521-3087-4e0a-81d6-cf4b9a5dd3dd",
- "name": "YouTube",
- "type": "n8n-nodes-base.youTube",
- "position": [
- 3640,
- 1500
- ],
- "parameters": {
- "options": {},
- "resource": "video",
- "operation": "upload"
- },
- "typeVersion": 1
- },
- {
- "id": "55f3decc-f952-4d2a-804d-2aec44fb2755",
- "name": "X",
- "type": "n8n-nodes-base.twitter",
- "position": [
- 3840,
- 1500
- ],
- "parameters": {
- "additionalFields": {}
- },
- "typeVersion": 2
- },
- {
- "id": "54c8b762-444d-4790-97a9-a2f84518492f",
- "name": "Instagram",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 4240,
- 1500
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "90040f15-95c0-4ebb-818f-dde508eb0689",
- "name": "LinkedIn",
- "type": "n8n-nodes-base.linkedIn",
- "position": [
- 4040,
- 1500
- ],
- "parameters": {
- "additionalFields": {}
- },
- "typeVersion": 1
- },
- {
- "id": "691eb779-5fae-4f65-89eb-b1b8e5488809",
- "name": "Leo - Improve Prompt2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2720,
- 500
- ],
- "parameters": {
- "url": "https://cloud.leonardo.ai/api/rest/v1/prompt/improve",
- "method": "POST",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "jsonBody": "={\n \"prompt\": \"{{ $('Article Prep').item.json.message.content['Image Prompt 2'] }}\"\n}",
- "sendBody": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "accept",
- "value": "application/json"
- }
- ]
- }
- },
- "credentials": {
- "httpCustomAuth": {
- "id": "hIzUsjbtHLmIe6uM",
- "name": "RunwayML Custom Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "076a745a-055b-459c-8af9-fa7b6740dc6f",
- "name": "Wait4",
- "type": "n8n-nodes-base.wait",
- "position": [
- 2720,
- 700
- ],
- "webhookId": "89b31515-b403-4644-a2c1-970e5e774008",
- "parameters": {
- "unit": "minutes",
- "amount": 3
- },
- "typeVersion": 1.1
- },
- {
- "id": "adc2c993-3f89-40df-96fc-eb3ff5eafb1c",
- "name": "Wait6",
- "type": "n8n-nodes-base.wait",
- "position": [
- 3120,
- 500
- ],
- "webhookId": "2efb873f-bcbd-41d9-99da-b2b57ef5ad93",
- "parameters": {
- "amount": 30
- },
- "typeVersion": 1.1
- },
- {
- "id": "156f5735-bc20-46a9-871c-143b0772ca45",
- "name": "Leo - Generate Image2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 500
- ],
- "parameters": {
- "url": "https://cloud.leonardo.ai/api/rest/v1/generations",
- "method": "POST",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "jsonBody": "={\n \"alchemy\": true,\n \"width\": 1024,\n \"height\": 768,\n \"modelId\": \"6b645e3a-d64f-4341-a6d8-7a3690fbf042\",\n \"num_images\": 1,\n \"presetStyle\": \"MONOCHROME\",\n \"prompt\": \"{{ $json.body.promptGeneration.prompt }}; Use the rule of thirds, leading lines, & balance. DO NOT INCLUDE ANY WORDS OR LABELS.\",\n \"guidance_scale\": 7,\n \"highResolution\": true,\n \"promptMagic\": false,\n \"promptMagicStrength\": 0.5,\n \"promptMagicVersion\": \"v3\",\n \"public\": false,\n \"ultra\": false,\n \"photoReal\": false,\n \"negative_prompt\": \"\"\n} ",
- "sendBody": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "accept",
- "value": "application/json"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "4f270fa8-4da2-44f0-927f-3509fd9f8f7d",
- "name": "Leo - Get imageId2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3320,
- 500
- ],
- "parameters": {
- "url": "=https://cloud.leonardo.ai/api/rest/v1/generations/{{ $json.body.sdGenerationJob.generationId }}",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "content-type",
- "value": "application/json"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "49c0e7ba-bf9c-4819-b479-61aa099ab9ab",
- "name": "Runway - Create Video2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2520,
- 700
- ],
- "parameters": {
- "url": "https://api.dev.runwayml.com/v1/image_to_video",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "promptImage",
- "value": "={{ $json.body.generations_by_pk.generated_images[0].url }}"
- },
- {
- "name": "promptText",
- "value": "string"
- },
- {
- "name": "model",
- "value": "gen3a_turbo"
- }
- ]
- },
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "X-Runway-Version",
- "value": "2024-11-06"
- }
- ]
- }
- },
- "credentials": {
- "httpCustomAuth": {
- "id": "hIzUsjbtHLmIe6uM",
- "name": "RunwayML Custom Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "d03eb190-5fc0-4b7e-ad65-88ece3ab833d",
- "name": "Runway - Get Video2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 700
- ],
- "parameters": {
- "url": "=https://api.dev.runwayml.com/v1/tasks/{{ $json.id }}",
- "options": {},
- "sendHeaders": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth",
- "headerParameters": {
- "parameters": [
- {
- "name": "X-Runway-Version",
- "value": "2024-11-06"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "0072563d-b87d-47c5-80fd-ed3c051b3287",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3580,
- 940
- ],
- "parameters": {
- "color": 6,
- "width": 882,
- "height": 372,
- "content": "# Upload Assets\nYou can extend this workflow further by uploading the generated assets to your storage option of choice."
- },
- "typeVersion": 1
- },
- {
- "id": "a0b2377e-57ea-47e9-83c9-3e58372610e5",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3580,
- 1360
- ],
- "parameters": {
- "color": 6,
- "width": 882,
- "height": 372,
- "content": "# Post to Social Media\nYou can extend this workflow further by posting the generated assets to social media."
- },
- "typeVersion": 1
- },
- {
- "id": "708fe6a0-4899-462b-9a08-fadea7c7e195",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2420,
- -40
- ],
- "parameters": {
- "color": 4,
- "width": 1114,
- "height": 943,
- "content": "# Generate Images and Videos"
- },
- "typeVersion": 1
- },
- {
- "id": "5bbb6552-ec3a-42ea-a911-993f67a6c8dc",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2420,
- 940
- ],
- "parameters": {
- "color": 5,
- "width": 1114,
- "height": 372,
- "content": "# Stitch it all together"
- },
- "typeVersion": 1
- },
- {
- "id": "25f4cc09-fbff-4c10-b706-30df5840b794",
- "name": "Cre - Generate Video1",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2880,
- 1020
- ],
- "parameters": {
- "url": "https://api.creatomate.com/v1/renders",
- "method": "POST",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "jsonBody": "={\n \"max_width\": 480,\n \"template_id\": \"enterTemplateID\",\n \"modifications\": {\n \"Scenes.elements\": [\n {\n \"name\": \"Intro Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Image-1\",\n \"type\": \"image\",\n \"source\": \"{{ $('Leo - Get imageId').item.json.body.generations_by_pk.generated_images[0].url }}\"\n },\n {\n \"name\": \"Subtitles-1\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-1\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,255,255,0.2)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff5900\"\n },\n {\n \"name\": \"Voiceover-1\",\n \"type\": \"audio\",\n \"source\": \"{{ $('News1').item.json.property_name }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n },\n {\n \"name\": \"Auto Scene Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Video-2\",\n \"type\": \"video\",\n \"source\": \"{{ $('Runway - Get Video').first().json.output[0] }}\",\n \"loop\": true\n },\n {\n \"name\": \"Subtitles-2\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-2\",\n \"y\": \"78.2173%\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,255,255,0.2)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff5900\"\n },\n {\n \"name\": \"Voiceover-2\",\n \"type\": \"audio\",\n \"source\": \"{{ $('Article Prep').item.json.message.content['Summary Blurb 1'] }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n },\n {\n \"name\": \"Auto Scene Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Video-3\",\n \"type\": \"video\",\n \"source\": \"{{ $('Runway - Get Video2').first().json.output[0] }}\",\n \"loop\": true\n },\n {\n \"name\": \"Subtitles-3\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-3\",\n \"y\": \"78.2173%\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,89,0,0.5)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff0040\"\n },\n {\n \"name\": \"Voiceover-3\",\n \"type\": \"audio\",\n \"source\": \"{{ $('Article Prep').item.json.message.content['Summary Blurb 2'] }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n }\n ]\n }\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth"
- },
- "credentials": {
- "httpCustomAuth": {
- "id": "hIzUsjbtHLmIe6uM",
- "name": "RunwayML Custom Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "7093de7b-a4e3-4363-8038-1002f7b20fbc",
- "name": "Cre - Get Video",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3280,
- 1020
- ],
- "parameters": {
- "url": "=https://api.creatomate.com/v1/renders/{{ $json.body.body[0].id }}",
- "options": {
- "response": {
- "response": {
- "fullResponse": true
- }
- }
- },
- "authentication": "genericCredentialType",
- "genericAuthType": "httpCustomAuth"
- },
- "credentials": {
- "httpCustomAuth": {
- "id": "hIzUsjbtHLmIe6uM",
- "name": "RunwayML Custom Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "a57b719f-b299-431e-9c85-fa333e38b6a7",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 660,
- -40
- ],
- "parameters": {
- "color": 3,
- "width": 1033,
- "height": 951,
- "content": "# Article Analysis - Is it the right topic?"
- },
- "typeVersion": 1
- },
- {
- "id": "60b879a0-8b7f-40f1-ae70-ac94e4675b38",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1740,
- -40
- ],
- "parameters": {
- "color": 3,
- "width": 630,
- "height": 947,
- "content": "# Prepare the article for content generation"
- },
- "typeVersion": 1
- },
- {
- "id": "afaf8437-ee52-434b-a267-8dbaff0e1922",
- "name": "Article Prep",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 1820,
- 320
- ],
- "parameters": {
- "modelId": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o-mini",
- "cachedResultName": "GPT-4O-MINI"
- },
- "options": {},
- "messages": {
- "values": [
- {
- "content": "=prepare the following summary for a newsletter where the article will be 1 of several presented in the newsletter:\n\n{{ $('Article Analysis').first().json.output.summary }}\n\nMake sure the Article Blurb lenght is less than 15 words.\n\nThen, create 2 Summary Blurbs, making sure each is less than 15 words.\n\nAlso create 2 image prompts that is less than 15 words long for each Summary Blurb"
- },
- {
- "role": "system",
- "content": "Output in markdown format\nArticle Title\nArticle Blurb\nSummary Blurb 1\nSummary Blurb 2\nArticle Image\nImage Prompt 1\nImage Prompt 2"
- }
- ]
- },
- "jsonOutput": true
- },
- "credentials": {
- "openAiApi": {
- "id": "ysxujEYFiY5ozRTS",
- "name": "AlexK OpenAi Key"
- }
- },
- "typeVersion": 1.6
- },
- {
- "id": "e7c95d56-86e1-4456-a6d3-9c8b9fc3a53c",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -620,
- -40
- ],
- "parameters": {
- "color": 6,
- "width": 252,
- "height": 946,
- "content": "# AlexK1919 \n\n\n#### Iโm Alex Kim, an AI-Native Workflow Automation Architect Building Solutions to Optimize your Personal and Professional Life.\n\n### Workflow Overview Video\nhttps://youtu.be/XaKybLDUlLk\n\n### About Me\nhttps://beacons.ai/alexk1919\n\n### Product Used \n[Leonardo.ai](https://leonardo.ai)\n[RunwayML](https://runwayml.com/)\n[Creatomate](https://creatomate.com/)\n"
- },
- "typeVersion": 1
- },
- {
- "id": "32e2803e-bf7c-4da4-a4ae-c9b6fa5ae226",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3280,
- 1180
- ],
- "parameters": {
- "color": 7,
- "width": 180,
- "height": 100,
- "content": "Don't forget to connect this last node to the loop to process additional items"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "c7ab1ecd-50cb-4e4b-b2f7-aade804bbd63",
- "connections": {
- "X": {
- "main": [
- [
- {
- "node": "LinkedIn",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Limit": {
- "main": [
- [
- {
- "node": "Loop Over Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "News1": {
- "main": [
- [
- {
- "node": "Prompt Settings1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait1": {
- "main": [
- [
- {
- "node": "Leo - Get imageId",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait2": {
- "main": [
- [
- {
- "node": "Runway - Get Video",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait3": {
- "main": [
- [
- {
- "node": "Cre - Get Video",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait4": {
- "main": [
- [
- {
- "node": "Runway - Get Video2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait6": {
- "main": [
- [
- {
- "node": "Leo - Get imageId2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Dropbox": {
- "main": [
- [
- {
- "node": "Google Drive",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "YouTube": {
- "main": [
- [
- {
- "node": "X",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If Topic": {
- "main": [
- [
- {
- "node": "Image Analysis",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Loop Over Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "LinkedIn": {
- "main": [
- [
- {
- "node": "Instagram",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Image": {
- "main": [
- [
- {
- "node": "Article Prep",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Hacker News": {
- "main": [
- [
- {
- "node": "Limit",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Article Prep": {
- "main": [
- [
- {
- "node": "News1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google Drive": {
- "main": [
- [
- {
- "node": "Microsoft OneDrive",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "HTTP Request1": {
- "ai_tool": [
- [
- {
- "node": "Article Analysis",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Image Analysis": {
- "main": [
- [
- {
- "node": "Get Image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Loop Over Items": {
- "main": [
- [],
- [
- {
- "node": "Article Analysis",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Article Analysis": {
- "main": [
- [
- {
- "node": "If Topic",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Loop Over Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prompt Settings1": {
- "main": [
- [
- {
- "node": "Leo - Improve Prompt",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Get imageId": {
- "main": [
- [
- {
- "node": "Runway - Create Video",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Get imageId2": {
- "main": [
- [
- {
- "node": "Runway - Create Video2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Microsoft OneDrive": {
- "main": [
- [
- {
- "node": "Upload to Minio",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model3": {
- "ai_languageModel": [
- [
- {
- "node": "Article Analysis",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Runway - Get Video": {
- "main": [
- [
- {
- "node": "Leo - Improve Prompt2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Runway - Get Video2": {
- "main": [
- [
- {
- "node": "Cre - Generate Video1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Generate Image": {
- "main": [
- [
- {
- "node": "Wait1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Improve Prompt": {
- "main": [
- [
- {
- "node": "Leo - Generate Image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Cre - Generate Video1": {
- "main": [
- [
- {
- "node": "Wait3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Generate Image2": {
- "main": [
- [
- {
- "node": "Wait6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Leo - Improve Prompt2": {
- "main": [
- [
- {
- "node": "Leo - Generate Image2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Runway - Create Video": {
- "main": [
- [
- {
- "node": "Wait2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Runway - Create Video2": {
- "main": [
- [
- {
- "node": "Wait4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Article Analysis",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Hacker News",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.txt b/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.txt
new file mode 100644
index 0000000..ef61f0f
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Hacker News to Video Content.txt
@@ -0,0 +1,1498 @@
+{
+"id": "744G7emgZe0pXaPB",
+"meta": {
+"instanceId": "d868e3d040e7bda892c81b17cf446053ea25d2556fcef89cbe19dd61a3e876e9"
+},
+"name": "Hacker News to Video Template - AlexK1919",
+"tags": [
+{
+"id": "04PL2irdWYmF2Dg3",
+"name": "RunwayML",
+"createdAt": "2024-11-15T05:55:30.783Z",
+"updatedAt": "2024-11-15T05:55:30.783Z"
+},
+{
+"id": "yrY6updwSCXMsT0z",
+"name": "Video",
+"createdAt": "2024-11-15T05:55:34.333Z",
+"updatedAt": "2024-11-15T05:55:34.333Z"
+},
+{
+"id": "QsH2EXuw2e7YCv0K",
+"name": "OpenAI",
+"createdAt": "2024-11-15T04:05:20.872Z",
+"updatedAt": "2024-11-15T04:05:20.872Z"
+},
+{
+"id": "lvPj9rYRsKOHCi4J",
+"name": "Creatomate",
+"createdAt": "2024-11-19T15:59:16.134Z",
+"updatedAt": "2024-11-19T15:59:16.134Z"
+},
+{
+"id": "9LXACqpQLNtrM6or",
+"name": "Leonardo",
+"createdAt": "2024-11-19T15:59:21.368Z",
+"updatedAt": "2024-11-19T15:59:21.368Z"
+}
+],
+"nodes": [
+{
+"id": "c777c41b-842d-4504-a1a0-ccbb034a0fdd",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-320,
+300
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "74fafd7c-55a4-46ec-b4a8-33d46f2b5b54",
+"name": "Hacker News",
+"type": "n8n-nodes-base.hackerNews",
+"position": [
+-20,
+300
+],
+"parameters": {
+"resource": "all",
+"additionalFields": {}
+},
+"typeVersion": 1
+},
+{
+"id": "9cd87fd2-6a38-463a-a22e-e0c34910818f",
+"name": "Loop Over Items",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+440,
+300
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "611b24cd-558b-4025-a0a8-ea355ba61988",
+"name": "OpenAI Chat Model3",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+720,
+580
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "f814682c-cf6f-49a8-8ea0-48fbc64a3ebe",
+"name": "HTTP Request1",
+"type": "@n8n/n8n-nodes-langchain.toolHttpRequest",
+"position": [
+900,
+580
+],
+"parameters": {
+"url": "={{ $json.url }}",
+"toolDescription": "grab the article for the ai agent to use"
+},
+"typeVersion": 1.1
+},
+{
+"id": "2a4bcf69-23f0-440d-a3b0-c8261e153c62",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1080,
+580
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n\t\"type\": \"object\",\n\t\"properties\": {\n\t\t\"summary\": {\n\t\t\t\"type\": \"string\"\n\t\t},\n\t\t\"related\": {\n\t\t\t\"type\": \"string\"\n\t\t},\n \"image urls\": {\n\t\t\t\"type\": \"string\"\n }\n\t}\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "83c3b8f0-8d67-48a2-a5ce-b777ea1d7b32",
+"name": "Upload to Minio",
+"type": "n8n-nodes-base.s3",
+"position": [
+4240,
+1080
+],
+"parameters": {
+"operation": "upload",
+"bucketName": "=",
+"additionalFields": {
+"grantRead": true,
+"parentFolderKey": "="
+}
+},
+"typeVersion": 1
+},
+{
+"id": "05b972ff-ccab-415b-8787-aafabb3b7292",
+"name": "News1",
+"type": "n8n-nodes-base.set",
+"position": [
+2180,
+320
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "ec8013d5-84b5-43c8-abcb-6986ef15939d",
+"name": "property_name",
+"type": "string",
+"value": "={{ $json.message.content['Article Title'] }}"
+},
+{
+"id": "4d91c4fc-12a2-4fe2-a58e-02284314e1de",
+"name": "property_text",
+"type": "string",
+"value": "={{ $json.message.content['Article Blurb'] }}"
+},
+{
+"id": "cad2b795-8b71-415f-a100-700d9ec62bbd",
+"name": "property_image_url",
+"type": "string",
+"value": "={{ $('If Topic').item.json.output['image urls'] }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "d175d366-e672-4452-b78e-a06336ef242b",
+"name": "Leo - Improve Prompt",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2720,
+100
+],
+"parameters": {
+"url": "https://cloud.leonardo.ai/api/rest/v1/prompt/improve",
+"method": "POST",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"jsonBody": "={\n \"prompt\": \"{{ $('Article Prep').item.json.message.content['Image Prompt 1'] }}\"\n}",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "accept",
+"value": "application/json"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "d8da7879-1a67-4da1-86db-f70e50b4e9da",
+"name": "Leo - Get imageId",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3320,
+100
+],
+"parameters": {
+"url": "=https://cloud.leonardo.ai/api/rest/v1/generations/{{ $json.body.sdGenerationJob.generationId }}",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "content-type",
+"value": "application/json"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "faf80246-3b1a-49c6-a277-0152428e46e1",
+"name": "Runway - Create Video",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2520,
+300
+],
+"parameters": {
+"url": "https://api.dev.runwayml.com/v1/image_to_video",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "promptImage",
+"value": "={{ $json.body.generations_by_pk.generated_images[0].url }}"
+},
+{
+"name": "promptText",
+"value": "string"
+},
+{
+"name": "model",
+"value": "gen3a_turbo"
+}
+]
+},
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "X-Runway-Version",
+"value": "2024-11-06"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e91c1f01-7870-4063-9557-24a6ba1d3db3",
+"name": "Runway - Get Video",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+300
+],
+"parameters": {
+"url": "=https://api.dev.runwayml.com/v1/tasks/{{ $json.id }}",
+"options": {},
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "X-Runway-Version",
+"value": "2024-11-06"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "41ee2665-e1aa-4d48-ade6-e37af568f211",
+"name": "Wait2",
+"type": "n8n-nodes-base.wait",
+"position": [
+2720,
+300
+],
+"webhookId": "ddca5833-a40b-404a-9140-686cd4fa26cb",
+"parameters": {
+"unit": "minutes",
+"amount": 3
+},
+"typeVersion": 1.1
+},
+{
+"id": "091e9e07-89ba-4fe3-9fc5-278fc333dbff",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-160,
+-40
+],
+"parameters": {
+"color": 5,
+"width": 341,
+"height": 951,
+"content": "# Choose your data source \n## This can be swapped for any other data source of your choosing."
+},
+"typeVersion": 1
+},
+{
+"id": "9660a593-9966-4ebe-bfd7-f884dc185d56",
+"name": "If Topic",
+"type": "n8n-nodes-base.if",
+"position": [
+1100,
+320
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "56219de5-244d-4b7f-a511-f3061572cf93",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.output.related }}",
+"rightValue": "yes"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "e47140ac-20cc-417b-a6cd-30f780dc8289",
+"name": "Get Image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1500,
+320
+],
+"parameters": {
+"url": "={{ $('Article Analysis').first().json.output['image urls'] }}",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "26f80f71-2c3a-46fe-a960-21cdbc18ce34",
+"name": "Prompt Settings1",
+"type": "n8n-nodes-base.set",
+"position": [
+2520,
+100
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "56c8f20d-d9d9-4be7-ac2a-38df6ffdd722",
+"name": "model",
+"type": "string",
+"value": "6b645e3a-d64f-4341-a6d8-7a3690fbf042"
+}
+]
+},
+"includeOtherFields": true
+},
+"typeVersion": 3.4
+},
+{
+"id": "ce697f6f-f8fc-4ba7-b776-17bbc2e870b7",
+"name": "Leo - Generate Image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+100
+],
+"parameters": {
+"url": "https://cloud.leonardo.ai/api/rest/v1/generations",
+"method": "POST",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"jsonBody": "={\n \"alchemy\": true,\n \"width\": 1024,\n \"height\": 768,\n \"modelId\": \"6b645e3a-d64f-4341-a6d8-7a3690fbf042\",\n \"num_images\": 1,\n \"presetStyle\": \"MONOCHROME\",\n \"prompt\": \"{{ $json.body.promptGeneration.prompt }}; Use the rule of thirds, leading lines, & balance. DO NOT INCLUDE ANY WORDS OR LABELS.\",\n \"guidance_scale\": 7,\n \"highResolution\": true,\n \"promptMagic\": false,\n \"promptMagicStrength\": 0.5,\n \"promptMagicVersion\": \"v3\",\n \"public\": false,\n \"ultra\": false,\n \"photoReal\": false,\n \"negative_prompt\": \"\"\n} ",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "accept",
+"value": "application/json"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e2067fe5-3fae-4f97-97c0-879967efd9b8",
+"name": "Wait1",
+"type": "n8n-nodes-base.wait",
+"position": [
+3120,
+100
+],
+"webhookId": "256c3814-6a52-4eb1-969a-30f9f3b8e04e",
+"parameters": {
+"amount": 30
+},
+"typeVersion": 1.1
+},
+{
+"id": "f0ba57a5-1d27-4c75-a422-4bc0e2cead9d",
+"name": "Limit",
+"type": "n8n-nodes-base.limit",
+"position": [
+240,
+300
+],
+"parameters": {
+"keep": "lastItems",
+"maxItems": 50
+},
+"typeVersion": 1
+},
+{
+"id": "e01152aa-961b-4e33-a1e3-186d47d81c55",
+"name": "Image Analysis",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+1300,
+320
+],
+"parameters": {
+"modelId": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o-mini",
+"cachedResultName": "GPT-4O-MINI"
+},
+"options": {
+"detail": "auto"
+},
+"resource": "image",
+"imageUrls": "={{ $json.output['image urls'] }}",
+"operation": "analyze"
+},
+"credentials": {
+"openAiApi": {
+"id": "ysxujEYFiY5ozRTS",
+"name": "AlexK OpenAi Key"
+}
+},
+"typeVersion": 1.6
+},
+{
+"id": "ab346129-c3d5-4f51-af5e-5d63cd154981",
+"name": "Wait3",
+"type": "n8n-nodes-base.wait",
+"disabled": true,
+"position": [
+3080,
+1020
+],
+"webhookId": "6e4a0b8d-6c31-4a98-8ec3-2509aa2087e8",
+"parameters": {
+"unit": "minutes"
+},
+"typeVersion": 1.1
+},
+{
+"id": "872c35a3-bdd5-4eec-9bac-0959f3ff78e7",
+"name": "Article Analysis",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"onError": "continueErrorOutput",
+"position": [
+740,
+300
+],
+"parameters": {
+"text": "=Can you tell me if the article at {{ $json.url }} is related to automation or ai? \n\nthen, create a 250 word summary of the article\n\nAlso, list any image url's related to the article content from the url. Limit to 1 image url.",
+"options": {
+"systemMessage": "You are a helpful assistant in summarizing and identifying articles related to automation and ai. \nOutput the results as:\nsummary: \nrelated: yes or no\nimage urls: "
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.7
+},
+{
+"id": "31c3a90e-10ee-4217-9b08-ff57bf17ea10",
+"name": "Dropbox",
+"type": "n8n-nodes-base.dropbox",
+"position": [
+3640,
+1080
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "22ccd0a0-f7f6-40ca-bd09-40ed4a7fcde1",
+"name": "Google Drive",
+"type": "n8n-nodes-base.googleDrive",
+"position": [
+3840,
+1080
+],
+"parameters": {
+"fileId": {
+"__rl": true,
+"mode": "list",
+"value": ""
+},
+"options": {},
+"operation": "update"
+},
+"credentials": {
+"googleDriveOAuth2Api": {
+"id": "m8K1mbAUn7yuiEwl",
+"name": "AlexK1919 Google Drive account"
+}
+},
+"typeVersion": 3
+},
+{
+"id": "ea75931d-c1ee-4139-9bdc-7901056ba016",
+"name": "Microsoft OneDrive",
+"type": "n8n-nodes-base.microsoftOneDrive",
+"position": [
+4040,
+1080
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "38888521-3087-4e0a-81d6-cf4b9a5dd3dd",
+"name": "YouTube",
+"type": "n8n-nodes-base.youTube",
+"position": [
+3640,
+1500
+],
+"parameters": {
+"options": {},
+"resource": "video",
+"operation": "upload"
+},
+"typeVersion": 1
+},
+{
+"id": "55f3decc-f952-4d2a-804d-2aec44fb2755",
+"name": "X",
+"type": "n8n-nodes-base.twitter",
+"position": [
+3840,
+1500
+],
+"parameters": {
+"additionalFields": {}
+},
+"typeVersion": 2
+},
+{
+"id": "54c8b762-444d-4790-97a9-a2f84518492f",
+"name": "Instagram",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+4240,
+1500
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "90040f15-95c0-4ebb-818f-dde508eb0689",
+"name": "LinkedIn",
+"type": "n8n-nodes-base.linkedIn",
+"position": [
+4040,
+1500
+],
+"parameters": {
+"additionalFields": {}
+},
+"typeVersion": 1
+},
+{
+"id": "691eb779-5fae-4f65-89eb-b1b8e5488809",
+"name": "Leo - Improve Prompt2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2720,
+500
+],
+"parameters": {
+"url": "https://cloud.leonardo.ai/api/rest/v1/prompt/improve",
+"method": "POST",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"jsonBody": "={\n \"prompt\": \"{{ $('Article Prep').item.json.message.content['Image Prompt 2'] }}\"\n}",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "accept",
+"value": "application/json"
+}
+]
+}
+},
+"credentials": {
+"httpCustomAuth": {
+"id": "hIzUsjbtHLmIe6uM",
+"name": "RunwayML Custom Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "076a745a-055b-459c-8af9-fa7b6740dc6f",
+"name": "Wait4",
+"type": "n8n-nodes-base.wait",
+"position": [
+2720,
+700
+],
+"webhookId": "89b31515-b403-4644-a2c1-970e5e774008",
+"parameters": {
+"unit": "minutes",
+"amount": 3
+},
+"typeVersion": 1.1
+},
+{
+"id": "adc2c993-3f89-40df-96fc-eb3ff5eafb1c",
+"name": "Wait6",
+"type": "n8n-nodes-base.wait",
+"position": [
+3120,
+500
+],
+"webhookId": "2efb873f-bcbd-41d9-99da-b2b57ef5ad93",
+"parameters": {
+"amount": 30
+},
+"typeVersion": 1.1
+},
+{
+"id": "156f5735-bc20-46a9-871c-143b0772ca45",
+"name": "Leo - Generate Image2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+500
+],
+"parameters": {
+"url": "https://cloud.leonardo.ai/api/rest/v1/generations",
+"method": "POST",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"jsonBody": "={\n \"alchemy\": true,\n \"width\": 1024,\n \"height\": 768,\n \"modelId\": \"6b645e3a-d64f-4341-a6d8-7a3690fbf042\",\n \"num_images\": 1,\n \"presetStyle\": \"MONOCHROME\",\n \"prompt\": \"{{ $json.body.promptGeneration.prompt }}; Use the rule of thirds, leading lines, & balance. DO NOT INCLUDE ANY WORDS OR LABELS.\",\n \"guidance_scale\": 7,\n \"highResolution\": true,\n \"promptMagic\": false,\n \"promptMagicStrength\": 0.5,\n \"promptMagicVersion\": \"v3\",\n \"public\": false,\n \"ultra\": false,\n \"photoReal\": false,\n \"negative_prompt\": \"\"\n} ",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "accept",
+"value": "application/json"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "4f270fa8-4da2-44f0-927f-3509fd9f8f7d",
+"name": "Leo - Get imageId2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3320,
+500
+],
+"parameters": {
+"url": "=https://cloud.leonardo.ai/api/rest/v1/generations/{{ $json.body.sdGenerationJob.generationId }}",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "content-type",
+"value": "application/json"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "49c0e7ba-bf9c-4819-b479-61aa099ab9ab",
+"name": "Runway - Create Video2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2520,
+700
+],
+"parameters": {
+"url": "https://api.dev.runwayml.com/v1/image_to_video",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "promptImage",
+"value": "={{ $json.body.generations_by_pk.generated_images[0].url }}"
+},
+{
+"name": "promptText",
+"value": "string"
+},
+{
+"name": "model",
+"value": "gen3a_turbo"
+}
+]
+},
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "X-Runway-Version",
+"value": "2024-11-06"
+}
+]
+}
+},
+"credentials": {
+"httpCustomAuth": {
+"id": "hIzUsjbtHLmIe6uM",
+"name": "RunwayML Custom Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "d03eb190-5fc0-4b7e-ad65-88ece3ab833d",
+"name": "Runway - Get Video2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+700
+],
+"parameters": {
+"url": "=https://api.dev.runwayml.com/v1/tasks/{{ $json.id }}",
+"options": {},
+"sendHeaders": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth",
+"headerParameters": {
+"parameters": [
+{
+"name": "X-Runway-Version",
+"value": "2024-11-06"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "0072563d-b87d-47c5-80fd-ed3c051b3287",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3580,
+940
+],
+"parameters": {
+"color": 6,
+"width": 882,
+"height": 372,
+"content": "# Upload Assets\nYou can extend this workflow further by uploading the generated assets to your storage option of choice."
+},
+"typeVersion": 1
+},
+{
+"id": "a0b2377e-57ea-47e9-83c9-3e58372610e5",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3580,
+1360
+],
+"parameters": {
+"color": 6,
+"width": 882,
+"height": 372,
+"content": "# Post to Social Media\nYou can extend this workflow further by posting the generated assets to social media."
+},
+"typeVersion": 1
+},
+{
+"id": "708fe6a0-4899-462b-9a08-fadea7c7e195",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2420,
+-40
+],
+"parameters": {
+"color": 4,
+"width": 1114,
+"height": 943,
+"content": "# Generate Images and Videos"
+},
+"typeVersion": 1
+},
+{
+"id": "5bbb6552-ec3a-42ea-a911-993f67a6c8dc",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2420,
+940
+],
+"parameters": {
+"color": 5,
+"width": 1114,
+"height": 372,
+"content": "# Stitch it all together"
+},
+"typeVersion": 1
+},
+{
+"id": "25f4cc09-fbff-4c10-b706-30df5840b794",
+"name": "Cre - Generate Video1",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2880,
+1020
+],
+"parameters": {
+"url": "https://api.creatomate.com/v1/renders",
+"method": "POST",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"jsonBody": "={\n \"max_width\": 480,\n \"template_id\": \"enterTemplateID\",\n \"modifications\": {\n \"Scenes.elements\": [\n {\n \"name\": \"Intro Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Image-1\",\n \"type\": \"image\",\n \"source\": \"{{ $('Leo - Get imageId').item.json.body.generations_by_pk.generated_images[0].url }}\"\n },\n {\n \"name\": \"Subtitles-1\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-1\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,255,255,0.2)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff5900\"\n },\n {\n \"name\": \"Voiceover-1\",\n \"type\": \"audio\",\n \"source\": \"{{ $('News1').item.json.property_name }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n },\n {\n \"name\": \"Auto Scene Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Video-2\",\n \"type\": \"video\",\n \"source\": \"{{ $('Runway - Get Video').first().json.output[0] }}\",\n \"loop\": true\n },\n {\n \"name\": \"Subtitles-2\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-2\",\n \"y\": \"78.2173%\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,255,255,0.2)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff5900\"\n },\n {\n \"name\": \"Voiceover-2\",\n \"type\": \"audio\",\n \"source\": \"{{ $('Article Prep').item.json.message.content['Summary Blurb 1'] }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n },\n {\n \"name\": \"Auto Scene Comp\",\n \"type\": \"composition\",\n \"track\": 1,\n \"elements\": [\n {\n \"name\": \"Video-3\",\n \"type\": \"video\",\n \"source\": \"{{ $('Runway - Get Video2').first().json.output[0] }}\",\n \"loop\": true\n },\n {\n \"name\": \"Subtitles-3\",\n \"type\": \"text\",\n \"transcript_source\": \"Voiceover-3\",\n \"y\": \"78.2173%\",\n \"width\": \"86.66%\",\n \"height\": \"37.71%\",\n \"x_alignment\": \"50%\",\n \"y_alignment\": \"50%\",\n \"fill_color\": \"#ffffff\",\n \"stroke_color\": \"#333333\",\n \"stroke_width\": \"1.05 vmin\",\n \"font_family\": \"Inter\",\n \"font_weight\": \"700\",\n \"font_size\": \"8 vmin\",\n \"background_color\": \"rgba(255,89,0,0.5)\",\n \"background_x_padding\": \"26%\",\n \"background_y_padding\": \"7%\",\n \"background_border_radius\": \"28%\",\n \"transcript_effect\": \"highlight\",\n \"transcript_color\": \"#ff0040\"\n },\n {\n \"name\": \"Voiceover-3\",\n \"type\": \"audio\",\n \"source\": \"{{ $('Article Prep').item.json.message.content['Summary Blurb 2'] }}\",\n \"provider\": \"openai model=tts-1 voice=onyx\"\n }\n ]\n }\n ]\n }\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth"
+},
+"credentials": {
+"httpCustomAuth": {
+"id": "hIzUsjbtHLmIe6uM",
+"name": "RunwayML Custom Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "7093de7b-a4e3-4363-8038-1002f7b20fbc",
+"name": "Cre - Get Video",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3280,
+1020
+],
+"parameters": {
+"url": "=https://api.creatomate.com/v1/renders/{{ $json.body.body[0].id }}",
+"options": {
+"response": {
+"response": {
+"fullResponse": true
+}
+}
+},
+"authentication": "genericCredentialType",
+"genericAuthType": "httpCustomAuth"
+},
+"credentials": {
+"httpCustomAuth": {
+"id": "hIzUsjbtHLmIe6uM",
+"name": "RunwayML Custom Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "a57b719f-b299-431e-9c85-fa333e38b6a7",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+660,
+-40
+],
+"parameters": {
+"color": 3,
+"width": 1033,
+"height": 951,
+"content": "# Article Analysis - Is it the right topic?"
+},
+"typeVersion": 1
+},
+{
+"id": "60b879a0-8b7f-40f1-ae70-ac94e4675b38",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1740,
+-40
+],
+"parameters": {
+"color": 3,
+"width": 630,
+"height": 947,
+"content": "# Prepare the article for content generation"
+},
+"typeVersion": 1
+},
+{
+"id": "afaf8437-ee52-434b-a267-8dbaff0e1922",
+"name": "Article Prep",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+1820,
+320
+],
+"parameters": {
+"modelId": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o-mini",
+"cachedResultName": "GPT-4O-MINI"
+},
+"options": {},
+"messages": {
+"values": [
+{
+"content": "=prepare the following summary for a newsletter where the article will be 1 of several presented in the newsletter:\n\n{{ $('Article Analysis').first().json.output.summary }}\n\nMake sure the Article Blurb lenght is less than 15 words.\n\nThen, create 2 Summary Blurbs, making sure each is less than 15 words.\n\nAlso create 2 image prompts that is less than 15 words long for each Summary Blurb"
+},
+{
+"role": "system",
+"content": "Output in markdown format\nArticle Title\nArticle Blurb\nSummary Blurb 1\nSummary Blurb 2\nArticle Image\nImage Prompt 1\nImage Prompt 2"
+}
+]
+},
+"jsonOutput": true
+},
+"credentials": {
+"openAiApi": {
+"id": "ysxujEYFiY5ozRTS",
+"name": "AlexK OpenAi Key"
+}
+},
+"typeVersion": 1.6
+},
+{
+"id": "e7c95d56-86e1-4456-a6d3-9c8b9fc3a53c",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-620,
+-40
+],
+"parameters": {
+"color": 6,
+"width": 252,
+"height": 946,
+"content": "# AlexK1919 \n\n\n#### Iโm Alex Kim, an AI-Native Workflow Automation Architect Building Solutions to Optimize your Personal and Professional Life.\n\n### Workflow Overview Video\nhttps://youtu.be/XaKybLDUlLk\n\n### About Me\nhttps://beacons.ai/alexk1919\n\n### Product Used \n[Leonardo.ai](https://leonardo.ai)\n[RunwayML](https://runwayml.com/)\n[Creatomate](https://creatomate.com/)\n"
+},
+"typeVersion": 1
+},
+{
+"id": "32e2803e-bf7c-4da4-a4ae-c9b6fa5ae226",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3280,
+1180
+],
+"parameters": {
+"color": 7,
+"width": 180,
+"height": 100,
+"content": "Don't forget to connect this last node to the loop to process additional items"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "c7ab1ecd-50cb-4e4b-b2f7-aade804bbd63",
+"connections": {
+"X": {
+"main": [
+[
+{
+"node": "LinkedIn",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Limit": {
+"main": [
+[
+{
+"node": "Loop Over Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"News1": {
+"main": [
+[
+{
+"node": "Prompt Settings1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait1": {
+"main": [
+[
+{
+"node": "Leo - Get imageId",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait2": {
+"main": [
+[
+{
+"node": "Runway - Get Video",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait3": {
+"main": [
+[
+{
+"node": "Cre - Get Video",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait4": {
+"main": [
+[
+{
+"node": "Runway - Get Video2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait6": {
+"main": [
+[
+{
+"node": "Leo - Get imageId2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Dropbox": {
+"main": [
+[
+{
+"node": "Google Drive",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"YouTube": {
+"main": [
+[
+{
+"node": "X",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If Topic": {
+"main": [
+[
+{
+"node": "Image Analysis",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Loop Over Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"LinkedIn": {
+"main": [
+[
+{
+"node": "Instagram",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Image": {
+"main": [
+[
+{
+"node": "Article Prep",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Hacker News": {
+"main": [
+[
+{
+"node": "Limit",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Article Prep": {
+"main": [
+[
+{
+"node": "News1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Drive": {
+"main": [
+[
+{
+"node": "Microsoft OneDrive",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"HTTP Request1": {
+"ai_tool": [
+[
+{
+"node": "Article Analysis",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Image Analysis": {
+"main": [
+[
+{
+"node": "Get Image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Loop Over Items": {
+"main": [
+[],
+[
+{
+"node": "Article Analysis",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Article Analysis": {
+"main": [
+[
+{
+"node": "If Topic",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Loop Over Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prompt Settings1": {
+"main": [
+[
+{
+"node": "Leo - Improve Prompt",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Get imageId": {
+"main": [
+[
+{
+"node": "Runway - Create Video",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Get imageId2": {
+"main": [
+[
+{
+"node": "Runway - Create Video2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Microsoft OneDrive": {
+"main": [
+[
+{
+"node": "Upload to Minio",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model3": {
+"ai_languageModel": [
+[
+{
+"node": "Article Analysis",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Runway - Get Video": {
+"main": [
+[
+{
+"node": "Leo - Improve Prompt2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Runway - Get Video2": {
+"main": [
+[
+{
+"node": "Cre - Generate Video1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Generate Image": {
+"main": [
+[
+{
+"node": "Wait1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Improve Prompt": {
+"main": [
+[
+{
+"node": "Leo - Generate Image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Cre - Generate Video1": {
+"main": [
+[
+{
+"node": "Wait3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Generate Image2": {
+"main": [
+[
+{
+"node": "Wait6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Leo - Improve Prompt2": {
+"main": [
+[
+{
+"node": "Leo - Generate Image2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Runway - Create Video": {
+"main": [
+[
+{
+"node": "Wait2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Runway - Create Video2": {
+"main": [
+[
+{
+"node": "Wait4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Article Analysis",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Hacker News",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.json b/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.json
deleted file mode 100644
index ad2151f..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.json
+++ /dev/null
@@ -1,2942 +0,0 @@
-{
- "meta": {
- "instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9",
- "templateCredsSetupCompleted": true
- },
- "nodes": [
- {
- "id": "645ae2b1-799e-49be-8bdf-12cd1bb739e6",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 1680,
- 1140
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"learnings\": {\n \"type\": \"array\",\n \"description\": \"List of learnings, max of 3.\",\n \"items\": { \"type\": \"string\" }\n },\n \"followUpQuestions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\",\n \"description\": \"List of follow-up questions to research the topic further, max of 3.\"\n }\n }\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "cbdb4e98-eeba-4609-91de-394c416b7904",
- "name": "Set Variables",
- "type": "n8n-nodes-base.set",
- "position": [
- -1360,
- -460
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "df28b12e-7c20-4ff5-b5b8-dc773aa14d4b",
- "name": "request_id",
- "type": "string",
- "value": "={{ $execution.id }}"
- },
- {
- "id": "9362c1e7-717d-444a-8ea2-6b5f958c9f3f",
- "name": "prompt",
- "type": "string",
- "value": "={{ $json['What would you like to research?'] }}"
- },
- {
- "id": "09094be4-7844-4a9e-af82-cc8e39322398",
- "name": "depth",
- "type": "number",
- "value": "={{ $json['Enter research depth (recommended 1-5, default 2)'] || 2 }}"
- },
- {
- "id": "3fc30a30-7806-4013-835d-97e27ddd7ae1",
- "name": "breadth",
- "type": "number",
- "value": "={{ $json['Enter research breadth (recommended 2-10, default 4)'] || 4 }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "c7096ab9-0b10-45b0-b178-a049bf57830b",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1500,
- 1140
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "id",
- "value": "o3-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "d0f1bc2f-6a10-4ac7-8d35-34f48f14fad5",
- "name": "OpenAI Chat Model1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -860,
- 1760
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "id",
- "value": "o3-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "bba3278c-0336-4305-887d-56515dfd87db",
- "name": "OpenAI Chat Model2",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -1060,
- -300
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "id",
- "value": "o3-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "f31f2fc7-0bec-4105-9d83-5f4f9a0eb35d",
- "name": "Structured Output Parser1",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- -840,
- -300
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"Follow up questions to clarify the research direction, max of 3.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "ea59c5ab-fa05-4c68-bc60-3f56e240478b",
- "name": "On form submission",
- "type": "n8n-nodes-base.formTrigger",
- "position": [
- -1760,
- -460
- ],
- "webhookId": "7ddfaa7c-a523-4d92-b033-d76cd5a313e9",
- "parameters": {
- "options": {
- "path": "deep_research",
- "ignoreBots": true,
- "buttonLabel": "Next"
- },
- "formTitle": " DeepResearcher",
- "formFields": {
- "values": [
- {
- "fieldType": "html",
- "fieldLabel": "placeholder"
- }
- ]
- },
- "formDescription": "=DeepResearcher is a multi-step, recursive approach using the internet to solve complex research tasks, accomplishing in tens of minutes what a human would take many hours.\n\nTo use, provide a short summary of what the research and how \"deep\" you'd like the workflow to investigate. Note, the higher the numbers the more time and cost will occur for the research.\n\nThe workflow is designed to complete independently and when finished, a report will be saved in a designated Notion Database."
- },
- "typeVersion": 2.2
- },
- {
- "id": "a8262288-a8c1-4967-9870-f728fa08b579",
- "name": "Generate SERP Queries",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -1040,
- 820
- ],
- "parameters": {
- "text": "=Given the following prompt from the user, generate a list of SERP queries to research the topic. Return a maximum of {{ $('JobType Router').first().json.data.breadth }} queries, but feel free to return less if the original prompt is clear. Make sure each query is unique and not similar to each other: {{ $('JobType Router').first().json.data.query.trim() }}\n\n{{\n$('JobType Router').first().json.data.learnings.length\n ? `Here are some learnings from previous research, use them to generate more specific queries: ${$('JobType Router').first().json.data.learnings.join('\\n')}`\n : ''\n}}",
- "messages": {
- "messageValues": [
- {
- "type": "HumanMessagePromptTemplate",
- "message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.5
- },
- {
- "id": "0534be47-22b7-4c2a-956b-d085e6b9f280",
- "name": "Structured Output Parser2",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- -860,
- 980
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"queries\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": {\n \"type\": \"string\",\n \"description\": \"The SERP query\"\n },\n \"researchGoal\": {\n \"type\": \"string\",\n \"description\": \"First talk about the goal of the research that this query is meant to accomplish, then go deeper into how to advance the research once the results are found, mention additional research directions. Be as specific as possible, especially for additional research directions.\"\n }\n }\n }\n }\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "4d8aa196-986f-442d-9b56-92c043ab785d",
- "name": "OpenAI Chat Model3",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -1040,
- 980
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "id",
- "value": "o3-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "7488b037-7422-4f62-8c37-1f6a901b3299",
- "name": "Set Initial Query",
- "type": "n8n-nodes-base.set",
- "position": [
- -580,
- 180
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "acb41e93-70c6-41a3-be0f-e5a74ec3ec88",
- "name": "query",
- "type": "string",
- "value": "={{ $('JobType Router').first().json.data.query }}"
- },
- {
- "id": "7fc54063-b610-42bc-a250-b1e8847c4d1e",
- "name": "learnings",
- "type": "array",
- "value": "={{ $('JobType Router').first().json.data.learnings }}"
- },
- {
- "id": "e8f1c158-56fb-41c8-8d86-96add16289bb",
- "name": "breadth",
- "type": "number",
- "value": "={{ $('JobType Router').first().json.data.breadth }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "12ae382e-d88a-4f1b-a71f-3bd63c892b17",
- "name": "SERP to Items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- -700,
- 820
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "output.queries"
- },
- "typeVersion": 1
- },
- {
- "id": "46700052-f48a-493c-aebf-cdf175d58550",
- "name": "Item Ref",
- "type": "n8n-nodes-base.noOp",
- "position": [
- -240,
- 980
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "2cef6f1d-e244-4ee6-bf25-6dc3e8042afa",
- "name": "Research Goal + Learnings",
- "type": "n8n-nodes-base.set",
- "position": [
- 1840,
- 1120
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "9acec2cc-64c8-4e62-bed4-c3d9ffab1379",
- "name": "researchGoal",
- "type": "string",
- "value": "={{ $('Item Ref').first().json.researchGoal }}"
- },
- {
- "id": "1b2d2dad-429b-4fc9-96c5-498f572a85c3",
- "name": "learnings",
- "type": "array",
- "value": "={{ $json.output.learnings }}"
- },
- {
- "id": "655b99f2-6045-4774-a634-49751bc9326f",
- "name": "followUpQuestions",
- "type": "array",
- "value": "={{ $json.output.followUpQuestions }}"
- },
- {
- "id": "c9e34ea4-5606-46d6-8d66-cb42d772a8b4",
- "name": "urls",
- "type": "array",
- "value": "={{\n$('Page Contents')\n .all()\n .filter(item => !item.json.error && item.json.body)\n .map(item => item.json.url)\n}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "4aebae86-2bd2-4f3d-8290-d34b9ac837c6",
- "name": "Accumulate Results",
- "type": "n8n-nodes-base.set",
- "position": [
- -200,
- 180
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "db509e90-9a86-431f-8149-4094d22666cc",
- "name": "should_stop",
- "type": "boolean",
- "value": "={{\n$runIndex >= ($('JobType Router').first().json.data.depth)\n}}"
- },
- {
- "id": "90986e2b-8aca-4a22-a9db-ed8809d6284d",
- "name": "all_learnings",
- "type": "array",
- "value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.learnings))\n } catch (e) {\n return []\n }\n })\n}}"
- },
- {
- "id": "3eade958-e8ab-4975-aac4-f4a4a983c163",
- "name": "all_urls",
- "type": "array",
- "value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.urls))\n } catch (e) {\n return []\n }\n })\n}}"
- }
- ]
- }
- },
- "executeOnce": true,
- "typeVersion": 3.4
- },
- {
- "id": "782baa36-ba07-4845-873c-c9400de6d463",
- "name": "DeepResearch Results",
- "type": "n8n-nodes-base.set",
- "position": [
- 160,
- 360
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "={{ $('Generate Learnings').item.json }}"
- },
- "typeVersion": 3.4
- },
- {
- "id": "89b09898-79ec-4924-975f-e9581d3bf774",
- "name": "Results to Items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 320,
- 360
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "data"
- },
- "typeVersion": 1
- },
- {
- "id": "122cd071-aade-4753-ba0a-8db4c58fa84e",
- "name": "Set Next Queries",
- "type": "n8n-nodes-base.set",
- "position": [
- 480,
- 360
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "d88bfe95-9e73-4d25-b45c-9f164b940b0e",
- "name": "query",
- "type": "string",
- "value": "=Previous research goal: {{ $json.researchGoal }}\nFollow-up research directions: {{ $json.followUpQuestions.map(q => `\\n${q}`).join('') }}"
- },
- {
- "id": "4aa20690-d998-458a-b1e4-0d72e6a68e6b",
- "name": "learnings",
- "type": "array",
- "value": "={{ $('Accumulate Results').item.json.all_learnings }}"
- },
- {
- "id": "89acafae-b04a-4d5d-b08b-656e715654e4",
- "name": "breadth",
- "type": "number",
- "value": "={{ $('JobType Router').first().json.data.breadth }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "9da01d8a-48d6-45b4-b8c6-9a0503b4bda6",
- "name": "Web Search",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- -80,
- 980
- ],
- "parameters": {
- "url": "https://api.apify.com/v2/acts/serping~fast-google-search-results-scraper/run-sync-get-dataset-items",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "authentication": "genericCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "searchTerms",
- "value": "={{\n[\n `${$json.query} -filetype:pdf`\n]\n}}"
- },
- {
- "name": "resultsPerPage",
- "value": "={{ 10 }}"
- }
- ]
- },
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "cO2w8RDNOZg8DRa8",
- "name": "Apify API"
- },
- "httpHeaderAuth": {
- "id": "SV9BDKc1cRbZBeoL",
- "name": "Apify.com (personal token)"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "99bd2c8e-5600-43a9-ab2f-7f2911efb16c",
- "name": "Top 5 Organic Results",
- "type": "n8n-nodes-base.set",
- "position": [
- 80,
- 980
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "29d1a759-d886-4a44-860b-9d16f9922043",
- "name": "results",
- "type": "array",
- "value": "={{\n$json.origin_search.results\n ? $json.origin_search\n .results\n .filter(res => res.type === 'normal')\n .slice(0, 5)\n .map(res => ({\n title: res.title,\n url: res.source.link\n }))\n : []\n}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "cb7c5a8b-5420-4fb9-b7f0-4e8e8d10034a",
- "name": "Convert to Markdown",
- "type": "n8n-nodes-base.markdown",
- "position": [
- 1320,
- 980
- ],
- "parameters": {
- "html": "={{ $json.body }}",
- "options": {
- "ignore": "a,img,picture,svg,video,audio,iframe"
- },
- "destinationKey": "markdown"
- },
- "typeVersion": 1
- },
- {
- "id": "818ccf2e-081d-492e-ba8d-de458b0c26db",
- "name": "For Each Query...",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -420,
- 820
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "1787b562-17e8-41af-9cdc-eb2d3e630916",
- "name": "Feedback to Items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- -720,
- -460
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "output.questions"
- },
- "typeVersion": 1
- },
- {
- "id": "4c695faa-74e3-456b-a1ef-aaea67e46743",
- "name": "Ask Clarity Questions",
- "type": "n8n-nodes-base.form",
- "position": [
- -360,
- -380
- ],
- "webhookId": "ab0d9b81-73f6-4baa-a3cd-ac3b31397708",
- "parameters": {
- "options": {
- "formTitle": "DeepResearcher",
- "buttonLabel": "Answer",
- "formDescription": "=\n
\nAnswer the following clarification questions to assist the DeepResearcher better under the research topic.\n
\n\n
\nTotal {{ $('Feedback to Items').all().length }} questions.\n
"
- },
- "formFields": {
- "values": [
- {
- "fieldType": "textarea",
- "fieldLabel": "={{ $json[\"output.questions\"] }}",
- "placeholder": "=",
- "requiredField": true
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "e07d8c3e-8bcd-4393-9892-f825433ab58d",
- "name": "For Each Question...",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -540,
- -460
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "e8d26351-52f4-40a6-ba5b-fb6bc816b734",
- "name": "DeepResearch Subworkflow",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- -1880,
- 820
- ],
- "parameters": {
- "workflowInputs": {
- "values": [
- {
- "name": "requestId",
- "type": "any"
- },
- {
- "name": "jobType"
- },
- {
- "name": "data",
- "type": "object"
- }
- ]
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "25a8055a-27aa-414f-856b-25a2e2f31974",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1140,
- -680
- ],
- "parameters": {
- "color": 7,
- "width": 1000,
- "height": 560,
- "content": "## 2. Ask Clarifying Questions\n[Read more about form nodes](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.form/)\n\nTo handle the clarification questions generated by the LLM, I used the same technique found in my \"AI Interviewer\" template ([link](https://n8n.io/workflows/2566-conversational-interviews-with-ai-agents-and-n8n-forms/)).\nThis involves a looping of dynamically generated forms to collect answers from the user."
- },
- "typeVersion": 1
- },
- {
- "id": "68398b92-eb35-48bf-885e-540074531cc4",
- "name": "Clarifying Questions",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -1040,
- -460
- ],
- "parameters": {
- "text": "=Given the following query from the user, ask some follow up questions to clarify the research direction. Return a maximum of 3 questions, but feel free to return less if the original query is clear: {{ $json.prompt }}`",
- "messages": {
- "messageValues": [
- {
- "type": "HumanMessagePromptTemplate",
- "message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.5
- },
- {
- "id": "65c4c293-67b8-4e64-af04-16e45e97c09a",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -660,
- -60
- ],
- "parameters": {
- "color": 7,
- "width": 1360,
- "height": 640,
- "content": "## 6. Perform DeepSearch Loop\n[Learn more about the Looping in n8n](https://docs.n8n.io/flow-logic/looping/#creating-loops)\n\nThe key of the Deep Research flow is its extensive data collection capability. In this implementation, this capability is represented by a recursive web search & scrape loop which starts with the original query and extended by AI-generated subqueries. How many subqueries to generate are determined the depth and breadth parameters specified.\n\n\"Learnings\" are generated for each subquery and accumulate on each iteration of the loop. When the loop finishes when depth limit is reached, all learnings are collected and it's these learnings are what we use to generate the report."
- },
- "typeVersion": 1
- },
- {
- "id": "43a5d93d-cae2-43ec-b9ae-b15d6b11b932",
- "name": "End Form",
- "type": "n8n-nodes-base.form",
- "position": [
- 960,
- -420
- ],
- "webhookId": "7b531f5d-942f-4c49-ac55-8ee480889600",
- "parameters": {
- "options": {},
- "operation": "completion",
- "completionTitle": "=Thank you for using DeepResearcher.",
- "completionMessage": "=You may now close this window."
- },
- "typeVersion": 1
- },
- {
- "id": "9a824011-e76f-433f-8735-44b358f4ff7d",
- "name": "Initiate DeepResearch",
- "type": "n8n-nodes-base.executeWorkflow",
- "position": [
- 600,
- -420
- ],
- "parameters": {
- "mode": "each",
- "options": {
- "waitForSubWorkflow": false
- },
- "workflowId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $workflow.id }}"
- },
- "workflowInputs": {
- "value": {
- "data": "={{\n{\n \"query\": $('Get Initial Query').first().json.query,\n \"learnings\": [],\n \"depth\": $('Set Variables').first().json.depth,\n \"breadth\": $('Set Variables').first().json.breadth,\n}\n}}",
- "jobType": "deepresearch_initiate",
- "requestId": "={{ $('Set Variables').first().json.request_id }}"
- },
- "schema": [
- {
- "id": "requestId",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "requestId",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "jobType",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "jobType",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "data",
- "type": "object",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "data",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [],
- "attemptToConvertTypes": false,
- "convertFieldsToString": true
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "c48ee4cd-bac1-4405-bb4c-5614e5eb25a0",
- "name": "Page Contents",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 560,
- 980
- ],
- "parameters": {
- "url": "https://api.apify.com/v2/acts/apify~web-scraper/run-sync-get-dataset-items",
- "options": {},
- "jsonBody": "={\n \"startUrls\": {{ [{ url: $json.url, method: 'GET' }].toJsonString() }},\n \"breakpointLocation\": \"NONE\",\n \"browserLog\": false,\n \"closeCookieModals\": false,\n \"debugLog\": false,\n \"downloadCss\": false,\n \"downloadMedia\": false,\n \"excludes\": [\n {\n \"glob\": \"/**/*.{png,jpg,jpeg,pdf}\"\n }\n ],\n \"headless\": true,\n \"ignoreCorsAndCsp\": false,\n \"ignoreSslErrors\": false,\n \"injectJQuery\": true,\n \"keepUrlFragments\": false,\n \"linkSelector\": \"\",\n \"maxCrawlingDepth\": 1,\n \"maxPagesPerCrawl\": 1,\n \"maxRequestRetries\": 1,\n \"maxResultsPerCrawl\": 1,\n \"pageFunction\": \"// The function accepts a single argument: the \\\"context\\\" object.\\n// For a complete list of its properties and functions,\\n// see https://apify.com/apify/web-scraper#page-function \\nasync function pageFunction(context) {\\n\\n await new Promise(res => { setTimeout(res, 6000) });\\n // This statement works as a breakpoint when you're trying to debug your code. Works only with Run mode: DEVELOPMENT!\\n // debugger; \\n\\n // jQuery is handy for finding DOM elements and extracting data from them.\\n // To use it, make sure to enable the \\\"Inject jQuery\\\" option.\\n const $ = context.jQuery;\\n const title = $('title').first().text();\\n\\n // Clone the body to avoid modifying the original content\\n const bodyClone = $('body').clone();\\n bodyClone.find('iframe, img, script, style, object, embed, noscript, svg, video, audio').remove();\\n const body = bodyClone.html();\\n\\n // Return an object with the data extracted from the page.\\n // It will be stored to the resulting dataset.\\n return {\\n url: context.request.url,\\n title,\\n body\\n };\\n}\",\n \"postNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept a single argument: the \\\"crawlingContext\\\" object.\\n[\\n async (crawlingContext) => {\\n // ...\\n },\\n]\",\n \"preNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept two arguments: the \\\"crawlingContext\\\" object\\n// and \\\"gotoOptions\\\".\\n[\\n async (crawlingContext, gotoOptions) => {\\n // ...\\n },\\n]\\n\",\n \"proxyConfiguration\": {\n \"useApifyProxy\": true\n },\n \"runMode\": \"PRODUCTION\",\n \"useChrome\": false,\n \"waitUntil\": [\n \"domcontentloaded\"\n ],\n \"globs\": [],\n \"pseudoUrls\": [],\n \"proxyRotation\": \"RECOMMENDED\",\n \"maxConcurrency\": 50,\n \"pageLoadTimeoutSecs\": 60,\n \"pageFunctionTimeoutSecs\": 60,\n \"maxScrollHeightPixels\": 5000,\n \"customData\": {}\n}",
- "sendBody": true,
- "sendQuery": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "memory",
- "value": "2048"
- },
- {
- "name": "timeout",
- "value": "90"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "cO2w8RDNOZg8DRa8",
- "name": "Apify API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "dc9f85ff-7565-4c29-981a-5ef65bba6ca3",
- "name": "Execution Data",
- "type": "n8n-nodes-base.executionData",
- "position": [
- -1700,
- 820
- ],
- "parameters": {
- "dataToSave": {
- "values": [
- {
- "key": "requestId",
- "value": "={{ $json.requestId }}"
- },
- {
- "key": "=jobType",
- "value": "={{ $json.jobType }}"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "26b33429-6d61-4758-9c76-3e998dd31fa4",
- "name": "JobType Router",
- "type": "n8n-nodes-base.switch",
- "position": [
- -1520,
- 820
- ],
- "parameters": {
- "rules": {
- "values": [
- {
- "outputKey": "initiate",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "operator": {
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.jobType }}",
- "rightValue": "deepresearch_initiate"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "learnings",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "ecbfa54d-fc97-48c5-8d3d-f0538b8d727b",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.jobType }}",
- "rightValue": "deepresearch_learnings"
- }
- ]
- },
- "renameOutput": true
- },
- {
- "outputKey": "report",
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "392f9a98-ec22-4e57-9c8e-0e1ed6b7dafa",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.jobType }}",
- "rightValue": "deepresearch_report"
- }
- ]
- },
- "renameOutput": true
- }
- ]
- },
- "options": {}
- },
- "typeVersion": 3.2
- },
- {
- "id": "a9637952-7c09-40ae-96ec-bdf0fc63d94e",
- "name": "Valid Pages",
- "type": "n8n-nodes-base.filter",
- "position": [
- 720,
- 980
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "f44691e4-f753-47b0-b66a-068a723b6beb",
- "operator": {
- "type": "boolean",
- "operation": "false",
- "singleValue": true
- },
- "leftValue": "={{ $json['#error'] }}",
- "rightValue": ""
- },
- {
- "id": "8e05df2b-0d4a-47da-9aab-da7e8907cbca",
- "operator": {
- "type": "string",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $json.body }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2,
- "alwaysOutputData": true
- },
- {
- "id": "204cfca2-05bb-46dd-ba96-b41866ed2cfe",
- "name": "OpenAI Chat Model4",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -20,
- -280
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "id",
- "value": "o3-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "45bc6261-35c8-4994-bb88-ed7a0f022767",
- "name": "Get Initial Query",
- "type": "n8n-nodes-base.set",
- "position": [
- -360,
- -540
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "14b77741-c3c3-4bd2-be6e-37bd09fcea2b",
- "name": "query",
- "type": "string",
- "value": "=Initial query: {{ $('Set Variables').first().json.prompt }}\nFollow-up Questions and Answers:\n{{\n$input.all()\n .map(item => {\n const q = Object.keys(item.json)[0];\n const a = item.json[q];\n return `question: ${q}\\nanswer: ${a}`;\n })\n .join('\\n')\n}}"
- }
- ]
- }
- },
- "executeOnce": true,
- "typeVersion": 3.4
- },
- {
- "id": "26d26e54-ee9b-4714-ae27-4f033dc825d3",
- "name": "Structured Output Parser4",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 160,
- -280
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\":\" A short title summarising the research topic\"\n },\n \"description\": {\n \"type\": \"string\",\n \"description\": \"A short description to summarise the research topic\"\n }\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "3842bc1d-d5f9-4879-bc06-db20fed3f55d",
- "name": "Create Row",
- "type": "n8n-nodes-base.notion",
- "position": [
- 300,
- -420
- ],
- "parameters": {
- "title": "={{ $json.output.title }}",
- "options": {},
- "resource": "databasePage",
- "databaseId": {
- "__rl": true,
- "mode": "list",
- "value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
- "cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
- "cachedResultName": "n8n DeepResearch"
- },
- "propertiesUi": {
- "propertyValues": [
- {
- "key": "Description|rich_text",
- "textContent": "={{ $json.output.description }}"
- },
- {
- "key": "Status|status",
- "statusValue": "Not started"
- },
- {
- "key": "Request ID|rich_text",
- "textContent": "={{ $('Set Variables').first().json.request_id }}"
- },
- {
- "key": "Name|title",
- "title": "={{ $json.output.title }}"
- }
- ]
- }
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "bfe98996-6ed5-4f60-afdd-a947a6fa6e36",
- "name": "Report Page Generator",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -20,
- -420
- ],
- "parameters": {
- "text": "=Create a suitable title for the research report which will be created from the user's query.\n{{ $json.query }}",
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.5
- },
- {
- "id": "ff05add8-94b0-4495-8f4e-3e8a10c556af",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -120,
- -680
- ],
- "parameters": {
- "color": 7,
- "width": 600,
- "height": 560,
- "content": "## 3. Create Empty Report Page in Notion\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)\n\nSome thought was given where to upload the final report and Notion was selected due to familiarity. This can be easily changed to whatever wiki tools you prefer.\n\nIf you're following along however, here's the Notion database you need to replicate - [Jim's n8n DeepResearcher Database](https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf)."
- },
- "typeVersion": 1
- },
- {
- "id": "5bc13d62-81e1-4730-b7e6-9e5579dff174",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 500,
- -680
- ],
- "parameters": {
- "color": 7,
- "width": 640,
- "height": 560,
- "content": "## 4. Trigger DeepResearch Asynchronously\n[Learn more about the Execute Trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/)\n\nn8n handles asynchronous jobs by spinning them off as separate executions. This basically means the user doesn't have to wait or keep their browser window open for our researcher to do its job.\n\nOnce we initiate the Deepresearcher job, we can close out the onboarding journey for a nice user experience."
- },
- "typeVersion": 1
- },
- {
- "id": "9fea6403-b2a2-4e67-99a2-b7a2f29a1e96",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1160,
- 620
- ],
- "parameters": {
- "color": 7,
- "width": 620,
- "height": 540,
- "content": "## 7. Generate Search Queries\n[Learn more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nMuch like a human researcher, the DeepResearcher will rely on web search and content as the preferred source of information. To ensure it can cover a wide range of sources, the AI can first generate relevant research queries of which each can be explored separately."
- },
- "typeVersion": 1
- },
- {
- "id": "0bccdc54-7570-4bca-93ec-cb140c5bd3a1",
- "name": "Is Depth Reached?",
- "type": "n8n-nodes-base.if",
- "position": [
- -40,
- 180
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "75d18d88-6ba6-43df-bef7-3e8ad99ad8bd",
- "operator": {
- "type": "boolean",
- "operation": "true",
- "singleValue": true
- },
- "leftValue": "={{ $json.should_stop }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "819aa5be-b71b-44a7-b062-b2a50209f290",
- "name": "URLs to Items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 400,
- 980
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "results"
- },
- "typeVersion": 1
- },
- {
- "id": "4b8e9936-4b24-4bd4-8fe7-75d58244cb6d",
- "name": "Get Research Results",
- "type": "n8n-nodes-base.set",
- "position": [
- 160,
- 180
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "90b3da00-dcd5-4289-bd45-953146a3b0ba",
- "name": "all_learnings",
- "type": "array",
- "value": "={{ $json.all_learnings }}"
- },
- {
- "id": "623dbb3d-83a1-44a9-8ad3-48d92bc42811",
- "name": "all_urls",
- "type": "array",
- "value": "={{ $json.all_urls }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "d371535a-2946-4ec5-9be6-2ee8e359ac44",
- "name": "Get Existing Row",
- "type": "n8n-nodes-base.notion",
- "position": [
- -1040,
- 180
- ],
- "parameters": {
- "limit": 1,
- "filters": {
- "conditions": [
- {
- "key": "Request ID|rich_text",
- "condition": "equals",
- "richTextValue": "={{ $json.requestId.toString() }}"
- }
- ]
- },
- "options": {},
- "resource": "databasePage",
- "matchType": "allFilters",
- "operation": "getAll",
- "databaseId": {
- "__rl": true,
- "mode": "list",
- "value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
- "cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
- "cachedResultName": "n8n DeepResearch"
- },
- "filterType": "manual"
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "fea4c30e-1193-494d-8823-dfbec5196a0d",
- "name": "Set In-Progress",
- "type": "n8n-nodes-base.notion",
- "position": [
- -840,
- 180
- ],
- "parameters": {
- "pageId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $json.id }}"
- },
- "options": {},
- "resource": "databasePage",
- "operation": "update",
- "propertiesUi": {
- "propertyValues": [
- {
- "key": "Status|status",
- "statusValue": "In progress"
- }
- ]
- }
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "37954acd-d8cb-4c74-afa8-d8973e017327",
- "name": "Set Done",
- "type": "n8n-nodes-base.notion",
- "position": [
- 1680,
- 1600
- ],
- "parameters": {
- "pageId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Get Existing Row1').first().json.id }}"
- },
- "options": {},
- "resource": "databasePage",
- "operation": "update",
- "propertiesUi": {
- "propertyValues": [
- {
- "key": "Status|status",
- "statusValue": "Done"
- },
- {
- "key": "Last Updated|date",
- "date": "={{ $now.toISO() }}"
- }
- ]
- }
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "executeOnce": true,
- "typeVersion": 2.2
- },
- {
- "id": "3db97ab8-b934-4567-a92e-92374a363df6",
- "name": "Tags to Items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- -60,
- 1600
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "tag"
- },
- "typeVersion": 1
- },
- {
- "id": "7f468bf7-762a-4818-86f9-54d172bb618a",
- "name": "Convert to HTML",
- "type": "n8n-nodes-base.markdown",
- "position": [
- -380,
- 1600
- ],
- "parameters": {
- "mode": "markdownToHtml",
- "options": {
- "tables": true
- },
- "markdown": "={{ $json.text }}"
- },
- "typeVersion": 1
- },
- {
- "id": "97914ee9-0ee8-408b-b2bb-a7193b2d0454",
- "name": "HTML to Array",
- "type": "n8n-nodes-base.set",
- "position": [
- -220,
- 1600
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "851b8a3f-c2d3-41ad-bf60-4e0e667f6c58",
- "name": "tag",
- "type": "array",
- "value": "={{ $json.data.match(/
|
|<[^>]+>[^<]*<\\/[^>]+>/g) }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "6ce79f16-51e3-4192-8103-738222be558b",
- "name": "Notion Block Generator",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 100,
- 1600
- ],
- "parameters": {
- "text": "={{ $json.tag.trim() }}",
- "messages": {
- "messageValues": [
- {
- "message": "=Convert the following html into its equivalent Notion Block as per Notion's API schema.\n* Ensure the content is always included and remains the same.\n* Return only a json response.\n* Generate child-level blocks. Should not define \"parent\" or \"children\" property.\n* Strongly prefer headings, paragraphs, tables and lists type blocks.\n* available headings are heading_1, heading_2 and heading_3 - h4,h5,h6 should use heading_3 type instead. ensure headings use the rich text definition.\n* ensure lists blocks include all list items.\n\n## Examples\n\n1. headings\n```\n
\n```\nwould convert to\n```\n{\n \"object\": \"block\",\n \"type\": \"table\",\n \"table\": {\n \"table_width\": 2,\n \"has_column_header\": true,\n \"has_row_header\": false,\n \"children\": [\n {\n \"object\": \"block\",\n \"type\": \"table_row\",\n \"table_row\": {\n \"cells\": [\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Technology\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Potential Impact\",\n \"link\": null\n }\n }\n ],\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"5G Connectivity\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Enables faster data speeds and advanced apps\",\n \"link\": null\n }\n }\n ]\n ]\n }\n }\n ]\n }\n}\n```\n4. anchor links\nSince Notion doesn't support anchor links, just convert them to rich text blocks instead.\n```\nModule 0: Pre-Course Setup and Learning Principles\n```\nconverts to\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Module 0: Pre-Course Setup and Learning Principles\"\n }\n }\n ]\n }\n}\n```\n5. Invalid html parts\nWhen the html is not syntax valid eg. orphaned closing tags, then just skip the conversion and use an empty rich text block.\n```\n\\n\n```\ncan be substituted with\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \" \"\n }\n }\n ]\n }\n}\n```"
- }
- ]
- },
- "promptType": "define"
- },
- "typeVersion": 1.5
- },
- {
- "id": "e3eeb9f0-7407-41f9-a814-def6c26b2ee1",
- "name": "Google Gemini Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
- "position": [
- 80,
- 1760
- ],
- "parameters": {
- "options": {},
- "modelName": "models/gemini-2.0-flash"
- },
- "credentials": {
- "googlePalmApi": {
- "id": "dSxo6ns5wn658r8N",
- "name": "Google Gemini(PaLM) Api account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "5b0aeaca-dce5-4afd-8a8a-0ef2c18b6f06",
- "name": "Parse JSON blocks",
- "type": "n8n-nodes-base.set",
- "onError": "continueRegularOutput",
- "position": [
- 420,
- 1600
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "73fcb8a0-2672-4bd5-86de-8075e1e02baf",
- "name": "=block",
- "type": "array",
- "value": "={{\n(function(){\n const block = $json.text\n .replace('```json', '')\n .replace('```', '')\n .trim()\n .parseJson();\n if (Array.isArray(block)) return block;\n if (block.type.startsWith('heading_')) {\n const prev = Number(block.type.split('_')[1]);\n const next = Math.max(1, prev - 1);\n if (next !== prev) {\n block.type = `heading_${next}`;\n block[`heading_${next}`] = Object.assign({}, block[`heading_${prev}`]);\n block[`heading_${prev}`] = undefined;\n }\n }\n return [block];\n})()\n}}"
- }
- ]
- }
- },
- "executeOnce": false,
- "typeVersion": 3.4
- },
- {
- "id": "e2a5a5bc-a3c8-42c5-9419-74ce3525f599",
- "name": "Upload to Notion Page",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "maxTries": 2,
- "position": [
- 1680,
- 1760
- ],
- "parameters": {
- "url": "=https://api.notion.com/v1/blocks/{{ $('Get Existing Row1').first().json.id }}/children",
- "method": "PATCH",
- "options": {
- "timeout": "={{ 1000 * 60 }}"
- },
- "jsonBody": "={{\n{\n \"children\": $json.block\n}\n}}",
- "sendBody": true,
- "sendHeaders": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "headerParameters": {
- "parameters": [
- {
- "name": "Notion-Version",
- "value": "2022-06-28"
- }
- ]
- },
- "nodeCredentialType": "notionApi"
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "retryOnFail": true,
- "typeVersion": 4.2,
- "waitBetweenTries": 3000
- },
- {
- "id": "9f18b2a5-ba74-40fc-8e35-a93ecd13507a",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- 620
- ],
- "parameters": {
- "color": 7,
- "width": 1740,
- "height": 740,
- "content": "## 8. Web Search and Extracting Web Page Contents using [APIFY.com](https://www.apify.com?fpr=414q6)\n[Read more about the HTTP Request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nHere is where I deviated a little from the reference implementation. I opted not to use Firecrawl.ai due to (1) high cost of the service and (2) a regular non-ai crawler would work just as well and probably quicker.\nInstead I'm using [APIFY.com](https://www.apify.com?fpr=414q6) which is a more performant, cost-effective and reliable web scraper service. If you don't want to use Apify, feel free to swap this out with your preferred service.\n\nThis step is the most exciting in terms of improvements and optimisations eg. mix in internal data sources! Add in Perplexity.ai or Jina.ai! Possibilities are endless."
- },
- "typeVersion": 1
- },
- {
- "id": "84c34a2a-d8bb-4e62-a5ea-df0a142aa2b4",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1140,
- 60
- ],
- "parameters": {
- "color": 7,
- "width": 460,
- "height": 360,
- "content": "## 5. Set Report to In-Progress\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)"
- },
- "typeVersion": 1
- },
- {
- "id": "bd022636-873d-4aca-8929-c189f8596cc1",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1240,
- 700
- ],
- "parameters": {
- "color": 7,
- "width": 780,
- "height": 660,
- "content": "## 9. Compile Learnings with Reasoning Model\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nWith our gathered sources, it's now just a case of giving it to our LLM to compile a list of \"learnings\" from them. For our DeepResearcher, we'll use OpenAI's o3-mini which is the latest reasoning model at time of writing. Reasoning perform better than regular chat models due their chain-of-thought or \"thinking\" process that they perform.\n\nThe \"Learnings\" are then combined with the generated research goal to complete one loop."
- },
- "typeVersion": 1
- },
- {
- "id": "b2f6e51d-cbe6-4459-9515-679f79063926",
- "name": "Get Existing Row1",
- "type": "n8n-nodes-base.notion",
- "position": [
- -1020,
- 1600
- ],
- "parameters": {
- "limit": 1,
- "filters": {
- "conditions": [
- {
- "key": "Request ID|rich_text",
- "condition": "equals",
- "richTextValue": "={{ $json.requestId.toString() }}"
- }
- ]
- },
- "options": {},
- "resource": "databasePage",
- "matchType": "allFilters",
- "operation": "getAll",
- "databaseId": {
- "__rl": true,
- "mode": "list",
- "value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
- "cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
- "cachedResultName": "n8n DeepResearch"
- },
- "filterType": "manual"
- },
- "credentials": {
- "notionApi": {
- "id": "iHBHe7ypzz4mZExM",
- "name": "Notion account"
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "7b315060-7e40-410c-ac9d-ef22acbb175a",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1140,
- 1400
- ],
- "parameters": {
- "color": 7,
- "width": 660,
- "height": 540,
- "content": "## 10. Generate DeepSearch Report using Learnings\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nFinally! After all learnings have been gathered - which may have taken up to an hour or more on the higher settings! - they are given to our LLM to generate the final research report in markdown format. Technically, the DeepResearch ends here but for this template, we need to push the output to Notion. If you're not using Notion, feel free to ignore the last few steps."
- },
- "typeVersion": 1
- },
- {
- "id": "01aaf2a8-c145-4f39-aa52-f40fc28f8767",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -460,
- 1400
- ],
- "parameters": {
- "color": 7,
- "width": 1060,
- "height": 540,
- "content": "## 11. Reformat Report as Notion Blocks\n[Learn more about the Markdown node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.markdown/)\n\nTo write our report to our Notion page, we'll have to convert it to Notion \"blocks\" - these are specialised json objects which are required by the Notion API. There are quite a number of ways to do this conversion not involving the use of AI but for kicks, I decided to do so anyway. In this step, we first convert to HTML as it allows us to split the report semantically and makes for easier parsing for the LLM."
- },
- "typeVersion": 1
- },
- {
- "id": "700a6f44-86bf-4aab-8a42-23bf6843f681",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1220,
- 1400
- ],
- "parameters": {
- "color": 7,
- "width": 800,
- "height": 580,
- "content": "## 13. Update Report in Notion\n[Read more about the HTTP request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nIn this step, we can use the Notion API to add the blocks to our page sequentially. A loop is used due to the unstable Notion API - the loop allows retries for blocks that require it."
- },
- "typeVersion": 1
- },
- {
- "id": "f8536052-c851-42ec-aaf3-5fc876570f6d",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1840,
- -680
- ],
- "parameters": {
- "color": 7,
- "width": 680,
- "height": 560,
- "content": "## 1. Let's Research!\n[Learn more about the form trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.formtrigger)\n\nn8n forms are a really nice way to get our frontend up and running quickly and compared to chat, offers a superior user interface for user input. I've gone perhaps a little extra with the custom html fields but I do enjoy adding a little customisation now and then."
- },
- "typeVersion": 1
- },
- {
- "id": "58aa92a2-e1fd-497d-a27e-40b733189bab",
- "name": "DeepResearch Report",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -860,
- 1600
- ],
- "parameters": {
- "text": "=You are are an expert and insightful researcher.\n* Given the following prompt from the user, write a final report on the topic using the learnings from research.\n* Make it as as detailed as possible, aim for 3 or more pages, include ALL the learnings from research.\n* Format the report in markdown. Use headings, lists and tables only and where appropriate.\n\n{{ $('JobType Router').first().json.data.query }}\n\nHere are all the learnings from previous research:\n\n\n{{\n$('JobType Router').first().json.data\n .all_learnings\n .map(item => `${item}`) \n .join('\\n')\n}}\n",
- "promptType": "define"
- },
- "typeVersion": 1.5
- },
- {
- "id": "0656be83-d510-46f1-aeeb-f62a69aa3cf2",
- "name": "DeepResearch Learnings",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 1500,
- 980
- ],
- "parameters": {
- "text": "=Given the following contents from a SERP search for the query {{ $('Item Ref').first().json.query }}, generate a list of learnings from the contents. Return a maximum of 3 learnings, but feel free to return less if the contents are clear. Make sure each learning is unique and not similar to each other. The learnings should be concise and to the point, as detailed and infromation dense as possible. Make sure to include any entities like people, places, companies, products, things, etc in the learnings, as well as any exact metrics, numbers, or dates. The learnings will be used to research the topic further.\n\n\n{{\n$('Convert to Markdown')\n .all()\n .map(item =>`\\n${item.json.markdown.substr(0, 25_000)}\\n`)\n .join('\\n')\n}}\n",
- "messages": {
- "messageValues": [
- {
- "type": "HumanMessagePromptTemplate",
- "message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "executeOnce": true,
- "typeVersion": 1.5
- },
- {
- "id": "9296a787-3226-44fe-8118-f84dda8e5167",
- "name": "Generate Report",
- "type": "n8n-nodes-base.executeWorkflow",
- "position": [
- 480,
- 180
- ],
- "parameters": {
- "options": {
- "waitForSubWorkflow": false
- },
- "workflowId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $workflow.id }}"
- },
- "workflowInputs": {
- "value": {
- "data": "={{\n{\n ...Object.assign({}, $json),\n query: $('JobType Router').first().json.data.query\n}\n}}",
- "jobType": "deepresearch_report",
- "requestId": "={{ $('JobType Router').first().json.requestId }}"
- },
- "schema": [
- {
- "id": "requestId",
- "display": true,
- "required": false,
- "displayName": "requestId",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "jobType",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "jobType",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "data",
- "type": "object",
- "display": true,
- "required": false,
- "displayName": "data",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [],
- "attemptToConvertTypes": false,
- "convertFieldsToString": true
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "471f9b9f-f331-4652-95de-1ec7136ea692",
- "name": "Generate Learnings",
- "type": "n8n-nodes-base.executeWorkflow",
- "position": [
- -380,
- 180
- ],
- "parameters": {
- "mode": "each",
- "options": {
- "waitForSubWorkflow": true
- },
- "workflowId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $workflow.id }}"
- },
- "workflowInputs": {
- "value": {
- "data": "={{ $json }}",
- "jobType": "deepresearch_learnings",
- "requestId": "={{ $('JobType Router').first().json.requestId }}"
- },
- "schema": [
- {
- "id": "requestId",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "requestId",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "jobType",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "jobType",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "data",
- "type": "object",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "data",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [],
- "attemptToConvertTypes": false,
- "convertFieldsToString": true
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "2e2fa1e6-9d5c-46ff-985c-58ada1139837",
- "name": "Confirmation",
- "type": "n8n-nodes-base.form",
- "position": [
- 780,
- -420
- ],
- "webhookId": "cf41a176-5d30-4274-955e-b0d5b483d37f",
- "parameters": {
- "options": {
- "formTitle": "DeepResearcher",
- "buttonLabel": "Done",
- "formDescription": "=\n
\nYour Report Is On Its Way!\n \nDeepResearcher will now work independently to conduct the research and the compiled report will be uploaded to the following Notion page below when finished.\n
\nPlease click the \"Done\" button to complete the form.\n
\n This value determines how many sources to explore.\n
\n \n \n
\n\n",
- "fieldType": "html",
- "fieldLabel": "Enter research breadth (recommended 2-10, default 4)"
- },
- {
- "fieldType": "dropdown",
- "fieldLabel": "={{ \"\" }}",
- "multiselect": true,
- "fieldOptions": {
- "values": [
- {
- "option": "=I understand higher depth and breath values I've selected may incur longer wait times and higher costs. I acknowledging this and wish to proceed with the research request."
- }
- ]
- },
- "requiredField": true
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "b3d11997-9c8c-4b72-b750-4fc22a2247b7",
- "name": "Valid Blocks",
- "type": "n8n-nodes-base.filter",
- "position": [
- 740,
- 1600
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "f68cefe0-e109-4d41-9aa3-043f3bc6c449",
- "operator": {
- "type": "string",
- "operation": "notExists",
- "singleValue": true
- },
- "leftValue": "={{ $json.error }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "ff90a1c1-b357-4012-8964-e007bef0c9db",
- "name": "Sticky Note12",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 620,
- 1400
- ],
- "parameters": {
- "color": 7,
- "width": 580,
- "height": 580,
- "content": "## 12. Append URL Sources List\n[Read more about the Code node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nFor our source URLs, we'll manually compose the Notion blocks for them - this is because there's usually a lot of them! We'll then append to the end of the other blocks."
- },
- "typeVersion": 1
- },
- {
- "id": "f6b50f06-6122-494c-bdb8-4215f473a27d",
- "name": "Append Blocks",
- "type": "n8n-nodes-base.merge",
- "position": [
- 1000,
- 1760
- ],
- "parameters": {},
- "typeVersion": 3
- },
- {
- "id": "ca39272a-828e-4314-80da-05dd5fd7b2e3",
- "name": "URL Sources to Lists",
- "type": "n8n-nodes-base.code",
- "position": [
- 740,
- 1760
- ],
- "parameters": {
- "jsCode": "const urls = $('JobType Router').first().json.data.all_urls;\nconst chunksize = 50;\nconst splits = Math.max(1, Math.floor(urls.length/chunksize));\n\nconst blocks = Array(splits).fill(0)\n .map((_, idx) => {\n const block = urls\n .slice(\n idx * chunksize, \n (idx * chunksize) + chunksize - 1\n )\n .map(url => {\n return {\n object: \"block\",\n type: \"bulleted_list_item\",\n bulleted_list_item: {\n rich_text: [\n { type: \"text\", text: { content: url } }\n ]\n }\n }\n });\n return { json: { block } }\n });\n\nreturn [\n { json: {\n block:[{\n \"object\": \"block\",\n \"type\": \"heading_2\",\n \"heading_2\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Sources\"\n }\n }\n ]\n }\n }]\n } },\n ...blocks\n];"
- },
- "typeVersion": 2
- },
- {
- "id": "e2e2b07a-4039-4859-b60c-f51982475282",
- "name": "Has Results?",
- "type": "n8n-nodes-base.if",
- "position": [
- 240,
- 980
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "9ef8d40c-1289-4654-9022-4a07f7102555",
- "operator": {
- "type": "array",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $json.results }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "5662dbf2-8877-4e83-982c-6bc5968b8835",
- "name": "Empty Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 1040,
- 1120
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "1de40158-338b-4db3-9e22-6fd63b21f825",
- "name": "ResearchGoal",
- "type": "string",
- "value": "={{ $('Item Ref').first().json.researchGoal }}"
- },
- {
- "id": "9f59a2d4-5e5a-4d0b-8adf-2832ce746f0f",
- "name": "learnings",
- "type": "array",
- "value": "={{ [] }}"
- },
- {
- "id": "972ab5f5-0537-4755-afcb-d1db4f09ad60",
- "name": "followUpQuestions",
- "type": "array",
- "value": "={{ [] }}"
- },
- {
- "id": "90cef471-76b0-465d-91a4-a0e256335cd3",
- "name": "urls",
- "type": "array",
- "value": "={{ [] }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "678b6af5-da74-4421-8c3d-0166aa52efd9",
- "name": "Has Content?",
- "type": "n8n-nodes-base.if",
- "position": [
- 880,
- 980
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "1ef1039a-4792-47f9-860b-d2ffcffd7129",
- "operator": {
- "type": "object",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $json }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "233f2e19-b4f2-4de3-8002-f79f3c01c1e7",
- "name": "Sticky Note13",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1820,
- -240
- ],
- "parameters": {
- "color": 5,
- "width": 300,
- "height": 100,
- "content": "### Not using forms?\nFeel free ot swap this out for chat or even webhooks to fit your existing workflows."
- },
- "typeVersion": 1
- },
- {
- "id": "453bb6eb-f2b0-4e21-b647-e095c80b7844",
- "name": "Sticky Note14",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1880,
- 540
- ],
- "parameters": {
- "color": 5,
- "width": 460,
- "height": 240,
- "content": "### ๐ The Subworkflow Event Pattern \nIf you're new to n8n, this advanced technique might need some explaining but in gist, we're using subworkflows to run different parts of our DeepResearcher workflow as separate executions.\n\n* Necessary to implement the recursive loop mechanism needed to enable this workflow.\n* Negates the need to split this workflow into multiple templates.\n* Great generally for building high performance n8n workflows (a topic for a future post!)"
- },
- "typeVersion": 1
- },
- {
- "id": "289cbe4c-c2e3-46b7-8799-197a7d78ab2a",
- "name": "Sticky Note15",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 720,
- -60
- ],
- "parameters": {
- "color": 5,
- "width": 340,
- "height": 200,
- "content": "### Recursive Looping\nThe recursive looping implemented for this workflow is an advanced item-linking technique. It works by specifically controlling which nodes \"execute once\" vs\" execute for each item\" because of this becareful of ermoving nodes! Always check the settings of the node you're replacing and ensure the settings match. "
- },
- "typeVersion": 1
- },
- {
- "id": "b95ffdcd-c0d1-4a12-a7a9-24135db7b467",
- "name": "Combine & Send back to Loop",
- "type": "n8n-nodes-base.aggregate",
- "position": [
- -240,
- 820
- ],
- "parameters": {
- "options": {},
- "aggregate": "aggregateAllItemData"
- },
- "typeVersion": 1
- },
- {
- "id": "f0a48ab5-70b9-49dc-a153-61e573803d1e",
- "name": "For Each Block...",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- 1440,
- 1600
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "2fd17fbd-005d-446e-b014-0da190cd3114",
- "name": "Sticky Note16",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -2420,
- -920
- ],
- "parameters": {
- "width": 520,
- "height": 1060,
- "content": "## n8n DeepResearcher\n### This template attempts to replicate OpenAI's DeepResearch feature which, at time of writing, is only available to their pro subscribers.\n\nThough the inner workings of DeepResearch have not been made public, it is presumed the feature relies on the ability to deep search the web, scrape web content and invoking reasoning models to generate reports. All of which n8n is really good at!\n\n### How it works\n* A form is used to first capture the user's research query and how deep they'd like the researcher to go.\n* Once submitted, a blank Notion page is created which will later hold the final report and the researcher gets to work.\n* The user's query goes through a recursive series of web serches and web scraping to collect data on the research topic to generate partial learnings.\n* Once complete, all learnings are combined and given to a reasoning LLM to generate the final report.\n* The report is then written to the placeholder Notion page created earlier. \n\n### How to use\n* Duplicate this Notion database to use with this template: https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf\n* Sign-up for [APIFY.com](https://www.apify.com?fpr=414q6) API Key for web search and scraping services.\n* Ensure you have access to OpenAI's o3-mini model. Alternatively, switch this out for o1 series.\n* You must publish this workflow and ensure the form url is publically accessible.\n\n### On Depth & Breadth Configuration\nFor more detailed reports, increase depth and breadth but be warned the workflow will take a exponentially more time and money to complete. The defaults are usually good enough.\n\nDepth=1 & Breadth=2 - will take about 10 - 15mins.\nDepth=1 & Breadth=4 - will take about 30 - 40mins.\nDpeth=3 & Breadth=5 - will take about 2 - 5 hours!\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
- },
- "typeVersion": 1
- },
- {
- "id": "ac6f2604-7439-4524-a27e-2f031ebce089",
- "name": "Sticky Note17",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -2420,
- -1180
- ],
- "parameters": {
- "color": 7,
- "width": 520,
- "height": 240,
- "content": ""
- },
- "typeVersion": 1
- },
- {
- "id": "40e3a0cf-7710-4537-b147-37ba8945fdbc",
- "name": "Sticky Note18",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -120,
- 960
- ],
- "parameters": {
- "width": 180,
- "height": 260,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
- },
- "typeVersion": 1
- },
- {
- "id": "5a96ecc2-eeea-4c33-b299-7a7f2ca7559c",
- "name": "Sticky Note19",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 520,
- 960
- ],
- "parameters": {
- "width": 180,
- "height": 260,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
- },
- "typeVersion": 1
- },
- {
- "id": "23bca6e2-e16a-48a4-a7fc-96ce25846764",
- "name": "Sticky Note20",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1640,
- 1740
- ],
- "parameters": {
- "width": 180,
- "height": 260,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE NOTION CREDENTIAL HERE!"
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Item Ref": {
- "main": [
- [
- {
- "node": "Web Search",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create Row": {
- "main": [
- [
- {
- "node": "Initiate DeepResearch",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Web Search": {
- "main": [
- [
- {
- "node": "Top 5 Organic Results",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Valid Pages": {
- "main": [
- [
- {
- "node": "Has Content?",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Confirmation": {
- "main": [
- [
- {
- "node": "End Form",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Has Content?": {
- "main": [
- [
- {
- "node": "Convert to Markdown",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Empty Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Has Results?": {
- "main": [
- [
- {
- "node": "URLs to Items",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Empty Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Valid Blocks": {
- "main": [
- [
- {
- "node": "Append Blocks",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Append Blocks": {
- "main": [
- [
- {
- "node": "For Each Block...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "HTML to Array": {
- "main": [
- [
- {
- "node": "Tags to Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Page Contents": {
- "main": [
- [
- {
- "node": "Valid Pages",
- "type": "main",
- "index": 0
- }
- ],
- []
- ]
- },
- "SERP to Items": {
- "main": [
- [
- {
- "node": "For Each Query...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Variables": {
- "main": [
- [
- {
- "node": "Clarifying Questions",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Tags to Items": {
- "main": [
- [
- {
- "node": "Notion Block Generator",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "URLs to Items": {
- "main": [
- [
- {
- "node": "Page Contents",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Empty Response": {
- "main": [
- [
- {
- "node": "For Each Query...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execution Data": {
- "main": [
- [
- {
- "node": "JobType Router",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "JobType Router": {
- "main": [
- [
- {
- "node": "Get Existing Row",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Generate SERP Queries",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Get Existing Row1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Convert to HTML": {
- "main": [
- [
- {
- "node": "HTML to Array",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set In-Progress": {
- "main": [
- [
- {
- "node": "Set Initial Query",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Existing Row": {
- "main": [
- [
- {
- "node": "Set In-Progress",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Research Request": {
- "main": [
- [
- {
- "node": "Set Variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Results to Items": {
- "main": [
- [
- {
- "node": "Set Next Queries",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Next Queries": {
- "main": [
- [
- {
- "node": "Generate Learnings",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Feedback to Items": {
- "main": [
- [
- {
- "node": "For Each Question...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "For Each Block...": {
- "main": [
- [
- {
- "node": "Set Done",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Upload to Notion Page",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "For Each Query...": {
- "main": [
- [
- {
- "node": "Combine & Send back to Loop",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Item Ref",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Existing Row1": {
- "main": [
- [
- {
- "node": "DeepResearch Report",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Initial Query": {
- "main": [
- [
- {
- "node": "Report Page Generator",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Is Depth Reached?": {
- "main": [
- [
- {
- "node": "Get Research Results",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "DeepResearch Results",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "DeepResearch Learnings",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Parse JSON blocks": {
- "main": [
- [
- {
- "node": "Valid Blocks",
- "type": "main",
- "index": 0
- },
- {
- "node": "URL Sources to Lists",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Initial Query": {
- "main": [
- [
- {
- "node": "Generate Learnings",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Accumulate Results": {
- "main": [
- [
- {
- "node": "Is Depth Reached?",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Generate Learnings": {
- "main": [
- [
- {
- "node": "Accumulate Results",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "On form submission": {
- "main": [
- [
- {
- "node": "Research Request",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model1": {
- "ai_languageModel": [
- [
- {
- "node": "DeepResearch Report",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model2": {
- "ai_languageModel": [
- [
- {
- "node": "Clarifying Questions",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model3": {
- "ai_languageModel": [
- [
- {
- "node": "Generate SERP Queries",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model4": {
- "ai_languageModel": [
- [
- {
- "node": "Report Page Generator",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Convert to Markdown": {
- "main": [
- [
- {
- "node": "DeepResearch Learnings",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "DeepResearch Report": {
- "main": [
- [
- {
- "node": "Convert to HTML",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Clarifying Questions": {
- "main": [
- [
- {
- "node": "Feedback to Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "DeepResearch Results": {
- "main": [
- [
- {
- "node": "Results to Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "For Each Question...": {
- "main": [
- [
- {
- "node": "Get Initial Query",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Ask Clarity Questions",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Research Results": {
- "main": [
- [
- {
- "node": "Generate Report",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "URL Sources to Lists": {
- "main": [
- [
- {
- "node": "Append Blocks",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Ask Clarity Questions": {
- "main": [
- [
- {
- "node": "For Each Question...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Generate SERP Queries": {
- "main": [
- [
- {
- "node": "SERP to Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Initiate DeepResearch": {
- "main": [
- [
- {
- "node": "Confirmation",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Report Page Generator": {
- "main": [
- [
- {
- "node": "Create Row",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Top 5 Organic Results": {
- "main": [
- [
- {
- "node": "Has Results?",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Upload to Notion Page": {
- "main": [
- [
- {
- "node": "For Each Block...",
- "type": "main",
- "index": 0
- }
- ],
- []
- ]
- },
- "DeepResearch Learnings": {
- "main": [
- [
- {
- "node": "Research Goal + Learnings",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Notion Block Generator": {
- "main": [
- [
- {
- "node": "Parse JSON blocks",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "DeepResearch Subworkflow": {
- "main": [
- [
- {
- "node": "Execution Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google Gemini Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Notion Block Generator",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "DeepResearch Learnings",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Research Goal + Learnings": {
- "main": [
- [
- {
- "node": "For Each Query...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser1": {
- "ai_outputParser": [
- [
- {
- "node": "Clarifying Questions",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser2": {
- "ai_outputParser": [
- [
- {
- "node": "Generate SERP Queries",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser4": {
- "ai_outputParser": [
- [
- {
- "node": "Report Page Generator",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.txt b/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.txt
new file mode 100644
index 0000000..2ba39c2
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Host Your Own AI Deep Research Agent with n8n, Apify and OpenAI o3.txt
@@ -0,0 +1,5883 @@
+{
+"meta": {
+"instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9",
+"templateCredsSetupCompleted": true
+},
+"nodes": [
+{
+"id": "645ae2b1-799e-49be-8bdf-12cd1bb739e6",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1680,
+1140
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"learnings\": {\n \"type\": \"array\",\n \"description\": \"List of learnings, max of 3.\",\n \"items\": { \"type\": \"string\" }\n },\n \"followUpQuestions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\",\n \"description\": \"List of follow-up questions to research the topic further, max of 3.\"\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "cbdb4e98-eeba-4609-91de-394c416b7904",
+"name": "Set Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-1360,
+-460
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "df28b12e-7c20-4ff5-b5b8-dc773aa14d4b",
+"name": "request_id",
+"type": "string",
+"value": "={{ $execution.id }}"
+},
+{
+"id": "9362c1e7-717d-444a-8ea2-6b5f958c9f3f",
+"name": "prompt",
+"type": "string",
+"value": "={{ $json['What would you like to research?'] }}"
+},
+{
+"id": "09094be4-7844-4a9e-af82-cc8e39322398",
+"name": "depth",
+"type": "number",
+"value": "={{ $json['Enter research depth (recommended 1-5, default 2)'] || 2 }}"
+},
+{
+"id": "3fc30a30-7806-4013-835d-97e27ddd7ae1",
+"name": "breadth",
+"type": "number",
+"value": "={{ $json['Enter research breadth (recommended 2-10, default 4)'] || 4 }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "c7096ab9-0b10-45b0-b178-a049bf57830b",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+1500,
+1140
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "d0f1bc2f-6a10-4ac7-8d35-34f48f14fad5",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-860,
+1760
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "bba3278c-0336-4305-887d-56515dfd87db",
+"name": "OpenAI Chat Model2",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-1060,
+-300
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "f31f2fc7-0bec-4105-9d83-5f4f9a0eb35d",
+"name": "Structured Output Parser1",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+-840,
+-300
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"Follow up questions to clarify the research direction, max of 3.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "ea59c5ab-fa05-4c68-bc60-3f56e240478b",
+"name": "On form submission",
+"type": "n8n-nodes-base.formTrigger",
+"position": [
+-1760,
+-460
+],
+"webhookId": "7ddfaa7c-a523-4d92-b033-d76cd5a313e9",
+"parameters": {
+"options": {
+"path": "deep_research",
+"ignoreBots": true,
+"buttonLabel": "Next"
+},
+"formTitle": " DeepResearcher",
+"formFields": {
+"values": [
+{
+"fieldType": "html",
+"fieldLabel": "placeholder"
+}
+]
+},
+"formDescription": "=DeepResearcher is a multi-step, recursive approach using the internet to solve complex research tasks, accomplishing in tens of minutes what a human would take many hours.\n\nTo use, provide a short summary of what the research and how \"deep\" you'd like the workflow to investigate. Note, the higher the numbers the more time and cost will occur for the research.\n\nThe workflow is designed to complete independently and when finished, a report will be saved in a designated Notion Database."
+},
+"typeVersion": 2.2
+},
+{
+"id": "a8262288-a8c1-4967-9870-f728fa08b579",
+"name": "Generate SERP Queries",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-1040,
+820
+],
+"parameters": {
+"text": "=Given the following prompt from the user, generate a list of SERP queries to research the topic. Return a maximum of {{ $('JobType Router').first().json.data.breadth }} queries, but feel free to return less if the original prompt is clear. Make sure each query is unique and not similar to each other: {{ $('JobType Router').first().json.data.query.trim() }}\n\n{{\n$('JobType Router').first().json.data.learnings.length\n ? `Here are some learnings from previous research, use them to generate more specific queries: ${$('JobType Router').first().json.data.learnings.join('\\n')}`\n : ''\n}}",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "0534be47-22b7-4c2a-956b-d085e6b9f280",
+"name": "Structured Output Parser2",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+-860,
+980
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"queries\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": {\n \"type\": \"string\",\n \"description\": \"The SERP query\"\n },\n \"researchGoal\": {\n \"type\": \"string\",\n \"description\": \"First talk about the goal of the research that this query is meant to accomplish, then go deeper into how to advance the research once the results are found, mention additional research directions. Be as specific as possible, especially for additional research directions.\"\n }\n }\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "4d8aa196-986f-442d-9b56-92c043ab785d",
+"name": "OpenAI Chat Model3",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-1040,
+980
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "7488b037-7422-4f62-8c37-1f6a901b3299",
+"name": "Set Initial Query",
+"type": "n8n-nodes-base.set",
+"position": [
+-580,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "acb41e93-70c6-41a3-be0f-e5a74ec3ec88",
+"name": "query",
+"type": "string",
+"value": "={{ $('JobType Router').first().json.data.query }}"
+},
+{
+"id": "7fc54063-b610-42bc-a250-b1e8847c4d1e",
+"name": "learnings",
+"type": "array",
+"value": "={{ $('JobType Router').first().json.data.learnings }}"
+},
+{
+"id": "e8f1c158-56fb-41c8-8d86-96add16289bb",
+"name": "breadth",
+"type": "number",
+"value": "={{ $('JobType Router').first().json.data.breadth }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "12ae382e-d88a-4f1b-a71f-3bd63c892b17",
+"name": "SERP to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-700,
+820
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output.queries"
+},
+"typeVersion": 1
+},
+{
+"id": "46700052-f48a-493c-aebf-cdf175d58550",
+"name": "Item Ref",
+"type": "n8n-nodes-base.noOp",
+"position": [
+-240,
+980
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "2cef6f1d-e244-4ee6-bf25-6dc3e8042afa",
+"name": "Research Goal + Learnings",
+"type": "n8n-nodes-base.set",
+"position": [
+1840,
+1120
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "9acec2cc-64c8-4e62-bed4-c3d9ffab1379",
+"name": "researchGoal",
+"type": "string",
+"value": "={{ $('Item Ref').first().json.researchGoal }}"
+},
+{
+"id": "1b2d2dad-429b-4fc9-96c5-498f572a85c3",
+"name": "learnings",
+"type": "array",
+"value": "={{ $json.output.learnings }}"
+},
+{
+"id": "655b99f2-6045-4774-a634-49751bc9326f",
+"name": "followUpQuestions",
+"type": "array",
+"value": "={{ $json.output.followUpQuestions }}"
+},
+{
+"id": "c9e34ea4-5606-46d6-8d66-cb42d772a8b4",
+"name": "urls",
+"type": "array",
+"value": "={{\n$('Page Contents')\n .all()\n .filter(item => !item.json.error && item.json.body)\n .map(item => item.json.url)\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "4aebae86-2bd2-4f3d-8290-d34b9ac837c6",
+"name": "Accumulate Results",
+"type": "n8n-nodes-base.set",
+"position": [
+-200,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "db509e90-9a86-431f-8149-4094d22666cc",
+"name": "should_stop",
+"type": "boolean",
+"value": "={{\n$runIndex >= ($('JobType Router').first().json.data.depth)\n}}"
+},
+{
+"id": "90986e2b-8aca-4a22-a9db-ed8809d6284d",
+"name": "all_learnings",
+"type": "array",
+"value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.learnings))\n } catch (e) {\n return []\n }\n })\n}}"
+},
+{
+"id": "3eade958-e8ab-4975-aac4-f4a4a983c163",
+"name": "all_urls",
+"type": "array",
+"value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.urls))\n } catch (e) {\n return []\n }\n })\n}}"
+}
+]
+}
+},
+"executeOnce": true,
+"typeVersion": 3.4
+},
+{
+"id": "782baa36-ba07-4845-873c-c9400de6d463",
+"name": "DeepResearch Results",
+"type": "n8n-nodes-base.set",
+"position": [
+160,
+360
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "={{ $('Generate Learnings').item.json }}"
+},
+"typeVersion": 3.4
+},
+{
+"id": "89b09898-79ec-4924-975f-e9581d3bf774",
+"name": "Results to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+320,
+360
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 1
+},
+{
+"id": "122cd071-aade-4753-ba0a-8db4c58fa84e",
+"name": "Set Next Queries",
+"type": "n8n-nodes-base.set",
+"position": [
+480,
+360
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "d88bfe95-9e73-4d25-b45c-9f164b940b0e",
+"name": "query",
+"type": "string",
+"value": "=Previous research goal: {{ $json.researchGoal }}\nFollow-up research directions: {{ $json.followUpQuestions.map(q => `\\n${q}`).join('') }}"
+},
+{
+"id": "4aa20690-d998-458a-b1e4-0d72e6a68e6b",
+"name": "learnings",
+"type": "array",
+"value": "={{ $('Accumulate Results').item.json.all_learnings }}"
+},
+{
+"id": "89acafae-b04a-4d5d-b08b-656e715654e4",
+"name": "breadth",
+"type": "number",
+"value": "={{ $('JobType Router').first().json.data.breadth }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "9da01d8a-48d6-45b4-b8c6-9a0503b4bda6",
+"name": "Web Search",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+-80,
+980
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/serping~fast-google-search-results-scraper/run-sync-get-dataset-items",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"authentication": "genericCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "searchTerms",
+"value": "={{\n[\n `${$json.query} -filetype:pdf`\n]\n}}"
+},
+{
+"name": "resultsPerPage",
+"value": "={{ 10 }}"
+}
+]
+},
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+},
+"httpHeaderAuth": {
+"id": "SV9BDKc1cRbZBeoL",
+"name": "Apify.com (personal token)"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "99bd2c8e-5600-43a9-ab2f-7f2911efb16c",
+"name": "Top 5 Organic Results",
+"type": "n8n-nodes-base.set",
+"position": [
+80,
+980
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "29d1a759-d886-4a44-860b-9d16f9922043",
+"name": "results",
+"type": "array",
+"value": "={{\n$json.origin_search.results\n ? $json.origin_search\n .results\n .filter(res => res.type === 'normal')\n .slice(0, 5)\n .map(res => ({\n title: res.title,\n url: res.source.link\n }))\n : []\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "cb7c5a8b-5420-4fb9-b7f0-4e8e8d10034a",
+"name": "Convert to Markdown",
+"type": "n8n-nodes-base.markdown",
+"position": [
+1320,
+980
+],
+"parameters": {
+"html": "={{ $json.body }}",
+"options": {
+"ignore": "a,img,picture,svg,video,audio,iframe"
+},
+"destinationKey": "markdown"
+},
+"typeVersion": 1
+},
+{
+"id": "818ccf2e-081d-492e-ba8d-de458b0c26db",
+"name": "For Each Query...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-420,
+820
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "1787b562-17e8-41af-9cdc-eb2d3e630916",
+"name": "Feedback to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-720,
+-460
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output.questions"
+},
+"typeVersion": 1
+},
+{
+"id": "4c695faa-74e3-456b-a1ef-aaea67e46743",
+"name": "Ask Clarity Questions",
+"type": "n8n-nodes-base.form",
+"position": [
+-360,
+-380
+],
+"webhookId": "ab0d9b81-73f6-4baa-a3cd-ac3b31397708",
+"parameters": {
+"options": {
+"formTitle": "DeepResearcher",
+"buttonLabel": "Answer",
+"formDescription": "=\n
\nAnswer the following clarification questions to assist the DeepResearcher better under the research topic.\n
\n\n
\nTotal {{ $('Feedback to Items').all().length }} questions.\n
"
+},
+"formFields": {
+"values": [
+{
+"fieldType": "textarea",
+"fieldLabel": "={{ $json[\"output.questions\"] }}",
+"placeholder": "=",
+"requiredField": true
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "e07d8c3e-8bcd-4393-9892-f825433ab58d",
+"name": "For Each Question...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-540,
+-460
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "e8d26351-52f4-40a6-ba5b-fb6bc816b734",
+"name": "DeepResearch Subworkflow",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-1880,
+820
+],
+"parameters": {
+"workflowInputs": {
+"values": [
+{
+"name": "requestId",
+"type": "any"
+},
+{
+"name": "jobType"
+},
+{
+"name": "data",
+"type": "object"
+}
+]
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "25a8055a-27aa-414f-856b-25a2e2f31974",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 1000,
+"height": 560,
+"content": "## 2. Ask Clarifying Questions\n[Read more about form nodes](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.form/)\n\nTo handle the clarification questions generated by the LLM, I used the same technique found in my \"AI Interviewer\" template ([link](https://n8n.io/workflows/2566-conversational-interviews-with-ai-agents-and-n8n-forms/)).\nThis involves a looping of dynamically generated forms to collect answers from the user."
+},
+"typeVersion": 1
+},
+{
+"id": "68398b92-eb35-48bf-885e-540074531cc4",
+"name": "Clarifying Questions",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-1040,
+-460
+],
+"parameters": {
+"text": "=Given the following query from the user, ask some follow up questions to clarify the research direction. Return a maximum of 3 questions, but feel free to return less if the original query is clear: {{ $json.prompt }}`",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "65c4c293-67b8-4e64-af04-16e45e97c09a",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-660,
+-60
+],
+"parameters": {
+"color": 7,
+"width": 1360,
+"height": 640,
+"content": "## 6. Perform DeepSearch Loop\n[Learn more about the Looping in n8n](https://docs.n8n.io/flow-logic/looping/#creating-loops)\n\nThe key of the Deep Research flow is its extensive data collection capability. In this implementation, this capability is represented by a recursive web search & scrape loop which starts with the original query and extended by AI-generated subqueries. How many subqueries to generate are determined the depth and breadth parameters specified.\n\n\"Learnings\" are generated for each subquery and accumulate on each iteration of the loop. When the loop finishes when depth limit is reached, all learnings are collected and it's these learnings are what we use to generate the report."
+},
+"typeVersion": 1
+},
+{
+"id": "43a5d93d-cae2-43ec-b9ae-b15d6b11b932",
+"name": "End Form",
+"type": "n8n-nodes-base.form",
+"position": [
+960,
+-420
+],
+"webhookId": "7b531f5d-942f-4c49-ac55-8ee480889600",
+"parameters": {
+"options": {},
+"operation": "completion",
+"completionTitle": "=Thank you for using DeepResearcher.",
+"completionMessage": "=You may now close this window."
+},
+"typeVersion": 1
+},
+{
+"id": "9a824011-e76f-433f-8735-44b358f4ff7d",
+"name": "Initiate DeepResearch",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+600,
+-420
+],
+"parameters": {
+"mode": "each",
+"options": {
+"waitForSubWorkflow": false
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{\n{\n \"query\": $('Get Initial Query').first().json.query,\n \"learnings\": [],\n \"depth\": $('Set Variables').first().json.depth,\n \"breadth\": $('Set Variables').first().json.breadth,\n}\n}}",
+"jobType": "deepresearch_initiate",
+"requestId": "={{ $('Set Variables').first().json.request_id }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "c48ee4cd-bac1-4405-bb4c-5614e5eb25a0",
+"name": "Page Contents",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+560,
+980
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/apify~web-scraper/run-sync-get-dataset-items",
+"options": {},
+"jsonBody": "={\n \"startUrls\": {{ [{ url: $json.url, method: 'GET' }].toJsonString() }},\n \"breakpointLocation\": \"NONE\",\n \"browserLog\": false,\n \"closeCookieModals\": false,\n \"debugLog\": false,\n \"downloadCss\": false,\n \"downloadMedia\": false,\n \"excludes\": [\n {\n \"glob\": \"/**/*.{png,jpg,jpeg,pdf}\"\n }\n ],\n \"headless\": true,\n \"ignoreCorsAndCsp\": false,\n \"ignoreSslErrors\": false,\n \"injectJQuery\": true,\n \"keepUrlFragments\": false,\n \"linkSelector\": \"\",\n \"maxCrawlingDepth\": 1,\n \"maxPagesPerCrawl\": 1,\n \"maxRequestRetries\": 1,\n \"maxResultsPerCrawl\": 1,\n \"pageFunction\": \"// The function accepts a single argument: the \\\"context\\\" object.\\n// For a complete list of its properties and functions,\\n// see https://apify.com/apify/web-scraper#page-function \\nasync function pageFunction(context) {\\n\\n await new Promise(res => { setTimeout(res, 6000) });\\n // This statement works as a breakpoint when you're trying to debug your code. Works only with Run mode: DEVELOPMENT!\\n // debugger; \\n\\n // jQuery is handy for finding DOM elements and extracting data from them.\\n // To use it, make sure to enable the \\\"Inject jQuery\\\" option.\\n const $ = context.jQuery;\\n const title = $('title').first().text();\\n\\n // Clone the body to avoid modifying the original content\\n const bodyClone = $('body').clone();\\n bodyClone.find('iframe, img, script, style, object, embed, noscript, svg, video, audio').remove();\\n const body = bodyClone.html();\\n\\n // Return an object with the data extracted from the page.\\n // It will be stored to the resulting dataset.\\n return {\\n url: context.request.url,\\n title,\\n body\\n };\\n}\",\n \"postNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept a single argument: the \\\"crawlingContext\\\" object.\\n[\\n async (crawlingContext) => {\\n // ...\\n },\\n]\",\n \"preNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept two arguments: the \\\"crawlingContext\\\" object\\n// and \\\"gotoOptions\\\".\\n[\\n async (crawlingContext, gotoOptions) => {\\n // ...\\n },\\n]\\n\",\n \"proxyConfiguration\": {\n \"useApifyProxy\": true\n },\n \"runMode\": \"PRODUCTION\",\n \"useChrome\": false,\n \"waitUntil\": [\n \"domcontentloaded\"\n ],\n \"globs\": [],\n \"pseudoUrls\": [],\n \"proxyRotation\": \"RECOMMENDED\",\n \"maxConcurrency\": 50,\n \"pageLoadTimeoutSecs\": 60,\n \"pageFunctionTimeoutSecs\": 60,\n \"maxScrollHeightPixels\": 5000,\n \"customData\": {}\n}",
+"sendBody": true,
+"sendQuery": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "memory",
+"value": "2048"
+},
+{
+"name": "timeout",
+"value": "90"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "dc9f85ff-7565-4c29-981a-5ef65bba6ca3",
+"name": "Execution Data",
+"type": "n8n-nodes-base.executionData",
+"position": [
+-1700,
+820
+],
+"parameters": {
+"dataToSave": {
+"values": [
+{
+"key": "requestId",
+"value": "={{ $json.requestId }}"
+},
+{
+"key": "=jobType",
+"value": "={{ $json.jobType }}"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "26b33429-6d61-4758-9c76-3e998dd31fa4",
+"name": "JobType Router",
+"type": "n8n-nodes-base.switch",
+"position": [
+-1520,
+820
+],
+"parameters": {
+"rules": {
+"values": [
+{
+"outputKey": "initiate",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"operator": {
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_initiate"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "learnings",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "ecbfa54d-fc97-48c5-8d3d-f0538b8d727b",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_learnings"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "report",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "392f9a98-ec22-4e57-9c8e-0e1ed6b7dafa",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_report"
+}
+]
+},
+"renameOutput": true
+}
+]
+},
+"options": {}
+},
+"typeVersion": 3.2
+},
+{
+"id": "a9637952-7c09-40ae-96ec-bdf0fc63d94e",
+"name": "Valid Pages",
+"type": "n8n-nodes-base.filter",
+"position": [
+720,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f44691e4-f753-47b0-b66a-068a723b6beb",
+"operator": {
+"type": "boolean",
+"operation": "false",
+"singleValue": true
+},
+"leftValue": "={{ $json['#error'] }}",
+"rightValue": ""
+},
+{
+"id": "8e05df2b-0d4a-47da-9aab-da7e8907cbca",
+"operator": {
+"type": "string",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json.body }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2,
+"alwaysOutputData": true
+},
+{
+"id": "204cfca2-05bb-46dd-ba96-b41866ed2cfe",
+"name": "OpenAI Chat Model4",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-20,
+-280
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "45bc6261-35c8-4994-bb88-ed7a0f022767",
+"name": "Get Initial Query",
+"type": "n8n-nodes-base.set",
+"position": [
+-360,
+-540
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "14b77741-c3c3-4bd2-be6e-37bd09fcea2b",
+"name": "query",
+"type": "string",
+"value": "=Initial query: {{ $('Set Variables').first().json.prompt }}\nFollow-up Questions and Answers:\n{{\n$input.all()\n .map(item => {\n const q = Object.keys(item.json)[0];\n const a = item.json[q];\n return `question: ${q}\\nanswer: ${a}`;\n })\n .join('\\n')\n}}"
+}
+]
+}
+},
+"executeOnce": true,
+"typeVersion": 3.4
+},
+{
+"id": "26d26e54-ee9b-4714-ae27-4f033dc825d3",
+"name": "Structured Output Parser4",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+160,
+-280
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\":\" A short title summarising the research topic\"\n },\n \"description\": {\n \"type\": \"string\",\n \"description\": \"A short description to summarise the research topic\"\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "3842bc1d-d5f9-4879-bc06-db20fed3f55d",
+"name": "Create Row",
+"type": "n8n-nodes-base.notion",
+"position": [
+300,
+-420
+],
+"parameters": {
+"title": "={{ $json.output.title }}",
+"options": {},
+"resource": "databasePage",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Description|rich_text",
+"textContent": "={{ $json.output.description }}"
+},
+{
+"key": "Status|status",
+"statusValue": "Not started"
+},
+{
+"key": "Request ID|rich_text",
+"textContent": "={{ $('Set Variables').first().json.request_id }}"
+},
+{
+"key": "Name|title",
+"title": "={{ $json.output.title }}"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "bfe98996-6ed5-4f60-afdd-a947a6fa6e36",
+"name": "Report Page Generator",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-20,
+-420
+],
+"parameters": {
+"text": "=Create a suitable title for the research report which will be created from the user's query.\n{{ $json.query }}",
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "ff05add8-94b0-4495-8f4e-3e8a10c556af",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-120,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 600,
+"height": 560,
+"content": "## 3. Create Empty Report Page in Notion\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)\n\nSome thought was given where to upload the final report and Notion was selected due to familiarity. This can be easily changed to whatever wiki tools you prefer.\n\nIf you're following along however, here's the Notion database you need to replicate - [Jim's n8n DeepResearcher Database](https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf)."
+},
+"typeVersion": 1
+},
+{
+"id": "5bc13d62-81e1-4730-b7e6-9e5579dff174",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+500,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 640,
+"height": 560,
+"content": "## 4. Trigger DeepResearch Asynchronously\n[Learn more about the Execute Trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/)\n\nn8n handles asynchronous jobs by spinning them off as separate executions. This basically means the user doesn't have to wait or keep their browser window open for our researcher to do its job.\n\nOnce we initiate the Deepresearcher job, we can close out the onboarding journey for a nice user experience."
+},
+"typeVersion": 1
+},
+{
+"id": "9fea6403-b2a2-4e67-99a2-b7a2f29a1e96",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1160,
+620
+],
+"parameters": {
+"color": 7,
+"width": 620,
+"height": 540,
+"content": "## 7. Generate Search Queries\n[Learn more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nMuch like a human researcher, the DeepResearcher will rely on web search and content as the preferred source of information. To ensure it can cover a wide range of sources, the AI can first generate relevant research queries of which each can be explored separately."
+},
+"typeVersion": 1
+},
+{
+"id": "0bccdc54-7570-4bca-93ec-cb140c5bd3a1",
+"name": "Is Depth Reached?",
+"type": "n8n-nodes-base.if",
+"position": [
+-40,
+180
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "75d18d88-6ba6-43df-bef7-3e8ad99ad8bd",
+"operator": {
+"type": "boolean",
+"operation": "true",
+"singleValue": true
+},
+"leftValue": "={{ $json.should_stop }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "819aa5be-b71b-44a7-b062-b2a50209f290",
+"name": "URLs to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+400,
+980
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "results"
+},
+"typeVersion": 1
+},
+{
+"id": "4b8e9936-4b24-4bd4-8fe7-75d58244cb6d",
+"name": "Get Research Results",
+"type": "n8n-nodes-base.set",
+"position": [
+160,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "90b3da00-dcd5-4289-bd45-953146a3b0ba",
+"name": "all_learnings",
+"type": "array",
+"value": "={{ $json.all_learnings }}"
+},
+{
+"id": "623dbb3d-83a1-44a9-8ad3-48d92bc42811",
+"name": "all_urls",
+"type": "array",
+"value": "={{ $json.all_urls }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "d371535a-2946-4ec5-9be6-2ee8e359ac44",
+"name": "Get Existing Row",
+"type": "n8n-nodes-base.notion",
+"position": [
+-1040,
+180
+],
+"parameters": {
+"limit": 1,
+"filters": {
+"conditions": [
+{
+"key": "Request ID|rich_text",
+"condition": "equals",
+"richTextValue": "={{ $json.requestId.toString() }}"
+}
+]
+},
+"options": {},
+"resource": "databasePage",
+"matchType": "allFilters",
+"operation": "getAll",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"filterType": "manual"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "fea4c30e-1193-494d-8823-dfbec5196a0d",
+"name": "Set In-Progress",
+"type": "n8n-nodes-base.notion",
+"position": [
+-840,
+180
+],
+"parameters": {
+"pageId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $json.id }}"
+},
+"options": {},
+"resource": "databasePage",
+"operation": "update",
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Status|status",
+"statusValue": "In progress"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "37954acd-d8cb-4c74-afa8-d8973e017327",
+"name": "Set Done",
+"type": "n8n-nodes-base.notion",
+"position": [
+1680,
+1600
+],
+"parameters": {
+"pageId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Get Existing Row1').first().json.id }}"
+},
+"options": {},
+"resource": "databasePage",
+"operation": "update",
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Status|status",
+"statusValue": "Done"
+},
+{
+"key": "Last Updated|date",
+"date": "={{ $now.toISO() }}"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"executeOnce": true,
+"typeVersion": 2.2
+},
+{
+"id": "3db97ab8-b934-4567-a92e-92374a363df6",
+"name": "Tags to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-60,
+1600
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "tag"
+},
+"typeVersion": 1
+},
+{
+"id": "7f468bf7-762a-4818-86f9-54d172bb618a",
+"name": "Convert to HTML",
+"type": "n8n-nodes-base.markdown",
+"position": [
+-380,
+1600
+],
+"parameters": {
+"mode": "markdownToHtml",
+"options": {
+"tables": true
+},
+"markdown": "={{ $json.text }}"
+},
+"typeVersion": 1
+},
+{
+"id": "97914ee9-0ee8-408b-b2bb-a7193b2d0454",
+"name": "HTML to Array",
+"type": "n8n-nodes-base.set",
+"position": [
+-220,
+1600
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "851b8a3f-c2d3-41ad-bf60-4e0e667f6c58",
+"name": "tag",
+"type": "array",
+"value": "={{ $json.data.match(/
|
|<[^>]+>[^<]*<\\/[^>]+>/g) }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "6ce79f16-51e3-4192-8103-738222be558b",
+"name": "Notion Block Generator",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+100,
+1600
+],
+"parameters": {
+"text": "={{ $json.tag.trim() }}",
+"messages": {
+"messageValues": [
+{
+"message": "=Convert the following html into its equivalent Notion Block as per Notion's API schema.\n* Ensure the content is always included and remains the same.\n* Return only a json response.\n* Generate child-level blocks. Should not define \"parent\" or \"children\" property.\n* Strongly prefer headings, paragraphs, tables and lists type blocks.\n* available headings are heading_1, heading_2 and heading_3 - h4,h5,h6 should use heading_3 type instead. ensure headings use the rich text definition.\n* ensure lists blocks include all list items.\n\n## Examples\n\n1. headings\n```\n
\n```\nwould convert to\n```\n{\n \"object\": \"block\",\n \"type\": \"table\",\n \"table\": {\n \"table_width\": 2,\n \"has_column_header\": true,\n \"has_row_header\": false,\n \"children\": [\n {\n \"object\": \"block\",\n \"type\": \"table_row\",\n \"table_row\": {\n \"cells\": [\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Technology\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Potential Impact\",\n \"link\": null\n }\n }\n ],\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"5G Connectivity\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Enables faster data speeds and advanced apps\",\n \"link\": null\n }\n }\n ]\n ]\n }\n }\n ]\n }\n}\n```\n4. anchor links\nSince Notion doesn't support anchor links, just convert them to rich text blocks instead.\n```\nModule 0: Pre-Course Setup and Learning Principles\n```\nconverts to\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Module 0: Pre-Course Setup and Learning Principles\"\n }\n }\n ]\n }\n}\n```\n5. Invalid html parts\nWhen the html is not syntax valid eg. orphaned closing tags, then just skip the conversion and use an empty rich text block.\n```\n\\n\n```\ncan be substituted with\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \" \"\n }\n }\n ]\n }\n}\n```"
+}
+]
+},
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "e3eeb9f0-7407-41f9-a814-def6c26b2ee1",
+"name": "Google Gemini Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+80,
+1760
+],
+"parameters": {
+"options": {},
+"modelName": "models/gemini-2.0-flash"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "dSxo6ns5wn658r8N",
+"name": "Google Gemini(PaLM) Api account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "5b0aeaca-dce5-4afd-8a8a-0ef2c18b6f06",
+"name": "Parse JSON blocks",
+"type": "n8n-nodes-base.set",
+"onError": "continueRegularOutput",
+"position": [
+420,
+1600
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "73fcb8a0-2672-4bd5-86de-8075e1e02baf",
+"name": "=block",
+"type": "array",
+"value": "={{\n(function(){\n const block = $json.text\n .replace('```json', '')\n .replace('```', '')\n .trim()\n .parseJson();\n if (Array.isArray(block)) return block;\n if (block.type.startsWith('heading_')) {\n const prev = Number(block.type.split('_')[1]);\n const next = Math.max(1, prev - 1);\n if (next !== prev) {\n block.type = `heading_${next}`;\n block[`heading_${next}`] = Object.assign({}, block[`heading_${prev}`]);\n block[`heading_${prev}`] = undefined;\n }\n }\n return [block];\n})()\n}}"
+}
+]
+}
+},
+"executeOnce": false,
+"typeVersion": 3.4
+},
+{
+"id": "e2a5a5bc-a3c8-42c5-9419-74ce3525f599",
+"name": "Upload to Notion Page",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"maxTries": 2,
+"position": [
+1680,
+1760
+],
+"parameters": {
+"url": "=https://api.notion.com/v1/blocks/{{ $('Get Existing Row1').first().json.id }}/children",
+"method": "PATCH",
+"options": {
+"timeout": "={{ 1000 * 60 }}"
+},
+"jsonBody": "={{\n{\n \"children\": $json.block\n}\n}}",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"headerParameters": {
+"parameters": [
+{
+"name": "Notion-Version",
+"value": "2022-06-28"
+}
+]
+},
+"nodeCredentialType": "notionApi"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"retryOnFail": true,
+"typeVersion": 4.2,
+"waitBetweenTries": 3000
+},
+{
+"id": "9f18b2a5-ba74-40fc-8e35-a93ecd13507a",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+620
+],
+"parameters": {
+"color": 7,
+"width": 1740,
+"height": 740,
+"content": "## 8. Web Search and Extracting Web Page Contents using [APIFY.com](https://www.apify.com?fpr=414q6)\n[Read more about the HTTP Request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nHere is where I deviated a little from the reference implementation. I opted not to use Firecrawl.ai due to (1) high cost of the service and (2) a regular non-ai crawler would work just as well and probably quicker.\nInstead I'm using [APIFY.com](https://www.apify.com?fpr=414q6) which is a more performant, cost-effective and reliable web scraper service. If you don't want to use Apify, feel free to swap this out with your preferred service.\n\nThis step is the most exciting in terms of improvements and optimisations eg. mix in internal data sources! Add in Perplexity.ai or Jina.ai! Possibilities are endless."
+},
+"typeVersion": 1
+},
+{
+"id": "84c34a2a-d8bb-4e62-a5ea-df0a142aa2b4",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+60
+],
+"parameters": {
+"color": 7,
+"width": 460,
+"height": 360,
+"content": "## 5. Set Report to In-Progress\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)"
+},
+"typeVersion": 1
+},
+{
+"id": "bd022636-873d-4aca-8929-c189f8596cc1",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1240,
+700
+],
+"parameters": {
+"color": 7,
+"width": 780,
+"height": 660,
+"content": "## 9. Compile Learnings with Reasoning Model\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nWith our gathered sources, it's now just a case of giving it to our LLM to compile a list of \"learnings\" from them. For our DeepResearcher, we'll use OpenAI's o3-mini which is the latest reasoning model at time of writing. Reasoning perform better than regular chat models due their chain-of-thought or \"thinking\" process that they perform.\n\nThe \"Learnings\" are then combined with the generated research goal to complete one loop."
+},
+"typeVersion": 1
+},
+{
+"id": "b2f6e51d-cbe6-4459-9515-679f79063926",
+"name": "Get Existing Row1",
+"type": "n8n-nodes-base.notion",
+"position": [
+-1020,
+1600
+],
+"parameters": {
+"limit": 1,
+"filters": {
+"conditions": [
+{
+"key": "Request ID|rich_text",
+"condition": "equals",
+"richTextValue": "={{ $json.requestId.toString() }}"
+}
+]
+},
+"options": {},
+"resource": "databasePage",
+"matchType": "allFilters",
+"operation": "getAll",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"filterType": "manual"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "7b315060-7e40-410c-ac9d-ef22acbb175a",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 660,
+"height": 540,
+"content": "## 10. Generate DeepSearch Report using Learnings\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nFinally! After all learnings have been gathered - which may have taken up to an hour or more on the higher settings! - they are given to our LLM to generate the final research report in markdown format. Technically, the DeepResearch ends here but for this template, we need to push the output to Notion. If you're not using Notion, feel free to ignore the last few steps."
+},
+"typeVersion": 1
+},
+{
+"id": "01aaf2a8-c145-4f39-aa52-f40fc28f8767",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-460,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 1060,
+"height": 540,
+"content": "## 11. Reformat Report as Notion Blocks\n[Learn more about the Markdown node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.markdown/)\n\nTo write our report to our Notion page, we'll have to convert it to Notion \"blocks\" - these are specialised json objects which are required by the Notion API. There are quite a number of ways to do this conversion not involving the use of AI but for kicks, I decided to do so anyway. In this step, we first convert to HTML as it allows us to split the report semantically and makes for easier parsing for the LLM."
+},
+"typeVersion": 1
+},
+{
+"id": "700a6f44-86bf-4aab-8a42-23bf6843f681",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1220,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 800,
+"height": 580,
+"content": "## 13. Update Report in Notion\n[Read more about the HTTP request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nIn this step, we can use the Notion API to add the blocks to our page sequentially. A loop is used due to the unstable Notion API - the loop allows retries for blocks that require it."
+},
+"typeVersion": 1
+},
+{
+"id": "f8536052-c851-42ec-aaf3-5fc876570f6d",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1840,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 680,
+"height": 560,
+"content": "## 1. Let's Research!\n[Learn more about the form trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.formtrigger)\n\nn8n forms are a really nice way to get our frontend up and running quickly and compared to chat, offers a superior user interface for user input. I've gone perhaps a little extra with the custom html fields but I do enjoy adding a little customisation now and then."
+},
+"typeVersion": 1
+},
+{
+"id": "58aa92a2-e1fd-497d-a27e-40b733189bab",
+"name": "DeepResearch Report",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-860,
+1600
+],
+"parameters": {
+"text": "=You are are an expert and insightful researcher.\n* Given the following prompt from the user, write a final report on the topic using the learnings from research.\n* Make it as as detailed as possible, aim for 3 or more pages, include ALL the learnings from research.\n* Format the report in markdown. Use headings, lists and tables only and where appropriate.\n\n{{ $('JobType Router').first().json.data.query }}\n\nHere are all the learnings from previous research:\n\n\n{{\n$('JobType Router').first().json.data\n .all_learnings\n .map(item => `${item}`) \n .join('\\n')\n}}\n",
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "0656be83-d510-46f1-aeeb-f62a69aa3cf2",
+"name": "DeepResearch Learnings",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+1500,
+980
+],
+"parameters": {
+"text": "=Given the following contents from a SERP search for the query {{ $('Item Ref').first().json.query }}, generate a list of learnings from the contents. Return a maximum of 3 learnings, but feel free to return less if the contents are clear. Make sure each learning is unique and not similar to each other. The learnings should be concise and to the point, as detailed and infromation dense as possible. Make sure to include any entities like people, places, companies, products, things, etc in the learnings, as well as any exact metrics, numbers, or dates. The learnings will be used to research the topic further.\n\n\n{{\n$('Convert to Markdown')\n .all()\n .map(item =>`\\n${item.json.markdown.substr(0, 25_000)}\\n`)\n .join('\\n')\n}}\n",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"executeOnce": true,
+"typeVersion": 1.5
+},
+{
+"id": "9296a787-3226-44fe-8118-f84dda8e5167",
+"name": "Generate Report",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+480,
+180
+],
+"parameters": {
+"options": {
+"waitForSubWorkflow": false
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{\n{\n ...Object.assign({}, $json),\n query: $('JobType Router').first().json.data.query\n}\n}}",
+"jobType": "deepresearch_report",
+"requestId": "={{ $('JobType Router').first().json.requestId }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "471f9b9f-f331-4652-95de-1ec7136ea692",
+"name": "Generate Learnings",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+-380,
+180
+],
+"parameters": {
+"mode": "each",
+"options": {
+"waitForSubWorkflow": true
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{ $json }}",
+"jobType": "deepresearch_learnings",
+"requestId": "={{ $('JobType Router').first().json.requestId }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "2e2fa1e6-9d5c-46ff-985c-58ada1139837",
+"name": "Confirmation",
+"type": "n8n-nodes-base.form",
+"position": [
+780,
+-420
+],
+"webhookId": "cf41a176-5d30-4274-955e-b0d5b483d37f",
+"parameters": {
+"options": {
+"formTitle": "DeepResearcher",
+"buttonLabel": "Done",
+"formDescription": "=\n
\nYour Report Is On Its Way!\n \nDeepResearcher will now work independently to conduct the research and the compiled report will be uploaded to the following Notion page below when finished.\n
\nPlease click the \"Done\" button to complete the form.\n
\n This value determines how many sources to explore.\n
\n \n \n
\n\n",
+"fieldType": "html",
+"fieldLabel": "Enter research breadth (recommended 2-10, default 4)"
+},
+{
+"fieldType": "dropdown",
+"fieldLabel": "={{ \"\" }}",
+"multiselect": true,
+"fieldOptions": {
+"values": [
+{
+"option": "=I understand higher depth and breath values I've selected may incur longer wait times and higher costs. I acknowledging this and wish to proceed with the research request."
+}
+]
+},
+"requiredField": true
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b3d11997-9c8c-4b72-b750-4fc22a2247b7",
+"name": "Valid Blocks",
+"type": "n8n-nodes-base.filter",
+"position": [
+740,
+1600
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f68cefe0-e109-4d41-9aa3-043f3bc6c449",
+"operator": {
+"type": "string",
+"operation": "notExists",
+"singleValue": true
+},
+"leftValue": "={{ $json.error }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "ff90a1c1-b357-4012-8964-e007bef0c9db",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+620,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 580,
+"height": 580,
+"content": "## 12. Append URL Sources List\n[Read more about the Code node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nFor our source URLs, we'll manually compose the Notion blocks for them - this is because there's usually a lot of them! We'll then append to the end of the other blocks."
+},
+"typeVersion": 1
+},
+{
+"id": "f6b50f06-6122-494c-bdb8-4215f473a27d",
+"name": "Append Blocks",
+"type": "n8n-nodes-base.merge",
+"position": [
+1000,
+1760
+],
+"parameters": {},
+"typeVersion": 3
+},
+{
+"id": "ca39272a-828e-4314-80da-05dd5fd7b2e3",
+"name": "URL Sources to Lists",
+"type": "n8n-nodes-base.code",
+"position": [
+740,
+1760
+],
+"parameters": {
+"jsCode": "const urls = $('JobType Router').first().json.data.all_urls;\nconst chunksize = 50;\nconst splits = Math.max(1, Math.floor(urls.length/chunksize));\n\nconst blocks = Array(splits).fill(0)\n .map((_, idx) => {\n const block = urls\n .slice(\n idx * chunksize, \n (idx * chunksize) + chunksize - 1\n )\n .map(url => {\n return {\n object: \"block\",\n type: \"bulleted_list_item\",\n bulleted_list_item: {\n rich_text: [\n { type: \"text\", text: { content: url } }\n ]\n }\n }\n });\n return { json: { block } }\n });\n\nreturn [\n { json: {\n block:[{\n \"object\": \"block\",\n \"type\": \"heading_2\",\n \"heading_2\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Sources\"\n }\n }\n ]\n }\n }]\n } },\n ...blocks\n];"
+},
+"typeVersion": 2
+},
+{
+"id": "e2e2b07a-4039-4859-b60c-f51982475282",
+"name": "Has Results?",
+"type": "n8n-nodes-base.if",
+"position": [
+240,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "9ef8d40c-1289-4654-9022-4a07f7102555",
+"operator": {
+"type": "array",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json.results }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "5662dbf2-8877-4e83-982c-6bc5968b8835",
+"name": "Empty Response",
+"type": "n8n-nodes-base.set",
+"position": [
+1040,
+1120
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "1de40158-338b-4db3-9e22-6fd63b21f825",
+"name": "ResearchGoal",
+"type": "string",
+"value": "={{ $('Item Ref').first().json.researchGoal }}"
+},
+{
+"id": "9f59a2d4-5e5a-4d0b-8adf-2832ce746f0f",
+"name": "learnings",
+"type": "array",
+"value": "={{ [] }}"
+},
+{
+"id": "972ab5f5-0537-4755-afcb-d1db4f09ad60",
+"name": "followUpQuestions",
+"type": "array",
+"value": "={{ [] }}"
+},
+{
+"id": "90cef471-76b0-465d-91a4-a0e256335cd3",
+"name": "urls",
+"type": "array",
+"value": "={{ [] }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "678b6af5-da74-4421-8c3d-0166aa52efd9",
+"name": "Has Content?",
+"type": "n8n-nodes-base.if",
+"position": [
+880,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "1ef1039a-4792-47f9-860b-d2ffcffd7129",
+"operator": {
+"type": "object",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "233f2e19-b4f2-4de3-8002-f79f3c01c1e7",
+"name": "Sticky Note13",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1820,
+-240
+],
+"parameters": {
+"color": 5,
+"width": 300,
+"height": 100,
+"content": "### Not using forms?\nFeel free ot swap this out for chat or even webhooks to fit your existing workflows."
+},
+"typeVersion": 1
+},
+{
+"id": "453bb6eb-f2b0-4e21-b647-e095c80b7844",
+"name": "Sticky Note14",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1880,
+540
+],
+"parameters": {
+"color": 5,
+"width": 460,
+"height": 240,
+"content": "### ๐ The Subworkflow Event Pattern \nIf you're new to n8n, this advanced technique might need some explaining but in gist, we're using subworkflows to run different parts of our DeepResearcher workflow as separate executions.\n\n* Necessary to implement the recursive loop mechanism needed to enable this workflow.\n* Negates the need to split this workflow into multiple templates.\n* Great generally for building high performance n8n workflows (a topic for a future post!)"
+},
+"typeVersion": 1
+},
+{
+"id": "289cbe4c-c2e3-46b7-8799-197a7d78ab2a",
+"name": "Sticky Note15",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+720,
+-60
+],
+"parameters": {
+"color": 5,
+"width": 340,
+"height": 200,
+"content": "### Recursive Looping\nThe recursive looping implemented for this workflow is an advanced item-linking technique. It works by specifically controlling which nodes \"execute once\" vs\" execute for each item\" because of this becareful of ermoving nodes! Always check the settings of the node you're replacing and ensure the settings match. "
+},
+"typeVersion": 1
+},
+{
+"id": "b95ffdcd-c0d1-4a12-a7a9-24135db7b467",
+"name": "Combine & Send back to Loop",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+-240,
+820
+],
+"parameters": {
+"options": {},
+"aggregate": "aggregateAllItemData"
+},
+"typeVersion": 1
+},
+{
+"id": "f0a48ab5-70b9-49dc-a153-61e573803d1e",
+"name": "For Each Block...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+1440,
+1600
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "2fd17fbd-005d-446e-b014-0da190cd3114",
+"name": "Sticky Note16",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2420,
+-920
+],
+"parameters": {
+"width": 520,
+"height": 1060,
+"content": "## n8n DeepResearcher\n### This template attempts to replicate OpenAI's DeepResearch feature which, at time of writing, is only available to their pro subscribers.\n\nThough the inner workings of DeepResearch have not been made public, it is presumed the feature relies on the ability to deep search the web, scrape web content and invoking reasoning models to generate reports. All of which n8n is really good at!\n\n### How it works\n* A form is used to first capture the user's research query and how deep they'd like the researcher to go.\n* Once submitted, a blank Notion page is created which will later hold the final report and the researcher gets to work.\n* The user's query goes through a recursive series of web serches and web scraping to collect data on the research topic to generate partial learnings.\n* Once complete, all learnings are combined and given to a reasoning LLM to generate the final report.\n* The report is then written to the placeholder Notion page created earlier. \n\n### How to use\n* Duplicate this Notion database to use with this template: https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf\n* Sign-up for [APIFY.com](https://www.apify.com?fpr=414q6) API Key for web search and scraping services.\n* Ensure you have access to OpenAI's o3-mini model. Alternatively, switch this out for o1 series.\n* You must publish this workflow and ensure the form url is publically accessible.\n\n### On Depth & Breadth Configuration\nFor more detailed reports, increase depth and breadth but be warned the workflow will take a exponentially more time and money to complete. The defaults are usually good enough.\n\nDepth=1 & Breadth=2 - will take about 10 - 15mins.\nDepth=1 & Breadth=4 - will take about 30 - 40mins.\nDpeth=3 & Breadth=5 - will take about 2 - 5 hours!\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "ac6f2604-7439-4524-a27e-2f031ebce089",
+"name": "Sticky Note17",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2420,
+-1180
+],
+"parameters": {
+"color": 7,
+"width": 520,
+"height": 240,
+"content": ""
+},
+"typeVersion": 1
+},
+{
+"id": "40e3a0cf-7710-4537-b147-37ba8945fdbc",
+"name": "Sticky Note18",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-120,
+960
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
+},
+"typeVersion": 1
+},
+{
+"id": "5a96ecc2-eeea-4c33-b299-7a7f2ca7559c",
+"name": "Sticky Note19",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+520,
+960
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
+},
+"typeVersion": 1
+},
+{
+"id": "23bca6e2-e16a-48a4-a7fc-96ce25846764",
+"name": "Sticky Note20",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1640,
+1740
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE NOTION CREDENTIAL HERE!"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Item Ref": {
+"main": [
+[
+{
+"node": "Web Search",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create Row": {
+"main": [
+[
+{
+"node": "Initiate DeepResearch",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Web Search": {
+"main": [
+[
+{
+"node": "Top 5 Organic Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Valid Pages": {
+"main": [
+[
+{
+"node": "Has Content?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Confirmation": {
+"main": [
+[
+{
+"node": "End Form",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Content?": {
+"main": [
+[
+{
+"node": "Convert to Markdown",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Empty Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Results?": {
+"main": [
+[
+{
+"node": "URLs to Items",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Empty Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Valid Blocks": {
+"main": [
+[
+{
+"node": "Append Blocks",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Append Blocks": {
+"main": [
+[
+{
+"node": "For Each Block...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"HTML to Array": {
+"main": [
+[
+{
+"node": "Tags to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Page Contents": {
+"main": [
+[
+{
+"node": "Valid Pages",
+"type": "main",
+"index": 0
+}
+],
+[]
+]
+},
+"SERP to Items": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Variables": {
+"main": [
+[
+{
+"node": "Clarifying Questions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Tags to Items": {
+"main": [
+[
+{
+"node": "Notion Block Generator",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"URLs to Items": {
+"main": [
+[
+{
+"node": "Page Contents",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Empty Response": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execution Data": {
+"main": [
+[
+{
+"node": "JobType Router",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"JobType Router": {
+"main": [
+[
+{
+"node": "Get Existing Row",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Generate SERP Queries",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Existing Row1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert to HTML": {
+"main": [
+[
+{
+"node": "HTML to Array",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set In-Progress": {
+"main": [
+[
+{
+"node": "Set Initial Query",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Existing Row": {
+"main": [
+[
+{
+"node": "Set In-Progress",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Research Request": {
+"main": [
+[
+{
+"node": "Set Variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Results to Items": {
+"main": [
+[
+{
+"node": "Set Next Queries",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Next Queries": {
+"main": [
+[
+{
+"node": "Generate Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Feedback to Items": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Block...": {
+"main": [
+[
+{
+"node": "Set Done",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Upload to Notion Page",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Query...": {
+"main": [
+[
+{
+"node": "Combine & Send back to Loop",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Item Ref",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Existing Row1": {
+"main": [
+[
+{
+"node": "DeepResearch Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Initial Query": {
+"main": [
+[
+{
+"node": "Report Page Generator",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Is Depth Reached?": {
+"main": [
+[
+{
+"node": "Get Research Results",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "DeepResearch Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Parse JSON blocks": {
+"main": [
+[
+{
+"node": "Valid Blocks",
+"type": "main",
+"index": 0
+},
+{
+"node": "URL Sources to Lists",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Initial Query": {
+"main": [
+[
+{
+"node": "Generate Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Accumulate Results": {
+"main": [
+[
+{
+"node": "Is Depth Reached?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate Learnings": {
+"main": [
+[
+{
+"node": "Accumulate Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"On form submission": {
+"main": [
+[
+{
+"node": "Research Request",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "DeepResearch Report",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model2": {
+"ai_languageModel": [
+[
+{
+"node": "Clarifying Questions",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model3": {
+"ai_languageModel": [
+[
+{
+"node": "Generate SERP Queries",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model4": {
+"ai_languageModel": [
+[
+{
+"node": "Report Page Generator",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Convert to Markdown": {
+"main": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Report": {
+"main": [
+[
+{
+"node": "Convert to HTML",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Clarifying Questions": {
+"main": [
+[
+{
+"node": "Feedback to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Results": {
+"main": [
+[
+{
+"node": "Results to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Question...": {
+"main": [
+[
+{
+"node": "Get Initial Query",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Ask Clarity Questions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Research Results": {
+"main": [
+[
+{
+"node": "Generate Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"URL Sources to Lists": {
+"main": [
+[
+{
+"node": "Append Blocks",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Ask Clarity Questions": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate SERP Queries": {
+"main": [
+[
+{
+"node": "SERP to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Initiate DeepResearch": {
+"main": [
+[
+{
+"node": "Confirmation",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Report Page Generator": {
+"main": [
+[
+{
+"node": "Create Row",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Top 5 Organic Results": {
+"main": [
+[
+{
+"node": "Has Results?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Upload to Notion Page": {
+"main": [
+[
+{
+"node": "For Each Block...",
+"type": "main",
+"index": 0
+}
+],
+[]
+]
+},
+"DeepResearch Learnings": {
+"main": [
+[
+{
+"node": "Research Goal + Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Notion Block Generator": {
+"main": [
+[
+{
+"node": "Parse JSON blocks",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Subworkflow": {
+"main": [
+[
+{
+"node": "Execution Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Gemini Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Notion Block Generator",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Research Goal + Learnings": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser1": {
+"ai_outputParser": [
+[
+{
+"node": "Clarifying Questions",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser2": {
+"ai_outputParser": [
+[
+{
+"node": "Generate SERP Queries",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser4": {
+"ai_outputParser": [
+[
+{
+"node": "Report Page Generator",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+}
+}
+}{
+"meta": {
+"instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9",
+"templateCredsSetupCompleted": true
+},
+"nodes": [
+{
+"id": "645ae2b1-799e-49be-8bdf-12cd1bb739e6",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1680,
+1140
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"learnings\": {\n \"type\": \"array\",\n \"description\": \"List of learnings, max of 3.\",\n \"items\": { \"type\": \"string\" }\n },\n \"followUpQuestions\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\",\n \"description\": \"List of follow-up questions to research the topic further, max of 3.\"\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "cbdb4e98-eeba-4609-91de-394c416b7904",
+"name": "Set Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-1360,
+-460
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "df28b12e-7c20-4ff5-b5b8-dc773aa14d4b",
+"name": "request_id",
+"type": "string",
+"value": "={{ $execution.id }}"
+},
+{
+"id": "9362c1e7-717d-444a-8ea2-6b5f958c9f3f",
+"name": "prompt",
+"type": "string",
+"value": "={{ $json['What would you like to research?'] }}"
+},
+{
+"id": "09094be4-7844-4a9e-af82-cc8e39322398",
+"name": "depth",
+"type": "number",
+"value": "={{ $json['Enter research depth (recommended 1-5, default 2)'] || 2 }}"
+},
+{
+"id": "3fc30a30-7806-4013-835d-97e27ddd7ae1",
+"name": "breadth",
+"type": "number",
+"value": "={{ $json['Enter research breadth (recommended 2-10, default 4)'] || 4 }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "c7096ab9-0b10-45b0-b178-a049bf57830b",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+1500,
+1140
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "d0f1bc2f-6a10-4ac7-8d35-34f48f14fad5",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-860,
+1760
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "bba3278c-0336-4305-887d-56515dfd87db",
+"name": "OpenAI Chat Model2",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-1060,
+-300
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "f31f2fc7-0bec-4105-9d83-5f4f9a0eb35d",
+"name": "Structured Output Parser1",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+-840,
+-300
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"questions\": {\n \"type\": \"array\",\n \"description\": \"Follow up questions to clarify the research direction, max of 3.\",\n \"items\": {\n \"type\": \"string\"\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "ea59c5ab-fa05-4c68-bc60-3f56e240478b",
+"name": "On form submission",
+"type": "n8n-nodes-base.formTrigger",
+"position": [
+-1760,
+-460
+],
+"webhookId": "7ddfaa7c-a523-4d92-b033-d76cd5a313e9",
+"parameters": {
+"options": {
+"path": "deep_research",
+"ignoreBots": true,
+"buttonLabel": "Next"
+},
+"formTitle": " DeepResearcher",
+"formFields": {
+"values": [
+{
+"fieldType": "html",
+"fieldLabel": "placeholder"
+}
+]
+},
+"formDescription": "=DeepResearcher is a multi-step, recursive approach using the internet to solve complex research tasks, accomplishing in tens of minutes what a human would take many hours.\n\nTo use, provide a short summary of what the research and how \"deep\" you'd like the workflow to investigate. Note, the higher the numbers the more time and cost will occur for the research.\n\nThe workflow is designed to complete independently and when finished, a report will be saved in a designated Notion Database."
+},
+"typeVersion": 2.2
+},
+{
+"id": "a8262288-a8c1-4967-9870-f728fa08b579",
+"name": "Generate SERP Queries",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-1040,
+820
+],
+"parameters": {
+"text": "=Given the following prompt from the user, generate a list of SERP queries to research the topic. Return a maximum of {{ $('JobType Router').first().json.data.breadth }} queries, but feel free to return less if the original prompt is clear. Make sure each query is unique and not similar to each other: {{ $('JobType Router').first().json.data.query.trim() }}\n\n{{\n$('JobType Router').first().json.data.learnings.length\n ? `Here are some learnings from previous research, use them to generate more specific queries: ${$('JobType Router').first().json.data.learnings.join('\\n')}`\n : ''\n}}",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "0534be47-22b7-4c2a-956b-d085e6b9f280",
+"name": "Structured Output Parser2",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+-860,
+980
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"queries\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"query\": {\n \"type\": \"string\",\n \"description\": \"The SERP query\"\n },\n \"researchGoal\": {\n \"type\": \"string\",\n \"description\": \"First talk about the goal of the research that this query is meant to accomplish, then go deeper into how to advance the research once the results are found, mention additional research directions. Be as specific as possible, especially for additional research directions.\"\n }\n }\n }\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "4d8aa196-986f-442d-9b56-92c043ab785d",
+"name": "OpenAI Chat Model3",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-1040,
+980
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "7488b037-7422-4f62-8c37-1f6a901b3299",
+"name": "Set Initial Query",
+"type": "n8n-nodes-base.set",
+"position": [
+-580,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "acb41e93-70c6-41a3-be0f-e5a74ec3ec88",
+"name": "query",
+"type": "string",
+"value": "={{ $('JobType Router').first().json.data.query }}"
+},
+{
+"id": "7fc54063-b610-42bc-a250-b1e8847c4d1e",
+"name": "learnings",
+"type": "array",
+"value": "={{ $('JobType Router').first().json.data.learnings }}"
+},
+{
+"id": "e8f1c158-56fb-41c8-8d86-96add16289bb",
+"name": "breadth",
+"type": "number",
+"value": "={{ $('JobType Router').first().json.data.breadth }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "12ae382e-d88a-4f1b-a71f-3bd63c892b17",
+"name": "SERP to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-700,
+820
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output.queries"
+},
+"typeVersion": 1
+},
+{
+"id": "46700052-f48a-493c-aebf-cdf175d58550",
+"name": "Item Ref",
+"type": "n8n-nodes-base.noOp",
+"position": [
+-240,
+980
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "2cef6f1d-e244-4ee6-bf25-6dc3e8042afa",
+"name": "Research Goal + Learnings",
+"type": "n8n-nodes-base.set",
+"position": [
+1840,
+1120
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "9acec2cc-64c8-4e62-bed4-c3d9ffab1379",
+"name": "researchGoal",
+"type": "string",
+"value": "={{ $('Item Ref').first().json.researchGoal }}"
+},
+{
+"id": "1b2d2dad-429b-4fc9-96c5-498f572a85c3",
+"name": "learnings",
+"type": "array",
+"value": "={{ $json.output.learnings }}"
+},
+{
+"id": "655b99f2-6045-4774-a634-49751bc9326f",
+"name": "followUpQuestions",
+"type": "array",
+"value": "={{ $json.output.followUpQuestions }}"
+},
+{
+"id": "c9e34ea4-5606-46d6-8d66-cb42d772a8b4",
+"name": "urls",
+"type": "array",
+"value": "={{\n$('Page Contents')\n .all()\n .filter(item => !item.json.error && item.json.body)\n .map(item => item.json.url)\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "4aebae86-2bd2-4f3d-8290-d34b9ac837c6",
+"name": "Accumulate Results",
+"type": "n8n-nodes-base.set",
+"position": [
+-200,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "db509e90-9a86-431f-8149-4094d22666cc",
+"name": "should_stop",
+"type": "boolean",
+"value": "={{\n$runIndex >= ($('JobType Router').first().json.data.depth)\n}}"
+},
+{
+"id": "90986e2b-8aca-4a22-a9db-ed8809d6284d",
+"name": "all_learnings",
+"type": "array",
+"value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.learnings))\n } catch (e) {\n return []\n }\n })\n}}"
+},
+{
+"id": "3eade958-e8ab-4975-aac4-f4a4a983c163",
+"name": "all_urls",
+"type": "array",
+"value": "={{\nArray($runIndex+1)\n .fill(0)\n .flatMap((_,idx) => {\n try {\n return $('Generate Learnings')\n .all(0,idx)\n .flatMap(item => item.json.data.flatMap(d => d.urls))\n } catch (e) {\n return []\n }\n })\n}}"
+}
+]
+}
+},
+"executeOnce": true,
+"typeVersion": 3.4
+},
+{
+"id": "782baa36-ba07-4845-873c-c9400de6d463",
+"name": "DeepResearch Results",
+"type": "n8n-nodes-base.set",
+"position": [
+160,
+360
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "={{ $('Generate Learnings').item.json }}"
+},
+"typeVersion": 3.4
+},
+{
+"id": "89b09898-79ec-4924-975f-e9581d3bf774",
+"name": "Results to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+320,
+360
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 1
+},
+{
+"id": "122cd071-aade-4753-ba0a-8db4c58fa84e",
+"name": "Set Next Queries",
+"type": "n8n-nodes-base.set",
+"position": [
+480,
+360
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "d88bfe95-9e73-4d25-b45c-9f164b940b0e",
+"name": "query",
+"type": "string",
+"value": "=Previous research goal: {{ $json.researchGoal }}\nFollow-up research directions: {{ $json.followUpQuestions.map(q => `\\n${q}`).join('') }}"
+},
+{
+"id": "4aa20690-d998-458a-b1e4-0d72e6a68e6b",
+"name": "learnings",
+"type": "array",
+"value": "={{ $('Accumulate Results').item.json.all_learnings }}"
+},
+{
+"id": "89acafae-b04a-4d5d-b08b-656e715654e4",
+"name": "breadth",
+"type": "number",
+"value": "={{ $('JobType Router').first().json.data.breadth }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "9da01d8a-48d6-45b4-b8c6-9a0503b4bda6",
+"name": "Web Search",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+-80,
+980
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/serping~fast-google-search-results-scraper/run-sync-get-dataset-items",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"authentication": "genericCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "searchTerms",
+"value": "={{\n[\n `${$json.query} -filetype:pdf`\n]\n}}"
+},
+{
+"name": "resultsPerPage",
+"value": "={{ 10 }}"
+}
+]
+},
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+},
+"httpHeaderAuth": {
+"id": "SV9BDKc1cRbZBeoL",
+"name": "Apify.com (personal token)"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "99bd2c8e-5600-43a9-ab2f-7f2911efb16c",
+"name": "Top 5 Organic Results",
+"type": "n8n-nodes-base.set",
+"position": [
+80,
+980
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "29d1a759-d886-4a44-860b-9d16f9922043",
+"name": "results",
+"type": "array",
+"value": "={{\n$json.origin_search.results\n ? $json.origin_search\n .results\n .filter(res => res.type === 'normal')\n .slice(0, 5)\n .map(res => ({\n title: res.title,\n url: res.source.link\n }))\n : []\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "cb7c5a8b-5420-4fb9-b7f0-4e8e8d10034a",
+"name": "Convert to Markdown",
+"type": "n8n-nodes-base.markdown",
+"position": [
+1320,
+980
+],
+"parameters": {
+"html": "={{ $json.body }}",
+"options": {
+"ignore": "a,img,picture,svg,video,audio,iframe"
+},
+"destinationKey": "markdown"
+},
+"typeVersion": 1
+},
+{
+"id": "818ccf2e-081d-492e-ba8d-de458b0c26db",
+"name": "For Each Query...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-420,
+820
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "1787b562-17e8-41af-9cdc-eb2d3e630916",
+"name": "Feedback to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-720,
+-460
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output.questions"
+},
+"typeVersion": 1
+},
+{
+"id": "4c695faa-74e3-456b-a1ef-aaea67e46743",
+"name": "Ask Clarity Questions",
+"type": "n8n-nodes-base.form",
+"position": [
+-360,
+-380
+],
+"webhookId": "ab0d9b81-73f6-4baa-a3cd-ac3b31397708",
+"parameters": {
+"options": {
+"formTitle": "DeepResearcher",
+"buttonLabel": "Answer",
+"formDescription": "=\n
\nAnswer the following clarification questions to assist the DeepResearcher better under the research topic.\n
\n\n
\nTotal {{ $('Feedback to Items').all().length }} questions.\n
"
+},
+"formFields": {
+"values": [
+{
+"fieldType": "textarea",
+"fieldLabel": "={{ $json[\"output.questions\"] }}",
+"placeholder": "=",
+"requiredField": true
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "e07d8c3e-8bcd-4393-9892-f825433ab58d",
+"name": "For Each Question...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-540,
+-460
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "e8d26351-52f4-40a6-ba5b-fb6bc816b734",
+"name": "DeepResearch Subworkflow",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-1880,
+820
+],
+"parameters": {
+"workflowInputs": {
+"values": [
+{
+"name": "requestId",
+"type": "any"
+},
+{
+"name": "jobType"
+},
+{
+"name": "data",
+"type": "object"
+}
+]
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "25a8055a-27aa-414f-856b-25a2e2f31974",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 1000,
+"height": 560,
+"content": "## 2. Ask Clarifying Questions\n[Read more about form nodes](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.form/)\n\nTo handle the clarification questions generated by the LLM, I used the same technique found in my \"AI Interviewer\" template ([link](https://n8n.io/workflows/2566-conversational-interviews-with-ai-agents-and-n8n-forms/)).\nThis involves a looping of dynamically generated forms to collect answers from the user."
+},
+"typeVersion": 1
+},
+{
+"id": "68398b92-eb35-48bf-885e-540074531cc4",
+"name": "Clarifying Questions",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-1040,
+-460
+],
+"parameters": {
+"text": "=Given the following query from the user, ask some follow up questions to clarify the research direction. Return a maximum of 3 questions, but feel free to return less if the original query is clear: {{ $json.prompt }}`",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "65c4c293-67b8-4e64-af04-16e45e97c09a",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-660,
+-60
+],
+"parameters": {
+"color": 7,
+"width": 1360,
+"height": 640,
+"content": "## 6. Perform DeepSearch Loop\n[Learn more about the Looping in n8n](https://docs.n8n.io/flow-logic/looping/#creating-loops)\n\nThe key of the Deep Research flow is its extensive data collection capability. In this implementation, this capability is represented by a recursive web search & scrape loop which starts with the original query and extended by AI-generated subqueries. How many subqueries to generate are determined the depth and breadth parameters specified.\n\n\"Learnings\" are generated for each subquery and accumulate on each iteration of the loop. When the loop finishes when depth limit is reached, all learnings are collected and it's these learnings are what we use to generate the report."
+},
+"typeVersion": 1
+},
+{
+"id": "43a5d93d-cae2-43ec-b9ae-b15d6b11b932",
+"name": "End Form",
+"type": "n8n-nodes-base.form",
+"position": [
+960,
+-420
+],
+"webhookId": "7b531f5d-942f-4c49-ac55-8ee480889600",
+"parameters": {
+"options": {},
+"operation": "completion",
+"completionTitle": "=Thank you for using DeepResearcher.",
+"completionMessage": "=You may now close this window."
+},
+"typeVersion": 1
+},
+{
+"id": "9a824011-e76f-433f-8735-44b358f4ff7d",
+"name": "Initiate DeepResearch",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+600,
+-420
+],
+"parameters": {
+"mode": "each",
+"options": {
+"waitForSubWorkflow": false
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{\n{\n \"query\": $('Get Initial Query').first().json.query,\n \"learnings\": [],\n \"depth\": $('Set Variables').first().json.depth,\n \"breadth\": $('Set Variables').first().json.breadth,\n}\n}}",
+"jobType": "deepresearch_initiate",
+"requestId": "={{ $('Set Variables').first().json.request_id }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "c48ee4cd-bac1-4405-bb4c-5614e5eb25a0",
+"name": "Page Contents",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+560,
+980
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/apify~web-scraper/run-sync-get-dataset-items",
+"options": {},
+"jsonBody": "={\n \"startUrls\": {{ [{ url: $json.url, method: 'GET' }].toJsonString() }},\n \"breakpointLocation\": \"NONE\",\n \"browserLog\": false,\n \"closeCookieModals\": false,\n \"debugLog\": false,\n \"downloadCss\": false,\n \"downloadMedia\": false,\n \"excludes\": [\n {\n \"glob\": \"/**/*.{png,jpg,jpeg,pdf}\"\n }\n ],\n \"headless\": true,\n \"ignoreCorsAndCsp\": false,\n \"ignoreSslErrors\": false,\n \"injectJQuery\": true,\n \"keepUrlFragments\": false,\n \"linkSelector\": \"\",\n \"maxCrawlingDepth\": 1,\n \"maxPagesPerCrawl\": 1,\n \"maxRequestRetries\": 1,\n \"maxResultsPerCrawl\": 1,\n \"pageFunction\": \"// The function accepts a single argument: the \\\"context\\\" object.\\n// For a complete list of its properties and functions,\\n// see https://apify.com/apify/web-scraper#page-function \\nasync function pageFunction(context) {\\n\\n await new Promise(res => { setTimeout(res, 6000) });\\n // This statement works as a breakpoint when you're trying to debug your code. Works only with Run mode: DEVELOPMENT!\\n // debugger; \\n\\n // jQuery is handy for finding DOM elements and extracting data from them.\\n // To use it, make sure to enable the \\\"Inject jQuery\\\" option.\\n const $ = context.jQuery;\\n const title = $('title').first().text();\\n\\n // Clone the body to avoid modifying the original content\\n const bodyClone = $('body').clone();\\n bodyClone.find('iframe, img, script, style, object, embed, noscript, svg, video, audio').remove();\\n const body = bodyClone.html();\\n\\n // Return an object with the data extracted from the page.\\n // It will be stored to the resulting dataset.\\n return {\\n url: context.request.url,\\n title,\\n body\\n };\\n}\",\n \"postNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept a single argument: the \\\"crawlingContext\\\" object.\\n[\\n async (crawlingContext) => {\\n // ...\\n },\\n]\",\n \"preNavigationHooks\": \"// We need to return array of (possibly async) functions here.\\n// The functions accept two arguments: the \\\"crawlingContext\\\" object\\n// and \\\"gotoOptions\\\".\\n[\\n async (crawlingContext, gotoOptions) => {\\n // ...\\n },\\n]\\n\",\n \"proxyConfiguration\": {\n \"useApifyProxy\": true\n },\n \"runMode\": \"PRODUCTION\",\n \"useChrome\": false,\n \"waitUntil\": [\n \"domcontentloaded\"\n ],\n \"globs\": [],\n \"pseudoUrls\": [],\n \"proxyRotation\": \"RECOMMENDED\",\n \"maxConcurrency\": 50,\n \"pageLoadTimeoutSecs\": 60,\n \"pageFunctionTimeoutSecs\": 60,\n \"maxScrollHeightPixels\": 5000,\n \"customData\": {}\n}",
+"sendBody": true,
+"sendQuery": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "memory",
+"value": "2048"
+},
+{
+"name": "timeout",
+"value": "90"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "dc9f85ff-7565-4c29-981a-5ef65bba6ca3",
+"name": "Execution Data",
+"type": "n8n-nodes-base.executionData",
+"position": [
+-1700,
+820
+],
+"parameters": {
+"dataToSave": {
+"values": [
+{
+"key": "requestId",
+"value": "={{ $json.requestId }}"
+},
+{
+"key": "=jobType",
+"value": "={{ $json.jobType }}"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "26b33429-6d61-4758-9c76-3e998dd31fa4",
+"name": "JobType Router",
+"type": "n8n-nodes-base.switch",
+"position": [
+-1520,
+820
+],
+"parameters": {
+"rules": {
+"values": [
+{
+"outputKey": "initiate",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"operator": {
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_initiate"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "learnings",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "ecbfa54d-fc97-48c5-8d3d-f0538b8d727b",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_learnings"
+}
+]
+},
+"renameOutput": true
+},
+{
+"outputKey": "report",
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "392f9a98-ec22-4e57-9c8e-0e1ed6b7dafa",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.jobType }}",
+"rightValue": "deepresearch_report"
+}
+]
+},
+"renameOutput": true
+}
+]
+},
+"options": {}
+},
+"typeVersion": 3.2
+},
+{
+"id": "a9637952-7c09-40ae-96ec-bdf0fc63d94e",
+"name": "Valid Pages",
+"type": "n8n-nodes-base.filter",
+"position": [
+720,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f44691e4-f753-47b0-b66a-068a723b6beb",
+"operator": {
+"type": "boolean",
+"operation": "false",
+"singleValue": true
+},
+"leftValue": "={{ $json['#error'] }}",
+"rightValue": ""
+},
+{
+"id": "8e05df2b-0d4a-47da-9aab-da7e8907cbca",
+"operator": {
+"type": "string",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json.body }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2,
+"alwaysOutputData": true
+},
+{
+"id": "204cfca2-05bb-46dd-ba96-b41866ed2cfe",
+"name": "OpenAI Chat Model4",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-20,
+-280
+],
+"parameters": {
+"model": {
+"__rl": true,
+"mode": "id",
+"value": "o3-mini"
+},
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "45bc6261-35c8-4994-bb88-ed7a0f022767",
+"name": "Get Initial Query",
+"type": "n8n-nodes-base.set",
+"position": [
+-360,
+-540
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "14b77741-c3c3-4bd2-be6e-37bd09fcea2b",
+"name": "query",
+"type": "string",
+"value": "=Initial query: {{ $('Set Variables').first().json.prompt }}\nFollow-up Questions and Answers:\n{{\n$input.all()\n .map(item => {\n const q = Object.keys(item.json)[0];\n const a = item.json[q];\n return `question: ${q}\\nanswer: ${a}`;\n })\n .join('\\n')\n}}"
+}
+]
+}
+},
+"executeOnce": true,
+"typeVersion": 3.4
+},
+{
+"id": "26d26e54-ee9b-4714-ae27-4f033dc825d3",
+"name": "Structured Output Parser4",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+160,
+-280
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\":\" A short title summarising the research topic\"\n },\n \"description\": {\n \"type\": \"string\",\n \"description\": \"A short description to summarise the research topic\"\n }\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "3842bc1d-d5f9-4879-bc06-db20fed3f55d",
+"name": "Create Row",
+"type": "n8n-nodes-base.notion",
+"position": [
+300,
+-420
+],
+"parameters": {
+"title": "={{ $json.output.title }}",
+"options": {},
+"resource": "databasePage",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Description|rich_text",
+"textContent": "={{ $json.output.description }}"
+},
+{
+"key": "Status|status",
+"statusValue": "Not started"
+},
+{
+"key": "Request ID|rich_text",
+"textContent": "={{ $('Set Variables').first().json.request_id }}"
+},
+{
+"key": "Name|title",
+"title": "={{ $json.output.title }}"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "bfe98996-6ed5-4f60-afdd-a947a6fa6e36",
+"name": "Report Page Generator",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-20,
+-420
+],
+"parameters": {
+"text": "=Create a suitable title for the research report which will be created from the user's query.\n{{ $json.query }}",
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.5
+},
+{
+"id": "ff05add8-94b0-4495-8f4e-3e8a10c556af",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-120,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 600,
+"height": 560,
+"content": "## 3. Create Empty Report Page in Notion\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)\n\nSome thought was given where to upload the final report and Notion was selected due to familiarity. This can be easily changed to whatever wiki tools you prefer.\n\nIf you're following along however, here's the Notion database you need to replicate - [Jim's n8n DeepResearcher Database](https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf)."
+},
+"typeVersion": 1
+},
+{
+"id": "5bc13d62-81e1-4730-b7e6-9e5579dff174",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+500,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 640,
+"height": 560,
+"content": "## 4. Trigger DeepResearch Asynchronously\n[Learn more about the Execute Trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow/)\n\nn8n handles asynchronous jobs by spinning them off as separate executions. This basically means the user doesn't have to wait or keep their browser window open for our researcher to do its job.\n\nOnce we initiate the Deepresearcher job, we can close out the onboarding journey for a nice user experience."
+},
+"typeVersion": 1
+},
+{
+"id": "9fea6403-b2a2-4e67-99a2-b7a2f29a1e96",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1160,
+620
+],
+"parameters": {
+"color": 7,
+"width": 620,
+"height": 540,
+"content": "## 7. Generate Search Queries\n[Learn more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nMuch like a human researcher, the DeepResearcher will rely on web search and content as the preferred source of information. To ensure it can cover a wide range of sources, the AI can first generate relevant research queries of which each can be explored separately."
+},
+"typeVersion": 1
+},
+{
+"id": "0bccdc54-7570-4bca-93ec-cb140c5bd3a1",
+"name": "Is Depth Reached?",
+"type": "n8n-nodes-base.if",
+"position": [
+-40,
+180
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "75d18d88-6ba6-43df-bef7-3e8ad99ad8bd",
+"operator": {
+"type": "boolean",
+"operation": "true",
+"singleValue": true
+},
+"leftValue": "={{ $json.should_stop }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "819aa5be-b71b-44a7-b062-b2a50209f290",
+"name": "URLs to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+400,
+980
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "results"
+},
+"typeVersion": 1
+},
+{
+"id": "4b8e9936-4b24-4bd4-8fe7-75d58244cb6d",
+"name": "Get Research Results",
+"type": "n8n-nodes-base.set",
+"position": [
+160,
+180
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "90b3da00-dcd5-4289-bd45-953146a3b0ba",
+"name": "all_learnings",
+"type": "array",
+"value": "={{ $json.all_learnings }}"
+},
+{
+"id": "623dbb3d-83a1-44a9-8ad3-48d92bc42811",
+"name": "all_urls",
+"type": "array",
+"value": "={{ $json.all_urls }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "d371535a-2946-4ec5-9be6-2ee8e359ac44",
+"name": "Get Existing Row",
+"type": "n8n-nodes-base.notion",
+"position": [
+-1040,
+180
+],
+"parameters": {
+"limit": 1,
+"filters": {
+"conditions": [
+{
+"key": "Request ID|rich_text",
+"condition": "equals",
+"richTextValue": "={{ $json.requestId.toString() }}"
+}
+]
+},
+"options": {},
+"resource": "databasePage",
+"matchType": "allFilters",
+"operation": "getAll",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"filterType": "manual"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "fea4c30e-1193-494d-8823-dfbec5196a0d",
+"name": "Set In-Progress",
+"type": "n8n-nodes-base.notion",
+"position": [
+-840,
+180
+],
+"parameters": {
+"pageId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $json.id }}"
+},
+"options": {},
+"resource": "databasePage",
+"operation": "update",
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Status|status",
+"statusValue": "In progress"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "37954acd-d8cb-4c74-afa8-d8973e017327",
+"name": "Set Done",
+"type": "n8n-nodes-base.notion",
+"position": [
+1680,
+1600
+],
+"parameters": {
+"pageId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Get Existing Row1').first().json.id }}"
+},
+"options": {},
+"resource": "databasePage",
+"operation": "update",
+"propertiesUi": {
+"propertyValues": [
+{
+"key": "Status|status",
+"statusValue": "Done"
+},
+{
+"key": "Last Updated|date",
+"date": "={{ $now.toISO() }}"
+}
+]
+}
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"executeOnce": true,
+"typeVersion": 2.2
+},
+{
+"id": "3db97ab8-b934-4567-a92e-92374a363df6",
+"name": "Tags to Items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-60,
+1600
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "tag"
+},
+"typeVersion": 1
+},
+{
+"id": "7f468bf7-762a-4818-86f9-54d172bb618a",
+"name": "Convert to HTML",
+"type": "n8n-nodes-base.markdown",
+"position": [
+-380,
+1600
+],
+"parameters": {
+"mode": "markdownToHtml",
+"options": {
+"tables": true
+},
+"markdown": "={{ $json.text }}"
+},
+"typeVersion": 1
+},
+{
+"id": "97914ee9-0ee8-408b-b2bb-a7193b2d0454",
+"name": "HTML to Array",
+"type": "n8n-nodes-base.set",
+"position": [
+-220,
+1600
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "851b8a3f-c2d3-41ad-bf60-4e0e667f6c58",
+"name": "tag",
+"type": "array",
+"value": "={{ $json.data.match(/
|
|<[^>]+>[^<]*<\\/[^>]+>/g) }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "6ce79f16-51e3-4192-8103-738222be558b",
+"name": "Notion Block Generator",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+100,
+1600
+],
+"parameters": {
+"text": "={{ $json.tag.trim() }}",
+"messages": {
+"messageValues": [
+{
+"message": "=Convert the following html into its equivalent Notion Block as per Notion's API schema.\n* Ensure the content is always included and remains the same.\n* Return only a json response.\n* Generate child-level blocks. Should not define \"parent\" or \"children\" property.\n* Strongly prefer headings, paragraphs, tables and lists type blocks.\n* available headings are heading_1, heading_2 and heading_3 - h4,h5,h6 should use heading_3 type instead. ensure headings use the rich text definition.\n* ensure lists blocks include all list items.\n\n## Examples\n\n1. headings\n```\n
\n```\nwould convert to\n```\n{\n \"object\": \"block\",\n \"type\": \"table\",\n \"table\": {\n \"table_width\": 2,\n \"has_column_header\": true,\n \"has_row_header\": false,\n \"children\": [\n {\n \"object\": \"block\",\n \"type\": \"table_row\",\n \"table_row\": {\n \"cells\": [\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Technology\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Potential Impact\",\n \"link\": null\n }\n }\n ],\n [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"5G Connectivity\",\n \"link\": null\n }\n },\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Enables faster data speeds and advanced apps\",\n \"link\": null\n }\n }\n ]\n ]\n }\n }\n ]\n }\n}\n```\n4. anchor links\nSince Notion doesn't support anchor links, just convert them to rich text blocks instead.\n```\nModule 0: Pre-Course Setup and Learning Principles\n```\nconverts to\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Module 0: Pre-Course Setup and Learning Principles\"\n }\n }\n ]\n }\n}\n```\n5. Invalid html parts\nWhen the html is not syntax valid eg. orphaned closing tags, then just skip the conversion and use an empty rich text block.\n```\n\\n\n```\ncan be substituted with\n```\n{\n \"object\": \"block\",\n \"type\": \"paragraph\",\n \"paragraph\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \" \"\n }\n }\n ]\n }\n}\n```"
+}
+]
+},
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "e3eeb9f0-7407-41f9-a814-def6c26b2ee1",
+"name": "Google Gemini Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+80,
+1760
+],
+"parameters": {
+"options": {},
+"modelName": "models/gemini-2.0-flash"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "dSxo6ns5wn658r8N",
+"name": "Google Gemini(PaLM) Api account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "5b0aeaca-dce5-4afd-8a8a-0ef2c18b6f06",
+"name": "Parse JSON blocks",
+"type": "n8n-nodes-base.set",
+"onError": "continueRegularOutput",
+"position": [
+420,
+1600
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "73fcb8a0-2672-4bd5-86de-8075e1e02baf",
+"name": "=block",
+"type": "array",
+"value": "={{\n(function(){\n const block = $json.text\n .replace('```json', '')\n .replace('```', '')\n .trim()\n .parseJson();\n if (Array.isArray(block)) return block;\n if (block.type.startsWith('heading_')) {\n const prev = Number(block.type.split('_')[1]);\n const next = Math.max(1, prev - 1);\n if (next !== prev) {\n block.type = `heading_${next}`;\n block[`heading_${next}`] = Object.assign({}, block[`heading_${prev}`]);\n block[`heading_${prev}`] = undefined;\n }\n }\n return [block];\n})()\n}}"
+}
+]
+}
+},
+"executeOnce": false,
+"typeVersion": 3.4
+},
+{
+"id": "e2a5a5bc-a3c8-42c5-9419-74ce3525f599",
+"name": "Upload to Notion Page",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"maxTries": 2,
+"position": [
+1680,
+1760
+],
+"parameters": {
+"url": "=https://api.notion.com/v1/blocks/{{ $('Get Existing Row1').first().json.id }}/children",
+"method": "PATCH",
+"options": {
+"timeout": "={{ 1000 * 60 }}"
+},
+"jsonBody": "={{\n{\n \"children\": $json.block\n}\n}}",
+"sendBody": true,
+"sendHeaders": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"headerParameters": {
+"parameters": [
+{
+"name": "Notion-Version",
+"value": "2022-06-28"
+}
+]
+},
+"nodeCredentialType": "notionApi"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"retryOnFail": true,
+"typeVersion": 4.2,
+"waitBetweenTries": 3000
+},
+{
+"id": "9f18b2a5-ba74-40fc-8e35-a93ecd13507a",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+620
+],
+"parameters": {
+"color": 7,
+"width": 1740,
+"height": 740,
+"content": "## 8. Web Search and Extracting Web Page Contents using [APIFY.com](https://www.apify.com?fpr=414q6)\n[Read more about the HTTP Request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nHere is where I deviated a little from the reference implementation. I opted not to use Firecrawl.ai due to (1) high cost of the service and (2) a regular non-ai crawler would work just as well and probably quicker.\nInstead I'm using [APIFY.com](https://www.apify.com?fpr=414q6) which is a more performant, cost-effective and reliable web scraper service. If you don't want to use Apify, feel free to swap this out with your preferred service.\n\nThis step is the most exciting in terms of improvements and optimisations eg. mix in internal data sources! Add in Perplexity.ai or Jina.ai! Possibilities are endless."
+},
+"typeVersion": 1
+},
+{
+"id": "84c34a2a-d8bb-4e62-a5ea-df0a142aa2b4",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+60
+],
+"parameters": {
+"color": 7,
+"width": 460,
+"height": 360,
+"content": "## 5. Set Report to In-Progress\n[Read more about the Notion node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.notion/)"
+},
+"typeVersion": 1
+},
+{
+"id": "bd022636-873d-4aca-8929-c189f8596cc1",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1240,
+700
+],
+"parameters": {
+"color": 7,
+"width": 780,
+"height": 660,
+"content": "## 9. Compile Learnings with Reasoning Model\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nWith our gathered sources, it's now just a case of giving it to our LLM to compile a list of \"learnings\" from them. For our DeepResearcher, we'll use OpenAI's o3-mini which is the latest reasoning model at time of writing. Reasoning perform better than regular chat models due their chain-of-thought or \"thinking\" process that they perform.\n\nThe \"Learnings\" are then combined with the generated research goal to complete one loop."
+},
+"typeVersion": 1
+},
+{
+"id": "b2f6e51d-cbe6-4459-9515-679f79063926",
+"name": "Get Existing Row1",
+"type": "n8n-nodes-base.notion",
+"position": [
+-1020,
+1600
+],
+"parameters": {
+"limit": 1,
+"filters": {
+"conditions": [
+{
+"key": "Request ID|rich_text",
+"condition": "equals",
+"richTextValue": "={{ $json.requestId.toString() }}"
+}
+]
+},
+"options": {},
+"resource": "databasePage",
+"matchType": "allFilters",
+"operation": "getAll",
+"databaseId": {
+"__rl": true,
+"mode": "list",
+"value": "19486dd6-0c0c-80da-9cb7-eb1468ea9afd",
+"cachedResultUrl": "https://www.notion.so/19486dd60c0c80da9cb7eb1468ea9afd",
+"cachedResultName": "n8n DeepResearch"
+},
+"filterType": "manual"
+},
+"credentials": {
+"notionApi": {
+"id": "iHBHe7ypzz4mZExM",
+"name": "Notion account"
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "7b315060-7e40-410c-ac9d-ef22acbb175a",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1140,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 660,
+"height": 540,
+"content": "## 10. Generate DeepSearch Report using Learnings\n[Read more about the Basic LLM node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nFinally! After all learnings have been gathered - which may have taken up to an hour or more on the higher settings! - they are given to our LLM to generate the final research report in markdown format. Technically, the DeepResearch ends here but for this template, we need to push the output to Notion. If you're not using Notion, feel free to ignore the last few steps."
+},
+"typeVersion": 1
+},
+{
+"id": "01aaf2a8-c145-4f39-aa52-f40fc28f8767",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-460,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 1060,
+"height": 540,
+"content": "## 11. Reformat Report as Notion Blocks\n[Learn more about the Markdown node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.markdown/)\n\nTo write our report to our Notion page, we'll have to convert it to Notion \"blocks\" - these are specialised json objects which are required by the Notion API. There are quite a number of ways to do this conversion not involving the use of AI but for kicks, I decided to do so anyway. In this step, we first convert to HTML as it allows us to split the report semantically and makes for easier parsing for the LLM."
+},
+"typeVersion": 1
+},
+{
+"id": "700a6f44-86bf-4aab-8a42-23bf6843f681",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1220,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 800,
+"height": 580,
+"content": "## 13. Update Report in Notion\n[Read more about the HTTP request node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/)\n\nIn this step, we can use the Notion API to add the blocks to our page sequentially. A loop is used due to the unstable Notion API - the loop allows retries for blocks that require it."
+},
+"typeVersion": 1
+},
+{
+"id": "f8536052-c851-42ec-aaf3-5fc876570f6d",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1840,
+-680
+],
+"parameters": {
+"color": 7,
+"width": 680,
+"height": 560,
+"content": "## 1. Let's Research!\n[Learn more about the form trigger node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.formtrigger)\n\nn8n forms are a really nice way to get our frontend up and running quickly and compared to chat, offers a superior user interface for user input. I've gone perhaps a little extra with the custom html fields but I do enjoy adding a little customisation now and then."
+},
+"typeVersion": 1
+},
+{
+"id": "58aa92a2-e1fd-497d-a27e-40b733189bab",
+"name": "DeepResearch Report",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-860,
+1600
+],
+"parameters": {
+"text": "=You are are an expert and insightful researcher.\n* Given the following prompt from the user, write a final report on the topic using the learnings from research.\n* Make it as as detailed as possible, aim for 3 or more pages, include ALL the learnings from research.\n* Format the report in markdown. Use headings, lists and tables only and where appropriate.\n\n{{ $('JobType Router').first().json.data.query }}\n\nHere are all the learnings from previous research:\n\n\n{{\n$('JobType Router').first().json.data\n .all_learnings\n .map(item => `${item}`) \n .join('\\n')\n}}\n",
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "0656be83-d510-46f1-aeeb-f62a69aa3cf2",
+"name": "DeepResearch Learnings",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+1500,
+980
+],
+"parameters": {
+"text": "=Given the following contents from a SERP search for the query {{ $('Item Ref').first().json.query }}, generate a list of learnings from the contents. Return a maximum of 3 learnings, but feel free to return less if the contents are clear. Make sure each learning is unique and not similar to each other. The learnings should be concise and to the point, as detailed and infromation dense as possible. Make sure to include any entities like people, places, companies, products, things, etc in the learnings, as well as any exact metrics, numbers, or dates. The learnings will be used to research the topic further.\n\n\n{{\n$('Convert to Markdown')\n .all()\n .map(item =>`\\n${item.json.markdown.substr(0, 25_000)}\\n`)\n .join('\\n')\n}}\n",
+"messages": {
+"messageValues": [
+{
+"type": "HumanMessagePromptTemplate",
+"message": "=You are an expert researcher. Today is {{ $now.toLocaleString() }}. Follow these instructions when responding:\n - You may be asked to research subjects that is after your knowledge cutoff, assume the user is right when presented with news.\n - The user is a highly experienced analyst, no need to simplify it, be as detailed as possible and make sure your response is correct.\n - Be highly organized.\n - Suggest solutions that I didn't think about.\n - Be proactive and anticipate my needs.\n - Treat me as an expert in all subject matter.\n - Mistakes erode my trust, so be accurate and thorough.\n - Provide detailed explanations, I'm comfortable with lots of detail.\n - Value good arguments over authorities, the source is irrelevant.\n - Consider new technologies and contrarian ideas, not just the conventional wisdom.\n - You may use high levels of speculation or prediction, just flag it for me."
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"executeOnce": true,
+"typeVersion": 1.5
+},
+{
+"id": "9296a787-3226-44fe-8118-f84dda8e5167",
+"name": "Generate Report",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+480,
+180
+],
+"parameters": {
+"options": {
+"waitForSubWorkflow": false
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{\n{\n ...Object.assign({}, $json),\n query: $('JobType Router').first().json.data.query\n}\n}}",
+"jobType": "deepresearch_report",
+"requestId": "={{ $('JobType Router').first().json.requestId }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "471f9b9f-f331-4652-95de-1ec7136ea692",
+"name": "Generate Learnings",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+-380,
+180
+],
+"parameters": {
+"mode": "each",
+"options": {
+"waitForSubWorkflow": true
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+},
+"workflowInputs": {
+"value": {
+"data": "={{ $json }}",
+"jobType": "deepresearch_learnings",
+"requestId": "={{ $('JobType Router').first().json.requestId }}"
+},
+"schema": [
+{
+"id": "requestId",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "requestId",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "jobType",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "jobType",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "data",
+"type": "object",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [],
+"attemptToConvertTypes": false,
+"convertFieldsToString": true
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "2e2fa1e6-9d5c-46ff-985c-58ada1139837",
+"name": "Confirmation",
+"type": "n8n-nodes-base.form",
+"position": [
+780,
+-420
+],
+"webhookId": "cf41a176-5d30-4274-955e-b0d5b483d37f",
+"parameters": {
+"options": {
+"formTitle": "DeepResearcher",
+"buttonLabel": "Done",
+"formDescription": "=\n
\nYour Report Is On Its Way!\n \nDeepResearcher will now work independently to conduct the research and the compiled report will be uploaded to the following Notion page below when finished.\n
\nPlease click the \"Done\" button to complete the form.\n
\n This value determines how many sources to explore.\n
\n \n \n
\n\n",
+"fieldType": "html",
+"fieldLabel": "Enter research breadth (recommended 2-10, default 4)"
+},
+{
+"fieldType": "dropdown",
+"fieldLabel": "={{ \"\" }}",
+"multiselect": true,
+"fieldOptions": {
+"values": [
+{
+"option": "=I understand higher depth and breath values I've selected may incur longer wait times and higher costs. I acknowledging this and wish to proceed with the research request."
+}
+]
+},
+"requiredField": true
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b3d11997-9c8c-4b72-b750-4fc22a2247b7",
+"name": "Valid Blocks",
+"type": "n8n-nodes-base.filter",
+"position": [
+740,
+1600
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f68cefe0-e109-4d41-9aa3-043f3bc6c449",
+"operator": {
+"type": "string",
+"operation": "notExists",
+"singleValue": true
+},
+"leftValue": "={{ $json.error }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "ff90a1c1-b357-4012-8964-e007bef0c9db",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+620,
+1400
+],
+"parameters": {
+"color": 7,
+"width": 580,
+"height": 580,
+"content": "## 12. Append URL Sources List\n[Read more about the Code node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nFor our source URLs, we'll manually compose the Notion blocks for them - this is because there's usually a lot of them! We'll then append to the end of the other blocks."
+},
+"typeVersion": 1
+},
+{
+"id": "f6b50f06-6122-494c-bdb8-4215f473a27d",
+"name": "Append Blocks",
+"type": "n8n-nodes-base.merge",
+"position": [
+1000,
+1760
+],
+"parameters": {},
+"typeVersion": 3
+},
+{
+"id": "ca39272a-828e-4314-80da-05dd5fd7b2e3",
+"name": "URL Sources to Lists",
+"type": "n8n-nodes-base.code",
+"position": [
+740,
+1760
+],
+"parameters": {
+"jsCode": "const urls = $('JobType Router').first().json.data.all_urls;\nconst chunksize = 50;\nconst splits = Math.max(1, Math.floor(urls.length/chunksize));\n\nconst blocks = Array(splits).fill(0)\n .map((_, idx) => {\n const block = urls\n .slice(\n idx * chunksize, \n (idx * chunksize) + chunksize - 1\n )\n .map(url => {\n return {\n object: \"block\",\n type: \"bulleted_list_item\",\n bulleted_list_item: {\n rich_text: [\n { type: \"text\", text: { content: url } }\n ]\n }\n }\n });\n return { json: { block } }\n });\n\nreturn [\n { json: {\n block:[{\n \"object\": \"block\",\n \"type\": \"heading_2\",\n \"heading_2\": {\n \"rich_text\": [\n {\n \"type\": \"text\",\n \"text\": {\n \"content\": \"Sources\"\n }\n }\n ]\n }\n }]\n } },\n ...blocks\n];"
+},
+"typeVersion": 2
+},
+{
+"id": "e2e2b07a-4039-4859-b60c-f51982475282",
+"name": "Has Results?",
+"type": "n8n-nodes-base.if",
+"position": [
+240,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "9ef8d40c-1289-4654-9022-4a07f7102555",
+"operator": {
+"type": "array",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json.results }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "5662dbf2-8877-4e83-982c-6bc5968b8835",
+"name": "Empty Response",
+"type": "n8n-nodes-base.set",
+"position": [
+1040,
+1120
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "1de40158-338b-4db3-9e22-6fd63b21f825",
+"name": "ResearchGoal",
+"type": "string",
+"value": "={{ $('Item Ref').first().json.researchGoal }}"
+},
+{
+"id": "9f59a2d4-5e5a-4d0b-8adf-2832ce746f0f",
+"name": "learnings",
+"type": "array",
+"value": "={{ [] }}"
+},
+{
+"id": "972ab5f5-0537-4755-afcb-d1db4f09ad60",
+"name": "followUpQuestions",
+"type": "array",
+"value": "={{ [] }}"
+},
+{
+"id": "90cef471-76b0-465d-91a4-a0e256335cd3",
+"name": "urls",
+"type": "array",
+"value": "={{ [] }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "678b6af5-da74-4421-8c3d-0166aa52efd9",
+"name": "Has Content?",
+"type": "n8n-nodes-base.if",
+"position": [
+880,
+980
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "1ef1039a-4792-47f9-860b-d2ffcffd7129",
+"operator": {
+"type": "object",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "233f2e19-b4f2-4de3-8002-f79f3c01c1e7",
+"name": "Sticky Note13",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1820,
+-240
+],
+"parameters": {
+"color": 5,
+"width": 300,
+"height": 100,
+"content": "### Not using forms?\nFeel free ot swap this out for chat or even webhooks to fit your existing workflows."
+},
+"typeVersion": 1
+},
+{
+"id": "453bb6eb-f2b0-4e21-b647-e095c80b7844",
+"name": "Sticky Note14",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1880,
+540
+],
+"parameters": {
+"color": 5,
+"width": 460,
+"height": 240,
+"content": "### ๐ The Subworkflow Event Pattern \nIf you're new to n8n, this advanced technique might need some explaining but in gist, we're using subworkflows to run different parts of our DeepResearcher workflow as separate executions.\n\n* Necessary to implement the recursive loop mechanism needed to enable this workflow.\n* Negates the need to split this workflow into multiple templates.\n* Great generally for building high performance n8n workflows (a topic for a future post!)"
+},
+"typeVersion": 1
+},
+{
+"id": "289cbe4c-c2e3-46b7-8799-197a7d78ab2a",
+"name": "Sticky Note15",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+720,
+-60
+],
+"parameters": {
+"color": 5,
+"width": 340,
+"height": 200,
+"content": "### Recursive Looping\nThe recursive looping implemented for this workflow is an advanced item-linking technique. It works by specifically controlling which nodes \"execute once\" vs\" execute for each item\" because of this becareful of ermoving nodes! Always check the settings of the node you're replacing and ensure the settings match. "
+},
+"typeVersion": 1
+},
+{
+"id": "b95ffdcd-c0d1-4a12-a7a9-24135db7b467",
+"name": "Combine & Send back to Loop",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+-240,
+820
+],
+"parameters": {
+"options": {},
+"aggregate": "aggregateAllItemData"
+},
+"typeVersion": 1
+},
+{
+"id": "f0a48ab5-70b9-49dc-a153-61e573803d1e",
+"name": "For Each Block...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+1440,
+1600
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "2fd17fbd-005d-446e-b014-0da190cd3114",
+"name": "Sticky Note16",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2420,
+-920
+],
+"parameters": {
+"width": 520,
+"height": 1060,
+"content": "## n8n DeepResearcher\n### This template attempts to replicate OpenAI's DeepResearch feature which, at time of writing, is only available to their pro subscribers.\n\nThough the inner workings of DeepResearch have not been made public, it is presumed the feature relies on the ability to deep search the web, scrape web content and invoking reasoning models to generate reports. All of which n8n is really good at!\n\n### How it works\n* A form is used to first capture the user's research query and how deep they'd like the researcher to go.\n* Once submitted, a blank Notion page is created which will later hold the final report and the researcher gets to work.\n* The user's query goes through a recursive series of web serches and web scraping to collect data on the research topic to generate partial learnings.\n* Once complete, all learnings are combined and given to a reasoning LLM to generate the final report.\n* The report is then written to the placeholder Notion page created earlier. \n\n### How to use\n* Duplicate this Notion database to use with this template: https://jimleuk.notion.site/19486dd60c0c80da9cb7eb1468ea9afd?v=19486dd60c0c805c8e0c000ce8c87acf\n* Sign-up for [APIFY.com](https://www.apify.com?fpr=414q6) API Key for web search and scraping services.\n* Ensure you have access to OpenAI's o3-mini model. Alternatively, switch this out for o1 series.\n* You must publish this workflow and ensure the form url is publically accessible.\n\n### On Depth & Breadth Configuration\nFor more detailed reports, increase depth and breadth but be warned the workflow will take a exponentially more time and money to complete. The defaults are usually good enough.\n\nDepth=1 & Breadth=2 - will take about 10 - 15mins.\nDepth=1 & Breadth=4 - will take about 30 - 40mins.\nDpeth=3 & Breadth=5 - will take about 2 - 5 hours!\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "ac6f2604-7439-4524-a27e-2f031ebce089",
+"name": "Sticky Note17",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2420,
+-1180
+],
+"parameters": {
+"color": 7,
+"width": 520,
+"height": 240,
+"content": ""
+},
+"typeVersion": 1
+},
+{
+"id": "40e3a0cf-7710-4537-b147-37ba8945fdbc",
+"name": "Sticky Note18",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-120,
+960
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
+},
+"typeVersion": 1
+},
+{
+"id": "5a96ecc2-eeea-4c33-b299-7a7f2ca7559c",
+"name": "Sticky Note19",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+520,
+960
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE APIFY CREDENTIAL HERE!"
+},
+"typeVersion": 1
+},
+{
+"id": "23bca6e2-e16a-48a4-a7fc-96ce25846764",
+"name": "Sticky Note20",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1640,
+1740
+],
+"parameters": {
+"width": 180,
+"height": 260,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n### UPDATE NOTION CREDENTIAL HERE!"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Item Ref": {
+"main": [
+[
+{
+"node": "Web Search",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create Row": {
+"main": [
+[
+{
+"node": "Initiate DeepResearch",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Web Search": {
+"main": [
+[
+{
+"node": "Top 5 Organic Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Valid Pages": {
+"main": [
+[
+{
+"node": "Has Content?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Confirmation": {
+"main": [
+[
+{
+"node": "End Form",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Content?": {
+"main": [
+[
+{
+"node": "Convert to Markdown",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Empty Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Results?": {
+"main": [
+[
+{
+"node": "URLs to Items",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Empty Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Valid Blocks": {
+"main": [
+[
+{
+"node": "Append Blocks",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Append Blocks": {
+"main": [
+[
+{
+"node": "For Each Block...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"HTML to Array": {
+"main": [
+[
+{
+"node": "Tags to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Page Contents": {
+"main": [
+[
+{
+"node": "Valid Pages",
+"type": "main",
+"index": 0
+}
+],
+[]
+]
+},
+"SERP to Items": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Variables": {
+"main": [
+[
+{
+"node": "Clarifying Questions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Tags to Items": {
+"main": [
+[
+{
+"node": "Notion Block Generator",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"URLs to Items": {
+"main": [
+[
+{
+"node": "Page Contents",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Empty Response": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execution Data": {
+"main": [
+[
+{
+"node": "JobType Router",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"JobType Router": {
+"main": [
+[
+{
+"node": "Get Existing Row",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Generate SERP Queries",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Get Existing Row1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert to HTML": {
+"main": [
+[
+{
+"node": "HTML to Array",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set In-Progress": {
+"main": [
+[
+{
+"node": "Set Initial Query",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Existing Row": {
+"main": [
+[
+{
+"node": "Set In-Progress",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Research Request": {
+"main": [
+[
+{
+"node": "Set Variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Results to Items": {
+"main": [
+[
+{
+"node": "Set Next Queries",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Next Queries": {
+"main": [
+[
+{
+"node": "Generate Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Feedback to Items": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Block...": {
+"main": [
+[
+{
+"node": "Set Done",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Upload to Notion Page",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Query...": {
+"main": [
+[
+{
+"node": "Combine & Send back to Loop",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Item Ref",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Existing Row1": {
+"main": [
+[
+{
+"node": "DeepResearch Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Initial Query": {
+"main": [
+[
+{
+"node": "Report Page Generator",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Is Depth Reached?": {
+"main": [
+[
+{
+"node": "Get Research Results",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "DeepResearch Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Parse JSON blocks": {
+"main": [
+[
+{
+"node": "Valid Blocks",
+"type": "main",
+"index": 0
+},
+{
+"node": "URL Sources to Lists",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Initial Query": {
+"main": [
+[
+{
+"node": "Generate Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Accumulate Results": {
+"main": [
+[
+{
+"node": "Is Depth Reached?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate Learnings": {
+"main": [
+[
+{
+"node": "Accumulate Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"On form submission": {
+"main": [
+[
+{
+"node": "Research Request",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "DeepResearch Report",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model2": {
+"ai_languageModel": [
+[
+{
+"node": "Clarifying Questions",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model3": {
+"ai_languageModel": [
+[
+{
+"node": "Generate SERP Queries",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model4": {
+"ai_languageModel": [
+[
+{
+"node": "Report Page Generator",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Convert to Markdown": {
+"main": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Report": {
+"main": [
+[
+{
+"node": "Convert to HTML",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Clarifying Questions": {
+"main": [
+[
+{
+"node": "Feedback to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Results": {
+"main": [
+[
+{
+"node": "Results to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Question...": {
+"main": [
+[
+{
+"node": "Get Initial Query",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Ask Clarity Questions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Research Results": {
+"main": [
+[
+{
+"node": "Generate Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"URL Sources to Lists": {
+"main": [
+[
+{
+"node": "Append Blocks",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Ask Clarity Questions": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate SERP Queries": {
+"main": [
+[
+{
+"node": "SERP to Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Initiate DeepResearch": {
+"main": [
+[
+{
+"node": "Confirmation",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Report Page Generator": {
+"main": [
+[
+{
+"node": "Create Row",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Top 5 Organic Results": {
+"main": [
+[
+{
+"node": "Has Results?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Upload to Notion Page": {
+"main": [
+[
+{
+"node": "For Each Block...",
+"type": "main",
+"index": 0
+}
+],
+[]
+]
+},
+"DeepResearch Learnings": {
+"main": [
+[
+{
+"node": "Research Goal + Learnings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Notion Block Generator": {
+"main": [
+[
+{
+"node": "Parse JSON blocks",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepResearch Subworkflow": {
+"main": [
+[
+{
+"node": "Execution Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Gemini Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Notion Block Generator",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "DeepResearch Learnings",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Research Goal + Learnings": {
+"main": [
+[
+{
+"node": "For Each Query...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser1": {
+"ai_outputParser": [
+[
+{
+"node": "Clarifying Questions",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser2": {
+"ai_outputParser": [
+[
+{
+"node": "Generate SERP Queries",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser4": {
+"ai_outputParser": [
+[
+{
+"node": "Report Page Generator",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.json b/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.json
deleted file mode 100644
index 96f73e9..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.json
+++ /dev/null
@@ -1,576 +0,0 @@
-{
- "id": "wa2uEnSIowqSrHoY",
- "meta": {
- "instanceId": "cca06617664f52c5a019ea575691fdbce675dd95dc0452af5f13dbe76d615b69"
- },
- "name": "Intelligent Web Query and Semantic Re-Ranking Flow",
- "tags": [],
- "nodes": [
- {
- "id": "8e7dc5cb-6822-4ef6-9e5a-2b350a1526bf",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -640,
- -620
- ],
- "parameters": {
- "color": 5,
- "width": 1172,
- "height": 970,
- "content": "\n## Step 1. Set Up a Free Brave Web Search Query API Key\n\nTo attain the free web search API tier from Brave, follow these steps:\n\n1. Visit api.search.brave.com\n2. Create an account\n3. Subscribe to the free plan (no charge)\n4. Navigate to the API Keys section\n5. Generate an API key. For the subscription type, choose \"Free\".\n6. Go to the \"Query\" Nodes and change the \"X-Subscription-Token\" value to your API Key.\n"
- },
- "typeVersion": 1
- },
- {
- "id": "5bb3e68f-7693-4d4b-b794-843f2c3535e0",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1580,
- -420
- ],
- "parameters": {
- "color": 4,
- "width": 680,
- "height": 360,
- "content": "## If you require to change this Node to Webhook Or any Other Item:\n\n- In case you want to change the input type from Webhook to any other item, Make sure to go to the Query 1 and Query 1 Ranker and replace the Webhook Input to your Node's input."
- },
- "typeVersion": 1
- },
- {
- "id": "f2fc02f9-a78a-4e87-be85-0032492a9f3f",
- "name": "Date & Time",
- "type": "n8n-nodes-base.dateTime",
- "position": [
- -820,
- -240
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 2
- },
- {
- "id": "6f18ebbd-83db-4900-bc2e-0a9f23d6e8c8",
- "name": "Webhook",
- "type": "n8n-nodes-base.webhook",
- "position": [
- -1340,
- -240
- ],
- "webhookId": "962f1468-c80f-4c0c-8555-a0acf648ede4",
- "parameters": {
- "path": "962f1468-c80f-4c0c-8555-a0acf648ede4",
- "options": {},
- "responseMode": "responseNode"
- },
- "typeVersion": 2
- },
- {
- "id": "ba5ea83e-1b47-475b-863f-269ae293729a",
- "name": "Auto-fixing Output Parser6",
- "type": "@n8n/n8n-nodes-langchain.outputParserAutofixing",
- "position": [
- 180,
- -140
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "ca426b6d-5412-4c5b-a55c-009a47c59a81",
- "name": "Auto-fixing Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserAutofixing",
- "position": [
- -580,
- -140
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "501d5390-5317-4973-a3e9-b0f502399c2b",
- "name": "Structured Output Parser1",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- -460,
- -60
- ],
- "parameters": {
- "jsonSchemaExample": "{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "a27e75c7-0307-4d71-9266-5a56b297a6e3",
- "name": "Query-1 Combined",
- "type": "n8n-nodes-base.code",
- "position": [
- -80,
- -240
- ],
- "parameters": {
- "jsCode": "// Initialize an empty string to store all title, url, and description pairs\nlet aggregatedOutputText = \"\";\n\n// Loop through all items passed to this Function node\nfor (let item of items) {\n // Access the JSON data from \"Query 1\" node for the current item\n const queryData = item.json;\n\n // Ensure there is a \"web.results\" array to process\n if (queryData.web?.results && Array.isArray(queryData.web.results)) {\n // Loop through all results in the \"web.results\" array\n for (let result of queryData.web.results) {\n // Extract the title, url, and description for each result\n const title = result.title || \"No Title\";\n const url = result.url || \"No URL\";\n const description = result.description || \"No Description\";\n\n // Append the values to the aggregated string\n aggregatedOutputText += `Title: ${title}\\nURL: ${url}\\nDescription: ${description}\\n\\n`;\n }\n } else {\n // If no results array, handle gracefully\n aggregatedOutputText += \"No results found for this item.\\n\\n\";\n }\n}\n\n// Trim the final string to remove any trailing newline and whitespace\naggregatedOutputText = aggregatedOutputText.trim();\n\n// Return a single item containing the aggregated output as a string\nreturn [\n {\n json: {\n aggregated_text: aggregatedOutputText\n }\n }\n];\n"
- },
- "typeVersion": 2
- },
- {
- "id": "acbdbe94-b5a7-4ec9-9fc8-c3ab147f42fa",
- "name": "Respond to Webhook",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 640,
- -240
- ],
- "parameters": {
- "options": {},
- "respondWith": "text",
- "responseBody": "={\n \"Highest_RANKEDURL_1\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['description'] }}\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['description'] }}\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['description'] }}\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['description'] }}\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['description'] }}\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['description'] }}\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['description'] }}\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['description'] }}\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['description'] }}\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['description'] }}\"\n },\n \"Information_extracted\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Information_extracted'] }}\"\n}\n"
- },
- "typeVersion": 1.1
- },
- {
- "id": "b8b6ae73-586a-406f-9641-57e2625f800c",
- "name": "Semantic Search - Result Re-Ranker",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "onError": "continueRegularOutput",
- "position": [
- 100,
- -240
- ],
- "parameters": {
- "text": "=\n**Objective:**\n\nFor the user's query, web search results are provided. Your tasks are:\n\n1. **Rank the links** based on how well they match the user's query.\n2. **Extract relevant information** from the descriptions provided. If no relevant information is found, return \"N/A\".\n\n---\n\n**Task:**\n\n**Step 1: Understand the User's Intent**\n\n- Determine what the user is truly and technically looking for.\n- The user's request query is: \"{{ $('Webhook').item.json.query['Research Question'] }}\"\n- The serach results below, however their performance seem, have been based on this query \"{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}\". If the result are not satisfactory or missing due to bad query making, you should note that as well for the neww query making.\n- To nesure being time aware , realize todays date is: \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n\n- Follow a three-step chain of thought to comprehend the user's needs. Think out loud.\n\n---\n\n**Step 2: Rank the Links**\n\n- From the URLs and description snippets provided, **rank the top 10 websites** that are most likely to contain the required information.\n- Use the titles, descriptions, and sources to inform your ranking.\n\n**Links, Titles, and Descriptions:**\n\n{{ $json.aggregated_text }}\n\n---\n\nThis list completes the structure up to 20 results as you requested. Let me know if thereโs anything more you need!\n\n---\n\n**Step 3: Analyze and Create a Follow-up Query**\n\n- Recognize that for the user's request:\n\n `\"{{ $('Webhook').item.json.query['Research Question'] }}\"`\n\n The results provided are based on the assistant's generated search query:\n\n `\"{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}\"`\n\n- Analyze and revise any issues or new insights through multi-step thinking to create a follow-up query.\n\n**Indications and Priorities:**\n\n1. **No Results Received:** If no search items are shared, the search query may have been ineffective (e.g., too specific, incorrect parameters).\n2. **Insufficient or Unpromising Results:** If fewer than 20 but more than 5 results are provided, and none seem promising, the search query may need refinement.\n3. **Successful Results with Potential Follow-up:** If none of the above issues occurred and the search results provide answers or suggest a follow-up, create a new query. This could be a new topic, a deep dive, or a parallel factor that offers additional benefits.\n\n- Provide your chain of thought that connects the user's request to the actual information.\n\n- Deliver precise, detailed, and value-oriented information relevant to the user's query.\n\n**Step 4: Query making notes and examples**: \n\nThe queries must not be long tails , as they result in 0 websearch reutrns. We give you some examples of good web search queries:\nExamples:\n\nUser Question: \"What is the current state of the U.S. economy in 2024?\"\nEffective Search Query: \"U.S. Economy Analysis Report 2024\"\n\nUser Question: \"What are the recent advancements in artificial intelligence?\"\nEffective Search Query: \"2024 Artificial Intelligence Developments\"\n\nUser Question: \"How is climate change affecting agriculture globally?\"\nEffective Search Query: \"Global Impact of Climate Change on Agriculture 2024\"\n\nUser Question: \"What are the latest trends in cybersecurity threats?\"\nEffective Search Query: \"Cybersecurity Threats and Trends 2024\"\n\nUser Question: \"What is the outlook for renewable energy investments?\"\nEffective Search Query: \"Renewable Energy Investment Outlook 2024\"\n\n**Step 5: Query making*: \nor query making remember as we said:\n - **Today's Date:** \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n **Search Inquiry:** \n - **Search Topic to create the query upon it:**{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}\"\"\n\n\n---\n\n**Step 6: Output Format**\n\nEnsure the response is in the following JSON format:\n\n{\n \"chain_of_thought\": \"Insert your step-by-step reasoning here.\",\n \"Highest_RANKEDURL_1\": {\n \"title\": \"Insert the First Ranked URL's Title here.\",\n \"link\": \"Insert the First Ranked URL here.\",\n \"description\": \"Insert the First Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"Insert the Second Ranked URL's Title here.\",\n \"link\": \"Insert the Second Ranked URL here.\",\n \"description\": \"Insert the Second Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"Insert the Third Ranked URL's Title here.\",\n \"link\": \"Insert the Third Ranked URL here.\",\n \"description\": \"Insert the Third Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"Insert the Fourth Ranked URL's Title here.\",\n \"link\": \"Insert the Fourth Ranked URL here.\",\n \"description\": \"Insert the Fourth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"Insert the Fifth Ranked URL's Title here.\",\n \"link\": \"Insert the Fifth Ranked URL here.\",\n \"description\": \"Insert the Fifth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"Insert the Sixth Ranked URL's Title here.\",\n \"link\": \"Insert the Sixth Ranked URL here.\",\n \"description\": \"Insert the Sixth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"Insert the Seventh Ranked URL's Title here.\",\n \"link\": \"Insert the Seventh Ranked URL here.\",\n \"description\": \"Insert the Seventh Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"Insert the Eighth Ranked URL's Title here.\",\n \"link\": \"Insert the Eighth Ranked URL here.\",\n \"description\": \"Insert the Eighth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"Insert the Ninth Ranked URL's Title here.\",\n \"link\": \"Insert the Ninth Ranked URL here.\",\n \"description\": \"Insert the Ninth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"Insert the Tenth Ranked URL's Title here.\",\n \"link\": \"Insert the Tenth Ranked URL here.\",\n \"description\": \"Insert the Tenth Ranked URL's Description here.\"\n },\n \"Information_extracted\": \"Insert all extracted information relevant to the user's query or 'N/A' if none.\"\n}\n",
- "messages": {
- "messageValues": [
- {
- "message": "=\nYou are an expert information retrieval and critical evaluation assistant designed to process, rank, and extract high-relevance content from web search results for complex user queries. You must provide value-oriented insights while refining searches based on relevance and context sensitivity. \n\n**Your Process and Priorities:**\n\n#### 1. **Determine the User's Technical Intent**\n - Interpret the user's core question provided as `{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}`, discerning underlying objectives and specialized needs.\n - Recognize that the search results may have been generated from a **secondary query**: `{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}`. \n - Judge the adequacy of this generated query. If it does not meet the userโs objectives, highlight the need for query refinement and prepare to adapt the approach.\n - Stay mindful of the date context, using `{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}` to assess the freshness of content or time-sensitive relevance.\n\n#### 2. **Rank Results Based on Analytical Relevance**\n - From the search results provided, **rank the top 3 URLs** that most closely align with the userโs intent and technical needs.\n - Use multi-dimensional analysis to assess how each linkโs title, description, and source match the userโs objective.\n - Prioritize results based on credibility, relevance, and their potential to add depth to the userโs inquiry.\n - Your goal is to select the highest-value links, disregarding results that offer superficial, off-topic, or outdated information.\n\n#### 3. **Extract Key Information**\n - For each of the top 3 ranked results, extract insights and details from the description snippets that directly address the userโs query.\n - If no pertinent information is available in a description, record `\"N/A\"` to indicate its lack of relevance.\n\n#### 4. **Evaluate for Potential Query Improvement**\n - Evaluate the relevance and coverage of search results:\n - If fewer than 5 relevant results are present, consider that the initial query may be too narrow, specific, or otherwise misaligned.\n - Generate a **refined query** that is adjusted to better match the userโs likely needs and produce higher-quality results.\n - Use advanced language modifications, new keyword suggestions, or rephrasing to formulate a search query that enhances alignment with the userโs goals.\n"
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "retryOnFail": true,
- "typeVersion": 1.4
- },
- {
- "id": "a1ca671d-0b0c-4717-9def-93fdb965de8d",
- "name": "Query",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -240,
- -240
- ],
- "parameters": {
- "url": "https://api.search.brave.com/res/v1/web/search",
- "options": {},
- "sendQuery": true,
- "sendHeaders": true,
- "queryParameters": {
- "parameters": [
- {
- "name": "q",
- "value": "={{ $json.output.final_search_query }}"
- }
- ]
- },
- "headerParameters": {
- "parameters": [
- {
- "name": "Accept",
- "value": "application/json"
- },
- {
- "name": "Accept-Encoding",
- "value": "gzip"
- },
- {
- "name": "X-Subscription-Token",
- "value": ""
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "d3cc4e7c-3ead-4d38-9b51-a11cd9d7faeb",
- "name": "Webhook Call",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -180,
- 1040
- ],
- "parameters": {
- "url": "https://primary-production-8aa4.up.railway.app/webhook-test/962f1468-c80f-4c0c-8555-a0acf648ede4",
- "options": {},
- "sendQuery": true,
- "queryParameters": {
- "parameters": [
- {
- "name": "Research Question",
- "value": "what is the latest news in global world in politics and economy?"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "6931404b-94d6-4b9d-9f0a-124012212eb5",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -640,
- 420
- ],
- "parameters": {
- "color": 3,
- "width": 1180,
- "height": 840,
- "content": "## Step 2. Setup the Webhook Call Node\n\n**Instructions for Setting Up the Webhook Call and Using It in Your Workflow**\n\nThis node is designed to send a **web search query** to the workflow (partly built in this chart) and return the results. Follow these steps to correctly configure and use it:\n\n1. **Locate the \"Webhook\" Node in the Workflow**:\n - Navigate to the workflow above, the first item, the \"Webhook\" node.\n - In the \"Webhook\" node, change the **Webhook URL option** from \"Test URL\" to \"Production URL.\"\n - Copy the generated **Production URL**.\n\n2. **Paste the Webhook URL in the HTTP Node**:\n - In your target workflow, locate the **HTTP Request** node.\n - Paste the copied **Production URL** into the URL field of the HTTP Request node. This connects the two workflows.\n\n3. **Send the Research Request**:\n - When sending the request to this workflow, make sure to include your web search query in the **\"Research Question\" parameter** of the HTTP Request node.\n\n4. **Move the Webhook Call Node**:\n - Move this **Webhook Call Node** into the workflow where you need the research results. Ensure that itโs correctly connected and configured to send the data to the main workflow.\n"
- },
- "typeVersion": 1
- },
- {
- "id": "01d73f91-1dd6-4b80-951c-9f944ea9d992",
- "name": "Semantic Search -Query Maker",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -560,
- -240
- ],
- "parameters": {
- "text": "=1. **Task:** `\"Your task is to develop a web search query that most effectively answers the research question given. Use meta-reasoning and multi-chain analysis to ensure a comprehensive approach.\"`\n\n2. **Structured Guidance for Chains of Thought:** \n a. **Chain 1:** Break down the research question, identifying keywords and relevant terms. \n b. **Chain 2:** Explore the context and potential sources, determining the types of results that would be most relevant. \n c. **Chain 3:** Refine the query for specificity and completeness, considering how to capture nuances of the question.\n\n3. **Final Query Generation:** Based on the insights from the three chains, generate a single, refined search query.\n\n\n4. Note, the queries must not be long tails , as they result in 0 websearch reutrns. We give you some examples of good web search queries:\nExamples:\n\nUser Question: \"What is the current state of the U.S. economy in 2024?\"\n\nEffective Search Query: \"U.S. Economy Analysis Report 2024\"\nUser Question: \"What are the recent advancements in artificial intelligence?\"\n\nEffective Search Query: \"2024 Artificial Intelligence Developments\"\nUser Question: \"How is climate change affecting agriculture globally?\"\n\nEffective Search Query: \"Global Impact of Climate Change on Agriculture 2024\"\nUser Question: \"What are the latest trends in cybersecurity threats?\"\n\nEffective Search Query: \"Cybersecurity Threats and Trends 2024\"\nUser Question: \"What is the outlook for renewable energy investments?\"\n\nEffective Search Query: \"Renewable Energy Investment Outlook 2024\"\n\n5. Data Input:\n - **Today's Date:** \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n **Search Inquiry:** \n - **Search Topic to create the query upon it:**{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}\"\"\n\n6. Now develop the best fit web search query given the user request above under number 5\n---\n\n**Output Requirements:** \nThe Assistantโs output should be in JSON format, structured as follows:\n\n{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}\n```\n\n---\n",
- "messages": {
- "messageValues": [
- {
- "message": "You are an advanced data and research retrieval through smart search queires via Bing and Brave websearch APIs. "
- },
- {
- "type": "HumanMessagePromptTemplate",
- "message": "1. **Task:** `\"Your task is to develop a web search query that most effectively answers the research question given. Use meta-reasoning and multi-chain analysis to ensure a comprehensive approach.\"`\n\n2. **Structured Guidance for Chains of Thought:** \n a. **Chain 1:** Break down the research question, identifying keywords and relevant terms. \n b. **Chain 2:** Explore the context and potential sources, determining the types of results that would be most relevant. \n c. **Chain 3:** Refine the query for specificity and completeness, considering how to capture nuances of the question.\n\n3. **Final Query Generation:** Based on the insights from the three chains, generate a single, refined search query.\n\n4. Data Input:\n - **Today's Date:** \"2024-11-12T10:21:33.764-05:00\"\n2. **Search Inquiry:** \n - **Search Topic to create the query upon it:** \"What is the latest stock Market Analysis in mid-term?\"\n\n\n---\n\n**Output Requirements:** \nThe Assistantโs output should be in JSON format, structured as follows:\n\n{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}\n```\n\n---\n"
- },
- {
- "type": "AIMessagePromptTemplate",
- "message": "Assessing query functionality\n\nIโm evaluating the viability of the assistant's proposed search query. The user questions if it would yield meaningful results on different search engines.\n\nClarifying the query\n\nOK, let me see. The task involves clarifying the user's needs, ensuring the response aligns with OpenAI's policies, and possibly re-evaluating and correcting any errors in the previous response.\n\nRefining search specifics\n\nI'm working through refining the search query to focus on specific terms like 'S&P 500', integrating dates such as 'November 2024', and emphasizing phrases for precision.\n\nSelf-reflecting on search queries\n\nToday, Iโm thinking about how to validate search queries for accuracy and realism, ensuring they are practical and likely to yield productive results.\n\nAcknowledging insights\n\nIโm recognizing and correcting a search query to ensure itโs optimal and effectively aligned with advanced user needs, leading to a more precise output.\n\nCrafting a search query\n\nIโm crafting a search query to pinpoint the most pertinent recent mid-term stock market analyses. Starting with relevant terms and then exploring potential sources for context and relevance.\n\nPinpointing sources and specifics\n\nI'm gathering info from financial news, investment reports, and economic research institutions. Iโm refining search terms like \"mid-term stock market analysis\" and \"market forecast\" to keep results current and nuanced.\n\nCrafting the query\n\nIโm finalizing the query by merging terms like 'latest' and 'stock market analysis' with 'mid-term', and considering synonyms such as 'market forecast' to refine the search for 2024.\n\nTweaking the search\n\nI adjusted the search terms to 'latest stock market mid-term outlook 2024,' removing excessive quotes for better clarity and relevance.\n\n\n{\n \"reasoning_summary\": \"Chain 1: Broke down the research question to identify key terms such as 'latest', 'stock market analysis', and 'mid-term'. Considered synonyms like 'mid-term outlook', 'market forecast', and included the current year '2024' for recency.\\n\\nChain 2: Explored the context where the user seeks recent analyses of the stock market with a mid-term perspective. Relevant sources would be financial news outlets, investment banks, and economic research firms. The most useful results would be up-to-date articles or reports offering mid-term forecasts.\\n\\nChain 3: Refined the query by removing unnecessary quotation marks and simplifying it to reflect how an advanced user might search. Included terms like 'latest', 'mid-term outlook', and '2024' to enhance specificity without limiting the search results unnecessarily.\",\n \"final_search_query\": \"latest stock market mid-term outlook 2024\"\n}"
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.4
- },
- {
- "id": "696b4f97-ad29-406b-9157-44ad9d05c9cd",
- "name": "Anthropic Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatAnthropic",
- "position": [
- 740,
- 180
- ],
- "parameters": {
- "model": "claude-3-5-haiku-20241022",
- "options": {
- "topP": 0.8,
- "temperature": 0.4,
- "maxTokensToSample": 4096
- }
- },
- "credentials": {
- "anthropicApi": {
- "id": "mVKB2CryW6bMm9Qo",
- "name": "Anthropic account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "82f25610-8b70-4aee-ad90-2616d3389f15",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 900,
- 180
- ],
- "parameters": {
- "options": {
- "topP": 0.7,
- "maxTokens": 4096,
- "maxRetries": 1,
- "temperature": 0.5
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "wQQZLwJO9A5nFu8h",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "f230bdf0-4a22-4abf-96cd-47f309f0c514",
- "name": "Structured Output Parser2",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 260,
- -60
- ],
- "parameters": {
- "jsonSchemaExample": "{\n \"chain_of_thought\": \"Insert your step-by-step reasoning here.\",\n \"Highest_RANKEDURL_1\": {\n \"title\": \"Insert the First Ranked URL's Title here.\",\n \"link\": \"Insert the First Ranked URL here.\",\n \"description\": \"Insert the First Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"Insert the Second Ranked URL's Title here.\",\n \"link\": \"Insert the Second Ranked URL here.\",\n \"description\": \"Insert the Second Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"Insert the Third Ranked URL's Title here.\",\n \"link\": \"Insert the Third Ranked URL here.\",\n \"description\": \"Insert the Third Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"Insert the Fourth Ranked URL's Title here.\",\n \"link\": \"Insert the Fourth Ranked URL here.\",\n \"description\": \"Insert the Fourth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"Insert the Fifth Ranked URL's Title here.\",\n \"link\": \"Insert the Fifth Ranked URL here.\",\n \"description\": \"Insert the Fifth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"Insert the Sixth Ranked URL's Title here.\",\n \"link\": \"Insert the Sixth Ranked URL here.\",\n \"description\": \"Insert the Sixth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"Insert the Seventh Ranked URL's Title here.\",\n \"link\": \"Insert the Seventh Ranked URL here.\",\n \"description\": \"Insert the Seventh Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"Insert the Eighth Ranked URL's Title here.\",\n \"link\": \"Insert the Eighth Ranked URL here.\",\n \"description\": \"Insert the Eighth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"Insert the Ninth Ranked URL's Title here.\",\n \"link\": \"Insert the Ninth Ranked URL here.\",\n \"description\": \"Insert the Ninth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"Insert the Tenth Ranked URL's Title here.\",\n \"link\": \"Insert the Tenth Ranked URL here.\",\n \"description\": \"Insert the Tenth Ranked URL's Description here.\"\n },\n \"Information_extracted\": \"Insert all extracted information relevant to the user's query or 'N/A' if none.\"\n}\n"
- },
- "typeVersion": 1.2
- },
- {
- "id": "3421ebe5-6a86-435e-b9c6-e3dcf6dd7833",
- "name": "Parser Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
- "position": [
- -180,
- 180
- ],
- "parameters": {
- "options": {
- "topP": 0.6,
- "temperature": 0.4,
- "maxOutputTokens": 4096
- },
- "modelName": "models/gemini-1.5-flash-002"
- },
- "credentials": {
- "googlePalmApi": {
- "id": "rTbWGMQGwWtjhNaA",
- "name": "Google Gemini(PaLM) Api account 4"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "fd19e9f6-d8c7-45df-93d9-ecf7956b461f",
- "name": "Agent Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
- "position": [
- -180,
- -20
- ],
- "parameters": {
- "options": {
- "topP": 0.6,
- "temperature": 0.4,
- "safetySettings": {
- "values": [
- {
- "category": "HARM_CATEGORY_HARASSMENT",
- "threshold": "BLOCK_NONE"
- },
- {
- "category": "HARM_CATEGORY_HATE_SPEECH",
- "threshold": "BLOCK_NONE"
- },
- {
- "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
- "threshold": "BLOCK_NONE"
- },
- {
- "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
- "threshold": "BLOCK_NONE"
- }
- ]
- },
- "maxOutputTokens": 4086
- },
- "modelName": "models/gemini-1.5-flash-002"
- },
- "credentials": {
- "googlePalmApi": {
- "id": "rTbWGMQGwWtjhNaA",
- "name": "Google Gemini(PaLM) Api account 4"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "156e83ff-928a-4aca-af8b-0c0fa51acd56",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 560,
- -20
- ],
- "parameters": {
- "color": 6,
- "width": 712,
- "height": 370,
- "content": "\n## Customized Models to Replace\n\nIn case you rather to use another LLM Model to Perform the Semantic Search and Re-Ranking, These nodes below are Optimized based on the LLM Structure of OpenAI GPT4o & Anthropic Claude.\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "b824ca9d-5676-4ca5-b97d-e5113d955de2",
- "connections": {
- "Query": {
- "main": [
- [
- {
- "node": "Query-1 Combined",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Webhook": {
- "main": [
- [
- {
- "node": "Date & Time",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Agent Model": {
- "ai_languageModel": [
- [
- {
- "node": "Semantic Search - Result Re-Ranker",
- "type": "ai_languageModel",
- "index": 0
- },
- {
- "node": "Semantic Search -Query Maker",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Date & Time": {
- "main": [
- [
- {
- "node": "Semantic Search -Query Maker",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parser Model": {
- "ai_languageModel": [
- [
- {
- "node": "Auto-fixing Output Parser6",
- "type": "ai_languageModel",
- "index": 0
- },
- {
- "node": "Auto-fixing Output Parser",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Query-1 Combined": {
- "main": [
- [
- {
- "node": "Semantic Search - Result Re-Ranker",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- []
- ]
- },
- "Anthropic Chat Model": {
- "ai_languageModel": [
- []
- ]
- },
- "Auto-fixing Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Semantic Search -Query Maker",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser1": {
- "ai_outputParser": [
- [
- {
- "node": "Auto-fixing Output Parser",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser2": {
- "ai_outputParser": [
- [
- {
- "node": "Auto-fixing Output Parser6",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Auto-fixing Output Parser6": {
- "ai_outputParser": [
- [
- {
- "node": "Semantic Search - Result Re-Ranker",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Semantic Search -Query Maker": {
- "main": [
- [
- {
- "node": "Query",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Semantic Search - Result Re-Ranker": {
- "main": [
- [
- {
- "node": "Respond to Webhook",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.txt b/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.txt
new file mode 100644
index 0000000..3e06ceb
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Intelligent Web Query and Semantic Re-Ranking Flow using Brave and Google Gemini.txt
@@ -0,0 +1,576 @@
+{
+"id": "wa2uEnSIowqSrHoY",
+"meta": {
+"instanceId": "cca06617664f52c5a019ea575691fdbce675dd95dc0452af5f13dbe76d615b69"
+},
+"name": "Intelligent Web Query and Semantic Re-Ranking Flow",
+"tags": [],
+"nodes": [
+{
+"id": "8e7dc5cb-6822-4ef6-9e5a-2b350a1526bf",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-640,
+-620
+],
+"parameters": {
+"color": 5,
+"width": 1172,
+"height": 970,
+"content": "\n## Step 1. Set Up a Free Brave Web Search Query API Key\n\nTo attain the free web search API tier from Brave, follow these steps:\n\n1. Visit api.search.brave.com\n2. Create an account\n3. Subscribe to the free plan (no charge)\n4. Navigate to the API Keys section\n5. Generate an API key. For the subscription type, choose \"Free\".\n6. Go to the \"Query\" Nodes and change the \"X-Subscription-Token\" value to your API Key.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "5bb3e68f-7693-4d4b-b794-843f2c3535e0",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1580,
+-420
+],
+"parameters": {
+"color": 4,
+"width": 680,
+"height": 360,
+"content": "## If you require to change this Node to Webhook Or any Other Item:\n\n- In case you want to change the input type from Webhook to any other item, Make sure to go to the Query 1 and Query 1 Ranker and replace the Webhook Input to your Node's input."
+},
+"typeVersion": 1
+},
+{
+"id": "f2fc02f9-a78a-4e87-be85-0032492a9f3f",
+"name": "Date & Time",
+"type": "n8n-nodes-base.dateTime",
+"position": [
+-820,
+-240
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 2
+},
+{
+"id": "6f18ebbd-83db-4900-bc2e-0a9f23d6e8c8",
+"name": "Webhook",
+"type": "n8n-nodes-base.webhook",
+"position": [
+-1340,
+-240
+],
+"webhookId": "962f1468-c80f-4c0c-8555-a0acf648ede4",
+"parameters": {
+"path": "962f1468-c80f-4c0c-8555-a0acf648ede4",
+"options": {},
+"responseMode": "responseNode"
+},
+"typeVersion": 2
+},
+{
+"id": "ba5ea83e-1b47-475b-863f-269ae293729a",
+"name": "Auto-fixing Output Parser6",
+"type": "@n8n/n8n-nodes-langchain.outputParserAutofixing",
+"position": [
+180,
+-140
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "ca426b6d-5412-4c5b-a55c-009a47c59a81",
+"name": "Auto-fixing Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserAutofixing",
+"position": [
+-580,
+-140
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "501d5390-5317-4973-a3e9-b0f502399c2b",
+"name": "Structured Output Parser1",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+-460,
+-60
+],
+"parameters": {
+"jsonSchemaExample": "{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "a27e75c7-0307-4d71-9266-5a56b297a6e3",
+"name": "Query-1 Combined",
+"type": "n8n-nodes-base.code",
+"position": [
+-80,
+-240
+],
+"parameters": {
+"jsCode": "// Initialize an empty string to store all title, url, and description pairs\nlet aggregatedOutputText = \"\";\n\n// Loop through all items passed to this Function node\nfor (let item of items) {\n // Access the JSON data from \"Query 1\" node for the current item\n const queryData = item.json;\n\n // Ensure there is a \"web.results\" array to process\n if (queryData.web?.results && Array.isArray(queryData.web.results)) {\n // Loop through all results in the \"web.results\" array\n for (let result of queryData.web.results) {\n // Extract the title, url, and description for each result\n const title = result.title || \"No Title\";\n const url = result.url || \"No URL\";\n const description = result.description || \"No Description\";\n\n // Append the values to the aggregated string\n aggregatedOutputText += `Title: ${title}\\nURL: ${url}\\nDescription: ${description}\\n\\n`;\n }\n } else {\n // If no results array, handle gracefully\n aggregatedOutputText += \"No results found for this item.\\n\\n\";\n }\n}\n\n// Trim the final string to remove any trailing newline and whitespace\naggregatedOutputText = aggregatedOutputText.trim();\n\n// Return a single item containing the aggregated output as a string\nreturn [\n {\n json: {\n aggregated_text: aggregatedOutputText\n }\n }\n];\n"
+},
+"typeVersion": 2
+},
+{
+"id": "acbdbe94-b5a7-4ec9-9fc8-c3ab147f42fa",
+"name": "Respond to Webhook",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+640,
+-240
+],
+"parameters": {
+"options": {},
+"respondWith": "text",
+"responseBody": "={\n \"Highest_RANKEDURL_1\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_1']['description'] }}\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_2']['description'] }}\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_3']['description'] }}\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_4']['description'] }}\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_5']['description'] }}\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_6']['description'] }}\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_7']['description'] }}\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_8']['description'] }}\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_9']['description'] }}\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['title'] }}\",\n \"link\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['link'] }}\",\n \"description\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Highest_RANKEDURL_10']['description'] }}\"\n },\n \"Information_extracted\": \"{{ $item('0').$node['Semantic Search - Result Re-Ranker'].json['output']['Information_extracted'] }}\"\n}\n"
+},
+"typeVersion": 1.1
+},
+{
+"id": "b8b6ae73-586a-406f-9641-57e2625f800c",
+"name": "Semantic Search - Result Re-Ranker",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"onError": "continueRegularOutput",
+"position": [
+100,
+-240
+],
+"parameters": {
+"text": "=\n**Objective:**\n\nFor the user's query, web search results are provided. Your tasks are:\n\n1. **Rank the links** based on how well they match the user's query.\n2. **Extract relevant information** from the descriptions provided. If no relevant information is found, return \"N/A\".\n\n---\n\n**Task:**\n\n**Step 1: Understand the User's Intent**\n\n- Determine what the user is truly and technically looking for.\n- The user's request query is: \"{{ $('Webhook').item.json.query['Research Question'] }}\"\n- The serach results below, however their performance seem, have been based on this query \"{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}\". If the result are not satisfactory or missing due to bad query making, you should note that as well for the neww query making.\n- To nesure being time aware , realize todays date is: \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n\n- Follow a three-step chain of thought to comprehend the user's needs. Think out loud.\n\n---\n\n**Step 2: Rank the Links**\n\n- From the URLs and description snippets provided, **rank the top 10 websites** that are most likely to contain the required information.\n- Use the titles, descriptions, and sources to inform your ranking.\n\n**Links, Titles, and Descriptions:**\n\n{{ $json.aggregated_text }}\n\n---\n\nThis list completes the structure up to 20 results as you requested. Let me know if thereโs anything more you need!\n\n---\n\n**Step 3: Analyze and Create a Follow-up Query**\n\n- Recognize that for the user's request:\n\n `\"{{ $('Webhook').item.json.query['Research Question'] }}\"`\n\n The results provided are based on the assistant's generated search query:\n\n `\"{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}\"`\n\n- Analyze and revise any issues or new insights through multi-step thinking to create a follow-up query.\n\n**Indications and Priorities:**\n\n1. **No Results Received:** If no search items are shared, the search query may have been ineffective (e.g., too specific, incorrect parameters).\n2. **Insufficient or Unpromising Results:** If fewer than 20 but more than 5 results are provided, and none seem promising, the search query may need refinement.\n3. **Successful Results with Potential Follow-up:** If none of the above issues occurred and the search results provide answers or suggest a follow-up, create a new query. This could be a new topic, a deep dive, or a parallel factor that offers additional benefits.\n\n- Provide your chain of thought that connects the user's request to the actual information.\n\n- Deliver precise, detailed, and value-oriented information relevant to the user's query.\n\n**Step 4: Query making notes and examples**: \n\nThe queries must not be long tails , as they result in 0 websearch reutrns. We give you some examples of good web search queries:\nExamples:\n\nUser Question: \"What is the current state of the U.S. economy in 2024?\"\nEffective Search Query: \"U.S. Economy Analysis Report 2024\"\n\nUser Question: \"What are the recent advancements in artificial intelligence?\"\nEffective Search Query: \"2024 Artificial Intelligence Developments\"\n\nUser Question: \"How is climate change affecting agriculture globally?\"\nEffective Search Query: \"Global Impact of Climate Change on Agriculture 2024\"\n\nUser Question: \"What are the latest trends in cybersecurity threats?\"\nEffective Search Query: \"Cybersecurity Threats and Trends 2024\"\n\nUser Question: \"What is the outlook for renewable energy investments?\"\nEffective Search Query: \"Renewable Energy Investment Outlook 2024\"\n\n**Step 5: Query making*: \nor query making remember as we said:\n - **Today's Date:** \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n **Search Inquiry:** \n - **Search Topic to create the query upon it:**{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}\"\"\n\n\n---\n\n**Step 6: Output Format**\n\nEnsure the response is in the following JSON format:\n\n{\n \"chain_of_thought\": \"Insert your step-by-step reasoning here.\",\n \"Highest_RANKEDURL_1\": {\n \"title\": \"Insert the First Ranked URL's Title here.\",\n \"link\": \"Insert the First Ranked URL here.\",\n \"description\": \"Insert the First Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"Insert the Second Ranked URL's Title here.\",\n \"link\": \"Insert the Second Ranked URL here.\",\n \"description\": \"Insert the Second Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"Insert the Third Ranked URL's Title here.\",\n \"link\": \"Insert the Third Ranked URL here.\",\n \"description\": \"Insert the Third Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"Insert the Fourth Ranked URL's Title here.\",\n \"link\": \"Insert the Fourth Ranked URL here.\",\n \"description\": \"Insert the Fourth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"Insert the Fifth Ranked URL's Title here.\",\n \"link\": \"Insert the Fifth Ranked URL here.\",\n \"description\": \"Insert the Fifth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"Insert the Sixth Ranked URL's Title here.\",\n \"link\": \"Insert the Sixth Ranked URL here.\",\n \"description\": \"Insert the Sixth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"Insert the Seventh Ranked URL's Title here.\",\n \"link\": \"Insert the Seventh Ranked URL here.\",\n \"description\": \"Insert the Seventh Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"Insert the Eighth Ranked URL's Title here.\",\n \"link\": \"Insert the Eighth Ranked URL here.\",\n \"description\": \"Insert the Eighth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"Insert the Ninth Ranked URL's Title here.\",\n \"link\": \"Insert the Ninth Ranked URL here.\",\n \"description\": \"Insert the Ninth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"Insert the Tenth Ranked URL's Title here.\",\n \"link\": \"Insert the Tenth Ranked URL here.\",\n \"description\": \"Insert the Tenth Ranked URL's Description here.\"\n },\n \"Information_extracted\": \"Insert all extracted information relevant to the user's query or 'N/A' if none.\"\n}\n",
+"messages": {
+"messageValues": [
+{
+"message": "=\nYou are an expert information retrieval and critical evaluation assistant designed to process, rank, and extract high-relevance content from web search results for complex user queries. You must provide value-oriented insights while refining searches based on relevance and context sensitivity. \n\n**Your Process and Priorities:**\n\n#### 1. **Determine the User's Technical Intent**\n - Interpret the user's core question provided as `{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}`, discerning underlying objectives and specialized needs.\n - Recognize that the search results may have been generated from a **secondary query**: `{{ $item(\"0\").$node[\"Semantic Search -Query Maker\"].json[\"output\"][\"final_search_query\"] }}`. \n - Judge the adequacy of this generated query. If it does not meet the userโs objectives, highlight the need for query refinement and prepare to adapt the approach.\n - Stay mindful of the date context, using `{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}` to assess the freshness of content or time-sensitive relevance.\n\n#### 2. **Rank Results Based on Analytical Relevance**\n - From the search results provided, **rank the top 3 URLs** that most closely align with the userโs intent and technical needs.\n - Use multi-dimensional analysis to assess how each linkโs title, description, and source match the userโs objective.\n - Prioritize results based on credibility, relevance, and their potential to add depth to the userโs inquiry.\n - Your goal is to select the highest-value links, disregarding results that offer superficial, off-topic, or outdated information.\n\n#### 3. **Extract Key Information**\n - For each of the top 3 ranked results, extract insights and details from the description snippets that directly address the userโs query.\n - If no pertinent information is available in a description, record `\"N/A\"` to indicate its lack of relevance.\n\n#### 4. **Evaluate for Potential Query Improvement**\n - Evaluate the relevance and coverage of search results:\n - If fewer than 5 relevant results are present, consider that the initial query may be too narrow, specific, or otherwise misaligned.\n - Generate a **refined query** that is adjusted to better match the userโs likely needs and produce higher-quality results.\n - Use advanced language modifications, new keyword suggestions, or rephrasing to formulate a search query that enhances alignment with the userโs goals.\n"
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"retryOnFail": true,
+"typeVersion": 1.4
+},
+{
+"id": "a1ca671d-0b0c-4717-9def-93fdb965de8d",
+"name": "Query",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-240,
+-240
+],
+"parameters": {
+"url": "https://api.search.brave.com/res/v1/web/search",
+"options": {},
+"sendQuery": true,
+"sendHeaders": true,
+"queryParameters": {
+"parameters": [
+{
+"name": "q",
+"value": "={{ $json.output.final_search_query }}"
+}
+]
+},
+"headerParameters": {
+"parameters": [
+{
+"name": "Accept",
+"value": "application/json"
+},
+{
+"name": "Accept-Encoding",
+"value": "gzip"
+},
+{
+"name": "X-Subscription-Token",
+"value": ""
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "d3cc4e7c-3ead-4d38-9b51-a11cd9d7faeb",
+"name": "Webhook Call",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-180,
+1040
+],
+"parameters": {
+"url": "https://primary-production-8aa4.up.railway.app/webhook-test/962f1468-c80f-4c0c-8555-a0acf648ede4",
+"options": {},
+"sendQuery": true,
+"queryParameters": {
+"parameters": [
+{
+"name": "Research Question",
+"value": "what is the latest news in global world in politics and economy?"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "6931404b-94d6-4b9d-9f0a-124012212eb5",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-640,
+420
+],
+"parameters": {
+"color": 3,
+"width": 1180,
+"height": 840,
+"content": "## Step 2. Setup the Webhook Call Node\n\n**Instructions for Setting Up the Webhook Call and Using It in Your Workflow**\n\nThis node is designed to send a **web search query** to the workflow (partly built in this chart) and return the results. Follow these steps to correctly configure and use it:\n\n1. **Locate the \"Webhook\" Node in the Workflow**:\n - Navigate to the workflow above, the first item, the \"Webhook\" node.\n - In the \"Webhook\" node, change the **Webhook URL option** from \"Test URL\" to \"Production URL.\"\n - Copy the generated **Production URL**.\n\n2. **Paste the Webhook URL in the HTTP Node**:\n - In your target workflow, locate the **HTTP Request** node.\n - Paste the copied **Production URL** into the URL field of the HTTP Request node. This connects the two workflows.\n\n3. **Send the Research Request**:\n - When sending the request to this workflow, make sure to include your web search query in the **\"Research Question\" parameter** of the HTTP Request node.\n\n4. **Move the Webhook Call Node**:\n - Move this **Webhook Call Node** into the workflow where you need the research results. Ensure that itโs correctly connected and configured to send the data to the main workflow.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "01d73f91-1dd6-4b80-951c-9f944ea9d992",
+"name": "Semantic Search -Query Maker",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-560,
+-240
+],
+"parameters": {
+"text": "=1. **Task:** `\"Your task is to develop a web search query that most effectively answers the research question given. Use meta-reasoning and multi-chain analysis to ensure a comprehensive approach.\"`\n\n2. **Structured Guidance for Chains of Thought:** \n a. **Chain 1:** Break down the research question, identifying keywords and relevant terms. \n b. **Chain 2:** Explore the context and potential sources, determining the types of results that would be most relevant. \n c. **Chain 3:** Refine the query for specificity and completeness, considering how to capture nuances of the question.\n\n3. **Final Query Generation:** Based on the insights from the three chains, generate a single, refined search query.\n\n\n4. Note, the queries must not be long tails , as they result in 0 websearch reutrns. We give you some examples of good web search queries:\nExamples:\n\nUser Question: \"What is the current state of the U.S. economy in 2024?\"\n\nEffective Search Query: \"U.S. Economy Analysis Report 2024\"\nUser Question: \"What are the recent advancements in artificial intelligence?\"\n\nEffective Search Query: \"2024 Artificial Intelligence Developments\"\nUser Question: \"How is climate change affecting agriculture globally?\"\n\nEffective Search Query: \"Global Impact of Climate Change on Agriculture 2024\"\nUser Question: \"What are the latest trends in cybersecurity threats?\"\n\nEffective Search Query: \"Cybersecurity Threats and Trends 2024\"\nUser Question: \"What is the outlook for renewable energy investments?\"\n\nEffective Search Query: \"Renewable Energy Investment Outlook 2024\"\n\n5. Data Input:\n - **Today's Date:** \"{{ $item(\"0\").$node[\"Date & Time\"].json[\"currentDate\"] }}\"\n **Search Inquiry:** \n - **Search Topic to create the query upon it:**{{ $item(\"0\").$node[\"Webhook\"].json[\"query\"][\"Research Question\"] }}\"\"\n\n6. Now develop the best fit web search query given the user request above under number 5\n---\n\n**Output Requirements:** \nThe Assistantโs output should be in JSON format, structured as follows:\n\n{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}\n```\n\n---\n",
+"messages": {
+"messageValues": [
+{
+"message": "You are an advanced data and research retrieval through smart search queires via Bing and Brave websearch APIs. "
+},
+{
+"type": "HumanMessagePromptTemplate",
+"message": "1. **Task:** `\"Your task is to develop a web search query that most effectively answers the research question given. Use meta-reasoning and multi-chain analysis to ensure a comprehensive approach.\"`\n\n2. **Structured Guidance for Chains of Thought:** \n a. **Chain 1:** Break down the research question, identifying keywords and relevant terms. \n b. **Chain 2:** Explore the context and potential sources, determining the types of results that would be most relevant. \n c. **Chain 3:** Refine the query for specificity and completeness, considering how to capture nuances of the question.\n\n3. **Final Query Generation:** Based on the insights from the three chains, generate a single, refined search query.\n\n4. Data Input:\n - **Today's Date:** \"2024-11-12T10:21:33.764-05:00\"\n2. **Search Inquiry:** \n - **Search Topic to create the query upon it:** \"What is the latest stock Market Analysis in mid-term?\"\n\n\n---\n\n**Output Requirements:** \nThe Assistantโs output should be in JSON format, structured as follows:\n\n{\n \"reasoning_summary\": \"Detailed explanation of each analytical chainโs purpose and insights, including key terms and considerations for query formulation.\",\n \"final_search_query\": \"The single, best-fit search query derived from the meta-reasoning and multi-chain analysis, optimized to answer the research question.\"\n}\n```\n\n---\n"
+},
+{
+"type": "AIMessagePromptTemplate",
+"message": "Assessing query functionality\n\nIโm evaluating the viability of the assistant's proposed search query. The user questions if it would yield meaningful results on different search engines.\n\nClarifying the query\n\nOK, let me see. The task involves clarifying the user's needs, ensuring the response aligns with OpenAI's policies, and possibly re-evaluating and correcting any errors in the previous response.\n\nRefining search specifics\n\nI'm working through refining the search query to focus on specific terms like 'S&P 500', integrating dates such as 'November 2024', and emphasizing phrases for precision.\n\nSelf-reflecting on search queries\n\nToday, Iโm thinking about how to validate search queries for accuracy and realism, ensuring they are practical and likely to yield productive results.\n\nAcknowledging insights\n\nIโm recognizing and correcting a search query to ensure itโs optimal and effectively aligned with advanced user needs, leading to a more precise output.\n\nCrafting a search query\n\nIโm crafting a search query to pinpoint the most pertinent recent mid-term stock market analyses. Starting with relevant terms and then exploring potential sources for context and relevance.\n\nPinpointing sources and specifics\n\nI'm gathering info from financial news, investment reports, and economic research institutions. Iโm refining search terms like \"mid-term stock market analysis\" and \"market forecast\" to keep results current and nuanced.\n\nCrafting the query\n\nIโm finalizing the query by merging terms like 'latest' and 'stock market analysis' with 'mid-term', and considering synonyms such as 'market forecast' to refine the search for 2024.\n\nTweaking the search\n\nI adjusted the search terms to 'latest stock market mid-term outlook 2024,' removing excessive quotes for better clarity and relevance.\n\n\n{\n \"reasoning_summary\": \"Chain 1: Broke down the research question to identify key terms such as 'latest', 'stock market analysis', and 'mid-term'. Considered synonyms like 'mid-term outlook', 'market forecast', and included the current year '2024' for recency.\\n\\nChain 2: Explored the context where the user seeks recent analyses of the stock market with a mid-term perspective. Relevant sources would be financial news outlets, investment banks, and economic research firms. The most useful results would be up-to-date articles or reports offering mid-term forecasts.\\n\\nChain 3: Refined the query by removing unnecessary quotation marks and simplifying it to reflect how an advanced user might search. Included terms like 'latest', 'mid-term outlook', and '2024' to enhance specificity without limiting the search results unnecessarily.\",\n \"final_search_query\": \"latest stock market mid-term outlook 2024\"\n}"
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.4
+},
+{
+"id": "696b4f97-ad29-406b-9157-44ad9d05c9cd",
+"name": "Anthropic Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatAnthropic",
+"position": [
+740,
+180
+],
+"parameters": {
+"model": "claude-3-5-haiku-20241022",
+"options": {
+"topP": 0.8,
+"temperature": 0.4,
+"maxTokensToSample": 4096
+}
+},
+"credentials": {
+"anthropicApi": {
+"id": "mVKB2CryW6bMm9Qo",
+"name": "Anthropic account"
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "82f25610-8b70-4aee-ad90-2616d3389f15",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+900,
+180
+],
+"parameters": {
+"options": {
+"topP": 0.7,
+"maxTokens": 4096,
+"maxRetries": 1,
+"temperature": 0.5
+}
+},
+"credentials": {
+"openAiApi": {
+"id": "wQQZLwJO9A5nFu8h",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "f230bdf0-4a22-4abf-96cd-47f309f0c514",
+"name": "Structured Output Parser2",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+260,
+-60
+],
+"parameters": {
+"jsonSchemaExample": "{\n \"chain_of_thought\": \"Insert your step-by-step reasoning here.\",\n \"Highest_RANKEDURL_1\": {\n \"title\": \"Insert the First Ranked URL's Title here.\",\n \"link\": \"Insert the First Ranked URL here.\",\n \"description\": \"Insert the First Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_2\": {\n \"title\": \"Insert the Second Ranked URL's Title here.\",\n \"link\": \"Insert the Second Ranked URL here.\",\n \"description\": \"Insert the Second Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_3\": {\n \"title\": \"Insert the Third Ranked URL's Title here.\",\n \"link\": \"Insert the Third Ranked URL here.\",\n \"description\": \"Insert the Third Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_4\": {\n \"title\": \"Insert the Fourth Ranked URL's Title here.\",\n \"link\": \"Insert the Fourth Ranked URL here.\",\n \"description\": \"Insert the Fourth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_5\": {\n \"title\": \"Insert the Fifth Ranked URL's Title here.\",\n \"link\": \"Insert the Fifth Ranked URL here.\",\n \"description\": \"Insert the Fifth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_6\": {\n \"title\": \"Insert the Sixth Ranked URL's Title here.\",\n \"link\": \"Insert the Sixth Ranked URL here.\",\n \"description\": \"Insert the Sixth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_7\": {\n \"title\": \"Insert the Seventh Ranked URL's Title here.\",\n \"link\": \"Insert the Seventh Ranked URL here.\",\n \"description\": \"Insert the Seventh Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_8\": {\n \"title\": \"Insert the Eighth Ranked URL's Title here.\",\n \"link\": \"Insert the Eighth Ranked URL here.\",\n \"description\": \"Insert the Eighth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_9\": {\n \"title\": \"Insert the Ninth Ranked URL's Title here.\",\n \"link\": \"Insert the Ninth Ranked URL here.\",\n \"description\": \"Insert the Ninth Ranked URL's Description here.\"\n },\n \"Highest_RANKEDURL_10\": {\n \"title\": \"Insert the Tenth Ranked URL's Title here.\",\n \"link\": \"Insert the Tenth Ranked URL here.\",\n \"description\": \"Insert the Tenth Ranked URL's Description here.\"\n },\n \"Information_extracted\": \"Insert all extracted information relevant to the user's query or 'N/A' if none.\"\n}\n"
+},
+"typeVersion": 1.2
+},
+{
+"id": "3421ebe5-6a86-435e-b9c6-e3dcf6dd7833",
+"name": "Parser Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+-180,
+180
+],
+"parameters": {
+"options": {
+"topP": 0.6,
+"temperature": 0.4,
+"maxOutputTokens": 4096
+},
+"modelName": "models/gemini-1.5-flash-002"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "rTbWGMQGwWtjhNaA",
+"name": "Google Gemini(PaLM) Api account 4"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "fd19e9f6-d8c7-45df-93d9-ecf7956b461f",
+"name": "Agent Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+-180,
+-20
+],
+"parameters": {
+"options": {
+"topP": 0.6,
+"temperature": 0.4,
+"safetySettings": {
+"values": [
+{
+"category": "HARM_CATEGORY_HARASSMENT",
+"threshold": "BLOCK_NONE"
+},
+{
+"category": "HARM_CATEGORY_HATE_SPEECH",
+"threshold": "BLOCK_NONE"
+},
+{
+"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
+"threshold": "BLOCK_NONE"
+},
+{
+"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
+"threshold": "BLOCK_NONE"
+}
+]
+},
+"maxOutputTokens": 4086
+},
+"modelName": "models/gemini-1.5-flash-002"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "rTbWGMQGwWtjhNaA",
+"name": "Google Gemini(PaLM) Api account 4"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "156e83ff-928a-4aca-af8b-0c0fa51acd56",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+560,
+-20
+],
+"parameters": {
+"color": 6,
+"width": 712,
+"height": 370,
+"content": "\n## Customized Models to Replace\n\nIn case you rather to use another LLM Model to Perform the Semantic Search and Re-Ranking, These nodes below are Optimized based on the LLM Structure of OpenAI GPT4o & Anthropic Claude.\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "b824ca9d-5676-4ca5-b97d-e5113d955de2",
+"connections": {
+"Query": {
+"main": [
+[
+{
+"node": "Query-1 Combined",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Webhook": {
+"main": [
+[
+{
+"node": "Date & Time",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Agent Model": {
+"ai_languageModel": [
+[
+{
+"node": "Semantic Search - Result Re-Ranker",
+"type": "ai_languageModel",
+"index": 0
+},
+{
+"node": "Semantic Search -Query Maker",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Date & Time": {
+"main": [
+[
+{
+"node": "Semantic Search -Query Maker",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parser Model": {
+"ai_languageModel": [
+[
+{
+"node": "Auto-fixing Output Parser6",
+"type": "ai_languageModel",
+"index": 0
+},
+{
+"node": "Auto-fixing Output Parser",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Query-1 Combined": {
+"main": [
+[
+{
+"node": "Semantic Search - Result Re-Ranker",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[]
+]
+},
+"Anthropic Chat Model": {
+"ai_languageModel": [
+[]
+]
+},
+"Auto-fixing Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Semantic Search -Query Maker",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser1": {
+"ai_outputParser": [
+[
+{
+"node": "Auto-fixing Output Parser",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser2": {
+"ai_outputParser": [
+[
+{
+"node": "Auto-fixing Output Parser6",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Auto-fixing Output Parser6": {
+"ai_outputParser": [
+[
+{
+"node": "Semantic Search - Result Re-Ranker",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Semantic Search -Query Maker": {
+"main": [
+[
+{
+"node": "Query",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Semantic Search - Result Re-Ranker": {
+"main": [
+[
+{
+"node": "Respond to Webhook",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.json b/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.json
deleted file mode 100644
index 1caad1d..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.json
+++ /dev/null
@@ -1,295 +0,0 @@
-{
- "nodes": [
- {
- "id": "41183066-0045-4a75-ba23-42f4efcfeccc",
- "name": "Google Gemini Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
- "position": [
- 720,
- 720
- ],
- "parameters": {
- "options": {},
- "modelName": "models/gemini-1.5-flash"
- },
- "credentials": {
- "googlePalmApi": {
- "id": "Hx1fn2jrUvojSKye",
- "name": "Google Gemini(PaLM) Api account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "eb061c39-7a4d-42e7-bb42-806504731b11",
- "name": "Basic LLM Chain",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 700,
- 560
- ],
- "parameters": {
- "text": "=Your Task is to find the best resources to learn {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}. \n\nI have scraped the HackerNews and The following is the list of comments from HackerNews on topic about Learning {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}\n\n\nFocus only on comments that provide any resouces or advice or insight about learning {{ $('GetTopicFromToLearn').item.json.Learn }}. Ignore all other comments that are off topic discussions.\n\nNow based on these comments, you need to find the top resources and list them. \n\nCategorize them based on resource type (course, book, article, youtube videos, lectures, etc) and also figure out the difficultiy level (beginner, intermediate, advanced, expert).\n\nYou don't always to have fill in these categories exactly, these are given here for reference. Use your intution to find the best categorization.\n\nNow based on these metrics and running a basic sentiment analysis on comments you need to figure out what the top resources are. \n\nRespond back in Markdown formatted text. In the following format\n\n**OUTPUT FORMAT**\n\n```\n\n## Top HN Recomended Resources To Learn \n\n### Category 1\n\n- **Resource 1** - One line description\n- **Resource 2** - One line description\n- ... \n\n\n\n### Category 2\n\n- **Resource 1** - One line description\n- **Resource 2** - One line description\n- ... \n\n\n\n...\n```\n\nHere is the list of HackerNews Comments.\n\n{{ $json.text }}",
- "promptType": "define"
- },
- "typeVersion": 1.5
- },
- {
- "id": "94073fe0-d25c-421e-9c99-67b6c4f0afad",
- "name": "SearchAskHN",
- "type": "n8n-nodes-base.hackerNews",
- "position": [
- -160,
- 560
- ],
- "parameters": {
- "limit": 150,
- "resource": "all",
- "additionalFields": {
- "tags": [
- "ask_hn"
- ],
- "keyword": "={{ $json[\"I want to learn\"] }}"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "eee4dfdf-53ab-42be-91ae-7b6c405df7c2",
- "name": "FindHNComments",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 260,
- 560
- ],
- "parameters": {
- "url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.children }}.json?print=pretty",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "e57d86ae-d7c1-4354-9e3c-528c76160cd9",
- "name": "CombineIntoSingleText",
- "type": "n8n-nodes-base.aggregate",
- "position": [
- 480,
- 560
- ],
- "parameters": {
- "options": {},
- "fieldsToAggregate": {
- "fieldToAggregate": [
- {
- "fieldToAggregate": "text"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "b2086d29-1de5-48f4-8c1e-affd509fb5f7",
- "name": "SplitOutChildrenIDs",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 40,
- 560
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "children"
- },
- "typeVersion": 1
- },
- {
- "id": "6fe68a4b-744b-48c8-9320-d2b19e3eb92b",
- "name": "GetTopicFromToLearn",
- "type": "n8n-nodes-base.formTrigger",
- "position": [
- -340,
- 560
- ],
- "webhookId": "4524d82f-86a6-4fab-ba09-1d24001e15f3",
- "parameters": {
- "options": {
- "path": "learn",
- "buttonLabel": "Submit",
- "respondWithOptions": {
- "values": {
- "formSubmittedText": "We'll shortly send you an email with top recommendations."
- }
- }
- },
- "formTitle": "What do You want to learn ?",
- "formFields": {
- "values": [
- {
- "fieldLabel": "I want to learn",
- "placeholder": "Python, DevOps, Ai, or just about anything"
- },
- {
- "fieldType": "email",
- "fieldLabel": "What's your email ?",
- "placeholder": "john.doe@example.com",
- "requiredField": true
- }
- ]
- },
- "formDescription": "We'll find the best resources from HackerNews and send you an email"
- },
- "typeVersion": 2.2
- },
- {
- "id": "72fcb7f3-6706-47cc-8a79-364b325aa8ae",
- "name": "SendEmailWithTopResources",
- "type": "n8n-nodes-base.emailSend",
- "position": [
- 1320,
- 560
- ],
- "parameters": {
- "html": "=FYI, We read through {{ $('SplitOutChildrenIDs').all().length }} comments in search for the best.\n\n{{ $json.data }}",
- "options": {},
- "subject": "=Here are Top HN Recommendations for Learning {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}",
- "toEmail": "={{ $('GetTopicFromToLearn').item.json[\"What's your email ?\"] }}",
- "fromEmail": "allsmallnocaps@gmail.com"
- },
- "credentials": {
- "smtp": {
- "id": "knhWxmnfY16ZQwBm",
- "name": "allsamll Gmail SMTP account"
- }
- },
- "typeVersion": 2.1
- },
- {
- "id": "b4d50b42-9e40-46b0-a411-90210b422de3",
- "name": "Convert2HTML",
- "type": "n8n-nodes-base.markdown",
- "position": [
- 1100,
- 560
- ],
- "parameters": {
- "mode": "markdownToHtml",
- "options": {},
- "markdown": "={{ $json.text }}"
- },
- "typeVersion": 1
- },
- {
- "id": "b79e867a-ea3b-4a94-9809-b5a01ee2820f",
- "name": "Finished",
- "type": "n8n-nodes-base.noOp",
- "position": [
- 1540,
- 560
- ],
- "parameters": {},
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "SearchAskHN": {
- "main": [
- [
- {
- "node": "SplitOutChildrenIDs",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Convert2HTML": {
- "main": [
- [
- {
- "node": "SendEmailWithTopResources",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "FindHNComments": {
- "main": [
- [
- {
- "node": "CombineIntoSingleText",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Basic LLM Chain": {
- "main": [
- [
- {
- "node": "Convert2HTML",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "GetTopicFromToLearn": {
- "main": [
- [
- {
- "node": "SearchAskHN",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "SplitOutChildrenIDs": {
- "main": [
- [
- {
- "node": "FindHNComments",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "CombineIntoSingleText": {
- "main": [
- [
- {
- "node": "Basic LLM Chain",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google Gemini Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Basic LLM Chain",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "SendEmailWithTopResources": {
- "main": [
- [
- {
- "node": "Finished",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.txt b/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.txt
new file mode 100644
index 0000000..ecf505d
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Learn Anything from HN - Get Top Resource Recommendations from Hacker News.txt
@@ -0,0 +1,295 @@
+{
+"nodes": [
+{
+"id": "41183066-0045-4a75-ba23-42f4efcfeccc",
+"name": "Google Gemini Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+720,
+720
+],
+"parameters": {
+"options": {},
+"modelName": "models/gemini-1.5-flash"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "Hx1fn2jrUvojSKye",
+"name": "Google Gemini(PaLM) Api account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "eb061c39-7a4d-42e7-bb42-806504731b11",
+"name": "Basic LLM Chain",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+700,
+560
+],
+"parameters": {
+"text": "=Your Task is to find the best resources to learn {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}. \n\nI have scraped the HackerNews and The following is the list of comments from HackerNews on topic about Learning {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}\n\n\nFocus only on comments that provide any resouces or advice or insight about learning {{ $('GetTopicFromToLearn').item.json.Learn }}. Ignore all other comments that are off topic discussions.\n\nNow based on these comments, you need to find the top resources and list them. \n\nCategorize them based on resource type (course, book, article, youtube videos, lectures, etc) and also figure out the difficultiy level (beginner, intermediate, advanced, expert).\n\nYou don't always to have fill in these categories exactly, these are given here for reference. Use your intution to find the best categorization.\n\nNow based on these metrics and running a basic sentiment analysis on comments you need to figure out what the top resources are. \n\nRespond back in Markdown formatted text. In the following format\n\n**OUTPUT FORMAT**\n\n```\n\n## Top HN Recomended Resources To Learn \n\n### Category 1\n\n- **Resource 1** - One line description\n- **Resource 2** - One line description\n- ... \n\n\n\n### Category 2\n\n- **Resource 1** - One line description\n- **Resource 2** - One line description\n- ... \n\n\n\n...\n```\n\nHere is the list of HackerNews Comments.\n\n{{ $json.text }}",
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "94073fe0-d25c-421e-9c99-67b6c4f0afad",
+"name": "SearchAskHN",
+"type": "n8n-nodes-base.hackerNews",
+"position": [
+-160,
+560
+],
+"parameters": {
+"limit": 150,
+"resource": "all",
+"additionalFields": {
+"tags": [
+"ask_hn"
+],
+"keyword": "={{ $json[\"I want to learn\"] }}"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "eee4dfdf-53ab-42be-91ae-7b6c405df7c2",
+"name": "FindHNComments",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+260,
+560
+],
+"parameters": {
+"url": "=https://hacker-news.firebaseio.com/v0/item/{{ $json.children }}.json?print=pretty",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e57d86ae-d7c1-4354-9e3c-528c76160cd9",
+"name": "CombineIntoSingleText",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+480,
+560
+],
+"parameters": {
+"options": {},
+"fieldsToAggregate": {
+"fieldToAggregate": [
+{
+"fieldToAggregate": "text"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b2086d29-1de5-48f4-8c1e-affd509fb5f7",
+"name": "SplitOutChildrenIDs",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+40,
+560
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "children"
+},
+"typeVersion": 1
+},
+{
+"id": "6fe68a4b-744b-48c8-9320-d2b19e3eb92b",
+"name": "GetTopicFromToLearn",
+"type": "n8n-nodes-base.formTrigger",
+"position": [
+-340,
+560
+],
+"webhookId": "4524d82f-86a6-4fab-ba09-1d24001e15f3",
+"parameters": {
+"options": {
+"path": "learn",
+"buttonLabel": "Submit",
+"respondWithOptions": {
+"values": {
+"formSubmittedText": "We'll shortly send you an email with top recommendations."
+}
+}
+},
+"formTitle": "What do You want to learn ?",
+"formFields": {
+"values": [
+{
+"fieldLabel": "I want to learn",
+"placeholder": "Python, DevOps, Ai, or just about anything"
+},
+{
+"fieldType": "email",
+"fieldLabel": "What's your email ?",
+"placeholder": "john.doe@example.com",
+"requiredField": true
+}
+]
+},
+"formDescription": "We'll find the best resources from HackerNews and send you an email"
+},
+"typeVersion": 2.2
+},
+{
+"id": "72fcb7f3-6706-47cc-8a79-364b325aa8ae",
+"name": "SendEmailWithTopResources",
+"type": "n8n-nodes-base.emailSend",
+"position": [
+1320,
+560
+],
+"parameters": {
+"html": "=FYI, We read through {{ $('SplitOutChildrenIDs').all().length }} comments in search for the best.\n\n{{ $json.data }}",
+"options": {},
+"subject": "=Here are Top HN Recommendations for Learning {{ $('GetTopicFromToLearn').item.json[\"I want to learn\"] }}",
+"toEmail": "={{ $('GetTopicFromToLearn').item.json[\"What's your email ?\"] }}",
+"fromEmail": "allsmallnocaps@gmail.com"
+},
+"credentials": {
+"smtp": {
+"id": "knhWxmnfY16ZQwBm",
+"name": "allsamll Gmail SMTP account"
+}
+},
+"typeVersion": 2.1
+},
+{
+"id": "b4d50b42-9e40-46b0-a411-90210b422de3",
+"name": "Convert2HTML",
+"type": "n8n-nodes-base.markdown",
+"position": [
+1100,
+560
+],
+"parameters": {
+"mode": "markdownToHtml",
+"options": {},
+"markdown": "={{ $json.text }}"
+},
+"typeVersion": 1
+},
+{
+"id": "b79e867a-ea3b-4a94-9809-b5a01ee2820f",
+"name": "Finished",
+"type": "n8n-nodes-base.noOp",
+"position": [
+1540,
+560
+],
+"parameters": {},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"SearchAskHN": {
+"main": [
+[
+{
+"node": "SplitOutChildrenIDs",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert2HTML": {
+"main": [
+[
+{
+"node": "SendEmailWithTopResources",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"FindHNComments": {
+"main": [
+[
+{
+"node": "CombineIntoSingleText",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Basic LLM Chain": {
+"main": [
+[
+{
+"node": "Convert2HTML",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"GetTopicFromToLearn": {
+"main": [
+[
+{
+"node": "SearchAskHN",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"SplitOutChildrenIDs": {
+"main": [
+[
+{
+"node": "FindHNComments",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"CombineIntoSingleText": {
+"main": [
+[
+{
+"node": "Basic LLM Chain",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Gemini Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Basic LLM Chain",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"SendEmailWithTopResources": {
+"main": [
+[
+{
+"node": "Finished",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.json b/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.json
deleted file mode 100644
index 08bac18..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.json
+++ /dev/null
@@ -1,502 +0,0 @@
-{
- "id": "5NAbfX550LJsfz6f",
- "meta": {
- "instanceId": "00493e38fecfc163cb182114bc2fab90114038eb9aad665a7a752d076920d3d5",
- "templateCredsSetupCompleted": true
- },
- "name": "Make OpenAI Citation for File Retrieval RAG",
- "tags": [
- {
- "id": "urxRtGxxLObZWPvX",
- "name": "sample",
- "createdAt": "2024-09-13T02:43:13.014Z",
- "updatedAt": "2024-09-13T02:43:13.014Z"
- },
- {
- "id": "nMXS3c9l1WqDwWF5",
- "name": "assist",
- "createdAt": "2024-12-23T16:09:38.737Z",
- "updatedAt": "2024-12-23T16:09:38.737Z"
- }
- ],
- "nodes": [
- {
- "id": "b9033511-3421-467a-9bfa-73af01b99c4f",
- "name": "Aggregate",
- "type": "n8n-nodes-base.aggregate",
- "position": [
- 740,
- 120
- ],
- "parameters": {
- "options": {},
- "aggregate": "aggregateAllItemData"
- },
- "typeVersion": 1,
- "alwaysOutputData": true
- },
- {
- "id": "a61dd9d3-4faa-4878-a6f3-ba8277279002",
- "name": "Window Buffer Memory",
- "type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
- "position": [
- 980,
- -320
- ],
- "parameters": {},
- "typeVersion": 1.3
- },
- {
- "id": "2daabca5-37ec-4cad-9157-29926367e1a7",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 220,
- 320
- ],
- "parameters": {
- "color": 3,
- "width": 840,
- "height": 80,
- "content": "## Within N8N, there will be a chat button to test"
- },
- "typeVersion": 1
- },
- {
- "id": "bf4485b1-cd94-41c8-a183-bf1b785f2761",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -440,
- -520
- ],
- "parameters": {
- "color": 5,
- "width": 500,
- "height": 720,
- "content": "## Make OpenAI Citation for File Retrieval RAG\n\n## Use case\n\nIn this example, we will ensure that all texts from the OpenAI assistant search for citations and sources in the vector store files. We can also format the output for Markdown or HTML tags.\n\nThis is necessary because the assistant sometimes generates strange characters, and we can also use dynamic references such as citations 1, 2, 3, for example.\n\n## What this workflow does\n\nIn this workflow, we will use an OpenAI assistant created within their interface, equipped with a vector store containing some files for file retrieval.\n\nThe assistant will perform the file search within the OpenAI infrastructure and will return the content with citations.\n\n- We will make an HTTP request to retrieve all the details we need to format the text output.\n\n## Setup\n\nInsert an OpenAI Key\n\n## How to adjust it to your needs\n\nAt the end of the workflow, we have a block of code that will format the output, and there we can add Markdown tags to create links. Optionally, we can transform the Markdown formatting into HTML.\n\n\nby Davi Saranszky Mesquita\nhttps://www.linkedin.com/in/mesquitadavi/"
- },
- "typeVersion": 1
- },
- {
- "id": "539a4e40-9745-4a26-aba8-2cc2b0dd6364",
- "name": "Create a simple Trigger to have the Chat button within N8N",
- "type": "@n8n/n8n-nodes-langchain.chatTrigger",
- "notes": "https://www.npmjs.com/package/@n8n/chat",
- "position": [
- 260,
- -520
- ],
- "webhookId": "8ccaa299-6f99-427b-9356-e783893a3d0c",
- "parameters": {
- "options": {}
- },
- "notesInFlow": true,
- "typeVersion": 1.1
- },
- {
- "id": "aa5b2951-df32-43ac-9939-83b02d818e73",
- "name": "OpenAI Assistant with Vector Store",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 580,
- -520
- ],
- "parameters": {
- "options": {
- "preserveOriginalTools": false
- },
- "resource": "assistant",
- "assistantId": {
- "__rl": true,
- "mode": "list",
- "value": "asst_QAfdobVCVCMJz8LmaEC7nlId",
- "cachedResultName": "Teste"
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "UfNrqPCRlD8FD9mk",
- "name": "OpenAi Lourival"
- }
- },
- "typeVersion": 1.7
- },
- {
- "id": "1817b673-6cb3-49aa-9f38-a5876eb0e6fa",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 560,
- -680
- ],
- "parameters": {
- "width": 300,
- "content": "## Setup\n\n- Configure OpenAI Key\n\n### In this step, we will use an assistant created within the OpenAI platform that contains a vector store a.k.a file retrieval"
- },
- "typeVersion": 1
- },
- {
- "id": "16429226-e850-4698-b419-fd9805a03fb7",
- "name": "Get ALL Thread Content",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1260,
- -520
- ],
- "parameters": {
- "url": "=https://api.openai.com/v1/threads/{{ $json.threadId }}/messages",
- "options": {},
- "sendHeaders": true,
- "authentication": "predefinedCredentialType",
- "headerParameters": {
- "parameters": [
- {
- "name": "OpenAI-Beta",
- "value": "assistants=v2"
- }
- ]
- },
- "nodeCredentialType": "openAiApi"
- },
- "credentials": {
- "openAiApi": {
- "id": "UfNrqPCRlD8FD9mk",
- "name": "OpenAi Lourival"
- }
- },
- "typeVersion": 4.2,
- "alwaysOutputData": true
- },
- {
- "id": "e8c88b08-5be2-4f7e-8b17-8cf804b3fe9f",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1160,
- -620
- ],
- "parameters": {
- "content": "### Retrieving all thread content is necessary because the OpenAI tool does not retrieve all citations upon request."
- },
- "typeVersion": 1
- },
- {
- "id": "0f51e09f-2782-4e2d-b797-d4d58fcabdaf",
- "name": "Split all message iterations from a thread",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 220,
- -300
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "data"
- },
- "typeVersion": 1,
- "alwaysOutputData": true
- },
- {
- "id": "4d569993-1ce3-4b32-beaf-382feac25da9",
- "name": "Split all content from a single message",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 460,
- -300
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "content"
- },
- "typeVersion": 1,
- "alwaysOutputData": true
- },
- {
- "id": "999e1c2b-1927-4483-aac1-6e8903f7ed25",
- "name": "Split all citations from a single message",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 700,
- -300
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "text.annotations"
- },
- "typeVersion": 1,
- "alwaysOutputData": true
- },
- {
- "id": "98af62f5-adb0-4e07-a146-fc2f13b851ce",
- "name": "Retrieve file name from a file ID",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 220,
- 120
- ],
- "parameters": {
- "url": "=https://api.openai.com/v1/files/{{ $json.file_citation.file_id }}",
- "options": {},
- "sendQuery": true,
- "authentication": "predefinedCredentialType",
- "queryParameters": {
- "parameters": [
- {
- "name": "limit",
- "value": "1"
- }
- ]
- },
- "nodeCredentialType": "openAiApi"
- },
- "credentials": {
- "openAiApi": {
- "id": "UfNrqPCRlD8FD9mk",
- "name": "OpenAi Lourival"
- }
- },
- "typeVersion": 4.2,
- "alwaysOutputData": true
- },
- {
- "id": "b11f0d3d-bdc4-4845-b14b-d0b0de214f01",
- "name": "Regularize output",
- "type": "n8n-nodes-base.set",
- "position": [
- 480,
- 120
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "2dcaafee-5037-4a97-942a-bcdd02bc2ad9",
- "name": "id",
- "type": "string",
- "value": "={{ $json.id }}"
- },
- {
- "id": "b63f967d-ceea-4aa8-98b9-91f5ab21bfe8",
- "name": "filename",
- "type": "string",
- "value": "={{ $json.filename }}"
- },
- {
- "id": "f611e749-054a-441d-8610-df8ba42de2e1",
- "name": "text",
- "type": "string",
- "value": "={{ $('Split all citations from a single message').item.json.text }}"
- }
- ]
- }
- },
- "typeVersion": 3.4,
- "alwaysOutputData": true
- },
- {
- "id": "0e999a0e-76ed-4897-989b-228f075e9bfb",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 440,
- -60
- ],
- "parameters": {
- "width": 200,
- "height": 220,
- "content": "### A file retrieval request contains a lot of information, and we want only the text that will be substituted and the file name.\n\n- id\n- filename\n- text\n"
- },
- "typeVersion": 1
- },
- {
- "id": "53c79a6c-7543-435f-b40e-966dff0904d4",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 700,
- -60
- ],
- "parameters": {
- "width": 200,
- "height": 220,
- "content": "### With the last three splits, we may have many citations and texts to substitute. By doing an aggregation, it will be possible to handle everything as a single request."
- },
- "typeVersion": 1
- },
- {
- "id": "381fb6d6-64fc-4668-9d3c-98aaa43a45ca",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 960,
- -60
- ],
- "parameters": {
- "height": 220,
- "content": "### This simple code will take all the previous files and citations and alter the original text, formatting the output. In this way, we can use Markdown tags to create links, or if you prefer, we can add an HTML transformation node."
- },
- "typeVersion": 1
- },
- {
- "id": "d0cbb943-57ab-4850-8370-1625610a852a",
- "name": "Optional Markdown to HTML",
- "type": "n8n-nodes-base.markdown",
- "disabled": true,
- "position": [
- 1220,
- 120
- ],
- "parameters": {
- "html": "={{ $json.output }}",
- "options": {},
- "destinationKey": "output"
- },
- "typeVersion": 1
- },
- {
- "id": "589e2418-5dec-47d0-ba08-420d84f09da7",
- "name": "Finnaly format the output",
- "type": "n8n-nodes-base.code",
- "position": [
- 980,
- 120
- ],
- "parameters": {
- "mode": "runOnceForEachItem",
- "jsCode": "let saida = $('OpenAI Assistant with Vector Store').item.json.output;\n\nfor (let i of $input.item.json.data) {\n saida = saida.replaceAll(i.text, \" _(\"+ i.filename+\")_ \");\n}\n\n$input.item.json.output = saida;\nreturn $input.item;"
- },
- "typeVersion": 2
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "0e621a5a-d99d-4db3-9ae4-ea98c31467e9",
- "connections": {
- "Aggregate": {
- "main": [
- [
- {
- "node": "Finnaly format the output",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Regularize output": {
- "main": [
- [
- {
- "node": "Aggregate",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Window Buffer Memory": {
- "ai_memory": [
- [
- {
- "node": "OpenAI Assistant with Vector Store",
- "type": "ai_memory",
- "index": 0
- }
- ]
- ]
- },
- "Get ALL Thread Content": {
- "main": [
- [
- {
- "node": "Split all message iterations from a thread",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Finnaly format the output": {
- "main": [
- [
- {
- "node": "Optional Markdown to HTML",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Retrieve file name from a file ID": {
- "main": [
- [
- {
- "node": "Regularize output",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Assistant with Vector Store": {
- "main": [
- [
- {
- "node": "Get ALL Thread Content",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split all content from a single message": {
- "main": [
- [
- {
- "node": "Split all citations from a single message",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split all citations from a single message": {
- "main": [
- [
- {
- "node": "Retrieve file name from a file ID",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split all message iterations from a thread": {
- "main": [
- [
- {
- "node": "Split all content from a single message",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create a simple Trigger to have the Chat button within N8N": {
- "main": [
- [
- {
- "node": "OpenAI Assistant with Vector Store",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.txt b/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.txt
new file mode 100644
index 0000000..59db43c
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Make OpenAI Citation for File Retrieval RAG.txt
@@ -0,0 +1,502 @@
+{
+"id": "5NAbfX550LJsfz6f",
+"meta": {
+"instanceId": "00493e38fecfc163cb182114bc2fab90114038eb9aad665a7a752d076920d3d5",
+"templateCredsSetupCompleted": true
+},
+"name": "Make OpenAI Citation for File Retrieval RAG",
+"tags": [
+{
+"id": "urxRtGxxLObZWPvX",
+"name": "sample",
+"createdAt": "2024-09-13T02:43:13.014Z",
+"updatedAt": "2024-09-13T02:43:13.014Z"
+},
+{
+"id": "nMXS3c9l1WqDwWF5",
+"name": "assist",
+"createdAt": "2024-12-23T16:09:38.737Z",
+"updatedAt": "2024-12-23T16:09:38.737Z"
+}
+],
+"nodes": [
+{
+"id": "b9033511-3421-467a-9bfa-73af01b99c4f",
+"name": "Aggregate",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+740,
+120
+],
+"parameters": {
+"options": {},
+"aggregate": "aggregateAllItemData"
+},
+"typeVersion": 1,
+"alwaysOutputData": true
+},
+{
+"id": "a61dd9d3-4faa-4878-a6f3-ba8277279002",
+"name": "Window Buffer Memory",
+"type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
+"position": [
+980,
+-320
+],
+"parameters": {},
+"typeVersion": 1.3
+},
+{
+"id": "2daabca5-37ec-4cad-9157-29926367e1a7",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+220,
+320
+],
+"parameters": {
+"color": 3,
+"width": 840,
+"height": 80,
+"content": "## Within N8N, there will be a chat button to test"
+},
+"typeVersion": 1
+},
+{
+"id": "bf4485b1-cd94-41c8-a183-bf1b785f2761",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-440,
+-520
+],
+"parameters": {
+"color": 5,
+"width": 500,
+"height": 720,
+"content": "## Make OpenAI Citation for File Retrieval RAG\n\n## Use case\n\nIn this example, we will ensure that all texts from the OpenAI assistant search for citations and sources in the vector store files. We can also format the output for Markdown or HTML tags.\n\nThis is necessary because the assistant sometimes generates strange characters, and we can also use dynamic references such as citations 1, 2, 3, for example.\n\n## What this workflow does\n\nIn this workflow, we will use an OpenAI assistant created within their interface, equipped with a vector store containing some files for file retrieval.\n\nThe assistant will perform the file search within the OpenAI infrastructure and will return the content with citations.\n\n- We will make an HTTP request to retrieve all the details we need to format the text output.\n\n## Setup\n\nInsert an OpenAI Key\n\n## How to adjust it to your needs\n\nAt the end of the workflow, we have a block of code that will format the output, and there we can add Markdown tags to create links. Optionally, we can transform the Markdown formatting into HTML.\n\n\nby Davi Saranszky Mesquita\nhttps://www.linkedin.com/in/mesquitadavi/"
+},
+"typeVersion": 1
+},
+{
+"id": "539a4e40-9745-4a26-aba8-2cc2b0dd6364",
+"name": "Create a simple Trigger to have the Chat button within N8N",
+"type": "@n8n/n8n-nodes-langchain.chatTrigger",
+"notes": "https://www.npmjs.com/package/@n8n/chat",
+"position": [
+260,
+-520
+],
+"webhookId": "8ccaa299-6f99-427b-9356-e783893a3d0c",
+"parameters": {
+"options": {}
+},
+"notesInFlow": true,
+"typeVersion": 1.1
+},
+{
+"id": "aa5b2951-df32-43ac-9939-83b02d818e73",
+"name": "OpenAI Assistant with Vector Store",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+580,
+-520
+],
+"parameters": {
+"options": {
+"preserveOriginalTools": false
+},
+"resource": "assistant",
+"assistantId": {
+"__rl": true,
+"mode": "list",
+"value": "asst_QAfdobVCVCMJz8LmaEC7nlId",
+"cachedResultName": "Teste"
+}
+},
+"credentials": {
+"openAiApi": {
+"id": "UfNrqPCRlD8FD9mk",
+"name": "OpenAi Lourival"
+}
+},
+"typeVersion": 1.7
+},
+{
+"id": "1817b673-6cb3-49aa-9f38-a5876eb0e6fa",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+560,
+-680
+],
+"parameters": {
+"width": 300,
+"content": "## Setup\n\n- Configure OpenAI Key\n\n### In this step, we will use an assistant created within the OpenAI platform that contains a vector store a.k.a file retrieval"
+},
+"typeVersion": 1
+},
+{
+"id": "16429226-e850-4698-b419-fd9805a03fb7",
+"name": "Get ALL Thread Content",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1260,
+-520
+],
+"parameters": {
+"url": "=https://api.openai.com/v1/threads/{{ $json.threadId }}/messages",
+"options": {},
+"sendHeaders": true,
+"authentication": "predefinedCredentialType",
+"headerParameters": {
+"parameters": [
+{
+"name": "OpenAI-Beta",
+"value": "assistants=v2"
+}
+]
+},
+"nodeCredentialType": "openAiApi"
+},
+"credentials": {
+"openAiApi": {
+"id": "UfNrqPCRlD8FD9mk",
+"name": "OpenAi Lourival"
+}
+},
+"typeVersion": 4.2,
+"alwaysOutputData": true
+},
+{
+"id": "e8c88b08-5be2-4f7e-8b17-8cf804b3fe9f",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1160,
+-620
+],
+"parameters": {
+"content": "### Retrieving all thread content is necessary because the OpenAI tool does not retrieve all citations upon request."
+},
+"typeVersion": 1
+},
+{
+"id": "0f51e09f-2782-4e2d-b797-d4d58fcabdaf",
+"name": "Split all message iterations from a thread",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+220,
+-300
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 1,
+"alwaysOutputData": true
+},
+{
+"id": "4d569993-1ce3-4b32-beaf-382feac25da9",
+"name": "Split all content from a single message",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+460,
+-300
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "content"
+},
+"typeVersion": 1,
+"alwaysOutputData": true
+},
+{
+"id": "999e1c2b-1927-4483-aac1-6e8903f7ed25",
+"name": "Split all citations from a single message",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+700,
+-300
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "text.annotations"
+},
+"typeVersion": 1,
+"alwaysOutputData": true
+},
+{
+"id": "98af62f5-adb0-4e07-a146-fc2f13b851ce",
+"name": "Retrieve file name from a file ID",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+220,
+120
+],
+"parameters": {
+"url": "=https://api.openai.com/v1/files/{{ $json.file_citation.file_id }}",
+"options": {},
+"sendQuery": true,
+"authentication": "predefinedCredentialType",
+"queryParameters": {
+"parameters": [
+{
+"name": "limit",
+"value": "1"
+}
+]
+},
+"nodeCredentialType": "openAiApi"
+},
+"credentials": {
+"openAiApi": {
+"id": "UfNrqPCRlD8FD9mk",
+"name": "OpenAi Lourival"
+}
+},
+"typeVersion": 4.2,
+"alwaysOutputData": true
+},
+{
+"id": "b11f0d3d-bdc4-4845-b14b-d0b0de214f01",
+"name": "Regularize output",
+"type": "n8n-nodes-base.set",
+"position": [
+480,
+120
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "2dcaafee-5037-4a97-942a-bcdd02bc2ad9",
+"name": "id",
+"type": "string",
+"value": "={{ $json.id }}"
+},
+{
+"id": "b63f967d-ceea-4aa8-98b9-91f5ab21bfe8",
+"name": "filename",
+"type": "string",
+"value": "={{ $json.filename }}"
+},
+{
+"id": "f611e749-054a-441d-8610-df8ba42de2e1",
+"name": "text",
+"type": "string",
+"value": "={{ $('Split all citations from a single message').item.json.text }}"
+}
+]
+}
+},
+"typeVersion": 3.4,
+"alwaysOutputData": true
+},
+{
+"id": "0e999a0e-76ed-4897-989b-228f075e9bfb",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+440,
+-60
+],
+"parameters": {
+"width": 200,
+"height": 220,
+"content": "### A file retrieval request contains a lot of information, and we want only the text that will be substituted and the file name.\n\n- id\n- filename\n- text\n"
+},
+"typeVersion": 1
+},
+{
+"id": "53c79a6c-7543-435f-b40e-966dff0904d4",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+700,
+-60
+],
+"parameters": {
+"width": 200,
+"height": 220,
+"content": "### With the last three splits, we may have many citations and texts to substitute. By doing an aggregation, it will be possible to handle everything as a single request."
+},
+"typeVersion": 1
+},
+{
+"id": "381fb6d6-64fc-4668-9d3c-98aaa43a45ca",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+960,
+-60
+],
+"parameters": {
+"height": 220,
+"content": "### This simple code will take all the previous files and citations and alter the original text, formatting the output. In this way, we can use Markdown tags to create links, or if you prefer, we can add an HTML transformation node."
+},
+"typeVersion": 1
+},
+{
+"id": "d0cbb943-57ab-4850-8370-1625610a852a",
+"name": "Optional Markdown to HTML",
+"type": "n8n-nodes-base.markdown",
+"disabled": true,
+"position": [
+1220,
+120
+],
+"parameters": {
+"html": "={{ $json.output }}",
+"options": {},
+"destinationKey": "output"
+},
+"typeVersion": 1
+},
+{
+"id": "589e2418-5dec-47d0-ba08-420d84f09da7",
+"name": "Finnaly format the output",
+"type": "n8n-nodes-base.code",
+"position": [
+980,
+120
+],
+"parameters": {
+"mode": "runOnceForEachItem",
+"jsCode": "let saida = $('OpenAI Assistant with Vector Store').item.json.output;\n\nfor (let i of $input.item.json.data) {\n saida = saida.replaceAll(i.text, \" _(\"+ i.filename+\")_ \");\n}\n\n$input.item.json.output = saida;\nreturn $input.item;"
+},
+"typeVersion": 2
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "0e621a5a-d99d-4db3-9ae4-ea98c31467e9",
+"connections": {
+"Aggregate": {
+"main": [
+[
+{
+"node": "Finnaly format the output",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Regularize output": {
+"main": [
+[
+{
+"node": "Aggregate",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Window Buffer Memory": {
+"ai_memory": [
+[
+{
+"node": "OpenAI Assistant with Vector Store",
+"type": "ai_memory",
+"index": 0
+}
+]
+]
+},
+"Get ALL Thread Content": {
+"main": [
+[
+{
+"node": "Split all message iterations from a thread",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Finnaly format the output": {
+"main": [
+[
+{
+"node": "Optional Markdown to HTML",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Retrieve file name from a file ID": {
+"main": [
+[
+{
+"node": "Regularize output",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Assistant with Vector Store": {
+"main": [
+[
+{
+"node": "Get ALL Thread Content",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split all content from a single message": {
+"main": [
+[
+{
+"node": "Split all citations from a single message",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split all citations from a single message": {
+"main": [
+[
+{
+"node": "Retrieve file name from a file ID",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split all message iterations from a thread": {
+"main": [
+[
+{
+"node": "Split all content from a single message",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create a simple Trigger to have the Chat button within N8N": {
+"main": [
+[
+{
+"node": "OpenAI Assistant with Vector Store",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.json b/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.json
deleted file mode 100644
index f22adda..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.json
+++ /dev/null
@@ -1,468 +0,0 @@
-{
- "id": "WLSqXECfQF7rOj2A",
- "meta": {
- "instanceId": "cba4a4a2eb5d7683330e2944837278938831ed3c042e20da6f5049c07ad14798"
- },
- "name": "Open Deep Research - AI-Powered Autonomous Research Workflow",
- "tags": [],
- "nodes": [
- {
- "id": "b7b70ba1-0267-4d2b-91f4-5cc4fd22fd03",
- "name": "Chat Message Trigger",
- "type": "@n8n/n8n-nodes-langchain.chatTrigger",
- "position": [
- -1940,
- 160
- ],
- "webhookId": "cb0b9dbe-1f35-441a-b062-29624b0ebc6a",
- "parameters": {
- "options": {}
- },
- "typeVersion": 1.1
- },
- {
- "id": "55a8a512-f2d4-4aed-93e5-dd9bfa2dcaad",
- "name": "Generate Search Queries using LLM",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- -1760,
- 160
- ],
- "parameters": {
- "text": "=User Query: {{ $('Chat Message Trigger').item.json.chatInput }}",
- "messages": {
- "messageValues": [
- {
- "message": "=You are an expert research assistant. Given a user's query, generate up to four distinct, precise search queries that would help gather comprehensive information on the topic. Return only a JSON list of strings, for example: ['query1', 'query2', 'query3']."
- }
- ]
- },
- "promptType": "define"
- },
- "typeVersion": 1.5
- },
- {
- "id": "5f92361a-b490-479d-8360-c87a100b470e",
- "name": "LLM Response Provider (OpenRouter)",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenRouter",
- "position": [
- -1760,
- 700
- ],
- "parameters": {
- "model": "google/gemini-2.0-flash-001",
- "options": {}
- },
- "credentials": {
- "openRouterApi": {
- "id": "WZWYWCfluxuKxZzV",
- "name": "OpenRouter account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "4ab360eb-858f-48b8-a00d-71867d4f0c93",
- "name": "Parse and Chunk JSON Data",
- "type": "n8n-nodes-base.code",
- "position": [
- -1420,
- 160
- ],
- "parameters": {
- "jsCode": "// Parse the input JSON string and split it into four chunks\nconst rawText = $json.text;\n\n// Remove Markdown JSON code blocks if present\nconst cleanedText = rawText.replace(/```json|```/g, '').trim();\n\ntry {\n const jsonArray = JSON.parse(cleanedText);\n if (!Array.isArray(jsonArray)) {\n throw new Error('The JSON is not an array.');\n }\n const chunkSize = Math.ceil(jsonArray.length / 4);\n const chunks = [];\n for (let i = 0; i < jsonArray.length; i += chunkSize) {\n chunks.push(jsonArray.slice(i, i + chunkSize));\n }\n return chunks.map(chunk => ({ json: { chunk } }));\n} catch (error) {\n return [{ json: { error: error.message } }];\n}\n"
- },
- "typeVersion": 2
- },
- {
- "id": "5a3ac393-8355-449f-93cb-b98e8bee9b80",
- "name": "Perform SerpAPI Search Request",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -780,
- 180
- ],
- "parameters": {
- "url": "https://serpapi.com/search",
- "options": {},
- "sendQuery": true,
- "queryParameters": {
- "parameters": [
- {
- "name": "q",
- "value": "={{ $('Parse and Chunk JSON Data').item.json.chunk }}"
- },
- {
- "name": "api_key",
- "value": "={{ $credentials.SerpAPI.key }}"
- },
- {
- "name": "engine",
- "value": "google"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "dad82469-830d-40fb-9f6b-b9fefef41267",
- "name": "Perform Jina AI Analysis Request",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 80,
- 160
- ],
- "parameters": {
- "url": "=https://r.jina.ai/{{ $json.url }}",
- "options": {},
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "iseKF5sPsvwtJhgT",
- "name": "Jina AI"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "e21bbdf6-a903-491e-920c-ef7576f9ce80",
- "name": "Format SerpAPI Organic Results",
- "type": "n8n-nodes-base.code",
- "position": [
- -460,
- 140
- ],
- "parameters": {
- "jsCode": "// Format the organic search results from SerpAPI\nconst results = $input.first().json.organic_results;\nif (results.length === 0) {\n return [{ json: { error: 'No search results found.' } }];\n}\nconst formattedResults = results.map(result => ({\n title: result.title || 'No title available',\n url: result.link || 'No link available',\n source: result.source || result.displayed_link || 'Unknown source'\n}));\nreturn formattedResults.map(result => ({ json: result }));\n"
- },
- "typeVersion": 2
- },
- {
- "id": "a856c8e8-5c3c-4a2f-9086-66deee1afd06",
- "name": "Extract Relevant Context via LLM",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- -1280,
- 520
- ],
- "parameters": {
- "text": "=User Queries: {{ $('Parse and Chunk JSON Data').all().map(item => item.json.chunk[0]).join(', ') }}\nWebpage Contents: \n\"\"\"\n{{ $json.data }}\n\"\"\"",
- "options": {
- "systemMessage": "=You are an expert information extractor. Given the user's query, the search query that led to this page, and the webpage content, extract all relevant pieces of information that are useful to answer the query. Return only the relevant context as plain text without any additional commentary."
- },
- "promptType": "define"
- },
- "typeVersion": 1.7
- },
- {
- "id": "6d5c6698-0b4f-438c-91b9-3597f5d3e904",
- "name": "Generate Comprehensive Research Report",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- -740,
- 520
- ],
- "parameters": {
- "text": "=Extracted Contexts (Merged):\n\"\"\"\n{{ $json.output }}\n\"\"\"",
- "options": {
- "systemMessage": "You are an expert researcher and report writer. Based on the gathered contexts and the original user query, generate a comprehensive, well-structured report. Include all relevant insights and conclusions without unnecessary commentary.\n\nFormat the report in Markdown with clear headings. For example:\n\n# Research Report: [User Query]\n\n## Key Findings\n- Point 1\n- Point 2\n\n## Detailed Analysis\n### Aspect 1\nSummary of findings.\n_Source:_ [Source Name](URL)\n\n### Aspect 2\nSummary of findings.\n_Source:_ [Another Source](URL)\n\nNow, generate the complete report."
- },
- "promptType": "define"
- },
- "typeVersion": 1.7
- },
- {
- "id": "05fea6a1-791e-4980-8f2a-2960455066d7",
- "name": "Split Data for SerpAPI Batching",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -1100,
- 160
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "df00e7e8-99b8-484a-8047-869474fefee9",
- "name": "Split Data for Jina AI Batching",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -220,
- 140
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "2edc683b-65f7-40c3-a22d-7fbf5b67de0a",
- "name": "LLM Memory Buffer (Input Context)",
- "type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
- "position": [
- -1160,
- 740
- ],
- "parameters": {
- "sessionKey": "my_test_session",
- "sessionIdType": "customKey",
- "contextWindowLength": 20
- },
- "typeVersion": 1.3
- },
- {
- "id": "23017ae7-72a7-45c7-8edf-d0ba72220675",
- "name": "LLM Memory Buffer (Report Context)",
- "type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
- "position": [
- -620,
- 760
- ],
- "parameters": {
- "sessionKey": "my_test_session",
- "sessionIdType": "customKey",
- "contextWindowLength": 20
- },
- "typeVersion": 1.3
- },
- {
- "id": "6bc9533b-e265-47b3-b93a-3a4f86ba0541",
- "name": "Fetch Wikipedia Information",
- "type": "@n8n/n8n-nodes-langchain.toolWikipedia",
- "position": [
- -580,
- 920
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "b25c148e-047d-40a7-8818-94c3504828dd",
- "name": "Sticky Note: SerpAPI Setup",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -940,
- -20
- ],
- "parameters": {
- "color": 7,
- "width": 420,
- "height": 140,
- "content": "## SerpAPI Setup Instructions\n1. Obtain your API key from https://serpapi.com/manage-api-key.\n2. Save your API key securely in n8n credentials (do not use plain text)."
- },
- "typeVersion": 1
- },
- {
- "id": "e69c9a85-31e4-42b9-a09a-683ec5bb97d1",
- "name": "Sticky Note: Jina AI Setup",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -60,
- -40
- ],
- "parameters": {
- "color": 7,
- "width": 420,
- "height": 140,
- "content": "## Jina AI Setup Instructions\n1. Obtain your API key from https://jina.ai/api-dashboard/key-manager.\n2. Configure your Jina AI credential in n8n to ensure secure API access."
- },
- "typeVersion": 1
- },
- {
- "id": "dbd204e0-da8e-41d8-814b-f409a23e9573",
- "name": "Sticky Note: OpenRouter API Setup",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1680,
- 460
- ],
- "parameters": {
- "color": 7,
- "width": 300,
- "height": 180,
- "content": "## OpenRouter API Setup Instructions\n1. Obtain your API key from https://openrouter.ai/settings/keys.\n2. Set up your OpenRouter credential in n8n for secure integration."
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "aa857bb3-84c1-4fe6-9464-90fc09163960",
- "connections": {
- "Chat Message Trigger": {
- "main": [
- [
- {
- "node": "Generate Search Queries using LLM",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse and Chunk JSON Data": {
- "main": [
- [
- {
- "node": "Split Data for SerpAPI Batching",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Fetch Wikipedia Information": {
- "ai_tool": [
- [
- {
- "node": "Generate Comprehensive Research Report",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Format SerpAPI Organic Results": {
- "main": [
- [
- {
- "node": "Split Data for Jina AI Batching",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Perform SerpAPI Search Request": {
- "main": [
- [
- {
- "node": "Split Data for SerpAPI Batching",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Data for Jina AI Batching": {
- "main": [
- [
- {
- "node": "Extract Relevant Context via LLM",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Perform Jina AI Analysis Request",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Data for SerpAPI Batching": {
- "main": [
- [
- {
- "node": "Format SerpAPI Organic Results",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Perform SerpAPI Search Request",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Relevant Context via LLM": {
- "main": [
- [
- {
- "node": "Generate Comprehensive Research Report",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Perform Jina AI Analysis Request": {
- "main": [
- [
- {
- "node": "Split Data for Jina AI Batching",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Generate Search Queries using LLM": {
- "main": [
- [
- {
- "node": "Parse and Chunk JSON Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "LLM Memory Buffer (Input Context)": {
- "ai_memory": [
- [
- {
- "node": "Extract Relevant Context via LLM",
- "type": "ai_memory",
- "index": 0
- }
- ]
- ]
- },
- "LLM Memory Buffer (Report Context)": {
- "ai_memory": [
- [
- {
- "node": "Generate Comprehensive Research Report",
- "type": "ai_memory",
- "index": 0
- }
- ]
- ]
- },
- "LLM Response Provider (OpenRouter)": {
- "ai_languageModel": [
- [
- {
- "node": "Generate Search Queries using LLM",
- "type": "ai_languageModel",
- "index": 0
- },
- {
- "node": "Extract Relevant Context via LLM",
- "type": "ai_languageModel",
- "index": 0
- },
- {
- "node": "Generate Comprehensive Research Report",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.txt b/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.txt
new file mode 100644
index 0000000..271ec08
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Open Deep Research - AI-Powered Autonomous Research Workflow.txt
@@ -0,0 +1,468 @@
+{
+"id": "WLSqXECfQF7rOj2A",
+"meta": {
+"instanceId": "cba4a4a2eb5d7683330e2944837278938831ed3c042e20da6f5049c07ad14798"
+},
+"name": "Open Deep Research - AI-Powered Autonomous Research Workflow",
+"tags": [],
+"nodes": [
+{
+"id": "b7b70ba1-0267-4d2b-91f4-5cc4fd22fd03",
+"name": "Chat Message Trigger",
+"type": "@n8n/n8n-nodes-langchain.chatTrigger",
+"position": [
+-1940,
+160
+],
+"webhookId": "cb0b9dbe-1f35-441a-b062-29624b0ebc6a",
+"parameters": {
+"options": {}
+},
+"typeVersion": 1.1
+},
+{
+"id": "55a8a512-f2d4-4aed-93e5-dd9bfa2dcaad",
+"name": "Generate Search Queries using LLM",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+-1760,
+160
+],
+"parameters": {
+"text": "=User Query: {{ $('Chat Message Trigger').item.json.chatInput }}",
+"messages": {
+"messageValues": [
+{
+"message": "=You are an expert research assistant. Given a user's query, generate up to four distinct, precise search queries that would help gather comprehensive information on the topic. Return only a JSON list of strings, for example: ['query1', 'query2', 'query3']."
+}
+]
+},
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "5f92361a-b490-479d-8360-c87a100b470e",
+"name": "LLM Response Provider (OpenRouter)",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenRouter",
+"position": [
+-1760,
+700
+],
+"parameters": {
+"model": "google/gemini-2.0-flash-001",
+"options": {}
+},
+"credentials": {
+"openRouterApi": {
+"id": "WZWYWCfluxuKxZzV",
+"name": "OpenRouter account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "4ab360eb-858f-48b8-a00d-71867d4f0c93",
+"name": "Parse and Chunk JSON Data",
+"type": "n8n-nodes-base.code",
+"position": [
+-1420,
+160
+],
+"parameters": {
+"jsCode": "// Parse the input JSON string and split it into four chunks\nconst rawText = $json.text;\n\n// Remove Markdown JSON code blocks if present\nconst cleanedText = rawText.replace(/```json|```/g, '').trim();\n\ntry {\n const jsonArray = JSON.parse(cleanedText);\n if (!Array.isArray(jsonArray)) {\n throw new Error('The JSON is not an array.');\n }\n const chunkSize = Math.ceil(jsonArray.length / 4);\n const chunks = [];\n for (let i = 0; i < jsonArray.length; i += chunkSize) {\n chunks.push(jsonArray.slice(i, i + chunkSize));\n }\n return chunks.map(chunk => ({ json: { chunk } }));\n} catch (error) {\n return [{ json: { error: error.message } }];\n}\n"
+},
+"typeVersion": 2
+},
+{
+"id": "5a3ac393-8355-449f-93cb-b98e8bee9b80",
+"name": "Perform SerpAPI Search Request",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-780,
+180
+],
+"parameters": {
+"url": "https://serpapi.com/search",
+"options": {},
+"sendQuery": true,
+"queryParameters": {
+"parameters": [
+{
+"name": "q",
+"value": "={{ $('Parse and Chunk JSON Data').item.json.chunk }}"
+},
+{
+"name": "api_key",
+"value": "={{ $credentials.SerpAPI.key }}"
+},
+{
+"name": "engine",
+"value": "google"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "dad82469-830d-40fb-9f6b-b9fefef41267",
+"name": "Perform Jina AI Analysis Request",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+80,
+160
+],
+"parameters": {
+"url": "=https://r.jina.ai/{{ $json.url }}",
+"options": {},
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "iseKF5sPsvwtJhgT",
+"name": "Jina AI"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e21bbdf6-a903-491e-920c-ef7576f9ce80",
+"name": "Format SerpAPI Organic Results",
+"type": "n8n-nodes-base.code",
+"position": [
+-460,
+140
+],
+"parameters": {
+"jsCode": "// Format the organic search results from SerpAPI\nconst results = $input.first().json.organic_results;\nif (results.length === 0) {\n return [{ json: { error: 'No search results found.' } }];\n}\nconst formattedResults = results.map(result => ({\n title: result.title || 'No title available',\n url: result.link || 'No link available',\n source: result.source || result.displayed_link || 'Unknown source'\n}));\nreturn formattedResults.map(result => ({ json: result }));\n"
+},
+"typeVersion": 2
+},
+{
+"id": "a856c8e8-5c3c-4a2f-9086-66deee1afd06",
+"name": "Extract Relevant Context via LLM",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+-1280,
+520
+],
+"parameters": {
+"text": "=User Queries: {{ $('Parse and Chunk JSON Data').all().map(item => item.json.chunk[0]).join(', ') }}\nWebpage Contents: \n\"\"\"\n{{ $json.data }}\n\"\"\"",
+"options": {
+"systemMessage": "=You are an expert information extractor. Given the user's query, the search query that led to this page, and the webpage content, extract all relevant pieces of information that are useful to answer the query. Return only the relevant context as plain text without any additional commentary."
+},
+"promptType": "define"
+},
+"typeVersion": 1.7
+},
+{
+"id": "6d5c6698-0b4f-438c-91b9-3597f5d3e904",
+"name": "Generate Comprehensive Research Report",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+-740,
+520
+],
+"parameters": {
+"text": "=Extracted Contexts (Merged):\n\"\"\"\n{{ $json.output }}\n\"\"\"",
+"options": {
+"systemMessage": "You are an expert researcher and report writer. Based on the gathered contexts and the original user query, generate a comprehensive, well-structured report. Include all relevant insights and conclusions without unnecessary commentary.\n\nFormat the report in Markdown with clear headings. For example:\n\n# Research Report: [User Query]\n\n## Key Findings\n- Point 1\n- Point 2\n\n## Detailed Analysis\n### Aspect 1\nSummary of findings.\n_Source:_ [Source Name](URL)\n\n### Aspect 2\nSummary of findings.\n_Source:_ [Another Source](URL)\n\nNow, generate the complete report."
+},
+"promptType": "define"
+},
+"typeVersion": 1.7
+},
+{
+"id": "05fea6a1-791e-4980-8f2a-2960455066d7",
+"name": "Split Data for SerpAPI Batching",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-1100,
+160
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "df00e7e8-99b8-484a-8047-869474fefee9",
+"name": "Split Data for Jina AI Batching",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-220,
+140
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "2edc683b-65f7-40c3-a22d-7fbf5b67de0a",
+"name": "LLM Memory Buffer (Input Context)",
+"type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
+"position": [
+-1160,
+740
+],
+"parameters": {
+"sessionKey": "my_test_session",
+"sessionIdType": "customKey",
+"contextWindowLength": 20
+},
+"typeVersion": 1.3
+},
+{
+"id": "23017ae7-72a7-45c7-8edf-d0ba72220675",
+"name": "LLM Memory Buffer (Report Context)",
+"type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
+"position": [
+-620,
+760
+],
+"parameters": {
+"sessionKey": "my_test_session",
+"sessionIdType": "customKey",
+"contextWindowLength": 20
+},
+"typeVersion": 1.3
+},
+{
+"id": "6bc9533b-e265-47b3-b93a-3a4f86ba0541",
+"name": "Fetch Wikipedia Information",
+"type": "@n8n/n8n-nodes-langchain.toolWikipedia",
+"position": [
+-580,
+920
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "b25c148e-047d-40a7-8818-94c3504828dd",
+"name": "Sticky Note: SerpAPI Setup",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-940,
+-20
+],
+"parameters": {
+"color": 7,
+"width": 420,
+"height": 140,
+"content": "## SerpAPI Setup Instructions\n1. Obtain your API key from https://serpapi.com/manage-api-key.\n2. Save your API key securely in n8n credentials (do not use plain text)."
+},
+"typeVersion": 1
+},
+{
+"id": "e69c9a85-31e4-42b9-a09a-683ec5bb97d1",
+"name": "Sticky Note: Jina AI Setup",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-60,
+-40
+],
+"parameters": {
+"color": 7,
+"width": 420,
+"height": 140,
+"content": "## Jina AI Setup Instructions\n1. Obtain your API key from https://jina.ai/api-dashboard/key-manager.\n2. Configure your Jina AI credential in n8n to ensure secure API access."
+},
+"typeVersion": 1
+},
+{
+"id": "dbd204e0-da8e-41d8-814b-f409a23e9573",
+"name": "Sticky Note: OpenRouter API Setup",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1680,
+460
+],
+"parameters": {
+"color": 7,
+"width": 300,
+"height": 180,
+"content": "## OpenRouter API Setup Instructions\n1. Obtain your API key from https://openrouter.ai/settings/keys.\n2. Set up your OpenRouter credential in n8n for secure integration."
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "aa857bb3-84c1-4fe6-9464-90fc09163960",
+"connections": {
+"Chat Message Trigger": {
+"main": [
+[
+{
+"node": "Generate Search Queries using LLM",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse and Chunk JSON Data": {
+"main": [
+[
+{
+"node": "Split Data for SerpAPI Batching",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Fetch Wikipedia Information": {
+"ai_tool": [
+[
+{
+"node": "Generate Comprehensive Research Report",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Format SerpAPI Organic Results": {
+"main": [
+[
+{
+"node": "Split Data for Jina AI Batching",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Perform SerpAPI Search Request": {
+"main": [
+[
+{
+"node": "Split Data for SerpAPI Batching",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Data for Jina AI Batching": {
+"main": [
+[
+{
+"node": "Extract Relevant Context via LLM",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Perform Jina AI Analysis Request",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Data for SerpAPI Batching": {
+"main": [
+[
+{
+"node": "Format SerpAPI Organic Results",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Perform SerpAPI Search Request",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Relevant Context via LLM": {
+"main": [
+[
+{
+"node": "Generate Comprehensive Research Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Perform Jina AI Analysis Request": {
+"main": [
+[
+{
+"node": "Split Data for Jina AI Batching",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Generate Search Queries using LLM": {
+"main": [
+[
+{
+"node": "Parse and Chunk JSON Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"LLM Memory Buffer (Input Context)": {
+"ai_memory": [
+[
+{
+"node": "Extract Relevant Context via LLM",
+"type": "ai_memory",
+"index": 0
+}
+]
+]
+},
+"LLM Memory Buffer (Report Context)": {
+"ai_memory": [
+[
+{
+"node": "Generate Comprehensive Research Report",
+"type": "ai_memory",
+"index": 0
+}
+]
+]
+},
+"LLM Response Provider (OpenRouter)": {
+"ai_languageModel": [
+[
+{
+"node": "Generate Search Queries using LLM",
+"type": "ai_languageModel",
+"index": 0
+},
+{
+"node": "Extract Relevant Context via LLM",
+"type": "ai_languageModel",
+"index": 0
+},
+{
+"node": "Generate Comprehensive Research Report",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.json b/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.json
deleted file mode 100644
index 94cd0b1..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.json
+++ /dev/null
@@ -1,176 +0,0 @@
-{
- "nodes": [
- {
- "id": "293b70f0-06e8-4db5-befd-bfaed1f3575a",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -460,
- 80
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "1c473546-6280-412d-9f8e-b43962365d78",
- "name": "Set Params",
- "type": "n8n-nodes-base.set",
- "position": [
- -160,
- -60
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "8b5c6ca0-5ca8-4f67-abc1-44341cf419bc",
- "name": "system_prompt",
- "type": "string",
- "value": "You are an n8n fanboy."
- },
- {
- "id": "7c36c362-6269-4564-b6fe-f82126bc8f5e",
- "name": "user_prompt",
- "type": "string",
- "value": "What are the differences between n8n and Make?"
- },
- {
- "id": "4366d2b5-ad22-445a-8589-fddab1caa1ab",
- "name": "domains",
- "type": "string",
- "value": "n8n.io, make.com"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "894bd6a4-5db7-45fb-a8e0-1a81af068bbf",
- "name": "Clean Output",
- "type": "n8n-nodes-base.set",
- "position": [
- 580,
- -100
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "5859093c-6b22-41db-ac6c-9a9f6f18b7e3",
- "name": "output",
- "type": "string",
- "value": "={{ $json.choices[0].message.content }}"
- },
- {
- "id": "13208fff-5153-45a7-a1cb-fe49e32d9a03",
- "name": "citations",
- "type": "array",
- "value": "={{ $json.citations }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "52d3a832-8c9b-4356-ad2a-377340678a58",
- "name": "Perplexity Request",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 240,
- 40
- ],
- "parameters": {
- "url": "https://api.perplexity.ai/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"sonar\",\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"{{ $json.system_prompt }}\"\n },\n {\n \"role\": \"user\",\n \"content\": \"{{ $json.user_prompt }}\"\n }\n ],\n \"temperature\": 0.2,\n \"top_p\": 0.9,\n \"search_domain_filter\": {{ (JSON.stringify($json.domains.split(','))) }},\n \"return_images\": false,\n \"return_related_questions\": false,\n \"search_recency_filter\": \"month\",\n \"top_k\": 0,\n \"stream\": false,\n \"presence_penalty\": 0,\n \"frequency_penalty\": 1,\n \"response_format\": null\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpBasicAuth": {
- "id": "yEocL0NSpUWzMsHG",
- "name": "Perplexity"
- },
- "httpHeaderAuth": {
- "id": "TngzgS09J1YvLIXl",
- "name": "Perplexity"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "48657f2c-d1dd-4d7e-8014-c27748e63e58",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -140,
- -440
- ],
- "parameters": {
- "width": 480,
- "height": 300,
- "content": "## Credentials Setup\n\n1/ Go to the perplexity dashboard, purchase some credits and create an API Key\n\nhttps://www.perplexity.ai/settings/api\n\n2/ In the perplexity Request node, use Generic Credentials, Header Auth. \n\nFor the name, use the value \"Authorization\"\nAnd for the value \"Bearer pplx-e4...59ea\" (Your Perplexity Api Key)\n\n"
- },
- "typeVersion": 1
- },
- {
- "id": "e0daabee-c145-469e-93c2-c759c303dc2a",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 100,
- 260
- ],
- "parameters": {
- "color": 5,
- "width": 480,
- "height": 120,
- "content": "**Sonar Pro** is the current top model used by perplexity. \nIf you want to use a different one, check this page: \n\nhttps://docs.perplexity.ai/guides/model-cards"
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Set Params": {
- "main": [
- [
- {
- "node": "Perplexity Request",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Perplexity Request": {
- "main": [
- [
- {
- "node": "Clean Output",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Set Params",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.txt b/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.txt
new file mode 100644
index 0000000..1dc5e71
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Query Perplexity AI from your n8n workflows.txt
@@ -0,0 +1,176 @@
+{
+"nodes": [
+{
+"id": "293b70f0-06e8-4db5-befd-bfaed1f3575a",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-460,
+80
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "1c473546-6280-412d-9f8e-b43962365d78",
+"name": "Set Params",
+"type": "n8n-nodes-base.set",
+"position": [
+-160,
+-60
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "8b5c6ca0-5ca8-4f67-abc1-44341cf419bc",
+"name": "system_prompt",
+"type": "string",
+"value": "You are an n8n fanboy."
+},
+{
+"id": "7c36c362-6269-4564-b6fe-f82126bc8f5e",
+"name": "user_prompt",
+"type": "string",
+"value": "What are the differences between n8n and Make?"
+},
+{
+"id": "4366d2b5-ad22-445a-8589-fddab1caa1ab",
+"name": "domains",
+"type": "string",
+"value": "n8n.io, make.com"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "894bd6a4-5db7-45fb-a8e0-1a81af068bbf",
+"name": "Clean Output",
+"type": "n8n-nodes-base.set",
+"position": [
+580,
+-100
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "5859093c-6b22-41db-ac6c-9a9f6f18b7e3",
+"name": "output",
+"type": "string",
+"value": "={{ $json.choices[0].message.content }}"
+},
+{
+"id": "13208fff-5153-45a7-a1cb-fe49e32d9a03",
+"name": "citations",
+"type": "array",
+"value": "={{ $json.citations }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "52d3a832-8c9b-4356-ad2a-377340678a58",
+"name": "Perplexity Request",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+240,
+40
+],
+"parameters": {
+"url": "https://api.perplexity.ai/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"sonar\",\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"{{ $json.system_prompt }}\"\n },\n {\n \"role\": \"user\",\n \"content\": \"{{ $json.user_prompt }}\"\n }\n ],\n \"temperature\": 0.2,\n \"top_p\": 0.9,\n \"search_domain_filter\": {{ (JSON.stringify($json.domains.split(','))) }},\n \"return_images\": false,\n \"return_related_questions\": false,\n \"search_recency_filter\": \"month\",\n \"top_k\": 0,\n \"stream\": false,\n \"presence_penalty\": 0,\n \"frequency_penalty\": 1,\n \"response_format\": null\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpBasicAuth": {
+"id": "yEocL0NSpUWzMsHG",
+"name": "Perplexity"
+},
+"httpHeaderAuth": {
+"id": "TngzgS09J1YvLIXl",
+"name": "Perplexity"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "48657f2c-d1dd-4d7e-8014-c27748e63e58",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-140,
+-440
+],
+"parameters": {
+"width": 480,
+"height": 300,
+"content": "## Credentials Setup\n\n1/ Go to the perplexity dashboard, purchase some credits and create an API Key\n\nhttps://www.perplexity.ai/settings/api\n\n2/ In the perplexity Request node, use Generic Credentials, Header Auth. \n\nFor the name, use the value \"Authorization\"\nAnd for the value \"Bearer pplx-e4...59ea\" (Your Perplexity Api Key)\n\n"
+},
+"typeVersion": 1
+},
+{
+"id": "e0daabee-c145-469e-93c2-c759c303dc2a",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+100,
+260
+],
+"parameters": {
+"color": 5,
+"width": 480,
+"height": 120,
+"content": "**Sonar Pro** is the current top model used by perplexity. \nIf you want to use a different one, check this page: \n\nhttps://docs.perplexity.ai/guides/model-cards"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Set Params": {
+"main": [
+[
+{
+"node": "Perplexity Request",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Perplexity Request": {
+"main": [
+[
+{
+"node": "Clean Output",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Set Params",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.json b/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.json
deleted file mode 100644
index 74345a8..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.json
+++ /dev/null
@@ -1,973 +0,0 @@
-{
- "meta": {
- "instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
- },
- "nodes": [
- {
- "id": "1eb82902-a1d6-4eff-82a2-26908a82cea2",
- "name": "When clicking \"Test workflow\"",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 720,
- 320
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "e0031fc3-27f1-45d9-910b-4c07dd322115",
- "name": "Get This Week's Menu",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 992,
- 370
- ],
- "parameters": {
- "url": "=https://www.hellofresh.co.uk/menus/{{ $now.year }}-W{{ $now.weekNumber }}",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "2c556cc7-7d4e-4d80-902f-9686e756ed8c",
- "name": "Extract Available Courses",
- "type": "n8n-nodes-base.code",
- "position": [
- 992,
- 650
- ],
- "parameters": {
- "jsCode": "const pageData = JSON.parse($input.first().json.data)\nreturn pageData.props.pageProps.ssrPayload.courses.slice(0, 10);"
- },
- "typeVersion": 2
- },
- {
- "id": "90c39db6-6116-4c37-8d48-a6d5e8f8c777",
- "name": "Extract Server Data",
- "type": "n8n-nodes-base.html",
- "position": [
- 992,
- 510
- ],
- "parameters": {
- "options": {
- "trimValues": false,
- "cleanUpText": true
- },
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "data",
- "cssSelector": "script#__NEXT_DATA__"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "fbd4ed97-0154-4991-bf16-d9c4cb3f4776",
- "name": "Get Course Metadata",
- "type": "n8n-nodes-base.set",
- "position": [
- 1172,
- 370
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "3c90fd1e-e9ac-49c1-a459-7cff8c87fe8d",
- "name": "name",
- "type": "string",
- "value": "={{ $json.recipe.name }}"
- },
- {
- "id": "c4f3a5df-346c-4e8d-90ba-a49ed6afdedf",
- "name": "cuisines",
- "type": "array",
- "value": "={{ $json.recipe.cuisines.map(item => item.name) }}"
- },
- {
- "id": "97917928-0956-497b-bb68-507df1783240",
- "name": "category",
- "type": "string",
- "value": "={{ $json.recipe.category.name }}"
- },
- {
- "id": "1e84cf1e-7ad7-4888-9606-d3f7a310ce5f",
- "name": "tags",
- "type": "array",
- "value": "={{ $json.recipe.tags.flatMap(tag => tag.preferences) }}"
- },
- {
- "id": "cf6e2174-e8cb-4935-8303-2f8ed067f510",
- "name": "nutrition",
- "type": "object",
- "value": "={{ $json.recipe.nutrition.reduce((acc,item) => ({ ...acc, [item.name]: item.amount + item.unit }), {}) }}"
- },
- {
- "id": "25ba3fe6-c2fa-4315-a2cb-112ec7e3620f",
- "name": "url",
- "type": "string",
- "value": "={{ $json.recipe.websiteUrl }}"
- },
- {
- "id": "8f444fb3-c2ee-4254-b505-440cca3c7b8b",
- "name": "id",
- "type": "string",
- "value": "={{ $json.recipe.id }}"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "5ab1a5fa-adc3-41e0-be6d-f680af301aca",
- "name": "Get Recipe",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1172,
- 510
- ],
- "parameters": {
- "url": "={{ $json.recipe.websiteUrl }}",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "5014dc62-8320-4968-b9bd-396a517a2b5c",
- "name": "Embeddings Mistral Cloud",
- "type": "@n8n/n8n-nodes-langchain.embeddingsMistralCloud",
- "position": [
- 1960,
- 420
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "mistralCloudApi": {
- "id": "EIl2QxhXAS9Hkg37",
- "name": "Mistral Cloud account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "2a8fad89-f74b-4808-8cb6-97c6b46a53ee",
- "name": "Default Data Loader",
- "type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
- "position": [
- 2080,
- 420
- ],
- "parameters": {
- "options": {
- "metadata": {
- "metadataValues": [
- {
- "name": "week",
- "value": "={{ $json.week }}"
- },
- {
- "name": "cuisine",
- "value": "={{ $json.cuisines }}"
- },
- {
- "name": "category",
- "value": "={{ $json.category }}"
- },
- {
- "name": "tag",
- "value": "={{ $json.tags }}"
- },
- {
- "name": "recipe_id",
- "value": "={{ $json.id }}"
- }
- ]
- }
- },
- "jsonData": "={{ $json.data }}",
- "jsonMode": "expressionData"
- },
- "typeVersion": 1
- },
- {
- "id": "44ceef5c-1d08-40d2-8ab4-227b551f72f5",
- "name": "Merge Course & Recipe",
- "type": "n8n-nodes-base.merge",
- "position": [
- 1480,
- 500
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combinationMode": "mergeByPosition"
- },
- "typeVersion": 2.1
- },
- {
- "id": "b56bd85e-f182-49d1-aeb1-062e905c316a",
- "name": "Prepare Documents",
- "type": "n8n-nodes-base.set",
- "position": [
- 1660,
- 500
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "462567fe-02ec-4747-ae33-407d2bc6d776",
- "name": "data",
- "type": "string",
- "value": "=# {{ $json.name }}\n{{ $json.description.replaceAll('\\n\\n','\\n') }}\n\n# Website\n{{ $json.url }}\n\n## Ingredients\n{{ $json.ingredients.replaceAll('\\n\\n','\\n') }}\n\n## Utensils\n{{ $json.utensils }}\n\n## Nutrition\n{{ Object.keys($json.nutrition).map(key => `* ${key}: ${$json.nutrition[key]}`).join('\\n') }}\n\n## Instructions\n{{ $json.instructions.replaceAll('\\n\\n','\\n') }}"
- },
- {
- "id": "5738e420-abfe-4a85-b7ad-541cfc181563",
- "name": "cuisine",
- "type": "array",
- "value": "={{ $json.cuisines }}"
- },
- {
- "id": "349f46d4-e230-4da8-a118-50227ceb7233",
- "name": "category",
- "type": "string",
- "value": "={{ $json.category }}"
- },
- {
- "id": "9588b347-4469-4aa5-93a2-e7bf41b4c468",
- "name": "tag",
- "type": "array",
- "value": "={{ $json.tags }}"
- },
- {
- "id": "7ddab229-fa52-4d27-84e1-83ed47280d29",
- "name": "week",
- "type": "string",
- "value": "={{ $now.year }}-W{{ $now.weekNumber }}"
- },
- {
- "id": "13163e45-5699-4d25-af3d-4c7910dd2926",
- "name": "id",
- "type": "string",
- "value": "={{ $json.id }}"
- },
- {
- "id": "a0c5d599-ff2b-420d-9173-2baf9218abc5",
- "name": "name",
- "type": "string",
- "value": "={{ $json.name }}"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "6b800632-f320-4fc3-bd2a-6a062834343d",
- "name": "Recursive Character Text Splitter",
- "type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
- "position": [
- 2080,
- 560
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "df7f17a2-8b27-4203-a2ff-091aaf6609b8",
- "name": "Chat Trigger",
- "type": "@n8n/n8n-nodes-langchain.chatTrigger",
- "position": [
- 2440,
- 360
- ],
- "webhookId": "745056ec-2d36-4ac3-9c70-6ff0b1055d0a",
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "ee38effe-5929-421e-a3c5-b1055a755242",
- "name": "Extract Recipe Details",
- "type": "n8n-nodes-base.html",
- "position": [
- 1172,
- 650
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "description",
- "cssSelector": "[data-test-id=\"recipe-description\"]"
- },
- {
- "key": "ingredients",
- "cssSelector": "[data-test-id=\"ingredients-list\"]"
- },
- {
- "key": "utensils",
- "cssSelector": "[data-test-id=\"utensils\"]"
- },
- {
- "key": "instructions",
- "cssSelector": "[data-test-id=\"instructions\"]",
- "skipSelectors": "img,a"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "dede108f-2fde-49cb-8a0e-fa5786c59d4b",
- "name": "Qdrant Recommend API",
- "type": "@n8n/n8n-nodes-langchain.toolWorkflow",
- "position": [
- 2840,
- 540
- ],
- "parameters": {
- "name": "get_recipe_recommendation",
- "fields": {
- "values": [
- {
- "name": "week",
- "stringValue": "={{ $now.year }}-W{{ $now.weekNumber }}"
- }
- ]
- },
- "schemaType": "manual",
- "workflowId": "={{ $workflow.id }}",
- "description": "Call this tool to get a recipe recommendation. Pass in the following params as a json object:\n* positives - a description of what the user wants to cook. This could be ingredients, flavours, utensils available, number of diners, type of meal etc.\n* negatives - a description of what the user wants to avoid in the recipe. This could be flavours to avoid, allergen considerations, conflicts with theme of meal etc.",
- "inputSchema": "{\n\"type\": \"object\",\n\"properties\": {\n\t\"positive\": {\n\t\t\"type\": \"string\",\n\t\t\"description\": \"a description of what the user wants to cook. This could be ingredients, flavours, utensils available, number of diners, type of meal etc.\"\n\t},\n \"negative\": {\n \"type\": \"string\",\n \"description\": \"a description of what the user wants to avoid in the recipe. This could be flavours to avoid, allergen considerations, conflicts with theme of meal etc.\"\n }\n}\n}",
- "specifyInputSchema": true
- },
- "typeVersion": 1.1
- },
- {
- "id": "5e703134-4dd9-464b-9ec9-dc6103907a1e",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- 2420,
- 940
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "9fb5f4fd-3b38-4a35-8986-d3955754c8d1",
- "name": "Mistral Cloud Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatMistralCloud",
- "position": [
- 2660,
- 540
- ],
- "parameters": {
- "model": "mistral-large-2402",
- "options": {}
- },
- "credentials": {
- "mistralCloudApi": {
- "id": "EIl2QxhXAS9Hkg37",
- "name": "Mistral Cloud account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "d38275e6-aede-4f1c-9b05-018f3cf4faab",
- "name": "Get Tool Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 3160,
- 940
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "10b55200-4610-4e9b-8be7-d487c6b56a78",
- "name": "response",
- "type": "string",
- "value": "={{ JSON.stringify($json.result) }}"
- }
- ]
- }
- },
- "typeVersion": 3.3
- },
- {
- "id": "dc3ceb2f-3c64-4b42-aeca-ddcdb84abf12",
- "name": "Wait for Rate Limits",
- "type": "n8n-nodes-base.wait",
- "position": [
- 2420,
- 1080
- ],
- "webhookId": "e86d8ae4-3b0d-4c40-9d12-a11d6501a043",
- "parameters": {
- "amount": 1.1
- },
- "typeVersion": 1.1
- },
- {
- "id": "ec36d6f8-c3da-4732-8d56-a092a3358864",
- "name": "Get Mistral Embeddings",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2620,
- 940
- ],
- "parameters": {
- "url": "https://api.mistral.ai/v1/embeddings",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "authentication": "predefinedCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "model",
- "value": "mistral-embed"
- },
- {
- "name": "encoding_format",
- "value": "float"
- },
- {
- "name": "input",
- "value": "={{ [$json.query.positive, $json.query.negative].compact() }}"
- }
- ]
- },
- "nodeCredentialType": "mistralCloudApi"
- },
- "credentials": {
- "mistralCloudApi": {
- "id": "EIl2QxhXAS9Hkg37",
- "name": "Mistral Cloud account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "aebcb860-d25c-4833-9e9d-0297101259c7",
- "name": "Use Qdrant Recommend API",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2800,
- 940
- ],
- "parameters": {
- "url": "=http://qdrant:6333/collections/hello_fresh/points/recommend/groups",
- "method": "POST",
- "options": {},
- "sendBody": true,
- "authentication": "predefinedCredentialType",
- "bodyParameters": {
- "parameters": [
- {
- "name": "strategy",
- "value": "average_vector"
- },
- {
- "name": "limit",
- "value": "={{ 3 }}"
- },
- {
- "name": "positive",
- "value": "={{ [$json.data[0].embedding] }}"
- },
- {
- "name": "negative",
- "value": "={{ [$json.data[1].embedding] }}"
- },
- {
- "name": "filter",
- "value": "={{ { \"must\": {\"key\": \"metadata.week\", \"match\": { \"value\": $('Execute Workflow Trigger').item.json.week } } } }}"
- },
- {
- "name": "with_payload",
- "value": "={{ true }}"
- },
- {
- "name": "group_by",
- "value": "metadata.recipe_id"
- },
- {
- "name": "group_size",
- "value": "={{ 3 }}"
- }
- ]
- },
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "NyinAS3Pgfik66w5",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "2474c97d-0d85-4acc-a95e-2eb6494786dc",
- "name": "Get Recipes From DB",
- "type": "n8n-nodes-base.code",
- "position": [
- 2980,
- 940
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "import sqlite3\ncon = sqlite3.connect(\"hello_fresh_1.db\")\n\nrecipe_ids = list(set([group.id for group in _input.all()[0].json.result.groups if group.hits[0].score > 0.5]))\nplaceholders = ','.join(['?' for i in range(0,len(recipe_ids))])\n\ncur = con.cursor()\nres = cur.execute(f'SELECT * FROM recipes WHERE id IN ({placeholders})', recipe_ids)\nrows = res.fetchall()\n\ncon.close()\n\nreturn [{ \"result\": [row[2] for row in rows] }]"
- },
- "typeVersion": 2
- },
- {
- "id": "54229c2a-6e26-4350-8a94-57f415ef2340",
- "name": "Save Recipes to DB",
- "type": "n8n-nodes-base.code",
- "position": [
- 1960,
- 940
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "import sqlite3\ncon = sqlite3.connect(\"hello_fresh_1.db\")\n\ncur = con.cursor()\ncur.execute(\"CREATE TABLE IF NOT EXISTS recipes (id TEXT PRIMARY KEY, name TEXT, data TEXT, cuisine TEXT, category TEXT, tag TEXT, week TEXT);\")\n\nfor item in _input.all():\n cur.execute('INSERT OR REPLACE INTO recipes VALUES(?,?,?,?,?,?,?)', (\n item.json.id,\n item.json.name,\n item.json.data,\n ','.join(item.json.cuisine),\n item.json.category,\n ','.join(item.json.tag),\n item.json.week\n ))\n\ncon.commit()\ncon.close()\n\nreturn [{ \"affected_rows\": len(_input.all()) }]"
- },
- "typeVersion": 2
- },
- {
- "id": "725c1f56-5373-4891-92b9-3f32dd28892b",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 901.1666225087287,
- 180.99920515712074
- ],
- "parameters": {
- "color": 7,
- "width": 484.12381677448207,
- "height": 674.1153489831718,
- "content": "## Step 1. Fetch Available Courses For the Current Week\n\nTo populate our vectorstore, we'll scrape the weekly menu off the HelloFresh Website. The pages are quite large so may take a while so please be patient."
- },
- "typeVersion": 1
- },
- {
- "id": "f4e882b8-3762-4e6b-9e95-b0d708d0c284",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1420,
- 300
- ],
- "parameters": {
- "color": 7,
- "width": 409.1756468632768,
- "height": 398.81415970574335,
- "content": "## Step 2. Create Recipe Documents For VectorStore\n\nTo populate our vectorstore, we'll scrape the weekly menu off the HelloFresh Website. The pages are quite large so may take a while so please be patient."
- },
- "typeVersion": 1
- },
- {
- "id": "fc3c2221-b67c-451c-9096-d6acd2a297fa",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1860,
- 19.326425127730317
- ],
- "parameters": {
- "color": 7,
- "width": 486.02284096214964,
- "height": 690.7816167755491,
- "content": "## Step 3. Vectorise Recipes For Recommendation Engine\n[Read more about Qdrant node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreqdrant)\n\nWe'll store our documents in our Qdrant vectorstore by converting to vectors using Mistral Embed. Our goal is to a build a recommendation engine for meals of the week which Qdrant is a perfect solution."
- },
- "typeVersion": 1
- },
- {
- "id": "43296173-b929-46cc-b6ea-58007837b8df",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1740,
- 740
- ],
- "parameters": {
- "color": 7,
- "width": 547.0098868353456,
- "height": 347.6002738958705,
- "content": "## Step 4. Save Original Document to Database\n[Read more about Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nFinally, let's have the original document stored in a more traditional datastore. USually our vectorsearch will return partial docs and those are enough for many use-cases, however in this instance we'll pull the full docs for the Agent get the info required to make the recommendation. "
- },
- "typeVersion": 1
- },
- {
- "id": "6e2e58d2-e0ad-4503-8ed6-891124c8035b",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2380,
- 160
- ],
- "parameters": {
- "color": 7,
- "width": 673.6008766895472,
- "height": 552.9202706743265,
- "content": "## 5. Chat with Our HelloFresh Recommendation AI Agent\n[Read more about AI Agents](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent)\n\nThis agent is designed to recommend HelloFresh recipes based on your current taste preferences. Need something hot and spicy, warm and comforting or fast and chilled? This agent will capture what you would like and not like and queries our Recipe Recommendation engine powered by Qdrant Vectorstore."
- },
- "typeVersion": 1
- },
- {
- "id": "ba692c21-38bc-48a1-8b40-bad298be8b9e",
- "name": "AI Agent",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 2660,
- 360
- ],
- "parameters": {
- "options": {
- "systemMessage": "=You are a recipe bot for the company, \"Hello fresh\". You will help the user choose which Hello Fresh recipe to choose from this week's menu. The current week is {{ $now.year }}-W{{ $now.weekNumber }}.\nDo not recommend any recipes other from the current week's menu. If there are no recipes to recommend, please ask the user to visit the website instead https://hellofresh.com."
- }
- },
- "typeVersion": 1.6
- },
- {
- "id": "d7ca0f97-72dc-4f4c-8b46-3ff57b9068a4",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2320,
- 740
- ],
- "parameters": {
- "color": 7,
- "width": 987.4785537889618,
- "height": 531.9173034334732,
- "content": "## 5. Using Qdrant's Recommend API & Grouping Functionality\n[Read more about Qdrant's Recommend API](https://qdrant.tech/documentation/concepts/explore/?q=recommend)\n\nUnlike basic similarity search, Qdrant's Recommend API takes a positive query to match against (eg. Roast Dinner) and a negative query to avoid (eg. Roast Chicken). This feature significantly improves results for a recommendation engine. Additionally, by utilising Qdrant's Grouping feature, we're able to group similar matches from the same recipe - meaning we can ensure unique recipes everytime."
- },
- "typeVersion": 1
- },
- {
- "id": "96a294e2-1437-4ded-9973-0999b444c999",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 440,
- -40
- ],
- "parameters": {
- "width": 432.916474478624,
- "height": 542.9295980774649,
- "content": "## Try it out!\n### This workflow does the following:\n* Fetches and stores this week's HelloFresh's menu\n* Builds the foundation of a recommendation engine by storing the recipes in a Qdrant Vectorstore and SQLite database.\n* Builds an AI Agent that allows for a chat interface to query for a the week's recipe recommendations.\n* AI agent uses the Qdrant Recommend API, providing what the user likes/dislikes as the query.\n* Qdrant returns the results which enable the AI Agent to make the recommendation to the user.\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
- },
- "typeVersion": 1
- },
- {
- "id": "72c98600-f21a-42d4-97be-836b8ef6dc77",
- "name": "Qdrant Vector Store",
- "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
- "position": [
- 1960,
- 240
- ],
- "parameters": {
- "mode": "insert",
- "options": {},
- "qdrantCollection": {
- "__rl": true,
- "mode": "list",
- "value": "hello_fresh",
- "cachedResultName": "hello_fresh"
- }
- },
- "credentials": {
- "qdrantApi": {
- "id": "NyinAS3Pgfik66w5",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "b7c4b597-ac2b-41d7-8f0f-1cbba25085de",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1860,
- -195.8987124522777
- ],
- "parameters": {
- "width": 382.47301504497716,
- "height": 195.8987124522777,
- "content": "### ๐จEnsure Qdrant collection exists!\nYou'll need to run the following command in Qdrant:\n```\nPUT collections/hello_fresh\n{\n \"vectors\": {\n \"distance\": \"Cosine\",\n \"size\": 1024\n }\n}\n```"
- },
- "typeVersion": 1
- },
- {
- "id": "39191834-ecc2-46f0-a31a-0a7e9c47ac5d",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2740,
- 920
- ],
- "parameters": {
- "width": 213.30551928619226,
- "height": 332.38559808882246,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n### ๐จConfigure Your Qdrant Connection\n* Be sure to enter your endpoint address"
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Get Recipe": {
- "main": [
- [
- {
- "node": "Extract Recipe Details",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Chat Trigger": {
- "main": [
- [
- {
- "node": "AI Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prepare Documents": {
- "main": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "main",
- "index": 0
- },
- {
- "node": "Save Recipes to DB",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Default Data Loader": {
- "ai_document": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "ai_document",
- "index": 0
- }
- ]
- ]
- },
- "Extract Server Data": {
- "main": [
- [
- {
- "node": "Extract Available Courses",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Course Metadata": {
- "main": [
- [
- {
- "node": "Merge Course & Recipe",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Recipes From DB": {
- "main": [
- [
- {
- "node": "Get Tool Response",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get This Week's Menu": {
- "main": [
- [
- {
- "node": "Extract Server Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant Recommend API": {
- "ai_tool": [
- [
- {
- "node": "AI Agent",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Wait for Rate Limits": {
- "main": [
- [
- {
- "node": "Get Mistral Embeddings",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Merge Course & Recipe": {
- "main": [
- [
- {
- "node": "Prepare Documents",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Recipe Details": {
- "main": [
- [
- {
- "node": "Merge Course & Recipe",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Get Mistral Embeddings": {
- "main": [
- [
- {
- "node": "Use Qdrant Recommend API",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Embeddings Mistral Cloud": {
- "ai_embedding": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "ai_embedding",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Wait for Rate Limits",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Mistral Cloud Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "AI Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Use Qdrant Recommend API": {
- "main": [
- [
- {
- "node": "Get Recipes From DB",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Available Courses": {
- "main": [
- [
- {
- "node": "Get Course Metadata",
- "type": "main",
- "index": 0
- },
- {
- "node": "Get Recipe",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking \"Test workflow\"": {
- "main": [
- [
- {
- "node": "Get This Week's Menu",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Recursive Character Text Splitter": {
- "ai_textSplitter": [
- [
- {
- "node": "Default Data Loader",
- "type": "ai_textSplitter",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.txt b/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.txt
new file mode 100644
index 0000000..e142bbc
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Recipe Recommendations with Qdrant and Mistral.txt
@@ -0,0 +1,973 @@
+{
+"meta": {
+"instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
+},
+"nodes": [
+{
+"id": "1eb82902-a1d6-4eff-82a2-26908a82cea2",
+"name": "When clicking \"Test workflow\"",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+720,
+320
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "e0031fc3-27f1-45d9-910b-4c07dd322115",
+"name": "Get This Week's Menu",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+992,
+370
+],
+"parameters": {
+"url": "=https://www.hellofresh.co.uk/menus/{{ $now.year }}-W{{ $now.weekNumber }}",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "2c556cc7-7d4e-4d80-902f-9686e756ed8c",
+"name": "Extract Available Courses",
+"type": "n8n-nodes-base.code",
+"position": [
+992,
+650
+],
+"parameters": {
+"jsCode": "const pageData = JSON.parse($input.first().json.data)\nreturn pageData.props.pageProps.ssrPayload.courses.slice(0, 10);"
+},
+"typeVersion": 2
+},
+{
+"id": "90c39db6-6116-4c37-8d48-a6d5e8f8c777",
+"name": "Extract Server Data",
+"type": "n8n-nodes-base.html",
+"position": [
+992,
+510
+],
+"parameters": {
+"options": {
+"trimValues": false,
+"cleanUpText": true
+},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "data",
+"cssSelector": "script#__NEXT_DATA__"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "fbd4ed97-0154-4991-bf16-d9c4cb3f4776",
+"name": "Get Course Metadata",
+"type": "n8n-nodes-base.set",
+"position": [
+1172,
+370
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "3c90fd1e-e9ac-49c1-a459-7cff8c87fe8d",
+"name": "name",
+"type": "string",
+"value": "={{ $json.recipe.name }}"
+},
+{
+"id": "c4f3a5df-346c-4e8d-90ba-a49ed6afdedf",
+"name": "cuisines",
+"type": "array",
+"value": "={{ $json.recipe.cuisines.map(item => item.name) }}"
+},
+{
+"id": "97917928-0956-497b-bb68-507df1783240",
+"name": "category",
+"type": "string",
+"value": "={{ $json.recipe.category.name }}"
+},
+{
+"id": "1e84cf1e-7ad7-4888-9606-d3f7a310ce5f",
+"name": "tags",
+"type": "array",
+"value": "={{ $json.recipe.tags.flatMap(tag => tag.preferences) }}"
+},
+{
+"id": "cf6e2174-e8cb-4935-8303-2f8ed067f510",
+"name": "nutrition",
+"type": "object",
+"value": "={{ $json.recipe.nutrition.reduce((acc,item) => ({ ...acc, [item.name]: item.amount + item.unit }), {}) }}"
+},
+{
+"id": "25ba3fe6-c2fa-4315-a2cb-112ec7e3620f",
+"name": "url",
+"type": "string",
+"value": "={{ $json.recipe.websiteUrl }}"
+},
+{
+"id": "8f444fb3-c2ee-4254-b505-440cca3c7b8b",
+"name": "id",
+"type": "string",
+"value": "={{ $json.recipe.id }}"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "5ab1a5fa-adc3-41e0-be6d-f680af301aca",
+"name": "Get Recipe",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1172,
+510
+],
+"parameters": {
+"url": "={{ $json.recipe.websiteUrl }}",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "5014dc62-8320-4968-b9bd-396a517a2b5c",
+"name": "Embeddings Mistral Cloud",
+"type": "@n8n/n8n-nodes-langchain.embeddingsMistralCloud",
+"position": [
+1960,
+420
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"mistralCloudApi": {
+"id": "EIl2QxhXAS9Hkg37",
+"name": "Mistral Cloud account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "2a8fad89-f74b-4808-8cb6-97c6b46a53ee",
+"name": "Default Data Loader",
+"type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
+"position": [
+2080,
+420
+],
+"parameters": {
+"options": {
+"metadata": {
+"metadataValues": [
+{
+"name": "week",
+"value": "={{ $json.week }}"
+},
+{
+"name": "cuisine",
+"value": "={{ $json.cuisines }}"
+},
+{
+"name": "category",
+"value": "={{ $json.category }}"
+},
+{
+"name": "tag",
+"value": "={{ $json.tags }}"
+},
+{
+"name": "recipe_id",
+"value": "={{ $json.id }}"
+}
+]
+}
+},
+"jsonData": "={{ $json.data }}",
+"jsonMode": "expressionData"
+},
+"typeVersion": 1
+},
+{
+"id": "44ceef5c-1d08-40d2-8ab4-227b551f72f5",
+"name": "Merge Course & Recipe",
+"type": "n8n-nodes-base.merge",
+"position": [
+1480,
+500
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combinationMode": "mergeByPosition"
+},
+"typeVersion": 2.1
+},
+{
+"id": "b56bd85e-f182-49d1-aeb1-062e905c316a",
+"name": "Prepare Documents",
+"type": "n8n-nodes-base.set",
+"position": [
+1660,
+500
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "462567fe-02ec-4747-ae33-407d2bc6d776",
+"name": "data",
+"type": "string",
+"value": "=# {{ $json.name }}\n{{ $json.description.replaceAll('\\n\\n','\\n') }}\n\n# Website\n{{ $json.url }}\n\n## Ingredients\n{{ $json.ingredients.replaceAll('\\n\\n','\\n') }}\n\n## Utensils\n{{ $json.utensils }}\n\n## Nutrition\n{{ Object.keys($json.nutrition).map(key => `* ${key}: ${$json.nutrition[key]}`).join('\\n') }}\n\n## Instructions\n{{ $json.instructions.replaceAll('\\n\\n','\\n') }}"
+},
+{
+"id": "5738e420-abfe-4a85-b7ad-541cfc181563",
+"name": "cuisine",
+"type": "array",
+"value": "={{ $json.cuisines }}"
+},
+{
+"id": "349f46d4-e230-4da8-a118-50227ceb7233",
+"name": "category",
+"type": "string",
+"value": "={{ $json.category }}"
+},
+{
+"id": "9588b347-4469-4aa5-93a2-e7bf41b4c468",
+"name": "tag",
+"type": "array",
+"value": "={{ $json.tags }}"
+},
+{
+"id": "7ddab229-fa52-4d27-84e1-83ed47280d29",
+"name": "week",
+"type": "string",
+"value": "={{ $now.year }}-W{{ $now.weekNumber }}"
+},
+{
+"id": "13163e45-5699-4d25-af3d-4c7910dd2926",
+"name": "id",
+"type": "string",
+"value": "={{ $json.id }}"
+},
+{
+"id": "a0c5d599-ff2b-420d-9173-2baf9218abc5",
+"name": "name",
+"type": "string",
+"value": "={{ $json.name }}"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "6b800632-f320-4fc3-bd2a-6a062834343d",
+"name": "Recursive Character Text Splitter",
+"type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
+"position": [
+2080,
+560
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "df7f17a2-8b27-4203-a2ff-091aaf6609b8",
+"name": "Chat Trigger",
+"type": "@n8n/n8n-nodes-langchain.chatTrigger",
+"position": [
+2440,
+360
+],
+"webhookId": "745056ec-2d36-4ac3-9c70-6ff0b1055d0a",
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "ee38effe-5929-421e-a3c5-b1055a755242",
+"name": "Extract Recipe Details",
+"type": "n8n-nodes-base.html",
+"position": [
+1172,
+650
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "description",
+"cssSelector": "[data-test-id=\"recipe-description\"]"
+},
+{
+"key": "ingredients",
+"cssSelector": "[data-test-id=\"ingredients-list\"]"
+},
+{
+"key": "utensils",
+"cssSelector": "[data-test-id=\"utensils\"]"
+},
+{
+"key": "instructions",
+"cssSelector": "[data-test-id=\"instructions\"]",
+"skipSelectors": "img,a"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "dede108f-2fde-49cb-8a0e-fa5786c59d4b",
+"name": "Qdrant Recommend API",
+"type": "@n8n/n8n-nodes-langchain.toolWorkflow",
+"position": [
+2840,
+540
+],
+"parameters": {
+"name": "get_recipe_recommendation",
+"fields": {
+"values": [
+{
+"name": "week",
+"stringValue": "={{ $now.year }}-W{{ $now.weekNumber }}"
+}
+]
+},
+"schemaType": "manual",
+"workflowId": "={{ $workflow.id }}",
+"description": "Call this tool to get a recipe recommendation. Pass in the following params as a json object:\n* positives - a description of what the user wants to cook. This could be ingredients, flavours, utensils available, number of diners, type of meal etc.\n* negatives - a description of what the user wants to avoid in the recipe. This could be flavours to avoid, allergen considerations, conflicts with theme of meal etc.",
+"inputSchema": "{\n\"type\": \"object\",\n\"properties\": {\n\t\"positive\": {\n\t\t\"type\": \"string\",\n\t\t\"description\": \"a description of what the user wants to cook. This could be ingredients, flavours, utensils available, number of diners, type of meal etc.\"\n\t},\n \"negative\": {\n \"type\": \"string\",\n \"description\": \"a description of what the user wants to avoid in the recipe. This could be flavours to avoid, allergen considerations, conflicts with theme of meal etc.\"\n }\n}\n}",
+"specifyInputSchema": true
+},
+"typeVersion": 1.1
+},
+{
+"id": "5e703134-4dd9-464b-9ec9-dc6103907a1e",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+2420,
+940
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "9fb5f4fd-3b38-4a35-8986-d3955754c8d1",
+"name": "Mistral Cloud Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatMistralCloud",
+"position": [
+2660,
+540
+],
+"parameters": {
+"model": "mistral-large-2402",
+"options": {}
+},
+"credentials": {
+"mistralCloudApi": {
+"id": "EIl2QxhXAS9Hkg37",
+"name": "Mistral Cloud account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "d38275e6-aede-4f1c-9b05-018f3cf4faab",
+"name": "Get Tool Response",
+"type": "n8n-nodes-base.set",
+"position": [
+3160,
+940
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "10b55200-4610-4e9b-8be7-d487c6b56a78",
+"name": "response",
+"type": "string",
+"value": "={{ JSON.stringify($json.result) }}"
+}
+]
+}
+},
+"typeVersion": 3.3
+},
+{
+"id": "dc3ceb2f-3c64-4b42-aeca-ddcdb84abf12",
+"name": "Wait for Rate Limits",
+"type": "n8n-nodes-base.wait",
+"position": [
+2420,
+1080
+],
+"webhookId": "e86d8ae4-3b0d-4c40-9d12-a11d6501a043",
+"parameters": {
+"amount": 1.1
+},
+"typeVersion": 1.1
+},
+{
+"id": "ec36d6f8-c3da-4732-8d56-a092a3358864",
+"name": "Get Mistral Embeddings",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2620,
+940
+],
+"parameters": {
+"url": "https://api.mistral.ai/v1/embeddings",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"authentication": "predefinedCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "model",
+"value": "mistral-embed"
+},
+{
+"name": "encoding_format",
+"value": "float"
+},
+{
+"name": "input",
+"value": "={{ [$json.query.positive, $json.query.negative].compact() }}"
+}
+]
+},
+"nodeCredentialType": "mistralCloudApi"
+},
+"credentials": {
+"mistralCloudApi": {
+"id": "EIl2QxhXAS9Hkg37",
+"name": "Mistral Cloud account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "aebcb860-d25c-4833-9e9d-0297101259c7",
+"name": "Use Qdrant Recommend API",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2800,
+940
+],
+"parameters": {
+"url": "=http://qdrant:6333/collections/hello_fresh/points/recommend/groups",
+"method": "POST",
+"options": {},
+"sendBody": true,
+"authentication": "predefinedCredentialType",
+"bodyParameters": {
+"parameters": [
+{
+"name": "strategy",
+"value": "average_vector"
+},
+{
+"name": "limit",
+"value": "={{ 3 }}"
+},
+{
+"name": "positive",
+"value": "={{ [$json.data[0].embedding] }}"
+},
+{
+"name": "negative",
+"value": "={{ [$json.data[1].embedding] }}"
+},
+{
+"name": "filter",
+"value": "={{ { \"must\": {\"key\": \"metadata.week\", \"match\": { \"value\": $('Execute Workflow Trigger').item.json.week } } } }}"
+},
+{
+"name": "with_payload",
+"value": "={{ true }}"
+},
+{
+"name": "group_by",
+"value": "metadata.recipe_id"
+},
+{
+"name": "group_size",
+"value": "={{ 3 }}"
+}
+]
+},
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "NyinAS3Pgfik66w5",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "2474c97d-0d85-4acc-a95e-2eb6494786dc",
+"name": "Get Recipes From DB",
+"type": "n8n-nodes-base.code",
+"position": [
+2980,
+940
+],
+"parameters": {
+"language": "python",
+"pythonCode": "import sqlite3\ncon = sqlite3.connect(\"hello_fresh_1.db\")\n\nrecipe_ids = list(set([group.id for group in _input.all()[0].json.result.groups if group.hits[0].score > 0.5]))\nplaceholders = ','.join(['?' for i in range(0,len(recipe_ids))])\n\ncur = con.cursor()\nres = cur.execute(f'SELECT * FROM recipes WHERE id IN ({placeholders})', recipe_ids)\nrows = res.fetchall()\n\ncon.close()\n\nreturn [{ \"result\": [row[2] for row in rows] }]"
+},
+"typeVersion": 2
+},
+{
+"id": "54229c2a-6e26-4350-8a94-57f415ef2340",
+"name": "Save Recipes to DB",
+"type": "n8n-nodes-base.code",
+"position": [
+1960,
+940
+],
+"parameters": {
+"language": "python",
+"pythonCode": "import sqlite3\ncon = sqlite3.connect(\"hello_fresh_1.db\")\n\ncur = con.cursor()\ncur.execute(\"CREATE TABLE IF NOT EXISTS recipes (id TEXT PRIMARY KEY, name TEXT, data TEXT, cuisine TEXT, category TEXT, tag TEXT, week TEXT);\")\n\nfor item in _input.all():\n cur.execute('INSERT OR REPLACE INTO recipes VALUES(?,?,?,?,?,?,?)', (\n item.json.id,\n item.json.name,\n item.json.data,\n ','.join(item.json.cuisine),\n item.json.category,\n ','.join(item.json.tag),\n item.json.week\n ))\n\ncon.commit()\ncon.close()\n\nreturn [{ \"affected_rows\": len(_input.all()) }]"
+},
+"typeVersion": 2
+},
+{
+"id": "725c1f56-5373-4891-92b9-3f32dd28892b",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+901.1666225087287,
+180.99920515712074
+],
+"parameters": {
+"color": 7,
+"width": 484.12381677448207,
+"height": 674.1153489831718,
+"content": "## Step 1. Fetch Available Courses For the Current Week\n\nTo populate our vectorstore, we'll scrape the weekly menu off the HelloFresh Website. The pages are quite large so may take a while so please be patient."
+},
+"typeVersion": 1
+},
+{
+"id": "f4e882b8-3762-4e6b-9e95-b0d708d0c284",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1420,
+300
+],
+"parameters": {
+"color": 7,
+"width": 409.1756468632768,
+"height": 398.81415970574335,
+"content": "## Step 2. Create Recipe Documents For VectorStore\n\nTo populate our vectorstore, we'll scrape the weekly menu off the HelloFresh Website. The pages are quite large so may take a while so please be patient."
+},
+"typeVersion": 1
+},
+{
+"id": "fc3c2221-b67c-451c-9096-d6acd2a297fa",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1860,
+19.326425127730317
+],
+"parameters": {
+"color": 7,
+"width": 486.02284096214964,
+"height": 690.7816167755491,
+"content": "## Step 3. Vectorise Recipes For Recommendation Engine\n[Read more about Qdrant node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreqdrant)\n\nWe'll store our documents in our Qdrant vectorstore by converting to vectors using Mistral Embed. Our goal is to a build a recommendation engine for meals of the week which Qdrant is a perfect solution."
+},
+"typeVersion": 1
+},
+{
+"id": "43296173-b929-46cc-b6ea-58007837b8df",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1740,
+740
+],
+"parameters": {
+"color": 7,
+"width": 547.0098868353456,
+"height": 347.6002738958705,
+"content": "## Step 4. Save Original Document to Database\n[Read more about Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nFinally, let's have the original document stored in a more traditional datastore. USually our vectorsearch will return partial docs and those are enough for many use-cases, however in this instance we'll pull the full docs for the Agent get the info required to make the recommendation. "
+},
+"typeVersion": 1
+},
+{
+"id": "6e2e58d2-e0ad-4503-8ed6-891124c8035b",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2380,
+160
+],
+"parameters": {
+"color": 7,
+"width": 673.6008766895472,
+"height": 552.9202706743265,
+"content": "## 5. Chat with Our HelloFresh Recommendation AI Agent\n[Read more about AI Agents](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent)\n\nThis agent is designed to recommend HelloFresh recipes based on your current taste preferences. Need something hot and spicy, warm and comforting or fast and chilled? This agent will capture what you would like and not like and queries our Recipe Recommendation engine powered by Qdrant Vectorstore."
+},
+"typeVersion": 1
+},
+{
+"id": "ba692c21-38bc-48a1-8b40-bad298be8b9e",
+"name": "AI Agent",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+2660,
+360
+],
+"parameters": {
+"options": {
+"systemMessage": "=You are a recipe bot for the company, \"Hello fresh\". You will help the user choose which Hello Fresh recipe to choose from this week's menu. The current week is {{ $now.year }}-W{{ $now.weekNumber }}.\nDo not recommend any recipes other from the current week's menu. If there are no recipes to recommend, please ask the user to visit the website instead https://hellofresh.com."
+}
+},
+"typeVersion": 1.6
+},
+{
+"id": "d7ca0f97-72dc-4f4c-8b46-3ff57b9068a4",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2320,
+740
+],
+"parameters": {
+"color": 7,
+"width": 987.4785537889618,
+"height": 531.9173034334732,
+"content": "## 5. Using Qdrant's Recommend API & Grouping Functionality\n[Read more about Qdrant's Recommend API](https://qdrant.tech/documentation/concepts/explore/?q=recommend)\n\nUnlike basic similarity search, Qdrant's Recommend API takes a positive query to match against (eg. Roast Dinner) and a negative query to avoid (eg. Roast Chicken). This feature significantly improves results for a recommendation engine. Additionally, by utilising Qdrant's Grouping feature, we're able to group similar matches from the same recipe - meaning we can ensure unique recipes everytime."
+},
+"typeVersion": 1
+},
+{
+"id": "96a294e2-1437-4ded-9973-0999b444c999",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+440,
+-40
+],
+"parameters": {
+"width": 432.916474478624,
+"height": 542.9295980774649,
+"content": "## Try it out!\n### This workflow does the following:\n* Fetches and stores this week's HelloFresh's menu\n* Builds the foundation of a recommendation engine by storing the recipes in a Qdrant Vectorstore and SQLite database.\n* Builds an AI Agent that allows for a chat interface to query for a the week's recipe recommendations.\n* AI agent uses the Qdrant Recommend API, providing what the user likes/dislikes as the query.\n* Qdrant returns the results which enable the AI Agent to make the recommendation to the user.\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "72c98600-f21a-42d4-97be-836b8ef6dc77",
+"name": "Qdrant Vector Store",
+"type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
+"position": [
+1960,
+240
+],
+"parameters": {
+"mode": "insert",
+"options": {},
+"qdrantCollection": {
+"__rl": true,
+"mode": "list",
+"value": "hello_fresh",
+"cachedResultName": "hello_fresh"
+}
+},
+"credentials": {
+"qdrantApi": {
+"id": "NyinAS3Pgfik66w5",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b7c4b597-ac2b-41d7-8f0f-1cbba25085de",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1860,
+-195.8987124522777
+],
+"parameters": {
+"width": 382.47301504497716,
+"height": 195.8987124522777,
+"content": "### ๐จEnsure Qdrant collection exists!\nYou'll need to run the following command in Qdrant:\n```\nPUT collections/hello_fresh\n{\n \"vectors\": {\n \"distance\": \"Cosine\",\n \"size\": 1024\n }\n}\n```"
+},
+"typeVersion": 1
+},
+{
+"id": "39191834-ecc2-46f0-a31a-0a7e9c47ac5d",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2740,
+920
+],
+"parameters": {
+"width": 213.30551928619226,
+"height": 332.38559808882246,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n### ๐จConfigure Your Qdrant Connection\n* Be sure to enter your endpoint address"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Get Recipe": {
+"main": [
+[
+{
+"node": "Extract Recipe Details",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Chat Trigger": {
+"main": [
+[
+{
+"node": "AI Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prepare Documents": {
+"main": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "main",
+"index": 0
+},
+{
+"node": "Save Recipes to DB",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Default Data Loader": {
+"ai_document": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "ai_document",
+"index": 0
+}
+]
+]
+},
+"Extract Server Data": {
+"main": [
+[
+{
+"node": "Extract Available Courses",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Course Metadata": {
+"main": [
+[
+{
+"node": "Merge Course & Recipe",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Recipes From DB": {
+"main": [
+[
+{
+"node": "Get Tool Response",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get This Week's Menu": {
+"main": [
+[
+{
+"node": "Extract Server Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Qdrant Recommend API": {
+"ai_tool": [
+[
+{
+"node": "AI Agent",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Wait for Rate Limits": {
+"main": [
+[
+{
+"node": "Get Mistral Embeddings",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Merge Course & Recipe": {
+"main": [
+[
+{
+"node": "Prepare Documents",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Recipe Details": {
+"main": [
+[
+{
+"node": "Merge Course & Recipe",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Get Mistral Embeddings": {
+"main": [
+[
+{
+"node": "Use Qdrant Recommend API",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Embeddings Mistral Cloud": {
+"ai_embedding": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "ai_embedding",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Wait for Rate Limits",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Mistral Cloud Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "AI Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Use Qdrant Recommend API": {
+"main": [
+[
+{
+"node": "Get Recipes From DB",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Available Courses": {
+"main": [
+[
+{
+"node": "Get Course Metadata",
+"type": "main",
+"index": 0
+},
+{
+"node": "Get Recipe",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking \"Test workflow\"": {
+"main": [
+[
+{
+"node": "Get This Week's Menu",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Recursive Character Text Splitter": {
+"ai_textSplitter": [
+[
+{
+"node": "Default Data Loader",
+"type": "ai_textSplitter",
+"index": 0
+}
+]
+]
+}
+}
+}sRecipe Recommendations with Qdrant and Mistral
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.json b/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.json
deleted file mode 100644
index e6608b8..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.json
+++ /dev/null
@@ -1,394 +0,0 @@
-{
- "meta": {
- "instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
- },
- "nodes": [
- {
- "id": "bebbf9cf-8103-4694-a3be-ae3ee1e9ebaf",
- "name": "Watch For Bank Statements",
- "type": "n8n-nodes-base.localFileTrigger",
- "position": [
- 780,
- 400
- ],
- "parameters": {
- "path": "/home/node/host_mount/reconciliation_project",
- "events": [
- "add"
- ],
- "options": {
- "ignored": "!**/*.csv"
- },
- "triggerOn": "folder"
- },
- "typeVersion": 1
- },
- {
- "id": "eca26bed-ba44-4507-97d4-9154e26908a5",
- "name": "Get Tenant Details",
- "type": "@n8n/n8n-nodes-langchain.toolCode",
- "position": [
- 1660,
- 540
- ],
- "parameters": {
- "name": "get_tenant_details",
- "jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').item.json;\nconst sheetName = 'tenants';\n\nconst wb = xlsx.readFile(spreadsheet_location, { sheets: [sheetName] });\nconst rows = xlsx.utils.sheet_to_json(wb.Sheets[sheetName], { raw: false });\n\nconst queryToList = [].concat(typeof query === 'string' ? query.split(',') : query);\n\nconst result = queryToList.map(q => (\n rows.find(row =>\n row['Tenant Name'].toLowerCase() === q.toLowerCase()\n || row['Tenant ID'].toLowerCase() === q.toString().toLowerCase()\n )\n));\n\nreturn result ? JSON.stringify(result) : `No results were found for ${query}`;",
- "description": "Call this tool to get a tenant's details which includes their tenancy terms, rent amount and any notes attached to their account. Pass in one or an array of either the tenant ID or the name of the tenant."
- },
- "typeVersion": 1.1
- },
- {
- "id": "76b68c2f-8d33-4f61-a442-732e784b733a",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 1920,
- 540
- ],
- "parameters": {
- "jsonSchemaExample": "[{\n \"tenant_id\": \"\",\n \"tenant_name\": \"\",\n \"property_id\": \"\",\n \"property_postcode\": \"\",\n \"action_required\": \"\",\n \"details\": \"\",\n \"date\": \"\"\n}]"
- },
- "typeVersion": 1.2
- },
- {
- "id": "be01720f-4617-4a2b-aaed-2474f9f0e25b",
- "name": "Get Bank Statement File",
- "type": "n8n-nodes-base.readWriteFile",
- "position": [
- 1100,
- 400
- ],
- "parameters": {
- "options": {},
- "fileSelector": "={{ $('Watch For Bank Statements').item.json.path }}"
- },
- "typeVersion": 1
- },
- {
- "id": "2aba5f6a-56b0-411f-9124-33025d90e325",
- "name": "Get CSV Data",
- "type": "n8n-nodes-base.extractFromFile",
- "position": [
- 1260,
- 400
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "a60d5851-f938-4696-855b-1f0845ffbc6c",
- "name": "Alert Actions To List",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 2260,
- 400
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "output"
- },
- "typeVersion": 1
- },
- {
- "id": "f804d9fb-f679-4e95-b70f-722e7c222c40",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 690.6721905682555,
- 177.80249392766257
- ],
- "parameters": {
- "color": 7,
- "width": 748.2548372021405,
- "height": 457.6238063670572,
- "content": "## Step 1. Wait For Incoming Bank Statements\n[Read more about the local file triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger)\n\nFor this demo, we'll show that n8n is more than capable working with the local filesystem. This gives great benefits in terms of privacy and data security.\n\nFor our datastore, we're using a locally hosted XLSX Excel file which we'll query and update throughout this workflow."
- },
- "typeVersion": 1
- },
- {
- "id": "01e9c335-320c-4fff-9ade-ad1cf808db00",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1460,
- 80
- ],
- "parameters": {
- "color": 7,
- "width": 634.3165117416636,
- "height": 675.2455596085985,
- "content": "## Step 2. Delegate to AI Agent to Quickly Identify Issues with Rental Payments\n[Read more about AI Agents](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/)\n\nAn AI agent can not only check against agreed amounts and compare due dates but also consider contract exceptions and tenant notes before deciding to take action. In a scenario of 10+ of tenants, this can save a lot of admin time.\n\nFor this demo, we're using a remote LLM Model but this can easily be swapped out for other self-hosted LLMS models that support function calling."
- },
- "typeVersion": 1
- },
- {
- "id": "2456b1e5-ceec-45c3-91a7-52e21125e6e5",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2120,
- 143.8836673253448
- ],
- "parameters": {
- "color": 7,
- "width": 618.3293247808133,
- "height": 473.7439917476675,
- "content": "## Step 3. Generate a Report to Action any Issues\n[Read more about using the Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nAfter the AI Agent has helped identify issues to action, we can generate a report and update a locally hosted xlsx file. This again helps keep workflows private to nothing senstive goes over the wire.\n\nThough n8n lacks a builtin node for editing local xlsx file, we can tap into the sheetJS library available to the \"Code\" node."
- },
- "typeVersion": 1
- },
- {
- "id": "7b32e8f9-b543-47e1-a08e-53ee47105966",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 260,
- 80
- ],
- "parameters": {
- "width": 399.5148533727183,
- "height": 558.2628336538015,
- "content": "## Try It Out!\n### This workflow ingests bank statements to analyses them against a list of tenants using an AI Agent. The agent then flags any issues such as missing payments or incorrect amounts which are exported to a XLSX spreadsheet.\n\n### Note: This workflow is intended to work with a self-hosted version of n8n and has access to the local file system.\n\n* Watches for CSV files (bank statements)\n* Imports into AI agent for analysis.\n* AI agent will query the Excel spreadsheet for tenant and property details.\n* AI agent will generate report on discrepancies or issues and write them to the Excel file.\n\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
- },
- "typeVersion": 1
- },
- {
- "id": "ba35ed0b-7ace-4b76-b915-0dc516a07fb1",
- "name": "Get Property Details",
- "type": "@n8n/n8n-nodes-langchain.toolCode",
- "position": [
- 1800,
- 540
- ],
- "parameters": {
- "name": "get_property_details",
- "jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').item.json;\nconst sheetName = 'properties'\n\nconst wb = xlsx.readFile(spreadsheet_location, { sheets: [sheetName] });\nconst rows = xlsx.utils.sheet_to_json(wb.Sheets[sheetName], { raw: false });\n\nconst queryToList = [].concat(typeof query === 'string' ? query.split(',') :query);\n\nconst result = queryToList.map(q => rows.find(row => row['Property ID'] === q));\n\nreturn result ? JSON.stringify(result) : `No results were found for ${query}`;",
- "description": "Call this tool to get a property details which includes the address, postcode and type of the property. Pass in one or an array of Property IDs."
- },
- "typeVersion": 1.1
- },
- {
- "id": "8c85a2f5-6741-41f4-b377-c74a74b14d0f",
- "name": "Set Variables",
- "type": "n8n-nodes-base.set",
- "position": [
- 940,
- 400
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "bcd3dd04-0082-4da6-b36b-e5ad09c4de30",
- "name": "spreadsheet_location",
- "type": "string",
- "value": "/home/node/host_mount/reconciliation_project/reconcilation-workbook.xlsx"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "bd75bad8-caa3-48f1-8892-3d1221765564",
- "name": "Append To Spreadsheet",
- "type": "n8n-nodes-base.code",
- "position": [
- 2480,
- 400
- ],
- "parameters": {
- "jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').first().json;\nconst sheetName = 'alerts';\n\nconst wb = xlsx.readFile(spreadsheet_location);\nxlsx.writeFile(wb, spreadsheet_location + '.bak.xlsx'); // create backup\n\nconst worksheet = wb.Sheets[sheetName];\n\nconst inputs = $input.all();\n\nfor (input of inputs) {\n xlsx.utils.sheet_add_aoa(worksheet, [\n [\n input.json.date,\n input.json[\"property_id\"],\n input.json[\"property_postcode\"],\n input.json[\"tenant_id\"],\n input.json[\"tenant_name\"],\n input.json[\"action_required\"],\n input.json[\"details\"],\n ]\n ], { origin: -1 });\n}\n\n// update sheet ref\nconst range = xlsx.utils.decode_range(worksheet['!ref']);\nconst rowIndex = range.e.r + 1; // The next row index to append\nworksheet['!ref'] = xlsx.utils.encode_range({\n s: range.s,\n e: { r: rowIndex, c: range.e.c }\n});\n\nxlsx.writeFile(wb, spreadsheet_location, {\n cellDates: true,\n cellStyles: true,\n bookType: 'xlsx',\n});\n\nreturn {\"json\": { \"output\": `${inputs.length} rows added` }}"
- },
- "typeVersion": 2
- },
- {
- "id": "c818ea7e-dc57-4680-b797-abb21cca87fb",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1540,
- 540
- ],
- "parameters": {
- "model": "gpt-4o",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "b2a97514-6020-49a6-bbdb-ee1251eb6aed",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2280,
- 640
- ],
- "parameters": {
- "color": 3,
- "width": 461.5505566920007,
- "height": 106.59049079746408,
- "content": "### ๐จWarning! Potentially Destructive Operations!\nWith code comes great responsibility! There is a risk you may overwrite/delete data you didn't intend. Always makes backups and test on a copy of your spreadsheets!"
- },
- "typeVersion": 1
- },
- {
- "id": "f869f6eb-cf19-4b14-bf3a-4db5d636646f",
- "name": "Reconcile Rental Payments",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 1640,
- 360
- ],
- "parameters": {
- "text": "=Bank Statement for {{ $input.first().json.date }} to {{ $input.last().json.date }}:\n|date|reference|money in|money out|\n|-|-|-|-|\n{{ $input.all().map(row => `|${row.json.date}|${row.json.reference}|${row.json.money_in || ''}|${row.json.money_out || ''}|`).join('\\n') }}",
- "options": {
- "systemMessage": "Your task is to help reconcile rent payments with the uploaded bank statement and alert only if there are any actions to be taken in regards to the tenants.\n* Identify and flag any tenants who have have missed their rent according to the month. Late payments which are within a few days of the due date are acceptable and should not be flagged.\n* Identify and flag if any tenants have not paid the correct ammount due, either less or more.\n* Identify and flag any tenants who are finishing their rentals within the time period of the current statement.\n* Identify and flag any remaining fees which are due and have not been paid from any tenant in the last month of their rental.\n\nIf the bank statement show incomplete months due to cut off, it is ok to assume the payment is pending and not actually missing.\n\nThe alert system requires a JSON formatted message. It is important that you format your response as follows:\n[{\n \"tenant_id\": \"\",\n \"tenant_name\": \"\",\n \"property_id\": \"\",\n \"property_postcode\": \"\",\n \"action required\": \"\",\n \"details\": \"\",\n \"date\": \"\"\n}]"
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "executeOnce": true,
- "typeVersion": 1.6
- },
- {
- "id": "510dc73c-f267-41f3-a981-58f5bfc229a6",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 360,
- 660
- ],
- "parameters": {
- "color": 5,
- "width": 302.6142384407349,
- "height": 86.00673806595168,
- "content": "### ๐กI'm designed to work self-hosted!\nSome nodes in this workflow are only available to the self-hosted version of n8n."
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Get CSV Data": {
- "main": [
- [
- {
- "node": "Reconcile Rental Payments",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Variables": {
- "main": [
- [
- {
- "node": "Get Bank Statement File",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Reconcile Rental Payments",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Get Tenant Details": {
- "ai_tool": [
- [
- {
- "node": "Reconcile Rental Payments",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Get Property Details": {
- "ai_tool": [
- [
- {
- "node": "Reconcile Rental Payments",
- "type": "ai_tool",
- "index": 0
- }
- ]
- ]
- },
- "Alert Actions To List": {
- "main": [
- [
- {
- "node": "Append To Spreadsheet",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Bank Statement File": {
- "main": [
- [
- {
- "node": "Get CSV Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Reconcile Rental Payments",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Reconcile Rental Payments": {
- "main": [
- [
- {
- "node": "Alert Actions To List",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Watch For Bank Statements": {
- "main": [
- [
- {
- "node": "Set Variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.txt b/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.txt
new file mode 100644
index 0000000..13bfebd
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Reconcile Rent Payments with Local Excel Spreadsheet and OpenAI.txt
@@ -0,0 +1,394 @@
+{
+"meta": {
+"instanceId": "26ba763460b97c249b82942b23b6384876dfeb9327513332e743c5f6219c2b8e"
+},
+"nodes": [
+{
+"id": "bebbf9cf-8103-4694-a3be-ae3ee1e9ebaf",
+"name": "Watch For Bank Statements",
+"type": "n8n-nodes-base.localFileTrigger",
+"position": [
+780,
+400
+],
+"parameters": {
+"path": "/home/node/host_mount/reconciliation_project",
+"events": [
+"add"
+],
+"options": {
+"ignored": "!**/*.csv"
+},
+"triggerOn": "folder"
+},
+"typeVersion": 1
+},
+{
+"id": "eca26bed-ba44-4507-97d4-9154e26908a5",
+"name": "Get Tenant Details",
+"type": "@n8n/n8n-nodes-langchain.toolCode",
+"position": [
+1660,
+540
+],
+"parameters": {
+"name": "get_tenant_details",
+"jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').item.json;\nconst sheetName = 'tenants';\n\nconst wb = xlsx.readFile(spreadsheet_location, { sheets: [sheetName] });\nconst rows = xlsx.utils.sheet_to_json(wb.Sheets[sheetName], { raw: false });\n\nconst queryToList = [].concat(typeof query === 'string' ? query.split(',') : query);\n\nconst result = queryToList.map(q => (\n rows.find(row =>\n row['Tenant Name'].toLowerCase() === q.toLowerCase()\n || row['Tenant ID'].toLowerCase() === q.toString().toLowerCase()\n )\n));\n\nreturn result ? JSON.stringify(result) : `No results were found for ${query}`;",
+"description": "Call this tool to get a tenant's details which includes their tenancy terms, rent amount and any notes attached to their account. Pass in one or an array of either the tenant ID or the name of the tenant."
+},
+"typeVersion": 1.1
+},
+{
+"id": "76b68c2f-8d33-4f61-a442-732e784b733a",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1920,
+540
+],
+"parameters": {
+"jsonSchemaExample": "[{\n \"tenant_id\": \"\",\n \"tenant_name\": \"\",\n \"property_id\": \"\",\n \"property_postcode\": \"\",\n \"action_required\": \"\",\n \"details\": \"\",\n \"date\": \"\"\n}]"
+},
+"typeVersion": 1.2
+},
+{
+"id": "be01720f-4617-4a2b-aaed-2474f9f0e25b",
+"name": "Get Bank Statement File",
+"type": "n8n-nodes-base.readWriteFile",
+"position": [
+1100,
+400
+],
+"parameters": {
+"options": {},
+"fileSelector": "={{ $('Watch For Bank Statements').item.json.path }}"
+},
+"typeVersion": 1
+},
+{
+"id": "2aba5f6a-56b0-411f-9124-33025d90e325",
+"name": "Get CSV Data",
+"type": "n8n-nodes-base.extractFromFile",
+"position": [
+1260,
+400
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "a60d5851-f938-4696-855b-1f0845ffbc6c",
+"name": "Alert Actions To List",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+2260,
+400
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output"
+},
+"typeVersion": 1
+},
+{
+"id": "f804d9fb-f679-4e95-b70f-722e7c222c40",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+690.6721905682555,
+177.80249392766257
+],
+"parameters": {
+"color": 7,
+"width": 748.2548372021405,
+"height": 457.6238063670572,
+"content": "## Step 1. Wait For Incoming Bank Statements\n[Read more about the local file triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.localfiletrigger)\n\nFor this demo, we'll show that n8n is more than capable working with the local filesystem. This gives great benefits in terms of privacy and data security.\n\nFor our datastore, we're using a locally hosted XLSX Excel file which we'll query and update throughout this workflow."
+},
+"typeVersion": 1
+},
+{
+"id": "01e9c335-320c-4fff-9ade-ad1cf808db00",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1460,
+80
+],
+"parameters": {
+"color": 7,
+"width": 634.3165117416636,
+"height": 675.2455596085985,
+"content": "## Step 2. Delegate to AI Agent to Quickly Identify Issues with Rental Payments\n[Read more about AI Agents](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.agent/)\n\nAn AI agent can not only check against agreed amounts and compare due dates but also consider contract exceptions and tenant notes before deciding to take action. In a scenario of 10+ of tenants, this can save a lot of admin time.\n\nFor this demo, we're using a remote LLM Model but this can easily be swapped out for other self-hosted LLMS models that support function calling."
+},
+"typeVersion": 1
+},
+{
+"id": "2456b1e5-ceec-45c3-91a7-52e21125e6e5",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2120,
+143.8836673253448
+],
+"parameters": {
+"color": 7,
+"width": 618.3293247808133,
+"height": 473.7439917476675,
+"content": "## Step 3. Generate a Report to Action any Issues\n[Read more about using the Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code)\n\nAfter the AI Agent has helped identify issues to action, we can generate a report and update a locally hosted xlsx file. This again helps keep workflows private to nothing senstive goes over the wire.\n\nThough n8n lacks a builtin node for editing local xlsx file, we can tap into the sheetJS library available to the \"Code\" node."
+},
+"typeVersion": 1
+},
+{
+"id": "7b32e8f9-b543-47e1-a08e-53ee47105966",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+260,
+80
+],
+"parameters": {
+"width": 399.5148533727183,
+"height": 558.2628336538015,
+"content": "## Try It Out!\n### This workflow ingests bank statements to analyses them against a list of tenants using an AI Agent. The agent then flags any issues such as missing payments or incorrect amounts which are exported to a XLSX spreadsheet.\n\n### Note: This workflow is intended to work with a self-hosted version of n8n and has access to the local file system.\n\n* Watches for CSV files (bank statements)\n* Imports into AI agent for analysis.\n* AI agent will query the Excel spreadsheet for tenant and property details.\n* AI agent will generate report on discrepancies or issues and write them to the Excel file.\n\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "ba35ed0b-7ace-4b76-b915-0dc516a07fb1",
+"name": "Get Property Details",
+"type": "@n8n/n8n-nodes-langchain.toolCode",
+"position": [
+1800,
+540
+],
+"parameters": {
+"name": "get_property_details",
+"jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').item.json;\nconst sheetName = 'properties'\n\nconst wb = xlsx.readFile(spreadsheet_location, { sheets: [sheetName] });\nconst rows = xlsx.utils.sheet_to_json(wb.Sheets[sheetName], { raw: false });\n\nconst queryToList = [].concat(typeof query === 'string' ? query.split(',') :query);\n\nconst result = queryToList.map(q => rows.find(row => row['Property ID'] === q));\n\nreturn result ? JSON.stringify(result) : `No results were found for ${query}`;",
+"description": "Call this tool to get a property details which includes the address, postcode and type of the property. Pass in one or an array of Property IDs."
+},
+"typeVersion": 1.1
+},
+{
+"id": "8c85a2f5-6741-41f4-b377-c74a74b14d0f",
+"name": "Set Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+940,
+400
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "bcd3dd04-0082-4da6-b36b-e5ad09c4de30",
+"name": "spreadsheet_location",
+"type": "string",
+"value": "/home/node/host_mount/reconciliation_project/reconcilation-workbook.xlsx"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "bd75bad8-caa3-48f1-8892-3d1221765564",
+"name": "Append To Spreadsheet",
+"type": "n8n-nodes-base.code",
+"position": [
+2480,
+400
+],
+"parameters": {
+"jsCode": "const xlsx = require('xlsx');\n\nconst { spreadsheet_location } = $('Set Variables').first().json;\nconst sheetName = 'alerts';\n\nconst wb = xlsx.readFile(spreadsheet_location);\nxlsx.writeFile(wb, spreadsheet_location + '.bak.xlsx'); // create backup\n\nconst worksheet = wb.Sheets[sheetName];\n\nconst inputs = $input.all();\n\nfor (input of inputs) {\n xlsx.utils.sheet_add_aoa(worksheet, [\n [\n input.json.date,\n input.json[\"property_id\"],\n input.json[\"property_postcode\"],\n input.json[\"tenant_id\"],\n input.json[\"tenant_name\"],\n input.json[\"action_required\"],\n input.json[\"details\"],\n ]\n ], { origin: -1 });\n}\n\n// update sheet ref\nconst range = xlsx.utils.decode_range(worksheet['!ref']);\nconst rowIndex = range.e.r + 1; // The next row index to append\nworksheet['!ref'] = xlsx.utils.encode_range({\n s: range.s,\n e: { r: rowIndex, c: range.e.c }\n});\n\nxlsx.writeFile(wb, spreadsheet_location, {\n cellDates: true,\n cellStyles: true,\n bookType: 'xlsx',\n});\n\nreturn {\"json\": { \"output\": `${inputs.length} rows added` }}"
+},
+"typeVersion": 2
+},
+{
+"id": "c818ea7e-dc57-4680-b797-abb21cca87fb",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+1540,
+540
+],
+"parameters": {
+"model": "gpt-4o",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b2a97514-6020-49a6-bbdb-ee1251eb6aed",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2280,
+640
+],
+"parameters": {
+"color": 3,
+"width": 461.5505566920007,
+"height": 106.59049079746408,
+"content": "### ๐จWarning! Potentially Destructive Operations!\nWith code comes great responsibility! There is a risk you may overwrite/delete data you didn't intend. Always makes backups and test on a copy of your spreadsheets!"
+},
+"typeVersion": 1
+},
+{
+"id": "f869f6eb-cf19-4b14-bf3a-4db5d636646f",
+"name": "Reconcile Rental Payments",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+1640,
+360
+],
+"parameters": {
+"text": "=Bank Statement for {{ $input.first().json.date }} to {{ $input.last().json.date }}:\n|date|reference|money in|money out|\n|-|-|-|-|\n{{ $input.all().map(row => `|${row.json.date}|${row.json.reference}|${row.json.money_in || ''}|${row.json.money_out || ''}|`).join('\\n') }}",
+"options": {
+"systemMessage": "Your task is to help reconcile rent payments with the uploaded bank statement and alert only if there are any actions to be taken in regards to the tenants.\n* Identify and flag any tenants who have have missed their rent according to the month. Late payments which are within a few days of the due date are acceptable and should not be flagged.\n* Identify and flag if any tenants have not paid the correct ammount due, either less or more.\n* Identify and flag any tenants who are finishing their rentals within the time period of the current statement.\n* Identify and flag any remaining fees which are due and have not been paid from any tenant in the last month of their rental.\n\nIf the bank statement show incomplete months due to cut off, it is ok to assume the payment is pending and not actually missing.\n\nThe alert system requires a JSON formatted message. It is important that you format your response as follows:\n[{\n \"tenant_id\": \"\",\n \"tenant_name\": \"\",\n \"property_id\": \"\",\n \"property_postcode\": \"\",\n \"action required\": \"\",\n \"details\": \"\",\n \"date\": \"\"\n}]"
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"executeOnce": true,
+"typeVersion": 1.6
+},
+{
+"id": "510dc73c-f267-41f3-a981-58f5bfc229a6",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+360,
+660
+],
+"parameters": {
+"color": 5,
+"width": 302.6142384407349,
+"height": 86.00673806595168,
+"content": "### ๐กI'm designed to work self-hosted!\nSome nodes in this workflow are only available to the self-hosted version of n8n."
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Get CSV Data": {
+"main": [
+[
+{
+"node": "Reconcile Rental Payments",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Variables": {
+"main": [
+[
+{
+"node": "Get Bank Statement File",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Reconcile Rental Payments",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Get Tenant Details": {
+"ai_tool": [
+[
+{
+"node": "Reconcile Rental Payments",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Get Property Details": {
+"ai_tool": [
+[
+{
+"node": "Reconcile Rental Payments",
+"type": "ai_tool",
+"index": 0
+}
+]
+]
+},
+"Alert Actions To List": {
+"main": [
+[
+{
+"node": "Append To Spreadsheet",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Bank Statement File": {
+"main": [
+[
+{
+"node": "Get CSV Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Reconcile Rental Payments",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Reconcile Rental Payments": {
+"main": [
+[
+{
+"node": "Alert Actions To List",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Watch For Bank Statements": {
+"main": [
+[
+{
+"node": "Set Variables",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.json b/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.json
deleted file mode 100644
index b24675d..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.json
+++ /dev/null
@@ -1,920 +0,0 @@
-{
- "id": "w434EiZ2z7klQAyp",
- "meta": {
- "instanceId": "a4bfc93e975ca233ac45ed7c9227d84cf5a2329310525917adaf3312e10d5462",
- "templateCredsSetupCompleted": true
- },
- "name": "Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI",
- "tags": [
- {
- "id": "2VG6RbmUdJ2VZbrj",
- "name": "Google Drive",
- "createdAt": "2024-12-04T16:50:56.177Z",
- "updatedAt": "2024-12-04T16:50:56.177Z"
- },
- {
- "id": "paTcf5QZDJsC2vKY",
- "name": "OpenAI",
- "createdAt": "2024-12-04T16:52:10.768Z",
- "updatedAt": "2024-12-04T16:52:10.768Z"
- }
- ],
- "nodes": [
- {
- "id": "095a8e10-1630-4a1a-b6c9-7950ae1ed803",
- "name": "Split Out",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 320,
- -380
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "recensioni"
- },
- "typeVersion": 1
- },
- {
- "id": "6ff4dd9d-eedd-4d84-b13a-b3c0db717409",
- "name": "Information Extractor",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- -440,
- 140
- ],
- "parameters": {
- "text": "=You need to extract the review from the following HTML: {{ $json.recensione }}",
- "options": {
- "systemPromptTemplate": "You are a review expert. You need to extract only the required information and report it without changing anything.\nAll the required information is in the text."
- },
- "attributes": {
- "attributes": [
- {
- "name": "autore",
- "required": true,
- "description": "Extract the name of the review author"
- },
- {
- "name": "valutazione",
- "type": "number",
- "required": true,
- "description": "Extract the rating given to the review (from 1 to 5)"
- },
- {
- "name": "data",
- "required": true,
- "description": "Extract review date in YYYY-MM-DD format"
- },
- {
- "name": "titolo",
- "required": true,
- "description": "Extract the review title"
- },
- {
- "name": "testo",
- "required": true,
- "description": "Extract the review text"
- },
- {
- "name": "n_recensioni",
- "type": "number",
- "required": true,
- "description": "Extract the total number of reviews made by the user"
- },
- {
- "name": "nazione",
- "required": true,
- "description": "Extract the country of the user who wrote the review. Must be two characters"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "0036f3b1-4832-4a35-8694-0893475a4119",
- "name": "If",
- "type": "n8n-nodes-base.if",
- "position": [
- 60,
- -100
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "loose"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "ab666549-4eec-40e2-a702-0575c094a2d4",
- "operator": {
- "type": "string",
- "operation": "empty",
- "singleValue": true
- },
- "leftValue": "={{ $json.Valutazione }}",
- "rightValue": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}"
- }
- ]
- },
- "looseTypeValidation": true
- },
- "executeOnce": false,
- "typeVersion": 2.2
- },
- {
- "id": "5423b55d-eb6c-41c6-9b26-410e3c92b85d",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -700,
- -380
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "506cdaa1-e0ba-4f29-b137-69d321b13c94",
- "name": "Limit1",
- "type": "n8n-nodes-base.limit",
- "position": [
- 540,
- -380
- ],
- "parameters": {
- "maxItems": 3
- },
- "typeVersion": 1
- },
- {
- "id": "40f1e30d-8aed-4995-b4e4-2239248bd6e7",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -460,
- -480
- ],
- "parameters": {
- "width": 212.25249169435213,
- "height": 245.55481727574733,
- "content": "Change to the name of the company registered on Trustpilot and the maximum number of pages to scrape"
- },
- "typeVersion": 1
- },
- {
- "id": "e6d2fec1-7255-4270-86b4-6d6f39f44ccb",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -460,
- 80
- ],
- "parameters": {
- "width": 381,
- "height": 177,
- "content": "Extract all information with DeepSeek (remember to change base_url with https://api.deepseek.com/v1)"
- },
- "typeVersion": 1
- },
- {
- "id": "af5e962c-4faf-41cc-a8b8-2fbb145b7af6",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -240,
- -160
- ],
- "parameters": {
- "width": 501.28903654485043,
- "height": 195.84053156146172,
- "content": "Check if the review has already been saved to Google Drive"
- },
- "typeVersion": 1
- },
- {
- "id": "400dff0c-8b2e-4fe2-933e-1f4d14624ca1",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 40,
- 80
- ],
- "parameters": {
- "width": 301.27574750830576,
- "height": 177.34219269102988,
- "content": "Analyze review sentiment"
- },
- "typeVersion": 1
- },
- {
- "id": "52757ade-4206-40f9-bf4f-c3aefb004d2e",
- "name": "Set Parameters",
- "type": "n8n-nodes-base.set",
- "position": [
- -440,
- -380
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "556e201d-242a-4c0e-bc13-787c2b60f800",
- "name": "company_id",
- "type": "string",
- "value": "COMPANY"
- },
- {
- "id": "a1f239df-df08-41d8-8b78-d6502266a581",
- "name": "max_page",
- "type": "number",
- "value": 2
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "cd7e9d36-7ecd-4d9c-b552-8f46b0cfcc03",
- "name": "Get reviews",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -200,
- -380
- ],
- "parameters": {
- "url": "=https://it.trustpilot.com/review/{{ $json.company_id }}",
- "options": {
- "pagination": {
- "pagination": {
- "parameters": {
- "parameters": [
- {
- "name": "page",
- "value": "={{ $pageCount + 1 }}"
- }
- ]
- },
- "maxRequests": "={{ $json.max_page }}",
- "requestInterval": 5000,
- "limitPagesFetched": true
- }
- }
- },
- "sendQuery": true,
- "queryParameters": {
- "parameters": [
- {
- "name": "sort",
- "value": "recency"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "476ff7b6-ab30-4674-a7fe-b032128ee51a",
- "name": "Extract",
- "type": "n8n-nodes-base.html",
- "position": [
- 60,
- -380
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "recensioni",
- "attribute": "href",
- "cssSelector": "article section a",
- "returnArray": true,
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "a2a35455-7d3e-4c4c-aa66-6cbbd48d867a",
- "name": "Get rows",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- -200,
- -100
- ],
- "parameters": {
- "options": {},
- "filtersUI": {
- "values": [
- {
- "lookupValue": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}",
- "lookupColumn": "Id"
- }
- ]
- },
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "",
- "cachedResultName": "Foglio1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
- "cachedResultUrl": "",
- "cachedResultName": "Trustpilot Review"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "JYR6a64Qecd6t8Hb",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.5
- },
- {
- "id": "2d507fe6-a4fc-42ff-97ff-dfd552c651ab",
- "name": "Get Google Sheets",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- -440,
- -100
- ],
- "parameters": {
- "columns": {
- "value": {
- "Id": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}"
- },
- "schema": [
- {
- "id": "Id",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "Id",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Data",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Data",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Nome",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Nome",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Titolo",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Titolo",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Testo",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Testo",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Localitร ",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Localitร ",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "N. Recensioni",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "N. Recensioni",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "URL",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "URL",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Valutazione",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Valutazione",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Sentiment",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "Sentiment",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [
- "Id"
- ],
- "attemptToConvertTypes": false,
- "convertFieldsToString": false
- },
- "options": {},
- "operation": "appendOrUpdate",
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "",
- "cachedResultName": "Foglio1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
- "cachedResultUrl": "",
- "cachedResultName": "Trustpilot Reviews"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "JYR6a64Qecd6t8Hb",
- "name": "Google Sheets account"
- }
- },
- "executeOnce": false,
- "typeVersion": 4.5
- },
- {
- "id": "0a1fab6e-96b7-403b-884e-f67be6e23fa5",
- "name": "Get Single review",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 320,
- -120
- ],
- "parameters": {
- "url": "=https://it.trustpilot.com{{ $('Split Out').item.json.recensioni }}",
- "options": {}
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "7d322d76-1032-405a-9d46-2958761a184d",
- "name": "Extract review",
- "type": "n8n-nodes-base.html",
- "position": [
- 540,
- -120
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "recensione",
- "cssSelector": "article",
- "returnArray": true
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "952484e5-8e87-4eb3-99a6-5bf26c701ba8",
- "name": "Update sheet",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 520,
- 120
- ],
- "parameters": {
- "columns": {
- "value": {
- "Id": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}",
- "URL": "=https://it.trustpilot.com{{ $('Split Out').item.json.recensioni }}",
- "Data": "={{ $('Information Extractor').item.json.output.data }}",
- "Nome": "={{ $json.output.autore }}",
- "Testo": "={{ $('Information Extractor').item.json.output.testo }}",
- "Titolo": "={{ $('Information Extractor').item.json.output.titolo }}",
- "Localitร ": "={{ $('Information Extractor').item.json.output.nazione }}",
- "Sentiment": "={{ $json.sentimentAnalysis.category }}",
- "Valutazione": "={{ $('Information Extractor').item.json.output.valutazione }}",
- "N. Recensioni": "={{ $('Information Extractor').item.json.output.n_recensioni }}"
- },
- "schema": [
- {
- "id": "Id",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "Id",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Data",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Data",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Nome",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Nome",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Titolo",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Titolo",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Testo",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Testo",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Localitร ",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Localitร ",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "N. Recensioni",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "N. Recensioni",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "URL",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "URL",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Valutazione",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "Valutazione",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Sentiment",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "Sentiment",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [
- "Id"
- ],
- "attemptToConvertTypes": false,
- "convertFieldsToString": false
- },
- "options": {},
- "operation": "appendOrUpdate",
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "",
- "cachedResultName": "Foglio1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
- "cachedResultUrl": "",
- "cachedResultName": "Trustpilot Reviews"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "JYR6a64Qecd6t8Hb",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.5
- },
- {
- "id": "eb853885-816d-4df7-b5ac-900fa89d3df9",
- "name": "Sentiment Analysis",
- "type": "@n8n/n8n-nodes-langchain.sentimentAnalysis",
- "position": [
- 60,
- 140
- ],
- "parameters": {
- "options": {
- "categories": "Positive, Neutral, Negative",
- "systemPromptTemplate": "You are highly intelligent and accurate sentiment analyzer. Analyze the sentiment of the provided text. Categorize it into one of the following: {categories}. Use the provided formatting instructions. Only output the JSON."
- },
- "inputText": "={{ $json.output.testo }}"
- },
- "typeVersion": 1
- },
- {
- "id": "79f1b9ea-6297-4735-9c0f-9f28dd65efa0",
- "name": "DeepSeek Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -460,
- 320
- ],
- "parameters": {
- "model": "deepseek-reasoner",
- "options": {
- "baseURL": "https://api.deepseek.com/v1"
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "97Cz4cqyiy1RdcQL",
- "name": "DeepSeek"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "159cc88e-1dd3-4bba-a3c8-59a9aad14c88",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 40,
- 320
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "CDX6QM4gLYanh0P4",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "43c8ee74-159c-4217-9cb4-554c63a3b183",
- "connections": {
- "If": {
- "main": [
- [
- {
- "node": "Get Single review",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Limit1": {
- "main": [
- [
- {
- "node": "Get Google Sheets",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract": {
- "main": [
- [
- {
- "node": "Split Out",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get rows": {
- "main": [
- [
- {
- "node": "If",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Out": {
- "main": [
- [
- {
- "node": "Limit1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get reviews": {
- "main": [
- [
- {
- "node": "Extract",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract review": {
- "main": [
- [
- {
- "node": "Information Extractor",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Parameters": {
- "main": [
- [
- {
- "node": "Get reviews",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Google Sheets": {
- "main": [
- [
- {
- "node": "Get rows",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Single review": {
- "main": [
- [
- {
- "node": "Extract review",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Sentiment Analysis",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Sentiment Analysis": {
- "main": [
- [
- {
- "node": "Update sheet",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Update sheet",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Update sheet",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "DeepSeek Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Information Extractor",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Information Extractor": {
- "main": [
- [
- {
- "node": "Sentiment Analysis",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Set Parameters",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.txt b/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.txt
new file mode 100644
index 0000000..b145f81
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI.txt
@@ -0,0 +1,920 @@
+{
+"id": "w434EiZ2z7klQAyp",
+"meta": {
+"instanceId": "a4bfc93e975ca233ac45ed7c9227d84cf5a2329310525917adaf3312e10d5462",
+"templateCredsSetupCompleted": true
+},
+"name": "Scrape Trustpilot Reviews with DeepSeek, Analyze Sentiment with OpenAI",
+"tags": [
+{
+"id": "2VG6RbmUdJ2VZbrj",
+"name": "Google Drive",
+"createdAt": "2024-12-04T16:50:56.177Z",
+"updatedAt": "2024-12-04T16:50:56.177Z"
+},
+{
+"id": "paTcf5QZDJsC2vKY",
+"name": "OpenAI",
+"createdAt": "2024-12-04T16:52:10.768Z",
+"updatedAt": "2024-12-04T16:52:10.768Z"
+}
+],
+"nodes": [
+{
+"id": "095a8e10-1630-4a1a-b6c9-7950ae1ed803",
+"name": "Split Out",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+320,
+-380
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "recensioni"
+},
+"typeVersion": 1
+},
+{
+"id": "6ff4dd9d-eedd-4d84-b13a-b3c0db717409",
+"name": "Information Extractor",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+-440,
+140
+],
+"parameters": {
+"text": "=You need to extract the review from the following HTML: {{ $json.recensione }}",
+"options": {
+"systemPromptTemplate": "You are a review expert. You need to extract only the required information and report it without changing anything.\nAll the required information is in the text."
+},
+"attributes": {
+"attributes": [
+{
+"name": "autore",
+"required": true,
+"description": "Extract the name of the review author"
+},
+{
+"name": "valutazione",
+"type": "number",
+"required": true,
+"description": "Extract the rating given to the review (from 1 to 5)"
+},
+{
+"name": "data",
+"required": true,
+"description": "Extract review date in YYYY-MM-DD format"
+},
+{
+"name": "titolo",
+"required": true,
+"description": "Extract the review title"
+},
+{
+"name": "testo",
+"required": true,
+"description": "Extract the review text"
+},
+{
+"name": "n_recensioni",
+"type": "number",
+"required": true,
+"description": "Extract the total number of reviews made by the user"
+},
+{
+"name": "nazione",
+"required": true,
+"description": "Extract the country of the user who wrote the review. Must be two characters"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "0036f3b1-4832-4a35-8694-0893475a4119",
+"name": "If",
+"type": "n8n-nodes-base.if",
+"position": [
+60,
+-100
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "loose"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "ab666549-4eec-40e2-a702-0575c094a2d4",
+"operator": {
+"type": "string",
+"operation": "empty",
+"singleValue": true
+},
+"leftValue": "={{ $json.Valutazione }}",
+"rightValue": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}"
+}
+]
+},
+"looseTypeValidation": true
+},
+"executeOnce": false,
+"typeVersion": 2.2
+},
+{
+"id": "5423b55d-eb6c-41c6-9b26-410e3c92b85d",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-700,
+-380
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "506cdaa1-e0ba-4f29-b137-69d321b13c94",
+"name": "Limit1",
+"type": "n8n-nodes-base.limit",
+"position": [
+540,
+-380
+],
+"parameters": {
+"maxItems": 3
+},
+"typeVersion": 1
+},
+{
+"id": "40f1e30d-8aed-4995-b4e4-2239248bd6e7",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-460,
+-480
+],
+"parameters": {
+"width": 212.25249169435213,
+"height": 245.55481727574733,
+"content": "Change to the name of the company registered on Trustpilot and the maximum number of pages to scrape"
+},
+"typeVersion": 1
+},
+{
+"id": "e6d2fec1-7255-4270-86b4-6d6f39f44ccb",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-460,
+80
+],
+"parameters": {
+"width": 381,
+"height": 177,
+"content": "Extract all information with DeepSeek (remember to change base_url with https://api.deepseek.com/v1)"
+},
+"typeVersion": 1
+},
+{
+"id": "af5e962c-4faf-41cc-a8b8-2fbb145b7af6",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-240,
+-160
+],
+"parameters": {
+"width": 501.28903654485043,
+"height": 195.84053156146172,
+"content": "Check if the review has already been saved to Google Drive"
+},
+"typeVersion": 1
+},
+{
+"id": "400dff0c-8b2e-4fe2-933e-1f4d14624ca1",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+40,
+80
+],
+"parameters": {
+"width": 301.27574750830576,
+"height": 177.34219269102988,
+"content": "Analyze review sentiment"
+},
+"typeVersion": 1
+},
+{
+"id": "52757ade-4206-40f9-bf4f-c3aefb004d2e",
+"name": "Set Parameters",
+"type": "n8n-nodes-base.set",
+"position": [
+-440,
+-380
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "556e201d-242a-4c0e-bc13-787c2b60f800",
+"name": "company_id",
+"type": "string",
+"value": "COMPANY"
+},
+{
+"id": "a1f239df-df08-41d8-8b78-d6502266a581",
+"name": "max_page",
+"type": "number",
+"value": 2
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "cd7e9d36-7ecd-4d9c-b552-8f46b0cfcc03",
+"name": "Get reviews",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-200,
+-380
+],
+"parameters": {
+"url": "=https://it.trustpilot.com/review/{{ $json.company_id }}",
+"options": {
+"pagination": {
+"pagination": {
+"parameters": {
+"parameters": [
+{
+"name": "page",
+"value": "={{ $pageCount + 1 }}"
+}
+]
+},
+"maxRequests": "={{ $json.max_page }}",
+"requestInterval": 5000,
+"limitPagesFetched": true
+}
+}
+},
+"sendQuery": true,
+"queryParameters": {
+"parameters": [
+{
+"name": "sort",
+"value": "recency"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "476ff7b6-ab30-4674-a7fe-b032128ee51a",
+"name": "Extract",
+"type": "n8n-nodes-base.html",
+"position": [
+60,
+-380
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "recensioni",
+"attribute": "href",
+"cssSelector": "article section a",
+"returnArray": true,
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "a2a35455-7d3e-4c4c-aa66-6cbbd48d867a",
+"name": "Get rows",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+-200,
+-100
+],
+"parameters": {
+"options": {},
+"filtersUI": {
+"values": [
+{
+"lookupValue": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}",
+"lookupColumn": "Id"
+}
+]
+},
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "",
+"cachedResultName": "Foglio1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
+"cachedResultUrl": "",
+"cachedResultName": "Trustpilot Review"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "JYR6a64Qecd6t8Hb",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.5
+},
+{
+"id": "2d507fe6-a4fc-42ff-97ff-dfd552c651ab",
+"name": "Get Google Sheets",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+-440,
+-100
+],
+"parameters": {
+"columns": {
+"value": {
+"Id": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}"
+},
+"schema": [
+{
+"id": "Id",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "Id",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Data",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Nome",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Nome",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Titolo",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Titolo",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Testo",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Testo",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Localitร ",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Localitร ",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "N. Recensioni",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "N. Recensioni",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "URL",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "URL",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Valutazione",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Valutazione",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Sentiment",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "Sentiment",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [
+"Id"
+],
+"attemptToConvertTypes": false,
+"convertFieldsToString": false
+},
+"options": {},
+"operation": "appendOrUpdate",
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "",
+"cachedResultName": "Foglio1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
+"cachedResultUrl": "",
+"cachedResultName": "Trustpilot Reviews"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "JYR6a64Qecd6t8Hb",
+"name": "Google Sheets account"
+}
+},
+"executeOnce": false,
+"typeVersion": 4.5
+},
+{
+"id": "0a1fab6e-96b7-403b-884e-f67be6e23fa5",
+"name": "Get Single review",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+320,
+-120
+],
+"parameters": {
+"url": "=https://it.trustpilot.com{{ $('Split Out').item.json.recensioni }}",
+"options": {}
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "7d322d76-1032-405a-9d46-2958761a184d",
+"name": "Extract review",
+"type": "n8n-nodes-base.html",
+"position": [
+540,
+-120
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "recensione",
+"cssSelector": "article",
+"returnArray": true
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "952484e5-8e87-4eb3-99a6-5bf26c701ba8",
+"name": "Update sheet",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+520,
+120
+],
+"parameters": {
+"columns": {
+"value": {
+"Id": "={{ $('Split Out').item.json.recensioni.replace('/reviews/','') }}",
+"URL": "=https://it.trustpilot.com{{ $('Split Out').item.json.recensioni }}",
+"Data": "={{ $('Information Extractor').item.json.output.data }}",
+"Nome": "={{ $json.output.autore }}",
+"Testo": "={{ $('Information Extractor').item.json.output.testo }}",
+"Titolo": "={{ $('Information Extractor').item.json.output.titolo }}",
+"Localitร ": "={{ $('Information Extractor').item.json.output.nazione }}",
+"Sentiment": "={{ $json.sentimentAnalysis.category }}",
+"Valutazione": "={{ $('Information Extractor').item.json.output.valutazione }}",
+"N. Recensioni": "={{ $('Information Extractor').item.json.output.n_recensioni }}"
+},
+"schema": [
+{
+"id": "Id",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "Id",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Data",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Data",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Nome",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Nome",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Titolo",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Titolo",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Testo",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Testo",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Localitร ",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Localitร ",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "N. Recensioni",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "N. Recensioni",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "URL",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "URL",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Valutazione",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "Valutazione",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Sentiment",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "Sentiment",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [
+"Id"
+],
+"attemptToConvertTypes": false,
+"convertFieldsToString": false
+},
+"options": {},
+"operation": "appendOrUpdate",
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "",
+"cachedResultName": "Foglio1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1QZhQqg79-HVBQh8Y2ihMq67UIYIRrJFKLQalcFvtDaY",
+"cachedResultUrl": "",
+"cachedResultName": "Trustpilot Reviews"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "JYR6a64Qecd6t8Hb",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.5
+},
+{
+"id": "eb853885-816d-4df7-b5ac-900fa89d3df9",
+"name": "Sentiment Analysis",
+"type": "@n8n/n8n-nodes-langchain.sentimentAnalysis",
+"position": [
+60,
+140
+],
+"parameters": {
+"options": {
+"categories": "Positive, Neutral, Negative",
+"systemPromptTemplate": "You are highly intelligent and accurate sentiment analyzer. Analyze the sentiment of the provided text. Categorize it into one of the following: {categories}. Use the provided formatting instructions. Only output the JSON."
+},
+"inputText": "={{ $json.output.testo }}"
+},
+"typeVersion": 1
+},
+{
+"id": "79f1b9ea-6297-4735-9c0f-9f28dd65efa0",
+"name": "DeepSeek Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+-460,
+320
+],
+"parameters": {
+"model": "deepseek-reasoner",
+"options": {
+"baseURL": "https://api.deepseek.com/v1"
+}
+},
+"credentials": {
+"openAiApi": {
+"id": "97Cz4cqyiy1RdcQL",
+"name": "DeepSeek"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "159cc88e-1dd3-4bba-a3c8-59a9aad14c88",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+40,
+320
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "CDX6QM4gLYanh0P4",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "43c8ee74-159c-4217-9cb4-554c63a3b183",
+"connections": {
+"If": {
+"main": [
+[
+{
+"node": "Get Single review",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Limit1": {
+"main": [
+[
+{
+"node": "Get Google Sheets",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract": {
+"main": [
+[
+{
+"node": "Split Out",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get rows": {
+"main": [
+[
+{
+"node": "If",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Out": {
+"main": [
+[
+{
+"node": "Limit1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get reviews": {
+"main": [
+[
+{
+"node": "Extract",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract review": {
+"main": [
+[
+{
+"node": "Information Extractor",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Parameters": {
+"main": [
+[
+{
+"node": "Get reviews",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Google Sheets": {
+"main": [
+[
+{
+"node": "Get rows",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Single review": {
+"main": [
+[
+{
+"node": "Extract review",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Sentiment Analysis",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Sentiment Analysis": {
+"main": [
+[
+{
+"node": "Update sheet",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Update sheet",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Update sheet",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"DeepSeek Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Information Extractor",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Information Extractor": {
+"main": [
+[
+{
+"node": "Sentiment Analysis",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Set Parameters",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.json b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.json
deleted file mode 100644
index db9c324..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.json
+++ /dev/null
@@ -1,885 +0,0 @@
-{
- "id": "xM8Z5vZVNTNjCySL",
- "meta": {
- "instanceId": "b8ef33547995f2a520f12118ac1f7819ea58faa7a1096148cac519fa08be8e99"
- },
- "name": "News Extraction",
- "tags": [],
- "nodes": [
- {
- "id": "97711d12-20de-40aa-b994-d2b10f20a5e5",
- "name": "Extract the HTML with the right css class",
- "type": "n8n-nodes-base.html",
- "position": [
- -500,
- 0
- ],
- "parameters": {
- "options": {
- "trimValues": true
- },
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "data",
- "attribute": "href",
- "cssSelector": "=div:nth-child(9) > div:nth-child(3) > a:nth-child(2)",
- "returnArray": true,
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "b874b570-daae-4878-b525-07ac30756eb1",
- "name": "Summary",
- "type": "n8n-nodes-base.openAi",
- "position": [
- -880,
- 440
- ],
- "parameters": {
- "model": "gpt-4-1106-preview",
- "prompt": {
- "messages": [
- {
- "content": "=Create a summary in less than 70 words {{ $json[\"content\"] }}"
- }
- ]
- },
- "options": {},
- "resource": "chat"
- },
- "credentials": {
- "openAiApi": {
- "id": "0Vdk5RlVe7AoUdAM",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "72696278-2d44-4073-936a-6fe9df1bc7d8",
- "name": "Keywords",
- "type": "n8n-nodes-base.openAi",
- "position": [
- -880,
- 620
- ],
- "parameters": {
- "model": "gpt-4-1106-preview",
- "prompt": {
- "messages": [
- {
- "content": "=name the 3 most important technical keywords in {{ $json[\"content\"] }} ? just name them without any explanations or other sentences"
- }
- ]
- },
- "options": {},
- "resource": "chat"
- },
- "credentials": {
- "openAiApi": {
- "id": "0Vdk5RlVe7AoUdAM",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "0bfdb3be-76ef-4bb3-902f-f0869342b83c",
- "name": "Rename keywords",
- "type": "n8n-nodes-base.set",
- "position": [
- -700,
- 620
- ],
- "parameters": {
- "fields": {
- "values": [
- {
- "name": "keywords",
- "stringValue": "={{ $json[\"message\"][\"content\"] }}"
- }
- ]
- },
- "include": "none",
- "options": {}
- },
- "typeVersion": 3.1
- },
- {
- "id": "0387cf34-41c9-4729-8570-1db7b17c42f4",
- "name": "Rename Summary",
- "type": "n8n-nodes-base.set",
- "position": [
- -700,
- 440
- ],
- "parameters": {
- "fields": {
- "values": [
- {
- "name": "=summary",
- "stringValue": "={{ $json[\"message\"][\"content\"] }}"
- }
- ]
- },
- "include": "none",
- "options": {}
- },
- "typeVersion": 3.1
- },
- {
- "id": "5fa1702c-f0bf-4524-bc8f-6f550dd83f1e",
- "name": "Merge",
- "type": "n8n-nodes-base.merge",
- "position": [
- -480,
- 560
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combinationMode": "mergeByPosition"
- },
- "typeVersion": 2.1
- },
- {
- "id": "25128a71-b0d5-49a4-adb8-c3fbe03c0a85",
- "name": "Extract date",
- "type": "n8n-nodes-base.html",
- "position": [
- -500,
- -160
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "data",
- "cssSelector": "div:nth-child(9) > div:nth-child(2) > span:nth-child(1)",
- "returnArray": true
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "138b3bd6-494a-49b9-b5b8-c9febcfef9fb",
- "name": "Select posts of last 7 days",
- "type": "n8n-nodes-base.code",
- "position": [
- 120,
- 0
- ],
- "parameters": {
- "jsCode": "const currentDate = new Date();\nconst sevenDaysAgo = new Date(currentDate.setDate(currentDate.getDate() - 70)); // Change the number of days going back to your liking (e.g. from -7 to -1) -> BUT sync with the cron job (first node)\n\nconst filteredItems = items.filter(item => {\n const postDate = new Date(item.json[\"Date\"]); // Assuming \"Date\" is the field name in the extracted html\n return postDate >= sevenDaysAgo;\n});\n\nreturn filteredItems;\n"
- },
- "typeVersion": 2
- },
- {
- "id": "1ace953b-e298-4fc2-8970-327f736889ec",
- "name": "Merge date & links",
- "type": "n8n-nodes-base.merge",
- "position": [
- -100,
- 0
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combinationMode": "mergeByPosition"
- },
- "typeVersion": 2.1
- },
- {
- "id": "bba692fc-c225-41be-a969-179d8b99c071",
- "name": "HTTP Request1",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 320,
- 0
- ],
- "parameters": {
- "url": "={{ $json[\"Link\"] }}",
- "options": {}
- },
- "typeVersion": 4.1
- },
- {
- "id": "26671065-631f-4684-9ee1-15f26b4cf1e4",
- "name": "Merge Content with Date & Link",
- "type": "n8n-nodes-base.merge",
- "position": [
- 500,
- 260
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combinationMode": "mergeByPosition"
- },
- "typeVersion": 2.1
- },
- {
- "id": "79beb744-97b8-4072-824a-6736b0a080ef",
- "name": "Extract individual posts",
- "type": "n8n-nodes-base.html",
- "position": [
- 500,
- 0
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "title",
- "cssSelector": "h1.fl-heading > span:nth-child(1)"
- },
- {
- "key": "content",
- "cssSelector": ".fl-node-5c7574ae7d5c6 > div:nth-child(1)"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "e89d9de5-875b-453e-825a-26f2bebcc8df",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 80,
- -107
- ],
- "parameters": {
- "width": 180.9747474601832,
- "height": 276.31054308676767,
- "content": "Select only the newest news: todays date going back xy days"
- },
- "typeVersion": 1
- },
- {
- "id": "8a603f2f-4208-48c7-b169-e5613f13fa7d",
- "name": "Merge ChatGPT output with Date & Link",
- "type": "n8n-nodes-base.merge",
- "position": [
- -180,
- 560
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combinationMode": "mergeByPosition"
- },
- "typeVersion": 2.1
- },
- {
- "id": "e1036421-9ce1-4121-a692-602410ec7c95",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "disabled": true,
- "position": [
- -539.7802584556148,
- -4.722020203185366
- ],
- "parameters": {
- "width": 182.2748213508401,
- "height": 304.2550759710132,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExtracting the individual links of the press release page in order to retrieve the individual posts on their respective **url**"
- },
- "typeVersion": 1
- },
- {
- "id": "3655ab22-6a17-429a-9d9b-d96bbcc78fee",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -538.404803912782,
- -304
- ],
- "parameters": {
- "width": 178.75185894039254,
- "height": 289.463147786618,
- "content": "Extracting the dates of the posts of the press release page.\nThe right CSS selector has to be chosen.\n[More info on datagrab.io](https://datagrab.io/blog/guide-to-css-selectors-for-web-scraping/)"
- },
- "typeVersion": 1
- },
- {
- "id": "2e27fb4c-426a-41e1-b5fb-9b2d78acd2a7",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1300,
- -299.82161760751774
- ],
- "parameters": {
- "width": 334.4404040637068,
- "height": 1127.2017245821128,
- "content": "# Scraping posts of a news site without RSS feed\n\n\nThe [News Site](https://www.colt.net/resources/type/news/) from Colt, a telecom company, does not offer an RSS feed, therefore web scraping is the \nchoice to extract and process the news.\n\nThe goal is to get only the newest posts, a summary of each post and their respective (technical) keywords.\n\nNote that the news site offers the links to each news post, but not the individual news. We collect first the links and dates of each post before extracting the newest ones.\n\nThe result is sent to a SQL database, in this case a NocoDB database.\n\nThis process happens each week thru a cron job.\n\n**Requirements**:\n- Basic understanding of CSS selectors and how to get them via browser (usually: right click → inspect)\n- ChatGPT API account - normal account is not sufficient\n- A NocoDB database - of course you may choose any type of output target\n\n**Assumptions**:\n- CSS selectors work on the news site\n- The post has a date with own CSS selector - meaning date is not part of the news content\n\n**\"Warnings\"**\n- Not every site likes to be scraped, especially not in high frequency\n- Each website is structured in different ways, the workflow may then need several adaptations.\n\n\nHappy about any suggestion to improve. You may contact me on **Mastodon**: https://bonn.social/@askans"
- },
- "typeVersion": 1
- },
- {
- "id": "d43bd5b7-2aff-4a07-8aca-ca4747ec6c4d",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -927.8447474890202,
- -80
- ],
- "parameters": {
- "width": 153.90180146729315,
- "height": 237.91333335255808,
- "content": "Weekly cron job"
- },
- "typeVersion": 1
- },
- {
- "id": "e732d136-fcf1-4fc3-8bb6-bdcea3c78d9e",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -760,
- -80
- ],
- "parameters": {
- "width": 185.41515152389002,
- "height": 241.454848504947,
- "content": "The html of the news site is being retrieved: https://www.colt.net/resources/type/news/"
- },
- "typeVersion": 1
- },
- {
- "id": "d5e29ec3-5ef2-42f3-b316-9350644dbba4",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -340,
- -306
- ],
- "parameters": {
- "width": 187.3613302133812,
- "height": 469.2923233086395,
- "content": "As the extraction are returned as arrays, they transformed into individual JSON items to enable looping with other nodes"
- },
- "typeVersion": 1
- },
- {
- "id": "1af15c45-32c0-4abf-a35d-be7206823569",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -120,
- -103.54151515238902
- ],
- "parameters": {
- "width": 150,
- "height": 274.50898992724416,
- "content": "The links of the individual posts and the dates of the posts "
- },
- "typeVersion": 1
- },
- {
- "id": "f7c42748-f227-42d0-a9e2-fcb16dbd0f75",
- "name": "Retrieve the web page for further processsing",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -720,
- 0
- ],
- "parameters": {
- "url": "https://www.colt.net/resources/type/news/",
- "options": {
- "response": {
- "response": {
- "responseFormat": "text"
- }
- }
- }
- },
- "typeVersion": 4.1
- },
- {
- "id": "b2c36f26-8221-478f-a4b0-22758b1e5e58",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 292,
- -100
- ],
- "parameters": {
- "width": 155.0036363426638,
- "height": 272.1479798256519,
- "content": "Get the html of each individual **newest** post"
- },
- "typeVersion": 1
- },
- {
- "id": "6ae05c31-c09a-4b4e-a013-41571937bc39",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 460,
- -100
- ],
- "parameters": {
- "width": 184.07417896879767,
- "height": 269.2504410842093,
- "content": "Extracting the title & content (text) of each individual news post with the right CSS selector"
- },
- "typeVersion": 1
- },
- {
- "id": "e2da76d4-0c8c-4c61-924f-50aa9387e9ab",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 460,
- 180
- ],
- "parameters": {
- "width": 191.87778190338406,
- "height": 234.13422787857044,
- "content": "Merge link to url, date with content (text) and title of each news psot"
- },
- "typeVersion": 1
- },
- {
- "id": "c124aaac-dce6-4658-9027-bdfe5c0c81e6",
- "name": "Sticky Note12",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -907.2264215202996,
- 331.0681740778203
- ],
- "parameters": {
- "width": 150,
- "height": 256.2444361932317,
- "content": "Create a summary of each news post with ChatGPT. You need a ChatGPT API account for this"
- },
- "typeVersion": 1
- },
- {
- "id": "c9037e74-007b-4e44-b7f9-90e78b853eb5",
- "name": "Sticky Note13",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -909.595196087218,
- 610.7495589157902
- ],
- "parameters": {
- "width": 152.85976723045226,
- "height": 218.52702200939785,
- "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\nGet the 3 keywords of each news post"
- },
- "typeVersion": 1
- },
- {
- "id": "756397d9-de80-4114-9dee-b4f4b9593333",
- "name": "Sticky Note14",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -740,
- 340
- ],
- "parameters": {
- "width": 182.7735784797001,
- "height": 489.05192374172555,
- "content": "Just a renaming of data fields and eliminating unnecessary ones"
- },
- "typeVersion": 1
- },
- {
- "id": "a0dcb254-f064-45ed-8e22-30a6d079085b",
- "name": "Sticky Note15",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- 480
- ],
- "parameters": {
- "width": 169.7675735887227,
- "height": 254.94383570413422,
- "content": "Merge summary and keywords of each news post"
- },
- "typeVersion": 1
- },
- {
- "id": "82993166-b273-4b82-a954-554c6892f825",
- "name": "Schedule Trigger each week",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- -900,
- 0
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks",
- "triggerAtDay": [
- 3
- ],
- "triggerAtHour": 4,
- "triggerAtMinute": 32
- }
- ]
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "3d670eb9-5a36-4cd9-8d2c-40adf848485e",
- "name": "Sticky Note16",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -220,
- 477.5081090810816
- ],
- "parameters": {
- "width": 180.1723775015045,
- "height": 260.5279202647822,
- "content": "Add title, link and date to summary and keywords of each news post"
- },
- "typeVersion": 1
- },
- {
- "id": "62021393-e988-4834-9fa2-75a957b42890",
- "name": "NocoDB news database",
- "type": "n8n-nodes-base.nocoDb",
- "position": [
- 60,
- 560
- ],
- "parameters": {
- "table": "mhbalmu9aaqcun6",
- "fieldsUi": {
- "fieldValues": [
- {
- "fieldName": "=News_Source",
- "fieldValue": "=Colt"
- },
- {
- "fieldName": "Title",
- "fieldValue": "={{ $json[\"title\"] }}"
- },
- {
- "fieldName": "Date",
- "fieldValue": "={{ $json[\"Date\"] }}"
- },
- {
- "fieldName": "Link",
- "fieldValue": "={{ $json[\"Link\"] }}"
- },
- {
- "fieldName": "Summary",
- "fieldValue": "={{ $json[\"summary\"] }}"
- },
- {
- "fieldName": "Keywords",
- "fieldValue": "={{ $json[\"keywords\"] }}"
- }
- ]
- },
- "operation": "create",
- "projectId": "prqu4e8bjj4bv1j",
- "authentication": "nocoDbApiToken"
- },
- "credentials": {
- "nocoDbApiToken": {
- "id": "gjNns0VJMS3P2RQ3",
- "name": "NocoDB Token account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "e59e9fab-10a7-470b-afa6-e1d4b4e57723",
- "name": "Sticky Note17",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 280,
- 480
- ],
- "parameters": {
- "width": 483.95825869942666,
- "height": 268.5678114630957,
- "content": "## News summaries and keywords → database\n\n[NocoDB](https://nocodb.com/) is an SQL database, here we store the news summaries and keywords for further processing. Any other output target can be chosen here, e.g. e-mail, Excel etc.\n\nYou need first have that database structured before appending the news summaries and additional fields. The you can shape this node.\n\nSome fields may be edited in the database itself (e.g. relevance of the news to you) and may be filled therefore with a default value or not at all"
- },
- "typeVersion": 1
- },
- {
- "id": "253b414b-9a5b-4a25-892b-9aa011d55d28",
- "name": "Sticky Note18",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 20,
- 480
- ],
- "parameters": {
- "width": 262.99083066277313,
- "height": 268.56781146309544,
- "content": ""
- },
- "typeVersion": 1
- },
- {
- "id": "438e8dde-ce0a-4e5e-8d62-d735d19ec189",
- "name": "Create single link items",
- "type": "n8n-nodes-base.itemLists",
- "position": [
- -300,
- 0
- ],
- "parameters": {
- "options": {
- "destinationFieldName": "Link"
- },
- "fieldToSplitOut": "data"
- },
- "typeVersion": 3
- },
- {
- "id": "d721776b-fefc-4e72-91ef-6710f10b0393",
- "name": "Create single date items",
- "type": "n8n-nodes-base.itemLists",
- "position": [
- -300,
- -160
- ],
- "parameters": {
- "options": {
- "destinationFieldName": "Date"
- },
- "fieldToSplitOut": "data"
- },
- "typeVersion": 3
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "ff89d802-3bcf-4b34-9cd9-776b1f3b5eab",
- "connections": {
- "Merge": {
- "main": [
- [
- {
- "node": "Merge ChatGPT output with Date & Link",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Summary": {
- "main": [
- [
- {
- "node": "Rename Summary",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Keywords": {
- "main": [
- [
- {
- "node": "Rename keywords",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract date": {
- "main": [
- [
- {
- "node": "Create single date items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "HTTP Request1": {
- "main": [
- [
- {
- "node": "Extract individual posts",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Rename Summary": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Rename keywords": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Merge date & links": {
- "main": [
- [
- {
- "node": "Select posts of last 7 days",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create single date items": {
- "main": [
- [
- {
- "node": "Merge date & links",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create single link items": {
- "main": [
- [
- {
- "node": "Merge date & links",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Extract individual posts": {
- "main": [
- [
- {
- "node": "Merge Content with Date & Link",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Schedule Trigger each week": {
- "main": [
- [
- {
- "node": "Retrieve the web page for further processsing",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Select posts of last 7 days": {
- "main": [
- [
- {
- "node": "Merge Content with Date & Link",
- "type": "main",
- "index": 1
- },
- {
- "node": "HTTP Request1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Merge Content with Date & Link": {
- "main": [
- [
- {
- "node": "Summary",
- "type": "main",
- "index": 0
- },
- {
- "node": "Keywords",
- "type": "main",
- "index": 0
- },
- {
- "node": "Merge ChatGPT output with Date & Link",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Merge ChatGPT output with Date & Link": {
- "main": [
- [
- {
- "node": "NocoDB news database",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract the HTML with the right css class": {
- "main": [
- [
- {
- "node": "Create single link items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Retrieve the web page for further processsing": {
- "main": [
- [
- {
- "node": "Extract the HTML with the right css class",
- "type": "main",
- "index": 0
- },
- {
- "node": "Extract date",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.txt b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.txt
new file mode 100644
index 0000000..4924e92
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize posts of a news site without RSS feed using AI and save them to a NocoDB.txt
@@ -0,0 +1,885 @@
+{
+"id": "xM8Z5vZVNTNjCySL",
+"meta": {
+"instanceId": "b8ef33547995f2a520f12118ac1f7819ea58faa7a1096148cac519fa08be8e99"
+},
+"name": "News Extraction",
+"tags": [],
+"nodes": [
+{
+"id": "97711d12-20de-40aa-b994-d2b10f20a5e5",
+"name": "Extract the HTML with the right css class",
+"type": "n8n-nodes-base.html",
+"position": [
+-500,
+0
+],
+"parameters": {
+"options": {
+"trimValues": true
+},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "data",
+"attribute": "href",
+"cssSelector": "=div:nth-child(9) > div:nth-child(3) > a:nth-child(2)",
+"returnArray": true,
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "b874b570-daae-4878-b525-07ac30756eb1",
+"name": "Summary",
+"type": "n8n-nodes-base.openAi",
+"position": [
+-880,
+440
+],
+"parameters": {
+"model": "gpt-4-1106-preview",
+"prompt": {
+"messages": [
+{
+"content": "=Create a summary in less than 70 words {{ $json[\"content\"] }}"
+}
+]
+},
+"options": {},
+"resource": "chat"
+},
+"credentials": {
+"openAiApi": {
+"id": "0Vdk5RlVe7AoUdAM",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "72696278-2d44-4073-936a-6fe9df1bc7d8",
+"name": "Keywords",
+"type": "n8n-nodes-base.openAi",
+"position": [
+-880,
+620
+],
+"parameters": {
+"model": "gpt-4-1106-preview",
+"prompt": {
+"messages": [
+{
+"content": "=name the 3 most important technical keywords in {{ $json[\"content\"] }} ? just name them without any explanations or other sentences"
+}
+]
+},
+"options": {},
+"resource": "chat"
+},
+"credentials": {
+"openAiApi": {
+"id": "0Vdk5RlVe7AoUdAM",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "0bfdb3be-76ef-4bb3-902f-f0869342b83c",
+"name": "Rename keywords",
+"type": "n8n-nodes-base.set",
+"position": [
+-700,
+620
+],
+"parameters": {
+"fields": {
+"values": [
+{
+"name": "keywords",
+"stringValue": "={{ $json[\"message\"][\"content\"] }}"
+}
+]
+},
+"include": "none",
+"options": {}
+},
+"typeVersion": 3.1
+},
+{
+"id": "0387cf34-41c9-4729-8570-1db7b17c42f4",
+"name": "Rename Summary",
+"type": "n8n-nodes-base.set",
+"position": [
+-700,
+440
+],
+"parameters": {
+"fields": {
+"values": [
+{
+"name": "=summary",
+"stringValue": "={{ $json[\"message\"][\"content\"] }}"
+}
+]
+},
+"include": "none",
+"options": {}
+},
+"typeVersion": 3.1
+},
+{
+"id": "5fa1702c-f0bf-4524-bc8f-6f550dd83f1e",
+"name": "Merge",
+"type": "n8n-nodes-base.merge",
+"position": [
+-480,
+560
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combinationMode": "mergeByPosition"
+},
+"typeVersion": 2.1
+},
+{
+"id": "25128a71-b0d5-49a4-adb8-c3fbe03c0a85",
+"name": "Extract date",
+"type": "n8n-nodes-base.html",
+"position": [
+-500,
+-160
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "data",
+"cssSelector": "div:nth-child(9) > div:nth-child(2) > span:nth-child(1)",
+"returnArray": true
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "138b3bd6-494a-49b9-b5b8-c9febcfef9fb",
+"name": "Select posts of last 7 days",
+"type": "n8n-nodes-base.code",
+"position": [
+120,
+0
+],
+"parameters": {
+"jsCode": "const currentDate = new Date();\nconst sevenDaysAgo = new Date(currentDate.setDate(currentDate.getDate() - 70)); // Change the number of days going back to your liking (e.g. from -7 to -1) -> BUT sync with the cron job (first node)\n\nconst filteredItems = items.filter(item => {\n const postDate = new Date(item.json[\"Date\"]); // Assuming \"Date\" is the field name in the extracted html\n return postDate >= sevenDaysAgo;\n});\n\nreturn filteredItems;\n"
+},
+"typeVersion": 2
+},
+{
+"id": "1ace953b-e298-4fc2-8970-327f736889ec",
+"name": "Merge date & links",
+"type": "n8n-nodes-base.merge",
+"position": [
+-100,
+0
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combinationMode": "mergeByPosition"
+},
+"typeVersion": 2.1
+},
+{
+"id": "bba692fc-c225-41be-a969-179d8b99c071",
+"name": "HTTP Request1",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+320,
+0
+],
+"parameters": {
+"url": "={{ $json[\"Link\"] }}",
+"options": {}
+},
+"typeVersion": 4.1
+},
+{
+"id": "26671065-631f-4684-9ee1-15f26b4cf1e4",
+"name": "Merge Content with Date & Link",
+"type": "n8n-nodes-base.merge",
+"position": [
+500,
+260
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combinationMode": "mergeByPosition"
+},
+"typeVersion": 2.1
+},
+{
+"id": "79beb744-97b8-4072-824a-6736b0a080ef",
+"name": "Extract individual posts",
+"type": "n8n-nodes-base.html",
+"position": [
+500,
+0
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "title",
+"cssSelector": "h1.fl-heading > span:nth-child(1)"
+},
+{
+"key": "content",
+"cssSelector": ".fl-node-5c7574ae7d5c6 > div:nth-child(1)"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "e89d9de5-875b-453e-825a-26f2bebcc8df",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+80,
+-107
+],
+"parameters": {
+"width": 180.9747474601832,
+"height": 276.31054308676767,
+"content": "Select only the newest news: todays date going back xy days"
+},
+"typeVersion": 1
+},
+{
+"id": "8a603f2f-4208-48c7-b169-e5613f13fa7d",
+"name": "Merge ChatGPT output with Date & Link",
+"type": "n8n-nodes-base.merge",
+"position": [
+-180,
+560
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combinationMode": "mergeByPosition"
+},
+"typeVersion": 2.1
+},
+{
+"id": "e1036421-9ce1-4121-a692-602410ec7c95",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"disabled": true,
+"position": [
+-539.7802584556148,
+-4.722020203185366
+],
+"parameters": {
+"width": 182.2748213508401,
+"height": 304.2550759710132,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nExtracting the individual links of the press release page in order to retrieve the individual posts on their respective **url**"
+},
+"typeVersion": 1
+},
+{
+"id": "3655ab22-6a17-429a-9d9b-d96bbcc78fee",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-538.404803912782,
+-304
+],
+"parameters": {
+"width": 178.75185894039254,
+"height": 289.463147786618,
+"content": "Extracting the dates of the posts of the press release page.\nThe right CSS selector has to be chosen.\n[More info on datagrab.io](https://datagrab.io/blog/guide-to-css-selectors-for-web-scraping/)"
+},
+"typeVersion": 1
+},
+{
+"id": "2e27fb4c-426a-41e1-b5fb-9b2d78acd2a7",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1300,
+-299.82161760751774
+],
+"parameters": {
+"width": 334.4404040637068,
+"height": 1127.2017245821128,
+"content": "# Scraping posts of a news site without RSS feed\n\n\nThe [News Site](https://www.colt.net/resources/type/news/) from Colt, a telecom company, does not offer an RSS feed, therefore web scraping is the \nchoice to extract and process the news.\n\nThe goal is to get only the newest posts, a summary of each post and their respective (technical) keywords.\n\nNote that the news site offers the links to each news post, but not the individual news. We collect first the links and dates of each post before extracting the newest ones.\n\nThe result is sent to a SQL database, in this case a NocoDB database.\n\nThis process happens each week thru a cron job.\n\n**Requirements**:\n- Basic understanding of CSS selectors and how to get them via browser (usually: right click → inspect)\n- ChatGPT API account - normal account is not sufficient\n- A NocoDB database - of course you may choose any type of output target\n\n**Assumptions**:\n- CSS selectors work on the news site\n- The post has a date with own CSS selector - meaning date is not part of the news content\n\n**\"Warnings\"**\n- Not every site likes to be scraped, especially not in high frequency\n- Each website is structured in different ways, the workflow may then need several adaptations.\n\n\nHappy about any suggestion to improve. You may contact me on **Mastodon**: https://bonn.social/@askans"
+},
+"typeVersion": 1
+},
+{
+"id": "d43bd5b7-2aff-4a07-8aca-ca4747ec6c4d",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-927.8447474890202,
+-80
+],
+"parameters": {
+"width": 153.90180146729315,
+"height": 237.91333335255808,
+"content": "Weekly cron job"
+},
+"typeVersion": 1
+},
+{
+"id": "e732d136-fcf1-4fc3-8bb6-bdcea3c78d9e",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-760,
+-80
+],
+"parameters": {
+"width": 185.41515152389002,
+"height": 241.454848504947,
+"content": "The html of the news site is being retrieved: https://www.colt.net/resources/type/news/"
+},
+"typeVersion": 1
+},
+{
+"id": "d5e29ec3-5ef2-42f3-b316-9350644dbba4",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-340,
+-306
+],
+"parameters": {
+"width": 187.3613302133812,
+"height": 469.2923233086395,
+"content": "As the extraction are returned as arrays, they transformed into individual JSON items to enable looping with other nodes"
+},
+"typeVersion": 1
+},
+{
+"id": "1af15c45-32c0-4abf-a35d-be7206823569",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-120,
+-103.54151515238902
+],
+"parameters": {
+"width": 150,
+"height": 274.50898992724416,
+"content": "The links of the individual posts and the dates of the posts "
+},
+"typeVersion": 1
+},
+{
+"id": "f7c42748-f227-42d0-a9e2-fcb16dbd0f75",
+"name": "Retrieve the web page for further processsing",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-720,
+0
+],
+"parameters": {
+"url": "https://www.colt.net/resources/type/news/",
+"options": {
+"response": {
+"response": {
+"responseFormat": "text"
+}
+}
+}
+},
+"typeVersion": 4.1
+},
+{
+"id": "b2c36f26-8221-478f-a4b0-22758b1e5e58",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+292,
+-100
+],
+"parameters": {
+"width": 155.0036363426638,
+"height": 272.1479798256519,
+"content": "Get the html of each individual **newest** post"
+},
+"typeVersion": 1
+},
+{
+"id": "6ae05c31-c09a-4b4e-a013-41571937bc39",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+460,
+-100
+],
+"parameters": {
+"width": 184.07417896879767,
+"height": 269.2504410842093,
+"content": "Extracting the title & content (text) of each individual news post with the right CSS selector"
+},
+"typeVersion": 1
+},
+{
+"id": "e2da76d4-0c8c-4c61-924f-50aa9387e9ab",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+460,
+180
+],
+"parameters": {
+"width": 191.87778190338406,
+"height": 234.13422787857044,
+"content": "Merge link to url, date with content (text) and title of each news psot"
+},
+"typeVersion": 1
+},
+{
+"id": "c124aaac-dce6-4658-9027-bdfe5c0c81e6",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-907.2264215202996,
+331.0681740778203
+],
+"parameters": {
+"width": 150,
+"height": 256.2444361932317,
+"content": "Create a summary of each news post with ChatGPT. You need a ChatGPT API account for this"
+},
+"typeVersion": 1
+},
+{
+"id": "c9037e74-007b-4e44-b7f9-90e78b853eb5",
+"name": "Sticky Note13",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-909.595196087218,
+610.7495589157902
+],
+"parameters": {
+"width": 152.85976723045226,
+"height": 218.52702200939785,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\nGet the 3 keywords of each news post"
+},
+"typeVersion": 1
+},
+{
+"id": "756397d9-de80-4114-9dee-b4f4b9593333",
+"name": "Sticky Note14",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-740,
+340
+],
+"parameters": {
+"width": 182.7735784797001,
+"height": 489.05192374172555,
+"content": "Just a renaming of data fields and eliminating unnecessary ones"
+},
+"typeVersion": 1
+},
+{
+"id": "a0dcb254-f064-45ed-8e22-30a6d079085b",
+"name": "Sticky Note15",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+480
+],
+"parameters": {
+"width": 169.7675735887227,
+"height": 254.94383570413422,
+"content": "Merge summary and keywords of each news post"
+},
+"typeVersion": 1
+},
+{
+"id": "82993166-b273-4b82-a954-554c6892f825",
+"name": "Schedule Trigger each week",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+-900,
+0
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks",
+"triggerAtDay": [
+3
+],
+"triggerAtHour": 4,
+"triggerAtMinute": 32
+}
+]
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "3d670eb9-5a36-4cd9-8d2c-40adf848485e",
+"name": "Sticky Note16",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-220,
+477.5081090810816
+],
+"parameters": {
+"width": 180.1723775015045,
+"height": 260.5279202647822,
+"content": "Add title, link and date to summary and keywords of each news post"
+},
+"typeVersion": 1
+},
+{
+"id": "62021393-e988-4834-9fa2-75a957b42890",
+"name": "NocoDB news database",
+"type": "n8n-nodes-base.nocoDb",
+"position": [
+60,
+560
+],
+"parameters": {
+"table": "mhbalmu9aaqcun6",
+"fieldsUi": {
+"fieldValues": [
+{
+"fieldName": "=News_Source",
+"fieldValue": "=Colt"
+},
+{
+"fieldName": "Title",
+"fieldValue": "={{ $json[\"title\"] }}"
+},
+{
+"fieldName": "Date",
+"fieldValue": "={{ $json[\"Date\"] }}"
+},
+{
+"fieldName": "Link",
+"fieldValue": "={{ $json[\"Link\"] }}"
+},
+{
+"fieldName": "Summary",
+"fieldValue": "={{ $json[\"summary\"] }}"
+},
+{
+"fieldName": "Keywords",
+"fieldValue": "={{ $json[\"keywords\"] }}"
+}
+]
+},
+"operation": "create",
+"projectId": "prqu4e8bjj4bv1j",
+"authentication": "nocoDbApiToken"
+},
+"credentials": {
+"nocoDbApiToken": {
+"id": "gjNns0VJMS3P2RQ3",
+"name": "NocoDB Token account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "e59e9fab-10a7-470b-afa6-e1d4b4e57723",
+"name": "Sticky Note17",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+280,
+480
+],
+"parameters": {
+"width": 483.95825869942666,
+"height": 268.5678114630957,
+"content": "## News summaries and keywords → database\n\n[NocoDB](https://nocodb.com/) is an SQL database, here we store the news summaries and keywords for further processing. Any other output target can be chosen here, e.g. e-mail, Excel etc.\n\nYou need first have that database structured before appending the news summaries and additional fields. The you can shape this node.\n\nSome fields may be edited in the database itself (e.g. relevance of the news to you) and may be filled therefore with a default value or not at all"
+},
+"typeVersion": 1
+},
+{
+"id": "253b414b-9a5b-4a25-892b-9aa011d55d28",
+"name": "Sticky Note18",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+20,
+480
+],
+"parameters": {
+"width": 262.99083066277313,
+"height": 268.56781146309544,
+"content": ""
+},
+"typeVersion": 1
+},
+{
+"id": "438e8dde-ce0a-4e5e-8d62-d735d19ec189",
+"name": "Create single link items",
+"type": "n8n-nodes-base.itemLists",
+"position": [
+-300,
+0
+],
+"parameters": {
+"options": {
+"destinationFieldName": "Link"
+},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 3
+},
+{
+"id": "d721776b-fefc-4e72-91ef-6710f10b0393",
+"name": "Create single date items",
+"type": "n8n-nodes-base.itemLists",
+"position": [
+-300,
+-160
+],
+"parameters": {
+"options": {
+"destinationFieldName": "Date"
+},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 3
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "ff89d802-3bcf-4b34-9cd9-776b1f3b5eab",
+"connections": {
+"Merge": {
+"main": [
+[
+{
+"node": "Merge ChatGPT output with Date & Link",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Summary": {
+"main": [
+[
+{
+"node": "Rename Summary",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Keywords": {
+"main": [
+[
+{
+"node": "Rename keywords",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract date": {
+"main": [
+[
+{
+"node": "Create single date items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"HTTP Request1": {
+"main": [
+[
+{
+"node": "Extract individual posts",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Rename Summary": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Rename keywords": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Merge date & links": {
+"main": [
+[
+{
+"node": "Select posts of last 7 days",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create single date items": {
+"main": [
+[
+{
+"node": "Merge date & links",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create single link items": {
+"main": [
+[
+{
+"node": "Merge date & links",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Extract individual posts": {
+"main": [
+[
+{
+"node": "Merge Content with Date & Link",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Schedule Trigger each week": {
+"main": [
+[
+{
+"node": "Retrieve the web page for further processsing",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Select posts of last 7 days": {
+"main": [
+[
+{
+"node": "Merge Content with Date & Link",
+"type": "main",
+"index": 1
+},
+{
+"node": "HTTP Request1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Merge Content with Date & Link": {
+"main": [
+[
+{
+"node": "Summary",
+"type": "main",
+"index": 0
+},
+{
+"node": "Keywords",
+"type": "main",
+"index": 0
+},
+{
+"node": "Merge ChatGPT output with Date & Link",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Merge ChatGPT output with Date & Link": {
+"main": [
+[
+{
+"node": "NocoDB news database",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract the HTML with the right css class": {
+"main": [
+[
+{
+"node": "Create single link items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Retrieve the web page for further processsing": {
+"main": [
+[
+{
+"node": "Extract the HTML with the right css class",
+"type": "main",
+"index": 0
+},
+{
+"node": "Extract date",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.json b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.json
deleted file mode 100644
index 1d2780c..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.json
+++ /dev/null
@@ -1,396 +0,0 @@
-{
- "meta": {
- "instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
- },
- "nodes": [
- {
- "id": "67850bd7-f9f4-4d5b-8c9e-bd1451247ba6",
- "name": "When clicking \"Execute Workflow\"",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -740,
- 1000
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "0d9133f9-b6d3-4101-95c6-3cd24cdb70c3",
- "name": "Fetch essay list",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -520,
- 1000
- ],
- "parameters": {
- "url": "http://www.paulgraham.com/articles.html",
- "options": {}
- },
- "typeVersion": 4.1
- },
- {
- "id": "ee634297-a456-4f70-a995-55b02950571e",
- "name": "Extract essay names",
- "type": "n8n-nodes-base.html",
- "position": [
- -300,
- 1000
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "dataPropertyName": "=data",
- "extractionValues": {
- "values": [
- {
- "key": "essay",
- "attribute": "href",
- "cssSelector": "table table a",
- "returnArray": true,
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "83d75693-dbb8-44c4-8533-da06f611c59c",
- "name": "Fetch essay texts",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 360,
- 1000
- ],
- "parameters": {
- "url": "=http://www.paulgraham.com/{{ $json.essay }}",
- "options": {}
- },
- "typeVersion": 4.1
- },
- {
- "id": "151022b5-8570-4176-bf3f-137f27ac7036",
- "name": "Extract title",
- "type": "n8n-nodes-base.html",
- "position": [
- 700,
- 700
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "title",
- "cssSelector": "title"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "07bcf095-3c4d-4a72-9bcb-341411750ff5",
- "name": "Clean up",
- "type": "n8n-nodes-base.set",
- "position": [
- 1360,
- 980
- ],
- "parameters": {
- "fields": {
- "values": [
- {
- "name": "title",
- "stringValue": "={{ $json.title }}"
- },
- {
- "name": "summary",
- "stringValue": "={{ $json.response.text }}"
- },
- {
- "name": "url",
- "stringValue": "=http://www.paulgraham.com/{{ $('Limit to first 3').item.json.essay }}"
- }
- ]
- },
- "include": "none",
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "11285de0-3c5d-4296-a322-9b7585af9acc",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -580,
- 920
- ],
- "parameters": {
- "width": 1071.752021563343,
- "height": 285.66037735849045,
- "content": "## Scrape latest Paul Graham essays"
- },
- "typeVersion": 1
- },
- {
- "id": "c32f905d-dd7a-4b68-bbe0-dd8115ee0944",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 620,
- 920
- ],
- "parameters": {
- "width": 465.3908355795153,
- "height": 606.7924528301882,
- "content": "## Summarize them with GPT"
- },
- "typeVersion": 1
- },
- {
- "id": "29d264f4-df6d-4a41-ab38-58e1b1becc9a",
- "name": "Split out into items",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- -80,
- 1000
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "essay"
- },
- "typeVersion": 1
- },
- {
- "id": "ccfa3a1d-f170-44b4-a1f2-3573c88cae98",
- "name": "Limit to first 3",
- "type": "n8n-nodes-base.limit",
- "position": [
- 140,
- 1000
- ],
- "parameters": {
- "maxItems": 3
- },
- "typeVersion": 1
- },
- {
- "id": "c3d05068-9d1a-4ef5-8249-e7384dc617ee",
- "name": "Default Data Loader",
- "type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
- "position": [
- 820,
- 1200
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "db75adad-cb16-4e72-b16e-34684a733b05",
- "name": "Recursive Character Text Splitter",
- "type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
- "position": [
- 820,
- 1340
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "022cc091-9b4c-45c2-bc8e-4037ec2d0d60",
- "name": "OpenAI Chat Model1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 680,
- 1200
- ],
- "parameters": {
- "model": "gpt-4o-mini",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "cda47bb7-36c5-4d15-a1ef-0c66b1194825",
- "name": "Merge",
- "type": "n8n-nodes-base.merge",
- "position": [
- 1160,
- 980
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combineBy": "combineByPosition"
- },
- "typeVersion": 3
- },
- {
- "id": "28144e4c-e425-428d-b3d1-f563bfd4e5b3",
- "name": "Summarization Chain",
- "type": "@n8n/n8n-nodes-langchain.chainSummarization",
- "position": [
- 720,
- 1000
- ],
- "parameters": {
- "options": {},
- "operationMode": "documentLoader"
- },
- "typeVersion": 2
- }
- ],
- "pinData": {},
- "connections": {
- "Merge": {
- "main": [
- [
- {
- "node": "Clean up",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract title": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Fetch essay list": {
- "main": [
- [
- {
- "node": "Extract essay names",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Limit to first 3": {
- "main": [
- [
- {
- "node": "Fetch essay texts",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Fetch essay texts": {
- "main": [
- [
- {
- "node": "Extract title",
- "type": "main",
- "index": 0
- },
- {
- "node": "Summarization Chain",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model1": {
- "ai_languageModel": [
- [
- {
- "node": "Summarization Chain",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Default Data Loader": {
- "ai_document": [
- [
- {
- "node": "Summarization Chain",
- "type": "ai_document",
- "index": 0
- }
- ]
- ]
- },
- "Extract essay names": {
- "main": [
- [
- {
- "node": "Split out into items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Summarization Chain": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Split out into items": {
- "main": [
- [
- {
- "node": "Limit to first 3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking \"Execute Workflow\"": {
- "main": [
- [
- {
- "node": "Fetch essay list",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Recursive Character Text Splitter": {
- "ai_textSplitter": [
- [
- {
- "node": "Default Data Loader",
- "type": "ai_textSplitter",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.txt b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.txt
new file mode 100644
index 0000000..8989f48
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Scrape and summarize webpages with AI.txt
@@ -0,0 +1,396 @@
+{
+"meta": {
+"instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
+},
+"nodes": [
+{
+"id": "67850bd7-f9f4-4d5b-8c9e-bd1451247ba6",
+"name": "When clicking \"Execute Workflow\"",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-740,
+1000
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "0d9133f9-b6d3-4101-95c6-3cd24cdb70c3",
+"name": "Fetch essay list",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-520,
+1000
+],
+"parameters": {
+"url": "http://www.paulgraham.com/articles.html",
+"options": {}
+},
+"typeVersion": 4.1
+},
+{
+"id": "ee634297-a456-4f70-a995-55b02950571e",
+"name": "Extract essay names",
+"type": "n8n-nodes-base.html",
+"position": [
+-300,
+1000
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"dataPropertyName": "=data",
+"extractionValues": {
+"values": [
+{
+"key": "essay",
+"attribute": "href",
+"cssSelector": "table table a",
+"returnArray": true,
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "83d75693-dbb8-44c4-8533-da06f611c59c",
+"name": "Fetch essay texts",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+360,
+1000
+],
+"parameters": {
+"url": "=http://www.paulgraham.com/{{ $json.essay }}",
+"options": {}
+},
+"typeVersion": 4.1
+},
+{
+"id": "151022b5-8570-4176-bf3f-137f27ac7036",
+"name": "Extract title",
+"type": "n8n-nodes-base.html",
+"position": [
+700,
+700
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "title",
+"cssSelector": "title"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "07bcf095-3c4d-4a72-9bcb-341411750ff5",
+"name": "Clean up",
+"type": "n8n-nodes-base.set",
+"position": [
+1360,
+980
+],
+"parameters": {
+"fields": {
+"values": [
+{
+"name": "title",
+"stringValue": "={{ $json.title }}"
+},
+{
+"name": "summary",
+"stringValue": "={{ $json.response.text }}"
+},
+{
+"name": "url",
+"stringValue": "=http://www.paulgraham.com/{{ $('Limit to first 3').item.json.essay }}"
+}
+]
+},
+"include": "none",
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "11285de0-3c5d-4296-a322-9b7585af9acc",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-580,
+920
+],
+"parameters": {
+"width": 1071.752021563343,
+"height": 285.66037735849045,
+"content": "## Scrape latest Paul Graham essays"
+},
+"typeVersion": 1
+},
+{
+"id": "c32f905d-dd7a-4b68-bbe0-dd8115ee0944",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+620,
+920
+],
+"parameters": {
+"width": 465.3908355795153,
+"height": 606.7924528301882,
+"content": "## Summarize them with GPT"
+},
+"typeVersion": 1
+},
+{
+"id": "29d264f4-df6d-4a41-ab38-58e1b1becc9a",
+"name": "Split out into items",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+-80,
+1000
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "essay"
+},
+"typeVersion": 1
+},
+{
+"id": "ccfa3a1d-f170-44b4-a1f2-3573c88cae98",
+"name": "Limit to first 3",
+"type": "n8n-nodes-base.limit",
+"position": [
+140,
+1000
+],
+"parameters": {
+"maxItems": 3
+},
+"typeVersion": 1
+},
+{
+"id": "c3d05068-9d1a-4ef5-8249-e7384dc617ee",
+"name": "Default Data Loader",
+"type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
+"position": [
+820,
+1200
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "db75adad-cb16-4e72-b16e-34684a733b05",
+"name": "Recursive Character Text Splitter",
+"type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
+"position": [
+820,
+1340
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "022cc091-9b4c-45c2-bc8e-4037ec2d0d60",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+680,
+1200
+],
+"parameters": {
+"model": "gpt-4o-mini",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "cda47bb7-36c5-4d15-a1ef-0c66b1194825",
+"name": "Merge",
+"type": "n8n-nodes-base.merge",
+"position": [
+1160,
+980
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combineBy": "combineByPosition"
+},
+"typeVersion": 3
+},
+{
+"id": "28144e4c-e425-428d-b3d1-f563bfd4e5b3",
+"name": "Summarization Chain",
+"type": "@n8n/n8n-nodes-langchain.chainSummarization",
+"position": [
+720,
+1000
+],
+"parameters": {
+"options": {},
+"operationMode": "documentLoader"
+},
+"typeVersion": 2
+}
+],
+"pinData": {},
+"connections": {
+"Merge": {
+"main": [
+[
+{
+"node": "Clean up",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract title": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Fetch essay list": {
+"main": [
+[
+{
+"node": "Extract essay names",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Limit to first 3": {
+"main": [
+[
+{
+"node": "Fetch essay texts",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Fetch essay texts": {
+"main": [
+[
+{
+"node": "Extract title",
+"type": "main",
+"index": 0
+},
+{
+"node": "Summarization Chain",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Summarization Chain",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Default Data Loader": {
+"ai_document": [
+[
+{
+"node": "Summarization Chain",
+"type": "ai_document",
+"index": 0
+}
+]
+]
+},
+"Extract essay names": {
+"main": [
+[
+{
+"node": "Split out into items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Summarization Chain": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Split out into items": {
+"main": [
+[
+{
+"node": "Limit to first 3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking \"Execute Workflow\"": {
+"main": [
+[
+{
+"node": "Fetch essay list",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Recursive Character Text Splitter": {
+"ai_textSplitter": [
+[
+{
+"node": "Default Data Loader",
+"type": "ai_textSplitter",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.json b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.json
deleted file mode 100644
index 6158806..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.json
+++ /dev/null
@@ -1,853 +0,0 @@
-{
- "id": "K3uf8aY8wipScEay",
- "meta": {
- "instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
- "templateCredsSetupCompleted": true
- },
- "name": "Google analytics template",
- "tags": [],
- "nodes": [
- {
- "id": "6a9fc442-d0a3-48be-8dff-94f8d9cd5cf1",
- "name": "Schedule Trigger",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- 460,
- 460
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "484cbc41-f57d-4c3d-a458-e439d480d290",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 460,
- 640
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "b1b66e9b-5fea-407b-9c1e-39bd2a9d4a90",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 460,
- 100
- ],
- "parameters": {
- "width": 714.172987012987,
- "content": "## Send Google analytics to A.I. and save results to baserow\n\nThis workflow will check for country views, page engagement and google search console results. It will take this week's data and compare it to last week's data.\n\n[You can read more about this workflow here](https://rumjahn.com/how-i-used-a-i-to-be-an-seo-expert-and-analyzed-my-google-analytics-data-in-n8n-and-make-com/)"
- },
- "typeVersion": 1
- },
- {
- "id": "adde29fc-ddb5-4b50-aa78-313ac9ede879",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 633.6540259740264,
- 320
- ],
- "parameters": {
- "color": 4,
- "width": 2097.92831168831,
- "height": 342.6576623376624,
- "content": "## Property ID\n\n1. Create your [Google Analytics Credentials](https://docs.n8n.io/integrations/builtin/credentials/google/oauth-single-service/?utm_source=n8n_app&utm_medium=credential_settings&utm_campaign=create_new_credentials_modal)\n2. Enter your [property ID](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id)."
- },
- "typeVersion": 1
- },
- {
- "id": "f2fb8535-e81e-4ca1-80df-ee68edba6386",
- "name": "Get Page Engagement Stats for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 700,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "screenPageViews",
- "listName": "other"
- },
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "screenPageViewsPerUser",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "unifiedScreenName",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "1d761425-cebf-4787-b286-b723a0851485",
- "name": "Get Page Engagement Stats for prior week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1060,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "2024-10-23T00:00:00",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "screenPageViews",
- "listName": "other"
- },
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "screenPageViewsPerUser",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "unifiedScreenName",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "typeVersion": 2
- },
- {
- "id": "f8dac36b-9e8a-407f-b923-b4cea368f1bc",
- "name": "Parse data from Google Analytics",
- "type": "n8n-nodes-base.code",
- "position": [
- 880,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
- },
- "typeVersion": 2
- },
- {
- "id": "ed880442-c92e-4347-b277-e8794aea6fbc",
- "name": "Parse GA data",
- "type": "n8n-nodes-base.code",
- "position": [
- 1240,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
- },
- "typeVersion": 2
- },
- {
- "id": "46e092cc-af94-4e64-aa92-931c56345eff",
- "name": "Get Google Search Results for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1420,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchAveragePosition",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClickThroughRate",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClicks",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchImpressions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "landingPagePlusQueryString",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "709d0aaf-bd3d-4d83-9e66-b7df495855bd",
- "name": "Get Google Search Results for last week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1780,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "={{$today.minus({days: 7})}}",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchAveragePosition",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClickThroughRate",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClicks",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchImpressions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "landingPagePlusQueryString",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "7d3835d6-d1f5-4159-8e34-871871e63989",
- "name": "Parse Google Analytics Data",
- "type": "n8n-nodes-base.code",
- "position": [
- 1600,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "c018fda4-a2e6-48f4-aabb-039c66374dc7",
- "name": "Parse Google Analytics Data1",
- "type": "n8n-nodes-base.code",
- "position": [
- 1940,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "d8f775cd-daf9-42de-a527-d932be46d945",
- "name": "Get Country views data for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 2120,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "newUsers",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "listName": "other"
- },
- {
- "name": "sessions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "country",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "7119e57c-cbf4-49a9-b0c9-1f3da1fd2af3",
- "name": "Get Country views data for last week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 2440,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "={{$today.minus({days: 7})}}",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "newUsers",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "listName": "other"
- },
- {
- "name": "sessions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "country",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "typeVersion": 2
- },
- {
- "id": "546d6cd2-6db6-4276-be35-abbe5a7e9b6a",
- "name": "Parse Google analytics data",
- "type": "n8n-nodes-base.code",
- "position": [
- 2280,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "87cb137c-686d-49a5-8657-06ed0c5f5c27",
- "name": "Parse Google analytics data1",
- "type": "n8n-nodes-base.code",
- "position": [
- 2600,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "06c4478d-a13a-4587-9f1f-451a68798a9f",
- "name": "Send page data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2760,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $json.urlString }} Data from last week: {{ $('Parse data from Google Analytics').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "4ad522b0-afe4-4eff-aa16-b86cc892ead8",
- "name": "Send page Search data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google Analytics Data1').item.json.urlString }} Data from last week:{{ $('Parse Google Analytics Data').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "07e1eebf-f16a-44c0-83b5-76bf65a3d3fc",
- "name": "Send country view data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3080,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google analytics data1').item.json.urlString }} Data from last week:{{ $('Parse Google analytics data').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "c4648ad8-2377-42a0-a431-931b53631c9d",
- "name": "Save A.I. output to Baserow",
- "type": "n8n-nodes-base.baserow",
- "position": [
- 3240,
- 460
- ],
- "parameters": {
- "tableId": 601,
- "fieldsUi": {
- "fieldValues": [
- {
- "fieldId": 5833,
- "fieldValue": "Name of your blog"
- },
- {
- "fieldId": 5831,
- "fieldValue": "={{ $('Send page data to A.I.').item.json.choices[0].message.content }}"
- },
- {
- "fieldId": 5830,
- "fieldValue": "={{ $('Send page Search data to A.I.').item.json.choices[0].message.content }}"
- },
- {
- "fieldId": 5832,
- "fieldValue": "={{ $json.choices[0].message.content }}"
- },
- {
- "fieldId": 5829,
- "fieldValue": "={{ DateTime.now() }}"
- }
- ]
- },
- "operation": "create",
- "databaseId": 121
- },
- "typeVersion": 1
- },
- {
- "id": "e185c836-c12f-4452-92bd-0daaf33b653a",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2760,
- 180
- ],
- "parameters": {
- "color": 5,
- "width": 441.7412987012988,
- "height": 508.95792207792226,
- "content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
- },
- "typeVersion": 1
- },
- {
- "id": "a1de2d16-d09e-4c74-8be1-f6bab8c34246",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3220,
- 180
- ],
- "parameters": {
- "color": 6,
- "width": 331.32883116883124,
- "height": 474.88,
- "content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Name\n- Country Views\n- Page Views\n- Search Report\n- Blog \n\nEnter the name of your website under \"Blog\" field."
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "ac4b5eac-1c84-49ce-9ff7-794f857265b4",
- "connections": {
- "Parse GA data": {
- "main": [
- [
- {
- "node": "Get Google Search Results for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Schedule Trigger": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send page data to A.I.": {
- "main": [
- [
- {
- "node": "Send page Search data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google Analytics Data": {
- "main": [
- [
- {
- "node": "Get Google Search Results for last week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google analytics data": {
- "main": [
- [
- {
- "node": "Get Country views data for last week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google Analytics Data1": {
- "main": [
- [
- {
- "node": "Get Country views data for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google analytics data1": {
- "main": [
- [
- {
- "node": "Send page data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send page Search data to A.I.": {
- "main": [
- [
- {
- "node": "Send country view data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send country view data to A.I.": {
- "main": [
- [
- {
- "node": "Save A.I. output to Baserow",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse data from Google Analytics": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for prior week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Country views data for last week": {
- "main": [
- [
- {
- "node": "Parse Google analytics data1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Country views data for this week": {
- "main": [
- [
- {
- "node": "Parse Google analytics data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Google Search Results for last week": {
- "main": [
- [
- {
- "node": "Parse Google Analytics Data1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Google Search Results for this week": {
- "main": [
- [
- {
- "node": "Parse Google Analytics Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Page Engagement Stats for this week": {
- "main": [
- [
- {
- "node": "Parse data from Google Analytics",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Page Engagement Stats for prior week": {
- "main": [
- [
- {
- "node": "Parse GA data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.txt b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.txt
new file mode 100644
index 0000000..451c870
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in Baserow.txt
@@ -0,0 +1,853 @@
+{
+"id": "K3uf8aY8wipScEay",
+"meta": {
+"instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
+"templateCredsSetupCompleted": true
+},
+"name": "Google analytics template",
+"tags": [],
+"nodes": [
+{
+"id": "6a9fc442-d0a3-48be-8dff-94f8d9cd5cf1",
+"name": "Schedule Trigger",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+460,
+460
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "484cbc41-f57d-4c3d-a458-e439d480d290",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+460,
+640
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "b1b66e9b-5fea-407b-9c1e-39bd2a9d4a90",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+460,
+100
+],
+"parameters": {
+"width": 714.172987012987,
+"content": "## Send Google analytics to A.I. and save results to baserow\n\nThis workflow will check for country views, page engagement and google search console results. It will take this week's data and compare it to last week's data.\n\n[You can read more about this workflow here](https://rumjahn.com/how-i-used-a-i-to-be-an-seo-expert-and-analyzed-my-google-analytics-data-in-n8n-and-make-com/)"
+},
+"typeVersion": 1
+},
+{
+"id": "adde29fc-ddb5-4b50-aa78-313ac9ede879",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+633.6540259740264,
+320
+],
+"parameters": {
+"color": 4,
+"width": 2097.92831168831,
+"height": 342.6576623376624,
+"content": "## Property ID\n\n1. Create your [Google Analytics Credentials](https://docs.n8n.io/integrations/builtin/credentials/google/oauth-single-service/?utm_source=n8n_app&utm_medium=credential_settings&utm_campaign=create_new_credentials_modal)\n2. Enter your [property ID](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id)."
+},
+"typeVersion": 1
+},
+{
+"id": "f2fb8535-e81e-4ca1-80df-ee68edba6386",
+"name": "Get Page Engagement Stats for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+700,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "screenPageViews",
+"listName": "other"
+},
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "screenPageViewsPerUser",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "unifiedScreenName",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "1d761425-cebf-4787-b286-b723a0851485",
+"name": "Get Page Engagement Stats for prior week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1060,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "2024-10-23T00:00:00",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "screenPageViews",
+"listName": "other"
+},
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "screenPageViewsPerUser",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "unifiedScreenName",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"typeVersion": 2
+},
+{
+"id": "f8dac36b-9e8a-407f-b923-b4cea368f1bc",
+"name": "Parse data from Google Analytics",
+"type": "n8n-nodes-base.code",
+"position": [
+880,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
+},
+"typeVersion": 2
+},
+{
+"id": "ed880442-c92e-4347-b277-e8794aea6fbc",
+"name": "Parse GA data",
+"type": "n8n-nodes-base.code",
+"position": [
+1240,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
+},
+"typeVersion": 2
+},
+{
+"id": "46e092cc-af94-4e64-aa92-931c56345eff",
+"name": "Get Google Search Results for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1420,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchAveragePosition",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClickThroughRate",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClicks",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchImpressions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "landingPagePlusQueryString",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "709d0aaf-bd3d-4d83-9e66-b7df495855bd",
+"name": "Get Google Search Results for last week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1780,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "={{$today.minus({days: 7})}}",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchAveragePosition",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClickThroughRate",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClicks",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchImpressions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "landingPagePlusQueryString",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "7d3835d6-d1f5-4159-8e34-871871e63989",
+"name": "Parse Google Analytics Data",
+"type": "n8n-nodes-base.code",
+"position": [
+1600,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "c018fda4-a2e6-48f4-aabb-039c66374dc7",
+"name": "Parse Google Analytics Data1",
+"type": "n8n-nodes-base.code",
+"position": [
+1940,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "d8f775cd-daf9-42de-a527-d932be46d945",
+"name": "Get Country views data for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+2120,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "newUsers",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"listName": "other"
+},
+{
+"name": "sessions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "country",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "7119e57c-cbf4-49a9-b0c9-1f3da1fd2af3",
+"name": "Get Country views data for last week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+2440,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "={{$today.minus({days: 7})}}",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "newUsers",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"listName": "other"
+},
+{
+"name": "sessions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "country",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"typeVersion": 2
+},
+{
+"id": "546d6cd2-6db6-4276-be35-abbe5a7e9b6a",
+"name": "Parse Google analytics data",
+"type": "n8n-nodes-base.code",
+"position": [
+2280,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "87cb137c-686d-49a5-8657-06ed0c5f5c27",
+"name": "Parse Google analytics data1",
+"type": "n8n-nodes-base.code",
+"position": [
+2600,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "06c4478d-a13a-4587-9f1f-451a68798a9f",
+"name": "Send page data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2760,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $json.urlString }} Data from last week: {{ $('Parse data from Google Analytics').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "4ad522b0-afe4-4eff-aa16-b86cc892ead8",
+"name": "Send page Search data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google Analytics Data1').item.json.urlString }} Data from last week:{{ $('Parse Google Analytics Data').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "07e1eebf-f16a-44c0-83b5-76bf65a3d3fc",
+"name": "Send country view data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3080,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google analytics data1').item.json.urlString }} Data from last week:{{ $('Parse Google analytics data').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "c4648ad8-2377-42a0-a431-931b53631c9d",
+"name": "Save A.I. output to Baserow",
+"type": "n8n-nodes-base.baserow",
+"position": [
+3240,
+460
+],
+"parameters": {
+"tableId": 601,
+"fieldsUi": {
+"fieldValues": [
+{
+"fieldId": 5833,
+"fieldValue": "Name of your blog"
+},
+{
+"fieldId": 5831,
+"fieldValue": "={{ $('Send page data to A.I.').item.json.choices[0].message.content }}"
+},
+{
+"fieldId": 5830,
+"fieldValue": "={{ $('Send page Search data to A.I.').item.json.choices[0].message.content }}"
+},
+{
+"fieldId": 5832,
+"fieldValue": "={{ $json.choices[0].message.content }}"
+},
+{
+"fieldId": 5829,
+"fieldValue": "={{ DateTime.now() }}"
+}
+]
+},
+"operation": "create",
+"databaseId": 121
+},
+"typeVersion": 1
+},
+{
+"id": "e185c836-c12f-4452-92bd-0daaf33b653a",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2760,
+180
+],
+"parameters": {
+"color": 5,
+"width": 441.7412987012988,
+"height": 508.95792207792226,
+"content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
+},
+"typeVersion": 1
+},
+{
+"id": "a1de2d16-d09e-4c74-8be1-f6bab8c34246",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3220,
+180
+],
+"parameters": {
+"color": 6,
+"width": 331.32883116883124,
+"height": 474.88,
+"content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Name\n- Country Views\n- Page Views\n- Search Report\n- Blog \n\nEnter the name of your website under \"Blog\" field."
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "ac4b5eac-1c84-49ce-9ff7-794f857265b4",
+"connections": {
+"Parse GA data": {
+"main": [
+[
+{
+"node": "Get Google Search Results for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Schedule Trigger": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send page data to A.I.": {
+"main": [
+[
+{
+"node": "Send page Search data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google Analytics Data": {
+"main": [
+[
+{
+"node": "Get Google Search Results for last week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google analytics data": {
+"main": [
+[
+{
+"node": "Get Country views data for last week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google Analytics Data1": {
+"main": [
+[
+{
+"node": "Get Country views data for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google analytics data1": {
+"main": [
+[
+{
+"node": "Send page data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send page Search data to A.I.": {
+"main": [
+[
+{
+"node": "Send country view data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send country view data to A.I.": {
+"main": [
+[
+{
+"node": "Save A.I. output to Baserow",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse data from Google Analytics": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for prior week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Country views data for last week": {
+"main": [
+[
+{
+"node": "Parse Google analytics data1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Country views data for this week": {
+"main": [
+[
+{
+"node": "Parse Google analytics data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Google Search Results for last week": {
+"main": [
+[
+{
+"node": "Parse Google Analytics Data1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Google Search Results for this week": {
+"main": [
+[
+{
+"node": "Parse Google Analytics Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Page Engagement Stats for this week": {
+"main": [
+[
+{
+"node": "Parse data from Google Analytics",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Page Engagement Stats for prior week": {
+"main": [
+[
+{
+"node": "Parse GA data",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.json b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.json
deleted file mode 100644
index 6158806..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.json
+++ /dev/null
@@ -1,853 +0,0 @@
-{
- "id": "K3uf8aY8wipScEay",
- "meta": {
- "instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
- "templateCredsSetupCompleted": true
- },
- "name": "Google analytics template",
- "tags": [],
- "nodes": [
- {
- "id": "6a9fc442-d0a3-48be-8dff-94f8d9cd5cf1",
- "name": "Schedule Trigger",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- 460,
- 460
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "484cbc41-f57d-4c3d-a458-e439d480d290",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 460,
- 640
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "b1b66e9b-5fea-407b-9c1e-39bd2a9d4a90",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 460,
- 100
- ],
- "parameters": {
- "width": 714.172987012987,
- "content": "## Send Google analytics to A.I. and save results to baserow\n\nThis workflow will check for country views, page engagement and google search console results. It will take this week's data and compare it to last week's data.\n\n[You can read more about this workflow here](https://rumjahn.com/how-i-used-a-i-to-be-an-seo-expert-and-analyzed-my-google-analytics-data-in-n8n-and-make-com/)"
- },
- "typeVersion": 1
- },
- {
- "id": "adde29fc-ddb5-4b50-aa78-313ac9ede879",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 633.6540259740264,
- 320
- ],
- "parameters": {
- "color": 4,
- "width": 2097.92831168831,
- "height": 342.6576623376624,
- "content": "## Property ID\n\n1. Create your [Google Analytics Credentials](https://docs.n8n.io/integrations/builtin/credentials/google/oauth-single-service/?utm_source=n8n_app&utm_medium=credential_settings&utm_campaign=create_new_credentials_modal)\n2. Enter your [property ID](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id)."
- },
- "typeVersion": 1
- },
- {
- "id": "f2fb8535-e81e-4ca1-80df-ee68edba6386",
- "name": "Get Page Engagement Stats for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 700,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "screenPageViews",
- "listName": "other"
- },
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "screenPageViewsPerUser",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "unifiedScreenName",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "1d761425-cebf-4787-b286-b723a0851485",
- "name": "Get Page Engagement Stats for prior week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1060,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "2024-10-23T00:00:00",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "screenPageViews",
- "listName": "other"
- },
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "screenPageViewsPerUser",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "unifiedScreenName",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "typeVersion": 2
- },
- {
- "id": "f8dac36b-9e8a-407f-b923-b4cea368f1bc",
- "name": "Parse data from Google Analytics",
- "type": "n8n-nodes-base.code",
- "position": [
- 880,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
- },
- "typeVersion": 2
- },
- {
- "id": "ed880442-c92e-4347-b277-e8794aea6fbc",
- "name": "Parse GA data",
- "type": "n8n-nodes-base.code",
- "position": [
- 1240,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
- },
- "typeVersion": 2
- },
- {
- "id": "46e092cc-af94-4e64-aa92-931c56345eff",
- "name": "Get Google Search Results for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1420,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchAveragePosition",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClickThroughRate",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClicks",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchImpressions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "landingPagePlusQueryString",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "709d0aaf-bd3d-4d83-9e66-b7df495855bd",
- "name": "Get Google Search Results for last week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 1780,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "={{$today.minus({days: 7})}}",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchAveragePosition",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClickThroughRate",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchClicks",
- "listName": "other"
- },
- {
- "name": "organicGoogleSearchImpressions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "landingPagePlusQueryString",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "7d3835d6-d1f5-4159-8e34-871871e63989",
- "name": "Parse Google Analytics Data",
- "type": "n8n-nodes-base.code",
- "position": [
- 1600,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "c018fda4-a2e6-48f4-aabb-039c66374dc7",
- "name": "Parse Google Analytics Data1",
- "type": "n8n-nodes-base.code",
- "position": [
- 1940,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "d8f775cd-daf9-42de-a527-d932be46d945",
- "name": "Get Country views data for this week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 2120,
- 460
- ],
- "parameters": {
- "simple": false,
- "returnAll": true,
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "newUsers",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "listName": "other"
- },
- {
- "name": "sessions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "country",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "credentials": {
- "googleAnalyticsOAuth2": {
- "id": "b1GX8VBMKCUNweV1",
- "name": "Google Analytics account"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "7119e57c-cbf4-49a9-b0c9-1f3da1fd2af3",
- "name": "Get Country views data for last week",
- "type": "n8n-nodes-base.googleAnalytics",
- "position": [
- 2440,
- 460
- ],
- "parameters": {
- "simple": false,
- "endDate": "={{$today.minus({days: 7})}}",
- "dateRange": "custom",
- "returnAll": true,
- "startDate": "={{$today.minus({days: 14})}}",
- "metricsGA4": {
- "metricValues": [
- {
- "name": "activeUsers",
- "listName": "other"
- },
- {
- "name": "newUsers",
- "listName": "other"
- },
- {
- "name": "engagementRate",
- "listName": "other"
- },
- {
- "name": "engagedSessions",
- "listName": "other"
- },
- {
- "name": "eventCount",
- "listName": "other"
- },
- {
- "listName": "other"
- },
- {
- "name": "sessions",
- "listName": "other"
- }
- ]
- },
- "propertyId": {
- "__rl": true,
- "mode": "id",
- "value": "460520224"
- },
- "dimensionsGA4": {
- "dimensionValues": [
- {
- "name": "country",
- "listName": "other"
- }
- ]
- },
- "additionalFields": {}
- },
- "typeVersion": 2
- },
- {
- "id": "546d6cd2-6db6-4276-be35-abbe5a7e9b6a",
- "name": "Parse Google analytics data",
- "type": "n8n-nodes-base.code",
- "position": [
- 2280,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "87cb137c-686d-49a5-8657-06ed0c5f5c27",
- "name": "Parse Google analytics data1",
- "type": "n8n-nodes-base.code",
- "position": [
- 2600,
- 460
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "06c4478d-a13a-4587-9f1f-451a68798a9f",
- "name": "Send page data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2760,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $json.urlString }} Data from last week: {{ $('Parse data from Google Analytics').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "4ad522b0-afe4-4eff-aa16-b86cc892ead8",
- "name": "Send page Search data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google Analytics Data1').item.json.urlString }} Data from last week:{{ $('Parse Google Analytics Data').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "07e1eebf-f16a-44c0-83b5-76bf65a3d3fc",
- "name": "Send country view data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3080,
- 460
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google analytics data1').item.json.urlString }} Data from last week:{{ $('Parse Google analytics data').item.json.urlString }}\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "c4648ad8-2377-42a0-a431-931b53631c9d",
- "name": "Save A.I. output to Baserow",
- "type": "n8n-nodes-base.baserow",
- "position": [
- 3240,
- 460
- ],
- "parameters": {
- "tableId": 601,
- "fieldsUi": {
- "fieldValues": [
- {
- "fieldId": 5833,
- "fieldValue": "Name of your blog"
- },
- {
- "fieldId": 5831,
- "fieldValue": "={{ $('Send page data to A.I.').item.json.choices[0].message.content }}"
- },
- {
- "fieldId": 5830,
- "fieldValue": "={{ $('Send page Search data to A.I.').item.json.choices[0].message.content }}"
- },
- {
- "fieldId": 5832,
- "fieldValue": "={{ $json.choices[0].message.content }}"
- },
- {
- "fieldId": 5829,
- "fieldValue": "={{ DateTime.now() }}"
- }
- ]
- },
- "operation": "create",
- "databaseId": 121
- },
- "typeVersion": 1
- },
- {
- "id": "e185c836-c12f-4452-92bd-0daaf33b653a",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2760,
- 180
- ],
- "parameters": {
- "color": 5,
- "width": 441.7412987012988,
- "height": 508.95792207792226,
- "content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
- },
- "typeVersion": 1
- },
- {
- "id": "a1de2d16-d09e-4c74-8be1-f6bab8c34246",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3220,
- 180
- ],
- "parameters": {
- "color": 6,
- "width": 331.32883116883124,
- "height": 474.88,
- "content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Name\n- Country Views\n- Page Views\n- Search Report\n- Blog \n\nEnter the name of your website under \"Blog\" field."
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "ac4b5eac-1c84-49ce-9ff7-794f857265b4",
- "connections": {
- "Parse GA data": {
- "main": [
- [
- {
- "node": "Get Google Search Results for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Schedule Trigger": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send page data to A.I.": {
- "main": [
- [
- {
- "node": "Send page Search data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google Analytics Data": {
- "main": [
- [
- {
- "node": "Get Google Search Results for last week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google analytics data": {
- "main": [
- [
- {
- "node": "Get Country views data for last week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google Analytics Data1": {
- "main": [
- [
- {
- "node": "Get Country views data for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Google analytics data1": {
- "main": [
- [
- {
- "node": "Send page data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send page Search data to A.I.": {
- "main": [
- [
- {
- "node": "Send country view data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send country view data to A.I.": {
- "main": [
- [
- {
- "node": "Save A.I. output to Baserow",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse data from Google Analytics": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for prior week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get Page Engagement Stats for this week",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Country views data for last week": {
- "main": [
- [
- {
- "node": "Parse Google analytics data1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Country views data for this week": {
- "main": [
- [
- {
- "node": "Parse Google analytics data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Google Search Results for last week": {
- "main": [
- [
- {
- "node": "Parse Google Analytics Data1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Google Search Results for this week": {
- "main": [
- [
- {
- "node": "Parse Google Analytics Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Page Engagement Stats for this week": {
- "main": [
- [
- {
- "node": "Parse data from Google Analytics",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Page Engagement Stats for prior week": {
- "main": [
- [
- {
- "node": "Parse GA data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.txt b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.txt
new file mode 100644
index 0000000..451c870
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Send Google analytics data to A.I. to analyze then save results in BaserowSend Google analytics data to A.I. to analyze then save results in Baserow.txt
@@ -0,0 +1,853 @@
+{
+"id": "K3uf8aY8wipScEay",
+"meta": {
+"instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
+"templateCredsSetupCompleted": true
+},
+"name": "Google analytics template",
+"tags": [],
+"nodes": [
+{
+"id": "6a9fc442-d0a3-48be-8dff-94f8d9cd5cf1",
+"name": "Schedule Trigger",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+460,
+460
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "484cbc41-f57d-4c3d-a458-e439d480d290",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+460,
+640
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "b1b66e9b-5fea-407b-9c1e-39bd2a9d4a90",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+460,
+100
+],
+"parameters": {
+"width": 714.172987012987,
+"content": "## Send Google analytics to A.I. and save results to baserow\n\nThis workflow will check for country views, page engagement and google search console results. It will take this week's data and compare it to last week's data.\n\n[You can read more about this workflow here](https://rumjahn.com/how-i-used-a-i-to-be-an-seo-expert-and-analyzed-my-google-analytics-data-in-n8n-and-make-com/)"
+},
+"typeVersion": 1
+},
+{
+"id": "adde29fc-ddb5-4b50-aa78-313ac9ede879",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+633.6540259740264,
+320
+],
+"parameters": {
+"color": 4,
+"width": 2097.92831168831,
+"height": 342.6576623376624,
+"content": "## Property ID\n\n1. Create your [Google Analytics Credentials](https://docs.n8n.io/integrations/builtin/credentials/google/oauth-single-service/?utm_source=n8n_app&utm_medium=credential_settings&utm_campaign=create_new_credentials_modal)\n2. Enter your [property ID](https://developers.google.com/analytics/devguides/reporting/data/v1/property-id)."
+},
+"typeVersion": 1
+},
+{
+"id": "f2fb8535-e81e-4ca1-80df-ee68edba6386",
+"name": "Get Page Engagement Stats for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+700,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "screenPageViews",
+"listName": "other"
+},
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "screenPageViewsPerUser",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "unifiedScreenName",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "1d761425-cebf-4787-b286-b723a0851485",
+"name": "Get Page Engagement Stats for prior week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1060,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "2024-10-23T00:00:00",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "screenPageViews",
+"listName": "other"
+},
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "screenPageViewsPerUser",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "unifiedScreenName",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"typeVersion": 2
+},
+{
+"id": "f8dac36b-9e8a-407f-b923-b4cea368f1bc",
+"name": "Parse data from Google Analytics",
+"type": "n8n-nodes-base.code",
+"position": [
+880,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
+},
+"typeVersion": 2
+},
+{
+"id": "ed880442-c92e-4347-b277-e8794aea6fbc",
+"name": "Parse GA data",
+"type": "n8n-nodes-base.code",
+"position": [
+1240,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // Debug logging\n console.log('Input items:', JSON.stringify(items, null, 2));\n \n // Check if items is an array and has content\n if (!Array.isArray(items) || items.length === 0) {\n console.log('Items is not an array or is empty');\n throw new Error('Invalid data structure');\n }\n\n // Check if first item exists and has json property\n if (!items[0] || !items[0].json) {\n console.log('First item is missing or has no json property');\n throw new Error('Invalid data structure');\n }\n\n // Get the analytics data\n const analyticsData = items[0].json;\n \n // Check if analyticsData has rows\n if (!analyticsData || !Array.isArray(analyticsData.rows)) {\n console.log('Analytics data is missing or has no rows array');\n throw new Error('Invalid data structure');\n }\n \n // Map each row to a simplified object\n const simplified = analyticsData.rows.map(row => {\n if (!row.dimensionValues?.[0]?.value || !row.metricValues?.length) {\n console.log('Invalid row structure:', row);\n throw new Error('Invalid row structure');\n }\n \n return {\n page: row.dimensionValues[0].value,\n pageViews: parseInt(row.metricValues[0].value) || 0,\n activeUsers: parseInt(row.metricValues[1].value) || 0,\n viewsPerUser: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0\n };\n });\n \n // Convert to JSON string and encode for URL\n return encodeURIComponent(JSON.stringify(simplified));\n}\n\n// Get input data and transform it\nconst urlString = transformToUrlString($input.all());\n\n// Return the result\nreturn { json: { urlString } };"
+},
+"typeVersion": 2
+},
+{
+"id": "46e092cc-af94-4e64-aa92-931c56345eff",
+"name": "Get Google Search Results for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1420,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchAveragePosition",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClickThroughRate",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClicks",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchImpressions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "landingPagePlusQueryString",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "709d0aaf-bd3d-4d83-9e66-b7df495855bd",
+"name": "Get Google Search Results for last week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+1780,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "={{$today.minus({days: 7})}}",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchAveragePosition",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClickThroughRate",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchClicks",
+"listName": "other"
+},
+{
+"name": "organicGoogleSearchImpressions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "landingPagePlusQueryString",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "7d3835d6-d1f5-4159-8e34-871871e63989",
+"name": "Parse Google Analytics Data",
+"type": "n8n-nodes-base.code",
+"position": [
+1600,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "c018fda4-a2e6-48f4-aabb-039c66374dc7",
+"name": "Parse Google Analytics Data1",
+"type": "n8n-nodes-base.code",
+"position": [
+1940,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n page: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n engagedSessions: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n eventCount: parseInt(row.metricValues[3].value) || 0,\n avgPosition: parseFloat(row.metricValues[4].value) || 0,\n ctr: parseFloat(row.metricValues[5].value) || 0,\n clicks: parseInt(row.metricValues[6].value) || 0,\n impressions: parseInt(row.metricValues[7].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "d8f775cd-daf9-42de-a527-d932be46d945",
+"name": "Get Country views data for this week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+2120,
+460
+],
+"parameters": {
+"simple": false,
+"returnAll": true,
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "newUsers",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"listName": "other"
+},
+{
+"name": "sessions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "country",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"credentials": {
+"googleAnalyticsOAuth2": {
+"id": "b1GX8VBMKCUNweV1",
+"name": "Google Analytics account"
+}
+},
+"typeVersion": 2
+},
+{
+"id": "7119e57c-cbf4-49a9-b0c9-1f3da1fd2af3",
+"name": "Get Country views data for last week",
+"type": "n8n-nodes-base.googleAnalytics",
+"position": [
+2440,
+460
+],
+"parameters": {
+"simple": false,
+"endDate": "={{$today.minus({days: 7})}}",
+"dateRange": "custom",
+"returnAll": true,
+"startDate": "={{$today.minus({days: 14})}}",
+"metricsGA4": {
+"metricValues": [
+{
+"name": "activeUsers",
+"listName": "other"
+},
+{
+"name": "newUsers",
+"listName": "other"
+},
+{
+"name": "engagementRate",
+"listName": "other"
+},
+{
+"name": "engagedSessions",
+"listName": "other"
+},
+{
+"name": "eventCount",
+"listName": "other"
+},
+{
+"listName": "other"
+},
+{
+"name": "sessions",
+"listName": "other"
+}
+]
+},
+"propertyId": {
+"__rl": true,
+"mode": "id",
+"value": "460520224"
+},
+"dimensionsGA4": {
+"dimensionValues": [
+{
+"name": "country",
+"listName": "other"
+}
+]
+},
+"additionalFields": {}
+},
+"typeVersion": 2
+},
+{
+"id": "546d6cd2-6db6-4276-be35-abbe5a7e9b6a",
+"name": "Parse Google analytics data",
+"type": "n8n-nodes-base.code",
+"position": [
+2280,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "87cb137c-686d-49a5-8657-06ed0c5f5c27",
+"name": "Parse Google analytics data1",
+"type": "n8n-nodes-base.code",
+"position": [
+2600,
+460
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data || !data.rows) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Process each row\n const simplified = data.rows.map(row => ({\n country: row.dimensionValues[0].value,\n activeUsers: parseInt(row.metricValues[0].value) || 0,\n newUsers: parseInt(row.metricValues[1].value) || 0,\n engagementRate: parseFloat(row.metricValues[2].value) || 0,\n engagedSessions: parseInt(row.metricValues[3].value) || 0,\n eventCount: parseInt(row.metricValues[4].value) || 0,\n totalUsers: parseInt(row.metricValues[5].value) || 0,\n sessions: parseInt(row.metricValues[6].value) || 0\n }));\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "06c4478d-a13a-4587-9f1f-451a68798a9f",
+"name": "Send page data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2760,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $json.urlString }} Data from last week: {{ $('Parse data from Google Analytics').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "4ad522b0-afe4-4eff-aa16-b86cc892ead8",
+"name": "Send page Search data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google Analytics Data1').item.json.urlString }} Data from last week:{{ $('Parse Google Analytics Data').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "07e1eebf-f16a-44c0-83b5-76bf65a3d3fc",
+"name": "Send country view data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3080,
+460
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Compare the data from past 2 weeks, give me a table in markdown. Then give me 5 suggestions to improve my SEO. Output the data so that it works with markdown editors. Data from 2 weeks ago:{{ $('Parse Google analytics data1').item.json.urlString }} Data from last week:{{ $('Parse Google analytics data').item.json.urlString }}\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "c4648ad8-2377-42a0-a431-931b53631c9d",
+"name": "Save A.I. output to Baserow",
+"type": "n8n-nodes-base.baserow",
+"position": [
+3240,
+460
+],
+"parameters": {
+"tableId": 601,
+"fieldsUi": {
+"fieldValues": [
+{
+"fieldId": 5833,
+"fieldValue": "Name of your blog"
+},
+{
+"fieldId": 5831,
+"fieldValue": "={{ $('Send page data to A.I.').item.json.choices[0].message.content }}"
+},
+{
+"fieldId": 5830,
+"fieldValue": "={{ $('Send page Search data to A.I.').item.json.choices[0].message.content }}"
+},
+{
+"fieldId": 5832,
+"fieldValue": "={{ $json.choices[0].message.content }}"
+},
+{
+"fieldId": 5829,
+"fieldValue": "={{ DateTime.now() }}"
+}
+]
+},
+"operation": "create",
+"databaseId": 121
+},
+"typeVersion": 1
+},
+{
+"id": "e185c836-c12f-4452-92bd-0daaf33b653a",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2760,
+180
+],
+"parameters": {
+"color": 5,
+"width": 441.7412987012988,
+"height": 508.95792207792226,
+"content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
+},
+"typeVersion": 1
+},
+{
+"id": "a1de2d16-d09e-4c74-8be1-f6bab8c34246",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3220,
+180
+],
+"parameters": {
+"color": 6,
+"width": 331.32883116883124,
+"height": 474.88,
+"content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Name\n- Country Views\n- Page Views\n- Search Report\n- Blog \n\nEnter the name of your website under \"Blog\" field."
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "ac4b5eac-1c84-49ce-9ff7-794f857265b4",
+"connections": {
+"Parse GA data": {
+"main": [
+[
+{
+"node": "Get Google Search Results for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Schedule Trigger": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send page data to A.I.": {
+"main": [
+[
+{
+"node": "Send page Search data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google Analytics Data": {
+"main": [
+[
+{
+"node": "Get Google Search Results for last week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google analytics data": {
+"main": [
+[
+{
+"node": "Get Country views data for last week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google Analytics Data1": {
+"main": [
+[
+{
+"node": "Get Country views data for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Google analytics data1": {
+"main": [
+[
+{
+"node": "Send page data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send page Search data to A.I.": {
+"main": [
+[
+{
+"node": "Send country view data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send country view data to A.I.": {
+"main": [
+[
+{
+"node": "Save A.I. output to Baserow",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse data from Google Analytics": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for prior week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get Page Engagement Stats for this week",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Country views data for last week": {
+"main": [
+[
+{
+"node": "Parse Google analytics data1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Country views data for this week": {
+"main": [
+[
+{
+"node": "Parse Google analytics data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Google Search Results for last week": {
+"main": [
+[
+{
+"node": "Parse Google Analytics Data1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Google Search Results for this week": {
+"main": [
+[
+{
+"node": "Parse Google Analytics Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Page Engagement Stats for this week": {
+"main": [
+[
+{
+"node": "Parse data from Google Analytics",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Page Engagement Stats for prior week": {
+"main": [
+[
+{
+"node": "Parse GA data",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.json b/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.json
deleted file mode 100644
index 808a87c..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.json
+++ /dev/null
@@ -1,1850 +0,0 @@
-{
- "id": "vzU9QRZsHcyRsord",
- "meta": {
- "instanceId": "a9f3b18652ddc96459b459de4fa8fa33252fb820a9e5a1593074f3580352864a",
- "templateCredsSetupCompleted": true
- },
- "name": "Spot Workplace Discrimination Patterns with AI",
- "tags": [
- {
- "id": "76EYz9X3GU4PtgSS",
- "name": "human_resources",
- "createdAt": "2025-01-30T18:52:17.614Z",
- "updatedAt": "2025-01-30T18:52:17.614Z"
- },
- {
- "id": "ey2Mx4vNaV8cKvao",
- "name": "openai",
- "createdAt": "2024-12-23T07:10:13.400Z",
- "updatedAt": "2024-12-23T07:10:13.400Z"
- }
- ],
- "nodes": [
- {
- "id": "b508ab50-158a-4cbf-a52e-f53e1804e770",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 280,
- 380
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "11a1a2d5-a274-44f7-97ca-5666a59fcb31",
- "name": "OpenAI Chat Model1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2220,
- 800
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "XXXXXX",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "395f7b67-c914-4aae-8727-0573fdbfc6ad",
- "name": "OpenAI Chat Model2",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2220,
- 380
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "XXXXXX",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "6ab194a9-b869-4296-aea9-19afcbffc0d7",
- "name": "Merge",
- "type": "n8n-nodes-base.merge",
- "position": [
- 2940,
- 600
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combineBy": "combineByPosition"
- },
- "typeVersion": 3
- },
- {
- "id": "1eba1dd7-a164-4c70-8c75-759532bd16a0",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 3840,
- 420
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "XXXXXX",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "f25f1b07-cded-4ca7-9655-8b8f463089ab",
- "name": "SET company_name",
- "type": "n8n-nodes-base.set",
- "position": [
- 540,
- 380
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "dd256ef7-013c-4769-8580-02c2d902d0b2",
- "name": "company_name",
- "type": "string",
- "value": "=Twilio"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "87264a93-ab97-4e39-8d40-43365189f704",
- "name": "Define dictionary of demographic keys",
- "type": "n8n-nodes-base.set",
- "position": [
- 740,
- 380
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "6ae671be-45d0-4a94-a443-2f1d4772d31b",
- "name": "asian",
- "type": "string",
- "value": "Asian"
- },
- {
- "id": "6c93370c-996c-44a6-a34c-4cd3baeeb846",
- "name": "hispanic",
- "type": "string",
- "value": "Hispanic or Latinx"
- },
- {
- "id": "dee79039-6051-4e9d-98b5-63a07d30f6b0",
- "name": "white",
- "type": "string",
- "value": "White"
- },
- {
- "id": "08d42380-8397-412f-8459-7553e9309b5d",
- "name": "pacific_islander",
- "type": "string",
- "value": "Native Hawaiian or other Pacific Islander"
- },
- {
- "id": "09e8ebc5-e7e7-449a-9036-9b9b54cdc828",
- "name": "black",
- "type": "string",
- "value": "Black or African American"
- },
- {
- "id": "39e910f8-3a8b-4233-a93a-3c5693e808c6",
- "name": "middle_eastern",
- "type": "string",
- "value": "Middle Eastern"
- },
- {
- "id": "169b3471-efa0-476e-aa83-e3f717c568f1",
- "name": "indigenous",
- "type": "string",
- "value": "Indigenous American or Native Alaskan"
- },
- {
- "id": "b6192296-4efa-4af5-ae02-1e31d28aae90",
- "name": "male",
- "type": "string",
- "value": "Men"
- },
- {
- "id": "4b322294-940c-459d-b083-8e91e38193f7",
- "name": "female",
- "type": "string",
- "value": "Women"
- },
- {
- "id": "1940eef0-6b76-4a26-9d8f-7c8536fbcb1b",
- "name": "trans",
- "type": "string",
- "value": "Transgender and/or Non-Binary"
- },
- {
- "id": "3dba3e18-2bb1-4078-bde9-9d187f9628dd",
- "name": "hetero",
- "type": "string",
- "value": "Heterosexual"
- },
- {
- "id": "9b7d10ad-1766-4b18-a230-3bd80142b48c",
- "name": "lgbtqia",
- "type": "string",
- "value": "LGBTQ+"
- },
- {
- "id": "458636f8-99e8-4245-9950-94e4cf68e371",
- "name": "nondisabled",
- "type": "string",
- "value": "Non-Disabled"
- },
- {
- "id": "a466e258-7de1-4453-a126-55f780094236",
- "name": "disabled",
- "type": "string",
- "value": "People with Disabilities"
- },
- {
- "id": "98735266-0451-432f-be7c-efcb09512cb1",
- "name": "caregiver",
- "type": "string",
- "value": "Caregivers"
- },
- {
- "id": "ebe2353c-9ff5-47bc-8c11-b66d3436f5b4",
- "name": "parent",
- "type": "string",
- "value": "Parents/Guardians"
- },
- {
- "id": "ab51c80c-d81d-41ab-94d9-c0a263743c17",
- "name": "nonparent",
- "type": "string",
- "value": "Not a Parent or Caregiver"
- },
- {
- "id": "cb7df429-c600-43f4-aa7e-dbc2382a85a0",
- "name": "nonveteran",
- "type": "string",
- "value": "Non-Veterans"
- },
- {
- "id": "dffbdb13-189a-462d-83d1-c5ec39a17d41",
- "name": "veteran",
- "type": "string",
- "value": "Veterans"
- }
- ]
- },
- "includeOtherFields": true
- },
- "typeVersion": 3.4
- },
- {
- "id": "862f1c77-44a8-4d79-abac-33351ebb731b",
- "name": "ScrapingBee Search Glassdoor",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 940,
- 380
- ],
- "parameters": {
- "url": "https://app.scrapingbee.com/api/v1",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "url",
- "value": "=https://www.glassdoor.com/Search/results.htm?keyword={{ $json.company_name.toLowerCase().urlEncode() }}"
- },
- {
- "name": "premium_proxy",
- "value": "true"
- },
- {
- "name": "block_resources",
- "value": "false"
- },
- {
- "name": "stealth_proxy",
- "value": "true"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "XXXXXX",
- "name": "ScrapingBee Query Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "4c9bf05e-9c50-4895-b20b-b7c329104615",
- "name": "Extract company url path",
- "type": "n8n-nodes-base.html",
- "position": [
- 1140,
- 380
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "url_path",
- "attribute": "href",
- "cssSelector": "body main div a",
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "d20bb0e7-4ca7-41d0-a3e9-41abc811b064",
- "name": "ScrapingBee GET company page contents",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1340,
- 380
- ],
- "parameters": {
- "url": "https://app.scrapingbee.com/api/v1",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "url",
- "value": "=https://www.glassdoor.com{{ $json.url_path }}"
- },
- {
- "name": "premium_proxy",
- "value": "true"
- },
- {
- "name": "block_resources",
- "value": "false"
- },
- {
- "name": "stealth_proxy",
- "value": "true"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "XXXXXX",
- "name": "ScrapingBee Query Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "fce70cab-8ce3-4ce2-b040-ce80d66b1e62",
- "name": "Extract reviews page url path",
- "type": "n8n-nodes-base.html",
- "position": [
- 1540,
- 380
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "url_path",
- "attribute": "href",
- "cssSelector": "#reviews a",
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "d2e7fee9-e3d4-42bf-8be6-38b352371273",
- "name": "ScrapingBee GET Glassdoor Reviews Content",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1760,
- 380
- ],
- "parameters": {
- "url": "https://app.scrapingbee.com/api/v1",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "url",
- "value": "=https://www.glassdoor.com{{ $json.url_path }}"
- },
- {
- "name": "premium_proxy",
- "value": "True"
- },
- {
- "name": "block_resources",
- "value": "False"
- },
- {
- "name": "stealth_proxy",
- "value": "true"
- }
- ]
- }
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "XXXXXX",
- "name": "ScrapingBee Query Auth"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "0c322823-0569-4bd5-9c4e-af3de0f8d7b4",
- "name": "Extract Overall Review Summary",
- "type": "n8n-nodes-base.html",
- "position": [
- 1980,
- 260
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "review_summary",
- "cssSelector": "div[data-test=\"review-summary\"]",
- "returnValue": "html"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "851305ba-0837-4be9-943d-7282e8d74aee",
- "name": "Extract Demographics Module",
- "type": "n8n-nodes-base.html",
- "position": [
- 1980,
- 520
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "demographics_content",
- "cssSelector": "div[data-test=\"demographics-module\"]",
- "returnValue": "html"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "cf9a6ee2-53b5-4fbf-a36c-4b9dab53b795",
- "name": "Extract overall ratings and distribution percentages",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 2200,
- 200
- ],
- "parameters": {
- "text": "={{ $json.review_summary }}",
- "options": {},
- "attributes": {
- "attributes": [
- {
- "name": "average_rating",
- "type": "number",
- "required": true,
- "description": "The overall average rating for this company."
- },
- {
- "name": "total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "The total number of reviews for this company."
- },
- {
- "name": "5_star_distribution_percentage",
- "type": "number",
- "required": true,
- "description": "The percentage distribution of 5 star reviews"
- },
- {
- "name": "4_star_distribution_percentage",
- "type": "number",
- "required": true,
- "description": "The percentage distribution of 4 star reviews"
- },
- {
- "name": "3_star_distribution_percentage",
- "type": "number",
- "required": true,
- "description": "The percentage distribution of 3 star reviews"
- },
- {
- "name": "2_star_distribution_percentage",
- "type": "number",
- "required": true,
- "description": "The percentage distribution of 2 star reviews"
- },
- {
- "name": "1_star_distribution_percentage",
- "type": "number",
- "required": true,
- "description": "The percentage distribution of 1 star reviews"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "ae164f6e-04e7-4d8b-951e-a17085956f4b",
- "name": "Extract demographic distributions",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 2200,
- 620
- ],
- "parameters": {
- "text": "={{ $json.demographics_content }}",
- "options": {
- "systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, you may use 0 for the attribute's value."
- },
- "attributes": {
- "attributes": [
- {
- "name": "asian_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as asian."
- },
- {
- "name": "asian_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as asian."
- },
- {
- "name": "hispanic_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as hispanic."
- },
- {
- "name": "hispanic_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as hispanic."
- },
- {
- "name": "white_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as white."
- },
- {
- "name": "white_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as white."
- },
- {
- "name": "pacific_islander_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as native hawaiian or pacific islander."
- },
- {
- "name": "pacific_islander_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as native hawaiian or pacific islander."
- },
- {
- "name": "black_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as black."
- },
- {
- "name": "black_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as black."
- },
- {
- "name": "middle_eastern_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as middle eastern."
- },
- {
- "name": "middle_eastern_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as middle_eastern."
- },
- {
- "name": "indigenous_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as indigenous american or native alaskan."
- },
- {
- "name": "indigenous_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as indigenous american or native alaskan."
- },
- {
- "name": "male_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as men."
- },
- {
- "name": "male_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as men."
- },
- {
- "name": "female_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as women."
- },
- {
- "name": "female_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as women."
- },
- {
- "name": "trans_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as transgender and/or non-binary."
- },
- {
- "name": "trans_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as trans and/or non-binary."
- },
- {
- "name": "hetero_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as heterosexual."
- },
- {
- "name": "hetero_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as heterosexual."
- },
- {
- "name": "lgbtqia_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as lgbtqia+."
- },
- {
- "name": "lgbtqia_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as lgbtqia+."
- },
- {
- "name": "nondisabled_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as non-disabled."
- },
- {
- "name": "nondisabled_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as non-disabled."
- },
- {
- "name": "disabled_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as people with disabilities."
- },
- {
- "name": "disabled_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as people with disabilities."
- },
- {
- "name": "caregiver_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as caregivers."
- },
- {
- "name": "caregiver_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as caregivers."
- },
- {
- "name": "parent_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as parents/guardians."
- },
- {
- "name": "parent_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as parents/guardians."
- },
- {
- "name": "nonparent_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as not a parent or caregiver."
- },
- {
- "name": "nonparent_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as not a parent or guardian."
- },
- {
- "name": "nonveteran_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as non-veterans."
- },
- {
- "name": "nonveteran_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as non-veterans."
- },
- {
- "name": "veteran_average_rating",
- "type": "number",
- "required": true,
- "description": "=The average rating for this company by employees who self identified as veterans."
- },
- {
- "name": "veteran_total_number_of_reviews",
- "type": "number",
- "required": true,
- "description": "=The number of reviews for this company by employees who self-identified as veterans."
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "c8d9e45c-7d41-47bd-b9a9-0fa70de5d154",
- "name": "Define contributions to variance",
- "type": "n8n-nodes-base.set",
- "position": [
- 2560,
- 200
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "7360b2c2-1e21-45de-8d1a-e72b8abcb56b",
- "name": "contribution_to_variance.5_star",
- "type": "number",
- "value": "={{ ($json.output['5_star_distribution_percentage'] / 100) * Math.pow(5 - $json.output.average_rating,2) }}"
- },
- {
- "id": "acdd308a-fa33-4e33-b71b-36b9441bfa06",
- "name": "contribution_to_variance.4_star",
- "type": "number",
- "value": "={{ ($json.output['4_star_distribution_percentage'] / 100) * Math.pow(4 - $json.output.average_rating,2) }}"
- },
- {
- "id": "376818f3-d429-4abe-8ece-e8e9c5585826",
- "name": "contribution_to_variance.3_star",
- "type": "number",
- "value": "={{ ($json.output['3_star_distribution_percentage'] / 100) * Math.pow(3 - $json.output.average_rating,2) }}"
- },
- {
- "id": "620d5c37-8b93-4d39-9963-b7ce3a7f431e",
- "name": "contribution_to_variance.2_star",
- "type": "number",
- "value": "={{ ($json.output['2_star_distribution_percentage'] / 100) * Math.pow(2 - $json.output.average_rating,2) }}"
- },
- {
- "id": "76357980-4f9b-4b14-be68-6498ba25af67",
- "name": "contribution_to_variance.1_star",
- "type": "number",
- "value": "={{ ($json.output['1_star_distribution_percentage'] / 100) * Math.pow(1 - $json.output.average_rating,2) }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "8ea03017-d5d6-46ef-a5f1-dae4372f6256",
- "name": "Set variance and std_dev",
- "type": "n8n-nodes-base.set",
- "position": [
- 2740,
- 200
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "3217d418-f1b0-45ff-9f9a-6e6145cc29ca",
- "name": "variance",
- "type": "number",
- "value": "={{ $json.contribution_to_variance.values().sum() }}"
- },
- {
- "id": "acdb9fea-15ec-46ed-bde9-073e93597f17",
- "name": "average_rating",
- "type": "number",
- "value": "={{ $('Extract overall ratings and distribution percentages').item.json.output.average_rating }}"
- },
- {
- "id": "1f3a8a29-4bd4-4b40-8694-c74a0285eadb",
- "name": "total_number_of_reviews",
- "type": "number",
- "value": "={{ $('Extract overall ratings and distribution percentages').item.json.output.total_number_of_reviews }}"
- },
- {
- "id": "1906c796-1964-446b-8b56-d856269da938",
- "name": "std_dev",
- "type": "number",
- "value": "={{ Math.sqrt($json.contribution_to_variance.values().sum()) }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "0570d531-8480-4446-8f02-18640b4b891e",
- "name": "Calculate P-Scores",
- "type": "n8n-nodes-base.code",
- "position": [
- 3340,
- 440
- ],
- "parameters": {
- "jsCode": "// Approximate CDF for standard normal distribution\nfunction normSDist(z) {\n const t = 1 / (1 + 0.3275911 * Math.abs(z));\n const d = 0.254829592 * t - 0.284496736 * t * t + 1.421413741 * t * t * t - 1.453152027 * t * t * t * t + 1.061405429 * t * t * t * t * t;\n return 0.5 * (1 + Math.sign(z) * d * Math.exp(-z * z / 2));\n}\n\nfor (const item of $input.all()) {\n if (!item.json.population_analysis.p_scores) {\n item.json.population_analysis.p_scores = {};\n }\n\n for (const score of Object.keys(item.json.population_analysis.z_scores)) {\n // Check if review count exists and is greater than zero\n if (item.json.population_analysis.review_count[score] > 0) {\n // Apply the p_score formula: 2 * NORM.S.DIST(-ABS(z_score))\n const p_score = 2 * normSDist(-Math.abs(item.json.population_analysis.z_scores[score]));\n\n // Store the calculated p_score\n item.json.population_analysis.p_scores[score] = p_score;\n } else {\n // Remove z_scores, effect_sizes, and p_scores for groups with no reviews\n delete item.json.population_analysis.z_scores[score];\n delete item.json.population_analysis.effect_sizes[score];\n delete item.json.population_analysis.p_scores[score];\n }\n }\n}\n\nreturn $input.all();"
- },
- "typeVersion": 2
- },
- {
- "id": "0bdb9732-67ef-440d-bdd2-42c4f64ff6b6",
- "name": "Sort Effect Sizes",
- "type": "n8n-nodes-base.set",
- "position": [
- 3540,
- 440
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "61cf92ba-bc4e-40b8-a234-9b993fd24019",
- "name": "population_analysis.effect_sizes",
- "type": "object",
- "value": "={{ Object.fromEntries(Object.entries($json.population_analysis.effect_sizes).sort(([,a],[,b]) => a-b )) }}"
- }
- ]
- },
- "includeOtherFields": true
- },
- "typeVersion": 3.4
- },
- {
- "id": "fd9026ef-e993-410a-87d6-40a3ad10b7a7",
- "name": "Calculate Z-Scores and Effect Sizes",
- "type": "n8n-nodes-base.set",
- "position": [
- 3140,
- 600
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "790a53e8-5599-45d3-880e-ab1ad7d165d2",
- "name": "population_analysis.z_scores.asian",
- "type": "number",
- "value": "={{ ($json.output.asian_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.asian_total_number_of_reviews)) }}"
- },
- {
- "id": "ebd61097-8773-45b9-a8e6-cdd840d73650",
- "name": "population_analysis.effect_sizes.asian",
- "type": "number",
- "value": "={{ ($json.output.asian_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "627b1293-efdc-485a-83c8-bd332d6dc225",
- "name": "population_analysis.z_scores.hispanic",
- "type": "number",
- "value": "={{ ($json.output.hispanic_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.hispanic_total_number_of_reviews)) }}"
- },
- {
- "id": "822028d0-e94f-4cf7-9e13-8f8cc5c72ec0",
- "name": "population_analysis.z_scores.white",
- "type": "number",
- "value": "={{ ($json.output.white_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.white_total_number_of_reviews)) }}"
- },
- {
- "id": "d32321f9-0fcf-4e54-9059-c3fd5a901ce0",
- "name": "population_analysis.z_scores.pacific_islander",
- "type": "number",
- "value": "={{ ($json.output.pacific_islander_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.pacific_islander_total_number_of_reviews)) }}"
- },
- {
- "id": "e212d683-247f-45c4-9668-c290230a10ed",
- "name": "population_analysis.z_scores.black",
- "type": "number",
- "value": "={{ ($json.output.black_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.black_total_number_of_reviews)) }}"
- },
- {
- "id": "882049c3-eb81-4c09-af0c-5c79b0ef0154",
- "name": "population_analysis.z_scores.middle_eastern",
- "type": "number",
- "value": "={{ ($json.output.middle_eastern_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.middle_eastern_total_number_of_reviews)) }}"
- },
- {
- "id": "9bdc187f-3d8d-4030-9143-479eff441b7e",
- "name": "population_analysis.z_scores.indigenous",
- "type": "number",
- "value": "={{ ($json.output.indigenous_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.indigenous_total_number_of_reviews)) }}"
- },
- {
- "id": "0cf11453-dbae-4250-a01a-c98e35aab224",
- "name": "population_analysis.z_scores.male",
- "type": "number",
- "value": "={{ ($json.output.male_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.male_total_number_of_reviews)) }}"
- },
- {
- "id": "35a18fbc-7c2c-40fe-829d-2fffbdb13bb8",
- "name": "population_analysis.z_scores.female",
- "type": "number",
- "value": "={{ ($json.output.female_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.female_total_number_of_reviews)) }}"
- },
- {
- "id": "a6e17c1b-a89b-4c05-8184-10f7248c159f",
- "name": "population_analysis.z_scores.trans",
- "type": "number",
- "value": "={{ ($json.output.trans_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.trans_total_number_of_reviews)) }}"
- },
- {
- "id": "5e7dbccf-3011-4dba-863c-5390c1ee9e50",
- "name": "population_analysis.z_scores.hetero",
- "type": "number",
- "value": "={{ ($json.output.hetero_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.hetero_total_number_of_reviews)) }}"
- },
- {
- "id": "1872152f-2c7e-4c24-bcd5-e2777616bfe2",
- "name": "population_analysis.z_scores.lgbtqia",
- "type": "number",
- "value": "={{ ($json.output.lgbtqia_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.lgbtqia_total_number_of_reviews)) }}"
- },
- {
- "id": "91b2cb00-173e-421a-929a-51d2a6654767",
- "name": "population_analysis.z_scores.nondisabled",
- "type": "number",
- "value": "={{ ($json.output.nondisabled_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nondisabled_total_number_of_reviews)) }}"
- },
- {
- "id": "8bb7429e-0500-482c-8e8d-d2c63733ffe1",
- "name": "population_analysis.z_scores.disabled",
- "type": "number",
- "value": "={{ ($json.output.disabled_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.disabled_total_number_of_reviews)) }}"
- },
- {
- "id": "89f00d0f-80db-4ad9-bf60-9385aa3d915b",
- "name": "population_analysis.z_scores.caregiver",
- "type": "number",
- "value": "={{ ($json.output.caregiver_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.caregiver_total_number_of_reviews)) }}"
- },
- {
- "id": "0bb2b96c-d882-4ac1-9432-9fce06b26cf5",
- "name": "population_analysis.z_scores.parent",
- "type": "number",
- "value": "={{ ($json.output.parent_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.parent_total_number_of_reviews)) }}"
- },
- {
- "id": "9aae7169-1a25-4fab-b940-7f2cd7ef39d9",
- "name": "population_analysis.z_scores.nonparent",
- "type": "number",
- "value": "={{ ($json.output.nonparent_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nonparent_total_number_of_reviews)) }}"
- },
- {
- "id": "aac189a0-d6fc-4581-a15d-3e75a0cb370a",
- "name": "population_analysis.z_scores.nonveteran",
- "type": "number",
- "value": "={{ ($json.output.nonveteran_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nonveteran_total_number_of_reviews)) }}"
- },
- {
- "id": "d40f014a-9c1d-4aea-88ac-d8a3de143931",
- "name": "population_analysis.z_scores.veteran",
- "type": "number",
- "value": "={{ ($json.output.veteran_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.veteran_total_number_of_reviews)) }}"
- },
- {
- "id": "67e0394f-6d55-4e80-8a7d-814635620b1d",
- "name": "population_analysis.effect_sizes.hispanic",
- "type": "number",
- "value": "={{ ($json.output.hispanic_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "65cd3a22-2c97-4da1-8fcc-cc1af39118f2",
- "name": "population_analysis.effect_sizes.white",
- "type": "number",
- "value": "={{ ($json.output.white_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "a03bdf0f-e294-4a01-bb08-ddc16e9997a5",
- "name": "population_analysis.effect_sizes.pacific_islander",
- "type": "number",
- "value": "={{ ($json.output.pacific_islander_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "b0bdc40e-ed5f-475b-9d8b-8cf5beff7002",
- "name": "population_analysis.effect_sizes.black",
- "type": "number",
- "value": "={{ ($json.output.black_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "45cac3f0-7270-4fa4-8fc4-94914245a77d",
- "name": "population_analysis.effect_sizes.middle_eastern",
- "type": "number",
- "value": "={{ ($json.output.middle_eastern_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "cf5b7650-8766-45f6-8241-49aea62bf619",
- "name": "population_analysis.effect_sizes.indigenous",
- "type": "number",
- "value": "={{ ($json.output.indigenous_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "7c6a8d38-02b7-47a1-af44-5eebfb4140ec",
- "name": "population_analysis.effect_sizes.male",
- "type": "number",
- "value": "={{ ($json.output.male_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "4bf3dba9-4d07-4315-83ce-5fba288a00c9",
- "name": "population_analysis.effect_sizes.female",
- "type": "number",
- "value": "={{ ($json.output.female_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "d5e980b8-d7a8-4d4c-bcd9-fd9cbd20c729",
- "name": "population_analysis.effect_sizes.trans",
- "type": "number",
- "value": "={{ ($json.output.trans_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "2c8271c1-b612-4292-9d48-92c342b83727",
- "name": "population_analysis.effect_sizes.hetero",
- "type": "number",
- "value": "={{ ($json.output.hetero_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "996f2ea0-2e46-424b-9797-2d58fd56b1d3",
- "name": "population_analysis.effect_sizes.lgbtqia",
- "type": "number",
- "value": "={{ ($json.output.lgbtqia_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "8c987b6e-764d-422e-82de-00bd89269b22",
- "name": "population_analysis.effect_sizes.nondisabled",
- "type": "number",
- "value": "={{ ($json.output.nondisabled_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "ab796bb7-06ff-4282-b4b3-eefd129c743e",
- "name": "population_analysis.effect_sizes.disabled",
- "type": "number",
- "value": "={{ ($json.output.disabled_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "a17bf413-a098-4f24-8162-821a6a0ddb5e",
- "name": "population_analysis.effect_sizes.caregiver",
- "type": "number",
- "value": "={{ ($json.output.caregiver_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "99911e1e-06e8-4bbd-915d-b92b8b37b374",
- "name": "population_analysis.effect_sizes.parent",
- "type": "number",
- "value": "={{ ($json.output.parent_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "4ddf729b-361e-4d81-a67c-b6c18509e60b",
- "name": "population_analysis.effect_sizes.nonparent",
- "type": "number",
- "value": "={{ ($json.output.nonparent_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "725b8abb-7f72-45fc-a0c0-0e0a4f2cb131",
- "name": "population_analysis.effect_sizes.nonveteran",
- "type": "number",
- "value": "={{ ($json.output.nonveteran_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "20e54fa5-2faa-4134-90e5-81224ec9659e",
- "name": "population_analysis.effect_sizes.veteran",
- "type": "number",
- "value": "={{ ($json.output.veteran_average_rating - $json.average_rating) / $json.std_dev }}"
- },
- {
- "id": "2cc6465a-3a1c-4eb5-9e5a-72d41049d81e",
- "name": "population_analysis.review_count.asian",
- "type": "number",
- "value": "={{ $json.output.asian_total_number_of_reviews }}"
- },
- {
- "id": "0a5f6aae-ba21-47b5-8af8-fec2256e4df6",
- "name": "population_analysis.review_count.hispanic",
- "type": "number",
- "value": "={{ $json.output.hispanic_total_number_of_reviews }}"
- },
- {
- "id": "ae124587-7e24-4c1a-a002-ed801f859c30",
- "name": "population_analysis.review_count.pacific_islander",
- "type": "number",
- "value": "={{ $json.output.pacific_islander_total_number_of_reviews }}"
- },
- {
- "id": "fc790196-ca8e-4069-a093-87a413ebbf3e",
- "name": "population_analysis.review_count.black",
- "type": "number",
- "value": "={{ $json.output.black_total_number_of_reviews }}"
- },
- {
- "id": "7fd72701-781e-4e33-b000-174a853b172b",
- "name": "population_analysis.review_count.middle_eastern",
- "type": "number",
- "value": "={{ $json.output.middle_eastern_total_number_of_reviews }}"
- },
- {
- "id": "3751e7da-11a7-4af3-8aa6-1c6d53bcf27d",
- "name": "population_analysis.review_count.indigenous",
- "type": "number",
- "value": "={{ $json.output.indigenous_total_number_of_reviews }}"
- },
- {
- "id": "9ee0cac9-d2dd-4ba0-90ee-b2cdd22d9b77",
- "name": "population_analysis.review_count.male",
- "type": "number",
- "value": "={{ $json.output.male_total_number_of_reviews }}"
- },
- {
- "id": "ae7fcdc7-d373-4c24-9a65-94bd2b5847a8",
- "name": "population_analysis.review_count.female",
- "type": "number",
- "value": "={{ $json.output.female_total_number_of_reviews }}"
- },
- {
- "id": "3f53d065-269f-425a-b27d-dc5a3dbb6141",
- "name": "population_analysis.review_count.trans",
- "type": "number",
- "value": "={{ $json.output.trans_total_number_of_reviews }}"
- },
- {
- "id": "d15e976e-7599-4df0-9e65-8047b7a4cda8",
- "name": "population_analysis.review_count.hetero",
- "type": "number",
- "value": "={{ $json.output.hetero_total_number_of_reviews }}"
- },
- {
- "id": "c8b786d3-a980-469f-bf0e-de70ad44f0ea",
- "name": "population_analysis.review_count.lgbtqia",
- "type": "number",
- "value": "={{ $json.output.lgbtqia_total_number_of_reviews }}"
- },
- {
- "id": "e9429215-0858-4482-964a-75de7978ecbb",
- "name": "population_analysis.review_count.nondisabled",
- "type": "number",
- "value": "={{ $json.output.nondisabled_total_number_of_reviews }}"
- },
- {
- "id": "2c6e53c4-eab1-42aa-b956-ee882832f569",
- "name": "population_analysis.review_count.disabled",
- "type": "number",
- "value": "={{ $json.output.disabled_total_number_of_reviews }}"
- },
- {
- "id": "b5edfa25-ab11-4b94-9670-4d5589a62498",
- "name": "population_analysis.review_count.caregiver",
- "type": "number",
- "value": "={{ $json.output.caregiver_total_number_of_reviews }}"
- },
- {
- "id": "41084e96-c42f-4bb0-ac1a-883b46537fca",
- "name": "population_analysis.review_count.parent",
- "type": "number",
- "value": "={{ $json.output.parent_total_number_of_reviews }}"
- },
- {
- "id": "96496a38-9311-4ade-bd2f-2943d1d92314",
- "name": "population_analysis.review_count.nonparent",
- "type": "number",
- "value": "={{ $json.output.nonparent_total_number_of_reviews }}"
- },
- {
- "id": "5071771d-5f41-43cb-a8ce-e4e40ed3519b",
- "name": "population_analysis.review_count.nonveteran",
- "type": "number",
- "value": "={{ $json.output.nonveteran_total_number_of_reviews }}"
- },
- {
- "id": "2358e782-70da-4964-b625-5fe1946b5250",
- "name": "population_analysis.review_count.veteran",
- "type": "number",
- "value": "={{ $json.output.veteran_total_number_of_reviews }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "85536931-839a-476b-b0dd-fa6d01c6d5c1",
- "name": "Format dataset for scatterplot",
- "type": "n8n-nodes-base.code",
- "position": [
- 3340,
- 760
- ],
- "parameters": {
- "jsCode": "// Iterate through the input data and format the dataset for QuickChart\nfor (const item of $input.all()) {\n // Ensure the data object exists and initialize datasets\n item.json.data = {\n datasets: []\n };\n\n const z_scores = item.json.population_analysis.z_scores;\n const effect_sizes = item.json.population_analysis.effect_sizes;\n const review_count = item.json.population_analysis.review_count;\n\n // Ensure z_scores, effect_sizes, and review_count are defined and are objects\n if (z_scores && effect_sizes && review_count && typeof z_scores === 'object' && typeof effect_sizes === 'object' && typeof review_count === 'object') {\n // Initialize the dataset object\n const dataset = {\n label: 'Demographics Data',\n data: []\n };\n\n // Iterate through the demographic keys\n for (const key in z_scores) {\n // Check if review count for the demographic is greater than 0\n if (z_scores.hasOwnProperty(key) && effect_sizes.hasOwnProperty(key) && review_count[key] > 0) {\n\n // Add each demographic point to the dataset\n dataset.data.push({\n x: z_scores[key], // x = z_score\n y: effect_sizes[key], // y = effect_size\n label: $('Define dictionary of demographic keys').first().json[key],\n });\n }\n }\n\n // Only add the dataset if it contains data\n if (dataset.data.length > 0) {\n item.json.data.datasets.push(dataset);\n }\n\n delete item.json.population_analysis\n }\n}\n\n// Return the updated input with the data object containing datasets and labels\nreturn $input.all();\n"
- },
- "typeVersion": 2
- },
- {
- "id": "957b9f6c-7cf8-4ec6-aec7-a7d59ed3a4ad",
- "name": "Specify additional parameters for scatterplot",
- "type": "n8n-nodes-base.set",
- "position": [
- 3540,
- 760
- ],
- "parameters": {
- "options": {
- "ignoreConversionErrors": false
- },
- "assignments": {
- "assignments": [
- {
- "id": "5cd507f6-6835-4d2e-8329-1b5d24a3fc15",
- "name": "type",
- "type": "string",
- "value": "scatter"
- },
- {
- "id": "80b6f981-e3c7-4c6e-a0a1-f30d028fe15e",
- "name": "options",
- "type": "object",
- "value": "={\n \"title\": {\n \"display\": true,\n \"position\": \"top\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\",\n \"fontColor\": \"#666666\",\n \"fontStyle\": \"bold\",\n \"padding\": 10,\n \"lineHeight\": 1.2,\n \"text\": \"{{ $('SET company_name').item.json.company_name }} Workplace Population Bias\"\n },\n \"legend\": {\n \"display\": false\n },\n \"scales\": {\n \"xAxes\": [\n {\n \"scaleLabel\": {\n \"display\": true,\n \"labelString\": \"Z-Score\",\n \"fontColor\": \"#666666\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\"\n }\n }\n ],\n \"yAxes\": [\n {\n \"scaleLabel\": {\n \"display\": true,\n \"labelString\": \"Effect Score\",\n \"fontColor\": \"#666666\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\"\n }\n }\n ]\n },\n \"plugins\": {\n \"datalabels\": {\n \"display\": true,\n \"align\": \"top\",\n \"anchor\": \"center\",\n \"backgroundColor\": \"#eee\",\n \"borderColor\": \"#ddd\",\n \"borderRadius\": 6,\n \"borderWidth\": 1,\n \"padding\": 4,\n \"color\": \"#000\",\n \"font\": {\n \"family\": \"sans-serif\",\n \"size\": 10,\n \"style\": \"normal\"\n }\n }\n }\n }"
- }
- ]
- },
- "includeOtherFields": true
- },
- "typeVersion": 3.4
- },
- {
- "id": "a937132c-43fc-4fa0-ae35-885da89e51d1",
- "name": "Quickchart Scatterplot",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3740,
- 760
- ],
- "parameters": {
- "url": "https://quickchart.io/chart",
- "options": {},
- "sendQuery": true,
- "queryParameters": {
- "parameters": [
- {
- "name": "c",
- "value": "={{ $json.toJsonString() }}"
- },
- {
- "name": "Content-Type",
- "value": "application/json"
- },
- {
- "name": "encoding",
- "value": "url"
- }
- ]
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "ede1931e-bac8-4279-b3a7-5980a190e324",
- "name": "QuickChart Bar Chart",
- "type": "n8n-nodes-base.quickChart",
- "position": [
- 3740,
- 560
- ],
- "parameters": {
- "data": "={{ $json.population_analysis.effect_sizes.values() }}",
- "output": "bar_chart",
- "labelsMode": "array",
- "labelsArray": "={{ $json.population_analysis.effect_sizes.keys() }}",
- "chartOptions": {
- "format": "png"
- },
- "datasetOptions": {
- "label": "={{ $('SET company_name').item.json.company_name }} Effect Size on Employee Experience"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "6122fec0-619c-48d3-ad2c-05ed55ba2275",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 480,
- 40
- ],
- "parameters": {
- "color": 7,
- "width": 3741.593083126444,
- "height": 1044.8111554136713,
- "content": "# Spot Workplace Discrimination Patterns using ScrapingBee, Glassdoor, OpenAI, and QuickChart\n"
- },
- "typeVersion": 1
- },
- {
- "id": "5cda63e8-f31b-46f6-8cb2-41d1856ac537",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 900,
- 180
- ],
- "parameters": {
- "color": 4,
- "width": 1237.3377621763516,
- "height": 575.9439659309116,
- "content": "## Use ScrapingBee to gather raw data from Glassdoor"
- },
- "typeVersion": 1
- },
- {
- "id": "28d247b2-9020-4280-83d2-d6583622c0b7",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 920,
- 240
- ],
- "parameters": {
- "color": 7,
- "width": 804.3951263154196,
- "height": 125.73173301324687,
- "content": "### Due to javascript restrictions, a normal HTTP request cannot be used to gather user-reported details from Glassdoor. \n\nInstead, [ScrapingBee](https://www.scrapingbee.com/) offers a great tool with a very generous package of free tokens per month, which works out to roughly 4-5 runs of this workflow."
- },
- "typeVersion": 1
- },
- {
- "id": "d65a239c-06d2-470b-b24a-23ec00a9f148",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2180,
- 99.69933502879758
- ],
- "parameters": {
- "color": 5,
- "width": 311.0523273992095,
- "height": 843.8786512173932,
- "content": "## Extract details with AI"
- },
- "typeVersion": 1
- },
- {
- "id": "3cffd188-62a1-43a7-a67f-548e21d2b187",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2516.1138215303854,
- 100
- ],
- "parameters": {
- "color": 7,
- "width": 423.41585047129973,
- "height": 309.71740416262054,
- "content": "### Calculate variance and standard deviation from review rating distributions."
- },
- "typeVersion": 1
- },
- {
- "id": "b5015c07-03e3-47d4-9469-e831b2c755c0",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3320,
- 706.46982689582
- ],
- "parameters": {
- "color": 5,
- "width": 639.5579220386832,
- "height": 242.80759628871897,
- "content": "## Formatting datasets for Scatterplot"
- },
- "typeVersion": 1
- },
- {
- "id": "e52bb9d9-617a-46f5-b217-a6f670b6714c",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 500,
- 120
- ],
- "parameters": {
- "width": 356.84794255678776,
- "height": 186.36110628732342,
- "content": "## How this workflow works\n1. Replace ScrapingBee and OpenAI credentials\n2. Replace company_name with company of choice (workflow performs better with larger US-based organizations)\n3. Preview QuickChart data visualizations and AI data analysis"
- },
- "typeVersion": 1
- },
- {
- "id": "d83c07a3-04ed-418f-94f1-e70828cba8b2",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 500,
- 880
- ],
- "parameters": {
- "color": 6,
- "width": 356.84794255678776,
- "height": 181.54335665904924,
- "content": "### Inspired by [Wes Medford's Medium Post](https://medium.com/@wryanmedford/an-open-letter-to-twilios-leadership-f06f661ecfb4)\n\nWes performed the initial data analysis highlighting problematic behaviors at Twilio. I wanted to try and democratize the data analysis they performed for those less technical.\n\n**Hi, Wes!**"
- },
- "typeVersion": 1
- },
- {
- "id": "ed0c1b4a-99fe-4a27-90bb-ac38dd20810b",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 4020,
- 880
- ],
- "parameters": {
- "color": 7,
- "width": 847.5931795867759,
- "height": 522.346478008115,
- "content": "%20%7B%0A%20%20%20%20%20%20%20%20%20%20var%20idx%20%3D%20context.dataIndex%3B%0A%20%20%20%20%20%20%20%20%20%20return%20context.chart.data.labels%5Bidx%5D%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A#full-width)"
- },
- "typeVersion": 1
- },
- {
- "id": "7b92edf8-3a58-4931-abf4-d9c2f57cfa32",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3980,
- 800
- ],
- "parameters": {
- "color": 6,
- "width": 989.7621518164046,
- "height": 636.6345107975716,
- "content": "## Example Scatterplot output"
- },
- "typeVersion": 1
- },
- {
- "id": "bd6859b4-096c-401e-9bce-91e970e1afd1",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2540,
- 800
- ],
- "parameters": {
- "color": 6,
- "width": 737.6316136259719,
- "height": 444.9087184962878,
- "content": "## Glossary\n**Z-Score** โ A statistical measure that indicates how many standard deviations a data point is from the mean. In this analysis, a negative z-score suggests a group rates their workplace experience lower than the average, while a positive z-score suggests a better-than-average experience.\n\n**Effect Size** โ A measure of the magnitude of difference between groups. Larger negative effect sizes indicate a more substantial disparity in workplace experiences for certain groups, making it useful for identifying meaningful gaps beyond just statistical significance.\n\n**P-Score (P-Value)** โ The probability that the observed differences occurred by chance. A lower p-score (typically below 0.05) suggests the difference is statistically significant and unlikely to be random. In this analysis, high p-scores confirm that the disparities in ratings for marginalized groups are unlikely to be due to chance alone.\n\n### Relevance to This Analysis\nThese metrics help quantify workplace disparities among demographic groups. Z-scores show which groups report better or worse experiences, effect sizes reveal the severity of these differences, and p-scores confirm whether the disparities are statistically meaningful. This data allows for a more informed discussion about workplace equity and areas needing improvement."
- },
- "typeVersion": 1
- },
- {
- "id": "5af3ef87-ed4b-481e-b1ba-d44ffb7551d8",
- "name": "Sticky Note12",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 4140,
- 80
- ],
- "parameters": {
- "color": 6,
- "width": 643.5995639515581,
- "height": 646.0030521944287,
- "content": "## Example AI Analysis (Twilio Example)\n\n### Key Takeaways\n1. **Significant Disparity Among Disabled Employees**\nDisabled employees reported the lowest average ratings, with a z-score of -5.50, indicating a far worse experience compared to their non-disabled peers. \n2. **LGBTQIA Community's Challenges**\nMembers of the LGBTQIA community showed significantly lower ratings (z-score of -2.53), suggesting they may experience a workplace environment that is less inclusive or supportive compared to others.\n3. **Transgender Experiences Are Particularly Negative**\nTransgender employees rated their experiences considerably lower (z-score of -2.91), highlighting a critical area for improvement in workplace culture and acceptance.\n4. **Veterans Report Higher Satisfaction**\nIn contrast, veterans had the highest ratings (z-score of 1.54), which could indicate a supportive environment or programs tailored to their needs.\n5. **Overall Gender Discrepancies**\nA noticeable gap exists in average ratings by gender, with female employees scoring below male employees, suggesting potential gender biases or challenges in workplace dynamics.\n\n### Employee Experiences\n#### Perceptions of Workplace Environment\nFor members of groups reporting significantly worse experiences, such as disabled, transgender, and LGBTQIA employees, the workplace may feel alienating or unwelcoming. These individuals might perceive that their contributions are undervalued or overlooked and that necessary support systems are lacking, creating a culture of exclusion rather than one of inclusivity. This feeling of being marginalized can lead to poorer engagement, higher turnover rates, and diminished overall job satisfaction, adversely impacting both employees and the organization."
- },
- "typeVersion": 1
- },
- {
- "id": "a39cdbe7-d6ae-4a84-98c7-52ebf98242f3",
- "name": "Text Analysis of Bias Data",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 3720,
- 280
- ],
- "parameters": {
- "text": "=This data compares the average rating given by different demographic groups against a baseline (the overall mean rating).\n\nObjective:\n1. Analyze the data and offer between 2 and 5 key takeaways with a title and short (one-sentence) summary.\n2. Below the key takeaways, Include a heading called \"Employee Experiences\". Under this heading, include a subheader and paragraph describing the possible perception of the workplace for members of any groups reporting significantly worse (or better) experiences than others.\n3. Ensure there are between 2-5 key takeaways and employee experiences\n\nData for analysis:\n{{ $json.population_analysis.toJsonString() }}",
- "promptType": "define"
- },
- "typeVersion": 1.4
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "ff1df786-ebaf-4ed0-aeca-1872b93ef275",
- "connections": {
- "Merge": {
- "main": [
- [
- {
- "node": "Calculate Z-Scores and Effect Sizes",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "SET company_name": {
- "main": [
- [
- {
- "node": "Define dictionary of demographic keys",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Text Analysis of Bias Data",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Sort Effect Sizes": {
- "main": [
- [
- {
- "node": "QuickChart Bar Chart",
- "type": "main",
- "index": 0
- },
- {
- "node": "Text Analysis of Bias Data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Calculate P-Scores": {
- "main": [
- [
- {
- "node": "Sort Effect Sizes",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model1": {
- "ai_languageModel": [
- [
- {
- "node": "Extract demographic distributions",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model2": {
- "ai_languageModel": [
- [
- {
- "node": "Extract overall ratings and distribution percentages",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Extract company url path": {
- "main": [
- [
- {
- "node": "ScrapingBee GET company page contents",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set variance and std_dev": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Demographics Module": {
- "main": [
- [
- {
- "node": "Extract demographic distributions",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "ScrapingBee Search Glassdoor": {
- "main": [
- [
- {
- "node": "Extract company url path",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract reviews page url path": {
- "main": [
- [
- {
- "node": "ScrapingBee GET Glassdoor Reviews Content",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Overall Review Summary": {
- "main": [
- [
- {
- "node": "Extract overall ratings and distribution percentages",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Format dataset for scatterplot": {
- "main": [
- [
- {
- "node": "Specify additional parameters for scatterplot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Define contributions to variance": {
- "main": [
- [
- {
- "node": "Set variance and std_dev",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract demographic distributions": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "SET company_name",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Calculate Z-Scores and Effect Sizes": {
- "main": [
- [
- {
- "node": "Calculate P-Scores",
- "type": "main",
- "index": 0
- },
- {
- "node": "Format dataset for scatterplot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Define dictionary of demographic keys": {
- "main": [
- [
- {
- "node": "ScrapingBee Search Glassdoor",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "ScrapingBee GET company page contents": {
- "main": [
- [
- {
- "node": "Extract reviews page url path",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "ScrapingBee GET Glassdoor Reviews Content": {
- "main": [
- [
- {
- "node": "Extract Demographics Module",
- "type": "main",
- "index": 0
- },
- {
- "node": "Extract Overall Review Summary",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Specify additional parameters for scatterplot": {
- "main": [
- [
- {
- "node": "Quickchart Scatterplot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract overall ratings and distribution percentages": {
- "main": [
- [
- {
- "node": "Define contributions to variance",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.txt b/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.txt
new file mode 100644
index 0000000..c8e5169
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Spot Workplace Discrimination Patterns with AI.txt
@@ -0,0 +1,1850 @@
+{
+"id": "vzU9QRZsHcyRsord",
+"meta": {
+"instanceId": "a9f3b18652ddc96459b459de4fa8fa33252fb820a9e5a1593074f3580352864a",
+"templateCredsSetupCompleted": true
+},
+"name": "Spot Workplace Discrimination Patterns with AI",
+"tags": [
+{
+"id": "76EYz9X3GU4PtgSS",
+"name": "human_resources",
+"createdAt": "2025-01-30T18:52:17.614Z",
+"updatedAt": "2025-01-30T18:52:17.614Z"
+},
+{
+"id": "ey2Mx4vNaV8cKvao",
+"name": "openai",
+"createdAt": "2024-12-23T07:10:13.400Z",
+"updatedAt": "2024-12-23T07:10:13.400Z"
+}
+],
+"nodes": [
+{
+"id": "b508ab50-158a-4cbf-a52e-f53e1804e770",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+280,
+380
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "11a1a2d5-a274-44f7-97ca-5666a59fcb31",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+2220,
+800
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "XXXXXX",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "395f7b67-c914-4aae-8727-0573fdbfc6ad",
+"name": "OpenAI Chat Model2",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+2220,
+380
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "XXXXXX",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "6ab194a9-b869-4296-aea9-19afcbffc0d7",
+"name": "Merge",
+"type": "n8n-nodes-base.merge",
+"position": [
+2940,
+600
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combineBy": "combineByPosition"
+},
+"typeVersion": 3
+},
+{
+"id": "1eba1dd7-a164-4c70-8c75-759532bd16a0",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+3840,
+420
+],
+"parameters": {
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "XXXXXX",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "f25f1b07-cded-4ca7-9655-8b8f463089ab",
+"name": "SET company_name",
+"type": "n8n-nodes-base.set",
+"position": [
+540,
+380
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "dd256ef7-013c-4769-8580-02c2d902d0b2",
+"name": "company_name",
+"type": "string",
+"value": "=Twilio"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "87264a93-ab97-4e39-8d40-43365189f704",
+"name": "Define dictionary of demographic keys",
+"type": "n8n-nodes-base.set",
+"position": [
+740,
+380
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "6ae671be-45d0-4a94-a443-2f1d4772d31b",
+"name": "asian",
+"type": "string",
+"value": "Asian"
+},
+{
+"id": "6c93370c-996c-44a6-a34c-4cd3baeeb846",
+"name": "hispanic",
+"type": "string",
+"value": "Hispanic or Latinx"
+},
+{
+"id": "dee79039-6051-4e9d-98b5-63a07d30f6b0",
+"name": "white",
+"type": "string",
+"value": "White"
+},
+{
+"id": "08d42380-8397-412f-8459-7553e9309b5d",
+"name": "pacific_islander",
+"type": "string",
+"value": "Native Hawaiian or other Pacific Islander"
+},
+{
+"id": "09e8ebc5-e7e7-449a-9036-9b9b54cdc828",
+"name": "black",
+"type": "string",
+"value": "Black or African American"
+},
+{
+"id": "39e910f8-3a8b-4233-a93a-3c5693e808c6",
+"name": "middle_eastern",
+"type": "string",
+"value": "Middle Eastern"
+},
+{
+"id": "169b3471-efa0-476e-aa83-e3f717c568f1",
+"name": "indigenous",
+"type": "string",
+"value": "Indigenous American or Native Alaskan"
+},
+{
+"id": "b6192296-4efa-4af5-ae02-1e31d28aae90",
+"name": "male",
+"type": "string",
+"value": "Men"
+},
+{
+"id": "4b322294-940c-459d-b083-8e91e38193f7",
+"name": "female",
+"type": "string",
+"value": "Women"
+},
+{
+"id": "1940eef0-6b76-4a26-9d8f-7c8536fbcb1b",
+"name": "trans",
+"type": "string",
+"value": "Transgender and/or Non-Binary"
+},
+{
+"id": "3dba3e18-2bb1-4078-bde9-9d187f9628dd",
+"name": "hetero",
+"type": "string",
+"value": "Heterosexual"
+},
+{
+"id": "9b7d10ad-1766-4b18-a230-3bd80142b48c",
+"name": "lgbtqia",
+"type": "string",
+"value": "LGBTQ+"
+},
+{
+"id": "458636f8-99e8-4245-9950-94e4cf68e371",
+"name": "nondisabled",
+"type": "string",
+"value": "Non-Disabled"
+},
+{
+"id": "a466e258-7de1-4453-a126-55f780094236",
+"name": "disabled",
+"type": "string",
+"value": "People with Disabilities"
+},
+{
+"id": "98735266-0451-432f-be7c-efcb09512cb1",
+"name": "caregiver",
+"type": "string",
+"value": "Caregivers"
+},
+{
+"id": "ebe2353c-9ff5-47bc-8c11-b66d3436f5b4",
+"name": "parent",
+"type": "string",
+"value": "Parents/Guardians"
+},
+{
+"id": "ab51c80c-d81d-41ab-94d9-c0a263743c17",
+"name": "nonparent",
+"type": "string",
+"value": "Not a Parent or Caregiver"
+},
+{
+"id": "cb7df429-c600-43f4-aa7e-dbc2382a85a0",
+"name": "nonveteran",
+"type": "string",
+"value": "Non-Veterans"
+},
+{
+"id": "dffbdb13-189a-462d-83d1-c5ec39a17d41",
+"name": "veteran",
+"type": "string",
+"value": "Veterans"
+}
+]
+},
+"includeOtherFields": true
+},
+"typeVersion": 3.4
+},
+{
+"id": "862f1c77-44a8-4d79-abac-33351ebb731b",
+"name": "ScrapingBee Search Glassdoor",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+940,
+380
+],
+"parameters": {
+"url": "https://app.scrapingbee.com/api/v1",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "url",
+"value": "=https://www.glassdoor.com/Search/results.htm?keyword={{ $json.company_name.toLowerCase().urlEncode() }}"
+},
+{
+"name": "premium_proxy",
+"value": "true"
+},
+{
+"name": "block_resources",
+"value": "false"
+},
+{
+"name": "stealth_proxy",
+"value": "true"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "XXXXXX",
+"name": "ScrapingBee Query Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "4c9bf05e-9c50-4895-b20b-b7c329104615",
+"name": "Extract company url path",
+"type": "n8n-nodes-base.html",
+"position": [
+1140,
+380
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "url_path",
+"attribute": "href",
+"cssSelector": "body main div a",
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "d20bb0e7-4ca7-41d0-a3e9-41abc811b064",
+"name": "ScrapingBee GET company page contents",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1340,
+380
+],
+"parameters": {
+"url": "https://app.scrapingbee.com/api/v1",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "url",
+"value": "=https://www.glassdoor.com{{ $json.url_path }}"
+},
+{
+"name": "premium_proxy",
+"value": "true"
+},
+{
+"name": "block_resources",
+"value": "false"
+},
+{
+"name": "stealth_proxy",
+"value": "true"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "XXXXXX",
+"name": "ScrapingBee Query Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "fce70cab-8ce3-4ce2-b040-ce80d66b1e62",
+"name": "Extract reviews page url path",
+"type": "n8n-nodes-base.html",
+"position": [
+1540,
+380
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "url_path",
+"attribute": "href",
+"cssSelector": "#reviews a",
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "d2e7fee9-e3d4-42bf-8be6-38b352371273",
+"name": "ScrapingBee GET Glassdoor Reviews Content",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1760,
+380
+],
+"parameters": {
+"url": "https://app.scrapingbee.com/api/v1",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "url",
+"value": "=https://www.glassdoor.com{{ $json.url_path }}"
+},
+{
+"name": "premium_proxy",
+"value": "True"
+},
+{
+"name": "block_resources",
+"value": "False"
+},
+{
+"name": "stealth_proxy",
+"value": "true"
+}
+]
+}
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "XXXXXX",
+"name": "ScrapingBee Query Auth"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "0c322823-0569-4bd5-9c4e-af3de0f8d7b4",
+"name": "Extract Overall Review Summary",
+"type": "n8n-nodes-base.html",
+"position": [
+1980,
+260
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "review_summary",
+"cssSelector": "div[data-test=\"review-summary\"]",
+"returnValue": "html"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "851305ba-0837-4be9-943d-7282e8d74aee",
+"name": "Extract Demographics Module",
+"type": "n8n-nodes-base.html",
+"position": [
+1980,
+520
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "demographics_content",
+"cssSelector": "div[data-test=\"demographics-module\"]",
+"returnValue": "html"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "cf9a6ee2-53b5-4fbf-a36c-4b9dab53b795",
+"name": "Extract overall ratings and distribution percentages",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+2200,
+200
+],
+"parameters": {
+"text": "={{ $json.review_summary }}",
+"options": {},
+"attributes": {
+"attributes": [
+{
+"name": "average_rating",
+"type": "number",
+"required": true,
+"description": "The overall average rating for this company."
+},
+{
+"name": "total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "The total number of reviews for this company."
+},
+{
+"name": "5_star_distribution_percentage",
+"type": "number",
+"required": true,
+"description": "The percentage distribution of 5 star reviews"
+},
+{
+"name": "4_star_distribution_percentage",
+"type": "number",
+"required": true,
+"description": "The percentage distribution of 4 star reviews"
+},
+{
+"name": "3_star_distribution_percentage",
+"type": "number",
+"required": true,
+"description": "The percentage distribution of 3 star reviews"
+},
+{
+"name": "2_star_distribution_percentage",
+"type": "number",
+"required": true,
+"description": "The percentage distribution of 2 star reviews"
+},
+{
+"name": "1_star_distribution_percentage",
+"type": "number",
+"required": true,
+"description": "The percentage distribution of 1 star reviews"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "ae164f6e-04e7-4d8b-951e-a17085956f4b",
+"name": "Extract demographic distributions",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+2200,
+620
+],
+"parameters": {
+"text": "={{ $json.demographics_content }}",
+"options": {
+"systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, you may use 0 for the attribute's value."
+},
+"attributes": {
+"attributes": [
+{
+"name": "asian_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as asian."
+},
+{
+"name": "asian_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as asian."
+},
+{
+"name": "hispanic_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as hispanic."
+},
+{
+"name": "hispanic_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as hispanic."
+},
+{
+"name": "white_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as white."
+},
+{
+"name": "white_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as white."
+},
+{
+"name": "pacific_islander_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as native hawaiian or pacific islander."
+},
+{
+"name": "pacific_islander_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as native hawaiian or pacific islander."
+},
+{
+"name": "black_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as black."
+},
+{
+"name": "black_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as black."
+},
+{
+"name": "middle_eastern_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as middle eastern."
+},
+{
+"name": "middle_eastern_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as middle_eastern."
+},
+{
+"name": "indigenous_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as indigenous american or native alaskan."
+},
+{
+"name": "indigenous_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as indigenous american or native alaskan."
+},
+{
+"name": "male_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as men."
+},
+{
+"name": "male_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as men."
+},
+{
+"name": "female_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as women."
+},
+{
+"name": "female_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as women."
+},
+{
+"name": "trans_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as transgender and/or non-binary."
+},
+{
+"name": "trans_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as trans and/or non-binary."
+},
+{
+"name": "hetero_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as heterosexual."
+},
+{
+"name": "hetero_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as heterosexual."
+},
+{
+"name": "lgbtqia_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as lgbtqia+."
+},
+{
+"name": "lgbtqia_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as lgbtqia+."
+},
+{
+"name": "nondisabled_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as non-disabled."
+},
+{
+"name": "nondisabled_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as non-disabled."
+},
+{
+"name": "disabled_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as people with disabilities."
+},
+{
+"name": "disabled_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as people with disabilities."
+},
+{
+"name": "caregiver_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as caregivers."
+},
+{
+"name": "caregiver_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as caregivers."
+},
+{
+"name": "parent_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as parents/guardians."
+},
+{
+"name": "parent_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as parents/guardians."
+},
+{
+"name": "nonparent_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as not a parent or caregiver."
+},
+{
+"name": "nonparent_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as not a parent or guardian."
+},
+{
+"name": "nonveteran_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as non-veterans."
+},
+{
+"name": "nonveteran_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as non-veterans."
+},
+{
+"name": "veteran_average_rating",
+"type": "number",
+"required": true,
+"description": "=The average rating for this company by employees who self identified as veterans."
+},
+{
+"name": "veteran_total_number_of_reviews",
+"type": "number",
+"required": true,
+"description": "=The number of reviews for this company by employees who self-identified as veterans."
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "c8d9e45c-7d41-47bd-b9a9-0fa70de5d154",
+"name": "Define contributions to variance",
+"type": "n8n-nodes-base.set",
+"position": [
+2560,
+200
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "7360b2c2-1e21-45de-8d1a-e72b8abcb56b",
+"name": "contribution_to_variance.5_star",
+"type": "number",
+"value": "={{ ($json.output['5_star_distribution_percentage'] / 100) * Math.pow(5 - $json.output.average_rating,2) }}"
+},
+{
+"id": "acdd308a-fa33-4e33-b71b-36b9441bfa06",
+"name": "contribution_to_variance.4_star",
+"type": "number",
+"value": "={{ ($json.output['4_star_distribution_percentage'] / 100) * Math.pow(4 - $json.output.average_rating,2) }}"
+},
+{
+"id": "376818f3-d429-4abe-8ece-e8e9c5585826",
+"name": "contribution_to_variance.3_star",
+"type": "number",
+"value": "={{ ($json.output['3_star_distribution_percentage'] / 100) * Math.pow(3 - $json.output.average_rating,2) }}"
+},
+{
+"id": "620d5c37-8b93-4d39-9963-b7ce3a7f431e",
+"name": "contribution_to_variance.2_star",
+"type": "number",
+"value": "={{ ($json.output['2_star_distribution_percentage'] / 100) * Math.pow(2 - $json.output.average_rating,2) }}"
+},
+{
+"id": "76357980-4f9b-4b14-be68-6498ba25af67",
+"name": "contribution_to_variance.1_star",
+"type": "number",
+"value": "={{ ($json.output['1_star_distribution_percentage'] / 100) * Math.pow(1 - $json.output.average_rating,2) }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "8ea03017-d5d6-46ef-a5f1-dae4372f6256",
+"name": "Set variance and std_dev",
+"type": "n8n-nodes-base.set",
+"position": [
+2740,
+200
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "3217d418-f1b0-45ff-9f9a-6e6145cc29ca",
+"name": "variance",
+"type": "number",
+"value": "={{ $json.contribution_to_variance.values().sum() }}"
+},
+{
+"id": "acdb9fea-15ec-46ed-bde9-073e93597f17",
+"name": "average_rating",
+"type": "number",
+"value": "={{ $('Extract overall ratings and distribution percentages').item.json.output.average_rating }}"
+},
+{
+"id": "1f3a8a29-4bd4-4b40-8694-c74a0285eadb",
+"name": "total_number_of_reviews",
+"type": "number",
+"value": "={{ $('Extract overall ratings and distribution percentages').item.json.output.total_number_of_reviews }}"
+},
+{
+"id": "1906c796-1964-446b-8b56-d856269da938",
+"name": "std_dev",
+"type": "number",
+"value": "={{ Math.sqrt($json.contribution_to_variance.values().sum()) }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "0570d531-8480-4446-8f02-18640b4b891e",
+"name": "Calculate P-Scores",
+"type": "n8n-nodes-base.code",
+"position": [
+3340,
+440
+],
+"parameters": {
+"jsCode": "// Approximate CDF for standard normal distribution\nfunction normSDist(z) {\n const t = 1 / (1 + 0.3275911 * Math.abs(z));\n const d = 0.254829592 * t - 0.284496736 * t * t + 1.421413741 * t * t * t - 1.453152027 * t * t * t * t + 1.061405429 * t * t * t * t * t;\n return 0.5 * (1 + Math.sign(z) * d * Math.exp(-z * z / 2));\n}\n\nfor (const item of $input.all()) {\n if (!item.json.population_analysis.p_scores) {\n item.json.population_analysis.p_scores = {};\n }\n\n for (const score of Object.keys(item.json.population_analysis.z_scores)) {\n // Check if review count exists and is greater than zero\n if (item.json.population_analysis.review_count[score] > 0) {\n // Apply the p_score formula: 2 * NORM.S.DIST(-ABS(z_score))\n const p_score = 2 * normSDist(-Math.abs(item.json.population_analysis.z_scores[score]));\n\n // Store the calculated p_score\n item.json.population_analysis.p_scores[score] = p_score;\n } else {\n // Remove z_scores, effect_sizes, and p_scores for groups with no reviews\n delete item.json.population_analysis.z_scores[score];\n delete item.json.population_analysis.effect_sizes[score];\n delete item.json.population_analysis.p_scores[score];\n }\n }\n}\n\nreturn $input.all();"
+},
+"typeVersion": 2
+},
+{
+"id": "0bdb9732-67ef-440d-bdd2-42c4f64ff6b6",
+"name": "Sort Effect Sizes",
+"type": "n8n-nodes-base.set",
+"position": [
+3540,
+440
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "61cf92ba-bc4e-40b8-a234-9b993fd24019",
+"name": "population_analysis.effect_sizes",
+"type": "object",
+"value": "={{ Object.fromEntries(Object.entries($json.population_analysis.effect_sizes).sort(([,a],[,b]) => a-b )) }}"
+}
+]
+},
+"includeOtherFields": true
+},
+"typeVersion": 3.4
+},
+{
+"id": "fd9026ef-e993-410a-87d6-40a3ad10b7a7",
+"name": "Calculate Z-Scores and Effect Sizes",
+"type": "n8n-nodes-base.set",
+"position": [
+3140,
+600
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "790a53e8-5599-45d3-880e-ab1ad7d165d2",
+"name": "population_analysis.z_scores.asian",
+"type": "number",
+"value": "={{ ($json.output.asian_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.asian_total_number_of_reviews)) }}"
+},
+{
+"id": "ebd61097-8773-45b9-a8e6-cdd840d73650",
+"name": "population_analysis.effect_sizes.asian",
+"type": "number",
+"value": "={{ ($json.output.asian_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "627b1293-efdc-485a-83c8-bd332d6dc225",
+"name": "population_analysis.z_scores.hispanic",
+"type": "number",
+"value": "={{ ($json.output.hispanic_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.hispanic_total_number_of_reviews)) }}"
+},
+{
+"id": "822028d0-e94f-4cf7-9e13-8f8cc5c72ec0",
+"name": "population_analysis.z_scores.white",
+"type": "number",
+"value": "={{ ($json.output.white_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.white_total_number_of_reviews)) }}"
+},
+{
+"id": "d32321f9-0fcf-4e54-9059-c3fd5a901ce0",
+"name": "population_analysis.z_scores.pacific_islander",
+"type": "number",
+"value": "={{ ($json.output.pacific_islander_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.pacific_islander_total_number_of_reviews)) }}"
+},
+{
+"id": "e212d683-247f-45c4-9668-c290230a10ed",
+"name": "population_analysis.z_scores.black",
+"type": "number",
+"value": "={{ ($json.output.black_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.black_total_number_of_reviews)) }}"
+},
+{
+"id": "882049c3-eb81-4c09-af0c-5c79b0ef0154",
+"name": "population_analysis.z_scores.middle_eastern",
+"type": "number",
+"value": "={{ ($json.output.middle_eastern_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.middle_eastern_total_number_of_reviews)) }}"
+},
+{
+"id": "9bdc187f-3d8d-4030-9143-479eff441b7e",
+"name": "population_analysis.z_scores.indigenous",
+"type": "number",
+"value": "={{ ($json.output.indigenous_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.indigenous_total_number_of_reviews)) }}"
+},
+{
+"id": "0cf11453-dbae-4250-a01a-c98e35aab224",
+"name": "population_analysis.z_scores.male",
+"type": "number",
+"value": "={{ ($json.output.male_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.male_total_number_of_reviews)) }}"
+},
+{
+"id": "35a18fbc-7c2c-40fe-829d-2fffbdb13bb8",
+"name": "population_analysis.z_scores.female",
+"type": "number",
+"value": "={{ ($json.output.female_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.female_total_number_of_reviews)) }}"
+},
+{
+"id": "a6e17c1b-a89b-4c05-8184-10f7248c159f",
+"name": "population_analysis.z_scores.trans",
+"type": "number",
+"value": "={{ ($json.output.trans_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.trans_total_number_of_reviews)) }}"
+},
+{
+"id": "5e7dbccf-3011-4dba-863c-5390c1ee9e50",
+"name": "population_analysis.z_scores.hetero",
+"type": "number",
+"value": "={{ ($json.output.hetero_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.hetero_total_number_of_reviews)) }}"
+},
+{
+"id": "1872152f-2c7e-4c24-bcd5-e2777616bfe2",
+"name": "population_analysis.z_scores.lgbtqia",
+"type": "number",
+"value": "={{ ($json.output.lgbtqia_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.lgbtqia_total_number_of_reviews)) }}"
+},
+{
+"id": "91b2cb00-173e-421a-929a-51d2a6654767",
+"name": "population_analysis.z_scores.nondisabled",
+"type": "number",
+"value": "={{ ($json.output.nondisabled_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nondisabled_total_number_of_reviews)) }}"
+},
+{
+"id": "8bb7429e-0500-482c-8e8d-d2c63733ffe1",
+"name": "population_analysis.z_scores.disabled",
+"type": "number",
+"value": "={{ ($json.output.disabled_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.disabled_total_number_of_reviews)) }}"
+},
+{
+"id": "89f00d0f-80db-4ad9-bf60-9385aa3d915b",
+"name": "population_analysis.z_scores.caregiver",
+"type": "number",
+"value": "={{ ($json.output.caregiver_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.caregiver_total_number_of_reviews)) }}"
+},
+{
+"id": "0bb2b96c-d882-4ac1-9432-9fce06b26cf5",
+"name": "population_analysis.z_scores.parent",
+"type": "number",
+"value": "={{ ($json.output.parent_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.parent_total_number_of_reviews)) }}"
+},
+{
+"id": "9aae7169-1a25-4fab-b940-7f2cd7ef39d9",
+"name": "population_analysis.z_scores.nonparent",
+"type": "number",
+"value": "={{ ($json.output.nonparent_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nonparent_total_number_of_reviews)) }}"
+},
+{
+"id": "aac189a0-d6fc-4581-a15d-3e75a0cb370a",
+"name": "population_analysis.z_scores.nonveteran",
+"type": "number",
+"value": "={{ ($json.output.nonveteran_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.nonveteran_total_number_of_reviews)) }}"
+},
+{
+"id": "d40f014a-9c1d-4aea-88ac-d8a3de143931",
+"name": "population_analysis.z_scores.veteran",
+"type": "number",
+"value": "={{ ($json.output.veteran_average_rating - $json.average_rating) / ($json.std_dev / Math.sqrt($json.output.veteran_total_number_of_reviews)) }}"
+},
+{
+"id": "67e0394f-6d55-4e80-8a7d-814635620b1d",
+"name": "population_analysis.effect_sizes.hispanic",
+"type": "number",
+"value": "={{ ($json.output.hispanic_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "65cd3a22-2c97-4da1-8fcc-cc1af39118f2",
+"name": "population_analysis.effect_sizes.white",
+"type": "number",
+"value": "={{ ($json.output.white_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "a03bdf0f-e294-4a01-bb08-ddc16e9997a5",
+"name": "population_analysis.effect_sizes.pacific_islander",
+"type": "number",
+"value": "={{ ($json.output.pacific_islander_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "b0bdc40e-ed5f-475b-9d8b-8cf5beff7002",
+"name": "population_analysis.effect_sizes.black",
+"type": "number",
+"value": "={{ ($json.output.black_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "45cac3f0-7270-4fa4-8fc4-94914245a77d",
+"name": "population_analysis.effect_sizes.middle_eastern",
+"type": "number",
+"value": "={{ ($json.output.middle_eastern_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "cf5b7650-8766-45f6-8241-49aea62bf619",
+"name": "population_analysis.effect_sizes.indigenous",
+"type": "number",
+"value": "={{ ($json.output.indigenous_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "7c6a8d38-02b7-47a1-af44-5eebfb4140ec",
+"name": "population_analysis.effect_sizes.male",
+"type": "number",
+"value": "={{ ($json.output.male_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "4bf3dba9-4d07-4315-83ce-5fba288a00c9",
+"name": "population_analysis.effect_sizes.female",
+"type": "number",
+"value": "={{ ($json.output.female_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "d5e980b8-d7a8-4d4c-bcd9-fd9cbd20c729",
+"name": "population_analysis.effect_sizes.trans",
+"type": "number",
+"value": "={{ ($json.output.trans_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "2c8271c1-b612-4292-9d48-92c342b83727",
+"name": "population_analysis.effect_sizes.hetero",
+"type": "number",
+"value": "={{ ($json.output.hetero_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "996f2ea0-2e46-424b-9797-2d58fd56b1d3",
+"name": "population_analysis.effect_sizes.lgbtqia",
+"type": "number",
+"value": "={{ ($json.output.lgbtqia_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "8c987b6e-764d-422e-82de-00bd89269b22",
+"name": "population_analysis.effect_sizes.nondisabled",
+"type": "number",
+"value": "={{ ($json.output.nondisabled_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "ab796bb7-06ff-4282-b4b3-eefd129c743e",
+"name": "population_analysis.effect_sizes.disabled",
+"type": "number",
+"value": "={{ ($json.output.disabled_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "a17bf413-a098-4f24-8162-821a6a0ddb5e",
+"name": "population_analysis.effect_sizes.caregiver",
+"type": "number",
+"value": "={{ ($json.output.caregiver_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "99911e1e-06e8-4bbd-915d-b92b8b37b374",
+"name": "population_analysis.effect_sizes.parent",
+"type": "number",
+"value": "={{ ($json.output.parent_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "4ddf729b-361e-4d81-a67c-b6c18509e60b",
+"name": "population_analysis.effect_sizes.nonparent",
+"type": "number",
+"value": "={{ ($json.output.nonparent_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "725b8abb-7f72-45fc-a0c0-0e0a4f2cb131",
+"name": "population_analysis.effect_sizes.nonveteran",
+"type": "number",
+"value": "={{ ($json.output.nonveteran_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "20e54fa5-2faa-4134-90e5-81224ec9659e",
+"name": "population_analysis.effect_sizes.veteran",
+"type": "number",
+"value": "={{ ($json.output.veteran_average_rating - $json.average_rating) / $json.std_dev }}"
+},
+{
+"id": "2cc6465a-3a1c-4eb5-9e5a-72d41049d81e",
+"name": "population_analysis.review_count.asian",
+"type": "number",
+"value": "={{ $json.output.asian_total_number_of_reviews }}"
+},
+{
+"id": "0a5f6aae-ba21-47b5-8af8-fec2256e4df6",
+"name": "population_analysis.review_count.hispanic",
+"type": "number",
+"value": "={{ $json.output.hispanic_total_number_of_reviews }}"
+},
+{
+"id": "ae124587-7e24-4c1a-a002-ed801f859c30",
+"name": "population_analysis.review_count.pacific_islander",
+"type": "number",
+"value": "={{ $json.output.pacific_islander_total_number_of_reviews }}"
+},
+{
+"id": "fc790196-ca8e-4069-a093-87a413ebbf3e",
+"name": "population_analysis.review_count.black",
+"type": "number",
+"value": "={{ $json.output.black_total_number_of_reviews }}"
+},
+{
+"id": "7fd72701-781e-4e33-b000-174a853b172b",
+"name": "population_analysis.review_count.middle_eastern",
+"type": "number",
+"value": "={{ $json.output.middle_eastern_total_number_of_reviews }}"
+},
+{
+"id": "3751e7da-11a7-4af3-8aa6-1c6d53bcf27d",
+"name": "population_analysis.review_count.indigenous",
+"type": "number",
+"value": "={{ $json.output.indigenous_total_number_of_reviews }}"
+},
+{
+"id": "9ee0cac9-d2dd-4ba0-90ee-b2cdd22d9b77",
+"name": "population_analysis.review_count.male",
+"type": "number",
+"value": "={{ $json.output.male_total_number_of_reviews }}"
+},
+{
+"id": "ae7fcdc7-d373-4c24-9a65-94bd2b5847a8",
+"name": "population_analysis.review_count.female",
+"type": "number",
+"value": "={{ $json.output.female_total_number_of_reviews }}"
+},
+{
+"id": "3f53d065-269f-425a-b27d-dc5a3dbb6141",
+"name": "population_analysis.review_count.trans",
+"type": "number",
+"value": "={{ $json.output.trans_total_number_of_reviews }}"
+},
+{
+"id": "d15e976e-7599-4df0-9e65-8047b7a4cda8",
+"name": "population_analysis.review_count.hetero",
+"type": "number",
+"value": "={{ $json.output.hetero_total_number_of_reviews }}"
+},
+{
+"id": "c8b786d3-a980-469f-bf0e-de70ad44f0ea",
+"name": "population_analysis.review_count.lgbtqia",
+"type": "number",
+"value": "={{ $json.output.lgbtqia_total_number_of_reviews }}"
+},
+{
+"id": "e9429215-0858-4482-964a-75de7978ecbb",
+"name": "population_analysis.review_count.nondisabled",
+"type": "number",
+"value": "={{ $json.output.nondisabled_total_number_of_reviews }}"
+},
+{
+"id": "2c6e53c4-eab1-42aa-b956-ee882832f569",
+"name": "population_analysis.review_count.disabled",
+"type": "number",
+"value": "={{ $json.output.disabled_total_number_of_reviews }}"
+},
+{
+"id": "b5edfa25-ab11-4b94-9670-4d5589a62498",
+"name": "population_analysis.review_count.caregiver",
+"type": "number",
+"value": "={{ $json.output.caregiver_total_number_of_reviews }}"
+},
+{
+"id": "41084e96-c42f-4bb0-ac1a-883b46537fca",
+"name": "population_analysis.review_count.parent",
+"type": "number",
+"value": "={{ $json.output.parent_total_number_of_reviews }}"
+},
+{
+"id": "96496a38-9311-4ade-bd2f-2943d1d92314",
+"name": "population_analysis.review_count.nonparent",
+"type": "number",
+"value": "={{ $json.output.nonparent_total_number_of_reviews }}"
+},
+{
+"id": "5071771d-5f41-43cb-a8ce-e4e40ed3519b",
+"name": "population_analysis.review_count.nonveteran",
+"type": "number",
+"value": "={{ $json.output.nonveteran_total_number_of_reviews }}"
+},
+{
+"id": "2358e782-70da-4964-b625-5fe1946b5250",
+"name": "population_analysis.review_count.veteran",
+"type": "number",
+"value": "={{ $json.output.veteran_total_number_of_reviews }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "85536931-839a-476b-b0dd-fa6d01c6d5c1",
+"name": "Format dataset for scatterplot",
+"type": "n8n-nodes-base.code",
+"position": [
+3340,
+760
+],
+"parameters": {
+"jsCode": "// Iterate through the input data and format the dataset for QuickChart\nfor (const item of $input.all()) {\n // Ensure the data object exists and initialize datasets\n item.json.data = {\n datasets: []\n };\n\n const z_scores = item.json.population_analysis.z_scores;\n const effect_sizes = item.json.population_analysis.effect_sizes;\n const review_count = item.json.population_analysis.review_count;\n\n // Ensure z_scores, effect_sizes, and review_count are defined and are objects\n if (z_scores && effect_sizes && review_count && typeof z_scores === 'object' && typeof effect_sizes === 'object' && typeof review_count === 'object') {\n // Initialize the dataset object\n const dataset = {\n label: 'Demographics Data',\n data: []\n };\n\n // Iterate through the demographic keys\n for (const key in z_scores) {\n // Check if review count for the demographic is greater than 0\n if (z_scores.hasOwnProperty(key) && effect_sizes.hasOwnProperty(key) && review_count[key] > 0) {\n\n // Add each demographic point to the dataset\n dataset.data.push({\n x: z_scores[key], // x = z_score\n y: effect_sizes[key], // y = effect_size\n label: $('Define dictionary of demographic keys').first().json[key],\n });\n }\n }\n\n // Only add the dataset if it contains data\n if (dataset.data.length > 0) {\n item.json.data.datasets.push(dataset);\n }\n\n delete item.json.population_analysis\n }\n}\n\n// Return the updated input with the data object containing datasets and labels\nreturn $input.all();\n"
+},
+"typeVersion": 2
+},
+{
+"id": "957b9f6c-7cf8-4ec6-aec7-a7d59ed3a4ad",
+"name": "Specify additional parameters for scatterplot",
+"type": "n8n-nodes-base.set",
+"position": [
+3540,
+760
+],
+"parameters": {
+"options": {
+"ignoreConversionErrors": false
+},
+"assignments": {
+"assignments": [
+{
+"id": "5cd507f6-6835-4d2e-8329-1b5d24a3fc15",
+"name": "type",
+"type": "string",
+"value": "scatter"
+},
+{
+"id": "80b6f981-e3c7-4c6e-a0a1-f30d028fe15e",
+"name": "options",
+"type": "object",
+"value": "={\n \"title\": {\n \"display\": true,\n \"position\": \"top\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\",\n \"fontColor\": \"#666666\",\n \"fontStyle\": \"bold\",\n \"padding\": 10,\n \"lineHeight\": 1.2,\n \"text\": \"{{ $('SET company_name').item.json.company_name }} Workplace Population Bias\"\n },\n \"legend\": {\n \"display\": false\n },\n \"scales\": {\n \"xAxes\": [\n {\n \"scaleLabel\": {\n \"display\": true,\n \"labelString\": \"Z-Score\",\n \"fontColor\": \"#666666\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\"\n }\n }\n ],\n \"yAxes\": [\n {\n \"scaleLabel\": {\n \"display\": true,\n \"labelString\": \"Effect Score\",\n \"fontColor\": \"#666666\",\n \"fontSize\": 12,\n \"fontFamily\": \"sans-serif\"\n }\n }\n ]\n },\n \"plugins\": {\n \"datalabels\": {\n \"display\": true,\n \"align\": \"top\",\n \"anchor\": \"center\",\n \"backgroundColor\": \"#eee\",\n \"borderColor\": \"#ddd\",\n \"borderRadius\": 6,\n \"borderWidth\": 1,\n \"padding\": 4,\n \"color\": \"#000\",\n \"font\": {\n \"family\": \"sans-serif\",\n \"size\": 10,\n \"style\": \"normal\"\n }\n }\n }\n }"
+}
+]
+},
+"includeOtherFields": true
+},
+"typeVersion": 3.4
+},
+{
+"id": "a937132c-43fc-4fa0-ae35-885da89e51d1",
+"name": "Quickchart Scatterplot",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3740,
+760
+],
+"parameters": {
+"url": "https://quickchart.io/chart",
+"options": {},
+"sendQuery": true,
+"queryParameters": {
+"parameters": [
+{
+"name": "c",
+"value": "={{ $json.toJsonString() }}"
+},
+{
+"name": "Content-Type",
+"value": "application/json"
+},
+{
+"name": "encoding",
+"value": "url"
+}
+]
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "ede1931e-bac8-4279-b3a7-5980a190e324",
+"name": "QuickChart Bar Chart",
+"type": "n8n-nodes-base.quickChart",
+"position": [
+3740,
+560
+],
+"parameters": {
+"data": "={{ $json.population_analysis.effect_sizes.values() }}",
+"output": "bar_chart",
+"labelsMode": "array",
+"labelsArray": "={{ $json.population_analysis.effect_sizes.keys() }}",
+"chartOptions": {
+"format": "png"
+},
+"datasetOptions": {
+"label": "={{ $('SET company_name').item.json.company_name }} Effect Size on Employee Experience"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "6122fec0-619c-48d3-ad2c-05ed55ba2275",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+480,
+40
+],
+"parameters": {
+"color": 7,
+"width": 3741.593083126444,
+"height": 1044.8111554136713,
+"content": "# Spot Workplace Discrimination Patterns using ScrapingBee, Glassdoor, OpenAI, and QuickChart\n"
+},
+"typeVersion": 1
+},
+{
+"id": "5cda63e8-f31b-46f6-8cb2-41d1856ac537",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+900,
+180
+],
+"parameters": {
+"color": 4,
+"width": 1237.3377621763516,
+"height": 575.9439659309116,
+"content": "## Use ScrapingBee to gather raw data from Glassdoor"
+},
+"typeVersion": 1
+},
+{
+"id": "28d247b2-9020-4280-83d2-d6583622c0b7",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+920,
+240
+],
+"parameters": {
+"color": 7,
+"width": 804.3951263154196,
+"height": 125.73173301324687,
+"content": "### Due to javascript restrictions, a normal HTTP request cannot be used to gather user-reported details from Glassdoor. \n\nInstead, [ScrapingBee](https://www.scrapingbee.com/) offers a great tool with a very generous package of free tokens per month, which works out to roughly 4-5 runs of this workflow."
+},
+"typeVersion": 1
+},
+{
+"id": "d65a239c-06d2-470b-b24a-23ec00a9f148",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2180,
+99.69933502879758
+],
+"parameters": {
+"color": 5,
+"width": 311.0523273992095,
+"height": 843.8786512173932,
+"content": "## Extract details with AI"
+},
+"typeVersion": 1
+},
+{
+"id": "3cffd188-62a1-43a7-a67f-548e21d2b187",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2516.1138215303854,
+100
+],
+"parameters": {
+"color": 7,
+"width": 423.41585047129973,
+"height": 309.71740416262054,
+"content": "### Calculate variance and standard deviation from review rating distributions."
+},
+"typeVersion": 1
+},
+{
+"id": "b5015c07-03e3-47d4-9469-e831b2c755c0",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3320,
+706.46982689582
+],
+"parameters": {
+"color": 5,
+"width": 639.5579220386832,
+"height": 242.80759628871897,
+"content": "## Formatting datasets for Scatterplot"
+},
+"typeVersion": 1
+},
+{
+"id": "e52bb9d9-617a-46f5-b217-a6f670b6714c",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+500,
+120
+],
+"parameters": {
+"width": 356.84794255678776,
+"height": 186.36110628732342,
+"content": "## How this workflow works\n1. Replace ScrapingBee and OpenAI credentials\n2. Replace company_name with company of choice (workflow performs better with larger US-based organizations)\n3. Preview QuickChart data visualizations and AI data analysis"
+},
+"typeVersion": 1
+},
+{
+"id": "d83c07a3-04ed-418f-94f1-e70828cba8b2",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+500,
+880
+],
+"parameters": {
+"color": 6,
+"width": 356.84794255678776,
+"height": 181.54335665904924,
+"content": "### Inspired by [Wes Medford's Medium Post](https://medium.com/@wryanmedford/an-open-letter-to-twilios-leadership-f06f661ecfb4)\n\nWes performed the initial data analysis highlighting problematic behaviors at Twilio. I wanted to try and democratize the data analysis they performed for those less technical.\n\n**Hi, Wes!**"
+},
+"typeVersion": 1
+},
+{
+"id": "ed0c1b4a-99fe-4a27-90bb-ac38dd20810b",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+4020,
+880
+],
+"parameters": {
+"color": 7,
+"width": 847.5931795867759,
+"height": 522.346478008115,
+"content": "%20%7B%0A%20%20%20%20%20%20%20%20%20%20var%20idx%20%3D%20context.dataIndex%3B%0A%20%20%20%20%20%20%20%20%20%20return%20context.chart.data.labels%5Bidx%5D%3B%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A#full-width)"
+},
+"typeVersion": 1
+},
+{
+"id": "7b92edf8-3a58-4931-abf4-d9c2f57cfa32",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3980,
+800
+],
+"parameters": {
+"color": 6,
+"width": 989.7621518164046,
+"height": 636.6345107975716,
+"content": "## Example Scatterplot output"
+},
+"typeVersion": 1
+},
+{
+"id": "bd6859b4-096c-401e-9bce-91e970e1afd1",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2540,
+800
+],
+"parameters": {
+"color": 6,
+"width": 737.6316136259719,
+"height": 444.9087184962878,
+"content": "## Glossary\n**Z-Score** โ A statistical measure that indicates how many standard deviations a data point is from the mean. In this analysis, a negative z-score suggests a group rates their workplace experience lower than the average, while a positive z-score suggests a better-than-average experience.\n\n**Effect Size** โ A measure of the magnitude of difference between groups. Larger negative effect sizes indicate a more substantial disparity in workplace experiences for certain groups, making it useful for identifying meaningful gaps beyond just statistical significance.\n\n**P-Score (P-Value)** โ The probability that the observed differences occurred by chance. A lower p-score (typically below 0.05) suggests the difference is statistically significant and unlikely to be random. In this analysis, high p-scores confirm that the disparities in ratings for marginalized groups are unlikely to be due to chance alone.\n\n### Relevance to This Analysis\nThese metrics help quantify workplace disparities among demographic groups. Z-scores show which groups report better or worse experiences, effect sizes reveal the severity of these differences, and p-scores confirm whether the disparities are statistically meaningful. This data allows for a more informed discussion about workplace equity and areas needing improvement."
+},
+"typeVersion": 1
+},
+{
+"id": "5af3ef87-ed4b-481e-b1ba-d44ffb7551d8",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+4140,
+80
+],
+"parameters": {
+"color": 6,
+"width": 643.5995639515581,
+"height": 646.0030521944287,
+"content": "## Example AI Analysis (Twilio Example)\n\n### Key Takeaways\n1. **Significant Disparity Among Disabled Employees**\nDisabled employees reported the lowest average ratings, with a z-score of -5.50, indicating a far worse experience compared to their non-disabled peers. \n2. **LGBTQIA Community's Challenges**\nMembers of the LGBTQIA community showed significantly lower ratings (z-score of -2.53), suggesting they may experience a workplace environment that is less inclusive or supportive compared to others.\n3. **Transgender Experiences Are Particularly Negative**\nTransgender employees rated their experiences considerably lower (z-score of -2.91), highlighting a critical area for improvement in workplace culture and acceptance.\n4. **Veterans Report Higher Satisfaction**\nIn contrast, veterans had the highest ratings (z-score of 1.54), which could indicate a supportive environment or programs tailored to their needs.\n5. **Overall Gender Discrepancies**\nA noticeable gap exists in average ratings by gender, with female employees scoring below male employees, suggesting potential gender biases or challenges in workplace dynamics.\n\n### Employee Experiences\n#### Perceptions of Workplace Environment\nFor members of groups reporting significantly worse experiences, such as disabled, transgender, and LGBTQIA employees, the workplace may feel alienating or unwelcoming. These individuals might perceive that their contributions are undervalued or overlooked and that necessary support systems are lacking, creating a culture of exclusion rather than one of inclusivity. This feeling of being marginalized can lead to poorer engagement, higher turnover rates, and diminished overall job satisfaction, adversely impacting both employees and the organization."
+},
+"typeVersion": 1
+},
+{
+"id": "a39cdbe7-d6ae-4a84-98c7-52ebf98242f3",
+"name": "Text Analysis of Bias Data",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+3720,
+280
+],
+"parameters": {
+"text": "=This data compares the average rating given by different demographic groups against a baseline (the overall mean rating).\n\nObjective:\n1. Analyze the data and offer between 2 and 5 key takeaways with a title and short (one-sentence) summary.\n2. Below the key takeaways, Include a heading called \"Employee Experiences\". Under this heading, include a subheader and paragraph describing the possible perception of the workplace for members of any groups reporting significantly worse (or better) experiences than others.\n3. Ensure there are between 2-5 key takeaways and employee experiences\n\nData for analysis:\n{{ $json.population_analysis.toJsonString() }}",
+"promptType": "define"
+},
+"typeVersion": 1.4
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "ff1df786-ebaf-4ed0-aeca-1872b93ef275",
+"connections": {
+"Merge": {
+"main": [
+[
+{
+"node": "Calculate Z-Scores and Effect Sizes",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"SET company_name": {
+"main": [
+[
+{
+"node": "Define dictionary of demographic keys",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Text Analysis of Bias Data",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Sort Effect Sizes": {
+"main": [
+[
+{
+"node": "QuickChart Bar Chart",
+"type": "main",
+"index": 0
+},
+{
+"node": "Text Analysis of Bias Data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Calculate P-Scores": {
+"main": [
+[
+{
+"node": "Sort Effect Sizes",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Extract demographic distributions",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model2": {
+"ai_languageModel": [
+[
+{
+"node": "Extract overall ratings and distribution percentages",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Extract company url path": {
+"main": [
+[
+{
+"node": "ScrapingBee GET company page contents",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set variance and std_dev": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Demographics Module": {
+"main": [
+[
+{
+"node": "Extract demographic distributions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"ScrapingBee Search Glassdoor": {
+"main": [
+[
+{
+"node": "Extract company url path",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract reviews page url path": {
+"main": [
+[
+{
+"node": "ScrapingBee GET Glassdoor Reviews Content",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Overall Review Summary": {
+"main": [
+[
+{
+"node": "Extract overall ratings and distribution percentages",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Format dataset for scatterplot": {
+"main": [
+[
+{
+"node": "Specify additional parameters for scatterplot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Define contributions to variance": {
+"main": [
+[
+{
+"node": "Set variance and std_dev",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract demographic distributions": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "SET company_name",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Calculate Z-Scores and Effect Sizes": {
+"main": [
+[
+{
+"node": "Calculate P-Scores",
+"type": "main",
+"index": 0
+},
+{
+"node": "Format dataset for scatterplot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Define dictionary of demographic keys": {
+"main": [
+[
+{
+"node": "ScrapingBee Search Glassdoor",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"ScrapingBee GET company page contents": {
+"main": [
+[
+{
+"node": "Extract reviews page url path",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"ScrapingBee GET Glassdoor Reviews Content": {
+"main": [
+[
+{
+"node": "Extract Demographics Module",
+"type": "main",
+"index": 0
+},
+{
+"node": "Extract Overall Review Summary",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Specify additional parameters for scatterplot": {
+"main": [
+[
+{
+"node": "Quickchart Scatterplot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract overall ratings and distribution percentages": {
+"main": [
+[
+{
+"node": "Define contributions to variance",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.json b/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.json
deleted file mode 100644
index 2c85a40..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.json
+++ /dev/null
@@ -1,277 +0,0 @@
-{
- "id": "qmmXKcpJOCm9qaCk",
- "meta": {
- "instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
- "templateCredsSetupCompleted": true
- },
- "name": "SERPBear analytics template",
- "tags": [],
- "nodes": [
- {
- "id": "2ad0eb40-6628-4c6b-bc15-7081e7712f1a",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 260,
- 380
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "5a3c9ad8-a562-4bb0-bb11-c325552d8101",
- "name": "Schedule Trigger",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- 260,
- 160
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "bdfa7388-f9b3-4145-90de-2e58138e14bf",
- "name": "Get data from SerpBear",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 580,
- 260
- ],
- "parameters": {
- "url": "https://myserpbearinstance.com/api/keyword?id=22",
- "options": {},
- "sendQuery": true,
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth",
- "queryParameters": {
- "parameters": [
- {
- "name": "domain",
- "value": "rumjahn.com"
- }
- ]
- }
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "3fshHb4fyI5XfLyq",
- "name": "Header Auth account 6"
- }
- },
- "executeOnce": false,
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "c169f4e3-ab60-4b46-9f49-cf27a13dd7c6",
- "name": "Parse data from SerpBear",
- "type": "n8n-nodes-base.code",
- "position": [
- 820,
- 260
- ],
- "parameters": {
- "jsCode": "const keywords = items[0].json.keywords;\nconst today = new Date().toISOString().split('T')[0];\n\n// Create summary for each keyword\nconst keywordSummaries = keywords.map(kw => {\n const position = kw.position || 0;\n const lastWeekPositions = Object.values(kw.history || {}).slice(-7);\n const avgPosition = lastWeekPositions.reduce((a, b) => a + b, 0) / lastWeekPositions.length;\n \n return {\n keyword: kw.keyword,\n currentPosition: position,\n averagePosition: Math.round(avgPosition * 10) / 10,\n trend: position < avgPosition ? 'improving' : position > avgPosition ? 'declining' : 'stable',\n url: kw.url || 'not ranking'\n };\n});\n\n// Create the prompt\nconst prompt = `Here's the SEO ranking data for rumjahn.com as of ${today}:\n\n${keywordSummaries.map(kw => `\nKeyword: \"${kw.keyword}\"\nCurrent Position: ${kw.currentPosition}\n7-Day Average: ${kw.averagePosition}\nTrend: ${kw.trend}\nRanking URL: ${kw.url}\n`).join('\\n')}\n\nPlease analyze this data and provide:\n1. Key observations about ranking performance\n2. Keywords showing the most improvement\n3. Keywords needing attention\n4. Suggested actions for improvement`;\n\nreturn {\n prompt\n};"
- },
- "typeVersion": 2
- },
- {
- "id": "cc6e16a7-db46-42fe-837a-59ce635c906c",
- "name": "Send data to A.I. for analysis",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1060,
- 260
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. This is keyword data for my site. Can you summarize the data into a table and then give me some suggestions:{{ encodeURIComponent($json.prompt)}}\" \n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "WY7UkF14ksPKq3S8",
- "name": "Header Auth account 2"
- }
- },
- "typeVersion": 4.2,
- "alwaysOutputData": false
- },
- {
- "id": "a623f06c-1dfe-4d04-a7fd-fed7049a7588",
- "name": "Save data to Baserow",
- "type": "n8n-nodes-base.baserow",
- "position": [
- 1340,
- 260
- ],
- "parameters": {
- "tableId": 644,
- "fieldsUi": {
- "fieldValues": [
- {
- "fieldId": 6264,
- "fieldValue": "={{ DateTime.now().toFormat('yyyy-MM-dd') }}"
- },
- {
- "fieldId": 6265,
- "fieldValue": "={{ $json.choices[0].message.content }}"
- },
- {
- "fieldId": 6266,
- "fieldValue": "Rumjahn"
- }
- ]
- },
- "operation": "create",
- "databaseId": 121
- },
- "credentials": {
- "baserowApi": {
- "id": "8w0zXhycIfCAgja3",
- "name": "Baserow account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "e8048faf-bbed-4e48-b273-d1a50a767e76",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 220,
- -360
- ],
- "parameters": {
- "color": 5,
- "width": 614.709677419355,
- "height": 208.51612903225802,
- "content": "## Send Matomo analytics to A.I. and save results to baserow\n\nThis workflow will check the Google keywords for your site and it's rank.\n\n[๐ก You can read more about this workflow here](https://rumjahn.com/how-to-create-an-a-i-agent-to-analyze-serpbear-keyword-rankings-using-n8n-for-free-without-any-coding-skills-required/)"
- },
- "typeVersion": 1
- },
- {
- "id": "1a18e685-79db-423f-992a-5e0d4ddeb672",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 520,
- -80
- ],
- "parameters": {
- "width": 214.75050403225822,
- "height": 531.7318548387107,
- "content": "## Get SERPBear Data\n \n1. Enter your SerpBear API keys and URL. You need to find your website ID which is probably 1.\n2. Navigate to Administration > Personal > Security > Auth tokens within your Matomo dashboard. Click on Create new token and provide a purpose for reference."
- },
- "typeVersion": 1
- },
- {
- "id": "99895baf-75d0-4af2-87de-5b8951186e78",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 980,
- -60
- ],
- "parameters": {
- "color": 3,
- "width": 225.99936321742769,
- "height": 508.95792207792226,
- "content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
- },
- "typeVersion": 1
- },
- {
- "id": "07d03511-98b0-4f4a-8e68-96ca177fb246",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1240,
- -40
- ],
- "parameters": {
- "color": 6,
- "width": 331.32883116883124,
- "height": 474.88,
- "content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Date\n- Note\n- Blog\n\nEnter the name of your website under \"Blog\" field."
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "8b7e7da7-1965-4ca4-8e15-889eda819723",
- "connections": {
- "Schedule Trigger": {
- "main": [
- [
- {
- "node": "Get data from SerpBear",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get data from SerpBear": {
- "main": [
- [
- {
- "node": "Parse data from SerpBear",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse data from SerpBear": {
- "main": [
- [
- {
- "node": "Send data to A.I. for analysis",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send data to A.I. for analysis": {
- "main": [
- [
- {
- "node": "Save data to Baserow",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get data from SerpBear",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.txt b/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.txt
new file mode 100644
index 0000000..5aba492
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Summarize SERPBear data with AI (via Openrouter) and save it to Baserow.txt
@@ -0,0 +1,277 @@
+{
+"id": "qmmXKcpJOCm9qaCk",
+"meta": {
+"instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
+"templateCredsSetupCompleted": true
+},
+"name": "SERPBear analytics template",
+"tags": [],
+"nodes": [
+{
+"id": "2ad0eb40-6628-4c6b-bc15-7081e7712f1a",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+260,
+380
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "5a3c9ad8-a562-4bb0-bb11-c325552d8101",
+"name": "Schedule Trigger",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+260,
+160
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "bdfa7388-f9b3-4145-90de-2e58138e14bf",
+"name": "Get data from SerpBear",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+580,
+260
+],
+"parameters": {
+"url": "https://myserpbearinstance.com/api/keyword?id=22",
+"options": {},
+"sendQuery": true,
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth",
+"queryParameters": {
+"parameters": [
+{
+"name": "domain",
+"value": "rumjahn.com"
+}
+]
+}
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "3fshHb4fyI5XfLyq",
+"name": "Header Auth account 6"
+}
+},
+"executeOnce": false,
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "c169f4e3-ab60-4b46-9f49-cf27a13dd7c6",
+"name": "Parse data from SerpBear",
+"type": "n8n-nodes-base.code",
+"position": [
+820,
+260
+],
+"parameters": {
+"jsCode": "const keywords = items[0].json.keywords;\nconst today = new Date().toISOString().split('T')[0];\n\n// Create summary for each keyword\nconst keywordSummaries = keywords.map(kw => {\n const position = kw.position || 0;\n const lastWeekPositions = Object.values(kw.history || {}).slice(-7);\n const avgPosition = lastWeekPositions.reduce((a, b) => a + b, 0) / lastWeekPositions.length;\n \n return {\n keyword: kw.keyword,\n currentPosition: position,\n averagePosition: Math.round(avgPosition * 10) / 10,\n trend: position < avgPosition ? 'improving' : position > avgPosition ? 'declining' : 'stable',\n url: kw.url || 'not ranking'\n };\n});\n\n// Create the prompt\nconst prompt = `Here's the SEO ranking data for rumjahn.com as of ${today}:\n\n${keywordSummaries.map(kw => `\nKeyword: \"${kw.keyword}\"\nCurrent Position: ${kw.currentPosition}\n7-Day Average: ${kw.averagePosition}\nTrend: ${kw.trend}\nRanking URL: ${kw.url}\n`).join('\\n')}\n\nPlease analyze this data and provide:\n1. Key observations about ranking performance\n2. Keywords showing the most improvement\n3. Keywords needing attention\n4. Suggested actions for improvement`;\n\nreturn {\n prompt\n};"
+},
+"typeVersion": 2
+},
+{
+"id": "cc6e16a7-db46-42fe-837a-59ce635c906c",
+"name": "Send data to A.I. for analysis",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1060,
+260
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. This is keyword data for my site. Can you summarize the data into a table and then give me some suggestions:{{ encodeURIComponent($json.prompt)}}\" \n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "WY7UkF14ksPKq3S8",
+"name": "Header Auth account 2"
+}
+},
+"typeVersion": 4.2,
+"alwaysOutputData": false
+},
+{
+"id": "a623f06c-1dfe-4d04-a7fd-fed7049a7588",
+"name": "Save data to Baserow",
+"type": "n8n-nodes-base.baserow",
+"position": [
+1340,
+260
+],
+"parameters": {
+"tableId": 644,
+"fieldsUi": {
+"fieldValues": [
+{
+"fieldId": 6264,
+"fieldValue": "={{ DateTime.now().toFormat('yyyy-MM-dd') }}"
+},
+{
+"fieldId": 6265,
+"fieldValue": "={{ $json.choices[0].message.content }}"
+},
+{
+"fieldId": 6266,
+"fieldValue": "Rumjahn"
+}
+]
+},
+"operation": "create",
+"databaseId": 121
+},
+"credentials": {
+"baserowApi": {
+"id": "8w0zXhycIfCAgja3",
+"name": "Baserow account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "e8048faf-bbed-4e48-b273-d1a50a767e76",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+220,
+-360
+],
+"parameters": {
+"color": 5,
+"width": 614.709677419355,
+"height": 208.51612903225802,
+"content": "## Send Matomo analytics to A.I. and save results to baserow\n\nThis workflow will check the Google keywords for your site and it's rank.\n\n[๐ก You can read more about this workflow here](https://rumjahn.com/how-to-create-an-a-i-agent-to-analyze-serpbear-keyword-rankings-using-n8n-for-free-without-any-coding-skills-required/)"
+},
+"typeVersion": 1
+},
+{
+"id": "1a18e685-79db-423f-992a-5e0d4ddeb672",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+520,
+-80
+],
+"parameters": {
+"width": 214.75050403225822,
+"height": 531.7318548387107,
+"content": "## Get SERPBear Data\n \n1. Enter your SerpBear API keys and URL. You need to find your website ID which is probably 1.\n2. Navigate to Administration > Personal > Security > Auth tokens within your Matomo dashboard. Click on Create new token and provide a purpose for reference."
+},
+"typeVersion": 1
+},
+{
+"id": "99895baf-75d0-4af2-87de-5b8951186e78",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+980,
+-60
+],
+"parameters": {
+"color": 3,
+"width": 225.99936321742769,
+"height": 508.95792207792226,
+"content": "## Send data to A.I.\n\nFill in your Openrouter A.I. credentials. Use Header Auth.\n- Username: Authorization\n- Password: Bearer {insert your API key}\n\nRemember to add a space after bearer. Also, feel free to modify the prompt to A.1."
+},
+"typeVersion": 1
+},
+{
+"id": "07d03511-98b0-4f4a-8e68-96ca177fb246",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1240,
+-40
+],
+"parameters": {
+"color": 6,
+"width": 331.32883116883124,
+"height": 474.88,
+"content": "## Send data to Baserow\n\nCreate a table first with the following columns:\n- Date\n- Note\n- Blog\n\nEnter the name of your website under \"Blog\" field."
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "8b7e7da7-1965-4ca4-8e15-889eda819723",
+"connections": {
+"Schedule Trigger": {
+"main": [
+[
+{
+"node": "Get data from SerpBear",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get data from SerpBear": {
+"main": [
+[
+{
+"node": "Parse data from SerpBear",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse data from SerpBear": {
+"main": [
+[
+{
+"node": "Send data to A.I. for analysis",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send data to A.I. for analysis": {
+"main": [
+[
+{
+"node": "Save data to Baserow",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get data from SerpBear",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.json b/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.json
deleted file mode 100644
index 627ec7f..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.json
+++ /dev/null
@@ -1,454 +0,0 @@
-{
- "id": "eZT6SZ4Kvmq5TzyQ",
- "meta": {
- "instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
- "templateCredsSetupCompleted": true
- },
- "name": "Umami analytics template",
- "tags": [],
- "nodes": [
- {
- "id": "8a54ac1c-a072-42e6-a3ba-8cde33475eb5",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 460,
- 220
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "e81c9be0-f59d-467e-9bda-eeb2d66ed31e",
- "name": "Schedule Trigger",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- 460,
- 380
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks",
- "triggerAtDay": [
- 4
- ]
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "01b04872-9aea-4834-8df5-f6c91914133d",
- "name": "Get view stats from Umami",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 760,
- 260
- ],
- "parameters": {
- "url": "=https://umami.mydomain.com/api/websites/86d4095c-a2a8-4fc8-9521-103e858e2b41/event-data/stats?startAt={{ DateTime.now().minus({ days: 7 }).toMillis() }}&endAt={{ DateTime.now().toMillis() }}&unit=hour&timezone=Asia%2FHong_Kong",
- "options": {},
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "FKsKXvQUlaX5qt9n",
- "name": "Header Auth account 3"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "38d342e3-10ad-4260-8f44-5a3233ec3166",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 660,
- -260
- ],
- "parameters": {
- "width": 504.88636363636317,
- "content": "## Send data from Umami to A.I. and then save to Baserow\n\nYou can find out more about the stats available in the [Umami API](https://umami.is/docs/api/website-stats-api)\n\nRead the [case study here](https://rumjahn.com/how-to-analyze-umami-data-using-n8n-and-a-i-to-improve-seo-and-uncover-hidden-insights-for-better-content-optimization/).\n\n"
- },
- "typeVersion": 1
- },
- {
- "id": "18c997fe-61b1-464a-8bb5-fcdc017dd1f6",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 660,
- -60
- ],
- "parameters": {
- "color": 4,
- "width": 393.16558441558414,
- "height": 504.17207792207796,
- "content": "## Get summary stats from Umami\n\nIt will get: Pageviews, Visitors, Visits, Bounces, Total Time\n\nYou need to change the URL to your website. https://{your website}/api/websites/{website ID}/\n\nYou can find your ID by going to your Umami account -> Settings -> Edit (next to domain)"
- },
- "typeVersion": 1
- },
- {
- "id": "bfdc04a2-57fa-4a8a-b412-39047cebb370",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1080,
- -60
- ],
- "parameters": {
- "color": 5,
- "width": 216.5746753246753,
- "height": 502.37012987012963,
- "content": "## Send data to A.I.\n\nTo use Openrouter, you need to register for an account.\nThen add header authorization credentials.\nUsername: Authroization\nPassword: Bearer {Your API Key}\n*It's Bearer space {API key}."
- },
- "typeVersion": 1
- },
- {
- "id": "fc373fd7-52fc-4729-8022-021c09d0c89c",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1320,
- -60
- ],
- "parameters": {
- "color": 6,
- "width": 746.3474025974022,
- "height": 505.9740259740257,
- "content": "## Get page specific stats for this week and last\n\nCalls Umami to get this week and last week's data. It will get the views for each page visited on your website for comparison."
- },
- "typeVersion": 1
- },
- {
- "id": "82bd35b6-8b49-4d77-8be2-033a8bff3f41",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2120,
- -60
- ],
- "parameters": {
- "color": 5,
- "width": 216.5746753246753,
- "height": 502.37012987012963,
- "content": "## Send data to A.I.\n\nTo use Openrouter, you need to register for an account.\nThen add header authorization credentials.\nUsername: Authroization\nPassword: Bearer {Your API Key}\n*It's Bearer space {API key}."
- },
- "typeVersion": 1
- },
- {
- "id": "503c4ca3-36da-41a8-9029-f844a34daa59",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2380,
- -60
- ],
- "parameters": {
- "color": 4,
- "width": 393.16558441558414,
- "height": 504.17207792207796,
- "content": "## Save analysis to baserow\n\nYou need to create a table in advance to save. \n- Date (date)\n- Summary (Long text)\n- Top pages (Long text)\n- Blog name (Long text)"
- },
- "typeVersion": 1
- },
- {
- "id": "f64cdfbd-712f-461c-b025-25f37e2bded8",
- "name": "Parse Umami data",
- "type": "n8n-nodes-base.code",
- "position": [
- 940,
- 260
- ],
- "parameters": {
- "jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Create a simplified object with the metrics\n const simplified = {\n pageviews: {\n value: parseInt(data.pageviews.value) || 0,\n prev: parseInt(data.pageviews.prev) || 0\n },\n visitors: {\n value: parseInt(data.visitors.value) || 0,\n prev: parseInt(data.visitors.prev) || 0\n },\n visits: {\n value: parseInt(data.visits.value) || 0,\n prev: parseInt(data.visits.prev) || 0\n },\n bounces: {\n value: parseInt(data.bounces.value) || 0,\n prev: parseInt(data.bounces.prev) || 0\n },\n totaltime: {\n value: parseInt(data.totaltime.value) || 0,\n prev: parseInt(data.totaltime.prev) || 0\n }\n };\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
- },
- "typeVersion": 2
- },
- {
- "id": "470715b6-0878-48b8-b6c6-40de27fbc966",
- "name": "Send data to A.I.",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1140,
- 260
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Here is data from Umami analytics of Pennibnotes.com. Where X is URL and Y is number of visitors. Give me a table summary of this data in markdown format:{{ $('Parse Umami data').item.json.urlString }}.\"\n }\n ]\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "WY7UkF14ksPKq3S8",
- "name": "Header Auth account 2"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "ea4bb37f-96d9-41b8-bf46-fb09865a6e0f",
- "name": "Get page data from Umami",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1380,
- 260
- ],
- "parameters": {
- "url": "=https://umami.rumjahn.synology.me/api/websites/f375d28c-1949-4597-8871-f1b942e3aa24/metrics?startAt={{Date.now() - (7 * 24 * 60 * 60 * 1000)}}&endAt={{Date.now()}}&type=url&tz=America/Los_Angeles",
- "options": {},
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "FKsKXvQUlaX5qt9n",
- "name": "Header Auth account 3"
- }
- },
- "typeVersion": 4
- },
- {
- "id": "d982606b-49c8-4d5b-ba79-bd0fdd2600b6",
- "name": "Parse Umami data1",
- "type": "n8n-nodes-base.code",
- "position": [
- 1560,
- 260
- ],
- "parameters": {
- "jsCode": "// Get input data\nconst data = $input.all();\n\n// Create URL-encoded string from the data\nconst encodedData = encodeURIComponent(JSON.stringify(data));\n\n// Return the encoded data\nreturn {\n json: {\n thisWeek: encodedData\n }\n};"
- },
- "typeVersion": 2
- },
- {
- "id": "f3734045-1318-4234-a3ac-61b766124609",
- "name": "Get page view data from Umami",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1760,
- 260
- ],
- "parameters": {
- "url": "=https://umami.rumjahn.synology.me/api/websites/f375d28c-1949-4597-8871-f1b942e3aa24/metrics?startAt={{Date.now() - (14 * 24 * 60 * 60 * 1000)}}&endAt={{Date.now() - (7 * 24 * 60 * 60 * 1000)}}&type=url&tz=America/Los_Angeles",
- "options": {},
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "FKsKXvQUlaX5qt9n",
- "name": "Header Auth account 3"
- }
- },
- "typeVersion": 4
- },
- {
- "id": "a0153ab0-3eaf-4f97-a2dc-ab63d45a9187",
- "name": "Parse Umami",
- "type": "n8n-nodes-base.code",
- "position": [
- 1920,
- 260
- ],
- "parameters": {
- "jsCode": "// Get input data\nconst data = $input.all();\n\n// Create URL-encoded string from the data\nconst encodedData = encodeURIComponent(JSON.stringify(data));\n\n// Return the encoded data\nreturn {\n json: {\n lastweek: encodedData\n }\n};"
- },
- "typeVersion": 2
- },
- {
- "id": "c2d3d396-09fa-4800-b56d-40ed7592cd3c",
- "name": "Send data to A.I.1",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2180,
- 260
- ],
- "parameters": {
- "url": "https://openrouter.ai/api/v1/chat/completions",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Here is data from Umami analytics of Pennibnotes.com. Where X is URL and Y is number of visitors. Compare the data from this week to last week. Present the data in a table using markdown and offer 5 improvement suggestions. This week:{{ $('Parse Umami data1').first().json.thisWeek }} Lastweek:{{ $json.lastweek }}\"\n }\n ]\n}\n\n",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "WY7UkF14ksPKq3S8",
- "name": "Header Auth account 2"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "ce58a556-c05a-4395-88b0-3edecbad80e5",
- "name": "Save data to Baserow",
- "type": "n8n-nodes-base.baserow",
- "position": [
- 2520,
- 260
- ],
- "parameters": {
- "tableId": 607,
- "fieldsUi": {
- "fieldValues": [
- {
- "fieldId": 5870,
- "fieldValue": "={{ $json.choices[0].message.content }}"
- },
- {
- "fieldId": 5869,
- "fieldValue": "={{ $('Send data to A.I.').first().json.choices[0].message.content }}"
- },
- {
- "fieldId": 5868,
- "fieldValue": "={{ DateTime.now().toFormat('yyyy-MM-dd') }}"
- },
- {
- "fieldId": 5871,
- "fieldValue": "Name of your blog"
- }
- ]
- },
- "operation": "create",
- "databaseId": 121
- },
- "credentials": {
- "baserowApi": {
- "id": "8w0zXhycIfCAgja3",
- "name": "Baserow account"
- }
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "e28e067d-9245-4879-9321-4d21925f951e",
- "connections": {
- "Parse Umami": {
- "main": [
- [
- {
- "node": "Send data to A.I.1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Umami data": {
- "main": [
- [
- {
- "node": "Send data to A.I.",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Schedule Trigger": {
- "main": [
- [
- {
- "node": "Get view stats from Umami",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Parse Umami data1": {
- "main": [
- [
- {
- "node": "Get page view data from Umami",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send data to A.I.": {
- "main": [
- [
- {
- "node": "Get page data from Umami",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Send data to A.I.1": {
- "main": [
- [
- {
- "node": "Save data to Baserow",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get page data from Umami": {
- "main": [
- [
- {
- "node": "Parse Umami data1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get view stats from Umami": {
- "main": [
- [
- {
- "node": "Parse Umami data",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get page view data from Umami": {
- "main": [
- [
- {
- "node": "Parse Umami",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get view stats from Umami",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.txt b/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.txt
new file mode 100644
index 0000000..ffd5fb5
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Summarize Umami data with AI (via Openrouter) and save it to Baserow.txt
@@ -0,0 +1,454 @@
+{
+"id": "eZT6SZ4Kvmq5TzyQ",
+"meta": {
+"instanceId": "558d88703fb65b2d0e44613bc35916258b0f0bf983c5d4730c00c424b77ca36a",
+"templateCredsSetupCompleted": true
+},
+"name": "Umami analytics template",
+"tags": [],
+"nodes": [
+{
+"id": "8a54ac1c-a072-42e6-a3ba-8cde33475eb5",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+460,
+220
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "e81c9be0-f59d-467e-9bda-eeb2d66ed31e",
+"name": "Schedule Trigger",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+460,
+380
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks",
+"triggerAtDay": [
+4
+]
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "01b04872-9aea-4834-8df5-f6c91914133d",
+"name": "Get view stats from Umami",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+760,
+260
+],
+"parameters": {
+"url": "=https://umami.mydomain.com/api/websites/86d4095c-a2a8-4fc8-9521-103e858e2b41/event-data/stats?startAt={{ DateTime.now().minus({ days: 7 }).toMillis() }}&endAt={{ DateTime.now().toMillis() }}&unit=hour&timezone=Asia%2FHong_Kong",
+"options": {},
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "FKsKXvQUlaX5qt9n",
+"name": "Header Auth account 3"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "38d342e3-10ad-4260-8f44-5a3233ec3166",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+660,
+-260
+],
+"parameters": {
+"width": 504.88636363636317,
+"content": "## Send data from Umami to A.I. and then save to Baserow\n\nYou can find out more about the stats available in the [Umami API](https://umami.is/docs/api/website-stats-api)\n\nRead the [case study here](https://rumjahn.com/how-to-analyze-umami-data-using-n8n-and-a-i-to-improve-seo-and-uncover-hidden-insights-for-better-content-optimization/).\n\n"
+},
+"typeVersion": 1
+},
+{
+"id": "18c997fe-61b1-464a-8bb5-fcdc017dd1f6",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+660,
+-60
+],
+"parameters": {
+"color": 4,
+"width": 393.16558441558414,
+"height": 504.17207792207796,
+"content": "## Get summary stats from Umami\n\nIt will get: Pageviews, Visitors, Visits, Bounces, Total Time\n\nYou need to change the URL to your website. https://{your website}/api/websites/{website ID}/\n\nYou can find your ID by going to your Umami account -> Settings -> Edit (next to domain)"
+},
+"typeVersion": 1
+},
+{
+"id": "bfdc04a2-57fa-4a8a-b412-39047cebb370",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1080,
+-60
+],
+"parameters": {
+"color": 5,
+"width": 216.5746753246753,
+"height": 502.37012987012963,
+"content": "## Send data to A.I.\n\nTo use Openrouter, you need to register for an account.\nThen add header authorization credentials.\nUsername: Authroization\nPassword: Bearer {Your API Key}\n*It's Bearer space {API key}."
+},
+"typeVersion": 1
+},
+{
+"id": "fc373fd7-52fc-4729-8022-021c09d0c89c",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1320,
+-60
+],
+"parameters": {
+"color": 6,
+"width": 746.3474025974022,
+"height": 505.9740259740257,
+"content": "## Get page specific stats for this week and last\n\nCalls Umami to get this week and last week's data. It will get the views for each page visited on your website for comparison."
+},
+"typeVersion": 1
+},
+{
+"id": "82bd35b6-8b49-4d77-8be2-033a8bff3f41",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2120,
+-60
+],
+"parameters": {
+"color": 5,
+"width": 216.5746753246753,
+"height": 502.37012987012963,
+"content": "## Send data to A.I.\n\nTo use Openrouter, you need to register for an account.\nThen add header authorization credentials.\nUsername: Authroization\nPassword: Bearer {Your API Key}\n*It's Bearer space {API key}."
+},
+"typeVersion": 1
+},
+{
+"id": "503c4ca3-36da-41a8-9029-f844a34daa59",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2380,
+-60
+],
+"parameters": {
+"color": 4,
+"width": 393.16558441558414,
+"height": 504.17207792207796,
+"content": "## Save analysis to baserow\n\nYou need to create a table in advance to save. \n- Date (date)\n- Summary (Long text)\n- Top pages (Long text)\n- Blog name (Long text)"
+},
+"typeVersion": 1
+},
+{
+"id": "f64cdfbd-712f-461c-b025-25f37e2bded8",
+"name": "Parse Umami data",
+"type": "n8n-nodes-base.code",
+"position": [
+940,
+260
+],
+"parameters": {
+"jsCode": "function transformToUrlString(items) {\n // In n8n, we need to check if items is an array and get the json property\n const data = items[0].json;\n \n if (!data) {\n console.log('No valid data found');\n return encodeURIComponent(JSON.stringify([]));\n }\n \n try {\n // Create a simplified object with the metrics\n const simplified = {\n pageviews: {\n value: parseInt(data.pageviews.value) || 0,\n prev: parseInt(data.pageviews.prev) || 0\n },\n visitors: {\n value: parseInt(data.visitors.value) || 0,\n prev: parseInt(data.visitors.prev) || 0\n },\n visits: {\n value: parseInt(data.visits.value) || 0,\n prev: parseInt(data.visits.prev) || 0\n },\n bounces: {\n value: parseInt(data.bounces.value) || 0,\n prev: parseInt(data.bounces.prev) || 0\n },\n totaltime: {\n value: parseInt(data.totaltime.value) || 0,\n prev: parseInt(data.totaltime.prev) || 0\n }\n };\n \n return encodeURIComponent(JSON.stringify(simplified));\n } catch (error) {\n console.log('Error processing data:', error);\n throw new Error('Invalid data structure');\n }\n}\n\n// Get the input data\nconst items = $input.all();\n\n// Process the data\nconst result = transformToUrlString(items);\n\n// Return the result\nreturn { json: { urlString: result } };"
+},
+"typeVersion": 2
+},
+{
+"id": "470715b6-0878-48b8-b6c6-40de27fbc966",
+"name": "Send data to A.I.",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1140,
+260
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Here is data from Umami analytics of Pennibnotes.com. Where X is URL and Y is number of visitors. Give me a table summary of this data in markdown format:{{ $('Parse Umami data').item.json.urlString }}.\"\n }\n ]\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "WY7UkF14ksPKq3S8",
+"name": "Header Auth account 2"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "ea4bb37f-96d9-41b8-bf46-fb09865a6e0f",
+"name": "Get page data from Umami",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1380,
+260
+],
+"parameters": {
+"url": "=https://umami.rumjahn.synology.me/api/websites/f375d28c-1949-4597-8871-f1b942e3aa24/metrics?startAt={{Date.now() - (7 * 24 * 60 * 60 * 1000)}}&endAt={{Date.now()}}&type=url&tz=America/Los_Angeles",
+"options": {},
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "FKsKXvQUlaX5qt9n",
+"name": "Header Auth account 3"
+}
+},
+"typeVersion": 4
+},
+{
+"id": "d982606b-49c8-4d5b-ba79-bd0fdd2600b6",
+"name": "Parse Umami data1",
+"type": "n8n-nodes-base.code",
+"position": [
+1560,
+260
+],
+"parameters": {
+"jsCode": "// Get input data\nconst data = $input.all();\n\n// Create URL-encoded string from the data\nconst encodedData = encodeURIComponent(JSON.stringify(data));\n\n// Return the encoded data\nreturn {\n json: {\n thisWeek: encodedData\n }\n};"
+},
+"typeVersion": 2
+},
+{
+"id": "f3734045-1318-4234-a3ac-61b766124609",
+"name": "Get page view data from Umami",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1760,
+260
+],
+"parameters": {
+"url": "=https://umami.rumjahn.synology.me/api/websites/f375d28c-1949-4597-8871-f1b942e3aa24/metrics?startAt={{Date.now() - (14 * 24 * 60 * 60 * 1000)}}&endAt={{Date.now() - (7 * 24 * 60 * 60 * 1000)}}&type=url&tz=America/Los_Angeles",
+"options": {},
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "FKsKXvQUlaX5qt9n",
+"name": "Header Auth account 3"
+}
+},
+"typeVersion": 4
+},
+{
+"id": "a0153ab0-3eaf-4f97-a2dc-ab63d45a9187",
+"name": "Parse Umami",
+"type": "n8n-nodes-base.code",
+"position": [
+1920,
+260
+],
+"parameters": {
+"jsCode": "// Get input data\nconst data = $input.all();\n\n// Create URL-encoded string from the data\nconst encodedData = encodeURIComponent(JSON.stringify(data));\n\n// Return the encoded data\nreturn {\n json: {\n lastweek: encodedData\n }\n};"
+},
+"typeVersion": 2
+},
+{
+"id": "c2d3d396-09fa-4800-b56d-40ed7592cd3c",
+"name": "Send data to A.I.1",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2180,
+260
+],
+"parameters": {
+"url": "https://openrouter.ai/api/v1/chat/completions",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"model\": \"meta-llama/llama-3.1-70b-instruct:free\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"You are an SEO expert. Here is data from Umami analytics of Pennibnotes.com. Where X is URL and Y is number of visitors. Compare the data from this week to last week. Present the data in a table using markdown and offer 5 improvement suggestions. This week:{{ $('Parse Umami data1').first().json.thisWeek }} Lastweek:{{ $json.lastweek }}\"\n }\n ]\n}\n\n",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "WY7UkF14ksPKq3S8",
+"name": "Header Auth account 2"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "ce58a556-c05a-4395-88b0-3edecbad80e5",
+"name": "Save data to Baserow",
+"type": "n8n-nodes-base.baserow",
+"position": [
+2520,
+260
+],
+"parameters": {
+"tableId": 607,
+"fieldsUi": {
+"fieldValues": [
+{
+"fieldId": 5870,
+"fieldValue": "={{ $json.choices[0].message.content }}"
+},
+{
+"fieldId": 5869,
+"fieldValue": "={{ $('Send data to A.I.').first().json.choices[0].message.content }}"
+},
+{
+"fieldId": 5868,
+"fieldValue": "={{ DateTime.now().toFormat('yyyy-MM-dd') }}"
+},
+{
+"fieldId": 5871,
+"fieldValue": "Name of your blog"
+}
+]
+},
+"operation": "create",
+"databaseId": 121
+},
+"credentials": {
+"baserowApi": {
+"id": "8w0zXhycIfCAgja3",
+"name": "Baserow account"
+}
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "e28e067d-9245-4879-9321-4d21925f951e",
+"connections": {
+"Parse Umami": {
+"main": [
+[
+{
+"node": "Send data to A.I.1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Umami data": {
+"main": [
+[
+{
+"node": "Send data to A.I.",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Schedule Trigger": {
+"main": [
+[
+{
+"node": "Get view stats from Umami",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Parse Umami data1": {
+"main": [
+[
+{
+"node": "Get page view data from Umami",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send data to A.I.": {
+"main": [
+[
+{
+"node": "Get page data from Umami",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Send data to A.I.1": {
+"main": [
+[
+{
+"node": "Save data to Baserow",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get page data from Umami": {
+"main": [
+[
+{
+"node": "Parse Umami data1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get view stats from Umami": {
+"main": [
+[
+{
+"node": "Parse Umami data",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get page view data from Umami": {
+"main": [
+[
+{
+"node": "Parse Umami",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get view stats from Umami",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.json b/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.json
deleted file mode 100644
index 38be286..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.json
+++ /dev/null
@@ -1,1272 +0,0 @@
-{
- "meta": {
- "instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
- },
- "nodes": [
- {
- "id": "0404384b-10b6-4666-84a4-8870db30c607",
- "name": "Embeddings OpenAI",
- "type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
- "position": [
- 1220,
- 280
- ],
- "parameters": {
- "model": "text-embedding-3-small",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "a6741f04-5a5b-47a9-ac08-eb562f9f6052",
- "name": "Default Data Loader",
- "type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
- "position": [
- 1340,
- 280
- ],
- "parameters": {
- "options": {
- "metadata": {
- "metadataValues": [
- {
- "name": "question",
- "value": "={{ $json.question }}"
- },
- {
- "name": "participant",
- "value": "={{ $json.participant }}"
- },
- {
- "name": "survey",
- "value": "={{ $('Get Survey Results').params.documentId.cachedResultName }}"
- }
- ]
- }
- },
- "jsonData": "={{ $json.answer }}",
- "jsonMode": "expressionData"
- },
- "typeVersion": 1
- },
- {
- "id": "7663c3dd-f713-4034-bef6-0c000285f54f",
- "name": "Convert to Question Answer Pairs",
- "type": "n8n-nodes-base.set",
- "position": [
- 720,
- 160
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "6b593ffb-ffbd-4cf5-a508-cd4f2a6d1004",
- "name": "data",
- "type": "array",
- "value": "={{\n Object.keys($json)\n .filter(key => !['row_number', 'Participant'].includes(key))\n .map(key => ({ question: key, answer: $json[key], participant: $json.Participant }))\n}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "84873f0c-81ce-442f-a33c-d7c6c2efa11b",
- "name": "Recursive Character Text Splitter",
- "type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
- "position": [
- 1340,
- 420
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "da9a8ee8-5e3f-49db-8d1f-26a61ca82344",
- "name": "Get Survey Results",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 540,
- 160
- ],
- "parameters": {
- "options": {},
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50/edit#gid=0",
- "cachedResultName": "Sheet1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50/edit?usp=drivesdk",
- "cachedResultName": "Remote Working Survey Responses"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.4
- },
- {
- "id": "4bad90b2-eefe-49c8-8caa-41cd4cb5e60f",
- "name": "Get Survey Headers",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 740,
- 940
- ],
- "parameters": {
- "options": {
- "dataLocationOnSheet": {
- "values": {
- "range": "A1:Z2",
- "rangeDefinition": "specifyRangeA1"
- }
- }
- },
- "sheetName": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Execute Workflow Trigger').first().json.sheetName }}"
- },
- "documentId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.4
- },
- {
- "id": "47c64994-9d1f-42ca-a849-3eeab5335b66",
- "name": "Extract Questions",
- "type": "n8n-nodes-base.set",
- "position": [
- 940,
- 940
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "d655b165-dfa2-46cb-bc27-140399bc4227",
- "name": "question",
- "type": "array",
- "value": "={{\n Object.keys($('Get Survey Headers').item.json)\n .filter(key => key.includes('?'))\n}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "c237d523-b290-41ca-b323-4cc7c7f6ff37",
- "name": "Questions to List",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 940,
- 1120
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "question"
- },
- "typeVersion": 1
- },
- {
- "id": "7f44a770-4c5d-4404-ae95-d9dee8348380",
- "name": "Find All Answers",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1460,
- 1120
- ],
- "parameters": {
- "url": "=http://qdrant:6333/collections/{{ $('Set Variables').item.json.collectionName }}/points/scroll",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"limit\": 500,\n \"filter\":{\n \"must\": [\n {\n \"key\": \"metadata.question\",\n \"match\": { \"value\": \"{{ $('For Each Question...').item.json.question }}\" }\n },\n {\n \"key\": \"metadata.survey\",\n \"match\": { \"value\": \"{{ $('Set Variables').item.json.surveyName }}\" }\n }\n ]\n },\n \"with_vector\":true\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "NyinAS3Pgfik66w5",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "2b6dc317-f8f3-4201-a9e1-d35ee578e79e",
- "name": "Get Payload of Points",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2380,
- 800
- ],
- "parameters": {
- "url": "=http://qdrant:6333/collections/{{ $('Set Variables').first().json.collectionName }}/points",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n {\n \"ids\": $json.points,\n \"with_payload\": true\n }\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "NyinAS3Pgfik66w5",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "d4a37d97-975a-4243-a7ea-81b3e30558a5",
- "name": "Clusters To List",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 2180,
- 800
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "output"
- },
- "typeVersion": 1
- },
- {
- "id": "c78f1bf6-8390-48ee-88f4-7d1a893a8ade",
- "name": "Set Variables",
- "type": "n8n-nodes-base.set",
- "position": [
- 200,
- 1060
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "b77c94a0-d865-4bd6-b078-a09b2ddb2a99",
- "name": "collectionName",
- "type": "string",
- "value": "ux_survey_insights"
- },
- {
- "id": "7b0a4d14-b5f9-4597-84c0-8cfdb363c3d3",
- "name": "surveyName",
- "type": "string",
- "value": "={{ $json.properties.title }}"
- },
- {
- "id": "45434b3b-3b74-4262-82e0-7ed02155caad",
- "name": "insightsSheetName",
- "type": "string",
- "value": "=Insights-{{ $now.format('yyyyMMdd') }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "fbb1f3c3-06ad-44b5-b020-6fc3c8eda7c4",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2560,
- 980
- ],
- "parameters": {
- "model": "gpt-4o-mini",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "8gccIjcuf3gvaoEr",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "83d3b413-a661-4c4c-9b8d-6ee395a15348",
- "name": "Prep Output For Export",
- "type": "n8n-nodes-base.set",
- "position": [
- 3160,
- 1300
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "={{ {\n ...$json.output,\n \"Number of Response\": $('Get Payload of Points').item.json.result.length,\n \"Participant IDs\": $('Get Payload of Points').item.json.result.map(item =>\n item.payload.metadata.participant\n ).join(','),\n \"Raw Responses\": $('Get Payload of Points').item.json.result.map(item =>\n `Participant ${item.payload.metadata.participant},${item.payload.content.replaceAll('\"', '\\\"')}`\n ).join('\\n')\n} }}\n"
- },
- "typeVersion": 3.4
- },
- {
- "id": "14784dff-a8ea-4b6b-8379-b0c9051a8f98",
- "name": "Export To Sheets",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 3360,
- 1300
- ],
- "parameters": {
- "columns": {
- "value": {},
- "schema": [
- {
- "id": "What is your name?",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "What is your name?",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "The responses indicate that two participants have the same name, 'Kwame Nkosi', which suggests a commonality in names or cultural naming traditions among the respondents. This could highlight the importance of understanding cultural context in survey responses.",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "The responses indicate that two participants have the same name, 'Kwame Nkosi', which suggests a commonality in names or cultural naming traditions among the respondents. This could highlight the importance of understanding cultural context in survey responses.",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "neutral",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "neutral",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "3",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "3",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "77,17,54",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "77,17,54",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "Participant 77,Kwame Nkosi\nParticipant 17,Kwame Nkosi\nParticipant 54,Kwame Nkansah",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "Participant 77,Kwame Nkosi\nParticipant 17,Kwame Nkosi\nParticipant 54,Kwame Nkansah",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "autoMapInputData",
- "matchingColumns": []
- },
- "options": {},
- "operation": "append",
- "sheetName": {
- "__rl": true,
- "mode": "name",
- "value": "={{ $('Set Variables').first().json.insightsSheetName }}"
- },
- "documentId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.4
- },
- {
- "id": "779b9411-db3e-44f3-ad2a-c9d40a70580d",
- "name": "Export To Sheets1",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 2360,
- 1300
- ],
- "parameters": {
- "columns": {
- "value": {},
- "schema": [],
- "mappingMode": "autoMapInputData",
- "matchingColumns": []
- },
- "options": {},
- "operation": "append",
- "sheetName": {
- "__rl": true,
- "mode": "name",
- "value": "={{ $('Set Variables').first().json.insightsSheetName }}"
- },
- "documentId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.4
- },
- {
- "id": "a31ab677-f57c-4b78-a290-d4a913ed4f8e",
- "name": "For Each Question...",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- 1280,
- 940
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "dcfaf927-6ecd-4ebe-aee0-5fb3367b2725",
- "name": "Trigger Insights",
- "type": "n8n-nodes-base.executeWorkflow",
- "position": [
- 1980,
- 160
- ],
- "parameters": {
- "options": {},
- "workflowId": "={{ $workflow.id }}"
- },
- "typeVersion": 1
- },
- {
- "id": "2579adf0-9c00-4b87-b53e-740044577ab0",
- "name": "Prep Values For Trigger",
- "type": "n8n-nodes-base.set",
- "position": [
- 1800,
- 160
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "24dd90ad-390f-444e-ba6c-8c06a41e836e",
- "name": "sheetID",
- "type": "string",
- "value": "={{ $('Get Survey Results').params.documentId.value }}"
- },
- {
- "id": "90199bbb-3938-411c-a7a8-faa7ccba6059",
- "name": "sheetName",
- "type": "string",
- "value": "={{ $('Get Survey Results').params.sheetName.value }}"
- }
- ]
- }
- },
- "executeOnce": true,
- "typeVersion": 3.4
- },
- {
- "id": "9b29e850-b9d0-4358-af62-92c20ab3b088",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- 20,
- 900
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "70a0dcec-9f74-4af2-bd64-0ab762a77e51",
- "name": "Create Insights Sheet",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- 420,
- 900
- ],
- "parameters": {
- "title": "={{ $('Set Variables').first().json.insightsSheetName }}",
- "options": {},
- "operation": "create",
- "documentId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.4,
- "alwaysOutputData": true
- },
- {
- "id": "f31400fb-dd7a-4c62-90ec-e9d78bbaa5e8",
- "name": "Prep Values For Export",
- "type": "n8n-nodes-base.set",
- "position": [
- 2180,
- 1300
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "={\n \"Question\": \"{{ $('For Each Question...').item.json.question }}\",\n \"Insight\": \"No Insight Found\"\n}\n"
- },
- "typeVersion": 3.4
- },
- {
- "id": "506c20df-5109-422c-8c9e-0eb50fbd3ff9",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 459.27570452141345,
- -42.168106366729035
- ],
- "parameters": {
- "color": 7,
- "width": 617.2130261221611,
- "height": 420.7389587470384,
- "content": "## Step 1. Import Survey Responses\n[Read more about Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets)\n\nOur approach requires to import all participant responses as vectors with metadata linking them to the questions being answered. To do this, we'll generate questiona and answer pairs from the survey."
- },
- "typeVersion": 1
- },
- {
- "id": "bddcafa8-6f54-4829-93c9-37bbb9e7edf3",
- "name": "QA Pairs to List",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 900,
- 160
- ],
- "parameters": {
- "options": {},
- "fieldToSplitOut": "data"
- },
- "typeVersion": 1
- },
- {
- "id": "8d6e6bf6-c94c-43cb-a29e-5d10207cb8bd",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1100,
- -102.05898437632061
- ],
- "parameters": {
- "color": 7,
- "width": 563.8350682199533,
- "height": 678.1641960508446,
- "content": "## Step 2. Vectorize Each Response Into Qdrant\n[Read more about using Qdrant](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreqdrant)\n\nSpecial attention is given to how metadata is captured as it becomes key to this workflow is being able to retrieve subsets of the data for analysis."
- },
- "typeVersion": 1
- },
- {
- "id": "613d4a32-a87a-423e-a1d1-ee23db0de6d1",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1680,
- -30.440883940004255
- ],
- "parameters": {
- "color": 7,
- "width": 519.6419932444072,
- "height": 429.11782776909047,
- "content": "## Step 3. Trigger Insights SubWorkflow\n[Learn more about Workflow Triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow)\n\nA subworkflow is used to trigger the analysis for the survey. This separation is optional but used here to better demonstrate the two part process."
- },
- "typeVersion": 1
- },
- {
- "id": "1e858e4a-b91b-4411-8e2a-6eb76647b796",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -57.47778952966382,
- 710.393394209128
- ],
- "parameters": {
- "color": 7,
- "width": 668.3083616841852,
- "height": 528.2386658883073,
- "content": "## Step 4. Create Insights Sheet\n[Learn more about Workflow Triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger)\n\nTo capture the generated insights, we'll create a new unique sheet within the survey spreadsheet. This is optional and you may want to capture in other datastores depending on your needs."
- },
- "typeVersion": 1
- },
- {
- "id": "9170c566-07d3-49dc-aafb-2dbe79940d2c",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 640,
- 683.5153164275844
- ],
- "parameters": {
- "color": 7,
- "width": 536.9288458983389,
- "height": 622.1362463986454,
- "content": "## Step 5. Get List Of Questions From Survey\n[Read more about using Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets)\n\nNext we'll fetch the survey for metadata and questions, splitting them into separate workflow items. Our intention is to process each question end-to-end before moving to the next. This approach is a little \"safer\" in the scenario where an interruption occurs we won't lose all our work."
- },
- "typeVersion": 1
- },
- {
- "id": "8488df77-055d-41cc-94f1-92ac5d54ef10",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1200,
- 673.291535602609
- ],
- "parameters": {
- "color": 7,
- "width": 823.147012265536,
- "height": 868.2579789328703,
- "content": "## Step 6. Find Groups of Similar Answers For Each Question\n[Learn more about using the Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code/)\n\nGiving all the responses to an LLM to analyse is the common but naive approach; the summarisation is usually too high level for real insights and loses a lot of detail such as the number and identity of respondants. What we want to do instead is find and group popular answers for each question to ensure all perspectives are considered.\n\nOur approach does this by mapping our answer vectors to a 2D grid and then identifying where the vector points are \"clustered\"; where a group of points are within close proximity to each other."
- },
- "typeVersion": 1
- },
- {
- "id": "f4748b6d-5bd8-48cf-942f-3a0dc681078d",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2060,
- 1180
- ],
- "parameters": {
- "color": 7,
- "width": 536.9288458983389,
- "height": 359.90385684071794,
- "content": "## Step 7b. Skip If No Clusters Found\nWhere no clusters were found, it means the answers were unique enough to not show any pattern. eg. \"What's you name?\""
- },
- "typeVersion": 1
- },
- {
- "id": "d55d6a47-da8c-46ae-bd10-0eb671dcd121",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2060,
- 611.6915003841909
- ],
- "parameters": {
- "color": 7,
- "width": 871.451300407926,
- "height": 541.1135860445843,
- "content": "## Step 7a. Summarise the Top Groups of Similar Answers\n[Read more about using the Information Extractor Node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.information-extractor)\n\nEach discovered cluster will return a reference vector which is used to fetch all related answers in the group.\nThe group is then sent to the LLM to summarise as well as assign a sentiment score."
- },
- "typeVersion": 1
- },
- {
- "id": "e5d5f88f-5832-43fc-a5b9-f747d08e7e77",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2620,
- 1180
- ],
- "parameters": {
- "color": 7,
- "width": 924.2798021207429,
- "height": 363.07347551845976,
- "content": "## Step 8. Write To Insights Sheet\nFinally, our completed insights to appended to\nthe Insights Sheet we created earlier in the workflow."
- },
- "typeVersion": 1
- },
- {
- "id": "49ac1504-7b43-4fa1-b4ce-15c7a53c9018",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 460,
- 400
- ],
- "parameters": {
- "color": 5,
- "width": 323.2987132716669,
- "height": 80,
- "content": "### Run this once! \nIf for any reason you need to run more than once, be sure to clear the existing data first."
- },
- "typeVersion": 1
- },
- {
- "id": "450f89c5-ef0f-4bf8-8db9-6347247c7f4d",
- "name": "Has Clusters?",
- "type": "n8n-nodes-base.if",
- "position": [
- 1820,
- 1120
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "40b6bb62-a2d6-4422-8fbb-7ae49898bad9",
- "operator": {
- "type": "array",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $json.output }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2
- },
- {
- "id": "1652a108-8fb8-4229-a76d-abf9fbcff626",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 20,
- -400
- ],
- "parameters": {
- "width": 400.381109509268,
- "height": 679.5610243514676,
- "content": "## Try It Out!\n\n### This workflow generates highly-detailed insights from survey responses. Works best when dealing with a large number of participants.\n\n* Import survey responses and vectorise in Qdrant vectorstore.\n* Identify clusters of popular responses to questions using K-means clustering algorithm. \n* Each valid cluster is analysed and summarised by LLM.\n* Export LLM response and cluster results back into sheet.\n\nCheck out the reference google sheet here: https://docs.google.com/spreadsheets/d/e/2PACX-1vT6m8XH8JWJTUAfwojc68NAUGC7q0lO7iV738J7aO5fuVjiVzdTRRPkMmT1C4N8TwejaiT0XrmF1Q48/pubhtml\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
- },
- "typeVersion": 1
- },
- {
- "id": "6eef981e-b2ce-433c-b71f-78be64812a56",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1260,
- 1340
- ],
- "parameters": {
- "color": 5,
- "width": 323.2987132716669,
- "height": 110.05160146874424,
- "content": "### First Time Running?\nThere is a slight delay on first run because the code node has to download the required packages."
- },
- "typeVersion": 1
- },
- {
- "id": "fa0c14be-03f4-4ed2-bd60-e93817382ded",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- 240,
- 100
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "30323019-59ba-4a19-a46e-196d469f097d",
- "name": "Get Sheet Details",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 200,
- 900
- ],
- "parameters": {
- "url": "=https://sheets.googleapis.com/v4/spreadsheets/{{ $json.sheetID }}",
- "options": {},
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "googleSheetsOAuth2Api"
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "6ced8012-1dd3-4da3-8c27-e4f4dfc959f6",
- "name": "Only Clusters With 3+ points",
- "type": "n8n-nodes-base.filter",
- "position": [
- 2180,
- 960
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "328f806c-0792-4d90-9bee-a1e10049e78f",
- "operator": {
- "type": "array",
- "operation": "lengthGt",
- "rightType": "number"
- },
- "leftValue": "={{ $json.points }}",
- "rightValue": 2
- }
- ]
- }
- },
- "typeVersion": 2
- },
- {
- "id": "8ae81a55-75e2-40a3-bef6-0935ff08128f",
- "name": "Apply K-means Clustering Algorithm",
- "type": "n8n-nodes-base.code",
- "position": [
- 1640,
- 1120
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "import numpy as np\nfrom sklearn.cluster import KMeans\n\n# get vectors for all answers\npoint_ids = [item.id for item in _input.first().json.result.points]\nvectors = [item.vector.to_py() for item in _input.first().json.result.points]\nvectors_array = np.array(vectors)\n\n# apply k-means clustering where n_clusters = 10\n# this is a max and we'll discard some of these clusters later\nkmeans = KMeans(n_clusters=min(len(vectors), 10), random_state=42).fit(vectors_array)\nlabels = kmeans.labels_\nunique_labels = set(labels)\n\n# Extract and print points in each cluster\nclusters = {}\nfor label in set(labels):\n clusters[label] = vectors_array[labels == label]\n\n# return Qdrant point ids for each cluster\n# we'll use these ids to fetch the payloads from the vector store.\noutput = []\nfor cluster_id, cluster_points in clusters.items():\n points = [point_ids[i] for i in range(len(labels)) if labels[i] == cluster_id]\n output.append({\n \"id\": f\"Cluster {cluster_id}\",\n \"total\": len(cluster_points),\n \"points\": points\n })\n\nreturn {\"json\": {\"output\": output } }"
- },
- "typeVersion": 2
- },
- {
- "id": "cbb42384-d46b-471f-a7d8-27e3de042492",
- "name": "Qdrant Vector Store",
- "type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
- "position": [
- 1220,
- 100
- ],
- "parameters": {
- "mode": "insert",
- "options": {},
- "qdrantCollection": {
- "__rl": true,
- "mode": "list",
- "value": "ux_survey_insights",
- "cachedResultName": "ux_survey_insights"
- }
- },
- "credentials": {
- "qdrantApi": {
- "id": "NyinAS3Pgfik66w5",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "17584901-15d6-421f-ad69-3ba872276055",
- "name": "Survey Insights Agent",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 2580,
- 800
- ],
- "parameters": {
- "text": "=The {{ $json.result.length }} participant responses were:\n{{\n$json.result.map(item =>\n`* Participant ${item.payload.metadata.participant}: \"${item.payload.content.replaceAll('\"', '\\\"')}\"`\n).join('\\n')\n}}",
- "options": {
- "systemPromptTemplate": "=You help summarise a selection of participant responses to a specific question for a survey called \"{{ $json.result[0].payload.metadata.survey }}\".\nThe question asked was \"{{ $json.result[0].payload.metadata.question }}\".\nThe {{ $json.result.length }} participant responses were selected because their answers were similar in context.\n\nYour task is to: \n* summarise the given participant responses into a short paragraph. Provide an insight from this summary and what we could learn from the answers.\n* determine if the overall sentiment of all the listed responses to be either negative, mildy negative, neutral, mildy positive or positive."
- },
- "schemaType": "fromJson",
- "jsonSchemaExample": "{\n\t\"Question\": \"What do you enjoy most about working remotely, and why?\",\n\t\"Insight\": \"\",\n \"Sentiment\": \"Positive\"\n}"
- },
- "typeVersion": 1
- }
- ],
- "pinData": {},
- "connections": {
- "Has Clusters?": {
- "main": [
- [
- {
- "node": "Clusters To List",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Prep Values For Export",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set Variables": {
- "main": [
- [
- {
- "node": "Create Insights Sheet",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Clusters To List": {
- "main": [
- [
- {
- "node": "Only Clusters With 3+ points",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Export To Sheets": {
- "main": [
- [
- {
- "node": "For Each Question...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Find All Answers": {
- "main": [
- [
- {
- "node": "Apply K-means Clustering Algorithm",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "QA Pairs to List": {
- "main": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Embeddings OpenAI": {
- "ai_embedding": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "ai_embedding",
- "index": 0
- }
- ]
- ]
- },
- "Export To Sheets1": {
- "main": [
- [
- {
- "node": "For Each Question...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract Questions": {
- "main": [
- [
- {
- "node": "Questions to List",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Sheet Details": {
- "main": [
- [
- {
- "node": "Set Variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Survey Insights Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Questions to List": {
- "main": [
- [
- {
- "node": "For Each Question...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Survey Headers": {
- "main": [
- [
- {
- "node": "Extract Questions",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Survey Results": {
- "main": [
- [
- {
- "node": "Convert to Question Answer Pairs",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Default Data Loader": {
- "ai_document": [
- [
- {
- "node": "Qdrant Vector Store",
- "type": "ai_document",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant Vector Store": {
- "main": [
- [
- {
- "node": "Prep Values For Trigger",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "For Each Question...": {
- "main": [
- null,
- [
- {
- "node": "Find All Answers",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create Insights Sheet": {
- "main": [
- [
- {
- "node": "Get Survey Headers",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Payload of Points": {
- "main": [
- [
- {
- "node": "Survey Insights Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Survey Insights Agent": {
- "main": [
- [
- {
- "node": "Prep Output For Export",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prep Output For Export": {
- "main": [
- [
- {
- "node": "Export To Sheets",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prep Values For Export": {
- "main": [
- [
- {
- "node": "Export To Sheets1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prep Values For Trigger": {
- "main": [
- [
- {
- "node": "Trigger Insights",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Get Sheet Details",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Only Clusters With 3+ points": {
- "main": [
- [
- {
- "node": "Get Payload of Points",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Convert to Question Answer Pairs": {
- "main": [
- [
- {
- "node": "QA Pairs to List",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Recursive Character Text Splitter": {
- "ai_textSplitter": [
- [
- {
- "node": "Default Data Loader",
- "type": "ai_textSplitter",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get Survey Results",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Apply K-means Clustering Algorithm": {
- "main": [
- [
- {
- "node": "Has Clusters?",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.txt b/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.txt
new file mode 100644
index 0000000..cd59f1c
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Survey Insights with Qdrant, Python and Information Extractor.txt
@@ -0,0 +1,1272 @@
+{
+"meta": {
+"instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
+},
+"nodes": [
+{
+"id": "0404384b-10b6-4666-84a4-8870db30c607",
+"name": "Embeddings OpenAI",
+"type": "@n8n/n8n-nodes-langchain.embeddingsOpenAi",
+"position": [
+1220,
+280
+],
+"parameters": {
+"model": "text-embedding-3-small",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "a6741f04-5a5b-47a9-ac08-eb562f9f6052",
+"name": "Default Data Loader",
+"type": "@n8n/n8n-nodes-langchain.documentDefaultDataLoader",
+"position": [
+1340,
+280
+],
+"parameters": {
+"options": {
+"metadata": {
+"metadataValues": [
+{
+"name": "question",
+"value": "={{ $json.question }}"
+},
+{
+"name": "participant",
+"value": "={{ $json.participant }}"
+},
+{
+"name": "survey",
+"value": "={{ $('Get Survey Results').params.documentId.cachedResultName }}"
+}
+]
+}
+},
+"jsonData": "={{ $json.answer }}",
+"jsonMode": "expressionData"
+},
+"typeVersion": 1
+},
+{
+"id": "7663c3dd-f713-4034-bef6-0c000285f54f",
+"name": "Convert to Question Answer Pairs",
+"type": "n8n-nodes-base.set",
+"position": [
+720,
+160
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "6b593ffb-ffbd-4cf5-a508-cd4f2a6d1004",
+"name": "data",
+"type": "array",
+"value": "={{\n Object.keys($json)\n .filter(key => !['row_number', 'Participant'].includes(key))\n .map(key => ({ question: key, answer: $json[key], participant: $json.Participant }))\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "84873f0c-81ce-442f-a33c-d7c6c2efa11b",
+"name": "Recursive Character Text Splitter",
+"type": "@n8n/n8n-nodes-langchain.textSplitterRecursiveCharacterTextSplitter",
+"position": [
+1340,
+420
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 1
+},
+{
+"id": "da9a8ee8-5e3f-49db-8d1f-26a61ca82344",
+"name": "Get Survey Results",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+540,
+160
+],
+"parameters": {
+"options": {},
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50/edit#gid=0",
+"cachedResultName": "Sheet1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1-168Vm-1kCeHkqGLAs6odha4DhPE93njfHlYIviKE50/edit?usp=drivesdk",
+"cachedResultName": "Remote Working Survey Responses"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.4
+},
+{
+"id": "4bad90b2-eefe-49c8-8caa-41cd4cb5e60f",
+"name": "Get Survey Headers",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+740,
+940
+],
+"parameters": {
+"options": {
+"dataLocationOnSheet": {
+"values": {
+"range": "A1:Z2",
+"rangeDefinition": "specifyRangeA1"
+}
+}
+},
+"sheetName": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Execute Workflow Trigger').first().json.sheetName }}"
+},
+"documentId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.4
+},
+{
+"id": "47c64994-9d1f-42ca-a849-3eeab5335b66",
+"name": "Extract Questions",
+"type": "n8n-nodes-base.set",
+"position": [
+940,
+940
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "d655b165-dfa2-46cb-bc27-140399bc4227",
+"name": "question",
+"type": "array",
+"value": "={{\n Object.keys($('Get Survey Headers').item.json)\n .filter(key => key.includes('?'))\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "c237d523-b290-41ca-b323-4cc7c7f6ff37",
+"name": "Questions to List",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+940,
+1120
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "question"
+},
+"typeVersion": 1
+},
+{
+"id": "7f44a770-4c5d-4404-ae95-d9dee8348380",
+"name": "Find All Answers",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1460,
+1120
+],
+"parameters": {
+"url": "=http://qdrant:6333/collections/{{ $('Set Variables').item.json.collectionName }}/points/scroll",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"limit\": 500,\n \"filter\":{\n \"must\": [\n {\n \"key\": \"metadata.question\",\n \"match\": { \"value\": \"{{ $('For Each Question...').item.json.question }}\" }\n },\n {\n \"key\": \"metadata.survey\",\n \"match\": { \"value\": \"{{ $('Set Variables').item.json.surveyName }}\" }\n }\n ]\n },\n \"with_vector\":true\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "NyinAS3Pgfik66w5",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "2b6dc317-f8f3-4201-a9e1-d35ee578e79e",
+"name": "Get Payload of Points",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2380,
+800
+],
+"parameters": {
+"url": "=http://qdrant:6333/collections/{{ $('Set Variables').first().json.collectionName }}/points",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n {\n \"ids\": $json.points,\n \"with_payload\": true\n }\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "NyinAS3Pgfik66w5",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "d4a37d97-975a-4243-a7ea-81b3e30558a5",
+"name": "Clusters To List",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+2180,
+800
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "output"
+},
+"typeVersion": 1
+},
+{
+"id": "c78f1bf6-8390-48ee-88f4-7d1a893a8ade",
+"name": "Set Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+200,
+1060
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "b77c94a0-d865-4bd6-b078-a09b2ddb2a99",
+"name": "collectionName",
+"type": "string",
+"value": "ux_survey_insights"
+},
+{
+"id": "7b0a4d14-b5f9-4597-84c0-8cfdb363c3d3",
+"name": "surveyName",
+"type": "string",
+"value": "={{ $json.properties.title }}"
+},
+{
+"id": "45434b3b-3b74-4262-82e0-7ed02155caad",
+"name": "insightsSheetName",
+"type": "string",
+"value": "=Insights-{{ $now.format('yyyyMMdd') }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "fbb1f3c3-06ad-44b5-b020-6fc3c8eda7c4",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+2560,
+980
+],
+"parameters": {
+"model": "gpt-4o-mini",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "8gccIjcuf3gvaoEr",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "83d3b413-a661-4c4c-9b8d-6ee395a15348",
+"name": "Prep Output For Export",
+"type": "n8n-nodes-base.set",
+"position": [
+3160,
+1300
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "={{ {\n ...$json.output,\n \"Number of Response\": $('Get Payload of Points').item.json.result.length,\n \"Participant IDs\": $('Get Payload of Points').item.json.result.map(item =>\n item.payload.metadata.participant\n ).join(','),\n \"Raw Responses\": $('Get Payload of Points').item.json.result.map(item =>\n `Participant ${item.payload.metadata.participant},${item.payload.content.replaceAll('\"', '\\\"')}`\n ).join('\\n')\n} }}\n"
+},
+"typeVersion": 3.4
+},
+{
+"id": "14784dff-a8ea-4b6b-8379-b0c9051a8f98",
+"name": "Export To Sheets",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+3360,
+1300
+],
+"parameters": {
+"columns": {
+"value": {},
+"schema": [
+{
+"id": "What is your name?",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "What is your name?",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "The responses indicate that two participants have the same name, 'Kwame Nkosi', which suggests a commonality in names or cultural naming traditions among the respondents. This could highlight the importance of understanding cultural context in survey responses.",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "The responses indicate that two participants have the same name, 'Kwame Nkosi', which suggests a commonality in names or cultural naming traditions among the respondents. This could highlight the importance of understanding cultural context in survey responses.",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "neutral",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "neutral",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "3",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "3",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "77,17,54",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "77,17,54",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "Participant 77,Kwame Nkosi\nParticipant 17,Kwame Nkosi\nParticipant 54,Kwame Nkansah",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "Participant 77,Kwame Nkosi\nParticipant 17,Kwame Nkosi\nParticipant 54,Kwame Nkansah",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "autoMapInputData",
+"matchingColumns": []
+},
+"options": {},
+"operation": "append",
+"sheetName": {
+"__rl": true,
+"mode": "name",
+"value": "={{ $('Set Variables').first().json.insightsSheetName }}"
+},
+"documentId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.4
+},
+{
+"id": "779b9411-db3e-44f3-ad2a-c9d40a70580d",
+"name": "Export To Sheets1",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+2360,
+1300
+],
+"parameters": {
+"columns": {
+"value": {},
+"schema": [],
+"mappingMode": "autoMapInputData",
+"matchingColumns": []
+},
+"options": {},
+"operation": "append",
+"sheetName": {
+"__rl": true,
+"mode": "name",
+"value": "={{ $('Set Variables').first().json.insightsSheetName }}"
+},
+"documentId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.4
+},
+{
+"id": "a31ab677-f57c-4b78-a290-d4a913ed4f8e",
+"name": "For Each Question...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+1280,
+940
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "dcfaf927-6ecd-4ebe-aee0-5fb3367b2725",
+"name": "Trigger Insights",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+1980,
+160
+],
+"parameters": {
+"options": {},
+"workflowId": "={{ $workflow.id }}"
+},
+"typeVersion": 1
+},
+{
+"id": "2579adf0-9c00-4b87-b53e-740044577ab0",
+"name": "Prep Values For Trigger",
+"type": "n8n-nodes-base.set",
+"position": [
+1800,
+160
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "24dd90ad-390f-444e-ba6c-8c06a41e836e",
+"name": "sheetID",
+"type": "string",
+"value": "={{ $('Get Survey Results').params.documentId.value }}"
+},
+{
+"id": "90199bbb-3938-411c-a7a8-faa7ccba6059",
+"name": "sheetName",
+"type": "string",
+"value": "={{ $('Get Survey Results').params.sheetName.value }}"
+}
+]
+}
+},
+"executeOnce": true,
+"typeVersion": 3.4
+},
+{
+"id": "9b29e850-b9d0-4358-af62-92c20ab3b088",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+20,
+900
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "70a0dcec-9f74-4af2-bd64-0ab762a77e51",
+"name": "Create Insights Sheet",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+420,
+900
+],
+"parameters": {
+"title": "={{ $('Set Variables').first().json.insightsSheetName }}",
+"options": {},
+"operation": "create",
+"documentId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $('Execute Workflow Trigger').first().json.sheetID }}"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.4,
+"alwaysOutputData": true
+},
+{
+"id": "f31400fb-dd7a-4c62-90ec-e9d78bbaa5e8",
+"name": "Prep Values For Export",
+"type": "n8n-nodes-base.set",
+"position": [
+2180,
+1300
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "={\n \"Question\": \"{{ $('For Each Question...').item.json.question }}\",\n \"Insight\": \"No Insight Found\"\n}\n"
+},
+"typeVersion": 3.4
+},
+{
+"id": "506c20df-5109-422c-8c9e-0eb50fbd3ff9",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+459.27570452141345,
+-42.168106366729035
+],
+"parameters": {
+"color": 7,
+"width": 617.2130261221611,
+"height": 420.7389587470384,
+"content": "## Step 1. Import Survey Responses\n[Read more about Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets)\n\nOur approach requires to import all participant responses as vectors with metadata linking them to the questions being answered. To do this, we'll generate questiona and answer pairs from the survey."
+},
+"typeVersion": 1
+},
+{
+"id": "bddcafa8-6f54-4829-93c9-37bbb9e7edf3",
+"name": "QA Pairs to List",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+900,
+160
+],
+"parameters": {
+"options": {},
+"fieldToSplitOut": "data"
+},
+"typeVersion": 1
+},
+{
+"id": "8d6e6bf6-c94c-43cb-a29e-5d10207cb8bd",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1100,
+-102.05898437632061
+],
+"parameters": {
+"color": 7,
+"width": 563.8350682199533,
+"height": 678.1641960508446,
+"content": "## Step 2. Vectorize Each Response Into Qdrant\n[Read more about using Qdrant](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreqdrant)\n\nSpecial attention is given to how metadata is captured as it becomes key to this workflow is being able to retrieve subsets of the data for analysis."
+},
+"typeVersion": 1
+},
+{
+"id": "613d4a32-a87a-423e-a1d1-ee23db0de6d1",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1680,
+-30.440883940004255
+],
+"parameters": {
+"color": 7,
+"width": 519.6419932444072,
+"height": 429.11782776909047,
+"content": "## Step 3. Trigger Insights SubWorkflow\n[Learn more about Workflow Triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflow)\n\nA subworkflow is used to trigger the analysis for the survey. This separation is optional but used here to better demonstrate the two part process."
+},
+"typeVersion": 1
+},
+{
+"id": "1e858e4a-b91b-4411-8e2a-6eb76647b796",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-57.47778952966382,
+710.393394209128
+],
+"parameters": {
+"color": 7,
+"width": 668.3083616841852,
+"height": 528.2386658883073,
+"content": "## Step 4. Create Insights Sheet\n[Learn more about Workflow Triggers](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.executeworkflowtrigger)\n\nTo capture the generated insights, we'll create a new unique sheet within the survey spreadsheet. This is optional and you may want to capture in other datastores depending on your needs."
+},
+"typeVersion": 1
+},
+{
+"id": "9170c566-07d3-49dc-aafb-2dbe79940d2c",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+640,
+683.5153164275844
+],
+"parameters": {
+"color": 7,
+"width": 536.9288458983389,
+"height": 622.1362463986454,
+"content": "## Step 5. Get List Of Questions From Survey\n[Read more about using Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets)\n\nNext we'll fetch the survey for metadata and questions, splitting them into separate workflow items. Our intention is to process each question end-to-end before moving to the next. This approach is a little \"safer\" in the scenario where an interruption occurs we won't lose all our work."
+},
+"typeVersion": 1
+},
+{
+"id": "8488df77-055d-41cc-94f1-92ac5d54ef10",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1200,
+673.291535602609
+],
+"parameters": {
+"color": 7,
+"width": 823.147012265536,
+"height": 868.2579789328703,
+"content": "## Step 6. Find Groups of Similar Answers For Each Question\n[Learn more about using the Code Node](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.code/)\n\nGiving all the responses to an LLM to analyse is the common but naive approach; the summarisation is usually too high level for real insights and loses a lot of detail such as the number and identity of respondants. What we want to do instead is find and group popular answers for each question to ensure all perspectives are considered.\n\nOur approach does this by mapping our answer vectors to a 2D grid and then identifying where the vector points are \"clustered\"; where a group of points are within close proximity to each other."
+},
+"typeVersion": 1
+},
+{
+"id": "f4748b6d-5bd8-48cf-942f-3a0dc681078d",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2060,
+1180
+],
+"parameters": {
+"color": 7,
+"width": 536.9288458983389,
+"height": 359.90385684071794,
+"content": "## Step 7b. Skip If No Clusters Found\nWhere no clusters were found, it means the answers were unique enough to not show any pattern. eg. \"What's you name?\""
+},
+"typeVersion": 1
+},
+{
+"id": "d55d6a47-da8c-46ae-bd10-0eb671dcd121",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2060,
+611.6915003841909
+],
+"parameters": {
+"color": 7,
+"width": 871.451300407926,
+"height": 541.1135860445843,
+"content": "## Step 7a. Summarise the Top Groups of Similar Answers\n[Read more about using the Information Extractor Node](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.information-extractor)\n\nEach discovered cluster will return a reference vector which is used to fetch all related answers in the group.\nThe group is then sent to the LLM to summarise as well as assign a sentiment score."
+},
+"typeVersion": 1
+},
+{
+"id": "e5d5f88f-5832-43fc-a5b9-f747d08e7e77",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2620,
+1180
+],
+"parameters": {
+"color": 7,
+"width": 924.2798021207429,
+"height": 363.07347551845976,
+"content": "## Step 8. Write To Insights Sheet\nFinally, our completed insights to appended to\nthe Insights Sheet we created earlier in the workflow."
+},
+"typeVersion": 1
+},
+{
+"id": "49ac1504-7b43-4fa1-b4ce-15c7a53c9018",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+460,
+400
+],
+"parameters": {
+"color": 5,
+"width": 323.2987132716669,
+"height": 80,
+"content": "### Run this once! \nIf for any reason you need to run more than once, be sure to clear the existing data first."
+},
+"typeVersion": 1
+},
+{
+"id": "450f89c5-ef0f-4bf8-8db9-6347247c7f4d",
+"name": "Has Clusters?",
+"type": "n8n-nodes-base.if",
+"position": [
+1820,
+1120
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "40b6bb62-a2d6-4422-8fbb-7ae49898bad9",
+"operator": {
+"type": "array",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $json.output }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2
+},
+{
+"id": "1652a108-8fb8-4229-a76d-abf9fbcff626",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+20,
+-400
+],
+"parameters": {
+"width": 400.381109509268,
+"height": 679.5610243514676,
+"content": "## Try It Out!\n\n### This workflow generates highly-detailed insights from survey responses. Works best when dealing with a large number of participants.\n\n* Import survey responses and vectorise in Qdrant vectorstore.\n* Identify clusters of popular responses to questions using K-means clustering algorithm. \n* Each valid cluster is analysed and summarised by LLM.\n* Export LLM response and cluster results back into sheet.\n\nCheck out the reference google sheet here: https://docs.google.com/spreadsheets/d/e/2PACX-1vT6m8XH8JWJTUAfwojc68NAUGC7q0lO7iV738J7aO5fuVjiVzdTRRPkMmT1C4N8TwejaiT0XrmF1Q48/pubhtml\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!"
+},
+"typeVersion": 1
+},
+{
+"id": "6eef981e-b2ce-433c-b71f-78be64812a56",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1260,
+1340
+],
+"parameters": {
+"color": 5,
+"width": 323.2987132716669,
+"height": 110.05160146874424,
+"content": "### First Time Running?\nThere is a slight delay on first run because the code node has to download the required packages."
+},
+"typeVersion": 1
+},
+{
+"id": "fa0c14be-03f4-4ed2-bd60-e93817382ded",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+240,
+100
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "30323019-59ba-4a19-a46e-196d469f097d",
+"name": "Get Sheet Details",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+200,
+900
+],
+"parameters": {
+"url": "=https://sheets.googleapis.com/v4/spreadsheets/{{ $json.sheetID }}",
+"options": {},
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "googleSheetsOAuth2Api"
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "6ced8012-1dd3-4da3-8c27-e4f4dfc959f6",
+"name": "Only Clusters With 3+ points",
+"type": "n8n-nodes-base.filter",
+"position": [
+2180,
+960
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "328f806c-0792-4d90-9bee-a1e10049e78f",
+"operator": {
+"type": "array",
+"operation": "lengthGt",
+"rightType": "number"
+},
+"leftValue": "={{ $json.points }}",
+"rightValue": 2
+}
+]
+}
+},
+"typeVersion": 2
+},
+{
+"id": "8ae81a55-75e2-40a3-bef6-0935ff08128f",
+"name": "Apply K-means Clustering Algorithm",
+"type": "n8n-nodes-base.code",
+"position": [
+1640,
+1120
+],
+"parameters": {
+"language": "python",
+"pythonCode": "import numpy as np\nfrom sklearn.cluster import KMeans\n\n# get vectors for all answers\npoint_ids = [item.id for item in _input.first().json.result.points]\nvectors = [item.vector.to_py() for item in _input.first().json.result.points]\nvectors_array = np.array(vectors)\n\n# apply k-means clustering where n_clusters = 10\n# this is a max and we'll discard some of these clusters later\nkmeans = KMeans(n_clusters=min(len(vectors), 10), random_state=42).fit(vectors_array)\nlabels = kmeans.labels_\nunique_labels = set(labels)\n\n# Extract and print points in each cluster\nclusters = {}\nfor label in set(labels):\n clusters[label] = vectors_array[labels == label]\n\n# return Qdrant point ids for each cluster\n# we'll use these ids to fetch the payloads from the vector store.\noutput = []\nfor cluster_id, cluster_points in clusters.items():\n points = [point_ids[i] for i in range(len(labels)) if labels[i] == cluster_id]\n output.append({\n \"id\": f\"Cluster {cluster_id}\",\n \"total\": len(cluster_points),\n \"points\": points\n })\n\nreturn {\"json\": {\"output\": output } }"
+},
+"typeVersion": 2
+},
+{
+"id": "cbb42384-d46b-471f-a7d8-27e3de042492",
+"name": "Qdrant Vector Store",
+"type": "@n8n/n8n-nodes-langchain.vectorStoreQdrant",
+"position": [
+1220,
+100
+],
+"parameters": {
+"mode": "insert",
+"options": {},
+"qdrantCollection": {
+"__rl": true,
+"mode": "list",
+"value": "ux_survey_insights",
+"cachedResultName": "ux_survey_insights"
+}
+},
+"credentials": {
+"qdrantApi": {
+"id": "NyinAS3Pgfik66w5",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "17584901-15d6-421f-ad69-3ba872276055",
+"name": "Survey Insights Agent",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+2580,
+800
+],
+"parameters": {
+"text": "=The {{ $json.result.length }} participant responses were:\n{{\n$json.result.map(item =>\n`* Participant ${item.payload.metadata.participant}: \"${item.payload.content.replaceAll('\"', '\\\"')}\"`\n).join('\\n')\n}}",
+"options": {
+"systemPromptTemplate": "=You help summarise a selection of participant responses to a specific question for a survey called \"{{ $json.result[0].payload.metadata.survey }}\".\nThe question asked was \"{{ $json.result[0].payload.metadata.question }}\".\nThe {{ $json.result.length }} participant responses were selected because their answers were similar in context.\n\nYour task is to: \n* summarise the given participant responses into a short paragraph. Provide an insight from this summary and what we could learn from the answers.\n* determine if the overall sentiment of all the listed responses to be either negative, mildy negative, neutral, mildy positive or positive."
+},
+"schemaType": "fromJson",
+"jsonSchemaExample": "{\n\t\"Question\": \"What do you enjoy most about working remotely, and why?\",\n\t\"Insight\": \"\",\n \"Sentiment\": \"Positive\"\n}"
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Has Clusters?": {
+"main": [
+[
+{
+"node": "Clusters To List",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Prep Values For Export",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set Variables": {
+"main": [
+[
+{
+"node": "Create Insights Sheet",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Clusters To List": {
+"main": [
+[
+{
+"node": "Only Clusters With 3+ points",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Export To Sheets": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Find All Answers": {
+"main": [
+[
+{
+"node": "Apply K-means Clustering Algorithm",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"QA Pairs to List": {
+"main": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Embeddings OpenAI": {
+"ai_embedding": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "ai_embedding",
+"index": 0
+}
+]
+]
+},
+"Export To Sheets1": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract Questions": {
+"main": [
+[
+{
+"node": "Questions to List",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Sheet Details": {
+"main": [
+[
+{
+"node": "Set Variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Survey Insights Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Questions to List": {
+"main": [
+[
+{
+"node": "For Each Question...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Survey Headers": {
+"main": [
+[
+{
+"node": "Extract Questions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Survey Results": {
+"main": [
+[
+{
+"node": "Convert to Question Answer Pairs",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Default Data Loader": {
+"ai_document": [
+[
+{
+"node": "Qdrant Vector Store",
+"type": "ai_document",
+"index": 0
+}
+]
+]
+},
+"Qdrant Vector Store": {
+"main": [
+[
+{
+"node": "Prep Values For Trigger",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Question...": {
+"main": [
+null,
+[
+{
+"node": "Find All Answers",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create Insights Sheet": {
+"main": [
+[
+{
+"node": "Get Survey Headers",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Payload of Points": {
+"main": [
+[
+{
+"node": "Survey Insights Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Survey Insights Agent": {
+"main": [
+[
+{
+"node": "Prep Output For Export",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prep Output For Export": {
+"main": [
+[
+{
+"node": "Export To Sheets",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prep Values For Export": {
+"main": [
+[
+{
+"node": "Export To Sheets1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prep Values For Trigger": {
+"main": [
+[
+{
+"node": "Trigger Insights",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Get Sheet Details",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Only Clusters With 3+ points": {
+"main": [
+[
+{
+"node": "Get Payload of Points",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert to Question Answer Pairs": {
+"main": [
+[
+{
+"node": "QA Pairs to List",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Recursive Character Text Splitter": {
+"ai_textSplitter": [
+[
+{
+"node": "Default Data Loader",
+"type": "ai_textSplitter",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get Survey Results",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Apply K-means Clustering Algorithm": {
+"main": [
+[
+{
+"node": "Has Clusters?",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.json b/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.json
deleted file mode 100644
index 7f14b7c..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.json
+++ /dev/null
@@ -1,1971 +0,0 @@
-{
- "id": "kZ3aL4r7xc96Q7lp",
- "meta": {
- "instanceId": "b8b2c0d20b02864cf66adc9cbefc86e9e56de0252b653d37ba6613341b5e0bef",
- "templateCredsSetupCompleted": true
- },
- "name": "Selenium Ultimate Scraper Workflow",
- "tags": [],
- "nodes": [
- {
- "id": "20d35d68-db49-4183-a913-85ad06c13912",
- "name": "Extract First Url Match",
- "type": "n8n-nodes-base.html",
- "position": [
- 1820,
- 540
- ],
- "parameters": {
- "options": {},
- "operation": "extractHtmlContent",
- "extractionValues": {
- "values": [
- {
- "key": "Url Find ",
- "attribute": "href",
- "cssSelector": "=a[href*=\"https://\"][href*=\"{{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}\"]\n",
- "returnArray": true,
- "returnValue": "attribute"
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "9167ea20-fc9c-4d75-bf4d-bb2016079dd0",
- "name": "OpenAI Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2060,
- 700
- ],
- "parameters": {
- "model": "gpt-4o",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "FmszNHDDVS32ud21",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "42a8646d-1b0b-4309-a87d-9c8aeb355a28",
- "name": "Clean Webdriver ",
- "type": "n8n-nodes-base.httpRequest",
- "notes": "Script to delete traces of selenium in the browser ",
- "position": [
- 3120,
- 560
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/execute/sync",
- "method": "POST",
- "options": {},
- "jsonBody": "{\n \"script\": \"Object.defineProperty(navigator, 'webdriver', { get: () => undefined }); window.navigator.chrome = { runtime: {} }; Object.defineProperty(navigator, 'languages', { get: () => ['en-US', 'en'] }); Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3, 4, 5] });\",\n \"args\": []\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "notesInFlow": false,
- "typeVersion": 4.2
- },
- {
- "id": "107dd8de-e341-4819-a493-94ed57fd0f33",
- "name": "Delete Session",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 5180,
- 920
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "8c7ec6bc-d417-48c2-a6f2-ecce27803671",
- "name": "Delete Session2",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 6740,
- -160
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "e43ecd94-b7f2-4f73-a9fa-b829de9e0296",
- "name": "If Block1",
- "type": "n8n-nodes-base.if",
- "position": [
- 6520,
- -20
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "e6e6e15d-1cfe-48be-8ea0-f112e9781c9d",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.content }}",
- "rightValue": "BLOCK"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "08e46f63-41b5-4606-8f2c-df9e96c9c34e",
- "name": "Delete Session3",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 6740,
- 60
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "b47d9b22-9a59-4c7a-8cba-9487f18207ee",
- "name": "Limit",
- "type": "n8n-nodes-base.limit",
- "position": [
- 5120,
- -100
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "541622f7-562b-4e8a-93e5-61e6e918ff52",
- "name": "Delete Session1",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 5180,
- 720
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "825be0d7-9dd3-4a2f-8c3d-fd405f59a5d6",
- "name": "Delete Session4",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 5780,
- 260
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "retryOnFail": false,
- "typeVersion": 4.2
- },
- {
- "id": "56f6f4f6-f737-4de8-bdfe-029546909677",
- "name": "Success with cookie",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 7260,
- 60
- ],
- "parameters": {
- "options": {
- "responseCode": 200
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "c6939773-e230-45e1-bf76-d0299c2c7066",
- "name": "Respond to Webhook2",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 6920,
- -160
- ],
- "parameters": {
- "options": {
- "responseCode": 200
- },
- "respondWith": "json",
- "responseBody": "{\n \"Success \": \"Request has been block by the targeted website\"\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "ea921f11-323f-4c79-8cc6-779b39498b05",
- "name": "Code",
- "type": "n8n-nodes-base.code",
- "position": [
- 4700,
- -100
- ],
- "parameters": {
- "jsCode": "// Rรฉcupรจre les donnรฉes du nลud Webhook (en remplaรงant \"Webhook\" par le nom du nลud Webhook dans votre workflow)\nconst webhookData = $node[\"Webhook\"].json;\n\n// Fonction pour convertir la valeur de sameSite\nfunction convertSameSite(value) {\n // Conversion spรฉcifique des valeurs de sameSite\n const conversionMap = {\n \"unspecified\": \"None\",\n \"lax\": \"Lax\",\n \"strict\": \"Strict\"\n };\n \n // Si la valeur existe dans le tableau de conversion, on la convertit\n if (value in conversionMap) {\n return conversionMap[value];\n }\n \n // Si la valeur est dรฉjร une des valeurs acceptรฉes par Selenium\n const allowedValues = [\"Strict\", \"Lax\", \"None\"];\n if (allowedValues.includes(value)) {\n return value;\n } else {\n // Si la valeur n'est pas reconnue, on la remplace par \"Lax\" (par dรฉfaut)\n return \"Lax\";\n }\n}\n\n// Vรฉrifiez et traitez les donnรฉes des cookies\nif (webhookData.body && webhookData.body.cookies) {\n let items = [];\n for (const cookieObject of webhookData.body.cookies) {\n if (cookieObject.cookie) {\n // Convertir la valeur de sameSite\n cookieObject.cookie.sameSite = convertSameSite(cookieObject.cookie.sameSite);\n \n // Ajouter le cookie ร la liste des items\n items.push({\n json: cookieObject.cookie\n });\n }\n }\n return items;\n}\n\n// Si les cookies ne sont pas trouvรฉs, renvoyer un tableau vide\nreturn [];\n"
- },
- "typeVersion": 2
- },
- {
- "id": "c3d77928-eefc-4903-9b4f-b14bd6f34e3c",
- "name": "Delete Session5",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 3940,
- 360
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "retryOnFail": false,
- "typeVersion": 4.2
- },
- {
- "id": "036cfce6-8082-4539-bb0e-980368679fe5",
- "name": "Error",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 4120,
- 360
- ],
- "parameters": {
- "options": {
- "responseCode": 404
- },
- "respondWith": "json",
- "responseBody": "{\n \"Error\": \"Cookies are note for the targeted url\"\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "09d6a99b-d8b3-40c9-b74a-14014e3647e2",
- "name": "Error1",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 6000,
- 260
- ],
- "parameters": {
- "options": {
- "responseCode": 500
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "0b1f3442-6b70-405f-b597-642e9c982b82",
- "name": "Error2",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 3060,
- 780
- ],
- "parameters": {
- "options": {
- "responseCode": 500
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "4d0112bb-cbfd-45c6-961a-964bd8f59cac",
- "name": "If",
- "type": "n8n-nodes-base.if",
- "position": [
- 3760,
- 200
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "1bffbc80-9913-46e7-a594-ebc26948c83b",
- "operator": {
- "type": "string",
- "operation": "contains"
- },
- "leftValue": "={{ $('Webhook').item.json.body.cookies[0].cookie.domain }}",
- "rightValue": "={{ $('Webhook').item.json.body.Url }}"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "58a50b80-df4c-4b6f-a682-72237f4dbdef",
- "name": "Inject Cookie",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 4900,
- -100
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/cookie",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"cookie\": {\n \"name\": \"{{ $json.name }}\",\n \"value\": \"{{ $json.value }}\",\n \"domain\": \"{{ $json.domain }}\",\n \"path\": \"{{ $json.path }}\",\n \"secure\": {{ $json.secure }},\n \"httpOnly\": {{ $json.httpOnly }},\n \"sameSite\": \"{{ $json.sameSite }}\",\n \"expirationDate\": {{ $json.expirationDate }}\n }\n}",
- "sendBody": true,
- "specifyBody": "json"
- },
- "typeVersion": 4.2
- },
- {
- "id": "39f7401b-b6b7-4f0c-9afc-8f144d394350",
- "name": "Respond to Webhook3",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 5400,
- 720
- ],
- "parameters": {
- "options": {
- "responseCode": 200
- },
- "respondWith": "json",
- "responseBody": "{\n \"Success \": \"Request has been block by the targeted website\"\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "80b107cc-2f6c-46f0-a597-e85594634492",
- "name": "Success",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 5740,
- 920
- ],
- "parameters": {
- "options": {
- "responseKey": "={{ $json.output }}",
- "responseCode": 200
- }
- },
- "typeVersion": 1.1
- },
- {
- "id": "94a97354-07d9-428e-989c-ef066f9b4d8a",
- "name": "Go on url",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 3900,
- 780
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"url\": \"{{ $('Webhook').item.json.body['Target Url'] }}\"\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- },
- {
- "id": "fd044cf3-594d-48af-bbd1-f2d9adedcbc1",
- "name": "Delete Session6",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 4360,
- 1200
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "retryOnFail": false,
- "typeVersion": 4.2
- },
- {
- "id": "7c28c3b6-1141-4609-8774-cb6b4d842b97",
- "name": "Error3",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 4520,
- 1200
- ],
- "parameters": {
- "options": {
- "responseCode": 500
- },
- "respondWith": "json",
- "responseBody": "{\n \"Error\": \"Page crash on the extracted url\"\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "52f78923-156f-4861-88ba-f0253c483bd9",
- "name": "Information Extractor",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 2040,
- 540
- ],
- "parameters": {
- "text": "={{ $json['Url Find '][1] }}{{ $json['Url Find '][2] }}{{ $json['Url Find '][3] }}",
- "options": {
- "systemPromptTemplate": "=You are an expert extraction algorithm.\nOnly extract relevant url from the unstructured urls array.\nA relevant url is a url whre you can find relevant information about this subject : {{ $('Edit Fields (For testing prupose )').item.json.Subject }}, on this domaine name : {{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}.\nIf you do not know the value of an attribute asked to extract, you need \\ attribute's value as NA."
- },
- "attributes": {
- "attributes": [
- {
- "name": "Good_url_for_etract_information",
- "required": true,
- "description": "=The url where I can extract relevant infroamtion on this subject : {{ $('Edit Fields (For testing prupose )').item.json.Subject }} on this domaine name : {{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "6ac249e2-a9d8-4590-b050-3a0a2472fa3c",
- "name": "Check if empty of NA",
- "type": "n8n-nodes-base.if",
- "position": [
- 2440,
- 540
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "or",
- "conditions": [
- {
- "id": "9470fb6c-e367-4af7-a697-275e724fe771",
- "operator": {
- "type": "string",
- "operation": "empty",
- "singleValue": true
- },
- "leftValue": "={{ $json.output.Good_url_for_etract_information }}",
- "rightValue": ""
- },
- {
- "id": "8518e9a9-5b0c-4699-97c5-d9b7b1943918",
- "operator": {
- "name": "filter.operator.equals",
- "type": "string",
- "operation": "equals"
- },
- "leftValue": "={{ $json.output.Good_url_for_etract_information }}",
- "rightValue": "NA"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "f380eff7-3d18-4791-9dac-8a88d3fdcc4f",
- "name": "If Block",
- "type": "n8n-nodes-base.if",
- "position": [
- 4960,
- 840
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "e6e6e15d-1cfe-48be-8ea0-f112e9781c9d",
- "operator": {
- "type": "string",
- "operation": "contains"
- },
- "leftValue": "={{ $json.content }}",
- "rightValue": "BLOCK"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "43382397-89b5-4b90-9016-49109ec04baf",
- "name": "Google search Query ",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1600,
- 540
- ],
- "parameters": {
- "url": "=https://www.google.com/search?q=site:{{ $json['Website Domaine'] }}+{{$json.Subject}}&oq=site&gs_lcrp=EgZjaHJvbWUqCAgAEEUYJxg7MggIABBFGCcYOzIICAEQRRgnGDsyBggCEEUYOzIRCAMQRRg5GEMYyQMYgAQYigUyBggEEEUYQDIGCAUQRRg9MgYIBhBFGD0yBggHEEUYPdIBCDEwNTRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "d34256af-1b43-4f64-853c-cf063b8c6b68",
- "name": "Create Selenium Session",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 2680,
- 640
- ],
- "parameters": {
- "url": "http://selenium_chrome:4444/wd/hub/session",
- "method": "POST",
- "options": {
- "timeout": 5000
- },
- "jsonBody": "{\n \"capabilities\": {\n \"alwaysMatch\": {\n \"browserName\": \"chrome\",\n \"goog:chromeOptions\": {\n \"args\": [ \n \"--disable-blink-features=AutomationControlled\",\n \"--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\"\n ]\n }\n }\n }\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- },
- {
- "id": "4f0f696c-9637-4c7d-82ae-1f5c36bb9cd1",
- "name": "Get ScreenShot 1",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 4420,
- 840
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "ba72c0cf-217a-4411-80f6-ca28ccdb0151",
- "name": "Refresh browser",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 5320,
- -100
- ],
- "parameters": {
- "url": "=http:///selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/refresh",
- "method": "POST",
- "options": {},
- "jsonBody": "{}",
- "sendBody": true,
- "specifyBody": "json"
- },
- "typeVersion": 4.2
- },
- {
- "id": "b6ba7068-399a-467d-ba58-7f47d650e2f1",
- "name": "Get ScreenShot ",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 5880,
- -20
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "792649be-0ee2-442f-bc21-d0c297cea227",
- "name": "Convert to File",
- "type": "n8n-nodes-base.convertToFile",
- "onError": "continueErrorOutput",
- "position": [
- 6160,
- -20
- ],
- "parameters": {
- "options": {},
- "operation": "toBinary",
- "sourceProperty": "value"
- },
- "typeVersion": 1.1
- },
- {
- "id": "49e58759-bedf-4f38-a96c-bd18e67b8aaf",
- "name": "Convert to File1",
- "type": "n8n-nodes-base.convertToFile",
- "onError": "continueErrorOutput",
- "position": [
- 4600,
- 840
- ],
- "parameters": {
- "options": {},
- "operation": "toBinary",
- "sourceProperty": "value"
- },
- "typeVersion": 1.1
- },
- {
- "id": "3735f5f5-665e-4649-b1c2-84a4a8699f70",
- "name": "Delete Session7",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 2920,
- 780
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "retryOnFail": false,
- "typeVersion": 4.2
- },
- {
- "id": "1b8b1e0c-f465-4963-869c-0e7086922151",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 920,
- -1023.3944834469928
- ],
- "parameters": {
- "color": 4,
- "width": 851.2111300888805,
- "height": 1333.3079943516484,
- "content": "## N8N Ultimate Scraper - Workflow\n\nThis workflow's objective is to collect data from any website page, whether it requires login or not.\n\nFor example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub.\n\n## Requirements\n**Selenium Container**: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages.\nYou can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: https://github.com/Touxan/n8n-ultimate-scraper\n\n**Residential Proxy Server**: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: https://geonode.com/invite/98895\n\n**OpenAI API Key**: For using GPT-4.\n\n## Optional\nSession Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. Follow the installation procedure to use it.\n\n## How to use \nDeploy the project with all the requiremnts and request your webhook.\n\n**Example of request**:\ncurl -X POST http://localhost:5678/webhook-test/yourwebhookid \\\n-H \"Content-Type: application/json\" \\\n-d '{\n \"subject\": \"Hugging Face\",\n \"Url\": \"github.com\",\n \"Target data\": [\n {\n \"DataName\": \"Followers\",\n \"description\": \"The number of followers of the GitHub page\"\n },\n {\n \"DataName\": \"Total Stars\",\n \"description\": \"The total numbers of stars on the different repos\"\n }\n ],\n \"cookies\": []\n}'\n\nYou can also scrape link like this : \ncurl -X POST http://localhost:5678/webhook-test/67d77918-2d5b-48c1-ae73-2004b32125f0 \\\n-H \"Content-Type: application/json\" \\\n-d '{\n \"Target Url\": \"https://github.com\",\n \"Target data\": [\n {\n \"DataName\": \"Followers\",\n \"description\": \"The number of followers of the GitHub page\"\n },\n {\n \"DataName\": \"Total Stars\",\n \"description\": \"The total numbers of stars on the different repo\"\n }\n]\n}'\n\n**Note**\nThe maximum nimber of Target data is 5."
- },
- "typeVersion": 1
- },
- {
- "id": "4d743518-4fcb-4e9f-aff7-a8959a78ccaf",
- "name": "Edit Fields (For testing prupose )",
- "type": "n8n-nodes-base.set",
- "position": [
- 1160,
- 540
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "3895040f-0a21-47ee-a73f-d3c7fd6edf36",
- "name": "Subject",
- "type": "string",
- "value": "={{ $json.body.subject }}"
- },
- {
- "id": "304e4240-513f-4c87-ae9d-4efda7d0c4ab",
- "name": "Website Domaine",
- "type": "string",
- "value": "={{ $json.body.Url }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "62b0a416-71a2-4d2b-83f9-8c5465c72006",
- "name": "Get ScreenShot 2",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 6200,
- 851
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "6a5b1a08-c47a-435e-8e0b-648cb8282a90",
- "name": "Convert to File2",
- "type": "n8n-nodes-base.convertToFile",
- "onError": "continueErrorOutput",
- "position": [
- 6440,
- 851
- ],
- "parameters": {
- "options": {},
- "operation": "toBinary",
- "sourceProperty": "value"
- },
- "typeVersion": 1.1
- },
- {
- "id": "a2aa5d45-5f41-41f7-a8ee-07c145b73d89",
- "name": "Go on ip-api.com",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 5960,
- 851
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"url\": \"https://ip-api.com/\"\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- },
- {
- "id": "8ddde1d2-0b09-45ca-88ef-db24352b095e",
- "name": "Delete Session8",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueRegularOutput",
- "position": [
- 6440,
- 1071
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
- "method": "DELETE",
- "options": {}
- },
- "retryOnFail": false,
- "typeVersion": 4.2
- },
- {
- "id": "78ffd8e1-b4b8-444c-8a7d-410172d3a7f8",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 5920,
- 727
- ],
- "parameters": {
- "color": 6,
- "width": 784.9798841202522,
- "height": 520.0741248156677,
- "content": "## Debug IP\n\nThis small debug flow aims to check the IP you're requesting with, in case you're using a proxy"
- },
- "typeVersion": 1
- },
- {
- "id": "be5de434-5f07-40bc-a1e6-aece9ad211b4",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1580,
- 420
- ],
- "parameters": {
- "width": 751.8596006980003,
- "height": 430.433007240277,
- "content": "## Search\n\n**Description** :\nThis part aims to search on Google for the subject and find the URL of the subject page based on the input URL."
- },
- "typeVersion": 1
- },
- {
- "id": "ffbb3c92-245b-4635-9adf-17d24f236bff",
- "name": "Error can't find url",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 2800,
- 280
- ],
- "parameters": {
- "options": {
- "responseCode": 404
- },
- "respondWith": "json",
- "responseBody": "{\n \"Error\": \"Can't find url\"\n}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "088ad72c-907a-409a-9fa4-00a16d396e1b",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2420,
- 420
- ],
- "parameters": {
- "width": 827.9448220213314,
- "height": 502.0185388323068,
- "content": "## Selenium Session\n\n**Description**:\nCreation and configuration of the Selenium session."
- },
- "typeVersion": 1
- },
- {
- "id": "00b8bf19-b34e-42ed-bb2a-3fbfa5f02a25",
- "name": "Resize browser window",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2920,
- 560
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $json.value.sessionId }}/window/rect",
- "method": "POST",
- "options": {},
- "jsonBody": "{\n \"width\": 1920,\n \"height\": 1080,\n \"x\": 0,\n \"y\": 0\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "typeVersion": 4.2
- },
- {
- "id": "007354a1-3f00-4ae9-ab53-54ded5eed563",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3500,
- -300
- ],
- "parameters": {
- "width": 3939.555135735299,
- "height": 821.0847869745435,
- "content": "## Scrape with cookies session\n\n**Description**\nThis part goes to the extracted URL, injects the cookies passed into the webhook, takes a screenshot of the webpage, and analyzes the image with GPT to extract the targeted data."
- },
- "typeVersion": 1
- },
- {
- "id": "5ab44e1b-6878-4af5-bfd8-1f1e5cbee3a7",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 3500,
- 580
- ],
- "parameters": {
- "width": 3336.952424000919,
- "height": 821.0847869745435,
- "content": "## Scrape without cookies session\n\n**Description**\nSame as the 'Scrape with cookies session' flow, but without the cookie injection"
- },
- "typeVersion": 1
- },
- {
- "id": "4fc7e290-0c60-4efe-ac3f-eb71ce5e457b",
- "name": "OpenAI",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "position": [
- 6340,
- -20
- ],
- "parameters": {
- "text": "=Analyse this image and extract revlant infromation about this subject : {{ $('Webhook').item.json.body.subject }}. \n\nIf the webpage seem block by waf, or don't have any relant information about the subject reurn BLOCK with out any aditinonal information.",
- "modelId": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o",
- "cachedResultName": "GPT-4O"
- },
- "options": {
- "detail": "auto",
- "maxTokens": 300
- },
- "resource": "image",
- "inputType": "base64",
- "operation": "analyze"
- },
- "credentials": {
- "openAiApi": {
- "id": "FmszNHDDVS32ud21",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.5
- },
- {
- "id": "b039ed2a-94da-4a37-b794-7fb1721a8ab3",
- "name": "OpenAI1",
- "type": "@n8n/n8n-nodes-langchain.openAi",
- "onError": "continueErrorOutput",
- "position": [
- 4780,
- 840
- ],
- "parameters": {
- "text": "=Analyse this image and extract revlant infromation about this subject : {{ $('Webhook').item.json.body.subject }}. \n\nIf the webpage seem block by waf, or don't have any relant information about the subject reurn BLOCK with out any aditinonal information.",
- "modelId": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o",
- "cachedResultName": "GPT-4O"
- },
- "options": {
- "detail": "auto",
- "maxTokens": 300
- },
- "resource": "image",
- "inputType": "base64",
- "operation": "analyze"
- },
- "credentials": {
- "openAiApi": {
- "id": "FmszNHDDVS32ud21",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.5
- },
- {
- "id": "c69364ce-c7e3-4f7a-ae0c-bad97643da30",
- "name": "Information Extractor1",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 5400,
- 920
- ],
- "parameters": {
- "text": "={{ $('OpenAI1').item.json.content }}",
- "options": {
- "systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, set the attribute's value to NA."
- },
- "attributes": {
- "attributes": [
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][0].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][0].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][1].DataName }}",
- "description": "=The total number of stars on all project"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][2].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][2].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][3].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][3].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][4].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][4].description }}"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "0e756adb-a6ba-421f-9d21-374e7fa74781",
- "name": "OpenAI Chat Model1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 5400,
- 1140
- ],
- "parameters": {
- "model": "gpt-4o-mini",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "FmszNHDDVS32ud21",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "920e9315-7de4-4a23-adbe-36338ea18097",
- "name": "Information Extractor2",
- "type": "@n8n/n8n-nodes-langchain.informationExtractor",
- "position": [
- 6920,
- 60
- ],
- "parameters": {
- "text": "={{ $('OpenAI').item.json.content }}",
- "options": {
- "systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, set the attribute's value to NA. If the attribute is empty you can omit it."
- },
- "attributes": {
- "attributes": [
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][0].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][0].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][1].DataName }}",
- "description": "=The total number of stars on all project"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][2].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][2].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][3].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][3].description }}"
- },
- {
- "name": "={{ $('Webhook').item.json.body['Target data'][4].DataName }}",
- "description": "={{ $('Webhook').item.json.body['Target data'][4].description }}"
- }
- ]
- }
- },
- "typeVersion": 1
- },
- {
- "id": "aa98d16e-d20c-4a8f-8eaf-1f64751dd8ea",
- "name": "OpenAI Chat Model2",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 6940,
- 220
- ],
- "parameters": {
- "model": "gpt-4o-mini",
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "FmszNHDDVS32ud21",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "ba41b87e-feb7-4753-95b3-d569d54d8756",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1820,
- -680
- ],
- "parameters": {
- "color": 3,
- "width": 813.0685668942513,
- "height": 507.4126722815008,
- "content": "## Proxy\n\n**Configuration**\n\nTo configure your proxy with the project, follow the instructions on the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. To configure the docker-compose, you also need to add this argument to the 'Create Selenium Session' node : --proxy-server=address:port.\n\n### โ ๏ธWarningโ ๏ธ\n Selenium does not support proxy authentication, so you need to add your server IP to the proxy whitelist. On GeoNode, it's here: https://app.geonode.com/whitelist-ip!"
- },
- "typeVersion": 1
- },
- {
- "id": "194bbecc-a5b3-4c5f-a17f-94703a44f196",
- "name": "Webhook",
- "type": "n8n-nodes-base.webhook",
- "position": [
- 940,
- 540
- ],
- "webhookId": "67d77918-2d5b-48c1-ae73-2004b32125f0",
- "parameters": {
- "path": "67d77918-2d5b-48c1-ae73-2004b32125f0",
- "options": {},
- "httpMethod": "POST",
- "responseMode": "responseNode"
- },
- "typeVersion": 2
- },
- {
- "id": "513389b0-0930-48d8-8cbb-e3575a0276ae",
- "name": "If Target Url",
- "type": "n8n-nodes-base.if",
- "position": [
- 1380,
- 620
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "4b608dcd-a175-4019-82c2-560320a2abce",
- "operator": {
- "type": "string",
- "operation": "empty",
- "singleValue": true
- },
- "leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "4ca0aee7-0dd2-4c78-b99b-8c188a3917f4",
- "name": "If1",
- "type": "n8n-nodes-base.if",
- "position": [
- 3700,
- 900
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "ff919945-b8c2-492a-b496-8617e9147389",
- "operator": {
- "type": "string",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "baa4dc94-67f3-4683-b8c7-6b6e856e7c64",
- "name": "Go on url1",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 3900,
- 960
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"url\": \"{{ $('Information Extractor').item.json.output.Good_url_for_etract_information }}\"\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- },
- {
- "id": "2c439b0e-7c78-4ae8-b653-3f02b3834aa8",
- "name": "If2",
- "type": "n8n-nodes-base.if",
- "position": [
- 3340,
- 560
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "loose"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "2a1bfc1e-28a6-45d1-9581-53b632af90e0",
- "operator": {
- "type": "string",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $('Webhook').item.json.body.cookies }}",
- "rightValue": ""
- }
- ]
- },
- "looseTypeValidation": true
- },
- "typeVersion": 2.2
- },
- {
- "id": "fc3260da-9131-4850-a581-55a27ce4428d",
- "name": "Go on url2",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 4260,
- -20
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"url\": \"{{ $('Webhook').item.json.body['Target Url'] }}\"\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- },
- {
- "id": "fe345010-1fa3-4d2c-8bc2-e87f6aeeb0d9",
- "name": "If3",
- "type": "n8n-nodes-base.if",
- "position": [
- 4060,
- 100
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "ff919945-b8c2-492a-b496-8617e9147389",
- "operator": {
- "type": "string",
- "operation": "notEmpty",
- "singleValue": true
- },
- "leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
- "rightValue": ""
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "1aae02ec-3a22-4dd5-aea4-819758f130c1",
- "name": "Go on url3",
- "type": "n8n-nodes-base.httpRequest",
- "onError": "continueErrorOutput",
- "position": [
- 4260,
- 160
- ],
- "parameters": {
- "url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"url\": \"{{ $('Information Extractor').item.json.output.Good_url_for_etract_information }}\"\n}\n",
- "sendBody": true,
- "specifyBody": "json"
- },
- "retryOnFail": true,
- "typeVersion": 4.2
- }
- ],
- "active": true,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "e0ae7ac4-4be7-4b9c-9247-1475ffd297b1",
- "connections": {
- "If": {
- "main": [
- [
- {
- "node": "If3",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session5",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If1": {
- "main": [
- [
- {
- "node": "Go on url",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Go on url1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If2": {
- "main": [
- [
- {
- "node": "If",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "If1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If3": {
- "main": [
- [
- {
- "node": "Go on url2",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Go on url3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Code": {
- "main": [
- [
- {
- "node": "Inject Cookie",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Limit": {
- "main": [
- [
- {
- "node": "Refresh browser",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI": {
- "main": [
- [
- {
- "node": "If Block1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI1": {
- "main": [
- [
- {
- "node": "If Block",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Webhook": {
- "main": [
- [
- {
- "node": "Edit Fields (For testing prupose )",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If Block": {
- "main": [
- [
- {
- "node": "Delete Session1",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Go on url": {
- "main": [
- [
- {
- "node": "Get ScreenShot 1",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If Block1": {
- "main": [
- [
- {
- "node": "Delete Session2",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Go on url1": {
- "main": [
- [
- {
- "node": "Get ScreenShot 1",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Go on url2": {
- "main": [
- [
- {
- "node": "Code",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Go on url3": {
- "main": [
- [
- {
- "node": "Code",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If Target Url": {
- "main": [
- [
- {
- "node": "Google search Query ",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Create Selenium Session",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Inject Cookie": {
- "main": [
- [
- {
- "node": "Limit",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session": {
- "main": [
- [
- {
- "node": "Information Extractor1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Convert to File": {
- "main": [
- [
- {
- "node": "OpenAI",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session1": {
- "main": [
- [
- {
- "node": "Respond to Webhook3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session2": {
- "main": [
- [
- {
- "node": "Respond to Webhook2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session3": {
- "main": [
- [
- {
- "node": "Information Extractor2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session4": {
- "main": [
- [
- {
- "node": "Error1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session5": {
- "main": [
- [
- {
- "node": "Error",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session6": {
- "main": [
- [
- {
- "node": "Error3",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Delete Session7": {
- "main": [
- [
- {
- "node": "Error2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get ScreenShot ": {
- "main": [
- [
- {
- "node": "Convert to File",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Refresh browser": {
- "main": [
- [
- {
- "node": "Get ScreenShot ",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session4",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Clean Webdriver ": {
- "main": [
- [
- {
- "node": "If2",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Convert to File1": {
- "main": [
- [
- {
- "node": "OpenAI1",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get ScreenShot 1": {
- "main": [
- [
- {
- "node": "Convert to File1",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session6",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get ScreenShot 2": {
- "main": [
- [
- {
- "node": "Convert to File2",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session8",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Go on ip-api.com": {
- "main": [
- [
- {
- "node": "Get ScreenShot 2",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session8",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Information Extractor",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model1": {
- "ai_languageModel": [
- [
- {
- "node": "Information Extractor1",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "OpenAI Chat Model2": {
- "ai_languageModel": [
- [
- {
- "node": "Information Extractor2",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Check if empty of NA": {
- "main": [
- [
- {
- "node": "Error can't find url",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Create Selenium Session",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google search Query ": {
- "main": [
- [
- {
- "node": "Extract First Url Match",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Information Extractor": {
- "main": [
- [
- {
- "node": "Check if empty of NA",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Resize browser window": {
- "main": [
- [
- {
- "node": "Clean Webdriver ",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Information Extractor1": {
- "main": [
- [
- {
- "node": "Success",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Information Extractor2": {
- "main": [
- [
- {
- "node": "Success with cookie",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Create Selenium Session": {
- "main": [
- [
- {
- "node": "Resize browser window",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Delete Session7",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Extract First Url Match": {
- "main": [
- [
- {
- "node": "Information Extractor",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Edit Fields (For testing prupose )": {
- "main": [
- [
- {
- "node": "If Target Url",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.txt b/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.txt
new file mode 100644
index 0000000..332b6aa
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Ultimate Scraper Workflow for n8n.txt
@@ -0,0 +1,1971 @@
+{
+"id": "kZ3aL4r7xc96Q7lp",
+"meta": {
+"instanceId": "b8b2c0d20b02864cf66adc9cbefc86e9e56de0252b653d37ba6613341b5e0bef",
+"templateCredsSetupCompleted": true
+},
+"name": "Selenium Ultimate Scraper Workflow",
+"tags": [],
+"nodes": [
+{
+"id": "20d35d68-db49-4183-a913-85ad06c13912",
+"name": "Extract First Url Match",
+"type": "n8n-nodes-base.html",
+"position": [
+1820,
+540
+],
+"parameters": {
+"options": {},
+"operation": "extractHtmlContent",
+"extractionValues": {
+"values": [
+{
+"key": "Url Find ",
+"attribute": "href",
+"cssSelector": "=a[href*=\"https://\"][href*=\"{{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}\"]\n",
+"returnArray": true,
+"returnValue": "attribute"
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "9167ea20-fc9c-4d75-bf4d-bb2016079dd0",
+"name": "OpenAI Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+2060,
+700
+],
+"parameters": {
+"model": "gpt-4o",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "FmszNHDDVS32ud21",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "42a8646d-1b0b-4309-a87d-9c8aeb355a28",
+"name": "Clean Webdriver ",
+"type": "n8n-nodes-base.httpRequest",
+"notes": "Script to delete traces of selenium in the browser ",
+"position": [
+3120,
+560
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/execute/sync",
+"method": "POST",
+"options": {},
+"jsonBody": "{\n \"script\": \"Object.defineProperty(navigator, 'webdriver', { get: () => undefined }); window.navigator.chrome = { runtime: {} }; Object.defineProperty(navigator, 'languages', { get: () => ['en-US', 'en'] }); Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3, 4, 5] });\",\n \"args\": []\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"notesInFlow": false,
+"typeVersion": 4.2
+},
+{
+"id": "107dd8de-e341-4819-a493-94ed57fd0f33",
+"name": "Delete Session",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+5180,
+920
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8c7ec6bc-d417-48c2-a6f2-ecce27803671",
+"name": "Delete Session2",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+6740,
+-160
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e43ecd94-b7f2-4f73-a9fa-b829de9e0296",
+"name": "If Block1",
+"type": "n8n-nodes-base.if",
+"position": [
+6520,
+-20
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "e6e6e15d-1cfe-48be-8ea0-f112e9781c9d",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.content }}",
+"rightValue": "BLOCK"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "08e46f63-41b5-4606-8f2c-df9e96c9c34e",
+"name": "Delete Session3",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+6740,
+60
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "b47d9b22-9a59-4c7a-8cba-9487f18207ee",
+"name": "Limit",
+"type": "n8n-nodes-base.limit",
+"position": [
+5120,
+-100
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "541622f7-562b-4e8a-93e5-61e6e918ff52",
+"name": "Delete Session1",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+5180,
+720
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "825be0d7-9dd3-4a2f-8c3d-fd405f59a5d6",
+"name": "Delete Session4",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+5780,
+260
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"retryOnFail": false,
+"typeVersion": 4.2
+},
+{
+"id": "56f6f4f6-f737-4de8-bdfe-029546909677",
+"name": "Success with cookie",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+7260,
+60
+],
+"parameters": {
+"options": {
+"responseCode": 200
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "c6939773-e230-45e1-bf76-d0299c2c7066",
+"name": "Respond to Webhook2",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+6920,
+-160
+],
+"parameters": {
+"options": {
+"responseCode": 200
+},
+"respondWith": "json",
+"responseBody": "{\n \"Success \": \"Request has been block by the targeted website\"\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "ea921f11-323f-4c79-8cc6-779b39498b05",
+"name": "Code",
+"type": "n8n-nodes-base.code",
+"position": [
+4700,
+-100
+],
+"parameters": {
+"jsCode": "// Rรฉcupรจre les donnรฉes du nลud Webhook (en remplaรงant \"Webhook\" par le nom du nลud Webhook dans votre workflow)\nconst webhookData = $node[\"Webhook\"].json;\n\n// Fonction pour convertir la valeur de sameSite\nfunction convertSameSite(value) {\n // Conversion spรฉcifique des valeurs de sameSite\n const conversionMap = {\n \"unspecified\": \"None\",\n \"lax\": \"Lax\",\n \"strict\": \"Strict\"\n };\n \n // Si la valeur existe dans le tableau de conversion, on la convertit\n if (value in conversionMap) {\n return conversionMap[value];\n }\n \n // Si la valeur est dรฉjร une des valeurs acceptรฉes par Selenium\n const allowedValues = [\"Strict\", \"Lax\", \"None\"];\n if (allowedValues.includes(value)) {\n return value;\n } else {\n // Si la valeur n'est pas reconnue, on la remplace par \"Lax\" (par dรฉfaut)\n return \"Lax\";\n }\n}\n\n// Vรฉrifiez et traitez les donnรฉes des cookies\nif (webhookData.body && webhookData.body.cookies) {\n let items = [];\n for (const cookieObject of webhookData.body.cookies) {\n if (cookieObject.cookie) {\n // Convertir la valeur de sameSite\n cookieObject.cookie.sameSite = convertSameSite(cookieObject.cookie.sameSite);\n \n // Ajouter le cookie ร la liste des items\n items.push({\n json: cookieObject.cookie\n });\n }\n }\n return items;\n}\n\n// Si les cookies ne sont pas trouvรฉs, renvoyer un tableau vide\nreturn [];\n"
+},
+"typeVersion": 2
+},
+{
+"id": "c3d77928-eefc-4903-9b4f-b14bd6f34e3c",
+"name": "Delete Session5",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+3940,
+360
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"retryOnFail": false,
+"typeVersion": 4.2
+},
+{
+"id": "036cfce6-8082-4539-bb0e-980368679fe5",
+"name": "Error",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+4120,
+360
+],
+"parameters": {
+"options": {
+"responseCode": 404
+},
+"respondWith": "json",
+"responseBody": "{\n \"Error\": \"Cookies are note for the targeted url\"\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "09d6a99b-d8b3-40c9-b74a-14014e3647e2",
+"name": "Error1",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+6000,
+260
+],
+"parameters": {
+"options": {
+"responseCode": 500
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "0b1f3442-6b70-405f-b597-642e9c982b82",
+"name": "Error2",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+3060,
+780
+],
+"parameters": {
+"options": {
+"responseCode": 500
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "4d0112bb-cbfd-45c6-961a-964bd8f59cac",
+"name": "If",
+"type": "n8n-nodes-base.if",
+"position": [
+3760,
+200
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "1bffbc80-9913-46e7-a594-ebc26948c83b",
+"operator": {
+"type": "string",
+"operation": "contains"
+},
+"leftValue": "={{ $('Webhook').item.json.body.cookies[0].cookie.domain }}",
+"rightValue": "={{ $('Webhook').item.json.body.Url }}"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "58a50b80-df4c-4b6f-a682-72237f4dbdef",
+"name": "Inject Cookie",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+4900,
+-100
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/cookie",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"cookie\": {\n \"name\": \"{{ $json.name }}\",\n \"value\": \"{{ $json.value }}\",\n \"domain\": \"{{ $json.domain }}\",\n \"path\": \"{{ $json.path }}\",\n \"secure\": {{ $json.secure }},\n \"httpOnly\": {{ $json.httpOnly }},\n \"sameSite\": \"{{ $json.sameSite }}\",\n \"expirationDate\": {{ $json.expirationDate }}\n }\n}",
+"sendBody": true,
+"specifyBody": "json"
+},
+"typeVersion": 4.2
+},
+{
+"id": "39f7401b-b6b7-4f0c-9afc-8f144d394350",
+"name": "Respond to Webhook3",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+5400,
+720
+],
+"parameters": {
+"options": {
+"responseCode": 200
+},
+"respondWith": "json",
+"responseBody": "{\n \"Success \": \"Request has been block by the targeted website\"\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "80b107cc-2f6c-46f0-a597-e85594634492",
+"name": "Success",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+5740,
+920
+],
+"parameters": {
+"options": {
+"responseKey": "={{ $json.output }}",
+"responseCode": 200
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "94a97354-07d9-428e-989c-ef066f9b4d8a",
+"name": "Go on url",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+3900,
+780
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"url\": \"{{ $('Webhook').item.json.body['Target Url'] }}\"\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+},
+{
+"id": "fd044cf3-594d-48af-bbd1-f2d9adedcbc1",
+"name": "Delete Session6",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+4360,
+1200
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"retryOnFail": false,
+"typeVersion": 4.2
+},
+{
+"id": "7c28c3b6-1141-4609-8774-cb6b4d842b97",
+"name": "Error3",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+4520,
+1200
+],
+"parameters": {
+"options": {
+"responseCode": 500
+},
+"respondWith": "json",
+"responseBody": "{\n \"Error\": \"Page crash on the extracted url\"\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "52f78923-156f-4861-88ba-f0253c483bd9",
+"name": "Information Extractor",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+2040,
+540
+],
+"parameters": {
+"text": "={{ $json['Url Find '][1] }}{{ $json['Url Find '][2] }}{{ $json['Url Find '][3] }}",
+"options": {
+"systemPromptTemplate": "=You are an expert extraction algorithm.\nOnly extract relevant url from the unstructured urls array.\nA relevant url is a url whre you can find relevant information about this subject : {{ $('Edit Fields (For testing prupose )').item.json.Subject }}, on this domaine name : {{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}.\nIf you do not know the value of an attribute asked to extract, you need \\ attribute's value as NA."
+},
+"attributes": {
+"attributes": [
+{
+"name": "Good_url_for_etract_information",
+"required": true,
+"description": "=The url where I can extract relevant infroamtion on this subject : {{ $('Edit Fields (For testing prupose )').item.json.Subject }} on this domaine name : {{ $('Edit Fields (For testing prupose )').item.json['Website Domaine'] }}"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "6ac249e2-a9d8-4590-b050-3a0a2472fa3c",
+"name": "Check if empty of NA",
+"type": "n8n-nodes-base.if",
+"position": [
+2440,
+540
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "or",
+"conditions": [
+{
+"id": "9470fb6c-e367-4af7-a697-275e724fe771",
+"operator": {
+"type": "string",
+"operation": "empty",
+"singleValue": true
+},
+"leftValue": "={{ $json.output.Good_url_for_etract_information }}",
+"rightValue": ""
+},
+{
+"id": "8518e9a9-5b0c-4699-97c5-d9b7b1943918",
+"operator": {
+"name": "filter.operator.equals",
+"type": "string",
+"operation": "equals"
+},
+"leftValue": "={{ $json.output.Good_url_for_etract_information }}",
+"rightValue": "NA"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "f380eff7-3d18-4791-9dac-8a88d3fdcc4f",
+"name": "If Block",
+"type": "n8n-nodes-base.if",
+"position": [
+4960,
+840
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "e6e6e15d-1cfe-48be-8ea0-f112e9781c9d",
+"operator": {
+"type": "string",
+"operation": "contains"
+},
+"leftValue": "={{ $json.content }}",
+"rightValue": "BLOCK"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "43382397-89b5-4b90-9016-49109ec04baf",
+"name": "Google search Query ",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1600,
+540
+],
+"parameters": {
+"url": "=https://www.google.com/search?q=site:{{ $json['Website Domaine'] }}+{{$json.Subject}}&oq=site&gs_lcrp=EgZjaHJvbWUqCAgAEEUYJxg7MggIABBFGCcYOzIICAEQRRgnGDsyBggCEEUYOzIRCAMQRRg5GEMYyQMYgAQYigUyBggEEEUYQDIGCAUQRRg9MgYIBhBFGD0yBggHEEUYPdIBCDEwNTRqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "d34256af-1b43-4f64-853c-cf063b8c6b68",
+"name": "Create Selenium Session",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+2680,
+640
+],
+"parameters": {
+"url": "http://selenium_chrome:4444/wd/hub/session",
+"method": "POST",
+"options": {
+"timeout": 5000
+},
+"jsonBody": "{\n \"capabilities\": {\n \"alwaysMatch\": {\n \"browserName\": \"chrome\",\n \"goog:chromeOptions\": {\n \"args\": [ \n \"--disable-blink-features=AutomationControlled\",\n \"--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\"\n ]\n }\n }\n }\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+},
+{
+"id": "4f0f696c-9637-4c7d-82ae-1f5c36bb9cd1",
+"name": "Get ScreenShot 1",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+4420,
+840
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "ba72c0cf-217a-4411-80f6-ca28ccdb0151",
+"name": "Refresh browser",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+5320,
+-100
+],
+"parameters": {
+"url": "=http:///selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/refresh",
+"method": "POST",
+"options": {},
+"jsonBody": "{}",
+"sendBody": true,
+"specifyBody": "json"
+},
+"typeVersion": 4.2
+},
+{
+"id": "b6ba7068-399a-467d-ba58-7f47d650e2f1",
+"name": "Get ScreenShot ",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+5880,
+-20
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "792649be-0ee2-442f-bc21-d0c297cea227",
+"name": "Convert to File",
+"type": "n8n-nodes-base.convertToFile",
+"onError": "continueErrorOutput",
+"position": [
+6160,
+-20
+],
+"parameters": {
+"options": {},
+"operation": "toBinary",
+"sourceProperty": "value"
+},
+"typeVersion": 1.1
+},
+{
+"id": "49e58759-bedf-4f38-a96c-bd18e67b8aaf",
+"name": "Convert to File1",
+"type": "n8n-nodes-base.convertToFile",
+"onError": "continueErrorOutput",
+"position": [
+4600,
+840
+],
+"parameters": {
+"options": {},
+"operation": "toBinary",
+"sourceProperty": "value"
+},
+"typeVersion": 1.1
+},
+{
+"id": "3735f5f5-665e-4649-b1c2-84a4a8699f70",
+"name": "Delete Session7",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+2920,
+780
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"retryOnFail": false,
+"typeVersion": 4.2
+},
+{
+"id": "1b8b1e0c-f465-4963-869c-0e7086922151",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+920,
+-1023.3944834469928
+],
+"parameters": {
+"color": 4,
+"width": 851.2111300888805,
+"height": 1333.3079943516484,
+"content": "## N8N Ultimate Scraper - Workflow\n\nThis workflow's objective is to collect data from any website page, whether it requires login or not.\n\nFor example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub.\n\n## Requirements\n**Selenium Container**: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages.\nYou can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: https://github.com/Touxan/n8n-ultimate-scraper\n\n**Residential Proxy Server**: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: https://geonode.com/invite/98895\n\n**OpenAI API Key**: For using GPT-4.\n\n## Optional\nSession Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. Follow the installation procedure to use it.\n\n## How to use \nDeploy the project with all the requiremnts and request your webhook.\n\n**Example of request**:\ncurl -X POST http://localhost:5678/webhook-test/yourwebhookid \\\n-H \"Content-Type: application/json\" \\\n-d '{\n \"subject\": \"Hugging Face\",\n \"Url\": \"github.com\",\n \"Target data\": [\n {\n \"DataName\": \"Followers\",\n \"description\": \"The number of followers of the GitHub page\"\n },\n {\n \"DataName\": \"Total Stars\",\n \"description\": \"The total numbers of stars on the different repos\"\n }\n ],\n \"cookies\": []\n}'\n\nYou can also scrape link like this : \ncurl -X POST http://localhost:5678/webhook-test/67d77918-2d5b-48c1-ae73-2004b32125f0 \\\n-H \"Content-Type: application/json\" \\\n-d '{\n \"Target Url\": \"https://github.com\",\n \"Target data\": [\n {\n \"DataName\": \"Followers\",\n \"description\": \"The number of followers of the GitHub page\"\n },\n {\n \"DataName\": \"Total Stars\",\n \"description\": \"The total numbers of stars on the different repo\"\n }\n]\n}'\n\n**Note**\nThe maximum nimber of Target data is 5."
+},
+"typeVersion": 1
+},
+{
+"id": "4d743518-4fcb-4e9f-aff7-a8959a78ccaf",
+"name": "Edit Fields (For testing prupose )",
+"type": "n8n-nodes-base.set",
+"position": [
+1160,
+540
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "3895040f-0a21-47ee-a73f-d3c7fd6edf36",
+"name": "Subject",
+"type": "string",
+"value": "={{ $json.body.subject }}"
+},
+{
+"id": "304e4240-513f-4c87-ae9d-4efda7d0c4ab",
+"name": "Website Domaine",
+"type": "string",
+"value": "={{ $json.body.Url }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "62b0a416-71a2-4d2b-83f9-8c5465c72006",
+"name": "Get ScreenShot 2",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+6200,
+851
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/screenshot",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "6a5b1a08-c47a-435e-8e0b-648cb8282a90",
+"name": "Convert to File2",
+"type": "n8n-nodes-base.convertToFile",
+"onError": "continueErrorOutput",
+"position": [
+6440,
+851
+],
+"parameters": {
+"options": {},
+"operation": "toBinary",
+"sourceProperty": "value"
+},
+"typeVersion": 1.1
+},
+{
+"id": "a2aa5d45-5f41-41f7-a8ee-07c145b73d89",
+"name": "Go on ip-api.com",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+5960,
+851
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"url\": \"https://ip-api.com/\"\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+},
+{
+"id": "8ddde1d2-0b09-45ca-88ef-db24352b095e",
+"name": "Delete Session8",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueRegularOutput",
+"position": [
+6440,
+1071
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}",
+"method": "DELETE",
+"options": {}
+},
+"retryOnFail": false,
+"typeVersion": 4.2
+},
+{
+"id": "78ffd8e1-b4b8-444c-8a7d-410172d3a7f8",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+5920,
+727
+],
+"parameters": {
+"color": 6,
+"width": 784.9798841202522,
+"height": 520.0741248156677,
+"content": "## Debug IP\n\nThis small debug flow aims to check the IP you're requesting with, in case you're using a proxy"
+},
+"typeVersion": 1
+},
+{
+"id": "be5de434-5f07-40bc-a1e6-aece9ad211b4",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1580,
+420
+],
+"parameters": {
+"width": 751.8596006980003,
+"height": 430.433007240277,
+"content": "## Search\n\n**Description** :\nThis part aims to search on Google for the subject and find the URL of the subject page based on the input URL."
+},
+"typeVersion": 1
+},
+{
+"id": "ffbb3c92-245b-4635-9adf-17d24f236bff",
+"name": "Error can't find url",
+"type": "n8n-nodes-base.respondToWebhook",
+"position": [
+2800,
+280
+],
+"parameters": {
+"options": {
+"responseCode": 404
+},
+"respondWith": "json",
+"responseBody": "{\n \"Error\": \"Can't find url\"\n}"
+},
+"typeVersion": 1.1
+},
+{
+"id": "088ad72c-907a-409a-9fa4-00a16d396e1b",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2420,
+420
+],
+"parameters": {
+"width": 827.9448220213314,
+"height": 502.0185388323068,
+"content": "## Selenium Session\n\n**Description**:\nCreation and configuration of the Selenium session."
+},
+"typeVersion": 1
+},
+{
+"id": "00b8bf19-b34e-42ed-bb2a-3fbfa5f02a25",
+"name": "Resize browser window",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2920,
+560
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $json.value.sessionId }}/window/rect",
+"method": "POST",
+"options": {},
+"jsonBody": "{\n \"width\": 1920,\n \"height\": 1080,\n \"x\": 0,\n \"y\": 0\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"typeVersion": 4.2
+},
+{
+"id": "007354a1-3f00-4ae9-ab53-54ded5eed563",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3500,
+-300
+],
+"parameters": {
+"width": 3939.555135735299,
+"height": 821.0847869745435,
+"content": "## Scrape with cookies session\n\n**Description**\nThis part goes to the extracted URL, injects the cookies passed into the webhook, takes a screenshot of the webpage, and analyzes the image with GPT to extract the targeted data."
+},
+"typeVersion": 1
+},
+{
+"id": "5ab44e1b-6878-4af5-bfd8-1f1e5cbee3a7",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+3500,
+580
+],
+"parameters": {
+"width": 3336.952424000919,
+"height": 821.0847869745435,
+"content": "## Scrape without cookies session\n\n**Description**\nSame as the 'Scrape with cookies session' flow, but without the cookie injection"
+},
+"typeVersion": 1
+},
+{
+"id": "4fc7e290-0c60-4efe-ac3f-eb71ce5e457b",
+"name": "OpenAI",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"position": [
+6340,
+-20
+],
+"parameters": {
+"text": "=Analyse this image and extract revlant infromation about this subject : {{ $('Webhook').item.json.body.subject }}. \n\nIf the webpage seem block by waf, or don't have any relant information about the subject reurn BLOCK with out any aditinonal information.",
+"modelId": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o",
+"cachedResultName": "GPT-4O"
+},
+"options": {
+"detail": "auto",
+"maxTokens": 300
+},
+"resource": "image",
+"inputType": "base64",
+"operation": "analyze"
+},
+"credentials": {
+"openAiApi": {
+"id": "FmszNHDDVS32ud21",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.5
+},
+{
+"id": "b039ed2a-94da-4a37-b794-7fb1721a8ab3",
+"name": "OpenAI1",
+"type": "@n8n/n8n-nodes-langchain.openAi",
+"onError": "continueErrorOutput",
+"position": [
+4780,
+840
+],
+"parameters": {
+"text": "=Analyse this image and extract revlant infromation about this subject : {{ $('Webhook').item.json.body.subject }}. \n\nIf the webpage seem block by waf, or don't have any relant information about the subject reurn BLOCK with out any aditinonal information.",
+"modelId": {
+"__rl": true,
+"mode": "list",
+"value": "gpt-4o",
+"cachedResultName": "GPT-4O"
+},
+"options": {
+"detail": "auto",
+"maxTokens": 300
+},
+"resource": "image",
+"inputType": "base64",
+"operation": "analyze"
+},
+"credentials": {
+"openAiApi": {
+"id": "FmszNHDDVS32ud21",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1.5
+},
+{
+"id": "c69364ce-c7e3-4f7a-ae0c-bad97643da30",
+"name": "Information Extractor1",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+5400,
+920
+],
+"parameters": {
+"text": "={{ $('OpenAI1').item.json.content }}",
+"options": {
+"systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, set the attribute's value to NA."
+},
+"attributes": {
+"attributes": [
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][0].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][0].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][1].DataName }}",
+"description": "=The total number of stars on all project"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][2].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][2].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][3].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][3].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][4].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][4].description }}"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "0e756adb-a6ba-421f-9d21-374e7fa74781",
+"name": "OpenAI Chat Model1",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+5400,
+1140
+],
+"parameters": {
+"model": "gpt-4o-mini",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "FmszNHDDVS32ud21",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "920e9315-7de4-4a23-adbe-36338ea18097",
+"name": "Information Extractor2",
+"type": "@n8n/n8n-nodes-langchain.informationExtractor",
+"position": [
+6920,
+60
+],
+"parameters": {
+"text": "={{ $('OpenAI').item.json.content }}",
+"options": {
+"systemPromptTemplate": "You are an expert extraction algorithm.\nOnly extract relevant information from the text.\nIf you do not know the value of an attribute asked to extract, set the attribute's value to NA. If the attribute is empty you can omit it."
+},
+"attributes": {
+"attributes": [
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][0].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][0].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][1].DataName }}",
+"description": "=The total number of stars on all project"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][2].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][2].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][3].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][3].description }}"
+},
+{
+"name": "={{ $('Webhook').item.json.body['Target data'][4].DataName }}",
+"description": "={{ $('Webhook').item.json.body['Target data'][4].description }}"
+}
+]
+}
+},
+"typeVersion": 1
+},
+{
+"id": "aa98d16e-d20c-4a8f-8eaf-1f64751dd8ea",
+"name": "OpenAI Chat Model2",
+"type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
+"position": [
+6940,
+220
+],
+"parameters": {
+"model": "gpt-4o-mini",
+"options": {}
+},
+"credentials": {
+"openAiApi": {
+"id": "FmszNHDDVS32ud21",
+"name": "OpenAi account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "ba41b87e-feb7-4753-95b3-d569d54d8756",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1820,
+-680
+],
+"parameters": {
+"color": 3,
+"width": 813.0685668942513,
+"height": 507.4126722815008,
+"content": "## Proxy\n\n**Configuration**\n\nTo configure your proxy with the project, follow the instructions on the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. To configure the docker-compose, you also need to add this argument to the 'Create Selenium Session' node : --proxy-server=address:port.\n\n### โ ๏ธWarningโ ๏ธ\n Selenium does not support proxy authentication, so you need to add your server IP to the proxy whitelist. On GeoNode, it's here: https://app.geonode.com/whitelist-ip!"
+},
+"typeVersion": 1
+},
+{
+"id": "194bbecc-a5b3-4c5f-a17f-94703a44f196",
+"name": "Webhook",
+"type": "n8n-nodes-base.webhook",
+"position": [
+940,
+540
+],
+"webhookId": "67d77918-2d5b-48c1-ae73-2004b32125f0",
+"parameters": {
+"path": "67d77918-2d5b-48c1-ae73-2004b32125f0",
+"options": {},
+"httpMethod": "POST",
+"responseMode": "responseNode"
+},
+"typeVersion": 2
+},
+{
+"id": "513389b0-0930-48d8-8cbb-e3575a0276ae",
+"name": "If Target Url",
+"type": "n8n-nodes-base.if",
+"position": [
+1380,
+620
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "4b608dcd-a175-4019-82c2-560320a2abce",
+"operator": {
+"type": "string",
+"operation": "empty",
+"singleValue": true
+},
+"leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "4ca0aee7-0dd2-4c78-b99b-8c188a3917f4",
+"name": "If1",
+"type": "n8n-nodes-base.if",
+"position": [
+3700,
+900
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "ff919945-b8c2-492a-b496-8617e9147389",
+"operator": {
+"type": "string",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "baa4dc94-67f3-4683-b8c7-6b6e856e7c64",
+"name": "Go on url1",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+3900,
+960
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"url\": \"{{ $('Information Extractor').item.json.output.Good_url_for_etract_information }}\"\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+},
+{
+"id": "2c439b0e-7c78-4ae8-b653-3f02b3834aa8",
+"name": "If2",
+"type": "n8n-nodes-base.if",
+"position": [
+3340,
+560
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "loose"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "2a1bfc1e-28a6-45d1-9581-53b632af90e0",
+"operator": {
+"type": "string",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $('Webhook').item.json.body.cookies }}",
+"rightValue": ""
+}
+]
+},
+"looseTypeValidation": true
+},
+"typeVersion": 2.2
+},
+{
+"id": "fc3260da-9131-4850-a581-55a27ce4428d",
+"name": "Go on url2",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+4260,
+-20
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"url\": \"{{ $('Webhook').item.json.body['Target Url'] }}\"\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+},
+{
+"id": "fe345010-1fa3-4d2c-8bc2-e87f6aeeb0d9",
+"name": "If3",
+"type": "n8n-nodes-base.if",
+"position": [
+4060,
+100
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "ff919945-b8c2-492a-b496-8617e9147389",
+"operator": {
+"type": "string",
+"operation": "notEmpty",
+"singleValue": true
+},
+"leftValue": "={{ $('Webhook').item.json.body['Target Url'] }}",
+"rightValue": ""
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "1aae02ec-3a22-4dd5-aea4-819758f130c1",
+"name": "Go on url3",
+"type": "n8n-nodes-base.httpRequest",
+"onError": "continueErrorOutput",
+"position": [
+4260,
+160
+],
+"parameters": {
+"url": "=http://selenium_chrome:4444/wd/hub/session/{{ $('Create Selenium Session').item.json.value.sessionId }}/url",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"url\": \"{{ $('Information Extractor').item.json.output.Good_url_for_etract_information }}\"\n}\n",
+"sendBody": true,
+"specifyBody": "json"
+},
+"retryOnFail": true,
+"typeVersion": 4.2
+}
+],
+"active": true,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "e0ae7ac4-4be7-4b9c-9247-1475ffd297b1",
+"connections": {
+"If": {
+"main": [
+[
+{
+"node": "If3",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session5",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If1": {
+"main": [
+[
+{
+"node": "Go on url",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Go on url1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If2": {
+"main": [
+[
+{
+"node": "If",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "If1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If3": {
+"main": [
+[
+{
+"node": "Go on url2",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Go on url3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Code": {
+"main": [
+[
+{
+"node": "Inject Cookie",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Limit": {
+"main": [
+[
+{
+"node": "Refresh browser",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI": {
+"main": [
+[
+{
+"node": "If Block1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI1": {
+"main": [
+[
+{
+"node": "If Block",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Webhook": {
+"main": [
+[
+{
+"node": "Edit Fields (For testing prupose )",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If Block": {
+"main": [
+[
+{
+"node": "Delete Session1",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Go on url": {
+"main": [
+[
+{
+"node": "Get ScreenShot 1",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If Block1": {
+"main": [
+[
+{
+"node": "Delete Session2",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Go on url1": {
+"main": [
+[
+{
+"node": "Get ScreenShot 1",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Go on url2": {
+"main": [
+[
+{
+"node": "Code",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Go on url3": {
+"main": [
+[
+{
+"node": "Code",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If Target Url": {
+"main": [
+[
+{
+"node": "Google search Query ",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Create Selenium Session",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Inject Cookie": {
+"main": [
+[
+{
+"node": "Limit",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session": {
+"main": [
+[
+{
+"node": "Information Extractor1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert to File": {
+"main": [
+[
+{
+"node": "OpenAI",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session1": {
+"main": [
+[
+{
+"node": "Respond to Webhook3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session2": {
+"main": [
+[
+{
+"node": "Respond to Webhook2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session3": {
+"main": [
+[
+{
+"node": "Information Extractor2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session4": {
+"main": [
+[
+{
+"node": "Error1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session5": {
+"main": [
+[
+{
+"node": "Error",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session6": {
+"main": [
+[
+{
+"node": "Error3",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Delete Session7": {
+"main": [
+[
+{
+"node": "Error2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get ScreenShot ": {
+"main": [
+[
+{
+"node": "Convert to File",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Refresh browser": {
+"main": [
+[
+{
+"node": "Get ScreenShot ",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session4",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Clean Webdriver ": {
+"main": [
+[
+{
+"node": "If2",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Convert to File1": {
+"main": [
+[
+{
+"node": "OpenAI1",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get ScreenShot 1": {
+"main": [
+[
+{
+"node": "Convert to File1",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session6",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get ScreenShot 2": {
+"main": [
+[
+{
+"node": "Convert to File2",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session8",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Go on ip-api.com": {
+"main": [
+[
+{
+"node": "Get ScreenShot 2",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session8",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Information Extractor",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Information Extractor1",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model2": {
+"ai_languageModel": [
+[
+{
+"node": "Information Extractor2",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Check if empty of NA": {
+"main": [
+[
+{
+"node": "Error can't find url",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Create Selenium Session",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google search Query ": {
+"main": [
+[
+{
+"node": "Extract First Url Match",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Information Extractor": {
+"main": [
+[
+{
+"node": "Check if empty of NA",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Resize browser window": {
+"main": [
+[
+{
+"node": "Clean Webdriver ",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Information Extractor1": {
+"main": [
+[
+{
+"node": "Success",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Information Extractor2": {
+"main": [
+[
+{
+"node": "Success with cookie",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Create Selenium Session": {
+"main": [
+[
+{
+"node": "Resize browser window",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Delete Session7",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Extract First Url Match": {
+"main": [
+[
+{
+"node": "Information Extractor",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Edit Fields (For testing prupose )": {
+"main": [
+[
+{
+"node": "If Target Url",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].json b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].json
deleted file mode 100644
index acf2db0..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].json
+++ /dev/null
@@ -1,688 +0,0 @@
-{
- "id": "pPtCy6qPfEv1qNRn",
- "meta": {
- "instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
- },
- "name": "[1/3 - anomaly detection] [1/2 - KNN classification] Batch upload dataset to Qdrant (crops dataset)",
- "tags": [
- {
- "id": "n3zAUYFhdqtjhcLf",
- "name": "qdrant",
- "createdAt": "2024-12-10T11:56:59.987Z",
- "updatedAt": "2024-12-10T11:56:59.987Z"
- }
- ],
- "nodes": [
- {
- "id": "53831410-b4f3-4374-8bdd-c2a33cd873cb",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -640,
- 0
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "e303ccea-c0e0-4fe5-bd31-48380a0e438f",
- "name": "Google Cloud Storage",
- "type": "n8n-nodes-base.googleCloudStorage",
- "position": [
- 820,
- 160
- ],
- "parameters": {
- "resource": "object",
- "returnAll": true,
- "bucketName": "n8n-qdrant-demo",
- "listFilters": {
- "prefix": "agricultural-crops"
- },
- "requestOptions": {}
- },
- "credentials": {
- "googleCloudStorageOAuth2Api": {
- "id": "fn0sr7grtfprVQvL",
- "name": "Google Cloud Storage account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "737bdb15-61cf-48eb-96af-569eb5986ee8",
- "name": "Get fields for Qdrant",
- "type": "n8n-nodes-base.set",
- "position": [
- 1080,
- 160
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "10d9147f-1c0c-4357-8413-3130829c2e24",
- "name": "=publicLink",
- "type": "string",
- "value": "=https://storage.googleapis.com/{{ $json.bucket }}/{{ $json.selfLink.split('/').splice(-1) }}"
- },
- {
- "id": "ff9e6a0b-e47a-4550-a13b-465507c75f8f",
- "name": "cropName",
- "type": "string",
- "value": "={{ $json.id.split('/').slice(-3, -2)[0].toLowerCase()}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "2b18ed0c-38d3-49e9-be3d-4f7b35f4d9e5",
- "name": "Qdrant cluster variables",
- "type": "n8n-nodes-base.set",
- "position": [
- -360,
- 0
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
- },
- {
- "id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
- "name": "collectionName",
- "type": "string",
- "value": "=agricultural-crops"
- },
- {
- "id": "33581e0a-307f-4380-9533-615791096de7",
- "name": "VoyageEmbeddingsDim",
- "type": "number",
- "value": 1024
- },
- {
- "id": "6e390343-2cd2-4559-aba9-82b13acb7f52",
- "name": "batchSize",
- "type": "number",
- "value": 4
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "f88d290e-3311-4322-b2a5-1350fc1f8768",
- "name": "Embed crop image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2120,
- 160
- ],
- "parameters": {
- "url": "https://api.voyageai.com/v1/multimodalembeddings",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"inputs\": $json.batchVoyage,\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "Vb0RNVDnIHmgnZOP",
- "name": "Voyage API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "250c6a8d-f545-4037-8069-c834437bbe15",
- "name": "Create Qdrant Collection",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 320,
- 160
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}",
- "method": "PUT",
- "options": {},
- "jsonBody": "={{\n{\n \"vectors\": {\n \"voyage\": { \n \"size\": $('Qdrant cluster variables').first().json.VoyageEmbeddingsDim, \n \"distance\": \"Cosine\" \n } \n }\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "20b612ff-4794-43ef-bf45-008a16a2f30f",
- "name": "Check Qdrant Collection Existence",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -100,
- 0
- ],
- "parameters": {
- "url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/exists",
- "options": {},
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "c067740b-5de3-452e-a614-bf14985a73a0",
- "name": "Batches in the API's format",
- "type": "n8n-nodes-base.set",
- "position": [
- 1860,
- 160
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "f14db112-6f15-4405-aa47-8cb56bb8ae7a",
- "name": "=batchVoyage",
- "type": "array",
- "value": "={{ $json.batch.map(item => ({ \"content\": ([{\"type\": \"image_url\", \"image_url\": item[\"publicLink\"]}])}))}}"
- },
- {
- "id": "3885fd69-66f5-4435-86a4-b80eaa568ac1",
- "name": "=batchPayloadQdrant",
- "type": "array",
- "value": "={{ $json.batch.map(item => ({\"crop_name\":item[\"cropName\"], \"image_path\":item[\"publicLink\"]})) }}"
- },
- {
- "id": "8ea7a91e-af27-49cb-9a29-41dae15c4e33",
- "name": "uuids",
- "type": "array",
- "value": "={{ $json.uuids }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "bf9a9532-db64-4c02-b91d-47e708ded4d3",
- "name": "Batch Upload to Qdrant",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2320,
- 160
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points",
- "method": "PUT",
- "options": {},
- "jsonBody": "={{\n{\n \"batch\": {\n \"ids\" : $('Batches in the API\\'s format').item.json.uuids,\n \"vectors\": {\"voyage\": $json.data.map(item => item[\"embedding\"]) },\n \"payloads\": $('Batches in the API\\'s format').item.json.batchPayloadQdrant\n }\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "3c30373f-c84c-405f-bb84-ec8b4c7419f4",
- "name": "Split in batches, generate uuids for Qdrant points",
- "type": "n8n-nodes-base.code",
- "position": [
- 1600,
- 160
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "import uuid\n\ncrops = [item.json for item in _input.all()]\nbatch_size = int(_('Qdrant cluster variables').first()['json']['batchSize'])\n\ndef split_into_batches_add_uuids(array, batch_size):\n return [\n {\n \"batch\": array[i:i + batch_size],\n \"uuids\": [str(uuid.uuid4()) for j in range(len(array[i:i + batch_size]))]\n }\n for i in range(0, len(array), batch_size)\n ]\n\n# Split crops into batches\nbatched_crops = split_into_batches_add_uuids(crops, batch_size)\n\nreturn batched_crops"
- },
- "typeVersion": 2
- },
- {
- "id": "2b028f8c-0a4c-4a3a-9e2b-14b1c2401c6d",
- "name": "If collection exists",
- "type": "n8n-nodes-base.if",
- "position": [
- 120,
- 0
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "2104b862-667c-4a34-8888-9cb81a2e10f8",
- "operator": {
- "type": "boolean",
- "operation": "true",
- "singleValue": true
- },
- "leftValue": "={{ $json.result.exists }}",
- "rightValue": "true"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "768793f6-391e-4cc9-b637-f32ee2f77156",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 500,
- 340
- ],
- "parameters": {
- "width": 280,
- "height": 200,
- "content": "In the next workflow, we're going to use Qdrant to get the number of images belonging to each crop type defined by `crop_name` (for example, *\"cucumber\"*). \nTo get this information about counts in payload fields, we need to create an index on that field to optimise the resources (it needs to be done once). That's what is happening here"
- },
- "typeVersion": 1
- },
- {
- "id": "0c8896f7-8c57-4add-bc4d-03c4a774bdf1",
- "name": "Payload index on crop_name",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 500,
- 160
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/index",
- "method": "PUT",
- "options": {},
- "jsonBody": "={\n \"field_name\": \"crop_name\",\n \"field_schema\": \"keyword\"\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "342186f6-41bf-46be-9be8-a9b1ca290d55",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -360,
- -360
- ],
- "parameters": {
- "height": 300,
- "content": "Setting up variables\n1) Cloud URL - to connect to Qdrant Cloud (your personal cluster URL)\n2) Collection name in Qdrant\n3) Size of Voyage embeddings (needed for collection creation in Qdrant) \n4) Batch size for batch embedding/batch uploading to Qdrant "
- },
- "typeVersion": 1
- },
- {
- "id": "fae9248c-dbcc-4b6d-b977-0047f120a587",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -100,
- -220
- ],
- "parameters": {
- "content": "In Qdrant, you can create a collection once; if you try to create it two times with the same name, you'll get an error, so I am adding here a check if a collection with this name exists already"
- },
- "typeVersion": 1
- },
- {
- "id": "f7aea242-3d98-4a1c-a98a-986ac2b4928b",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 180,
- 340
- ],
- "parameters": {
- "height": 280,
- "content": "If a collection with the name set up in variables doesn't exist yet, I create an empty one; \n\nCollection will contain [named vectors](https://qdrant.tech/documentation/concepts/vectors/#named-vectors), with a name *\"voyage\"*\nFor these named vectors, I define two parameters:\n1) Vectors size (in our case, Voyage embeddings size)\n2) Similarity metric to compare embeddings: in our case, **\"Cosine\"**.\n"
- },
- "typeVersion": 1
- },
- {
- "id": "b84045c1-f66a-4543-8d42-1e76de0b6e91",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 800,
- -280
- ],
- "parameters": {
- "height": 400,
- "content": "Now it's time to embed & upload to Qdrant our image datasets;\nBoth of them, [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) and [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) were uploaded to our Google Cloud Storage bucket, and in this workflow we're fetching **the crops dataset** (for lands it will be a nearly identical workflow, up to variable names)\n(you should replace it with your image datasets)\n\nDatasets consist of **image URLs**; images are grouped by folders based on their class. For example, we have a system of subfolders like *\"tomato\"* and *\"cucumber\"* for the crops dataset with image URLs of the respective class.\n"
- },
- "typeVersion": 1
- },
- {
- "id": "255dfad8-c545-4d75-bc9c-529aa50447a9",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1080,
- -140
- ],
- "parameters": {
- "height": 240,
- "content": "Google Storage node returns **mediaLink**, which can be used directly for downloading images; however, we just need a public image URL so that Voyage API can process it; so here we construct this public link and extract a crop name from the folder in which image was stored (for example, *\"cucumber\"*)\n"
- },
- "typeVersion": 1
- },
- {
- "id": "a6acce75-cce0-4de3-bc64-37592c97359b",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1600,
- -80
- ],
- "parameters": {
- "height": 180,
- "content": "I regroup images into batches of `batchSize` size and, to make batch upload to Qdrant possible, generate UUIDs to use them as batch [point IDs](https://qdrant.tech/documentation/concepts/points/#point-ids) (Qdrant doesn't set up id's for the user; users have to choose them themselves)"
- },
- "typeVersion": 1
- },
- {
- "id": "cab3cc83-b50c-41f4-8d51-59e04bba5556",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1340,
- -60
- ],
- "parameters": {
- "content": "Since we build anomaly detection based on the crops dataset, to test it properly, I didn't upload to Qdrant pictures of tomatoes at all; I filter them out here"
- },
- "typeVersion": 1
- },
- {
- "id": "e5cdcce5-efdc-41f2-9796-656bd345f783",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1860,
- -100
- ],
- "parameters": {
- "height": 200,
- "content": "Since Voyage API requires a [specific json structure](https://docs.voyageai.com/reference/multimodal-embeddings-api) for batch embeddings, as does [Qdrant's API for uploading points in batches](https://api.qdrant.tech/api-reference/points/upsert-points), I am adapting the structure of jsons\n\n[NB] - [payload = meta data in Qdrant]"
- },
- "typeVersion": 1
- },
- {
- "id": "a7f15c44-3d5c-4b43-bfb2-94fe27a32071",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2120,
- 20
- ],
- "parameters": {
- "width": 180,
- "height": 80,
- "content": "Embedding images with Voyage model (mind `input_type`)"
- },
- "typeVersion": 1
- },
- {
- "id": "01b92e7e-d954-4d58-85b1-109c336546c4",
- "name": "Filtering out tomato to test anomalies",
- "type": "n8n-nodes-base.filter",
- "position": [
- 1340,
- 160
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "f7953ae2-5333-4805-abe5-abf6da645c5e",
- "operator": {
- "type": "string",
- "operation": "notEquals"
- },
- "leftValue": "={{ $json.cropName }}",
- "rightValue": "tomato"
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "8d564817-885e-453a-a087-900b34b84d9c",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1160,
- -280
- ],
- "parameters": {
- "width": 440,
- "height": 460,
- "content": "## Batch Uploading Dataset to Qdrant \n### This template imports dataset images from storage, creates embeddings for them in batches, and uploads them to Qdrant in batches. In this particular template, we work with [crops dataset](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification). However, it's analogous to [lands dataset](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification), and in general, it's adaptable to any dataset consisting of image URLs (as the following pipelines are).\n\n* First, check for an existing Qdrant collection to use; otherwise, create it here. Additionally, when creating the collection, we'll create a [payload index](https://qdrant.tech/documentation/concepts/indexing/#payload-index), which is required for a particular type of Qdrant requests we will use later.\n* Next, import all (dataset) images from Google Storage but keep only non-tomato-related ones (for anomaly detection testing).\n* Create (per batch) embeddings for all imported images using the Voyage AI multimodal embeddings API.\n* Finally, upload the resulting embeddings and image descriptors to Qdrant via batch uploading."
- },
- "typeVersion": 1
- },
- {
- "id": "0233d3d0-bbdf-4d5b-a366-53cbfa4b6f9c",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -860,
- 360
- ],
- "parameters": {
- "color": 4,
- "width": 540,
- "height": 420,
- "content": "### For anomaly detection\n**1. This is the first pipeline to upload (crops) dataset to Qdrant's collection.**\n2. The second pipeline is to set up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.\n3. The third is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.\n\n### For KNN (k nearest neighbours) classification\n**1. This is the first pipeline to upload (lands) dataset to Qdrant's collection.**\n2. The second is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.\n\n### To recreate both\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) and [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) datasets from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "27776c4a-3bf9-4704-9c13-345b75ffacc0",
- "connections": {
- "Embed crop image": {
- "main": [
- [
- {
- "node": "Batch Upload to Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google Cloud Storage": {
- "main": [
- [
- {
- "node": "Get fields for Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "If collection exists": {
- "main": [
- [
- {
- "node": "Google Cloud Storage",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Create Qdrant Collection",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get fields for Qdrant": {
- "main": [
- [
- {
- "node": "Filtering out tomato to test anomalies",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Batch Upload to Qdrant": {
- "main": [
- []
- ]
- },
- "Create Qdrant Collection": {
- "main": [
- [
- {
- "node": "Payload index on crop_name",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant cluster variables": {
- "main": [
- [
- {
- "node": "Check Qdrant Collection Existence",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Payload index on crop_name": {
- "main": [
- [
- {
- "node": "Google Cloud Storage",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Batches in the API's format": {
- "main": [
- [
- {
- "node": "Embed crop image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Check Qdrant Collection Existence": {
- "main": [
- [
- {
- "node": "If collection exists",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Qdrant cluster variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Filtering out tomato to test anomalies": {
- "main": [
- [
- {
- "node": "Split in batches, generate uuids for Qdrant points",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split in batches, generate uuids for Qdrant points": {
- "main": [
- [
- {
- "node": "Batches in the API's format",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].txt b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].txt
new file mode 100644
index 0000000..de84941
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [1_3 anomaly][1_2 KNN].txt
@@ -0,0 +1,688 @@
+{
+"id": "pPtCy6qPfEv1qNRn",
+"meta": {
+"instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
+},
+"name": "[1/3 - anomaly detection] [1/2 - KNN classification] Batch upload dataset to Qdrant (crops dataset)",
+"tags": [
+{
+"id": "n3zAUYFhdqtjhcLf",
+"name": "qdrant",
+"createdAt": "2024-12-10T11:56:59.987Z",
+"updatedAt": "2024-12-10T11:56:59.987Z"
+}
+],
+"nodes": [
+{
+"id": "53831410-b4f3-4374-8bdd-c2a33cd873cb",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-640,
+0
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "e303ccea-c0e0-4fe5-bd31-48380a0e438f",
+"name": "Google Cloud Storage",
+"type": "n8n-nodes-base.googleCloudStorage",
+"position": [
+820,
+160
+],
+"parameters": {
+"resource": "object",
+"returnAll": true,
+"bucketName": "n8n-qdrant-demo",
+"listFilters": {
+"prefix": "agricultural-crops"
+},
+"requestOptions": {}
+},
+"credentials": {
+"googleCloudStorageOAuth2Api": {
+"id": "fn0sr7grtfprVQvL",
+"name": "Google Cloud Storage account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "737bdb15-61cf-48eb-96af-569eb5986ee8",
+"name": "Get fields for Qdrant",
+"type": "n8n-nodes-base.set",
+"position": [
+1080,
+160
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "10d9147f-1c0c-4357-8413-3130829c2e24",
+"name": "=publicLink",
+"type": "string",
+"value": "=https://storage.googleapis.com/{{ $json.bucket }}/{{ $json.selfLink.split('/').splice(-1) }}"
+},
+{
+"id": "ff9e6a0b-e47a-4550-a13b-465507c75f8f",
+"name": "cropName",
+"type": "string",
+"value": "={{ $json.id.split('/').slice(-3, -2)[0].toLowerCase()}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "2b18ed0c-38d3-49e9-be3d-4f7b35f4d9e5",
+"name": "Qdrant cluster variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-360,
+0
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
+},
+{
+"id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
+"name": "collectionName",
+"type": "string",
+"value": "=agricultural-crops"
+},
+{
+"id": "33581e0a-307f-4380-9533-615791096de7",
+"name": "VoyageEmbeddingsDim",
+"type": "number",
+"value": 1024
+},
+{
+"id": "6e390343-2cd2-4559-aba9-82b13acb7f52",
+"name": "batchSize",
+"type": "number",
+"value": 4
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "f88d290e-3311-4322-b2a5-1350fc1f8768",
+"name": "Embed crop image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2120,
+160
+],
+"parameters": {
+"url": "https://api.voyageai.com/v1/multimodalembeddings",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"inputs\": $json.batchVoyage,\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "Vb0RNVDnIHmgnZOP",
+"name": "Voyage API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "250c6a8d-f545-4037-8069-c834437bbe15",
+"name": "Create Qdrant Collection",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+320,
+160
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}",
+"method": "PUT",
+"options": {},
+"jsonBody": "={{\n{\n \"vectors\": {\n \"voyage\": { \n \"size\": $('Qdrant cluster variables').first().json.VoyageEmbeddingsDim, \n \"distance\": \"Cosine\" \n } \n }\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "20b612ff-4794-43ef-bf45-008a16a2f30f",
+"name": "Check Qdrant Collection Existence",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-100,
+0
+],
+"parameters": {
+"url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/exists",
+"options": {},
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "c067740b-5de3-452e-a614-bf14985a73a0",
+"name": "Batches in the API's format",
+"type": "n8n-nodes-base.set",
+"position": [
+1860,
+160
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "f14db112-6f15-4405-aa47-8cb56bb8ae7a",
+"name": "=batchVoyage",
+"type": "array",
+"value": "={{ $json.batch.map(item => ({ \"content\": ([{\"type\": \"image_url\", \"image_url\": item[\"publicLink\"]}])}))}}"
+},
+{
+"id": "3885fd69-66f5-4435-86a4-b80eaa568ac1",
+"name": "=batchPayloadQdrant",
+"type": "array",
+"value": "={{ $json.batch.map(item => ({\"crop_name\":item[\"cropName\"], \"image_path\":item[\"publicLink\"]})) }}"
+},
+{
+"id": "8ea7a91e-af27-49cb-9a29-41dae15c4e33",
+"name": "uuids",
+"type": "array",
+"value": "={{ $json.uuids }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "bf9a9532-db64-4c02-b91d-47e708ded4d3",
+"name": "Batch Upload to Qdrant",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2320,
+160
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points",
+"method": "PUT",
+"options": {},
+"jsonBody": "={{\n{\n \"batch\": {\n \"ids\" : $('Batches in the API\\'s format').item.json.uuids,\n \"vectors\": {\"voyage\": $json.data.map(item => item[\"embedding\"]) },\n \"payloads\": $('Batches in the API\\'s format').item.json.batchPayloadQdrant\n }\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "3c30373f-c84c-405f-bb84-ec8b4c7419f4",
+"name": "Split in batches, generate uuids for Qdrant points",
+"type": "n8n-nodes-base.code",
+"position": [
+1600,
+160
+],
+"parameters": {
+"language": "python",
+"pythonCode": "import uuid\n\ncrops = [item.json for item in _input.all()]\nbatch_size = int(_('Qdrant cluster variables').first()['json']['batchSize'])\n\ndef split_into_batches_add_uuids(array, batch_size):\n return [\n {\n \"batch\": array[i:i + batch_size],\n \"uuids\": [str(uuid.uuid4()) for j in range(len(array[i:i + batch_size]))]\n }\n for i in range(0, len(array), batch_size)\n ]\n\n# Split crops into batches\nbatched_crops = split_into_batches_add_uuids(crops, batch_size)\n\nreturn batched_crops"
+},
+"typeVersion": 2
+},
+{
+"id": "2b028f8c-0a4c-4a3a-9e2b-14b1c2401c6d",
+"name": "If collection exists",
+"type": "n8n-nodes-base.if",
+"position": [
+120,
+0
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "2104b862-667c-4a34-8888-9cb81a2e10f8",
+"operator": {
+"type": "boolean",
+"operation": "true",
+"singleValue": true
+},
+"leftValue": "={{ $json.result.exists }}",
+"rightValue": "true"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "768793f6-391e-4cc9-b637-f32ee2f77156",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+500,
+340
+],
+"parameters": {
+"width": 280,
+"height": 200,
+"content": "In the next workflow, we're going to use Qdrant to get the number of images belonging to each crop type defined by `crop_name` (for example, *\"cucumber\"*). \nTo get this information about counts in payload fields, we need to create an index on that field to optimise the resources (it needs to be done once). That's what is happening here"
+},
+"typeVersion": 1
+},
+{
+"id": "0c8896f7-8c57-4add-bc4d-03c4a774bdf1",
+"name": "Payload index on crop_name",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+500,
+160
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/index",
+"method": "PUT",
+"options": {},
+"jsonBody": "={\n \"field_name\": \"crop_name\",\n \"field_schema\": \"keyword\"\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "342186f6-41bf-46be-9be8-a9b1ca290d55",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-360,
+-360
+],
+"parameters": {
+"height": 300,
+"content": "Setting up variables\n1) Cloud URL - to connect to Qdrant Cloud (your personal cluster URL)\n2) Collection name in Qdrant\n3) Size of Voyage embeddings (needed for collection creation in Qdrant) \n4) Batch size for batch embedding/batch uploading to Qdrant "
+},
+"typeVersion": 1
+},
+{
+"id": "fae9248c-dbcc-4b6d-b977-0047f120a587",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-100,
+-220
+],
+"parameters": {
+"content": "In Qdrant, you can create a collection once; if you try to create it two times with the same name, you'll get an error, so I am adding here a check if a collection with this name exists already"
+},
+"typeVersion": 1
+},
+{
+"id": "f7aea242-3d98-4a1c-a98a-986ac2b4928b",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+180,
+340
+],
+"parameters": {
+"height": 280,
+"content": "If a collection with the name set up in variables doesn't exist yet, I create an empty one; \n\nCollection will contain [named vectors](https://qdrant.tech/documentation/concepts/vectors/#named-vectors), with a name *\"voyage\"*\nFor these named vectors, I define two parameters:\n1) Vectors size (in our case, Voyage embeddings size)\n2) Similarity metric to compare embeddings: in our case, **\"Cosine\"**.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "b84045c1-f66a-4543-8d42-1e76de0b6e91",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+800,
+-280
+],
+"parameters": {
+"height": 400,
+"content": "Now it's time to embed & upload to Qdrant our image datasets;\nBoth of them, [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) and [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) were uploaded to our Google Cloud Storage bucket, and in this workflow we're fetching **the crops dataset** (for lands it will be a nearly identical workflow, up to variable names)\n(you should replace it with your image datasets)\n\nDatasets consist of **image URLs**; images are grouped by folders based on their class. For example, we have a system of subfolders like *\"tomato\"* and *\"cucumber\"* for the crops dataset with image URLs of the respective class.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "255dfad8-c545-4d75-bc9c-529aa50447a9",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1080,
+-140
+],
+"parameters": {
+"height": 240,
+"content": "Google Storage node returns **mediaLink**, which can be used directly for downloading images; however, we just need a public image URL so that Voyage API can process it; so here we construct this public link and extract a crop name from the folder in which image was stored (for example, *\"cucumber\"*)\n"
+},
+"typeVersion": 1
+},
+{
+"id": "a6acce75-cce0-4de3-bc64-37592c97359b",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1600,
+-80
+],
+"parameters": {
+"height": 180,
+"content": "I regroup images into batches of `batchSize` size and, to make batch upload to Qdrant possible, generate UUIDs to use them as batch [point IDs](https://qdrant.tech/documentation/concepts/points/#point-ids) (Qdrant doesn't set up id's for the user; users have to choose them themselves)"
+},
+"typeVersion": 1
+},
+{
+"id": "cab3cc83-b50c-41f4-8d51-59e04bba5556",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1340,
+-60
+],
+"parameters": {
+"content": "Since we build anomaly detection based on the crops dataset, to test it properly, I didn't upload to Qdrant pictures of tomatoes at all; I filter them out here"
+},
+"typeVersion": 1
+},
+{
+"id": "e5cdcce5-efdc-41f2-9796-656bd345f783",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1860,
+-100
+],
+"parameters": {
+"height": 200,
+"content": "Since Voyage API requires a [specific json structure](https://docs.voyageai.com/reference/multimodal-embeddings-api) for batch embeddings, as does [Qdrant's API for uploading points in batches](https://api.qdrant.tech/api-reference/points/upsert-points), I am adapting the structure of jsons\n\n[NB] - [payload = meta data in Qdrant]"
+},
+"typeVersion": 1
+},
+{
+"id": "a7f15c44-3d5c-4b43-bfb2-94fe27a32071",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2120,
+20
+],
+"parameters": {
+"width": 180,
+"height": 80,
+"content": "Embedding images with Voyage model (mind `input_type`)"
+},
+"typeVersion": 1
+},
+{
+"id": "01b92e7e-d954-4d58-85b1-109c336546c4",
+"name": "Filtering out tomato to test anomalies",
+"type": "n8n-nodes-base.filter",
+"position": [
+1340,
+160
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "f7953ae2-5333-4805-abe5-abf6da645c5e",
+"operator": {
+"type": "string",
+"operation": "notEquals"
+},
+"leftValue": "={{ $json.cropName }}",
+"rightValue": "tomato"
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "8d564817-885e-453a-a087-900b34b84d9c",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1160,
+-280
+],
+"parameters": {
+"width": 440,
+"height": 460,
+"content": "## Batch Uploading Dataset to Qdrant \n### This template imports dataset images from storage, creates embeddings for them in batches, and uploads them to Qdrant in batches. In this particular template, we work with [crops dataset](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification). However, it's analogous to [lands dataset](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification), and in general, it's adaptable to any dataset consisting of image URLs (as the following pipelines are).\n\n* First, check for an existing Qdrant collection to use; otherwise, create it here. Additionally, when creating the collection, we'll create a [payload index](https://qdrant.tech/documentation/concepts/indexing/#payload-index), which is required for a particular type of Qdrant requests we will use later.\n* Next, import all (dataset) images from Google Storage but keep only non-tomato-related ones (for anomaly detection testing).\n* Create (per batch) embeddings for all imported images using the Voyage AI multimodal embeddings API.\n* Finally, upload the resulting embeddings and image descriptors to Qdrant via batch uploading."
+},
+"typeVersion": 1
+},
+{
+"id": "0233d3d0-bbdf-4d5b-a366-53cbfa4b6f9c",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-860,
+360
+],
+"parameters": {
+"color": 4,
+"width": 540,
+"height": 420,
+"content": "### For anomaly detection\n**1. This is the first pipeline to upload (crops) dataset to Qdrant's collection.**\n2. The second pipeline is to set up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.\n3. The third is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.\n\n### For KNN (k nearest neighbours) classification\n**1. This is the first pipeline to upload (lands) dataset to Qdrant's collection.**\n2. The second is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.\n\n### To recreate both\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) and [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) datasets from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "27776c4a-3bf9-4704-9c13-345b75ffacc0",
+"connections": {
+"Embed crop image": {
+"main": [
+[
+{
+"node": "Batch Upload to Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Cloud Storage": {
+"main": [
+[
+{
+"node": "Get fields for Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"If collection exists": {
+"main": [
+[
+{
+"node": "Google Cloud Storage",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Create Qdrant Collection",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get fields for Qdrant": {
+"main": [
+[
+{
+"node": "Filtering out tomato to test anomalies",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Batch Upload to Qdrant": {
+"main": [
+[]
+]
+},
+"Create Qdrant Collection": {
+"main": [
+[
+{
+"node": "Payload index on crop_name",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Qdrant cluster variables": {
+"main": [
+[
+{
+"node": "Check Qdrant Collection Existence",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Payload index on crop_name": {
+"main": [
+[
+{
+"node": "Google Cloud Storage",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Batches in the API's format": {
+"main": [
+[
+{
+"node": "Embed crop image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Check Qdrant Collection Existence": {
+"main": [
+[
+{
+"node": "If collection exists",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Qdrant cluster variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Filtering out tomato to test anomalies": {
+"main": [
+[
+{
+"node": "Split in batches, generate uuids for Qdrant points",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split in batches, generate uuids for Qdrant points": {
+"main": [
+[
+{
+"node": "Batches in the API's format",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).json b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).json
deleted file mode 100644
index 541e862..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).json
+++ /dev/null
@@ -1,544 +0,0 @@
-{
- "id": "itzURpN5wbUNOXOw",
- "meta": {
- "instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
- },
- "name": "[2/2] KNN classifier (lands dataset)",
- "tags": [
- {
- "id": "QN7etptCmdcGIpkS",
- "name": "classifier",
- "createdAt": "2024-12-08T22:08:15.968Z",
- "updatedAt": "2024-12-09T19:25:04.113Z"
- }
- ],
- "nodes": [
- {
- "id": "33373ccb-164e-431c-8a9a-d68668fc70be",
- "name": "Embed image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -140,
- -240
- ],
- "parameters": {
- "url": "https://api.voyageai.com/v1/multimodalembeddings",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "Vb0RNVDnIHmgnZOP",
- "name": "Voyage API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "58adecfa-45c7-4928-b850-053ea6f3b1c5",
- "name": "Query Qdrant",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 440,
- -240
- ],
- "parameters": {
- "url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.ImageEmbedding,\n \"using\": \"voyage\",\n \"limit\": $json.limitKNN,\n \"with_payload\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "258026b7-2dda-4165-bfe1-c4163b9caf78",
- "name": "Majority Vote",
- "type": "n8n-nodes-base.code",
- "position": [
- 840,
- -240
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "from collections import Counter\n\ninput_json = _input.all()[0]\npoints = input_json['json']['result']['points']\nmajority_vote_two_most_common = Counter([point[\"payload\"][\"landscape_name\"] for point in points]).most_common(2)\n\nreturn [{\n \"json\": {\n \"result\": majority_vote_two_most_common \n }\n}]\n"
- },
- "typeVersion": 2
- },
- {
- "id": "e83e7a0c-cb36-46d0-8908-86ee1bddf638",
- "name": "Increase limitKNN",
- "type": "n8n-nodes-base.set",
- "position": [
- 1240,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "0b5d257b-1b27-48bc-bec2-78649bc844cc",
- "name": "limitKNN",
- "type": "number",
- "value": "={{ $('Propagate loop variables').item.json.limitKNN + 5}}"
- },
- {
- "id": "afee4bb3-f78b-4355-945d-3776e33337a4",
- "name": "ImageEmbedding",
- "type": "array",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.ImageEmbedding }}"
- },
- {
- "id": "701ed7ba-d112-4699-a611-c0c134757a6c",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.qdrantCloudURL }}"
- },
- {
- "id": "f5612f78-e7d8-4124-9c3a-27bd5870c9bf",
- "name": "collectionName",
- "type": "string",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.collectionName }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "8edbff53-cba6-4491-9d5e-bac7ad6db418",
- "name": "Propagate loop variables",
- "type": "n8n-nodes-base.set",
- "position": [
- 640,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "880838bf-2be2-4f5f-9417-974b3cbee163",
- "name": "=limitKNN",
- "type": "number",
- "value": "={{ $json.result.points.length}}"
- },
- {
- "id": "5fff2bea-f644-4fd9-ad04-afbecd19a5bc",
- "name": "result",
- "type": "object",
- "value": "={{ $json.result }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "6fad4cc0-f02c-429d-aa4e-0d69ebab9d65",
- "name": "Image Test URL",
- "type": "n8n-nodes-base.set",
- "position": [
- -320,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
- "name": "imageURL",
- "type": "string",
- "value": "={{ $json.query.imageURL }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "f02e79e2-32c8-4af0-8bf9-281119b23cc0",
- "name": "Return class",
- "type": "n8n-nodes-base.set",
- "position": [
- 1240,
- 0
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "bd8ca541-8758-4551-b667-1de373231364",
- "name": "class",
- "type": "string",
- "value": "={{ $json.result[0][0] }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "83ca90fb-d5d5-45f4-8957-4363a4baf8ed",
- "name": "Check tie",
- "type": "n8n-nodes-base.if",
- "position": [
- 1040,
- -240
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "980663f6-9d7d-4e88-87b9-02030882472c",
- "operator": {
- "type": "number",
- "operation": "gt"
- },
- "leftValue": "={{ $json.result.length }}",
- "rightValue": 1
- },
- {
- "id": "9f46fdeb-0f89-4010-99af-624c1c429d6a",
- "operator": {
- "type": "number",
- "operation": "equals"
- },
- "leftValue": "={{ $json.result[0][1] }}",
- "rightValue": "={{ $json.result[1][1] }}"
- },
- {
- "id": "c59bc4fe-6821-4639-8595-fdaf4194c1e1",
- "operator": {
- "type": "number",
- "operation": "lte"
- },
- "leftValue": "={{ $('Propagate loop variables').item.json.limitKNN }}",
- "rightValue": 100
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "847ced21-4cfd-45d8-98fa-b578adc054d6",
- "name": "Qdrant variables + embedding + KNN neigbours",
- "type": "n8n-nodes-base.set",
- "position": [
- 120,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "de66070d-5e74-414e-8af7-d094cbc26f62",
- "name": "ImageEmbedding",
- "type": "array",
- "value": "={{ $json.data[0].embedding }}"
- },
- {
- "id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
- },
- {
- "id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
- "name": "collectionName",
- "type": "string",
- "value": "=land-use"
- },
- {
- "id": "db37e18d-340b-4624-84f6-df993af866d6",
- "name": "limitKNN",
- "type": "number",
- "value": "=10"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "d1bc4edc-37d2-43ac-8d8b-560453e68d1f",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -940,
- -120
- ],
- "parameters": {
- "color": 6,
- "width": 320,
- "height": 540,
- "content": "Here we're classifying existing types of satellite imagery of land types:\n- 'agricultural',\n- 'airplane',\n- 'baseballdiamond',\n- 'beach',\n- 'buildings',\n- 'chaparral',\n- 'denseresidential',\n- 'forest',\n- 'freeway',\n- 'golfcourse',\n- 'harbor',\n- 'intersection',\n- 'mediumresidential',\n- 'mobilehomepark',\n- 'overpass',\n- 'parkinglot',\n- 'river',\n- 'runway',\n- 'sparseresidential',\n- 'storagetanks',\n- 'tenniscourt'\n"
- },
- "typeVersion": 1
- },
- {
- "id": "13560a31-3c72-43b8-9635-3f9ca11f23c9",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- -460
- ],
- "parameters": {
- "color": 6,
- "content": "I tested this KNN classifier on a whole `test` set of a dataset (it's not a part of the collection, only `validation` + `train` parts). Accuracy of classification on `test` is **93.24%**, no fine-tuning, no metric learning."
- },
- "typeVersion": 1
- },
- {
- "id": "8c9dcbcb-a1ad-430f-b7dd-e19b5645b0f6",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- -520,
- -240
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "b36fb270-2101-45e9-bb5c-06c4e07b769c",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1080,
- -520
- ],
- "parameters": {
- "width": 460,
- "height": 380,
- "content": "## KNN classification workflow-tool\n### This n8n template takes an image URL (as anomaly detection tool does), and as output, it returns a class of the object on the image (out of land types list)\n\n* An image URL is received via the Execute Workflow Trigger, which is then sent to the Voyage.ai Multimodal Embeddings API to fetch its embedding.\n* The image's embedding vector is then used to query Qdrant, returning a set of X similar images with pre-labeled classes.\n* Majority voting is done for classes of neighbouring images.\n* A loop is used to resolve scenarios where there is a tie in Majority Voting (for example, we have 5 \"forest\" and 5 \"beach\"), and we increase the number of neighbours to retrieve.\n* When the loop finally resolves, the identified class is returned to the calling workflow."
- },
- "typeVersion": 1
- },
- {
- "id": "51ece7fc-fd85-4d20-ae26-4df2d3893251",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 120,
- -40
- ],
- "parameters": {
- "height": 200,
- "content": "Variables define another Qdrant's collection with landscapes (uploaded similarly as the crops collection, don't forget to switch it with your data) + amount of neighbours **limitKNN** in the database we'll use for an input image classification."
- },
- "typeVersion": 1
- },
- {
- "id": "7aad5904-eb0b-4389-9d47-cc91780737ba",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -180,
- -60
- ],
- "parameters": {
- "height": 80,
- "content": "Similarly to anomaly detection tool, we're embedding input image with the Voyage model"
- },
- "typeVersion": 1
- },
- {
- "id": "d3702707-ee4a-481f-82ca-d9386f5b7c8a",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 440,
- -500
- ],
- "parameters": {
- "width": 740,
- "height": 200,
- "content": "## Tie loop\nHere we're [querying](https://api.qdrant.tech/api-reference/search/query-points) Qdrant, getting **limitKNN** nearest neighbours to our image <*Query Qdrant node*>, parsing their classes from payloads (images were pre-labeled & uploaded with their labels to Qdrant) & calculating the most frequent class name <*Majority Vote node*>. If there is a tie <*check tie node*> in 2 most common classes, for example, we have 5 \"forest\" and 5 \"harbor\", we repeat the procedure with the number of neighbours increased by 5 <*propagate loop variables node* and *increase limitKNN node*>.\nIf there is no tie, or we have already checked 100 neighbours, we exit the loop <*check tie node*> and return the class-answer."
- },
- "typeVersion": 1
- },
- {
- "id": "d26911bb-0442-4adc-8511-7cec2d232393",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1240,
- 160
- ],
- "parameters": {
- "height": 80,
- "content": "Here, we extract the name of the input image class decided by the Majority Vote\n"
- },
- "typeVersion": 1
- },
- {
- "id": "84ffc859-1d5c-4063-9051-3587f30a0017",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- 80
- ],
- "parameters": {
- "color": 4,
- "width": 540,
- "height": 260,
- "content": "### KNN (k nearest neighbours) classification\n1. The first pipeline is uploading (lands) dataset to Qdrant's collection.\n2. **This is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.**\n\n### To recreate it\nYou'll have to upload [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {
- "Execute Workflow Trigger": [
- {
- "json": {
- "query": {
- "imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/land-use/images_train_test_val/test/buildings/buildings_000323.png"
- }
- }
- }
- ]
- },
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "c8cfe732-fd78-4985-9540-ed8cb2de7ef3",
- "connections": {
- "Check tie": {
- "main": [
- [
- {
- "node": "Increase limitKNN",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Return class",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Embed image": {
- "main": [
- [
- {
- "node": "Qdrant variables + embedding + KNN neigbours",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Query Qdrant": {
- "main": [
- [
- {
- "node": "Propagate loop variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Majority Vote": {
- "main": [
- [
- {
- "node": "Check tie",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Image Test URL": {
- "main": [
- [
- {
- "node": "Embed image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Increase limitKNN": {
- "main": [
- [
- {
- "node": "Query Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Image Test URL",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Propagate loop variables": {
- "main": [
- [
- {
- "node": "Majority Vote",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant variables + embedding + KNN neigbours": {
- "main": [
- [
- {
- "node": "Query Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).txt b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).txt
new file mode 100644
index 0000000..1606a3e
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN] (1).txt
@@ -0,0 +1,544 @@
+{
+"id": "itzURpN5wbUNOXOw",
+"meta": {
+"instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
+},
+"name": "[2/2] KNN classifier (lands dataset)",
+"tags": [
+{
+"id": "QN7etptCmdcGIpkS",
+"name": "classifier",
+"createdAt": "2024-12-08T22:08:15.968Z",
+"updatedAt": "2024-12-09T19:25:04.113Z"
+}
+],
+"nodes": [
+{
+"id": "33373ccb-164e-431c-8a9a-d68668fc70be",
+"name": "Embed image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-140,
+-240
+],
+"parameters": {
+"url": "https://api.voyageai.com/v1/multimodalembeddings",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "Vb0RNVDnIHmgnZOP",
+"name": "Voyage API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "58adecfa-45c7-4928-b850-053ea6f3b1c5",
+"name": "Query Qdrant",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+440,
+-240
+],
+"parameters": {
+"url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.ImageEmbedding,\n \"using\": \"voyage\",\n \"limit\": $json.limitKNN,\n \"with_payload\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "258026b7-2dda-4165-bfe1-c4163b9caf78",
+"name": "Majority Vote",
+"type": "n8n-nodes-base.code",
+"position": [
+840,
+-240
+],
+"parameters": {
+"language": "python",
+"pythonCode": "from collections import Counter\n\ninput_json = _input.all()[0]\npoints = input_json['json']['result']['points']\nmajority_vote_two_most_common = Counter([point[\"payload\"][\"landscape_name\"] for point in points]).most_common(2)\n\nreturn [{\n \"json\": {\n \"result\": majority_vote_two_most_common \n }\n}]\n"
+},
+"typeVersion": 2
+},
+{
+"id": "e83e7a0c-cb36-46d0-8908-86ee1bddf638",
+"name": "Increase limitKNN",
+"type": "n8n-nodes-base.set",
+"position": [
+1240,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "0b5d257b-1b27-48bc-bec2-78649bc844cc",
+"name": "limitKNN",
+"type": "number",
+"value": "={{ $('Propagate loop variables').item.json.limitKNN + 5}}"
+},
+{
+"id": "afee4bb3-f78b-4355-945d-3776e33337a4",
+"name": "ImageEmbedding",
+"type": "array",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.ImageEmbedding }}"
+},
+{
+"id": "701ed7ba-d112-4699-a611-c0c134757a6c",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.qdrantCloudURL }}"
+},
+{
+"id": "f5612f78-e7d8-4124-9c3a-27bd5870c9bf",
+"name": "collectionName",
+"type": "string",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.collectionName }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "8edbff53-cba6-4491-9d5e-bac7ad6db418",
+"name": "Propagate loop variables",
+"type": "n8n-nodes-base.set",
+"position": [
+640,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "880838bf-2be2-4f5f-9417-974b3cbee163",
+"name": "=limitKNN",
+"type": "number",
+"value": "={{ $json.result.points.length}}"
+},
+{
+"id": "5fff2bea-f644-4fd9-ad04-afbecd19a5bc",
+"name": "result",
+"type": "object",
+"value": "={{ $json.result }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "6fad4cc0-f02c-429d-aa4e-0d69ebab9d65",
+"name": "Image Test URL",
+"type": "n8n-nodes-base.set",
+"position": [
+-320,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
+"name": "imageURL",
+"type": "string",
+"value": "={{ $json.query.imageURL }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "f02e79e2-32c8-4af0-8bf9-281119b23cc0",
+"name": "Return class",
+"type": "n8n-nodes-base.set",
+"position": [
+1240,
+0
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "bd8ca541-8758-4551-b667-1de373231364",
+"name": "class",
+"type": "string",
+"value": "={{ $json.result[0][0] }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "83ca90fb-d5d5-45f4-8957-4363a4baf8ed",
+"name": "Check tie",
+"type": "n8n-nodes-base.if",
+"position": [
+1040,
+-240
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "980663f6-9d7d-4e88-87b9-02030882472c",
+"operator": {
+"type": "number",
+"operation": "gt"
+},
+"leftValue": "={{ $json.result.length }}",
+"rightValue": 1
+},
+{
+"id": "9f46fdeb-0f89-4010-99af-624c1c429d6a",
+"operator": {
+"type": "number",
+"operation": "equals"
+},
+"leftValue": "={{ $json.result[0][1] }}",
+"rightValue": "={{ $json.result[1][1] }}"
+},
+{
+"id": "c59bc4fe-6821-4639-8595-fdaf4194c1e1",
+"operator": {
+"type": "number",
+"operation": "lte"
+},
+"leftValue": "={{ $('Propagate loop variables').item.json.limitKNN }}",
+"rightValue": 100
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "847ced21-4cfd-45d8-98fa-b578adc054d6",
+"name": "Qdrant variables + embedding + KNN neigbours",
+"type": "n8n-nodes-base.set",
+"position": [
+120,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "de66070d-5e74-414e-8af7-d094cbc26f62",
+"name": "ImageEmbedding",
+"type": "array",
+"value": "={{ $json.data[0].embedding }}"
+},
+{
+"id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
+},
+{
+"id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
+"name": "collectionName",
+"type": "string",
+"value": "=land-use"
+},
+{
+"id": "db37e18d-340b-4624-84f6-df993af866d6",
+"name": "limitKNN",
+"type": "number",
+"value": "=10"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "d1bc4edc-37d2-43ac-8d8b-560453e68d1f",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-940,
+-120
+],
+"parameters": {
+"color": 6,
+"width": 320,
+"height": 540,
+"content": "Here we're classifying existing types of satellite imagery of land types:\n- 'agricultural',\n- 'airplane',\n- 'baseballdiamond',\n- 'beach',\n- 'buildings',\n- 'chaparral',\n- 'denseresidential',\n- 'forest',\n- 'freeway',\n- 'golfcourse',\n- 'harbor',\n- 'intersection',\n- 'mediumresidential',\n- 'mobilehomepark',\n- 'overpass',\n- 'parkinglot',\n- 'river',\n- 'runway',\n- 'sparseresidential',\n- 'storagetanks',\n- 'tenniscourt'\n"
+},
+"typeVersion": 1
+},
+{
+"id": "13560a31-3c72-43b8-9635-3f9ca11f23c9",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+-460
+],
+"parameters": {
+"color": 6,
+"content": "I tested this KNN classifier on a whole `test` set of a dataset (it's not a part of the collection, only `validation` + `train` parts). Accuracy of classification on `test` is **93.24%**, no fine-tuning, no metric learning."
+},
+"typeVersion": 1
+},
+{
+"id": "8c9dcbcb-a1ad-430f-b7dd-e19b5645b0f6",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-520,
+-240
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "b36fb270-2101-45e9-bb5c-06c4e07b769c",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1080,
+-520
+],
+"parameters": {
+"width": 460,
+"height": 380,
+"content": "## KNN classification workflow-tool\n### This n8n template takes an image URL (as anomaly detection tool does), and as output, it returns a class of the object on the image (out of land types list)\n\n* An image URL is received via the Execute Workflow Trigger, which is then sent to the Voyage.ai Multimodal Embeddings API to fetch its embedding.\n* The image's embedding vector is then used to query Qdrant, returning a set of X similar images with pre-labeled classes.\n* Majority voting is done for classes of neighbouring images.\n* A loop is used to resolve scenarios where there is a tie in Majority Voting (for example, we have 5 \"forest\" and 5 \"beach\"), and we increase the number of neighbours to retrieve.\n* When the loop finally resolves, the identified class is returned to the calling workflow."
+},
+"typeVersion": 1
+},
+{
+"id": "51ece7fc-fd85-4d20-ae26-4df2d3893251",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+120,
+-40
+],
+"parameters": {
+"height": 200,
+"content": "Variables define another Qdrant's collection with landscapes (uploaded similarly as the crops collection, don't forget to switch it with your data) + amount of neighbours **limitKNN** in the database we'll use for an input image classification."
+},
+"typeVersion": 1
+},
+{
+"id": "7aad5904-eb0b-4389-9d47-cc91780737ba",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-180,
+-60
+],
+"parameters": {
+"height": 80,
+"content": "Similarly to anomaly detection tool, we're embedding input image with the Voyage model"
+},
+"typeVersion": 1
+},
+{
+"id": "d3702707-ee4a-481f-82ca-d9386f5b7c8a",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+440,
+-500
+],
+"parameters": {
+"width": 740,
+"height": 200,
+"content": "## Tie loop\nHere we're [querying](https://api.qdrant.tech/api-reference/search/query-points) Qdrant, getting **limitKNN** nearest neighbours to our image <*Query Qdrant node*>, parsing their classes from payloads (images were pre-labeled & uploaded with their labels to Qdrant) & calculating the most frequent class name <*Majority Vote node*>. If there is a tie <*check tie node*> in 2 most common classes, for example, we have 5 \"forest\" and 5 \"harbor\", we repeat the procedure with the number of neighbours increased by 5 <*propagate loop variables node* and *increase limitKNN node*>.\nIf there is no tie, or we have already checked 100 neighbours, we exit the loop <*check tie node*> and return the class-answer."
+},
+"typeVersion": 1
+},
+{
+"id": "d26911bb-0442-4adc-8511-7cec2d232393",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1240,
+160
+],
+"parameters": {
+"height": 80,
+"content": "Here, we extract the name of the input image class decided by the Majority Vote\n"
+},
+"typeVersion": 1
+},
+{
+"id": "84ffc859-1d5c-4063-9051-3587f30a0017",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+80
+],
+"parameters": {
+"color": 4,
+"width": 540,
+"height": 260,
+"content": "### KNN (k nearest neighbours) classification\n1. The first pipeline is uploading (lands) dataset to Qdrant's collection.\n2. **This is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.**\n\n### To recreate it\nYou'll have to upload [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {
+"Execute Workflow Trigger": [
+{
+"json": {
+"query": {
+"imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/land-use/images_train_test_val/test/buildings/buildings_000323.png"
+}
+}
+}
+]
+},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "c8cfe732-fd78-4985-9540-ed8cb2de7ef3",
+"connections": {
+"Check tie": {
+"main": [
+[
+{
+"node": "Increase limitKNN",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Return class",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Embed image": {
+"main": [
+[
+{
+"node": "Qdrant variables + embedding + KNN neigbours",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Query Qdrant": {
+"main": [
+[
+{
+"node": "Propagate loop variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Majority Vote": {
+"main": [
+[
+{
+"node": "Check tie",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Image Test URL": {
+"main": [
+[
+{
+"node": "Embed image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Increase limitKNN": {
+"main": [
+[
+{
+"node": "Query Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Image Test URL",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Propagate loop variables": {
+"main": [
+[
+{
+"node": "Majority Vote",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Qdrant variables + embedding + KNN neigbours": {
+"main": [
+[
+{
+"node": "Query Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].json b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].json
deleted file mode 100644
index 541e862..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].json
+++ /dev/null
@@ -1,544 +0,0 @@
-{
- "id": "itzURpN5wbUNOXOw",
- "meta": {
- "instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
- },
- "name": "[2/2] KNN classifier (lands dataset)",
- "tags": [
- {
- "id": "QN7etptCmdcGIpkS",
- "name": "classifier",
- "createdAt": "2024-12-08T22:08:15.968Z",
- "updatedAt": "2024-12-09T19:25:04.113Z"
- }
- ],
- "nodes": [
- {
- "id": "33373ccb-164e-431c-8a9a-d68668fc70be",
- "name": "Embed image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -140,
- -240
- ],
- "parameters": {
- "url": "https://api.voyageai.com/v1/multimodalembeddings",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "Vb0RNVDnIHmgnZOP",
- "name": "Voyage API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "58adecfa-45c7-4928-b850-053ea6f3b1c5",
- "name": "Query Qdrant",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 440,
- -240
- ],
- "parameters": {
- "url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.ImageEmbedding,\n \"using\": \"voyage\",\n \"limit\": $json.limitKNN,\n \"with_payload\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "258026b7-2dda-4165-bfe1-c4163b9caf78",
- "name": "Majority Vote",
- "type": "n8n-nodes-base.code",
- "position": [
- 840,
- -240
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "from collections import Counter\n\ninput_json = _input.all()[0]\npoints = input_json['json']['result']['points']\nmajority_vote_two_most_common = Counter([point[\"payload\"][\"landscape_name\"] for point in points]).most_common(2)\n\nreturn [{\n \"json\": {\n \"result\": majority_vote_two_most_common \n }\n}]\n"
- },
- "typeVersion": 2
- },
- {
- "id": "e83e7a0c-cb36-46d0-8908-86ee1bddf638",
- "name": "Increase limitKNN",
- "type": "n8n-nodes-base.set",
- "position": [
- 1240,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "0b5d257b-1b27-48bc-bec2-78649bc844cc",
- "name": "limitKNN",
- "type": "number",
- "value": "={{ $('Propagate loop variables').item.json.limitKNN + 5}}"
- },
- {
- "id": "afee4bb3-f78b-4355-945d-3776e33337a4",
- "name": "ImageEmbedding",
- "type": "array",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.ImageEmbedding }}"
- },
- {
- "id": "701ed7ba-d112-4699-a611-c0c134757a6c",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.qdrantCloudURL }}"
- },
- {
- "id": "f5612f78-e7d8-4124-9c3a-27bd5870c9bf",
- "name": "collectionName",
- "type": "string",
- "value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.collectionName }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "8edbff53-cba6-4491-9d5e-bac7ad6db418",
- "name": "Propagate loop variables",
- "type": "n8n-nodes-base.set",
- "position": [
- 640,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "880838bf-2be2-4f5f-9417-974b3cbee163",
- "name": "=limitKNN",
- "type": "number",
- "value": "={{ $json.result.points.length}}"
- },
- {
- "id": "5fff2bea-f644-4fd9-ad04-afbecd19a5bc",
- "name": "result",
- "type": "object",
- "value": "={{ $json.result }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "6fad4cc0-f02c-429d-aa4e-0d69ebab9d65",
- "name": "Image Test URL",
- "type": "n8n-nodes-base.set",
- "position": [
- -320,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
- "name": "imageURL",
- "type": "string",
- "value": "={{ $json.query.imageURL }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "f02e79e2-32c8-4af0-8bf9-281119b23cc0",
- "name": "Return class",
- "type": "n8n-nodes-base.set",
- "position": [
- 1240,
- 0
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "bd8ca541-8758-4551-b667-1de373231364",
- "name": "class",
- "type": "string",
- "value": "={{ $json.result[0][0] }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "83ca90fb-d5d5-45f4-8957-4363a4baf8ed",
- "name": "Check tie",
- "type": "n8n-nodes-base.if",
- "position": [
- 1040,
- -240
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "980663f6-9d7d-4e88-87b9-02030882472c",
- "operator": {
- "type": "number",
- "operation": "gt"
- },
- "leftValue": "={{ $json.result.length }}",
- "rightValue": 1
- },
- {
- "id": "9f46fdeb-0f89-4010-99af-624c1c429d6a",
- "operator": {
- "type": "number",
- "operation": "equals"
- },
- "leftValue": "={{ $json.result[0][1] }}",
- "rightValue": "={{ $json.result[1][1] }}"
- },
- {
- "id": "c59bc4fe-6821-4639-8595-fdaf4194c1e1",
- "operator": {
- "type": "number",
- "operation": "lte"
- },
- "leftValue": "={{ $('Propagate loop variables').item.json.limitKNN }}",
- "rightValue": 100
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "847ced21-4cfd-45d8-98fa-b578adc054d6",
- "name": "Qdrant variables + embedding + KNN neigbours",
- "type": "n8n-nodes-base.set",
- "position": [
- 120,
- -240
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "de66070d-5e74-414e-8af7-d094cbc26f62",
- "name": "ImageEmbedding",
- "type": "array",
- "value": "={{ $json.data[0].embedding }}"
- },
- {
- "id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
- },
- {
- "id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
- "name": "collectionName",
- "type": "string",
- "value": "=land-use"
- },
- {
- "id": "db37e18d-340b-4624-84f6-df993af866d6",
- "name": "limitKNN",
- "type": "number",
- "value": "=10"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "d1bc4edc-37d2-43ac-8d8b-560453e68d1f",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -940,
- -120
- ],
- "parameters": {
- "color": 6,
- "width": 320,
- "height": 540,
- "content": "Here we're classifying existing types of satellite imagery of land types:\n- 'agricultural',\n- 'airplane',\n- 'baseballdiamond',\n- 'beach',\n- 'buildings',\n- 'chaparral',\n- 'denseresidential',\n- 'forest',\n- 'freeway',\n- 'golfcourse',\n- 'harbor',\n- 'intersection',\n- 'mediumresidential',\n- 'mobilehomepark',\n- 'overpass',\n- 'parkinglot',\n- 'river',\n- 'runway',\n- 'sparseresidential',\n- 'storagetanks',\n- 'tenniscourt'\n"
- },
- "typeVersion": 1
- },
- {
- "id": "13560a31-3c72-43b8-9635-3f9ca11f23c9",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- -460
- ],
- "parameters": {
- "color": 6,
- "content": "I tested this KNN classifier on a whole `test` set of a dataset (it's not a part of the collection, only `validation` + `train` parts). Accuracy of classification on `test` is **93.24%**, no fine-tuning, no metric learning."
- },
- "typeVersion": 1
- },
- {
- "id": "8c9dcbcb-a1ad-430f-b7dd-e19b5645b0f6",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- -520,
- -240
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "b36fb270-2101-45e9-bb5c-06c4e07b769c",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1080,
- -520
- ],
- "parameters": {
- "width": 460,
- "height": 380,
- "content": "## KNN classification workflow-tool\n### This n8n template takes an image URL (as anomaly detection tool does), and as output, it returns a class of the object on the image (out of land types list)\n\n* An image URL is received via the Execute Workflow Trigger, which is then sent to the Voyage.ai Multimodal Embeddings API to fetch its embedding.\n* The image's embedding vector is then used to query Qdrant, returning a set of X similar images with pre-labeled classes.\n* Majority voting is done for classes of neighbouring images.\n* A loop is used to resolve scenarios where there is a tie in Majority Voting (for example, we have 5 \"forest\" and 5 \"beach\"), and we increase the number of neighbours to retrieve.\n* When the loop finally resolves, the identified class is returned to the calling workflow."
- },
- "typeVersion": 1
- },
- {
- "id": "51ece7fc-fd85-4d20-ae26-4df2d3893251",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 120,
- -40
- ],
- "parameters": {
- "height": 200,
- "content": "Variables define another Qdrant's collection with landscapes (uploaded similarly as the crops collection, don't forget to switch it with your data) + amount of neighbours **limitKNN** in the database we'll use for an input image classification."
- },
- "typeVersion": 1
- },
- {
- "id": "7aad5904-eb0b-4389-9d47-cc91780737ba",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -180,
- -60
- ],
- "parameters": {
- "height": 80,
- "content": "Similarly to anomaly detection tool, we're embedding input image with the Voyage model"
- },
- "typeVersion": 1
- },
- {
- "id": "d3702707-ee4a-481f-82ca-d9386f5b7c8a",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 440,
- -500
- ],
- "parameters": {
- "width": 740,
- "height": 200,
- "content": "## Tie loop\nHere we're [querying](https://api.qdrant.tech/api-reference/search/query-points) Qdrant, getting **limitKNN** nearest neighbours to our image <*Query Qdrant node*>, parsing their classes from payloads (images were pre-labeled & uploaded with their labels to Qdrant) & calculating the most frequent class name <*Majority Vote node*>. If there is a tie <*check tie node*> in 2 most common classes, for example, we have 5 \"forest\" and 5 \"harbor\", we repeat the procedure with the number of neighbours increased by 5 <*propagate loop variables node* and *increase limitKNN node*>.\nIf there is no tie, or we have already checked 100 neighbours, we exit the loop <*check tie node*> and return the class-answer."
- },
- "typeVersion": 1
- },
- {
- "id": "d26911bb-0442-4adc-8511-7cec2d232393",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1240,
- 160
- ],
- "parameters": {
- "height": 80,
- "content": "Here, we extract the name of the input image class decided by the Majority Vote\n"
- },
- "typeVersion": 1
- },
- {
- "id": "84ffc859-1d5c-4063-9051-3587f30a0017",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -520,
- 80
- ],
- "parameters": {
- "color": 4,
- "width": 540,
- "height": 260,
- "content": "### KNN (k nearest neighbours) classification\n1. The first pipeline is uploading (lands) dataset to Qdrant's collection.\n2. **This is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.**\n\n### To recreate it\nYou'll have to upload [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {
- "Execute Workflow Trigger": [
- {
- "json": {
- "query": {
- "imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/land-use/images_train_test_val/test/buildings/buildings_000323.png"
- }
- }
- }
- ]
- },
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "c8cfe732-fd78-4985-9540-ed8cb2de7ef3",
- "connections": {
- "Check tie": {
- "main": [
- [
- {
- "node": "Increase limitKNN",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Return class",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Embed image": {
- "main": [
- [
- {
- "node": "Qdrant variables + embedding + KNN neigbours",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Query Qdrant": {
- "main": [
- [
- {
- "node": "Propagate loop variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Majority Vote": {
- "main": [
- [
- {
- "node": "Check tie",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Image Test URL": {
- "main": [
- [
- {
- "node": "Embed image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Increase limitKNN": {
- "main": [
- [
- {
- "node": "Query Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Image Test URL",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Propagate loop variables": {
- "main": [
- [
- {
- "node": "Majority Vote",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant variables + embedding + KNN neigbours": {
- "main": [
- [
- {
- "node": "Query Qdrant",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].txt b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].txt
new file mode 100644
index 0000000..1606a3e
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_2 KNN].txt
@@ -0,0 +1,544 @@
+{
+"id": "itzURpN5wbUNOXOw",
+"meta": {
+"instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
+},
+"name": "[2/2] KNN classifier (lands dataset)",
+"tags": [
+{
+"id": "QN7etptCmdcGIpkS",
+"name": "classifier",
+"createdAt": "2024-12-08T22:08:15.968Z",
+"updatedAt": "2024-12-09T19:25:04.113Z"
+}
+],
+"nodes": [
+{
+"id": "33373ccb-164e-431c-8a9a-d68668fc70be",
+"name": "Embed image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-140,
+-240
+],
+"parameters": {
+"url": "https://api.voyageai.com/v1/multimodalembeddings",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "Vb0RNVDnIHmgnZOP",
+"name": "Voyage API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "58adecfa-45c7-4928-b850-053ea6f3b1c5",
+"name": "Query Qdrant",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+440,
+-240
+],
+"parameters": {
+"url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.ImageEmbedding,\n \"using\": \"voyage\",\n \"limit\": $json.limitKNN,\n \"with_payload\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "258026b7-2dda-4165-bfe1-c4163b9caf78",
+"name": "Majority Vote",
+"type": "n8n-nodes-base.code",
+"position": [
+840,
+-240
+],
+"parameters": {
+"language": "python",
+"pythonCode": "from collections import Counter\n\ninput_json = _input.all()[0]\npoints = input_json['json']['result']['points']\nmajority_vote_two_most_common = Counter([point[\"payload\"][\"landscape_name\"] for point in points]).most_common(2)\n\nreturn [{\n \"json\": {\n \"result\": majority_vote_two_most_common \n }\n}]\n"
+},
+"typeVersion": 2
+},
+{
+"id": "e83e7a0c-cb36-46d0-8908-86ee1bddf638",
+"name": "Increase limitKNN",
+"type": "n8n-nodes-base.set",
+"position": [
+1240,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "0b5d257b-1b27-48bc-bec2-78649bc844cc",
+"name": "limitKNN",
+"type": "number",
+"value": "={{ $('Propagate loop variables').item.json.limitKNN + 5}}"
+},
+{
+"id": "afee4bb3-f78b-4355-945d-3776e33337a4",
+"name": "ImageEmbedding",
+"type": "array",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.ImageEmbedding }}"
+},
+{
+"id": "701ed7ba-d112-4699-a611-c0c134757a6c",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.qdrantCloudURL }}"
+},
+{
+"id": "f5612f78-e7d8-4124-9c3a-27bd5870c9bf",
+"name": "collectionName",
+"type": "string",
+"value": "={{ $('Qdrant variables + embedding + KNN neigbours').first().json.collectionName }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "8edbff53-cba6-4491-9d5e-bac7ad6db418",
+"name": "Propagate loop variables",
+"type": "n8n-nodes-base.set",
+"position": [
+640,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "880838bf-2be2-4f5f-9417-974b3cbee163",
+"name": "=limitKNN",
+"type": "number",
+"value": "={{ $json.result.points.length}}"
+},
+{
+"id": "5fff2bea-f644-4fd9-ad04-afbecd19a5bc",
+"name": "result",
+"type": "object",
+"value": "={{ $json.result }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "6fad4cc0-f02c-429d-aa4e-0d69ebab9d65",
+"name": "Image Test URL",
+"type": "n8n-nodes-base.set",
+"position": [
+-320,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
+"name": "imageURL",
+"type": "string",
+"value": "={{ $json.query.imageURL }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "f02e79e2-32c8-4af0-8bf9-281119b23cc0",
+"name": "Return class",
+"type": "n8n-nodes-base.set",
+"position": [
+1240,
+0
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "bd8ca541-8758-4551-b667-1de373231364",
+"name": "class",
+"type": "string",
+"value": "={{ $json.result[0][0] }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "83ca90fb-d5d5-45f4-8957-4363a4baf8ed",
+"name": "Check tie",
+"type": "n8n-nodes-base.if",
+"position": [
+1040,
+-240
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "980663f6-9d7d-4e88-87b9-02030882472c",
+"operator": {
+"type": "number",
+"operation": "gt"
+},
+"leftValue": "={{ $json.result.length }}",
+"rightValue": 1
+},
+{
+"id": "9f46fdeb-0f89-4010-99af-624c1c429d6a",
+"operator": {
+"type": "number",
+"operation": "equals"
+},
+"leftValue": "={{ $json.result[0][1] }}",
+"rightValue": "={{ $json.result[1][1] }}"
+},
+{
+"id": "c59bc4fe-6821-4639-8595-fdaf4194c1e1",
+"operator": {
+"type": "number",
+"operation": "lte"
+},
+"leftValue": "={{ $('Propagate loop variables').item.json.limitKNN }}",
+"rightValue": 100
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "847ced21-4cfd-45d8-98fa-b578adc054d6",
+"name": "Qdrant variables + embedding + KNN neigbours",
+"type": "n8n-nodes-base.set",
+"position": [
+120,
+-240
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "de66070d-5e74-414e-8af7-d094cbc26f62",
+"name": "ImageEmbedding",
+"type": "array",
+"value": "={{ $json.data[0].embedding }}"
+},
+{
+"id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
+},
+{
+"id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
+"name": "collectionName",
+"type": "string",
+"value": "=land-use"
+},
+{
+"id": "db37e18d-340b-4624-84f6-df993af866d6",
+"name": "limitKNN",
+"type": "number",
+"value": "=10"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "d1bc4edc-37d2-43ac-8d8b-560453e68d1f",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-940,
+-120
+],
+"parameters": {
+"color": 6,
+"width": 320,
+"height": 540,
+"content": "Here we're classifying existing types of satellite imagery of land types:\n- 'agricultural',\n- 'airplane',\n- 'baseballdiamond',\n- 'beach',\n- 'buildings',\n- 'chaparral',\n- 'denseresidential',\n- 'forest',\n- 'freeway',\n- 'golfcourse',\n- 'harbor',\n- 'intersection',\n- 'mediumresidential',\n- 'mobilehomepark',\n- 'overpass',\n- 'parkinglot',\n- 'river',\n- 'runway',\n- 'sparseresidential',\n- 'storagetanks',\n- 'tenniscourt'\n"
+},
+"typeVersion": 1
+},
+{
+"id": "13560a31-3c72-43b8-9635-3f9ca11f23c9",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+-460
+],
+"parameters": {
+"color": 6,
+"content": "I tested this KNN classifier on a whole `test` set of a dataset (it's not a part of the collection, only `validation` + `train` parts). Accuracy of classification on `test` is **93.24%**, no fine-tuning, no metric learning."
+},
+"typeVersion": 1
+},
+{
+"id": "8c9dcbcb-a1ad-430f-b7dd-e19b5645b0f6",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-520,
+-240
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "b36fb270-2101-45e9-bb5c-06c4e07b769c",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1080,
+-520
+],
+"parameters": {
+"width": 460,
+"height": 380,
+"content": "## KNN classification workflow-tool\n### This n8n template takes an image URL (as anomaly detection tool does), and as output, it returns a class of the object on the image (out of land types list)\n\n* An image URL is received via the Execute Workflow Trigger, which is then sent to the Voyage.ai Multimodal Embeddings API to fetch its embedding.\n* The image's embedding vector is then used to query Qdrant, returning a set of X similar images with pre-labeled classes.\n* Majority voting is done for classes of neighbouring images.\n* A loop is used to resolve scenarios where there is a tie in Majority Voting (for example, we have 5 \"forest\" and 5 \"beach\"), and we increase the number of neighbours to retrieve.\n* When the loop finally resolves, the identified class is returned to the calling workflow."
+},
+"typeVersion": 1
+},
+{
+"id": "51ece7fc-fd85-4d20-ae26-4df2d3893251",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+120,
+-40
+],
+"parameters": {
+"height": 200,
+"content": "Variables define another Qdrant's collection with landscapes (uploaded similarly as the crops collection, don't forget to switch it with your data) + amount of neighbours **limitKNN** in the database we'll use for an input image classification."
+},
+"typeVersion": 1
+},
+{
+"id": "7aad5904-eb0b-4389-9d47-cc91780737ba",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-180,
+-60
+],
+"parameters": {
+"height": 80,
+"content": "Similarly to anomaly detection tool, we're embedding input image with the Voyage model"
+},
+"typeVersion": 1
+},
+{
+"id": "d3702707-ee4a-481f-82ca-d9386f5b7c8a",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+440,
+-500
+],
+"parameters": {
+"width": 740,
+"height": 200,
+"content": "## Tie loop\nHere we're [querying](https://api.qdrant.tech/api-reference/search/query-points) Qdrant, getting **limitKNN** nearest neighbours to our image <*Query Qdrant node*>, parsing their classes from payloads (images were pre-labeled & uploaded with their labels to Qdrant) & calculating the most frequent class name <*Majority Vote node*>. If there is a tie <*check tie node*> in 2 most common classes, for example, we have 5 \"forest\" and 5 \"harbor\", we repeat the procedure with the number of neighbours increased by 5 <*propagate loop variables node* and *increase limitKNN node*>.\nIf there is no tie, or we have already checked 100 neighbours, we exit the loop <*check tie node*> and return the class-answer."
+},
+"typeVersion": 1
+},
+{
+"id": "d26911bb-0442-4adc-8511-7cec2d232393",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1240,
+160
+],
+"parameters": {
+"height": 80,
+"content": "Here, we extract the name of the input image class decided by the Majority Vote\n"
+},
+"typeVersion": 1
+},
+{
+"id": "84ffc859-1d5c-4063-9051-3587f30a0017",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-520,
+80
+],
+"parameters": {
+"color": 4,
+"width": 540,
+"height": 260,
+"content": "### KNN (k nearest neighbours) classification\n1. The first pipeline is uploading (lands) dataset to Qdrant's collection.\n2. **This is the KNN classifier tool, which takes any image as input and classifies it based on queries to the Qdrant (lands) collection.**\n\n### To recreate it\nYou'll have to upload [lands](https://www.kaggle.com/datasets/apollo2506/landuse-scene-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {
+"Execute Workflow Trigger": [
+{
+"json": {
+"query": {
+"imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/land-use/images_train_test_val/test/buildings/buildings_000323.png"
+}
+}
+}
+]
+},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "c8cfe732-fd78-4985-9540-ed8cb2de7ef3",
+"connections": {
+"Check tie": {
+"main": [
+[
+{
+"node": "Increase limitKNN",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Return class",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Embed image": {
+"main": [
+[
+{
+"node": "Qdrant variables + embedding + KNN neigbours",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Query Qdrant": {
+"main": [
+[
+{
+"node": "Propagate loop variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Majority Vote": {
+"main": [
+[
+{
+"node": "Check tie",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Image Test URL": {
+"main": [
+[
+{
+"node": "Embed image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Increase limitKNN": {
+"main": [
+[
+{
+"node": "Query Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Image Test URL",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Propagate loop variables": {
+"main": [
+[
+{
+"node": "Majority Vote",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Qdrant variables + embedding + KNN neigbours": {
+"main": [
+[
+{
+"node": "Query Qdrant",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].json b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].json
deleted file mode 100644
index 2971290..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].json
+++ /dev/null
@@ -1,1250 +0,0 @@
-{
- "id": "m9aACcHqydEbH4nR",
- "meta": {
- "instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
- },
- "name": "[2/3] Set up medoids (2 types) for anomaly detection (crops dataset)",
- "tags": [
- {
- "id": "spMntyrlE9ydvWFA",
- "name": "anomaly-detection",
- "createdAt": "2024-12-08T22:05:15.945Z",
- "updatedAt": "2024-12-09T12:50:19.287Z"
- }
- ],
- "nodes": [
- {
- "id": "edaa871e-2b79-400e-8328-333d250bfdd2",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -660,
- -220
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "ebd964de-faa4-4dc0-9245-cc9154b9ce02",
- "name": "Total Points in Collection",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 180,
- -220
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').item.json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').item.json.collectionName }}/points/count",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"exact\": true\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "b51f6344-d090-4341-a908-581b78664b07",
- "name": "Cluster Distance Matrix",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1200,
- -360
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/search/matrix/offsets",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"sample\": $json.maxClusterSize,\n \"limit\": $json.maxClusterSize,\n \"using\": \"voyage\",\n \"filter\": {\n \"must\": {\n \"key\": \"crop_name\",\n \"match\": { \"value\": $json.cropName }\n }\n }\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "bebe5249-b138-4d7a-84b8-51eaed4331b8",
- "name": "Scipy Sparse Matrix",
- "type": "n8n-nodes-base.code",
- "position": [
- 1460,
- -360
- ],
- "parameters": {
- "mode": "runOnceForEachItem",
- "language": "python",
- "pythonCode": "from scipy.sparse import coo_array\n\ncluster = _input.item.json['result']\n\nscores = list(cluster['scores'])\noffsets_row = list(cluster['offsets_row'])\noffsets_col = list(cluster['offsets_col'])\n\ncluster_matrix = coo_array((scores, (offsets_row, offsets_col)))\nthe_most_similar_to_others = cluster_matrix.sum(axis=1).argmax()\n\nreturn {\n \"json\": {\n \"medoid_id\": cluster[\"ids\"][the_most_similar_to_others]\n }\n}\n"
- },
- "typeVersion": 2
- },
- {
- "id": "006c38bb-a271-40e1-9c5b-5a0a29ea96de",
- "name": "Set medoid id",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2000,
- -680
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"payload\": {\"is_medoid\": true},\n \"points\": [$json.medoid_id]\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "aeeccfc5-67bf-4047-8a5a-8830e4fc87e8",
- "name": "Get Medoid Vector",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2000,
- -360
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"ids\": [$json.medoid_id],\n \"with_vector\": true,\n \"with_payload\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "11fe54d5-9dc8-49ce-9e3f-1103ace0a3d5",
- "name": "Prepare for Searching Threshold",
- "type": "n8n-nodes-base.set",
- "position": [
- 2240,
- -360
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "6faa5949-968c-42bf-8ce8-cf2403566eba",
- "name": "oppositeOfCenterVector",
- "type": "array",
- "value": "={{ $json.result[0].vector.voyage.map(value => value * -1)}}"
- },
- {
- "id": "84eb42be-2ea5-4a76-9c76-f21a962360a3",
- "name": "cropName",
- "type": "string",
- "value": "={{ $json.result[0].payload.crop_name }}"
- },
- {
- "id": "b68d2e42-0dde-4875-bb59-056f29b6ac0a",
- "name": "centerId",
- "type": "string",
- "value": "={{ $json.result[0].id }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "4051b488-2e2e-4d33-9cc9-e1403c9173ed",
- "name": "Searching Score",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2500,
- -360
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.oppositeOfCenterVector,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $json.cropName }\n }\n ]\n },\n \"limit\": $('Medoids Variables').first().json.furthestFromCenter,\n \"with_payload\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "1c6cb6ee-ce3a-4d1a-b1b4-1e59e9a8f5b6",
- "name": "Threshold Score",
- "type": "n8n-nodes-base.set",
- "position": [
- 2760,
- -360
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "579a2ee4-0ab2-4fde-909a-01166624c9d8",
- "name": "thresholdScore",
- "type": "number",
- "value": "={{ $json.result.points.last().score * -1 }}"
- },
- {
- "id": "11eab775-f709-40a9-b0fe-d1059b67de05",
- "name": "centerId",
- "type": "string",
- "value": "={{ $('Prepare for Searching Threshold').item.json.centerId }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "1bab1b9e-7b80-4ef3-8e3d-be4874792e58",
- "name": "Set medoid threshold score",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2940,
- -360
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"payload\": {\"is_medoid_cluster_threshold\": $json.thresholdScore },\n \"points\": [$json.centerId]\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "cd5af197-4d79-49c2-aba6-a20571bd5c2e",
- "name": "Split Out1",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 860,
- 80
- ],
- "parameters": {
- "options": {
- "destinationFieldName": ""
- },
- "fieldToSplitOut": "['text anchors']"
- },
- "typeVersion": 1
- },
- {
- "id": "956c126c-8bd6-4390-8704-3f0a5a2ce479",
- "name": "Merge",
- "type": "n8n-nodes-base.merge",
- "position": [
- 1200,
- -80
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "fieldsToMatchString": "cropName"
- },
- "typeVersion": 3
- },
- {
- "id": "54a5d467-4985-49b5-9f13-e6563acf08b3",
- "name": "Textual (visual) crop descriptions",
- "type": "n8n-nodes-base.set",
- "position": [
- 380,
- 80
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "{\"text anchors\": [{\"cropName\": \"pearl_millet(bajra)\", \"cropDescription\": \"pearl_millet(bajra) - Tall stalks with cylindrical, spiked green grain heads.\"},\n{\"cropName\": \"tobacco-plant\", \"cropDescription\": \"tobacco-plant - Broad, oval leaves and small tubular flowers, typically pink or white.\"},\n{\"cropName\": \"cherry\", \"cropDescription\": \"cherry - Small, glossy red fruits on a medium-sized tree with slender branches and serrated leaves.\"},\n{\"cropName\": \"cotton\", \"cropDescription\": \"cotton - Bushy plant with fluffy white fiber-filled pods and lobed green leaves.\"},\n{\"cropName\": \"banana\", \"cropDescription\": \"banana - Tall herbaceous plant with broad, elongated green leaves and hanging bunches of yellow fruits.\"},\n{\"cropName\": \"cucumber\", \"cropDescription\": \"cucumber - Creeping vine with yellow flowers and elongated green cylindrical fruits.\"},\n{\"cropName\": \"maize\", \"cropDescription\": \"maize - Tall stalks with broad leaves, tassels at the top, and ears of corn sheathed in husks.\"},\n{\"cropName\": \"wheat\", \"cropDescription\": \"wheat - Slender, upright stalks with narrow green leaves and golden, spiky grain heads.\"},\n{\"cropName\": \"clove\", \"cropDescription\": \"clove - Small tree with oval green leaves and clusters of unopened reddish flower buds.\"},\n{\"cropName\": \"jowar\", \"cropDescription\": \"jowar - Tall grass-like plant with broad leaves and round, compact grain clusters at the top.\"},\n{\"cropName\": \"olive-tree\", \"cropDescription\": \"olive-tree - Medium-sized tree with silvery-green leaves and small oval green or black fruits.\"},\n{\"cropName\": \"soyabean\", \"cropDescription\": \"soyabean - Bushy plant with trifoliate green leaves and small pods containing rounded beans.\"},\n{\"cropName\": \"coffee-plant\", \"cropDescription\": \"coffee-plant - Shrub with shiny dark green leaves and clusters of small white flowers, followed by red berries.\"},\n{\"cropName\": \"rice\", \"cropDescription\": \"rice - Short, water-loving grass with narrow green leaves and drooping golden grain heads.\"},\n{\"cropName\": \"lemon\", \"cropDescription\": \"lemon - Small tree with glossy green leaves and oval yellow fruits.\"},\n{\"cropName\": \"mustard-oil\", \"cropDescription\": \"mustard-oil - Small herbaceous plant with yellow flowers and slender seed pods.\"},\n{\"cropName\": \"vigna-radiati(mung)\", \"cropDescription\": \"vigna-radiati(mung) - Low-growing plant with trifoliate leaves and small green pods containing mung beans.\"},\n{\"cropName\": \"coconut\", \"cropDescription\": \"coconut - Tall palm tree with feathery leaves and large round fibrous fruits.\"},\n{\"cropName\": \"gram\", \"cropDescription\": \"gram - Low bushy plant with feathery leaves and small pods containing round seeds.\"},\n{\"cropName\": \"pineapple\", \"cropDescription\": \"pineapple - Low plant with spiky, sword-shaped leaves and large, spiky golden fruits.\"},\n{\"cropName\": \"sugarcane\", \"cropDescription\": \"sugarcane - Tall, jointed stalks with long narrow leaves and a sweet interior.\"},\n{\"cropName\": \"sunflower\", \"cropDescription\": \"sunflower - Tall plant with rough green leaves and large bright yellow flower heads.\"},\n{\"cropName\": \"chilli\", \"cropDescription\": \"chilli - Small bushy plant with slender green or red elongated fruits.\"},\n{\"cropName\": \"fox_nut(makhana)\", \"cropDescription\": \"fox_nut(makhana) - Aquatic plant with floating round leaves and spiny white seeds.\"},\n{\"cropName\": \"jute\", \"cropDescription\": \"jute - Tall plant with long, straight stalks and narrow green leaves.\"},\n{\"cropName\": \"papaya\", \"cropDescription\": \"papaya - Medium-sized tree with hollow trunk, large lobed leaves, and yellow-orange pear-shaped fruits.\"},\n{\"cropName\": \"tea\", \"cropDescription\": \"tea - Small shrub with glossy dark green leaves and small white flowers.\"},\n{\"cropName\": \"cardamom\", \"cropDescription\": \"cardamom - Low tropical plant with broad leaves and clusters of small, light green pods.\"},\n{\"cropName\": \"almond\", \"cropDescription\": \"almond - Medium-sized tree with serrated leaves and oval green pods containing edible nuts.\"}]}\n"
- },
- "typeVersion": 3.4
- },
- {
- "id": "14c25e76-8a2c-4df8-98ea-b2f31b15fd1f",
- "name": "Embed text",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1460,
- -80
- ],
- "parameters": {
- "url": "https://api.voyageai.com/v1/multimodalembeddings",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": $json.cropDescription\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"query\"\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "Vb0RNVDnIHmgnZOP",
- "name": "Voyage API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "8763db0a-9a92-4ffd-8a40-c7db614b735f",
- "name": "Get Medoid by Text",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 1640,
- -80
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.data[0].embedding,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $('Merge').item.json.cropName }\n }\n ]\n },\n \"limit\": 1,\n \"with_payload\": true,\n \"with_vector\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "5c770ca2-6e1a-4c4b-80e0-dcbeeda43a0f",
- "name": "Set text medoid id",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2000,
- 160
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"payload\": {\"is_text_anchor_medoid\": true},\n \"points\": [$json.result.points[0].id]\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "c08ff472-51ab-4c3d-b9c0-2170fda2ccef",
- "name": "Prepare for Searching Threshold1",
- "type": "n8n-nodes-base.set",
- "position": [
- 2300,
- 80
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "6faa5949-968c-42bf-8ce8-cf2403566eba",
- "name": "oppositeOfCenterVector",
- "type": "array",
- "value": "={{ $json.result.points[0].vector.voyage.map(value => value * -1)}}"
- },
- {
- "id": "84eb42be-2ea5-4a76-9c76-f21a962360a3",
- "name": "cropName",
- "type": "string",
- "value": "={{ $json.result.points[0].payload.crop_name }}"
- },
- {
- "id": "b68d2e42-0dde-4875-bb59-056f29b6ac0a",
- "name": "centerId",
- "type": "string",
- "value": "={{ $json.result.points[0].id }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "84ba4de5-aa9b-43fb-89cb-70db0b3ca334",
- "name": "Threshold Score1",
- "type": "n8n-nodes-base.set",
- "position": [
- 2820,
- 80
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "579a2ee4-0ab2-4fde-909a-01166624c9d8",
- "name": "thresholdScore",
- "type": "number",
- "value": "={{ $json.result.points.last().score * -1 }}"
- },
- {
- "id": "11eab775-f709-40a9-b0fe-d1059b67de05",
- "name": "centerId",
- "type": "string",
- "value": "={{ $('Prepare for Searching Threshold1').item.json.centerId }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "f490d224-38a8-4087-889d-1addb4472471",
- "name": "Searching Text Medoid Score",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 2560,
- 80
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.oppositeOfCenterVector,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $json.cropName }\n }\n ]\n },\n \"limit\": $('Text Medoids Variables').first().json.furthestFromCenter,\n \"with_payload\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "f5035aca-1706-4c8d-bd26-49b3451ae04b",
- "name": "Medoids Variables",
- "type": "n8n-nodes-base.set",
- "position": [
- -140,
- -220
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "5eb23ad2-aacd-468f-9a27-ef2b63e6bd08",
- "name": "furthestFromCenter",
- "type": "number",
- "value": 5
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "c9cad66d-4a76-4092-bfd6-4860493f942a",
- "name": "Text Medoids Variables",
- "type": "n8n-nodes-base.set",
- "position": [
- -140,
- 80
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "5eb23ad2-aacd-468f-9a27-ef2b63e6bd08",
- "name": "furthestFromCenter",
- "type": "number",
- "value": 1
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "ecab63f7-7a72-425a-8f5a-0c707e7f77bc",
- "name": "Qdrant cluster variables",
- "type": "n8n-nodes-base.set",
- "position": [
- -420,
- -220
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
- },
- {
- "id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
- "name": "collectionName",
- "type": "string",
- "value": "=agricultural-crops"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "6e81f0b0-3843-467e-9c93-40026e57fa91",
- "name": "Info About Crop Clusters",
- "type": "n8n-nodes-base.set",
- "position": [
- 600,
- -220
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "5327b254-b703-4a34-a398-f82edb1d6d6b",
- "name": "=cropsNumber",
- "type": "number",
- "value": "={{ $json.result.hits.length }}"
- },
- {
- "id": "79168efa-11b8-4a7b-8851-da9c8cbd700b",
- "name": "maxClusterSize",
- "type": "number",
- "value": "={{ Math.max(...$json.result.hits.map(item => item.count)) }}"
- },
- {
- "id": "e1367cec-9629-4c69-a8d7-3eeae3ac94d3",
- "name": "cropNames",
- "type": "array",
- "value": "={{ $json.result.hits.map(item => item.value)}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "20191c0a-5310-48f2-8be4-1d160f237db2",
- "name": "Crop Counts",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 380,
- -220
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/facet",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"key\": \"crop_name\",\n \"limit\": $json.result.count,\n \"exact\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "a81103bb-6522-49a2-8102-83c7e004b9b3",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1260,
- -340
- ],
- "parameters": {
- "width": 520,
- "height": 240,
- "content": "## Setting Up Medoids for Anomaly Detection\n### Preparatory workflow to set cluster centres and cluster threshold scores, so anomalies can be detected based on these thresholds\nHere, we're using two approaches to set up these centres: the upper branch is the *\"distance matrix approach\"*, and the lower is the *\"multimodal embedding model approach\"*."
- },
- "typeVersion": 1
- },
- {
- "id": "38fc8252-7e27-450d-b09e-59ceaebc5378",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -420,
- -340
- ],
- "parameters": {
- "height": 80,
- "content": "Once again, variables for Qdrant: cluster URL and a collection we're working with"
- },
- "typeVersion": 1
- },
- {
- "id": "2d0e3b52-d382-428c-9b37-870f4c53b8e7",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -140,
- -360
- ],
- "parameters": {
- "height": 100,
- "content": "Which point in the cluster we're using to draw threshold on: the furthest one from center, or the 2nd, ... Xth furthest one;"
- },
- "typeVersion": 1
- },
- {
- "id": "b0b300f3-e2c9-4c36-8a1d-6705932c296c",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 380,
- -500
- ],
- "parameters": {
- "width": 180,
- "height": 240,
- "content": "Here we are getting [facet counts](https://qdrant.tech/documentation/concepts/payload/?q=facet#facet-counts): information which unique values are there behind *\"crop_name\"* payload and how many points have these values (for example, we have 31 *\"cucumber\"* and 29 *\"cotton\"*)"
- },
- "typeVersion": 1
- },
- {
- "id": "0d2584da-5fd0-4830-b329-c78b0debf584",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -140,
- 260
- ],
- "parameters": {
- "height": 120,
- "content": "Which point in the cluster we're using to draw threshold on: the furthest one from center, or the 2nd, ... Xth furthest one;\n"
- },
- "typeVersion": 1
- },
- {
- "id": "f4c98469-d426-415c-916d-1bc442cf6a21",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 120,
- -400
- ],
- "parameters": {
- "height": 140,
- "content": "We need to get the [total amount of points](https://qdrant.tech/documentation/concepts/points/?q=count#counting-points) in Qdrant collection to use it as a `limit` in the *\"Crop Counts\"* node, so we won't lose any information;\n"
- },
- "typeVersion": 1
- },
- {
- "id": "037af9df-34c4-488d-8c89-561ac25247c4",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 600,
- -640
- ],
- "parameters": {
- "width": 220,
- "height": 380,
- "content": "Here we're extracting and gathering all the information about crop clusters, so we can call [Qdrant distance matrix API](https://qdrant.tech/documentation/concepts/explore/?q=distance+#distance-matrix) for each cluster.\nWe're propagating **the biggest cluster size** (of labeled data, in our case all data is labeled; for real use cases don't call distance matrix API if your labeled data is more than a couple of hundreds), **the number of unique crop values** and **unique crop values** themselves. We will run the algorithm once per unique crop cluster (to find it's center and threshold)."
- },
- "typeVersion": 1
- },
- {
- "id": "b4e635e3-233d-4358-ad11-250a2b14a2f7",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 380,
- 260
- ],
- "parameters": {
- "height": 200,
- "content": "Hardcoded descriptions on how each crop usually looks; They were generated with chatGPT, and that can be technically done directly in n8n based on the crop name or a crop picture (we need a good description of how the most normal specimen of a crop looks like)"
- },
- "typeVersion": 1
- },
- {
- "id": "4fda1841-e7e3-4bd2-acf2-ee7338598184",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1200,
- -800
- ],
- "parameters": {
- "height": 400,
- "content": "Calling [distance matrix API](https://qdrant.tech/documentation/concepts/explore/?q=distance+#distance-matrix) once per cluster. \n\n`sample` - how many points we are sampling (here filtered by `crop_name` field, so we are sampling within each cluster, and since we are passing the biggest cluster size to `sample`, we will get all points from each cluster.\n\n`limit` is the number of neighbours distance to which we will see calculated. Since we want all pairwise distances between the points within a cluster, here we're once again setting an upper limit equal to the biggest cluster size; "
- },
- "typeVersion": 1
- },
- {
- "id": "19c4bb6d-abcb-423b-b883-48c779d0307d",
- "name": "Split Out",
- "type": "n8n-nodes-base.splitOut",
- "position": [
- 860,
- -220
- ],
- "parameters": {
- "include": "allOtherFields",
- "options": {
- "destinationFieldName": "cropName"
- },
- "fieldToSplitOut": "cropNames"
- },
- "typeVersion": 1
- },
- {
- "id": "f6d74ced-1998-4dbd-ab04-ca1b6ea409a5",
- "name": "Sticky Note10",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 840,
- -60
- ],
- "parameters": {
- "width": 150,
- "height": 80,
- "content": "Splitting out into each unique crop cluster"
- },
- "typeVersion": 1
- },
- {
- "id": "b3adb2bc-61f5-42ff-bb5d-11faa12189b7",
- "name": "Sticky Note11",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1460,
- -640
- ],
- "parameters": {
- "width": 180,
- "height": 240,
- "content": "Using distance matrix generated by Qdrant and `coo_array` from `scipy`, we're finding a **representative** for each cluster (point which is the most similar to all other points within a cluster, based on the **Cosine** distance)"
- },
- "typeVersion": 1
- },
- {
- "id": "d9d3953e-8b69-4b6a-86f2-b2d2db28d4ad",
- "name": "Sticky Note12",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1200,
- 100
- ],
- "parameters": {
- "height": 280,
- "content": "To find a **representative** with this approach, we:\n1) Embed descriptions of crops with the same Voyage model we used for images (we can do so, since model is multimodal)\n2) For each (crop) cluster, find an image the closest by **Cosine** similarity metric to this embedded description. We will consider it a perfect representative of the cluster"
- },
- "typeVersion": 1
- },
- {
- "id": "8751efd4-d85e-4dc8-86ef-90073d49b6df",
- "name": "Sticky Note13",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1460,
- 100
- ],
- "parameters": {
- "width": 160,
- "height": 140,
- "content": "Embedding descriptions with Voyage model \n[Note] mind `input_type`, it's *\"query\"*"
- },
- "typeVersion": 1
- },
- {
- "id": "652bc70a-4e6f-416a-977b-5d29ae9cb4f0",
- "name": "Sticky Note14",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1640,
- 100
- ],
- "parameters": {
- "height": 260,
- "content": "Find the closest image to the description embeddings (done per cluster)\n[Note] Mind `exact` parameter\n[Note] `limit` is 1 because vector database always returns points sorted by distance from the most similar one to the least\n[Note] `using` parameter is here because our vectors uploaded in the previous pipeline are named *\"voyage\"*."
- },
- "typeVersion": 1
- },
- {
- "id": "a5836982-0de0-4692-883c-267602468ed2",
- "name": "Set text medoid threshold score",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 3000,
- 80
- ],
- "parameters": {
- "url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"payload\": {\"is_text_anchor_medoid_cluster_threshold\": $json.thresholdScore },\n \"points\": [$json.centerId]\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "5354d197-be5e-4add-b721-9e5e3943e53d",
- "name": "Sticky Note15",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1960,
- -460
- ],
- "parameters": {
- "width": 200,
- "height": 80,
- "content": "Fetching vectors of centres by their IDs"
- },
- "typeVersion": 1
- },
- {
- "id": "93043602-92bc-40ac-b967-ddb7289e5d22",
- "name": "Sticky Note16",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2000,
- -820
- ],
- "parameters": {
- "height": 100,
- "content": "Set in Qdrant *\"is_medoid\"* [payloads](https://qdrant.tech/documentation/concepts/payload/) for points which were defined as centres by *\"distance matrix approach\"*"
- },
- "typeVersion": 1
- },
- {
- "id": "cb1364ad-e21c-4336-9a5b-15e80c2ed2f2",
- "name": "Sticky Note17",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2280,
- 260
- ],
- "parameters": {
- "height": 180,
- "content": "Here, we don't have to fetch a vector by point id as in the *\"distance matrix approach\"*, since [an API call in the previous node](https://api.qdrant.tech/api-reference/search/query-points) is able to return vectors stored in Qdrant as a response, while the distance matrix API returns only points IDs."
- },
- "typeVersion": 1
- },
- {
- "id": "6d735a28-a93e-41f1-9889-2557a1dd7aec",
- "name": "Sticky Note18",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1980,
- 320
- ],
- "parameters": {
- "height": 140,
- "content": "Set in Qdrant *\"is_text_anchor_medoid\"* [payloads](https://qdrant.tech/documentation/concepts/payload/) for points which were defined as centres by *\"multimodal embedding model approach\"*."
- },
- "typeVersion": 1
- },
- {
- "id": "7c6796a9-260b-41c0-9ac7-feb5d4d95c19",
- "name": "Sticky Note19",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2240,
- -500
- ],
- "parameters": {
- "width": 440,
- "height": 100,
- "content": "Starting from here, this and the three following nodes are analogous for both methods, with a difference only in variable names. The goal is to find a **class (cluster) threshold score** so we can use it for anomaly detection (for each class).\n"
- },
- "typeVersion": 1
- },
- {
- "id": "5025936d-d49c-4cc1-a675-3bde71627c40",
- "name": "Sticky Note20",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2280,
- -180
- ],
- "parameters": {
- "height": 220,
- "content": "Finding the most dissimilar point to a centre vector (within each class) is equivalent to finding the most similar point to the [opposite](https://mathinsight.org/image/vector_opposite) of a centre vector, aka the centre vector with all coordinates multiplied by -1. It is always true with **Cosine** vector similarity metric (that we're using)."
- },
- "typeVersion": 1
- },
- {
- "id": "fa9026e4-0c92-4755-92a0-5e400b5f04c9",
- "name": "Sticky Note21",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 2580,
- -140
- ],
- "parameters": {
- "width": 520,
- "height": 140,
- "content": "So here, we found the most dissimilar point within the crop class to the class centre (or the Xth dissimilar point, depending on a variable set in the beginning of this pipeline). Our **threshold score** is the similarity score between this point and the class centre. Now we're saving it as meta information of each class centre point. All preparatory work for anomaly detection is done."
- },
- "typeVersion": 1
- },
- {
- "id": "8e172a7c-6865-4daf-9d9c-86e0dba2c0a2",
- "name": "Sticky Note22",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -900,
- -820
- ],
- "parameters": {
- "color": 4,
- "width": 540,
- "height": 300,
- "content": "### For anomaly detection\n1. The first pipeline is uploading (crops) dataset to Qdrant's collection.\n2. **This is the second pipeline, to set up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.**\n3. The third one is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.\n\n### To recreate it\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {},
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "a23fc305-7ecd-4754-b208-2d964d9b1eda",
- "connections": {
- "Merge": {
- "main": [
- [
- {
- "node": "Embed text",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Out": {
- "main": [
- [
- {
- "node": "Cluster Distance Matrix",
- "type": "main",
- "index": 0
- },
- {
- "node": "Merge",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Embed text": {
- "main": [
- [
- {
- "node": "Get Medoid by Text",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Split Out1": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Crop Counts": {
- "main": [
- [
- {
- "node": "Info About Crop Clusters",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Set medoid id": {
- "main": [
- []
- ]
- },
- "Searching Score": {
- "main": [
- [
- {
- "node": "Threshold Score",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Threshold Score": {
- "main": [
- [
- {
- "node": "Set medoid threshold score",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Threshold Score1": {
- "main": [
- [
- {
- "node": "Set text medoid threshold score",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Medoid Vector": {
- "main": [
- [
- {
- "node": "Prepare for Searching Threshold",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Medoids Variables": {
- "main": [
- [
- {
- "node": "Total Points in Collection",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Medoid by Text": {
- "main": [
- [
- {
- "node": "Set text medoid id",
- "type": "main",
- "index": 0
- },
- {
- "node": "Prepare for Searching Threshold1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Scipy Sparse Matrix": {
- "main": [
- [
- {
- "node": "Set medoid id",
- "type": "main",
- "index": 0
- },
- {
- "node": "Get Medoid Vector",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Text Medoids Variables": {
- "main": [
- [
- {
- "node": "Textual (visual) crop descriptions",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Cluster Distance Matrix": {
- "main": [
- [
- {
- "node": "Scipy Sparse Matrix",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Info About Crop Clusters": {
- "main": [
- [
- {
- "node": "Split Out",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Qdrant cluster variables": {
- "main": [
- [
- {
- "node": "Medoids Variables",
- "type": "main",
- "index": 0
- },
- {
- "node": "Text Medoids Variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Total Points in Collection": {
- "main": [
- [
- {
- "node": "Crop Counts",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Searching Text Medoid Score": {
- "main": [
- [
- {
- "node": "Threshold Score1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prepare for Searching Threshold": {
- "main": [
- [
- {
- "node": "Searching Score",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Prepare for Searching Threshold1": {
- "main": [
- [
- {
- "node": "Searching Text Medoid Score",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Qdrant cluster variables",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Textual (visual) crop descriptions": {
- "main": [
- [
- {
- "node": "Split Out1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].txt b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].txt
new file mode 100644
index 0000000..562d039
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [2_3 - anomaly].txt
@@ -0,0 +1,1250 @@
+{
+"id": "m9aACcHqydEbH4nR",
+"meta": {
+"instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
+},
+"name": "[2/3] Set up medoids (2 types) for anomaly detection (crops dataset)",
+"tags": [
+{
+"id": "spMntyrlE9ydvWFA",
+"name": "anomaly-detection",
+"createdAt": "2024-12-08T22:05:15.945Z",
+"updatedAt": "2024-12-09T12:50:19.287Z"
+}
+],
+"nodes": [
+{
+"id": "edaa871e-2b79-400e-8328-333d250bfdd2",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-660,
+-220
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "ebd964de-faa4-4dc0-9245-cc9154b9ce02",
+"name": "Total Points in Collection",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+180,
+-220
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').item.json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').item.json.collectionName }}/points/count",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"exact\": true\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "b51f6344-d090-4341-a908-581b78664b07",
+"name": "Cluster Distance Matrix",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1200,
+-360
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/search/matrix/offsets",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"sample\": $json.maxClusterSize,\n \"limit\": $json.maxClusterSize,\n \"using\": \"voyage\",\n \"filter\": {\n \"must\": {\n \"key\": \"crop_name\",\n \"match\": { \"value\": $json.cropName }\n }\n }\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "bebe5249-b138-4d7a-84b8-51eaed4331b8",
+"name": "Scipy Sparse Matrix",
+"type": "n8n-nodes-base.code",
+"position": [
+1460,
+-360
+],
+"parameters": {
+"mode": "runOnceForEachItem",
+"language": "python",
+"pythonCode": "from scipy.sparse import coo_array\n\ncluster = _input.item.json['result']\n\nscores = list(cluster['scores'])\noffsets_row = list(cluster['offsets_row'])\noffsets_col = list(cluster['offsets_col'])\n\ncluster_matrix = coo_array((scores, (offsets_row, offsets_col)))\nthe_most_similar_to_others = cluster_matrix.sum(axis=1).argmax()\n\nreturn {\n \"json\": {\n \"medoid_id\": cluster[\"ids\"][the_most_similar_to_others]\n }\n}\n"
+},
+"typeVersion": 2
+},
+{
+"id": "006c38bb-a271-40e1-9c5b-5a0a29ea96de",
+"name": "Set medoid id",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2000,
+-680
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"payload\": {\"is_medoid\": true},\n \"points\": [$json.medoid_id]\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "aeeccfc5-67bf-4047-8a5a-8830e4fc87e8",
+"name": "Get Medoid Vector",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2000,
+-360
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"ids\": [$json.medoid_id],\n \"with_vector\": true,\n \"with_payload\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "11fe54d5-9dc8-49ce-9e3f-1103ace0a3d5",
+"name": "Prepare for Searching Threshold",
+"type": "n8n-nodes-base.set",
+"position": [
+2240,
+-360
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "6faa5949-968c-42bf-8ce8-cf2403566eba",
+"name": "oppositeOfCenterVector",
+"type": "array",
+"value": "={{ $json.result[0].vector.voyage.map(value => value * -1)}}"
+},
+{
+"id": "84eb42be-2ea5-4a76-9c76-f21a962360a3",
+"name": "cropName",
+"type": "string",
+"value": "={{ $json.result[0].payload.crop_name }}"
+},
+{
+"id": "b68d2e42-0dde-4875-bb59-056f29b6ac0a",
+"name": "centerId",
+"type": "string",
+"value": "={{ $json.result[0].id }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "4051b488-2e2e-4d33-9cc9-e1403c9173ed",
+"name": "Searching Score",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2500,
+-360
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.oppositeOfCenterVector,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $json.cropName }\n }\n ]\n },\n \"limit\": $('Medoids Variables').first().json.furthestFromCenter,\n \"with_payload\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "1c6cb6ee-ce3a-4d1a-b1b4-1e59e9a8f5b6",
+"name": "Threshold Score",
+"type": "n8n-nodes-base.set",
+"position": [
+2760,
+-360
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "579a2ee4-0ab2-4fde-909a-01166624c9d8",
+"name": "thresholdScore",
+"type": "number",
+"value": "={{ $json.result.points.last().score * -1 }}"
+},
+{
+"id": "11eab775-f709-40a9-b0fe-d1059b67de05",
+"name": "centerId",
+"type": "string",
+"value": "={{ $('Prepare for Searching Threshold').item.json.centerId }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "1bab1b9e-7b80-4ef3-8e3d-be4874792e58",
+"name": "Set medoid threshold score",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2940,
+-360
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"payload\": {\"is_medoid_cluster_threshold\": $json.thresholdScore },\n \"points\": [$json.centerId]\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "cd5af197-4d79-49c2-aba6-a20571bd5c2e",
+"name": "Split Out1",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+860,
+80
+],
+"parameters": {
+"options": {
+"destinationFieldName": ""
+},
+"fieldToSplitOut": "['text anchors']"
+},
+"typeVersion": 1
+},
+{
+"id": "956c126c-8bd6-4390-8704-3f0a5a2ce479",
+"name": "Merge",
+"type": "n8n-nodes-base.merge",
+"position": [
+1200,
+-80
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"fieldsToMatchString": "cropName"
+},
+"typeVersion": 3
+},
+{
+"id": "54a5d467-4985-49b5-9f13-e6563acf08b3",
+"name": "Textual (visual) crop descriptions",
+"type": "n8n-nodes-base.set",
+"position": [
+380,
+80
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "{\"text anchors\": [{\"cropName\": \"pearl_millet(bajra)\", \"cropDescription\": \"pearl_millet(bajra) - Tall stalks with cylindrical, spiked green grain heads.\"},\n{\"cropName\": \"tobacco-plant\", \"cropDescription\": \"tobacco-plant - Broad, oval leaves and small tubular flowers, typically pink or white.\"},\n{\"cropName\": \"cherry\", \"cropDescription\": \"cherry - Small, glossy red fruits on a medium-sized tree with slender branches and serrated leaves.\"},\n{\"cropName\": \"cotton\", \"cropDescription\": \"cotton - Bushy plant with fluffy white fiber-filled pods and lobed green leaves.\"},\n{\"cropName\": \"banana\", \"cropDescription\": \"banana - Tall herbaceous plant with broad, elongated green leaves and hanging bunches of yellow fruits.\"},\n{\"cropName\": \"cucumber\", \"cropDescription\": \"cucumber - Creeping vine with yellow flowers and elongated green cylindrical fruits.\"},\n{\"cropName\": \"maize\", \"cropDescription\": \"maize - Tall stalks with broad leaves, tassels at the top, and ears of corn sheathed in husks.\"},\n{\"cropName\": \"wheat\", \"cropDescription\": \"wheat - Slender, upright stalks with narrow green leaves and golden, spiky grain heads.\"},\n{\"cropName\": \"clove\", \"cropDescription\": \"clove - Small tree with oval green leaves and clusters of unopened reddish flower buds.\"},\n{\"cropName\": \"jowar\", \"cropDescription\": \"jowar - Tall grass-like plant with broad leaves and round, compact grain clusters at the top.\"},\n{\"cropName\": \"olive-tree\", \"cropDescription\": \"olive-tree - Medium-sized tree with silvery-green leaves and small oval green or black fruits.\"},\n{\"cropName\": \"soyabean\", \"cropDescription\": \"soyabean - Bushy plant with trifoliate green leaves and small pods containing rounded beans.\"},\n{\"cropName\": \"coffee-plant\", \"cropDescription\": \"coffee-plant - Shrub with shiny dark green leaves and clusters of small white flowers, followed by red berries.\"},\n{\"cropName\": \"rice\", \"cropDescription\": \"rice - Short, water-loving grass with narrow green leaves and drooping golden grain heads.\"},\n{\"cropName\": \"lemon\", \"cropDescription\": \"lemon - Small tree with glossy green leaves and oval yellow fruits.\"},\n{\"cropName\": \"mustard-oil\", \"cropDescription\": \"mustard-oil - Small herbaceous plant with yellow flowers and slender seed pods.\"},\n{\"cropName\": \"vigna-radiati(mung)\", \"cropDescription\": \"vigna-radiati(mung) - Low-growing plant with trifoliate leaves and small green pods containing mung beans.\"},\n{\"cropName\": \"coconut\", \"cropDescription\": \"coconut - Tall palm tree with feathery leaves and large round fibrous fruits.\"},\n{\"cropName\": \"gram\", \"cropDescription\": \"gram - Low bushy plant with feathery leaves and small pods containing round seeds.\"},\n{\"cropName\": \"pineapple\", \"cropDescription\": \"pineapple - Low plant with spiky, sword-shaped leaves and large, spiky golden fruits.\"},\n{\"cropName\": \"sugarcane\", \"cropDescription\": \"sugarcane - Tall, jointed stalks with long narrow leaves and a sweet interior.\"},\n{\"cropName\": \"sunflower\", \"cropDescription\": \"sunflower - Tall plant with rough green leaves and large bright yellow flower heads.\"},\n{\"cropName\": \"chilli\", \"cropDescription\": \"chilli - Small bushy plant with slender green or red elongated fruits.\"},\n{\"cropName\": \"fox_nut(makhana)\", \"cropDescription\": \"fox_nut(makhana) - Aquatic plant with floating round leaves and spiny white seeds.\"},\n{\"cropName\": \"jute\", \"cropDescription\": \"jute - Tall plant with long, straight stalks and narrow green leaves.\"},\n{\"cropName\": \"papaya\", \"cropDescription\": \"papaya - Medium-sized tree with hollow trunk, large lobed leaves, and yellow-orange pear-shaped fruits.\"},\n{\"cropName\": \"tea\", \"cropDescription\": \"tea - Small shrub with glossy dark green leaves and small white flowers.\"},\n{\"cropName\": \"cardamom\", \"cropDescription\": \"cardamom - Low tropical plant with broad leaves and clusters of small, light green pods.\"},\n{\"cropName\": \"almond\", \"cropDescription\": \"almond - Medium-sized tree with serrated leaves and oval green pods containing edible nuts.\"}]}\n"
+},
+"typeVersion": 3.4
+},
+{
+"id": "14c25e76-8a2c-4df8-98ea-b2f31b15fd1f",
+"name": "Embed text",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1460,
+-80
+],
+"parameters": {
+"url": "https://api.voyageai.com/v1/multimodalembeddings",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": $json.cropDescription\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"query\"\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "Vb0RNVDnIHmgnZOP",
+"name": "Voyage API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8763db0a-9a92-4ffd-8a40-c7db614b735f",
+"name": "Get Medoid by Text",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+1640,
+-80
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.data[0].embedding,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $('Merge').item.json.cropName }\n }\n ]\n },\n \"limit\": 1,\n \"with_payload\": true,\n \"with_vector\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "5c770ca2-6e1a-4c4b-80e0-dcbeeda43a0f",
+"name": "Set text medoid id",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2000,
+160
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"payload\": {\"is_text_anchor_medoid\": true},\n \"points\": [$json.result.points[0].id]\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "c08ff472-51ab-4c3d-b9c0-2170fda2ccef",
+"name": "Prepare for Searching Threshold1",
+"type": "n8n-nodes-base.set",
+"position": [
+2300,
+80
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "6faa5949-968c-42bf-8ce8-cf2403566eba",
+"name": "oppositeOfCenterVector",
+"type": "array",
+"value": "={{ $json.result.points[0].vector.voyage.map(value => value * -1)}}"
+},
+{
+"id": "84eb42be-2ea5-4a76-9c76-f21a962360a3",
+"name": "cropName",
+"type": "string",
+"value": "={{ $json.result.points[0].payload.crop_name }}"
+},
+{
+"id": "b68d2e42-0dde-4875-bb59-056f29b6ac0a",
+"name": "centerId",
+"type": "string",
+"value": "={{ $json.result.points[0].id }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "84ba4de5-aa9b-43fb-89cb-70db0b3ca334",
+"name": "Threshold Score1",
+"type": "n8n-nodes-base.set",
+"position": [
+2820,
+80
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "579a2ee4-0ab2-4fde-909a-01166624c9d8",
+"name": "thresholdScore",
+"type": "number",
+"value": "={{ $json.result.points.last().score * -1 }}"
+},
+{
+"id": "11eab775-f709-40a9-b0fe-d1059b67de05",
+"name": "centerId",
+"type": "string",
+"value": "={{ $('Prepare for Searching Threshold1').item.json.centerId }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "f490d224-38a8-4087-889d-1addb4472471",
+"name": "Searching Text Medoid Score",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+2560,
+80
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.oppositeOfCenterVector,\n \"using\": \"voyage\",\n \"exact\": true,\n \"filter\": {\n \"must\": [\n {\n \"key\": \"crop_name\",\n \"match\": {\"value\": $json.cropName }\n }\n ]\n },\n \"limit\": $('Text Medoids Variables').first().json.furthestFromCenter,\n \"with_payload\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "f5035aca-1706-4c8d-bd26-49b3451ae04b",
+"name": "Medoids Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-140,
+-220
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "5eb23ad2-aacd-468f-9a27-ef2b63e6bd08",
+"name": "furthestFromCenter",
+"type": "number",
+"value": 5
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "c9cad66d-4a76-4092-bfd6-4860493f942a",
+"name": "Text Medoids Variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-140,
+80
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "5eb23ad2-aacd-468f-9a27-ef2b63e6bd08",
+"name": "furthestFromCenter",
+"type": "number",
+"value": 1
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "ecab63f7-7a72-425a-8f5a-0c707e7f77bc",
+"name": "Qdrant cluster variables",
+"type": "n8n-nodes-base.set",
+"position": [
+-420,
+-220
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "58b7384d-fd0c-44aa-9f8e-0306a99be431",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "=https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
+},
+{
+"id": "e34c4d88-b102-43cc-a09e-e0553f2da23a",
+"name": "collectionName",
+"type": "string",
+"value": "=agricultural-crops"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "6e81f0b0-3843-467e-9c93-40026e57fa91",
+"name": "Info About Crop Clusters",
+"type": "n8n-nodes-base.set",
+"position": [
+600,
+-220
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "5327b254-b703-4a34-a398-f82edb1d6d6b",
+"name": "=cropsNumber",
+"type": "number",
+"value": "={{ $json.result.hits.length }}"
+},
+{
+"id": "79168efa-11b8-4a7b-8851-da9c8cbd700b",
+"name": "maxClusterSize",
+"type": "number",
+"value": "={{ Math.max(...$json.result.hits.map(item => item.count)) }}"
+},
+{
+"id": "e1367cec-9629-4c69-a8d7-3eeae3ac94d3",
+"name": "cropNames",
+"type": "array",
+"value": "={{ $json.result.hits.map(item => item.value)}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "20191c0a-5310-48f2-8be4-1d160f237db2",
+"name": "Crop Counts",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+380,
+-220
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/facet",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"key\": \"crop_name\",\n \"limit\": $json.result.count,\n \"exact\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "a81103bb-6522-49a2-8102-83c7e004b9b3",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1260,
+-340
+],
+"parameters": {
+"width": 520,
+"height": 240,
+"content": "## Setting Up Medoids for Anomaly Detection\n### Preparatory workflow to set cluster centres and cluster threshold scores, so anomalies can be detected based on these thresholds\nHere, we're using two approaches to set up these centres: the upper branch is the *\"distance matrix approach\"*, and the lower is the *\"multimodal embedding model approach\"*."
+},
+"typeVersion": 1
+},
+{
+"id": "38fc8252-7e27-450d-b09e-59ceaebc5378",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-420,
+-340
+],
+"parameters": {
+"height": 80,
+"content": "Once again, variables for Qdrant: cluster URL and a collection we're working with"
+},
+"typeVersion": 1
+},
+{
+"id": "2d0e3b52-d382-428c-9b37-870f4c53b8e7",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-140,
+-360
+],
+"parameters": {
+"height": 100,
+"content": "Which point in the cluster we're using to draw threshold on: the furthest one from center, or the 2nd, ... Xth furthest one;"
+},
+"typeVersion": 1
+},
+{
+"id": "b0b300f3-e2c9-4c36-8a1d-6705932c296c",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+380,
+-500
+],
+"parameters": {
+"width": 180,
+"height": 240,
+"content": "Here we are getting [facet counts](https://qdrant.tech/documentation/concepts/payload/?q=facet#facet-counts): information which unique values are there behind *\"crop_name\"* payload and how many points have these values (for example, we have 31 *\"cucumber\"* and 29 *\"cotton\"*)"
+},
+"typeVersion": 1
+},
+{
+"id": "0d2584da-5fd0-4830-b329-c78b0debf584",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-140,
+260
+],
+"parameters": {
+"height": 120,
+"content": "Which point in the cluster we're using to draw threshold on: the furthest one from center, or the 2nd, ... Xth furthest one;\n"
+},
+"typeVersion": 1
+},
+{
+"id": "f4c98469-d426-415c-916d-1bc442cf6a21",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+120,
+-400
+],
+"parameters": {
+"height": 140,
+"content": "We need to get the [total amount of points](https://qdrant.tech/documentation/concepts/points/?q=count#counting-points) in Qdrant collection to use it as a `limit` in the *\"Crop Counts\"* node, so we won't lose any information;\n"
+},
+"typeVersion": 1
+},
+{
+"id": "037af9df-34c4-488d-8c89-561ac25247c4",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+600,
+-640
+],
+"parameters": {
+"width": 220,
+"height": 380,
+"content": "Here we're extracting and gathering all the information about crop clusters, so we can call [Qdrant distance matrix API](https://qdrant.tech/documentation/concepts/explore/?q=distance+#distance-matrix) for each cluster.\nWe're propagating **the biggest cluster size** (of labeled data, in our case all data is labeled; for real use cases don't call distance matrix API if your labeled data is more than a couple of hundreds), **the number of unique crop values** and **unique crop values** themselves. We will run the algorithm once per unique crop cluster (to find it's center and threshold)."
+},
+"typeVersion": 1
+},
+{
+"id": "b4e635e3-233d-4358-ad11-250a2b14a2f7",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+380,
+260
+],
+"parameters": {
+"height": 200,
+"content": "Hardcoded descriptions on how each crop usually looks; They were generated with chatGPT, and that can be technically done directly in n8n based on the crop name or a crop picture (we need a good description of how the most normal specimen of a crop looks like)"
+},
+"typeVersion": 1
+},
+{
+"id": "4fda1841-e7e3-4bd2-acf2-ee7338598184",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1200,
+-800
+],
+"parameters": {
+"height": 400,
+"content": "Calling [distance matrix API](https://qdrant.tech/documentation/concepts/explore/?q=distance+#distance-matrix) once per cluster. \n\n`sample` - how many points we are sampling (here filtered by `crop_name` field, so we are sampling within each cluster, and since we are passing the biggest cluster size to `sample`, we will get all points from each cluster.\n\n`limit` is the number of neighbours distance to which we will see calculated. Since we want all pairwise distances between the points within a cluster, here we're once again setting an upper limit equal to the biggest cluster size; "
+},
+"typeVersion": 1
+},
+{
+"id": "19c4bb6d-abcb-423b-b883-48c779d0307d",
+"name": "Split Out",
+"type": "n8n-nodes-base.splitOut",
+"position": [
+860,
+-220
+],
+"parameters": {
+"include": "allOtherFields",
+"options": {
+"destinationFieldName": "cropName"
+},
+"fieldToSplitOut": "cropNames"
+},
+"typeVersion": 1
+},
+{
+"id": "f6d74ced-1998-4dbd-ab04-ca1b6ea409a5",
+"name": "Sticky Note10",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+840,
+-60
+],
+"parameters": {
+"width": 150,
+"height": 80,
+"content": "Splitting out into each unique crop cluster"
+},
+"typeVersion": 1
+},
+{
+"id": "b3adb2bc-61f5-42ff-bb5d-11faa12189b7",
+"name": "Sticky Note11",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1460,
+-640
+],
+"parameters": {
+"width": 180,
+"height": 240,
+"content": "Using distance matrix generated by Qdrant and `coo_array` from `scipy`, we're finding a **representative** for each cluster (point which is the most similar to all other points within a cluster, based on the **Cosine** distance)"
+},
+"typeVersion": 1
+},
+{
+"id": "d9d3953e-8b69-4b6a-86f2-b2d2db28d4ad",
+"name": "Sticky Note12",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1200,
+100
+],
+"parameters": {
+"height": 280,
+"content": "To find a **representative** with this approach, we:\n1) Embed descriptions of crops with the same Voyage model we used for images (we can do so, since model is multimodal)\n2) For each (crop) cluster, find an image the closest by **Cosine** similarity metric to this embedded description. We will consider it a perfect representative of the cluster"
+},
+"typeVersion": 1
+},
+{
+"id": "8751efd4-d85e-4dc8-86ef-90073d49b6df",
+"name": "Sticky Note13",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1460,
+100
+],
+"parameters": {
+"width": 160,
+"height": 140,
+"content": "Embedding descriptions with Voyage model \n[Note] mind `input_type`, it's *\"query\"*"
+},
+"typeVersion": 1
+},
+{
+"id": "652bc70a-4e6f-416a-977b-5d29ae9cb4f0",
+"name": "Sticky Note14",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1640,
+100
+],
+"parameters": {
+"height": 260,
+"content": "Find the closest image to the description embeddings (done per cluster)\n[Note] Mind `exact` parameter\n[Note] `limit` is 1 because vector database always returns points sorted by distance from the most similar one to the least\n[Note] `using` parameter is here because our vectors uploaded in the previous pipeline are named *\"voyage\"*."
+},
+"typeVersion": 1
+},
+{
+"id": "a5836982-0de0-4692-883c-267602468ed2",
+"name": "Set text medoid threshold score",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+3000,
+80
+],
+"parameters": {
+"url": "={{ $('Qdrant cluster variables').first().json.qdrantCloudURL }}/collections/{{ $('Qdrant cluster variables').first().json.collectionName }}/points/payload",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"payload\": {\"is_text_anchor_medoid_cluster_threshold\": $json.thresholdScore },\n \"points\": [$json.centerId]\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "5354d197-be5e-4add-b721-9e5e3943e53d",
+"name": "Sticky Note15",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1960,
+-460
+],
+"parameters": {
+"width": 200,
+"height": 80,
+"content": "Fetching vectors of centres by their IDs"
+},
+"typeVersion": 1
+},
+{
+"id": "93043602-92bc-40ac-b967-ddb7289e5d22",
+"name": "Sticky Note16",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2000,
+-820
+],
+"parameters": {
+"height": 100,
+"content": "Set in Qdrant *\"is_medoid\"* [payloads](https://qdrant.tech/documentation/concepts/payload/) for points which were defined as centres by *\"distance matrix approach\"*"
+},
+"typeVersion": 1
+},
+{
+"id": "cb1364ad-e21c-4336-9a5b-15e80c2ed2f2",
+"name": "Sticky Note17",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2280,
+260
+],
+"parameters": {
+"height": 180,
+"content": "Here, we don't have to fetch a vector by point id as in the *\"distance matrix approach\"*, since [an API call in the previous node](https://api.qdrant.tech/api-reference/search/query-points) is able to return vectors stored in Qdrant as a response, while the distance matrix API returns only points IDs."
+},
+"typeVersion": 1
+},
+{
+"id": "6d735a28-a93e-41f1-9889-2557a1dd7aec",
+"name": "Sticky Note18",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+1980,
+320
+],
+"parameters": {
+"height": 140,
+"content": "Set in Qdrant *\"is_text_anchor_medoid\"* [payloads](https://qdrant.tech/documentation/concepts/payload/) for points which were defined as centres by *\"multimodal embedding model approach\"*."
+},
+"typeVersion": 1
+},
+{
+"id": "7c6796a9-260b-41c0-9ac7-feb5d4d95c19",
+"name": "Sticky Note19",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2240,
+-500
+],
+"parameters": {
+"width": 440,
+"height": 100,
+"content": "Starting from here, this and the three following nodes are analogous for both methods, with a difference only in variable names. The goal is to find a **class (cluster) threshold score** so we can use it for anomaly detection (for each class).\n"
+},
+"typeVersion": 1
+},
+{
+"id": "5025936d-d49c-4cc1-a675-3bde71627c40",
+"name": "Sticky Note20",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2280,
+-180
+],
+"parameters": {
+"height": 220,
+"content": "Finding the most dissimilar point to a centre vector (within each class) is equivalent to finding the most similar point to the [opposite](https://mathinsight.org/image/vector_opposite) of a centre vector, aka the centre vector with all coordinates multiplied by -1. It is always true with **Cosine** vector similarity metric (that we're using)."
+},
+"typeVersion": 1
+},
+{
+"id": "fa9026e4-0c92-4755-92a0-5e400b5f04c9",
+"name": "Sticky Note21",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+2580,
+-140
+],
+"parameters": {
+"width": 520,
+"height": 140,
+"content": "So here, we found the most dissimilar point within the crop class to the class centre (or the Xth dissimilar point, depending on a variable set in the beginning of this pipeline). Our **threshold score** is the similarity score between this point and the class centre. Now we're saving it as meta information of each class centre point. All preparatory work for anomaly detection is done."
+},
+"typeVersion": 1
+},
+{
+"id": "8e172a7c-6865-4daf-9d9c-86e0dba2c0a2",
+"name": "Sticky Note22",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-900,
+-820
+],
+"parameters": {
+"color": 4,
+"width": 540,
+"height": 300,
+"content": "### For anomaly detection\n1. The first pipeline is uploading (crops) dataset to Qdrant's collection.\n2. **This is the second pipeline, to set up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.**\n3. The third one is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.\n\n### To recreate it\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "a23fc305-7ecd-4754-b208-2d964d9b1eda",
+"connections": {
+"Merge": {
+"main": [
+[
+{
+"node": "Embed text",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Out": {
+"main": [
+[
+{
+"node": "Cluster Distance Matrix",
+"type": "main",
+"index": 0
+},
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Embed text": {
+"main": [
+[
+{
+"node": "Get Medoid by Text",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Split Out1": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Crop Counts": {
+"main": [
+[
+{
+"node": "Info About Crop Clusters",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Set medoid id": {
+"main": [
+[]
+]
+},
+"Searching Score": {
+"main": [
+[
+{
+"node": "Threshold Score",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Threshold Score": {
+"main": [
+[
+{
+"node": "Set medoid threshold score",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Threshold Score1": {
+"main": [
+[
+{
+"node": "Set text medoid threshold score",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Medoid Vector": {
+"main": [
+[
+{
+"node": "Prepare for Searching Threshold",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Medoids Variables": {
+"main": [
+[
+{
+"node": "Total Points in Collection",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Medoid by Text": {
+"main": [
+[
+{
+"node": "Set text medoid id",
+"type": "main",
+"index": 0
+},
+{
+"node": "Prepare for Searching Threshold1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Scipy Sparse Matrix": {
+"main": [
+[
+{
+"node": "Set medoid id",
+"type": "main",
+"index": 0
+},
+{
+"node": "Get Medoid Vector",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Text Medoids Variables": {
+"main": [
+[
+{
+"node": "Textual (visual) crop descriptions",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Cluster Distance Matrix": {
+"main": [
+[
+{
+"node": "Scipy Sparse Matrix",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Info About Crop Clusters": {
+"main": [
+[
+{
+"node": "Split Out",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Qdrant cluster variables": {
+"main": [
+[
+{
+"node": "Medoids Variables",
+"type": "main",
+"index": 0
+},
+{
+"node": "Text Medoids Variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Total Points in Collection": {
+"main": [
+[
+{
+"node": "Crop Counts",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Searching Text Medoid Score": {
+"main": [
+[
+{
+"node": "Threshold Score1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prepare for Searching Threshold": {
+"main": [
+[
+{
+"node": "Searching Score",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Prepare for Searching Threshold1": {
+"main": [
+[
+{
+"node": "Searching Text Medoid Score",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Qdrant cluster variables",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Textual (visual) crop descriptions": {
+"main": [
+[
+{
+"node": "Split Out1",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].json b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].json
deleted file mode 100644
index fd5ec31..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].json
+++ /dev/null
@@ -1,461 +0,0 @@
-{
- "id": "G8jRDBvwsMkkMiLN",
- "meta": {
- "instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
- },
- "name": "[3/3] Anomaly detection tool (crops dataset)",
- "tags": [
- {
- "id": "spMntyrlE9ydvWFA",
- "name": "anomaly-detection",
- "createdAt": "2024-12-08T22:05:15.945Z",
- "updatedAt": "2024-12-09T12:50:19.287Z"
- }
- ],
- "nodes": [
- {
- "id": "e01bafec-eb24-44c7-b3c4-a60f91666350",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1200,
- 180
- ],
- "parameters": {
- "color": 6,
- "width": 400,
- "height": 740,
- "content": "We are working here with crops dataset: \nExisting (so not anomalies) crops images in dataset are:\n- 'pearl_millet(bajra)',\n- 'tobacco-plant',\n- 'cherry',\n- 'cotton',\n- 'banana',\n- 'cucumber',\n- 'maize',\n- 'wheat',\n- 'clove',\n- 'jowar',\n- 'olive-tree',\n- 'soyabean',\n- 'coffee-plant',\n- 'rice',\n- 'lemon',\n- 'mustard-oil',\n- 'vigna-radiati(mung)',\n- 'coconut',\n- 'gram',\n- 'pineapple',\n- 'sugarcane',\n- 'sunflower',\n- 'chilli',\n- 'fox_nut(makhana)',\n- 'jute',\n- 'papaya',\n- 'tea',\n- 'cardamom',\n- 'almond'\n"
- },
- "typeVersion": 1
- },
- {
- "id": "b9943781-de1f-4129-9b81-ed836e9ebb11",
- "name": "Embed image",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 680,
- 60
- ],
- "parameters": {
- "url": "https://api.voyageai.com/v1/multimodalembeddings",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $('Image URL hardcode').first().json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpHeaderAuth"
- },
- "credentials": {
- "httpHeaderAuth": {
- "id": "Vb0RNVDnIHmgnZOP",
- "name": "Voyage API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "47b72bc2-4817-48c6-b517-c1328e402468",
- "name": "Get similarity of medoids",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 940,
- 60
- ],
- "parameters": {
- "url": "={{ $('Variables for medoids').first().json.qdrantCloudURL }}/collections/{{ $('Variables for medoids').first().json.collectionName }}/points/query",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"query\": $json.data[0].embedding,\n \"using\": \"voyage\",\n \"limit\": $('Info About Crop Labeled Clusters').first().json.cropsNumber,\n \"with_payload\": true,\n \"filter\": {\n \"must\": [\n { \n \"key\": $('Variables for medoids').first().json.clusterCenterType,\n \"match\": {\n \"value\": true\n }\n }\n ]\n }\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "42d7eb27-ec38-4406-b5c4-27eb45358e93",
- "name": "Compare scores",
- "type": "n8n-nodes-base.code",
- "position": [
- 1140,
- 60
- ],
- "parameters": {
- "language": "python",
- "pythonCode": "points = _input.first()['json']['result']['points']\nthreshold_type = _('Variables for medoids').first()['json']['clusterThresholdCenterType']\n\nmax_score = -1\ncrop_with_max_score = None\nundefined = True\n\nfor center in points:\n if center['score'] >= center['payload'][threshold_type]:\n undefined = False\n if center['score'] > max_score:\n max_score = center['score']\n crop_with_max_score = center['payload']['crop_name']\n\nif undefined:\n result_message = \"ALERT, we might have a new undefined crop!\"\nelse:\n result_message = f\"Looks similar to {crop_with_max_score}\"\n\nreturn [{\n \"json\": {\n \"result\": result_message\n }\n}]\n"
- },
- "typeVersion": 2
- },
- {
- "id": "23aa604a-ff0b-4948-bcd5-af39300198c0",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1200,
- -220
- ],
- "parameters": {
- "width": 400,
- "height": 380,
- "content": "## Crop Anomaly Detection Tool\n### This is the tool that can be used directly for anomalous crops detection. \nIt takes as input (any) **image URL** and returns a **text message** telling if whatever this image depicts is anomalous to the crop dataset stored in Qdrant. \n\n* An Image URL is received via the Execute Workflow Trigger which is used to generate embedding vectors via the Voyage.ai Embeddings API.\n* The returned vectors are used to query the Qdrant collection to determine if the given crop is known by comparing it to **threshold scores** of each image class (crop type).\n* If the image scores lower than all thresholds, then the image is considered an anomaly for the dataset."
- },
- "typeVersion": 1
- },
- {
- "id": "3a79eca2-44f9-4aee-8a0d-9c7ca2f9149d",
- "name": "Variables for medoids",
- "type": "n8n-nodes-base.set",
- "position": [
- -200,
- 60
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "dbbc1e7b-c63e-4ff1-9524-8ef3e9f6cd48",
- "name": "clusterCenterType",
- "type": "string",
- "value": "is_medoid"
- },
- {
- "id": "a994ce37-2530-4030-acfb-ec777a7ddb05",
- "name": "qdrantCloudURL",
- "type": "string",
- "value": "https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
- },
- {
- "id": "12f0a9e6-686d-416e-a61b-72d034ec21ba",
- "name": "collectionName",
- "type": "string",
- "value": "=agricultural-crops"
- },
- {
- "id": "4c88a617-d44f-4776-b457-8a1dffb1d03c",
- "name": "clusterThresholdCenterType",
- "type": "string",
- "value": "is_medoid_cluster_threshold"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "13b25434-bd66-4293-93f1-26c67b9ec7dd",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -340,
- 260
- ],
- "parameters": {
- "color": 6,
- "width": 360,
- "height": 200,
- "content": "**clusterCenterType** - either\n* \"is_text_anchor_medoid\" or\n* \"is_medoid\"\n\n\n**clusterThresholdCenterType** - either\n* \"is_text_anchor_medoid_cluster_threshold\" or\n* \"is_medoid_cluster_threshold\""
- },
- "typeVersion": 1
- },
- {
- "id": "869b0962-6cae-487d-8230-539a0cc4c14c",
- "name": "Info About Crop Labeled Clusters",
- "type": "n8n-nodes-base.set",
- "position": [
- 440,
- 60
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "5327b254-b703-4a34-a398-f82edb1d6d6b",
- "name": "=cropsNumber",
- "type": "number",
- "value": "={{ $json.result.hits.length }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "5d3956f8-f43b-439e-b176-a594a21d8011",
- "name": "Total Points in Collection",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 40,
- 60
- ],
- "parameters": {
- "url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/count",
- "method": "POST",
- "options": {},
- "jsonBody": "={\n \"exact\": true\n}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "14ba3db9-3965-4b20-b333-145616d45c3a",
- "name": "Each Crop Counts",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 240,
- 60
- ],
- "parameters": {
- "url": "={{ $('Variables for medoids').first().json.qdrantCloudURL }}/collections/{{ $('Variables for medoids').first().json.collectionName }}/facet",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"key\": \"crop_name\",\n \"limit\": $json.result.count,\n \"exact\": true\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "predefinedCredentialType",
- "nodeCredentialType": "qdrantApi"
- },
- "credentials": {
- "qdrantApi": {
- "id": "it3j3hP9FICqhgX6",
- "name": "QdrantApi account"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "e37c6758-0556-4a56-ab14-d4df663cb53a",
- "name": "Image URL hardcode",
- "type": "n8n-nodes-base.set",
- "position": [
- -480,
- 60
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
- "name": "imageURL",
- "type": "string",
- "value": "={{ $json.query.imageURL }}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "b24ad1a7-0cf8-4acc-9c18-6fe9d58b10f2",
- "name": "Execute Workflow Trigger",
- "type": "n8n-nodes-base.executeWorkflowTrigger",
- "position": [
- -720,
- 60
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "50424f2b-6831-41bf-8de4-81f69d901ce1",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -240,
- -80
- ],
- "parameters": {
- "width": 180,
- "height": 120,
- "content": "Variables to access Qdrant's collection we uploaded & prepared for anomaly detection in 2 previous pipelines\n"
- },
- "typeVersion": 1
- },
- {
- "id": "2e8ed3ca-1bba-4214-b34b-376a237842ff",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 40,
- -120
- ],
- "parameters": {
- "width": 560,
- "height": 140,
- "content": "These three nodes are needed just to figure out how many different classes (crops) we have in our Qdrant collection: **cropsNumber** (needed in *\"Get similarity of medoids\"* node. \n[Note] *\"Total Points in Collection\"* -> *\"Each Crop Counts\"* were used&explained already in *\"[2/4] Set up medoids (2 types) for anomaly detection (crops dataset)\"* pipeline.\n"
- },
- "typeVersion": 1
- },
- {
- "id": "e2fa5763-6e97-4ff5-8919-1cb85a3c6968",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 620,
- 240
- ],
- "parameters": {
- "height": 120,
- "content": "Here, we're embedding the image passed to this workflow tool with the Voyage embedding model to compare the image to all crop images in the database."
- },
- "typeVersion": 1
- },
- {
- "id": "cdb6b8d3-f7f4-4d66-850f-ce16c8ed98b9",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 920,
- 220
- ],
- "parameters": {
- "width": 400,
- "height": 180,
- "content": "Checking how similar the image is to all the centres of clusters (crops).\nIf it's more similar to the thresholds we set up and stored in centres in the previous workflow, the image probably belongs to this crop class; otherwise, it's anomalous to the class. \nIf image is anomalous to all the classes, it's an anomaly."
- },
- "typeVersion": 1
- },
- {
- "id": "03b4699f-ba43-4f5f-ad69-6f81deea2641",
- "name": "Sticky Note22",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -620,
- 580
- ],
- "parameters": {
- "color": 4,
- "width": 540,
- "height": 300,
- "content": "### For anomaly detection\n1. The first pipeline is uploading (crops) dataset to Qdrant's collection.\n2. The second pipeline sets up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.\n3. **This is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.**\n\n### To recreate it\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
- },
- "typeVersion": 1
- }
- ],
- "active": false,
- "pinData": {
- "Execute Workflow Trigger": [
- {
- "json": {
- "query": {
- "imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/agricultural-crops%2Fcotton%2Fimage%20(36).jpg"
- }
- }
- }
- ]
- },
- "settings": {
- "executionOrder": "v1"
- },
- "versionId": "f67b764b-9e1a-4db0-b9f2-490077a62f74",
- "connections": {
- "Embed image": {
- "main": [
- [
- {
- "node": "Get similarity of medoids",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Each Crop Counts": {
- "main": [
- [
- {
- "node": "Info About Crop Labeled Clusters",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Image URL hardcode": {
- "main": [
- [
- {
- "node": "Variables for medoids",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Variables for medoids": {
- "main": [
- [
- {
- "node": "Total Points in Collection",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Execute Workflow Trigger": {
- "main": [
- [
- {
- "node": "Image URL hardcode",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get similarity of medoids": {
- "main": [
- [
- {
- "node": "Compare scores",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Total Points in Collection": {
- "main": [
- [
- {
- "node": "Each Crop Counts",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Info About Crop Labeled Clusters": {
- "main": [
- [
- {
- "node": "Embed image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].txt b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].txt
new file mode 100644
index 0000000..4534f39
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Vector Database as a Big Data Analysis Tool for AI Agents [3_3 - anomaly].txt
@@ -0,0 +1,461 @@
+{
+"id": "G8jRDBvwsMkkMiLN",
+"meta": {
+"instanceId": "205b3bc06c96f2dc835b4f00e1cbf9a937a74eeb3b47c99d0c30b0586dbf85aa"
+},
+"name": "[3/3] Anomaly detection tool (crops dataset)",
+"tags": [
+{
+"id": "spMntyrlE9ydvWFA",
+"name": "anomaly-detection",
+"createdAt": "2024-12-08T22:05:15.945Z",
+"updatedAt": "2024-12-09T12:50:19.287Z"
+}
+],
+"nodes": [
+{
+"id": "e01bafec-eb24-44c7-b3c4-a60f91666350",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1200,
+180
+],
+"parameters": {
+"color": 6,
+"width": 400,
+"height": 740,
+"content": "We are working here with crops dataset: \nExisting (so not anomalies) crops images in dataset are:\n- 'pearl_millet(bajra)',\n- 'tobacco-plant',\n- 'cherry',\n- 'cotton',\n- 'banana',\n- 'cucumber',\n- 'maize',\n- 'wheat',\n- 'clove',\n- 'jowar',\n- 'olive-tree',\n- 'soyabean',\n- 'coffee-plant',\n- 'rice',\n- 'lemon',\n- 'mustard-oil',\n- 'vigna-radiati(mung)',\n- 'coconut',\n- 'gram',\n- 'pineapple',\n- 'sugarcane',\n- 'sunflower',\n- 'chilli',\n- 'fox_nut(makhana)',\n- 'jute',\n- 'papaya',\n- 'tea',\n- 'cardamom',\n- 'almond'\n"
+},
+"typeVersion": 1
+},
+{
+"id": "b9943781-de1f-4129-9b81-ed836e9ebb11",
+"name": "Embed image",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+680,
+60
+],
+"parameters": {
+"url": "https://api.voyageai.com/v1/multimodalembeddings",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"inputs\": [\n {\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": $('Image URL hardcode').first().json.imageURL\n }\n ]\n }\n ],\n \"model\": \"voyage-multimodal-3\",\n \"input_type\": \"document\"\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpHeaderAuth"
+},
+"credentials": {
+"httpHeaderAuth": {
+"id": "Vb0RNVDnIHmgnZOP",
+"name": "Voyage API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "47b72bc2-4817-48c6-b517-c1328e402468",
+"name": "Get similarity of medoids",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+940,
+60
+],
+"parameters": {
+"url": "={{ $('Variables for medoids').first().json.qdrantCloudURL }}/collections/{{ $('Variables for medoids').first().json.collectionName }}/points/query",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"query\": $json.data[0].embedding,\n \"using\": \"voyage\",\n \"limit\": $('Info About Crop Labeled Clusters').first().json.cropsNumber,\n \"with_payload\": true,\n \"filter\": {\n \"must\": [\n { \n \"key\": $('Variables for medoids').first().json.clusterCenterType,\n \"match\": {\n \"value\": true\n }\n }\n ]\n }\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "42d7eb27-ec38-4406-b5c4-27eb45358e93",
+"name": "Compare scores",
+"type": "n8n-nodes-base.code",
+"position": [
+1140,
+60
+],
+"parameters": {
+"language": "python",
+"pythonCode": "points = _input.first()['json']['result']['points']\nthreshold_type = _('Variables for medoids').first()['json']['clusterThresholdCenterType']\n\nmax_score = -1\ncrop_with_max_score = None\nundefined = True\n\nfor center in points:\n if center['score'] >= center['payload'][threshold_type]:\n undefined = False\n if center['score'] > max_score:\n max_score = center['score']\n crop_with_max_score = center['payload']['crop_name']\n\nif undefined:\n result_message = \"ALERT, we might have a new undefined crop!\"\nelse:\n result_message = f\"Looks similar to {crop_with_max_score}\"\n\nreturn [{\n \"json\": {\n \"result\": result_message\n }\n}]\n"
+},
+"typeVersion": 2
+},
+{
+"id": "23aa604a-ff0b-4948-bcd5-af39300198c0",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1200,
+-220
+],
+"parameters": {
+"width": 400,
+"height": 380,
+"content": "## Crop Anomaly Detection Tool\n### This is the tool that can be used directly for anomalous crops detection. \nIt takes as input (any) **image URL** and returns a **text message** telling if whatever this image depicts is anomalous to the crop dataset stored in Qdrant. \n\n* An Image URL is received via the Execute Workflow Trigger which is used to generate embedding vectors via the Voyage.ai Embeddings API.\n* The returned vectors are used to query the Qdrant collection to determine if the given crop is known by comparing it to **threshold scores** of each image class (crop type).\n* If the image scores lower than all thresholds, then the image is considered an anomaly for the dataset."
+},
+"typeVersion": 1
+},
+{
+"id": "3a79eca2-44f9-4aee-8a0d-9c7ca2f9149d",
+"name": "Variables for medoids",
+"type": "n8n-nodes-base.set",
+"position": [
+-200,
+60
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "dbbc1e7b-c63e-4ff1-9524-8ef3e9f6cd48",
+"name": "clusterCenterType",
+"type": "string",
+"value": "is_medoid"
+},
+{
+"id": "a994ce37-2530-4030-acfb-ec777a7ddb05",
+"name": "qdrantCloudURL",
+"type": "string",
+"value": "https://152bc6e2-832a-415c-a1aa-fb529f8baf8d.eu-central-1-0.aws.cloud.qdrant.io"
+},
+{
+"id": "12f0a9e6-686d-416e-a61b-72d034ec21ba",
+"name": "collectionName",
+"type": "string",
+"value": "=agricultural-crops"
+},
+{
+"id": "4c88a617-d44f-4776-b457-8a1dffb1d03c",
+"name": "clusterThresholdCenterType",
+"type": "string",
+"value": "is_medoid_cluster_threshold"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "13b25434-bd66-4293-93f1-26c67b9ec7dd",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-340,
+260
+],
+"parameters": {
+"color": 6,
+"width": 360,
+"height": 200,
+"content": "**clusterCenterType** - either\n* \"is_text_anchor_medoid\" or\n* \"is_medoid\"\n\n\n**clusterThresholdCenterType** - either\n* \"is_text_anchor_medoid_cluster_threshold\" or\n* \"is_medoid_cluster_threshold\""
+},
+"typeVersion": 1
+},
+{
+"id": "869b0962-6cae-487d-8230-539a0cc4c14c",
+"name": "Info About Crop Labeled Clusters",
+"type": "n8n-nodes-base.set",
+"position": [
+440,
+60
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "5327b254-b703-4a34-a398-f82edb1d6d6b",
+"name": "=cropsNumber",
+"type": "number",
+"value": "={{ $json.result.hits.length }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "5d3956f8-f43b-439e-b176-a594a21d8011",
+"name": "Total Points in Collection",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+40,
+60
+],
+"parameters": {
+"url": "={{ $json.qdrantCloudURL }}/collections/{{ $json.collectionName }}/points/count",
+"method": "POST",
+"options": {},
+"jsonBody": "={\n \"exact\": true\n}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "14ba3db9-3965-4b20-b333-145616d45c3a",
+"name": "Each Crop Counts",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+240,
+60
+],
+"parameters": {
+"url": "={{ $('Variables for medoids').first().json.qdrantCloudURL }}/collections/{{ $('Variables for medoids').first().json.collectionName }}/facet",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"key\": \"crop_name\",\n \"limit\": $json.result.count,\n \"exact\": true\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "predefinedCredentialType",
+"nodeCredentialType": "qdrantApi"
+},
+"credentials": {
+"qdrantApi": {
+"id": "it3j3hP9FICqhgX6",
+"name": "QdrantApi account"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "e37c6758-0556-4a56-ab14-d4df663cb53a",
+"name": "Image URL hardcode",
+"type": "n8n-nodes-base.set",
+"position": [
+-480,
+60
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "46ceba40-fb25-450c-8550-d43d8b8aa94c",
+"name": "imageURL",
+"type": "string",
+"value": "={{ $json.query.imageURL }}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "b24ad1a7-0cf8-4acc-9c18-6fe9d58b10f2",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-720,
+60
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "50424f2b-6831-41bf-8de4-81f69d901ce1",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-240,
+-80
+],
+"parameters": {
+"width": 180,
+"height": 120,
+"content": "Variables to access Qdrant's collection we uploaded & prepared for anomaly detection in 2 previous pipelines\n"
+},
+"typeVersion": 1
+},
+{
+"id": "2e8ed3ca-1bba-4214-b34b-376a237842ff",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+40,
+-120
+],
+"parameters": {
+"width": 560,
+"height": 140,
+"content": "These three nodes are needed just to figure out how many different classes (crops) we have in our Qdrant collection: **cropsNumber** (needed in *\"Get similarity of medoids\"* node. \n[Note] *\"Total Points in Collection\"* -> *\"Each Crop Counts\"* were used&explained already in *\"[2/4] Set up medoids (2 types) for anomaly detection (crops dataset)\"* pipeline.\n"
+},
+"typeVersion": 1
+},
+{
+"id": "e2fa5763-6e97-4ff5-8919-1cb85a3c6968",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+620,
+240
+],
+"parameters": {
+"height": 120,
+"content": "Here, we're embedding the image passed to this workflow tool with the Voyage embedding model to compare the image to all crop images in the database."
+},
+"typeVersion": 1
+},
+{
+"id": "cdb6b8d3-f7f4-4d66-850f-ce16c8ed98b9",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+920,
+220
+],
+"parameters": {
+"width": 400,
+"height": 180,
+"content": "Checking how similar the image is to all the centres of clusters (crops).\nIf it's more similar to the thresholds we set up and stored in centres in the previous workflow, the image probably belongs to this crop class; otherwise, it's anomalous to the class. \nIf image is anomalous to all the classes, it's an anomaly."
+},
+"typeVersion": 1
+},
+{
+"id": "03b4699f-ba43-4f5f-ad69-6f81deea2641",
+"name": "Sticky Note22",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-620,
+580
+],
+"parameters": {
+"color": 4,
+"width": 540,
+"height": 300,
+"content": "### For anomaly detection\n1. The first pipeline is uploading (crops) dataset to Qdrant's collection.\n2. The second pipeline sets up cluster (class) centres in this Qdrant collection & cluster (class) threshold scores.\n3. **This is the anomaly detection tool, which takes any image as input and uses all preparatory work done with Qdrant (crops) collection.**\n\n### To recreate it\nYou'll have to upload [crops](https://www.kaggle.com/datasets/mdwaquarazam/agricultural-crops-image-classification) dataset from Kaggle to your own Google Storage bucket, and re-create APIs/connections to [Qdrant Cloud](https://qdrant.tech/documentation/quickstart-cloud/) (you can use **Free Tier** cluster), Voyage AI API & Google Cloud Storage\n\n**In general, pipelines are adaptable to any dataset of images**\n"
+},
+"typeVersion": 1
+}
+],
+"active": false,
+"pinData": {
+"Execute Workflow Trigger": [
+{
+"json": {
+"query": {
+"imageURL": "https://storage.googleapis.com/n8n-qdrant-demo/agricultural-crops%2Fcotton%2Fimage%20(36).jpg"
+}
+}
+}
+]
+},
+"settings": {
+"executionOrder": "v1"
+},
+"versionId": "f67b764b-9e1a-4db0-b9f2-490077a62f74",
+"connections": {
+"Embed image": {
+"main": [
+[
+{
+"node": "Get similarity of medoids",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Each Crop Counts": {
+"main": [
+[
+{
+"node": "Info About Crop Labeled Clusters",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Image URL hardcode": {
+"main": [
+[
+{
+"node": "Variables for medoids",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Variables for medoids": {
+"main": [
+[
+{
+"node": "Total Points in Collection",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Image URL hardcode",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get similarity of medoids": {
+"main": [
+[
+{
+"node": "Compare scores",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Total Points in Collection": {
+"main": [
+[
+{
+"node": "Each Crop Counts",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Info About Crop Labeled Clusters": {
+"main": [
+[
+{
+"node": "Embed image",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.json b/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.json
deleted file mode 100644
index 6edb3bb..0000000
--- a/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.json
+++ /dev/null
@@ -1,1001 +0,0 @@
-{
- "meta": {
- "instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
- },
- "nodes": [
- {
- "id": "cb62c9a5-2f43-4328-af94-84c2cb731d9c",
- "name": "Base Image",
- "type": "n8n-nodes-base.googleDrive",
- "position": [
- 260,
- 660
- ],
- "parameters": {
- "fileId": {
- "__rl": true,
- "mode": "id",
- "value": "={{ $json.base }}"
- },
- "options": {
- "binaryPropertyName": "data_1"
- },
- "operation": "download"
- },
- "credentials": {
- "googleDriveOAuth2Api": {
- "id": "yOwz41gMQclOadgu",
- "name": "Google Drive account"
- }
- },
- "typeVersion": 3
- },
- {
- "id": "b1c304cc-9949-441a-ac2a-275c8d4c51fc",
- "name": "Google Gemini Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
- "position": [
- 1120,
- 900
- ],
- "parameters": {
- "options": {},
- "modelName": "models/gemini-1.5-pro-latest"
- },
- "credentials": {
- "googlePalmApi": {
- "id": "dSxo6ns5wn658r8N",
- "name": "Google Gemini(PaLM) Api account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "964d94bf-be2a-424e-ab0e-c1c1fe260ebd",
- "name": "Structured Output Parser",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 1320,
- 900
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n\t\"properties\": {\n\t\t\"type\": {\n \t\t\"type\": \"string\",\n \"description\": \"type of regression. One of text, number, image, color or position.\"\n \t\t},\n\t\t\"description\": { \"type\": \"string\" },\n \"previous_state\": { \"type\": \"string\" },\n \"current_state\": { \"type\": \"string\" }\n\t}\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "67195eb2-1729-42b0-8275-bdd6128b81aa",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -2340,
- 20
- ],
- "parameters": {
- "color": 4,
- "width": 405.95003875719203,
- "height": 180.74812634463558,
- "content": "### Part A. Generate Base Images\nBefore we can run our visual regression tests, we must generate a series of base screenshots to compare against. This part of the workflow uses an external website screenshotting service, [Apify.com](https://www.apify.com?fpr=414q6), to achieve this. This part of the workflow should only be run when we want to update our base screenshots."
- },
- "typeVersion": 1
- },
- {
- "id": "85f9b371-1710-4c9c-a0ed-210d9c0e5d64",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 162.7495769165307,
- 500
- ],
- "parameters": {
- "color": 7,
- "width": 702.1744987652204,
- "height": 548.4621171664835,
- "content": "## 5. Download Base and Generate new Webpage Screenshot\n[Learn more about Apify.com](https://www.apify.com?fpr=414q6)\n\nLooping for each webpage, we'll do 2 tasks (1) download the base screenshot for the url and (2) and use our [Apify.com](https://www.apify.com?fpr=414q6) webpage screenshot actor again to generate a fresh screenshot."
- },
- "typeVersion": 1
- },
- {
- "id": "8bff4efc-d9f9-485c-b51d-a8edc29d1105",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 900,
- 500
- ],
- "parameters": {
- "color": 7,
- "width": 759.5372282495247,
- "height": 548.702446115556,
- "content": "## 6. Compare Screenshots using Vision Model\n[Read more about the basic LLM chain](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nTo carry out our visual regression test, we need to send both screenshots simultaneously to our Vision model. This is easily achieved using n8n's built-in basic LLM chain where we can define two user messages of the binary type. For our vision model, we'll use Google's Gemini but any capable vision model will also do the job. A Structured Output Parser is used here to return the AI's response in JSON format, this is for easier formatting purposes which we'll get to in the next step."
- },
- "typeVersion": 1
- },
- {
- "id": "a92d11e5-0985-4a8f-bc43-8bc0ca48e744",
- "name": "Sticky Note3",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 397.518987341772,
- 93.8157360237642
- ],
- "parameters": {
- "color": 7,
- "width": 885.2402868841493,
- "height": 388.92815062755926,
- "content": "## 7. Create Report In Linear\n[Learn more about integrating with Linear.app](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.linear)\n\nFor the final step, we'll generate a simple report which will capture any changes detected in our webpages list. Let's do this by first combining our webpages with their test results and filter out any in the page where no changes were detected. Next, we'll aggregate all changes into the Linear.app node which will be formatted into a markdown snippet and used to create a new issue in Linear. If you don't use Linear, feel free to swap this out for JIRA or even Slack."
- },
- "typeVersion": 1
- },
- {
- "id": "3f52c006-6c0a-456d-ab3c-ee5a16726299",
- "name": "Loop Over Items",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -1680,
- 580
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "478ee25d-3f0f-4f6c-bf34-add1dc14c3cb",
- "name": "Wait",
- "type": "n8n-nodes-base.wait",
- "position": [
- -1240,
- 820
- ],
- "webhookId": "f06eab66-30a7-48ad-90ee-cb3394eb2edb",
- "parameters": {
- "amount": 2
- },
- "typeVersion": 1.1
- },
- {
- "id": "64b5f755-a85e-4ae5-ad81-113c1ef9b64c",
- "name": "Download Screenshot",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -1260,
- 360
- ],
- "parameters": {
- "url": "={{ $json.screenshotUrl }}",
- "options": {}
- },
- "typeVersion": 4.2
- },
- {
- "id": "8f99ef1f-1cdc-4d80-b858-e9960a805dd4",
- "name": "Upload to Drive",
- "type": "n8n-nodes-base.googleDrive",
- "position": [
- -1080,
- 360
- ],
- "parameters": {
- "name": "={{ $('Merge').item.json.url.urlEncode() }}",
- "driveId": {
- "__rl": true,
- "mode": "list",
- "value": "My Drive"
- },
- "options": {
- "simplifyOutput": true
- },
- "folderId": {
- "__rl": true,
- "mode": "list",
- "value": "1lAFxJPWcA_sOcjr3UUKKfFfoTwd4Stkh",
- "cachedResultUrl": "https://drive.google.com/drive/folders/1lAFxJPWcA_sOcjr3UUKKfFfoTwd4Stkh",
- "cachedResultName": "base_images"
- }
- },
- "credentials": {
- "googleDriveOAuth2Api": {
- "id": "yOwz41gMQclOadgu",
- "name": "Google Drive account"
- }
- },
- "typeVersion": 3
- },
- {
- "id": "5e253123-89ba-42d5-b743-60bfd1ebae5b",
- "name": "Update Base Image",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- -900,
- 360
- ],
- "parameters": {
- "columns": {
- "value": {
- "url": "={{ $('Merge').item.json.url }}",
- "base": "={{ $json.id }}"
- },
- "schema": [
- {
- "id": "service",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "service",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "url",
- "type": "string",
- "display": true,
- "removed": false,
- "required": false,
- "displayName": "url",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- },
- {
- "id": "base",
- "type": "string",
- "display": true,
- "required": false,
- "displayName": "base",
- "defaultMatch": false,
- "canBeUsedToMatch": true
- }
- ],
- "mappingMode": "defineBelow",
- "matchingColumns": [
- "url"
- ]
- },
- "options": {},
- "operation": "appendOrUpdate",
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
- "cachedResultName": "Sheet1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
- "cachedResultName": "Visual Regression List"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.5
- },
- {
- "id": "fa7339b7-b7dd-4ecd-8dc2-f42f6549adb6",
- "name": "Merge",
- "type": "n8n-nodes-base.merge",
- "position": [
- -1440,
- 360
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combineBy": "combineByPosition"
- },
- "typeVersion": 3
- },
- {
- "id": "47845df9-a50e-429e-b81e-5eefd996d5c7",
- "name": "Schedule Trigger",
- "type": "n8n-nodes-base.scheduleTrigger",
- "position": [
- -560,
- 380
- ],
- "parameters": {
- "rule": {
- "interval": [
- {
- "field": "weeks",
- "triggerAtDay": [
- 1
- ],
- "triggerAtHour": 6
- }
- ]
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "63492aa4-3535-4832-a9d0-0a949e46ec81",
- "name": "Get URLs with Missing Base Images",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- -1980,
- 480
- ],
- "parameters": {
- "options": {},
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
- "cachedResultName": "Sheet1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
- "cachedResultName": "Visual Regression List"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.5
- },
- {
- "id": "8907f3b9-0613-4057-8adb-fd5c4e25cf72",
- "name": "Run Webpage Screenshot",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- -1420,
- 820
- ],
- "parameters": {
- "url": "https://api.apify.com/v2/acts/apify~screenshot-url/run-sync-get-dataset-items",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"delay\": 0,\n \"format\": \"png\",\n \"proxy\": {\n \"useApifyProxy\": true\n },\n \"scrollToBottom\": false,\n \"urls\": [\n {\n \"url\": $json.url\n }\n ],\n \"viewportWidth\": 1280,\n \"waitUntil\": \"domcontentloaded\",\n \"waitUntilNetworkIdleAfterScroll\": false\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth"
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "cO2w8RDNOZg8DRa8",
- "name": "Apify API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "3dc45b2d-4c4a-44d5-9b45-3e2144479603",
- "name": "Run Webpage Screenshot1",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 273,
- 833
- ],
- "parameters": {
- "url": "https://api.apify.com/v2/acts/apify~screenshot-url/run-sync-get-dataset-items",
- "method": "POST",
- "options": {},
- "jsonBody": "={{\n{\n \"delay\": 0,\n \"format\": \"png\",\n \"proxy\": {\n \"useApifyProxy\": true\n },\n \"scrollToBottom\": false,\n \"urls\": [\n {\n \"url\": $json.url\n }\n ],\n \"viewportWidth\": 1280,\n \"waitUntil\": \"domcontentloaded\",\n \"waitUntilNetworkIdleAfterScroll\": false\n}\n}}",
- "sendBody": true,
- "specifyBody": "json",
- "authentication": "genericCredentialType",
- "genericAuthType": "httpQueryAuth"
- },
- "credentials": {
- "httpQueryAuth": {
- "id": "cO2w8RDNOZg8DRa8",
- "name": "Apify API"
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "672d64fb-7782-427e-8779-953e51118fbc",
- "name": "Has Changes",
- "type": "n8n-nodes-base.filter",
- "position": [
- 680,
- 300
- ],
- "parameters": {
- "options": {},
- "conditions": {
- "options": {
- "version": 2,
- "leftValue": "",
- "caseSensitive": true,
- "typeValidation": "strict"
- },
- "combinator": "and",
- "conditions": [
- {
- "id": "20b18a7e-bf98-4f39-baa9-4d965097526a",
- "operator": {
- "type": "array",
- "operation": "lengthGt",
- "rightType": "number"
- },
- "leftValue": "={{ $json.output }}",
- "rightValue": 0
- }
- ]
- }
- },
- "typeVersion": 2.2
- },
- {
- "id": "efa168ec-ff05-471b-869f-cee5a222594a",
- "name": "Combine Row and Result",
- "type": "n8n-nodes-base.set",
- "position": [
- 500,
- 300
- ],
- "parameters": {
- "mode": "raw",
- "options": {},
- "jsonOutput": "={{\n{\n ...$('Get Webpages List').item.json,\n output: $json.output\n}\n}}\n"
- },
- "typeVersion": 3.4
- },
- {
- "id": "1fe901dc-f460-41b8-8042-0fcb0474092f",
- "name": "Wait1",
- "type": "n8n-nodes-base.wait",
- "position": [
- 1580,
- 900
- ],
- "webhookId": "6bbf2e65-bed1-4efc-bb31-09d12c644dc5",
- "parameters": {
- "amount": 1
- },
- "typeVersion": 1.1
- },
- {
- "id": "7891f052-4073-4746-a04b-27c7c4fa1e63",
- "name": "Aggregate",
- "type": "n8n-nodes-base.aggregate",
- "position": [
- 860,
- 300
- ],
- "parameters": {
- "options": {},
- "aggregate": "aggregateAllItemData"
- },
- "typeVersion": 1
- },
- {
- "id": "ef2b2ddb-51f9-4576-bd99-9efa39be5163",
- "name": "Create Report",
- "type": "n8n-nodes-base.linear",
- "position": [
- 1040,
- 300
- ],
- "parameters": {
- "title": "=Visual Regression Report {{ $now.format('yyyy-MM-dd') }}",
- "teamId": "1c721608-321d-4132-ac32-6e92d04bb487",
- "additionalFields": {
- "description": "=Visual Regression Workflow picked up the following changes:\n\n{{\n$json.data.map(row =>\n`### ${row.url}\n${row.output.map(issue =>\n`* **${issue.description}** - expected \"${issue.previous_state}\" but got \"${issue.current_state}\"`\n).join('\\n')}`\n).join('\\n');\n}}"
- }
- },
- "credentials": {
- "linearApi": {
- "id": "Nn0F7T9FtvRUtEbe",
- "name": "Linear account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "477b89f7-00ca-4001-a246-0887bcb553eb",
- "name": "When clicking โTest workflowโ",
- "type": "n8n-nodes-base.manualTrigger",
- "position": [
- -2180,
- 480
- ],
- "parameters": {},
- "typeVersion": 1
- },
- {
- "id": "eb7f6310-5465-4638-b702-5ecbd98a0199",
- "name": "Get Webpages List",
- "type": "n8n-nodes-base.googleSheets",
- "position": [
- -360,
- 380
- ],
- "parameters": {
- "options": {},
- "filtersUI": {
- "values": [
- {
- "lookupValue": "2",
- "lookupColumn": "=row_number"
- }
- ]
- },
- "sheetName": {
- "__rl": true,
- "mode": "list",
- "value": "gid=0",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
- "cachedResultName": "Sheet1"
- },
- "documentId": {
- "__rl": true,
- "mode": "list",
- "value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
- "cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
- "cachedResultName": "Visual Regression List"
- }
- },
- "credentials": {
- "googleSheetsOAuth2Api": {
- "id": "XHvC7jIRR8A2TlUl",
- "name": "Google Sheets account"
- }
- },
- "typeVersion": 4.5
- },
- {
- "id": "6c0f7341-14c9-48c2-9447-edab0ad18df7",
- "name": "For Each Webpage...",
- "type": "n8n-nodes-base.splitInBatches",
- "position": [
- -40,
- 440
- ],
- "parameters": {
- "options": {}
- },
- "typeVersion": 3
- },
- {
- "id": "62e13166-458d-4c63-8911-740f9ceaeb54",
- "name": "Sticky Note4",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -660,
- 160
- ],
- "parameters": {
- "color": 7,
- "width": 561.2038065501644,
- "height": 408.0284015307624,
- "content": "## 4. Trigger Visual Regression Test Run\n[Read more about the Schedule Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.scheduletrigger/)\n\nOnce we've generated our base images to compare with in Part A, we can now run our Visual Regression Tests. These tests are intended to check for unexpected changes to a webpage by using some form of image detection. To trigger Part B, we'll start with a schedule trigger and pull a list of webpages to test from Google Sheets."
- },
- "typeVersion": 1
- },
- {
- "id": "8d958f44-fd2c-49b4-adbd-d8a99b2614c8",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -2340,
- 218.0216140230686
- ],
- "parameters": {
- "color": 7,
- "width": 626.9985071319608,
- "height": 487.40071048786325,
- "content": "## 1. Get List of Webpages to Generate Base Images\n[Learn more about using Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets/)\n\nThis workflow is split into 2 parts: Part A will generate the \"base\" screenshots to compare new screenshots against. To capture these base screenshots, we'll use Google Sheets to hold our list of webpages and their screenshot references (we'll come on to that later).\n\nExample Sheet: https://docs.google.com/spreadsheets/d/e/2PACX-1vTXRZRi55dUbLAsWZboJqH5U-EK0ZRSse8pkqANAV4Ss70otpQ97zgT8YBd3dL4d2u2UC1TTx_o1o1R/pubhtml"
- },
- "typeVersion": 1
- },
- {
- "id": "ee776b4d-4532-4c08-ac38-35d40afbd8ad",
- "name": "Sticky Note6",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1480,
- 580
- ],
- "parameters": {
- "color": 7,
- "width": 653.369086691465,
- "height": 443.1120543367141,
- "content": "## 2. Generate Webpage Screenshot via Apify\n[Learn more about Apify.com](https://www.apify.com?fpr=414q6)\n\nTo generate a screenshot of the webpage, we'll need a third party service since this functionality is outside the scope of n8n. Feel free to pick whichever internal or external service works for you but I've had great experience using [Apify.com](https://www.apify.com?fpr=414q6) - a popular webscraping SaaS who offer a generous free plan and require very little configuration to get started.\n\nThe Apify \"actor\" (ie. a type of scraper) we'll be using is specifically designed to take webpage screenshots."
- },
- "typeVersion": 1
- },
- {
- "id": "3d90e103-2829-4075-b3d4-5ba848af4843",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1520,
- 160
- ],
- "parameters": {
- "color": 7,
- "width": 808.188722669735,
- "height": 397.73072497123115,
- "content": "## 3. Upload Screenshot to Google Drive\n[Read more about using the Google Drive node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googledrive/)\n\nOnce we have our screenshots, we'll download them from Apify and upload them to our Google Drive for safe keeping. After uploading, we'll capture the new Google Drive IDs for the images into our Google Sheet, this will allow us to reference them again when we perform the visual regression testing."
- },
- "typeVersion": 1
- },
- {
- "id": "e47d14ec-ad78-42c8-a294-301dcd581a67",
- "name": "Download New Screenshot",
- "type": "n8n-nodes-base.httpRequest",
- "position": [
- 453,
- 833
- ],
- "parameters": {
- "url": "={{ $json.screenshotUrl }}",
- "options": {
- "response": {
- "response": {
- "responseFormat": "file",
- "outputPropertyName": "data_2"
- }
- }
- }
- },
- "typeVersion": 4.2
- },
- {
- "id": "8ca118bc-3d19-48ac-9d9c-0892993da736",
- "name": "Combine Screenshots",
- "type": "n8n-nodes-base.merge",
- "position": [
- 660,
- 660
- ],
- "parameters": {
- "mode": "combine",
- "options": {},
- "combineBy": "combineByPosition"
- },
- "typeVersion": 3
- },
- {
- "id": "03359cbb-d7af-4118-a32a-3fe24062dc9f",
- "name": "Sticky Note8",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -660,
- 20
- ],
- "parameters": {
- "color": 4,
- "width": 394.03359370567625,
- "height": 111.52173490405977,
- "content": "### Part B. Run Visual Regression Test\nIn this part of the workflow, we'll retrieve our list of webpages to test with our AI vision model. This part can be run as many times as required."
- },
- "typeVersion": 1
- },
- {
- "id": "a78c0f92-aa61-483b-95bf-dd60958f182d",
- "name": "Sticky Note9",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -2920,
- 220
- ],
- "parameters": {
- "width": 553.2963720930223,
- "height": 473.4987906976746,
- "content": "## Try It Out!\n\n### This workflow implements an approach to Visual Regression Testing - a means to test websites for defects - using AI Vision Models.\n\nThis workflow uses a Google Sheet to track a list of webpages to test and is split into 2 parts; Part A generates the base screenshots of the list and Part B runs the visual regression testing.\n\nThe example spreadsheet can be found here: https://docs.google.com/spreadsheets/d/e/2PACX-1vTXRZRi55dUbLAsWZboJqH5U-EK0ZRSse8pkqANAV4Ss70otpQ97zgT8YBd3dL4d2u2UC1TTx_o1o1R/pubhtml\n\n**[Apify.com](https://www.apify.com?fpr=414q6)** is the screenshot generator of choice and a free account with $5 in credit is available via this [link](https://www.apify.com?fpr=414q6).\n\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!"
- },
- "typeVersion": 1
- },
- {
- "id": "a0b257e5-99f8-409a-bc67-2468db377d6c",
- "name": "Visual Regression Agent",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 1120,
- 740
- ],
- "parameters": {
- "text": "Identify changes between the base image and test image.",
- "messages": {
- "messageValues": [
- {
- "message": "=You help with visual regression testing for websites. Identify changes to text content, images, colors, position and layouts of the elements in the screenshots. Ignore text styling or casing changes.\nThe first image will be the base image and the second image will be the test. Note all changes to the test image which differ from the base. If there are no changes, it is okay to return an empty array."
- },
- {
- "type": "HumanMessagePromptTemplate",
- "messageType": "imageBinary",
- "binaryImageDataKey": "data_1"
- },
- {
- "type": "HumanMessagePromptTemplate",
- "messageType": "imageBinary",
- "binaryImageDataKey": "data_2"
- }
- ]
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.4
- }
- ],
- "pinData": {},
- "connections": {
- "Wait": {
- "main": [
- [
- {
- "node": "Loop Over Items",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Merge": {
- "main": [
- [
- {
- "node": "Download Screenshot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Wait1": {
- "main": [
- [
- {
- "node": "For Each Webpage...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Aggregate": {
- "main": [
- [
- {
- "node": "Create Report",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Base Image": {
- "main": [
- [
- {
- "node": "Combine Screenshots",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Has Changes": {
- "main": [
- [
- {
- "node": "Aggregate",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Loop Over Items": {
- "main": [
- [
- {
- "node": "Merge",
- "type": "main",
- "index": 1
- }
- ],
- [
- {
- "node": "Run Webpage Screenshot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Upload to Drive": {
- "main": [
- [
- {
- "node": "Update Base Image",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Schedule Trigger": {
- "main": [
- [
- {
- "node": "Get Webpages List",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Get Webpages List": {
- "main": [
- [
- {
- "node": "For Each Webpage...",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Combine Screenshots": {
- "main": [
- [
- {
- "node": "Visual Regression Agent",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Download Screenshot": {
- "main": [
- [
- {
- "node": "Upload to Drive",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "For Each Webpage...": {
- "main": [
- [
- {
- "node": "Combine Row and Result",
- "type": "main",
- "index": 0
- }
- ],
- [
- {
- "node": "Base Image",
- "type": "main",
- "index": 0
- },
- {
- "node": "Run Webpage Screenshot1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Combine Row and Result": {
- "main": [
- [
- {
- "node": "Has Changes",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Run Webpage Screenshot": {
- "main": [
- [
- {
- "node": "Wait",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Download New Screenshot": {
- "main": [
- [
- {
- "node": "Combine Screenshots",
- "type": "main",
- "index": 1
- }
- ]
- ]
- },
- "Run Webpage Screenshot1": {
- "main": [
- [
- {
- "node": "Download New Screenshot",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Visual Regression Agent": {
- "main": [
- [
- {
- "node": "Wait1",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "Google Gemini Chat Model": {
- "ai_languageModel": [
- [
- {
- "node": "Visual Regression Agent",
- "type": "ai_languageModel",
- "index": 0
- }
- ]
- ]
- },
- "Structured Output Parser": {
- "ai_outputParser": [
- [
- {
- "node": "Visual Regression Agent",
- "type": "ai_outputParser",
- "index": 0
- }
- ]
- ]
- },
- "Get URLs with Missing Base Images": {
- "main": [
- [
- {
- "node": "Loop Over Items",
- "type": "main",
- "index": 0
- },
- {
- "node": "Merge",
- "type": "main",
- "index": 0
- }
- ]
- ]
- },
- "When clicking โTest workflowโ": {
- "main": [
- [
- {
- "node": "Get URLs with Missing Base Images",
- "type": "main",
- "index": 0
- }
- ]
- ]
- }
- }
-}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.txt b/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.txt
new file mode 100644
index 0000000..124b565
--- /dev/null
+++ b/AI_Research_RAG_and_Data_Analysis/Visual Regression Testing with Apify and AI Vision Model.txt
@@ -0,0 +1,1001 @@
+{
+"meta": {
+"instanceId": "408f9fb9940c3cb18ffdef0e0150fe342d6e655c3a9fac21f0f644e8bedabcd9"
+},
+"nodes": [
+{
+"id": "cb62c9a5-2f43-4328-af94-84c2cb731d9c",
+"name": "Base Image",
+"type": "n8n-nodes-base.googleDrive",
+"position": [
+260,
+660
+],
+"parameters": {
+"fileId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $json.base }}"
+},
+"options": {
+"binaryPropertyName": "data_1"
+},
+"operation": "download"
+},
+"credentials": {
+"googleDriveOAuth2Api": {
+"id": "yOwz41gMQclOadgu",
+"name": "Google Drive account"
+}
+},
+"typeVersion": 3
+},
+{
+"id": "b1c304cc-9949-441a-ac2a-275c8d4c51fc",
+"name": "Google Gemini Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
+"position": [
+1120,
+900
+],
+"parameters": {
+"options": {},
+"modelName": "models/gemini-1.5-pro-latest"
+},
+"credentials": {
+"googlePalmApi": {
+"id": "dSxo6ns5wn658r8N",
+"name": "Google Gemini(PaLM) Api account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "964d94bf-be2a-424e-ab0e-c1c1fe260ebd",
+"name": "Structured Output Parser",
+"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
+"position": [
+1320,
+900
+],
+"parameters": {
+"schemaType": "manual",
+"inputSchema": "{\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n\t\"properties\": {\n\t\t\"type\": {\n \t\t\"type\": \"string\",\n \"description\": \"type of regression. One of text, number, image, color or position.\"\n \t\t},\n\t\t\"description\": { \"type\": \"string\" },\n \"previous_state\": { \"type\": \"string\" },\n \"current_state\": { \"type\": \"string\" }\n\t}\n }\n}"
+},
+"typeVersion": 1.2
+},
+{
+"id": "67195eb2-1729-42b0-8275-bdd6128b81aa",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2340,
+20
+],
+"parameters": {
+"color": 4,
+"width": 405.95003875719203,
+"height": 180.74812634463558,
+"content": "### Part A. Generate Base Images\nBefore we can run our visual regression tests, we must generate a series of base screenshots to compare against. This part of the workflow uses an external website screenshotting service, [Apify.com](https://www.apify.com?fpr=414q6), to achieve this. This part of the workflow should only be run when we want to update our base screenshots."
+},
+"typeVersion": 1
+},
+{
+"id": "85f9b371-1710-4c9c-a0ed-210d9c0e5d64",
+"name": "Sticky Note1",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+162.7495769165307,
+500
+],
+"parameters": {
+"color": 7,
+"width": 702.1744987652204,
+"height": 548.4621171664835,
+"content": "## 5. Download Base and Generate new Webpage Screenshot\n[Learn more about Apify.com](https://www.apify.com?fpr=414q6)\n\nLooping for each webpage, we'll do 2 tasks (1) download the base screenshot for the url and (2) and use our [Apify.com](https://www.apify.com?fpr=414q6) webpage screenshot actor again to generate a fresh screenshot."
+},
+"typeVersion": 1
+},
+{
+"id": "8bff4efc-d9f9-485c-b51d-a8edc29d1105",
+"name": "Sticky Note2",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+900,
+500
+],
+"parameters": {
+"color": 7,
+"width": 759.5372282495247,
+"height": 548.702446115556,
+"content": "## 6. Compare Screenshots using Vision Model\n[Read more about the basic LLM chain](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.chainllm/)\n\nTo carry out our visual regression test, we need to send both screenshots simultaneously to our Vision model. This is easily achieved using n8n's built-in basic LLM chain where we can define two user messages of the binary type. For our vision model, we'll use Google's Gemini but any capable vision model will also do the job. A Structured Output Parser is used here to return the AI's response in JSON format, this is for easier formatting purposes which we'll get to in the next step."
+},
+"typeVersion": 1
+},
+{
+"id": "a92d11e5-0985-4a8f-bc43-8bc0ca48e744",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+397.518987341772,
+93.8157360237642
+],
+"parameters": {
+"color": 7,
+"width": 885.2402868841493,
+"height": 388.92815062755926,
+"content": "## 7. Create Report In Linear\n[Learn more about integrating with Linear.app](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.linear)\n\nFor the final step, we'll generate a simple report which will capture any changes detected in our webpages list. Let's do this by first combining our webpages with their test results and filter out any in the page where no changes were detected. Next, we'll aggregate all changes into the Linear.app node which will be formatted into a markdown snippet and used to create a new issue in Linear. If you don't use Linear, feel free to swap this out for JIRA or even Slack."
+},
+"typeVersion": 1
+},
+{
+"id": "3f52c006-6c0a-456d-ab3c-ee5a16726299",
+"name": "Loop Over Items",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-1680,
+580
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "478ee25d-3f0f-4f6c-bf34-add1dc14c3cb",
+"name": "Wait",
+"type": "n8n-nodes-base.wait",
+"position": [
+-1240,
+820
+],
+"webhookId": "f06eab66-30a7-48ad-90ee-cb3394eb2edb",
+"parameters": {
+"amount": 2
+},
+"typeVersion": 1.1
+},
+{
+"id": "64b5f755-a85e-4ae5-ad81-113c1ef9b64c",
+"name": "Download Screenshot",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-1260,
+360
+],
+"parameters": {
+"url": "={{ $json.screenshotUrl }}",
+"options": {}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8f99ef1f-1cdc-4d80-b858-e9960a805dd4",
+"name": "Upload to Drive",
+"type": "n8n-nodes-base.googleDrive",
+"position": [
+-1080,
+360
+],
+"parameters": {
+"name": "={{ $('Merge').item.json.url.urlEncode() }}",
+"driveId": {
+"__rl": true,
+"mode": "list",
+"value": "My Drive"
+},
+"options": {
+"simplifyOutput": true
+},
+"folderId": {
+"__rl": true,
+"mode": "list",
+"value": "1lAFxJPWcA_sOcjr3UUKKfFfoTwd4Stkh",
+"cachedResultUrl": "https://drive.google.com/drive/folders/1lAFxJPWcA_sOcjr3UUKKfFfoTwd4Stkh",
+"cachedResultName": "base_images"
+}
+},
+"credentials": {
+"googleDriveOAuth2Api": {
+"id": "yOwz41gMQclOadgu",
+"name": "Google Drive account"
+}
+},
+"typeVersion": 3
+},
+{
+"id": "5e253123-89ba-42d5-b743-60bfd1ebae5b",
+"name": "Update Base Image",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+-900,
+360
+],
+"parameters": {
+"columns": {
+"value": {
+"url": "={{ $('Merge').item.json.url }}",
+"base": "={{ $json.id }}"
+},
+"schema": [
+{
+"id": "service",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "service",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "url",
+"type": "string",
+"display": true,
+"removed": false,
+"required": false,
+"displayName": "url",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+},
+{
+"id": "base",
+"type": "string",
+"display": true,
+"required": false,
+"displayName": "base",
+"defaultMatch": false,
+"canBeUsedToMatch": true
+}
+],
+"mappingMode": "defineBelow",
+"matchingColumns": [
+"url"
+]
+},
+"options": {},
+"operation": "appendOrUpdate",
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
+"cachedResultName": "Sheet1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
+"cachedResultName": "Visual Regression List"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.5
+},
+{
+"id": "fa7339b7-b7dd-4ecd-8dc2-f42f6549adb6",
+"name": "Merge",
+"type": "n8n-nodes-base.merge",
+"position": [
+-1440,
+360
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combineBy": "combineByPosition"
+},
+"typeVersion": 3
+},
+{
+"id": "47845df9-a50e-429e-b81e-5eefd996d5c7",
+"name": "Schedule Trigger",
+"type": "n8n-nodes-base.scheduleTrigger",
+"position": [
+-560,
+380
+],
+"parameters": {
+"rule": {
+"interval": [
+{
+"field": "weeks",
+"triggerAtDay": [
+1
+],
+"triggerAtHour": 6
+}
+]
+}
+},
+"typeVersion": 1.2
+},
+{
+"id": "63492aa4-3535-4832-a9d0-0a949e46ec81",
+"name": "Get URLs with Missing Base Images",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+-1980,
+480
+],
+"parameters": {
+"options": {},
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
+"cachedResultName": "Sheet1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
+"cachedResultName": "Visual Regression List"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.5
+},
+{
+"id": "8907f3b9-0613-4057-8adb-fd5c4e25cf72",
+"name": "Run Webpage Screenshot",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+-1420,
+820
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/apify~screenshot-url/run-sync-get-dataset-items",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"delay\": 0,\n \"format\": \"png\",\n \"proxy\": {\n \"useApifyProxy\": true\n },\n \"scrollToBottom\": false,\n \"urls\": [\n {\n \"url\": $json.url\n }\n ],\n \"viewportWidth\": 1280,\n \"waitUntil\": \"domcontentloaded\",\n \"waitUntilNetworkIdleAfterScroll\": false\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth"
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "3dc45b2d-4c4a-44d5-9b45-3e2144479603",
+"name": "Run Webpage Screenshot1",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+273,
+833
+],
+"parameters": {
+"url": "https://api.apify.com/v2/acts/apify~screenshot-url/run-sync-get-dataset-items",
+"method": "POST",
+"options": {},
+"jsonBody": "={{\n{\n \"delay\": 0,\n \"format\": \"png\",\n \"proxy\": {\n \"useApifyProxy\": true\n },\n \"scrollToBottom\": false,\n \"urls\": [\n {\n \"url\": $json.url\n }\n ],\n \"viewportWidth\": 1280,\n \"waitUntil\": \"domcontentloaded\",\n \"waitUntilNetworkIdleAfterScroll\": false\n}\n}}",
+"sendBody": true,
+"specifyBody": "json",
+"authentication": "genericCredentialType",
+"genericAuthType": "httpQueryAuth"
+},
+"credentials": {
+"httpQueryAuth": {
+"id": "cO2w8RDNOZg8DRa8",
+"name": "Apify API"
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "672d64fb-7782-427e-8779-953e51118fbc",
+"name": "Has Changes",
+"type": "n8n-nodes-base.filter",
+"position": [
+680,
+300
+],
+"parameters": {
+"options": {},
+"conditions": {
+"options": {
+"version": 2,
+"leftValue": "",
+"caseSensitive": true,
+"typeValidation": "strict"
+},
+"combinator": "and",
+"conditions": [
+{
+"id": "20b18a7e-bf98-4f39-baa9-4d965097526a",
+"operator": {
+"type": "array",
+"operation": "lengthGt",
+"rightType": "number"
+},
+"leftValue": "={{ $json.output }}",
+"rightValue": 0
+}
+]
+}
+},
+"typeVersion": 2.2
+},
+{
+"id": "efa168ec-ff05-471b-869f-cee5a222594a",
+"name": "Combine Row and Result",
+"type": "n8n-nodes-base.set",
+"position": [
+500,
+300
+],
+"parameters": {
+"mode": "raw",
+"options": {},
+"jsonOutput": "={{\n{\n ...$('Get Webpages List').item.json,\n output: $json.output\n}\n}}\n"
+},
+"typeVersion": 3.4
+},
+{
+"id": "1fe901dc-f460-41b8-8042-0fcb0474092f",
+"name": "Wait1",
+"type": "n8n-nodes-base.wait",
+"position": [
+1580,
+900
+],
+"webhookId": "6bbf2e65-bed1-4efc-bb31-09d12c644dc5",
+"parameters": {
+"amount": 1
+},
+"typeVersion": 1.1
+},
+{
+"id": "7891f052-4073-4746-a04b-27c7c4fa1e63",
+"name": "Aggregate",
+"type": "n8n-nodes-base.aggregate",
+"position": [
+860,
+300
+],
+"parameters": {
+"options": {},
+"aggregate": "aggregateAllItemData"
+},
+"typeVersion": 1
+},
+{
+"id": "ef2b2ddb-51f9-4576-bd99-9efa39be5163",
+"name": "Create Report",
+"type": "n8n-nodes-base.linear",
+"position": [
+1040,
+300
+],
+"parameters": {
+"title": "=Visual Regression Report {{ $now.format('yyyy-MM-dd') }}",
+"teamId": "1c721608-321d-4132-ac32-6e92d04bb487",
+"additionalFields": {
+"description": "=Visual Regression Workflow picked up the following changes:\n\n{{\n$json.data.map(row =>\n`### ${row.url}\n${row.output.map(issue =>\n`* **${issue.description}** - expected \"${issue.previous_state}\" but got \"${issue.current_state}\"`\n).join('\\n')}`\n).join('\\n');\n}}"
+}
+},
+"credentials": {
+"linearApi": {
+"id": "Nn0F7T9FtvRUtEbe",
+"name": "Linear account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "477b89f7-00ca-4001-a246-0887bcb553eb",
+"name": "When clicking โTest workflowโ",
+"type": "n8n-nodes-base.manualTrigger",
+"position": [
+-2180,
+480
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "eb7f6310-5465-4638-b702-5ecbd98a0199",
+"name": "Get Webpages List",
+"type": "n8n-nodes-base.googleSheets",
+"position": [
+-360,
+380
+],
+"parameters": {
+"options": {},
+"filtersUI": {
+"values": [
+{
+"lookupValue": "2",
+"lookupColumn": "=row_number"
+}
+]
+},
+"sheetName": {
+"__rl": true,
+"mode": "list",
+"value": "gid=0",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit#gid=0",
+"cachedResultName": "Sheet1"
+},
+"documentId": {
+"__rl": true,
+"mode": "list",
+"value": "1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84",
+"cachedResultUrl": "https://docs.google.com/spreadsheets/d/1RbobwHCJiYKnic6T-VA3kPr-Grd4Y_ZSQXqy2st_T84/edit?usp=drivesdk",
+"cachedResultName": "Visual Regression List"
+}
+},
+"credentials": {
+"googleSheetsOAuth2Api": {
+"id": "XHvC7jIRR8A2TlUl",
+"name": "Google Sheets account"
+}
+},
+"typeVersion": 4.5
+},
+{
+"id": "6c0f7341-14c9-48c2-9447-edab0ad18df7",
+"name": "For Each Webpage...",
+"type": "n8n-nodes-base.splitInBatches",
+"position": [
+-40,
+440
+],
+"parameters": {
+"options": {}
+},
+"typeVersion": 3
+},
+{
+"id": "62e13166-458d-4c63-8911-740f9ceaeb54",
+"name": "Sticky Note4",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-660,
+160
+],
+"parameters": {
+"color": 7,
+"width": 561.2038065501644,
+"height": 408.0284015307624,
+"content": "## 4. Trigger Visual Regression Test Run\n[Read more about the Schedule Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.scheduletrigger/)\n\nOnce we've generated our base images to compare with in Part A, we can now run our Visual Regression Tests. These tests are intended to check for unexpected changes to a webpage by using some form of image detection. To trigger Part B, we'll start with a schedule trigger and pull a list of webpages to test from Google Sheets."
+},
+"typeVersion": 1
+},
+{
+"id": "8d958f44-fd2c-49b4-adbd-d8a99b2614c8",
+"name": "Sticky Note5",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2340,
+218.0216140230686
+],
+"parameters": {
+"color": 7,
+"width": 626.9985071319608,
+"height": 487.40071048786325,
+"content": "## 1. Get List of Webpages to Generate Base Images\n[Learn more about using Google Sheets](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googlesheets/)\n\nThis workflow is split into 2 parts: Part A will generate the \"base\" screenshots to compare new screenshots against. To capture these base screenshots, we'll use Google Sheets to hold our list of webpages and their screenshot references (we'll come on to that later).\n\nExample Sheet: https://docs.google.com/spreadsheets/d/e/2PACX-1vTXRZRi55dUbLAsWZboJqH5U-EK0ZRSse8pkqANAV4Ss70otpQ97zgT8YBd3dL4d2u2UC1TTx_o1o1R/pubhtml"
+},
+"typeVersion": 1
+},
+{
+"id": "ee776b4d-4532-4c08-ac38-35d40afbd8ad",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1480,
+580
+],
+"parameters": {
+"color": 7,
+"width": 653.369086691465,
+"height": 443.1120543367141,
+"content": "## 2. Generate Webpage Screenshot via Apify\n[Learn more about Apify.com](https://www.apify.com?fpr=414q6)\n\nTo generate a screenshot of the webpage, we'll need a third party service since this functionality is outside the scope of n8n. Feel free to pick whichever internal or external service works for you but I've had great experience using [Apify.com](https://www.apify.com?fpr=414q6) - a popular webscraping SaaS who offer a generous free plan and require very little configuration to get started.\n\nThe Apify \"actor\" (ie. a type of scraper) we'll be using is specifically designed to take webpage screenshots."
+},
+"typeVersion": 1
+},
+{
+"id": "3d90e103-2829-4075-b3d4-5ba848af4843",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1520,
+160
+],
+"parameters": {
+"color": 7,
+"width": 808.188722669735,
+"height": 397.73072497123115,
+"content": "## 3. Upload Screenshot to Google Drive\n[Read more about using the Google Drive node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.googledrive/)\n\nOnce we have our screenshots, we'll download them from Apify and upload them to our Google Drive for safe keeping. After uploading, we'll capture the new Google Drive IDs for the images into our Google Sheet, this will allow us to reference them again when we perform the visual regression testing."
+},
+"typeVersion": 1
+},
+{
+"id": "e47d14ec-ad78-42c8-a294-301dcd581a67",
+"name": "Download New Screenshot",
+"type": "n8n-nodes-base.httpRequest",
+"position": [
+453,
+833
+],
+"parameters": {
+"url": "={{ $json.screenshotUrl }}",
+"options": {
+"response": {
+"response": {
+"responseFormat": "file",
+"outputPropertyName": "data_2"
+}
+}
+}
+},
+"typeVersion": 4.2
+},
+{
+"id": "8ca118bc-3d19-48ac-9d9c-0892993da736",
+"name": "Combine Screenshots",
+"type": "n8n-nodes-base.merge",
+"position": [
+660,
+660
+],
+"parameters": {
+"mode": "combine",
+"options": {},
+"combineBy": "combineByPosition"
+},
+"typeVersion": 3
+},
+{
+"id": "03359cbb-d7af-4118-a32a-3fe24062dc9f",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-660,
+20
+],
+"parameters": {
+"color": 4,
+"width": 394.03359370567625,
+"height": 111.52173490405977,
+"content": "### Part B. Run Visual Regression Test\nIn this part of the workflow, we'll retrieve our list of webpages to test with our AI vision model. This part can be run as many times as required."
+},
+"typeVersion": 1
+},
+{
+"id": "a78c0f92-aa61-483b-95bf-dd60958f182d",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2920,
+220
+],
+"parameters": {
+"width": 553.2963720930223,
+"height": 473.4987906976746,
+"content": "## Try It Out!\n\n### This workflow implements an approach to Visual Regression Testing - a means to test websites for defects - using AI Vision Models.\n\nThis workflow uses a Google Sheet to track a list of webpages to test and is split into 2 parts; Part A generates the base screenshots of the list and Part B runs the visual regression testing.\n\nThe example spreadsheet can be found here: https://docs.google.com/spreadsheets/d/e/2PACX-1vTXRZRi55dUbLAsWZboJqH5U-EK0ZRSse8pkqANAV4Ss70otpQ97zgT8YBd3dL4d2u2UC1TTx_o1o1R/pubhtml\n\n**[Apify.com](https://www.apify.com?fpr=414q6)** is the screenshot generator of choice and a free account with $5 in credit is available via this [link](https://www.apify.com?fpr=414q6).\n\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!"
+},
+"typeVersion": 1
+},
+{
+"id": "a0b257e5-99f8-409a-bc67-2468db377d6c",
+"name": "Visual Regression Agent",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+1120,
+740
+],
+"parameters": {
+"text": "Identify changes between the base image and test image.",
+"messages": {
+"messageValues": [
+{
+"message": "=You help with visual regression testing for websites. Identify changes to text content, images, colors, position and layouts of the elements in the screenshots. Ignore text styling or casing changes.\nThe first image will be the base image and the second image will be the test. Note all changes to the test image which differ from the base. If there are no changes, it is okay to return an empty array."
+},
+{
+"type": "HumanMessagePromptTemplate",
+"messageType": "imageBinary",
+"binaryImageDataKey": "data_1"
+},
+{
+"type": "HumanMessagePromptTemplate",
+"messageType": "imageBinary",
+"binaryImageDataKey": "data_2"
+}
+]
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.4
+}
+],
+"pinData": {},
+"connections": {
+"Wait": {
+"main": [
+[
+{
+"node": "Loop Over Items",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Merge": {
+"main": [
+[
+{
+"node": "Download Screenshot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait1": {
+"main": [
+[
+{
+"node": "For Each Webpage...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Aggregate": {
+"main": [
+[
+{
+"node": "Create Report",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Base Image": {
+"main": [
+[
+{
+"node": "Combine Screenshots",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Changes": {
+"main": [
+[
+{
+"node": "Aggregate",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Loop Over Items": {
+"main": [
+[
+{
+"node": "Merge",
+"type": "main",
+"index": 1
+}
+],
+[
+{
+"node": "Run Webpage Screenshot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Upload to Drive": {
+"main": [
+[
+{
+"node": "Update Base Image",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Schedule Trigger": {
+"main": [
+[
+{
+"node": "Get Webpages List",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Webpages List": {
+"main": [
+[
+{
+"node": "For Each Webpage...",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Combine Screenshots": {
+"main": [
+[
+{
+"node": "Visual Regression Agent",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Download Screenshot": {
+"main": [
+[
+{
+"node": "Upload to Drive",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"For Each Webpage...": {
+"main": [
+[
+{
+"node": "Combine Row and Result",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Base Image",
+"type": "main",
+"index": 0
+},
+{
+"node": "Run Webpage Screenshot1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Combine Row and Result": {
+"main": [
+[
+{
+"node": "Has Changes",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Run Webpage Screenshot": {
+"main": [
+[
+{
+"node": "Wait",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Download New Screenshot": {
+"main": [
+[
+{
+"node": "Combine Screenshots",
+"type": "main",
+"index": 1
+}
+]
+]
+},
+"Run Webpage Screenshot1": {
+"main": [
+[
+{
+"node": "Download New Screenshot",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Visual Regression Agent": {
+"main": [
+[
+{
+"node": "Wait1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Google Gemini Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Visual Regression Agent",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Structured Output Parser": {
+"ai_outputParser": [
+[
+{
+"node": "Visual Regression Agent",
+"type": "ai_outputParser",
+"index": 0
+}
+]
+]
+},
+"Get URLs with Missing Base Images": {
+"main": [
+[
+{
+"node": "Loop Over Items",
+"type": "main",
+"index": 0
+},
+{
+"node": "Merge",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"When clicking โTest workflowโ": {
+"main": [
+[
+{
+"node": "Get URLs with Missing Base Images",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/AI_Research_RAG_and_Data_Analysis/๐ Perplexity Research to HTML_ AI-Powered Content Creation.json b/AI_Research_RAG_and_Data_Analysis/๐ Perplexity Research to HTML_ AI-Powered Content Creation.json
deleted file mode 100644
index bba07ad..0000000
--- a/AI_Research_RAG_and_Data_Analysis/๐ Perplexity Research to HTML_ AI-Powered Content Creation.json
+++ /dev/null
@@ -1,1389 +0,0 @@
-{
- "id": "HnqGW0eq5asKfZxf",
- "meta": {
- "instanceId": "03907a25f048377a8789a4332f28148522ba31ee907fababf704f1d88130b1b6",
- "templateCredsSetupCompleted": true
- },
- "name": "๐๐ ๏ธPerplexity Researcher to HTML Web Page",
- "tags": [],
- "nodes": [
- {
- "id": "ad5d96c6-941a-4ab3-b349-10bae99e5988",
- "name": "Sticky Note",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 320,
- 1360
- ],
- "parameters": {
- "color": 3,
- "width": 625.851492623043,
- "height": 465.2493344282225,
- "content": "## Create Article from Perplexity Research"
- },
- "typeVersion": 1
- },
- {
- "id": "19b3ca66-5fd2-4d04-b25a-a17fb38642f8",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 1240,
- 1360
- ],
- "parameters": {
- "color": 4,
- "width": 479.02028317328745,
- "height": 464.14912719677955,
- "content": "## Convert Article into HTML"
- },
- "typeVersion": 1
- },
- {
- "id": "7fad54e8-5a50-42da-b38d-08f6912615ab",
- "name": "gpt-4o-mini",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1380,
- 1660
- ],
- "parameters": {
- "model": "gpt-4o-mini-2024-07-18",
- "options": {
- "responseFormat": "text"
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "h597GY4ZJQD47RQd",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "5291869f-3ac6-4ce2-88f3-b572924b6082",
- "name": "gpt-4o-mini1",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 1560,
- 1040
- ],
- "parameters": {
- "options": {
- "topP": 1,
- "timeout": 60000,
- "maxTokens": -1,
- "maxRetries": 2,
- "temperature": 0,
- "responseFormat": "text",
- "presencePenalty": 0,
- "frequencyPenalty": 0
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "h597GY4ZJQD47RQd",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "a232f6ca-ad4c-40fa-a641-f0dd83c8f18a",
- "name": "Structured Output Parser1",
- "type": "@n8n/n8n-nodes-langchain.outputParserStructured",
- "position": [
- 640,
- 1660
- ],
- "parameters": {
- "schemaType": "manual",
- "inputSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"article\": {\n \"type\": \"object\",\n \"required\": [\"category\", \"title\", \"metadata\", \"content\", \"hashtags\"],\n \"properties\": {\n \"category\": {\n \"type\": \"string\",\n \"description\": \"Article category\"\n },\n \"title\": {\n \"type\": \"string\",\n \"description\": \"Article title\"\n },\n \"metadata\": {\n \"type\": \"object\",\n \"properties\": {\n \"timePosted\": {\n \"type\": \"string\",\n \"description\": \"Time since article was posted\"\n },\n \"author\": {\n \"type\": \"string\",\n \"description\": \"Article author name\"\n },\n \"tag\": {\n \"type\": \"string\",\n \"description\": \"Article primary tag\"\n }\n },\n \"required\": [\"timePosted\", \"author\", \"tag\"]\n },\n \"content\": {\n \"type\": \"object\",\n \"properties\": {\n \"mainText\": {\n \"type\": \"string\",\n \"description\": \"Main article content\"\n },\n \"sections\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"title\": {\n \"type\": \"string\",\n \"description\": \"Section title\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"Section content\"\n },\n \"quote\": {\n \"type\": \"string\",\n \"description\": \"Blockquote text\"\n }\n },\n \"required\": [\"title\", \"text\", \"quote\"]\n }\n }\n },\n \"required\": [\"mainText\", \"sections\"]\n },\n \"hashtags\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"string\"\n },\n \"description\": \"Article hashtags\"\n }\n }\n }\n }\n}"
- },
- "typeVersion": 1.2
- },
- {
- "id": "e7d1adac-88aa-4f76-92bf-bbac3aa6386a",
- "name": "gpt-4o-mini2",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 420,
- 1660
- ],
- "parameters": {
- "options": {
- "topP": 1,
- "timeout": 60000,
- "maxTokens": -1,
- "maxRetries": 2,
- "temperature": 0,
- "responseFormat": "json_object",
- "presencePenalty": 0,
- "frequencyPenalty": 0
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "h597GY4ZJQD47RQd",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "156e51db-03f7-4099-afe8-6f0361c5b497",
- "name": "Webhook",
- "type": "n8n-nodes-base.webhook",
- "position": [
- 160,
- 860
- ],
- "webhookId": "6a8e3ae7-02ae-4663-a27a-07df448550ab",
- "parameters": {
- "path": "pblog",
- "options": {},
- "responseMode": "responseNode"
- },
- "typeVersion": 2
- },
- {
- "id": "6dd3eba7-e779-4e4a-960e-c5a7b6b3a929",
- "name": "Respond to Webhook",
- "type": "n8n-nodes-base.respondToWebhook",
- "position": [
- 2820,
- 1480
- ],
- "parameters": {
- "options": {},
- "respondWith": "text",
- "responseBody": "={{ $json.text }}"
- },
- "typeVersion": 1.1
- },
- {
- "id": "27ee681e-4259-4323-b4fe-629f99cb33d0",
- "name": "Telegram",
- "type": "n8n-nodes-base.telegram",
- "position": [
- 2320,
- 880
- ],
- "parameters": {
- "text": "={{ $('Perplexity Topic Agent').item.json.output.slice(0, 300) }}",
- "chatId": "={{ $json.telegram_chat_id }}",
- "additionalFields": {
- "parse_mode": "HTML",
- "appendAttribution": false
- }
- },
- "credentials": {
- "telegramApi": {
- "id": "BIE64nzfpGeesXUn",
- "name": "Telegram account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "f437d40c-2bf6-43e2-b77b-e5c2cdc35055",
- "name": "gpt-4o-mini5",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- 2480,
- 1660
- ],
- "parameters": {
- "options": {
- "topP": 1,
- "timeout": 60000,
- "maxTokens": -1,
- "maxRetries": 2,
- "temperature": 0,
- "responseFormat": "text",
- "presencePenalty": 0,
- "frequencyPenalty": 0
- }
- },
- "credentials": {
- "openAiApi": {
- "id": "h597GY4ZJQD47RQd",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "275bce4a-4252-41d4-bcba-174f0c51bf4a",
- "name": "Basic LLM Chain",
- "type": "@n8n/n8n-nodes-langchain.chainLlm",
- "position": [
- 2340,
- 1480
- ],
- "parameters": {
- "text": "=Create a modern, responsive single-line HTML document. Convert any markdown to Tailwind CSS classes. Replace markdown lists with proper HTML list elements. Remove all newline characters while preserving tags in content. Enhance the layout with Tailwind CSS cards where appropriate. Use the following base structure, but improve the styling and responsiveness:\n\n\n\n\n\n \n \n Comprehensive Overview of DeepSeek V3\n \n\n\n\n
\n
\n
\n
\n
Comprehensive Overview of DeepSeek V3
\n
\n
Time Posted: Just now
\n
Author: AI Research Team
\n
Tag: AI Models
\n
\n
DeepSeek V3 is a state-of-the-art AI model that leverages\n advanced architectures and techniques to deliver high performance across various applications.\n This overview covers its key concepts, practical applications, advantages, limitations, and best\n practices for implementation.
\n \n
Key Concepts and Core Components
\n
1. Mixture-of-Experts (MoE) Architecture: DeepSeek V3\n employs a Mixture-of-Experts (MoE) architecture, which consists of multiple neural networks,\n each optimized for different tasks. This architecture allows for efficient processing by\n activating only a portion of the network for each task, reducing hardware costs.
\n
2. Parameters: The model boasts a total of 671\n billion\n parameters, with 37 billion active parameters for each token during processing. The addition\n of\n the Multi-Token Prediction (MTP) module increases the total parameters to 685 billion,\n making it\n significantly larger than other models like Meta's Llama 3.1 (405B).
\n
3. Multi-head Latent Attention (MLA): DeepSeek V3\n uses\n Multi-head Latent Attention (MLA) to extract key details from text multiple times, improving\n its\n accuracy.
\n
4. Multi-Token Prediction (MTP): The model utilizes\n Multi-Token Prediction to generate several tokens at once, speeding up inference and\n enabling\n speculative decoding.
\n
\n DeepSeek V3 employs a Mixture-of-Experts architecture for efficient processing.
\n \n \n
Practical Applications
\n \n
Translation, Coding, and Content Generation:\n DeepSeek V3 is designed for a wide range of tasks including translation, coding, content\n generation, and reasoning. It excels in English, Chinese, coding, and mathematics,\n rivaling leading commercial models like OpenAI's GPT-4.
\n
Research and Development: The open-source nature\n of DeepSeek V3 fuels innovation, allowing researchers to experiment with and build upon\n its technology.
\n
Commercial Applications: The licensing of\n DeepSeek V3 makes it permissible for commercial use, opening it up to numerous\n applications across different industries.
\n
Democratization of AI: By making powerful AI\n accessible, DeepSeek V3 levels the playing field, allowing smaller organizations to\n compete with larger ones.
\n \n
\n DeepSeek V3 democratizes AI access for smaller organizations.
\n \n \n
Advantages
\n \n
Speed and Efficiency: DeepSeek V3 processes\n information at a blistering 60 tokens per second, a threefold increase over its\n predecessor. It uses advanced inference capabilities, deploying 32 H800 GPUs for prefill\n and 320 H800 GPUs for decoding.
\n
Cost-Effectiveness: The model was trained for a\n mere $5.5 million, a fraction of the estimated over $100 million invested by OpenAI in\n GPT-4. DeepSeek V3 offers significantly lower prices for its online services, with 1\n million tokens priced at just $1.1, currently offered at a promotional rate of $0.28.\n
\n
Innovation in Inference: The model's advanced\n inference capabilities set the standard for future model deployment, making it a\n powerful tool in the digital realm.
\n \n
\n DeepSeek V3 processes information at 60 tokens per second.
\n \n \n
Limitations
\n \n
Deployment Complexity: Deploying DeepSeek V3\n requires advanced hardware and a deployment strategy that separates the prefilling and\n decoding stages, which might be unachievable for small companies due to a lack of\n resources. The recommended deployment unit for DeepSeek V3 is relatively large, posing a\n burden for small-sized teams.
\n
Potential for Further Enhancement: Although\n DeepSeek V3 has achieved an end-to-end generation speed of more than two times that of\n DeepSeek V2, there still remains potential for further enhancement with the development\n of more advanced hardware.
\n \n
\n Deployment of DeepSeek V3 may be complex for small companies.
\n \n \n
Best Practices for Implementation
\n \n
Hardware Requirements: Ensure that the\n deployment environment has the necessary advanced hardware to handle the model's\n requirements, including multiple GPUs for prefill and decoding.
\n
Deployment Strategy: Implement a deployment\n strategy that separates the prefilling and decoding stages to optimize performance and\n efficiency.
\n
Monitoring and Optimization: Continuously\n monitor the model's performance and optimize it as needed to address any limitations and\n improve efficiency.
\n
Community Engagement: Engage with the\n open-source community to leverage the collective knowledge and resources available,\n which can help in addressing any challenges and improving the model further.
\n \n
\n Engage with the open-source community for better implementation.
\n\t\n\n"
- },
- "typeVersion": 1.2
- },
- {
- "id": "0e968154-ead5-4194-834e-0d1175e7c1d9",
- "name": "AI Researcher",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- 2900,
- 920
- ],
- "parameters": {
- "text": "={{ $json.answer }}",
- "options": {
- "systemMessage": "=You are a user research expert interviewing a user on the topic of \"{{ $('Set Interview Topic').first().json.interview_topic }}\".\n\n* Your task is to ask open-ended questions relevant to the interview topic.\n* Ask only one question at a time. Analyse the previous question and ask new question each time. If there is an opportunity to dig deeper into a previous answer, do so but limit to 1 follow-on question.\n* Keep asking questions until the user requests to stop the interview. When the user requests to stop the interview and no question is required, \"question\" is an empty string.\n* Use a friendly and polite tone when asking questions.\n* If the user answers are inrelevant to the question, ask the question again or move on to another question.\n* If the user's answer is beyond the scope of the interview, ignore the answer and ask if the user would like to stop the interview.\n*You must format your response using the following json schema as we require pre processing before responding to the user.\n```\n{\n \"type\":\"object\",\n \"properties\": {\n \"stop_interview\": { \"type\": \"boolean\" },\n \"question\": { \"type\": [\"string\", \"null\"] }\n }\n}\n```\n* Output only the json object and do not prefix or suffix the message with extraneous text."
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.7
- },
- {
- "id": "969d4094-1046-4f53-bf8b-5ae7e50bd3ed",
- "name": "Parse Response",
- "type": "n8n-nodes-base.set",
- "position": [
- 3220,
- 920
- ],
- "parameters": {
- "options": {},
- "assignments": {
- "assignments": [
- {
- "id": "bf61134c-e24c-453e-97ef-5edd25726148",
- "name": "output",
- "type": "object",
- "value": "={{\n$json.output\n .replace('```json', '')\n .replace('```', '')\n .parseJson()\n}}"
- }
- ]
- }
- },
- "typeVersion": 3.4
- },
- {
- "id": "323b73c4-8c77-48a9-a549-f3e863ba72c2",
- "name": "Groq Chat Model",
- "type": "@n8n/n8n-nodes-langchain.lmChatGroq",
- "position": [
- 2860,
- 1120
- ],
- "parameters": {
- "model": "llama-3.2-90b-text-preview",
- "options": {}
- },
- "credentials": {
- "groqApi": {
- "id": "YQVoV5K9FREww7t1",
- "name": "Groq account"
- }
- },
- "typeVersion": 1
- },
- {
- "id": "bf4518c4-8e59-450e-be5a-92f31cf38528",
- "name": "Show Transcript",
- "type": "n8n-nodes-base.html",
- "position": [
- 5260,
- 1140
- ],
- "parameters": {
- "html": "\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n\t\tAI Interviewer Transcripts\n\n\t\t\n\t\n\n\t\n\t\t
\n\t\n\n"
+},
+"typeVersion": 1.2
+},
+{
+"id": "0e968154-ead5-4194-834e-0d1175e7c1d9",
+"name": "AI Researcher",
+"type": "@n8n/n8n-nodes-langchain.agent",
+"position": [
+2900,
+920
+],
+"parameters": {
+"text": "={{ $json.answer }}",
+"options": {
+"systemMessage": "=You are a user research expert interviewing a user on the topic of \"{{ $('Set Interview Topic').first().json.interview_topic }}\".\n\n* Your task is to ask open-ended questions relevant to the interview topic.\n* Ask only one question at a time. Analyse the previous question and ask new question each time. If there is an opportunity to dig deeper into a previous answer, do so but limit to 1 follow-on question.\n* Keep asking questions until the user requests to stop the interview. When the user requests to stop the interview and no question is required, \"question\" is an empty string.\n* Use a friendly and polite tone when asking questions.\n* If the user answers are inrelevant to the question, ask the question again or move on to another question.\n* If the user's answer is beyond the scope of the interview, ignore the answer and ask if the user would like to stop the interview.\n*You must format your response using the following json schema as we require pre processing before responding to the user.\n```\n{\n \"type\":\"object\",\n \"properties\": {\n \"stop_interview\": { \"type\": \"boolean\" },\n \"question\": { \"type\": [\"string\", \"null\"] }\n }\n}\n```\n* Output only the json object and do not prefix or suffix the message with extraneous text."
+},
+"promptType": "define",
+"hasOutputParser": true
+},
+"typeVersion": 1.7
+},
+{
+"id": "969d4094-1046-4f53-bf8b-5ae7e50bd3ed",
+"name": "Parse Response",
+"type": "n8n-nodes-base.set",
+"position": [
+3220,
+920
+],
+"parameters": {
+"options": {},
+"assignments": {
+"assignments": [
+{
+"id": "bf61134c-e24c-453e-97ef-5edd25726148",
+"name": "output",
+"type": "object",
+"value": "={{\n$json.output\n .replace('```json', '')\n .replace('```', '')\n .parseJson()\n}}"
+}
+]
+}
+},
+"typeVersion": 3.4
+},
+{
+"id": "323b73c4-8c77-48a9-a549-f3e863ba72c2",
+"name": "Groq Chat Model",
+"type": "@n8n/n8n-nodes-langchain.lmChatGroq",
+"position": [
+2860,
+1120
+],
+"parameters": {
+"model": "llama-3.2-90b-text-preview",
+"options": {}
+},
+"credentials": {
+"groqApi": {
+"id": "YQVoV5K9FREww7t1",
+"name": "Groq account"
+}
+},
+"typeVersion": 1
+},
+{
+"id": "bf4518c4-8e59-450e-be5a-92f31cf38528",
+"name": "Show Transcript",
+"type": "n8n-nodes-base.html",
+"position": [
+5260,
+1140
+],
+"parameters": {
+"html": "\n\n\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n\t\tAI Interviewer Transcripts\n\n\t\t\n\t\n\n\t\n\t\t
Unfortunately, we cannot schedule the requested appointment at the requested time.
\n
Kind regards
\n",
+"options": {},
+"subject": "=Appointment Request Rejected for {{ DateTime.fromISO($('Execute Workflow Trigger').first().json.dateTime).format('EEE, dd MMM @ t') }}"
+},
+"credentials": {
+"gmailOAuth2": {
+"id": "Sf5Gfl9NiFTNXFWb",
+"name": "Gmail account"
+}
+},
+"typeVersion": 2.1
+},
+{
+"id": "40785eca-943c-45f6-b4a9-0c95538621ed",
+"name": "Sticky Note3",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-2580,
+-555.2889298043726
+],
+"parameters": {
+"color": 7,
+"width": 763.0427617951669,
+"height": 611.898918296892,
+"content": "## 1. Qualify Enquiries Using AI\n[Learn more about the text classifier](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.text-classifier/)\n\nWith n8n's multi-forms, youโre no longer stuck creating long, overwhelming forms. Instead, you have more flexibility and control to design smarter, more engaging form experiences.\n\nIn this demo, weโll explore an appointment request scenario where a user wants to schedule a call to discuss their inquiry. However, not all inquiries require a meeting, making it a perfect use case for AI to pre-qualify the request. We can handle this validation using the text classifier node."
+},
+"typeVersion": 1
+},
+{
+"id": "985be8d1-e77a-475b-9ac2-dba163dbd950",
+"name": "Sticky Note6",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1800,
+-549.8684464902185
+],
+"parameters": {
+"color": 7,
+"width": 781.472405063291,
+"height": 606.0718987341766,
+"content": "## 2. Split Form For Better User Experience\n[Learn more about the forms](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.form)\n\nOnboarding is a great reason to split your big form into smaller ones. Taking the user through a step by step process ensures a smooth experience and keeps them engaged throughout.\n\nHere, we take the opportunity of the extra context space to display a terms and conditions which the user must agree to making their request. The next form then asks for desired date and time of the event."
+},
+"typeVersion": 1
+},
+{
+"id": "9b0a3f0e-e15d-4d0e-b620-1acc78bf812c",
+"name": "Decline",
+"type": "n8n-nodes-base.form",
+"position": [
+-2020,
+-160
+],
+"webhookId": "4353eadb-b7a0-45f2-8dd8-5f6cd882d8d8",
+"parameters": {
+"options": {},
+"operation": "completion",
+"completionTitle": "Send me a DM Instead!",
+"completionMessage": "Thanks for your enquiry but it may not necessarily need an appointment. Please feel free to email me instead at jim@example.com."
+},
+"typeVersion": 1
+},
+{
+"id": "fcd3eb7d-6389-4c07-97cc-275ae387c963",
+"name": "Decline1",
+"type": "n8n-nodes-base.form",
+"position": [
+-1260,
+-160
+],
+"webhookId": "4353eadb-b7a0-45f2-8dd8-5f6cd882d8d8",
+"parameters": {
+"options": {},
+"operation": "completion",
+"completionTitle": "Send me a DM Instead!",
+"completionMessage": "Thanks for your enquiry but it may not necessarily need an appointment. Please feel free to email me instead at jim@example.com."
+},
+"typeVersion": 1
+},
+{
+"id": "d89427cb-fffb-4aa4-b55c-b315fa0e92be",
+"name": "Sticky Note7",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-1000,
+-498.80432681242814
+],
+"parameters": {
+"color": 7,
+"width": 792.9401150747982,
+"height": 497.4250863060987,
+"content": "## 3. Send Acknowledgement to User and Start Approval Process\n[Learn more about the Gmail node](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.gmail/)\n\nOnce all form steps are concluded, we can send a notification to the requester via email and in the background, trigger another email to the admin to initiate the approval process. The approval process works in a separate execution so doesn't interrupt the user's form experience."
+},
+"typeVersion": 1
+},
+{
+"id": "041081e1-ee98-4b40-aa14-1980b23f4031",
+"name": "Sticky Note8",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-160,
+-620
+],
+"parameters": {
+"color": 7,
+"width": 609.4228768699652,
+"height": 287.178089758343,
+"content": "## 4. Approve or Decline Appointment\n[Learn more about the Waiting for Approval](https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.gmail/message-operations/#send-a-message-and-wait-for-approval)\n\nThe Wait for Approval feature for Gmail is a special operation which allows for human-in-the-loop interaction in n8n workflows. In this example, the human interaction is the approval of the appointment request. The feature will put the workflow in a waiting state where a message is sent to the admin with 2 buttons: confirm and decline.\n\nWhen the admin clicks on the confirm button, the workflow resumes from the Gmail node and a meeting event is created for the requesting user in Google Calendar.\n\nWhen declined, a rejection email is sent to the requester instead."
+},
+"typeVersion": 1
+},
+{
+"id": "d6af0f50-234f-46ca-aa41-7f3891aff8a3",
+"name": "Trigger Approval Process",
+"type": "n8n-nodes-base.executeWorkflow",
+"position": [
+-740,
+-260
+],
+"parameters": {
+"mode": "each",
+"options": {
+"waitForSubWorkflow": false
+},
+"workflowId": {
+"__rl": true,
+"mode": "id",
+"value": "={{ $workflow.id }}"
+}
+},
+"typeVersion": 1.1
+},
+{
+"id": "e524d6df-9b6d-4d61-8e71-08a0d3a751d7",
+"name": "Execute Workflow Trigger",
+"type": "n8n-nodes-base.executeWorkflowTrigger",
+"position": [
+-160,
+-260
+],
+"parameters": {},
+"typeVersion": 1
+},
+{
+"id": "74dccbc1-7728-4336-a18a-2541007fd369",
+"name": "Summarise Enquiry",
+"type": "@n8n/n8n-nodes-langchain.chainLlm",
+"position": [
+0,
+-260
+],
+"parameters": {
+"text": "=The enquiry is as follows:\n{{ $('Execute Workflow Trigger').first().json.enquiry.substring(0, 500) }}",
+"messages": {
+"messageValues": [
+{
+"message": "Summarise the given enquiry"
+}
+]
+},
+"promptType": "define"
+},
+"typeVersion": 1.5
+},
+{
+"id": "b74f0f5a-39f0-4db3-beba-03caf981c5d2",
+"name": "Sticky Note9",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+-3080,
+-640
+],
+"parameters": {
+"width": 468.6766398158801,
+"height": 690.6653164556957,
+"content": "## Try it out!\n\n### This n8n template is a simple appointment scheduling workflow using n8n forms with AI thrown in the mix for good measure. It also uses n8n's wait for approval feature which allows the ability to confirm appointment requests and create events in Google Calendar.\n\n### How it works\n* We start with a form trigger which asks for the purpose of the appointment.\n* Instantly, we can qualify this by using a text classifier node which uses AI's contextual understanding to ensure the appointment is worthwhile. If not, an alternative is suggested instead.\n* Multi-page forms are then used to set the terms of the appointment and ask the user for a desired date and time.\n* An acknowledgement is sent to the user while an approval by email process is triggered in the background.\n* In a subworkflow, we use Gmail with the wait for approval operation to send an approval form to the admin user who can either confirm or decline the appointment request.\n* When approved, a Google Calendar event is created. When declined, the user is notified via email that the appointment request was declined.\n\n### Need Help?\nJoin the [Discord](https://discord.com/invite/XPKeKXeB7d) or ask in the [Forum](https://community.n8n.io/)!\n\nHappy Hacking!\n"
+},
+"typeVersion": 1
+},
+{
+"id": "d3c87dfa-d6e5-402a-89e5-6d8f93b824a6",
+"name": "Sticky Note",
+"type": "n8n-nodes-base.stickyNote",
+"position": [
+299,
+-280
+],
+"parameters": {
+"width": 177.66444188722656,
+"height": 257.56869965477557,
+"content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n### ๐จ Set your admin email here!"
+},
+"typeVersion": 1
+},
+{
+"id": "6351121d-6ebe-432d-b370-13296fd58e1a",
+"name": "Enquiry Classifier",
+"type": "@n8n/n8n-nodes-langchain.textClassifier",
+"position": [
+-2340,
+-280
+],
+"parameters": {
+"options": {
+"fallback": "other"
+},
+"inputText": "={{ $json.Enquiry }}",
+"categories": {
+"categories": [
+{
+"category": "relevant enquiry",
+"description": "Enquire about AI, automation, digital products and product engineering."
+}
+]
+}
+},
+"typeVersion": 1
+}
+],
+"pinData": {},
+"connections": {
+"Send Receipt": {
+"main": [
+[
+{
+"node": "Form End",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Accepted?": {
+"main": [
+[
+{
+"node": "Enter Date & Time",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Decline1",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Has Approval?": {
+"main": [
+[
+{
+"node": "Create Appointment",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Send Rejection",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Get Form Values": {
+"main": [
+[
+{
+"node": "Trigger Approval Process",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"n8n Form Trigger": {
+"main": [
+[
+{
+"node": "Enquiry Classifier",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Enter Date & Time": {
+"main": [
+[
+{
+"node": "Get Form Values",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model": {
+"ai_languageModel": [
+[
+{
+"node": "Enquiry Classifier",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Summarise Enquiry": {
+"main": [
+[
+{
+"node": "Wait for Approval",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Wait for Approval": {
+"main": [
+[
+{
+"node": "Has Approval?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Enquiry Classifier": {
+"main": [
+[
+{
+"node": "Terms & Conditions",
+"type": "main",
+"index": 0
+}
+],
+[
+{
+"node": "Decline",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"OpenAI Chat Model1": {
+"ai_languageModel": [
+[
+{
+"node": "Summarise Enquiry",
+"type": "ai_languageModel",
+"index": 0
+}
+]
+]
+},
+"Terms & Conditions": {
+"main": [
+[
+{
+"node": "Has Accepted?",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Execute Workflow Trigger": {
+"main": [
+[
+{
+"node": "Summarise Enquiry",
+"type": "main",
+"index": 0
+}
+]
+]
+},
+"Trigger Approval Process": {
+"main": [
+[
+{
+"node": "Send Receipt",
+"type": "main",
+"index": 0
+}
+]
+]
+}
+}
+}
\ No newline at end of file
diff --git a/Gmail_and_Email_Automation/A Very Simple _Human in the Loop_ Email Response System Using AI and IMAP.json b/Gmail_and_Email_Automation/A Very Simple _Human in the Loop_ Email Response System Using AI and IMAP.json
deleted file mode 100644
index 7a2626d..0000000
--- a/Gmail_and_Email_Automation/A Very Simple _Human in the Loop_ Email Response System Using AI and IMAP.json
+++ /dev/null
@@ -1,436 +0,0 @@
-{
- "id": "Nvn78tMRNnKji7Fg",
- "meta": {
- "instanceId": "a4bfc93e975ca233ac45ed7c9227d84cf5a2329310525917adaf3312e10d5462",
- "templateCredsSetupCompleted": true
- },
- "name": "Very simple Human in the loop system email with AI e IMAP",
- "tags": [],
- "nodes": [
- {
- "id": "271bb16f-9b62-41d9-ab76-114cd7ba915a",
- "name": "Email Trigger (IMAP)",
- "type": "n8n-nodes-base.emailReadImap",
- "position": [
- -1300,
- 1340
- ],
- "parameters": {
- "options": {}
- },
- "credentials": {
- "imap": {
- "id": "k31W9oGddl9pMDy4",
- "name": "IMAP info@n3witalia.com"
- }
- },
- "typeVersion": 2
- },
- {
- "id": "42d150d8-d574-49f9-9c0e-71a2cdea3b79",
- "name": "Markdown",
- "type": "n8n-nodes-base.markdown",
- "position": [
- -1040,
- 1340
- ],
- "parameters": {
- "html": "={{ $json.textHtml }}",
- "options": {}
- },
- "typeVersion": 1
- },
- {
- "id": "e9498a60-0078-4581-b269-7ff552f4047a",
- "name": "Send Email",
- "type": "n8n-nodes-base.emailSend",
- "position": [
- 920,
- 1320
- ],
- "webhookId": "a79ae1b4-648c-4cb4-b6cd-04ea3c1d9314",
- "parameters": {
- "html": "={{ $('Set Email text').item.json.email }}",
- "options": {},
- "subject": "=Re: {{ $('Email Trigger (IMAP)').item.json.subject }}",
- "toEmail": "={{ $('Email Trigger (IMAP)').item.json.from }}",
- "fromEmail": "={{ $('Email Trigger (IMAP)').item.json.to }}"
- },
- "credentials": {
- "smtp": {
- "id": "hRjP3XbDiIQqvi7x",
- "name": "SMTP info@n3witalia.com"
- }
- },
- "typeVersion": 2.1
- },
- {
- "id": "ab9f6ac3-2095-44df-aeba-2eab96ecf425",
- "name": "Email Summarization Chain",
- "type": "@n8n/n8n-nodes-langchain.chainSummarization",
- "position": [
- -780,
- 1340
- ],
- "parameters": {
- "options": {
- "binaryDataKey": "={{ $json.data }}",
- "summarizationMethodAndPrompts": {
- "values": {
- "prompt": "=Write a concise summary of the following in max 100 words:\n\n\"{{ $json.data }}\"\n\nDo not enter the total number of words used.",
- "combineMapPrompt": "=Write a concise summary of the following in max 100 words:\n\n\"{{ $json.data }}\"\n\nDo not enter the total number of words used."
- }
- }
- },
- "operationMode": "nodeInputBinary"
- },
- "typeVersion": 2
- },
- {
- "id": "86b7c3d0-e1f2-4e2f-b293-8042700d6816",
- "name": "Write email",
- "type": "@n8n/n8n-nodes-langchain.agent",
- "position": [
- -340,
- 1340
- ],
- "parameters": {
- "text": "=Write the text to reply to the following email:\n\n{{ $json.response.text }}",
- "options": {
- "systemMessage": "You are an expert at answering emails. You need to answer them professionally based on the information you have. This is a business email. Be concise and never exceed 100 words. Only the body of the email, not create the subject"
- },
- "promptType": "define",
- "hasOutputParser": true
- },
- "typeVersion": 1.7
- },
- {
- "id": "5d5a397f-f9c3-4691-afd0-9a6102679eac",
- "name": "OpenAI",
- "type": "@n8n/n8n-nodes-langchain.lmChatOpenAi",
- "position": [
- -400,
- 1560
- ],
- "parameters": {
- "model": {
- "__rl": true,
- "mode": "list",
- "value": "gpt-4o-mini",
- "cachedResultName": "gpt-4o-mini"
- },
- "options": {}
- },
- "credentials": {
- "openAiApi": {
- "id": "CDX6QM4gLYanh0P4",
- "name": "OpenAi account"
- }
- },
- "typeVersion": 1.2
- },
- {
- "id": "5b36a295-fda6-4174-9078-0a8ec57620d2",
- "name": "Sticky Note2",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -800,
- 1260
- ],
- "parameters": {
- "width": 320,
- "height": 240,
- "content": "Chain that summarizes the received email"
- },
- "typeVersion": 1
- },
- {
- "id": "7110fe1f-0099-49aa-9095-96e733aa468f",
- "name": "Sticky Note5",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -360,
- 1260
- ],
- "parameters": {
- "width": 340,
- "height": 240,
- "content": "Agent that retrieves business information from a vector database and processes the response"
- },
- "typeVersion": 1
- },
- {
- "id": "e2bdbd64-3c37-4867-ae2c-0f6937d82b81",
- "name": "Sticky Note1",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- -1100,
- 1260
- ],
- "parameters": {
- "height": 240,
- "content": "Convert email to Markdown format for better understanding of LLM models"
- },
- "typeVersion": 1
- },
- {
- "id": "8ae5d216-5897-4c33-800a-27ff939b174a",
- "name": "Sticky Note7",
- "type": "n8n-nodes-base.stickyNote",
- "position": [
- 620,
- 1300
- ],
- "parameters": {
- "height": 180,
- "content": "If the feedback is OK send email"
- },
- "typeVersion": 1
- },
- {
- "id": "4cfce63c-5931-45c5-99ca-eb85dca962b5",
- "name": "Approve Email",
- "type": "n8n-nodes-base.emailSend",
- "position": [
- 380,
- 1340
- ],
- "webhookId": "4f9f06e7-9b2b-4896-9b51-245972341d12",
- "parameters": {
- "message": "=