NeroBot AI

Working with LLMs to get structured outputs can be extremely annoying, they often fail on the simplest of tasks! 🔥Structured Output, Constrained Generation, JSON Mode are the holy grail for LLMs 🔥 Why?? For problem solving, instead of needing to write new code at runtime, they enable the delegation of cognitive tasks across specialized pre-defined agents, chains, functions and tools. This connectivity results in a symphony of logic, enabling intricate problem-solving, at a low cost.

Going to regularly tweet about our technical stack and design decisions from a non CTO perspective 😂. #buildinpublic We wanted to achieve these goals: 1. Get to market as quickly as possible 2. Focus resources on building LLM applications not commoditized app components 3. Maximize scalability and minimize vender lock in For @nerobotai, as an api first product, today I will cover our API management layer using @Zuplo.

Quick demo of @nerobotai AI Crawl Assistant in action. #buildinpublic It's able to navigate sitemaps for desired pages all from natural language inputs. It does so by leveraging a set of tools and/or code interpreter to handle as many types of website structures as possible. Currently we are using @OpenAI Assistant API and Mistral models. The beautiful frontend was made possible by the boilerplate framework @makerkit_dev from @gc_psk And shoutout to @KillerTopic who did a lot of the heavy lifting on the backend.

New blog post on why I’m building yet another crawler. 1/ After months of digging into various crawlers, it hit me – there's no easy, automated way to scrape knowledge bases! We tried them all, Diffbot, Apify, you name it. The struggle was real. 😧 2/ These tools would throw everything at us, not just the page content. Navigating through sidebars and other extras? Not fun. For us dealing with LLMs, losing the text structure, especially HTML tables, is a nightmare. 😓 3/ And don’t get me started on the crawl itself. Every client website is a new beast – dealing with JS rendering, bot blocks, and even language issues. And if you don’t have the sitemap, good luck! But wait, there’s hope...🤞 4/ Introducing NeroBot! We’re talking about a tool that thinks like a crawling pro, examines site structures, understands client goals, and uses LLMs and AI to make everything automated and smooth. No more bot blocks or messy extractions! 🎉 5/ Our AI is on point – it gives you only the content you need. And we’re not stopping there. Soon, it’ll tell you the category of the article, as the publisher defines it. Ask it anything about your entities, and get answers in real time! 🧠💬 6/ To kick things off, NeroBot is diving into docs, knowledge bases, support centers, & news. Your thoughts during our closed beta will be the magic touch helping us tweak and grow. And yes, more features are coming! 📈 7/ This isn’t just another crawler, folks. We’re stepping up the game, making things efficient, user-friendly, and ready to meet all sorts of needs. Come join the party, drop your two cents, and let’s make web crawling epic together! 🎊 8/ Let me know what you think. Your thoughts are the key to making NeroBot the best it can be. Let’s turn those web crawling headaches into wins, and make some internet magic! #AI #Langchain #LlamaIndex