Here is a huge trick to cut down on hallucinations: I have an agent that processes phone receipts using GPT-4o. Whenever I go to a restaurant or pay for something, I take a photo of the receipt and upload it to Google Drive. The agent goes through the folder and extracts the following information from each receipt: • Name of the place • Category of the place • Amount • Tax amount • Tip amount Each field requires an individual prompt. The result is a JSON object that I export to a separate folder. This is awesome, but there's a problem: The agent occasionally makes mistakes when interpreting a receipt; unfortunately, I couldn't detect them as they happened. Here is the trick: I'm now using a challenger model (o3-mini) to check the work of GPT-4o. This increases the cost of processing receipts but reduces the mistakes to almost nothing. The best part: 1. Whenever the challenger disagrees with the main model, I produce detailed information about the problem to improve the prompts and make the extraction more robust. 2. It's very easy to turn this process into a Human-in-the-Loop workflow where I get notifications of every disagreement and an opportunity to settle them. I recorded the attached video to show you how this works using @GetUnstract.

I'm using @GetUnstract to build unstructured ETL pipelines. They partnered with me on this post. Unstract is open-source, and it takes 3 minutes to run it locally. It's pretty popular: they process over 7 million pages monthly and turn them into actionable structured data. I like to think about it as an ETL platform where the T uses LLMs. Here is their GitHub repository: github.com/Zipstack/unstract

GPT-4o is bad at processing PDF documents. Whoever tells you otherwise is not living in the real world. In 2024, people fill out forms using pen and paper. Try to answer questions from those forms using modern models, and you'll be disappointed. I recorded a video to show you how to fix this. The answer is simple: stop letting the model see your PDF document. Instead, preprocess it and stick to showing the model text. Watch the video. You'll go from "THIS IS CRAP" to "EXCELLENT" in no time. I'm using @GetUnstract to turn the documents into text while keeping the original format (this is crucial!) You can use them to process up to 100 pages for free. They collaborated with me on this post. You can find the code I wrote right here: github.com/svpino/unstract-l…