A few strategies I implemented in my Data Engineering production systems to make it more "cost effective💰" and "less cool" 1. Maximize the use of caching wherever feasible. adding a cache layer between the client and data warehouse, resulting in over 60% cache hits weekly 2. Data warehouse(s) for poor. Established multiple cost-effective data warehouses using @duckdb endpoints for varied data sets, with a custom client handling request redirection. This approach led to significant cost savings. Users only hit the big DWH when duckdb endpoints have no references. 3. Avoided using big data technologies for non-big data tasks, reducing reliance on Spark and transitioning many processes to @DataPolars. 4. Reevaluated the need for orchestrators in data engineering jobs, discovering that 45% of them didn't require such complexity. Shifted to simpler cron jobs or Kubernetes executors for these tasks. 5. Don't code if LLM can generate it. Leveraged LLMs for generating various infrastructure templates, reducing coding requirements. Developed a finely-tuned(local) internal model to address frequently required tasks efficiently.