Leverage the power of Delta Lake with Sling. Perfect for analytics on large-scale data stored in Delta format. Read our latest blog post to learn more about streamlining your data workflows with Sling and Delta Lake. #DataEngineering #DeltaLake #ETL
blog.slingdata.io/efficient-…
Sling now supports reading the Apache Iceberg format. Try it out!
blog.slingdata.io/reading-ap…
With Sling, you can see column level metadata of your tables from the command line. Here we are using the --pattern (or -p) and --columns flags.
SlingData.io retweeted
DuckDB is introducing support for vector similarity search through the new VSS extension.
Read @Maxxen_'s blog post for a sneak preview on the new extension's capabilities: duckdb.org/2024/05/03/vector…
Use Sling to quickly discover your tables from the command line. There is even a --pattern (or -p) flag to filter the output.
SlingData.io retweeted
What content do you believe to be missing from existing data engineering courses?
#dataengineering
reddit.com/r/dataengineering…
Got Prometheus running? Want to export the data elsewhere? Try sling.
blog.slingdata.io/export-dat…
SlingData.io retweeted
Have you heard of the 12-Factor App?
The "12 Factor App" offers a set of best practices for building modern software applications. Following these 12 principles can help developers and teams in building reliable, scalable, and manageable applications.
Here's a brief overview of each principle:
1. Codebase:
Have one place to keep all your code, and manage it using version control like Git.
2. Dependencies:
List all the things your app needs to work properly, and make sure they're easy to install.
3. Config:
Keep important settings like database credentials separate from your code, so you can change them without rewriting code.
4. Backing Services:
Use other services (like databases or payment processors) as separate components that your app connects to.
5. Build, Release, Run:
Make a clear distinction between preparing your app, releasing it, and running it in production.
6. Processes:
Design your app so that each part doesn't rely on a specific computer or memory. It's like making LEGO blocks that fit together.
7. Port Binding:
Let your app be accessible through a network port, and make sure it doesn't store critical information on a single computer.
8. Concurrency:
Make your app able to handle more work by adding more copies of the same thing, like hiring more workers for a busy restaurant.
9. Disposability:
Your app should start quickly and shut down gracefully, like turning off a light switch instead of yanking out the power cord.
10. Dev/Prod Parity:
Ensure that what you use for developing your app is very similar to what you use in production, to avoid surprises.
11. Logs:
Keep a record of what happens in your app so you can understand and fix issues, like a diary for your software.
12. Admin Processes:
Run special tasks separately from your app, like doing maintenance work in a workshop instead of on the factory floor.
Over to you: Where do you think these principles can have the most impact in improving software development practices?
--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): bit.ly/3KCnWXq
SlingData.io retweeted
Data engineers often become bored of data engineering!
After a while of SQL + Python + airflow, you start thinking all pipelines are the same and it’s copy and paste work.
Some strategies to help with this:
- become more end-to-end
Maybe that means building a dashboard. Maybe that means fixing logging. Maybe that means updating some ML model.
- transition to a new role
Personally, I can only do data engineering for 2 years at a time before I want some different. I find transitioning to backend the most natural but transitioning to ML engineer the most seductive.
- try other things to build up other skills.
Making content really helped me deal with the boredom. So did building side projects in NextJS. Maybe painting is your thing and you’re wasting your time in computers?
#dataengineering
Data News — Mistral AI, Klarna AI customer support agent, extract and load still unsolved blef.fr/data-news-week-24-09…












