Cutting edge tools for AI creators | Join our waitlist @ https://xitter.cf/t.co/koVD9SXRaz Discord: https://xitter.cf/t.co/nVT5y6g3Cc
Joined December 2022
- Tweets567
- Following353
- Followers1.4K
- Likes1.5K
MagickML retweeted
We have sub-spells. Nested graphs are coming to Magick.
The next step will be sub-spells as tools. The socket inputs and outputs on the sub-spell are given descriptions, and we can transform them into tool schemas. Then we parse and call the sub-spell, returning the result.
This is going to open up all kinds of possibilities.
MagickML retweeted
OpenAI is expected to demo a real-time voice assistant tomorrow. What does it take to deliver an immersive, or even magical experience?
Almost all voice AI go through 3 stages:
1. Speech recognition or "ASR": audio -> text1, think Whisper;
2. LLM that plans what to say next: text1 -> text2;
3. Speech synthesis or "TTS": text2 -> audio, think ElevenLabs or VALL-E.
Last year, I made the figure below to show how to make Siri/Alexa 10x better. However, naively going through 3 stages results in huge latency. User experience falls off the cliff if we have to wait 5 seconds for *each* reply. It breaks the immersion and feels lifeless even if the synthesized audio itself sounds real.
Natural dialogues fundamentally don't work like this. We humans
> think about what to say next at the same time as we listen & speak;
> inject "yes, hmm, huh" at appropriate moments;
> predict when the other person finishes and immediately take over;
> decide to talk over the other person organically, without being offensive;
> handle interruptions gracefully. Currently, AI assistants either cannot be interrupted (super frustrating) or simply stop when they detect an audio event and lose train of thought;
> engage in group chat. We are so good at multi-agent conversations.
It's not as simple as making each of the 3 neural nets faster, sequentially. Solving real-time dialogue requires us to rethink the whole stack, overlap each component as much as possible, and learn how to make interventions in real time.
Or perhaps even better - just have 1 NN mapping audio to audio. End-to-end always wins.
I'll sketch out how to design such a model and its training pipeline. Meanwhile, let's wait and see how far OpenAI pushes it!
MagickML retweeted
Check out our brand-new course on Building Agentic RAG with LlamaIndex with @DeepLearningAI + @AndrewYNg 🔥
Learn how to build an autonomous research assistant that can answer complex questions over multiple documents - handle a much broader range of inputs than a standard RAG pipeline! 📖🕵️
The course helps you learn how to build an agent ingredient by ingredient from a standard RAG pipeline. Once you're done with the course check out the following @llama_index resources to take your agents to the next level 👇
Building a custom agent: docs.llamaindex.ai/en/stable…
Multi-document agents (diagram by @clusteredbytes): docs.llamaindex.ai/en/stable…
Course: deeplearning.ai/short-course…
I’m excited to kick off the first of our short courses focused on agents, starting with Building Agentic RAG with LlamaIndex, taught by @jerryjliu0, CEO of @llama_index.
This covers an important shift in RAG (retrieval augmented generation), in which rather than having the developer write explicit routines to retrieve information to feed into the LLM context, we instead build a RAG agent that that has access to tools for retrieving information. This lets the agent decide what information to fetch, and enables it to answer more complex questions using multi-step reasoning.
In detail, you'll learn about:
- Routing: Where your agent will use decision-making to route requests to multiple tools.
- Tool Use: Where you'll create an interface for agents to select what tool (function call) to use as well as generate the right arguments.
- Multi-step reasoning with tool use: Where you'll use an LLM to carry out multiple steps of reasoning, while retaining memory throughout the process.
You’ll also learn how to step through what your agent is doing to debug and improve it iteratively.
It’s an exciting time to build agents. Sign up and get started here! deeplearning.ai/short-course…
MagickML retweeted
Another small UX improvement. Hiding unused sockets. It isn't a full node collapse, which I want to do next, but that will have to wait. In demo crunch time.
MagickML retweeted
I love adding these little UX improvements that make my experience building way better. Renaming a nodes title goes a long way to legibility. Even more so possibly than comments.
MagickML retweeted
I have just decided that I love multi-gates. I find this incredibly satisfying. Using this in the demo I am working on, and will have to document this.
MagickML retweeted
The most epic AI panel in a while! We at NVIDIA have gathered ALL 8 authors of "Attention is All You Need" for a panel at GTC, hosted by none other than the GOAT himself, Jensen Huang.
In 2017, 8 researchers had a flash of genius and invented Transformer, the seminal work that transformed AI once and for all.
How did they come up with the idea?
What challenges did they face?
What surprised them in the following years?
What have they been up to these days?
How do they envision a post-transformer future?
Thanks @maggie_albrecht for making this happen! I'm so glad to play a part in reaching out to some of the coauthors.
Join us for a behind-the-scenes look at modern AI's greatest moment. Mar. 20, 11 am PDT, at San Jose McEnery Convention Center, NVIDIA GTC: nvidia.com/gtc/session-catal…
MagickML retweeted
Brad Templeton asks if it is a copyright violation to recite a poem or song because you have a copy of it in your brain
MagickML retweeted
Build RAG over the Web 🌐
AgentSearch is an awesome project by @ocolegro - the AgentSearch-v1 includes over 1 billion embeddings from 50million > documents from the web.
It’s an awesome data source if you want to build search/retrieval systems over internet content without having to use Bing/Google APIs.
Excited to present that it’s now as a data loader and as a Retriever in LlamaHub!
Huge props to @ocolegro for putting this together.
Loader: llamahub.ai/l/agent_search?f…
Retriever LlamaPack: llamahub.ai/l/llama_packs-ag…
MagickML retweeted
Agentic Chunking?
We should chunk our text off its meaning...not physical characters
Use an agent-like app to look a text, have it determine if it should be joined with a chunk, or create a new one
Yes it's slow and expensive, but most solid and tunable results I've seen
MagickML retweeted
Introducing r1. Watch the keynote.
Order now: rabbit.tech
#CES2024
MagickML retweeted
Build Custom Agent Loops 💫🛠️
You may want to build agentic reasoning beyond what’s available in prepackaged agent frameworks (ReAct, ToT, LLMCompiler).
This lets you tackle complex questions over your data that’s best suited for your use case.
We’re excited to launch a new abstraction + guide that lets you build your own agent reasoning loop from scratch! 🧑🍳
✅ The abstraction is super simple, lets you define any step-wise reasoning behavior
✅ Can plug in directly on top of any RAG/SQL/other tools over your data
Your agent can be as simple/sophisticated as you want - our walkthrough builds a simple agent that adds retry logic on top of a router, so it can keep retrying the question if the initial query is incomplete.
We build this over a SQL tool + RAG tool and show that you can now answer more complex, multi-part questions beyond simply using a router. It can correct SQL queries, combine information from both data sources, and more.
Check it out: github.com/run-llama/llama_i…
Docs: docs.llamaindex.ai/en/latest…
MagickML retweeted
Meta just announced Fairy
Fast Parallelized Instruction-Guided Video-to-Video Synthesis
paper page: huggingface.co/papers/2312.1…
introduce Fairy, a minimalist yet robust adaptation of image-editing diffusion models, enhancing them for video editing applications. Our approach centers on the concept of anchor-based cross-frame attention, a mechanism that implicitly propagates diffusion features across frames, ensuring superior temporal coherence and high-fidelity synthesis. Fairy not only addresses limitations of previous models, including memory and processing speed. It also improves temporal consistency through a unique data augmentation strategy. This strategy renders the model equivariant to affine transformations in both source and target images. Remarkably efficient, Fairy generates 120-frame 512x384 videos (4-second duration at 30 FPS) in just 14 seconds, outpacing prior works by at least 44x. A comprehensive user study, involving 1000 generated samples, confirms that our approach delivers superior quality, decisively outperforming established methods.
MagickML retweeted
Introducing Comfy Deploy. A `vercel` for the generative workflow era.
From your Comfy UI workflow to production-ready API, flexibility at the speed of thought. Open source on day 1.
Built with @shadcn UI @DrizzleORM @neondatabase @nextjs
oss/acc
See thread for more
MagickML retweeted
High quality AI generated human videos are coming!
Animate Anyone can generate videos of anyone with a single image and a bit of pose guidance 🤯
humanaigc.github.io/animate-…
