oryan (@oryanmoshe) header
oryan (@oryanmoshe) profile picture

Software Engineer & AI nerd

Tel Aviv, Israel
Joined March 2020
Photo by oryan (@oryanmoshe)Photo by oryan (@oryanmoshe)Photo by oryan (@oryanmoshe)
📢 Excited to announce that starting today I’m #OpenForWork and providing services in anything front-end related, from hands-on development to consulting, DevRel, and more. Please reach out for details, DMs are open. * Barbenheimer is so last week, here's the new blockbuster:
6
18
1
105
35,291
Now this is the right way to give criticism! Now let's close some dataset holes and crush them.
Are open-sourced LLMs really good? 👀 We introduce FLASK🧪, a fine-grained evaluation based on skill sets! Even SOTA open-sourced LLMs such as LLaMA2 Chat 70B lag behind proprietary LLMs for some abilities. 🤯 Paper: arxiv.org/abs/2307.10928 Demo: kaistai.github.io/FLASK
3
9
67
8,187
Adam8bit + model.half() + gradient_checkpointing_enable() + batch_size = 1 + flash attention + zero_optimization: stage: 3 + offload_optimizer + offload_param + eps = 1e-4 + weight_decay = 0 + dropout = 0 Going to make some coffee No way the loss is not 𝚗𝚊𝚗 when I come back.
8
10
1
155
28,213
The struggle is real
This quoted post is unavailable.
42
Introduce the newest WizardLM models trained with only 1k high-quality evolved data! WizardLM-13B-V1.1 achieves: (1). 6.74 on MT-Bench (2). 86.32% on Alpaca Eval (ChatGPT is 86.09%) (3). 99.3% on WizardLM Eval Github: github.com/nlpxucan/WizardLM Weights: huggingface.co/WizardLM
25
91
11
386
108,671
This is absolutely insane, go try it out!
You can get ChatGPT Code Interpreter for right now. 1. Log off -> Log in to ChatGPT. 2. Go to settings -> Beta features 3. Enable Code interpreter. Breaking!
1
405
בשקט בשקט ובאופן עצמאי @Yampeleg בונה לנו מודל שפה גדול בעברית ראשון מסוגו. הוא משחרר בקהילה טיזרים קטנים ומראה לנו על מה הוא עובד וזה פשוט מדהים. אני לא זוכר מתי פעם האחרונה שבאמת חיכיתי שמשהו כזה יצא לאוויר. זה באמת יכול לשנות את חוקי המשחק עבור כל התחום בישראל.
17
17
2
442
69,988
One. Billion. Tokens. Context Window. 🤯 I didn't want to get sucked into twitter but I couldn't help it. Here is how they do it: Dilated Attention They use an approach similar to dilated convolutions: "convolutions with holes" that are expanding as the depth of the network increases. Allowing the layer to "see" parts of the input that are farther apart from each other. Illustration: The filter is the red part (moving through the input seq) Standard Convolution: 🟩🟩🟥🟥🟥🟩🟩🟩 Dilated Convolution: 🟩🟥🟩🟥🟩🟥🟩🟩 Note: It is the same filter size but the dilated can "see far". In order not to lose information: We usually increase the dilation rate as the depth of the layer increases (Wavenet, Google 2016). So they apply the same idea but to the 2D attention maps. 1. Input Segmentation: Dilated attention takes and splits it into equally sized segments. 2. Sparsification: Each of the segments is then sparsified based on a given interval 'r'. (Meaning: Every 'r'th row is selected, others are discarded) 3. Outputs: The outputs for the sparsified segments are then gathered and concatenated to form the final output of the attention layer. 4. Mixing with dense attentions: To capture both short-range and long-range, the final network use a mix of dilated & dense attention layers with different segment sizes and dilation rates. Note about the code: I didn't go into it too much but the codebase has references to many architecture tricks (with papers). Looks like a cool reading list, I didn't know about some of the papers there.
LongNet: Scaling Transformers to 1,000,000,000 Tokens Presents LONGNET, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences abs: arxiv.org/abs/2307.02486 repo: github.com/microsoft/torchsc…
9
60
8
359
145,690
BatGPT from Wuhan university. (Yes.) - Paper: arxiv.org/pdf/2307.00360.pdf Jokes aside, pay attention to the following tricks: - Trick 1: Reversing the sequence during training. [learn to predict the next token & learn to predict the previous token] - Trick 2: Reinforcement learning from inhuman. Using feedback from models to train and RL model. - Trick 3: Formalizing the RLHF data collection to simple pairs comparison [see picture] Results: They do not measure themselves on known benchmarks but rather on Chinese benchmarks. They are SOTA but this might be hard to estimate what it means. Anyway, the tricks are important to know about. - For more on trick 2: See my post "the missing pieces of GPT-4. - Apologist about the previous tweet about this paper - I was misled.
5
27
12
132
61,232
🚀 "The Secret Behind My Latest Blog Post: ChatGPT Did the Heavy Lifting" 🚀 Learn how I collaborated with ChatGPT to create high-quality content and the tips & tricks that made our teamwork a success. Check it out! ➡️ dev.to/oryanmoshe/the-secret… #ChatGPT
38
🚀 Just published a new blog post: "Mock Data & Stubs: Fake It Till You Make It"! 🚀 Learn how to reduce iteration times by using mock data, stubs, and fakers. Check it out and let me know your thoughts! 👉 dev.to/oryanmoshe/mock-data-… #programming #development #productivity
47
Wordle 213 4/6 ⬛🟩⬛⬛⬛ ⬛🟩⬛⬛⬛ ⬛🟩🟩⬛⬛ 🟩🟩🟩🟩🟩 מדויקת 18.1.2022 - 6/6 ⬜🟨⬜⬜⬜ ⬜⬜🟨🟨🟨 ⬜🟨⬜🟨🟨 ⬜🟩⬜🟩🟨 🟨🟩⬜🟩🟩 🟩🟩🟩🟩🟩
1
Wordle 211 4/6 ⬛🟨🟨⬛⬛ 🟨🟨⬛⬛🟨 🟩⬛🟨🟨⬛ 🟩🟩🟩🟩🟩 מדויקת 16.1.2022 - 5/6 🟨⬜⬜⬜⬜ ⬜⬜🟨⬜⬜ ⬜🟨⬜⬜🟩 🟩⬜⬜🟨🟩 🟩🟩🟩🟩🟩
1
Wordle 198 2/6 🟩🟩⬛⬛⬛ 🟩🟩🟩🟩🟩
Wordle 195 6/6 ⬛⬛⬛🟨🟨 ⬛⬛🟩🟨⬛ 🟨⬛⬛⬛⬛ ⬛🟩🟩⬛⬛ 🟩🟩🟩⬛🟩 🟩🟩🟩🟩🟩
Wordle 194 3/6 🟩🟩⬛⬛⬛ 🟩🟩⬛⬛⬛ 🟩🟩🟩🟩🟩
Looking to follow some white men in tech with 50k followers or more. Any ideas?
137
59
12
1,387