oryan retweeted
📢 Excited to announce that starting today I’m #OpenForWork and providing services in anything front-end related, from hands-on development to consulting, DevRel, and more.
Please reach out for details, DMs are open.
* Barbenheimer is so last week, here's the new blockbuster:
oryan retweeted
Now this is the right way to give criticism!
Now let's close some dataset holes and crush them.
Are open-sourced LLMs really good? 👀
We introduce FLASK🧪, a fine-grained evaluation based on skill sets! Even SOTA open-sourced LLMs such as LLaMA2 Chat 70B lag behind proprietary LLMs for some abilities. 🤯
Paper: arxiv.org/abs/2307.10928
Demo: kaistai.github.io/FLASK
oryan retweeted
Adam8bit
+ model.half()
+ gradient_checkpointing_enable()
+ batch_size = 1
+ flash attention
+ zero_optimization: stage: 3
+ offload_optimizer
+ offload_param
+ eps = 1e-4
+ weight_decay = 0
+ dropout = 0
Going to make some coffee
No way the loss is not 𝚗𝚊𝚗 when I come back.
oryan retweeted
Introduce the newest WizardLM models trained with only 1k high-quality evolved data!
WizardLM-13B-V1.1 achieves:
(1). 6.74 on MT-Bench
(2). 86.32% on Alpaca Eval (ChatGPT is 86.09%)
(3). 99.3% on WizardLM Eval
Github: github.com/nlpxucan/WizardLM
Weights: huggingface.co/WizardLM
oryan retweeted
בשקט בשקט ובאופן עצמאי @Yampeleg בונה לנו מודל שפה גדול בעברית ראשון מסוגו. הוא משחרר בקהילה טיזרים קטנים ומראה לנו על מה הוא עובד וזה פשוט מדהים.
אני לא זוכר מתי פעם האחרונה שבאמת חיכיתי שמשהו כזה יצא לאוויר. זה באמת יכול לשנות את חוקי המשחק עבור כל התחום בישראל.
oryan retweeted
One. Billion. Tokens. Context Window. 🤯
I didn't want to get sucked into twitter but I couldn't help it.
Here is how they do it:
Dilated Attention
They use an approach similar to dilated convolutions: "convolutions with holes" that are expanding as the depth of the network increases.
Allowing the layer to "see" parts of the input that are farther apart from each other.
Illustration:
The filter is the red part (moving through the input seq)
Standard Convolution:
🟩🟩🟥🟥🟥🟩🟩🟩
Dilated Convolution:
🟩🟥🟩🟥🟩🟥🟩🟩
Note: It is the same filter size but the dilated can "see far".
In order not to lose information: We usually increase the dilation rate as the depth of the layer increases (Wavenet, Google 2016).
So they apply the same idea but to the 2D attention maps.
1. Input Segmentation: Dilated attention takes and splits it into equally sized segments.
2. Sparsification: Each of the segments is then sparsified based on a given interval 'r'.
(Meaning: Every 'r'th row is selected, others are discarded)
3. Outputs: The outputs for the sparsified segments are then gathered and concatenated to form the final output of the attention layer.
4. Mixing with dense attentions: To capture both short-range and long-range, the final network use a mix of dilated & dense attention layers with different segment sizes and dilation rates.
Note about the code:
I didn't go into it too much but the codebase has references to many architecture tricks (with papers).
Looks like a cool reading list, I didn't know about some of the papers there.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Presents LONGNET, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences
abs: arxiv.org/abs/2307.02486
repo: github.com/microsoft/torchsc…
oryan retweeted
BatGPT from Wuhan university. (Yes.)
- Paper: arxiv.org/pdf/2307.00360.pdf
Jokes aside, pay attention to the following tricks:
- Trick 1: Reversing the sequence during training. [learn to predict the next token & learn to predict the previous token]
- Trick 2: Reinforcement learning from inhuman. Using feedback from models to train and RL model.
- Trick 3: Formalizing the RLHF data collection to simple pairs comparison [see picture]
Results: They do not measure themselves on known benchmarks but rather on Chinese benchmarks.
They are SOTA but this might be hard to estimate what it means.
Anyway, the tricks are important to know about.
- For more on trick 2: See my post "the missing pieces of GPT-4.
- Apologist about the previous tweet about this paper - I was misled.
🚀 "The Secret Behind My Latest Blog Post: ChatGPT Did the Heavy Lifting" 🚀
Learn how I collaborated with ChatGPT to create high-quality content and the tips & tricks that made our teamwork a success. Check it out! ➡️ dev.to/oryanmoshe/the-secret… #ChatGPT
38
🚀 Just published a new blog post: "Mock Data & Stubs: Fake It Till You Make It"! 🚀
Learn how to reduce iteration times by using mock data, stubs, and fakers.
Check it out and let me know your thoughts! 👉 dev.to/oryanmoshe/mock-data-…
#programming #development #productivity
47
Go check out Panda's Nickvitational's fundraiser for BREAST CANCER RESEARCH FOUNDATION on @Tiltify | tiltify.com/+pandas-nickvita…
I just published my new blog post (first one in a while!)
It's about @debezium new SPI, and how we're using Custom Converters to drive our CDC capabilities here at @riveryio!
in.dev/blog/debezium-custom-…
@ThePracticalDev @indevofficial
#indev #java #cdc #debezium #kafka #blog


















