LOVE UNIX. fascinated by & interested in distributed systems & machine learning. programming lang design enthusiast. enjoy Haskell. cyber security dilettante.
San Francisco, CA
Joined March 2011
- Tweets6K
- Following269
- Followers253
- Likes436
This is a great list of amazing books on Programming and Software Development compiled by @liza -> safaribooksonline.com/list/m… [fantastic]
Trying to achieve 2 things with that ^^ if possible:
- Won't emit UPDATED result for a window due to LATE irrelevant events (as opposed to a LATE DESIRED event)
- Won't keep windows open simultaneously ANY longer than necessary
.@snntrable @rmetzger_ If you emit result of a window with key "foo" & PURGE it (since a desired event is received for the window & subsequent events for the window are IRRELEVANT); will @ApacheFlink open a NEW window again for key "foo" when later events with that key arrive?
A good description of Chandy-Lamport algorithm for taking distributed snapshots and capturing the global state of a distributed system in a consistent fashion by @adriancolyer! (@ApacheFlink uses a variation of this algorithm for its checkpoints)
blog.acolyer.org/2015/04/22/…
Even though it's an old article and things have quite evolved across most/all of those systems (especially in the case of Spark Structured Streaming); it lays out the ultimate desires of streaming frameworks & challenges involved quite nicely!
ververica.com/blog/high-thro…
why open-source Firmware matters & more in "Steps Into The World Behind Kernel" by Jessie Frazelle
queue.acm.org/detail.cfm?id=…
~~ To enable focusing on business logic w/o having to worry about complexity involved in a distributed systems, it is our goal to provide a rich set of composable operators that can be easily plugged into a streaming job DAG. ~~ @NetflixEng
medium.com/netflix-techblog/…
--
we take a layered approach to ensure two things:
1. Remotely attacking our vehicle over cellular is absurdly difficult
2. In the event of a compromise, affecting the physical aspects of our AV are equally as challenging
--
medium.com/cruise/how-to-pri…
A look into the science behind @ApacheFlink by @StefanRRichter which lays out certain choices of data structures and strategies when it comes to checkpointing barriers, rescaling stateful operators, credit based flow control and other interesting aspects!
youtube.com/watch?v=pPqNNkLF…
-- Theory and practice sometimes clash. And when that happens, theory loses. Every single time. --
lwn.net/Articles/326505/
-- So rather than come up with new barriers that nobody will use, filesystem people should aim to make "badly written" code "just work" unless people are really really unlucky. Because like it or not, that's what 99% of all code is. -- ~ Linus Torvalds
A thorough look into @ApacheFlink's support for EXACTY-ONCE semantics in and end-to-end fashion (e.g. SINK into an external system such as @apachekafka), by leveraging the Two-Phase Commit protocol!
flink.apache.org/features/20…
A peek into @ApacheFlink's engine room by @fhueske which lays out different shuffling and local strategies when it comes to JOIN operations.
flink.apache.org/news/2015/0…
@evanpriestley any way to detect "file removals" in
a DIFF via Phabricator's Herald Rules?! I know that capability exists for the removed "content" of files.
A super cool article on Fourier Transforms by @jezzamonn
which you don't wanna miss :) jezzamon.com/fourier/
For some reason the default GitHub IO went through "HTTP" as opposed to "HTTPS". Here is the "HTTPS" link to that post. sam-serpoosh.github.io/flink…
Big thanks to @snntrable for reviewing the first draft of this post and providing valuable feedbacks! cc: @VervericaData
A detailed & concrete dive into @ApacheFlink's window and its boundaries. sam-serpoosh.github.io/flink…