@BrianHulette

I like to make things. Software engineer @Google, @ApacheArrow JS committer

 Seattle, WA
theneuralbit.com
Joined August 2012
  • Tweets193
  • Following218
  • Followers123
27Media
Replying to @BrianHulette @flightaware @algoriffic
Also made this gif showing one day of MD-11s and DC-10s, used by FedEx (and some UPS). FedEx flights are in/out of Memphis. UPS - Louisville
4
9
This morning a 🐝 was happily buzzing along, searching for flowers to pollinate, when suddenly it found itself IN MY NOSTRIL. Neither one of us is happy with the outcome.
2
Brian Hulette retweeted
lak lakshmanan
@lak_luster19 Jul 2022
Used to be that you had to read in Beam, add a schema, and then convert to a Dataframe. But now, super-familiar pandas-like read_csv etc just works
1
2
7
Replying to @ApacheBeam
@ApacheBeam 2.29.0 was released last week! It has several bugfixes and improvements for the Beam DataFrame API as we march towards exiting experimental. beam.apache.org/blog/beam-2.…
1
2
Most notably, we now have support for pandas' groupby.apply() method. It's an incredibly flexible operation for performing aggregations, but just like in standard pandas there's a performance penalty. In Beam's case it's because we can't do combiner lifting.
2
Brian Hulette retweeted
Dominik Moritz@domoritz27 Apr 2021
Today, @ApacheArrow JS 4.0 dropped. I implemented an improvement that speeds up iterating over Tables and Vectors by ~2X. observablehq.com/@domoritz/a…

Arrow JS 3 vs 4 Iterator Benchmark

An Observable notebook by Dominik Moritz

observablehq.com
1
2
21
Really excited to see this @ApacheArrow work out of @uwdata!
Jeffrey Heer@jeffrey_heer4 Feb 2021
Replying to @jeffrey_heer
For example, here are benchmark results (in ms) for sequentially parsing, counting, and filtering 5M row Arrow tables (node v14, 2017 MacBook Pro). In many cases, Arquero matches or exceeds the performance of the reference Arrow JS implementation!
Arrow processing with 5M values, 0% nulls: init table | arrow-js: 0ms | arquero: 0ms count dictionary │ arrow-js: 15ms │ arquero: 16ms filter dictionary | arrow-js: 24ms │ arquero: 15ms filter numbers 0 │ arrow-js: 70ms │ arquero: 102ms filter numbers 1 │ arrow-js: 103ms │ arquero: 110ms Arrow processing with 5M values, 5% nulls init table │ arrow-js: 0ms | arquero: 27ms count dictionary │ arrow-js: 130ms │ arquero: 14ms filter dictionary │ arrow-js: 169ms │ arquero: 49ms filter numbers 0 │ arrow-js: 209ms │ arquero: 128ms filter numbers 1 │ arrow-js: 204ms │ arquero: 128ms

ALT Arrow processing with 5M values, 0% nulls: init table | arrow-js: 0ms | arquero: 0ms count dictionary │ arrow-js: 15ms │ arquero: 16ms filter dictionary | arrow-js: 24ms │ arquero: 15ms filter numbers 0 │ arrow-js: 70ms │ arquero: 102ms filter numbers 1 │ arrow-js: 103ms │ arquero: 110ms Arrow processing with 5M values, 5% nulls init table │ arrow-js: 0ms | arquero: 27ms count dictionary │ arrow-js: 130ms │ arquero: 14ms filter dictionary │ arrow-js: 169ms │ arquero: 49ms filter numbers 0 │ arrow-js: 209ms │ arquero: 128ms filter numbers 1 │ arrow-js: 204ms │ arquero: 128ms

1
3
Brian Hulette retweeted
Jeffrey Heer@jeffrey_heer4 Feb 2021
For example, here are benchmark results (in ms) for sequentially parsing, counting, and filtering 5M row Arrow tables (node v14, 2017 MacBook Pro). In many cases, Arquero matches or exceeds the performance of the reference Arrow JS implementation!
Arrow processing with 5M values, 0% nulls: init table | arrow-js: 0ms | arquero: 0ms count dictionary │ arrow-js: 15ms │ arquero: 16ms filter dictionary | arrow-js: 24ms │ arquero: 15ms filter numbers 0 │ arrow-js: 70ms │ arquero: 102ms filter numbers 1 │ arrow-js: 103ms │ arquero: 110ms Arrow processing with 5M values, 5% nulls init table │ arrow-js: 0ms | arquero: 27ms count dictionary │ arrow-js: 130ms │ arquero: 14ms filter dictionary │ arrow-js: 169ms │ arquero: 49ms filter numbers 0 │ arrow-js: 209ms │ arquero: 128ms filter numbers 1 │ arrow-js: 204ms │ arquero: 128ms

ALT Arrow processing with 5M values, 0% nulls: init table | arrow-js: 0ms | arquero: 0ms count dictionary │ arrow-js: 15ms │ arquero: 16ms filter dictionary | arrow-js: 24ms │ arquero: 15ms filter numbers 0 │ arrow-js: 70ms │ arquero: 102ms filter numbers 1 │ arrow-js: 103ms │ arquero: 110ms Arrow processing with 5M values, 5% nulls init table │ arrow-js: 0ms | arquero: 27ms count dictionary │ arrow-js: 130ms │ arquero: 14ms filter dictionary │ arrow-js: 169ms │ arquero: 49ms filter numbers 0 │ arrow-js: 209ms │ arquero: 128ms filter numbers 1 │ arrow-js: 204ms │ arquero: 128ms

2
1
17
Spark has Koalas, and Beam has ... well, Pandas. Pandas DataFrame API is now supported in @ApacheBeam. Develop pipelines using DataFrame, and run them on top of highly-scalable distributed infrastructure. I will let you decide which mascot is cuter. beam.apache.org/blog/datafra…
3
22
8
96
Really exciting to show off these new features with Robert :)
Aizhamal Nurmamat kyzy@iamaijamal26 Aug 2020
Check out all of these new, nifty utilities to develop nicer @ApacheBeam Python pipelines with #schemas! If you are not watching this talk by @BrianHulette and Robert Bradshaw at @BeamSummit, here are their slides and video will be uploaded soon💥 s.apache.org/simpler-python-…
1
Someone is running around my neighborhood putting googly eyes on everything. @MrTrashWheel would be proud.
1
My wife is in healthcare so I'm alone in the apartment all day working from home. At least I have this joker to keep me company. I call him bumpy.
4
Saw the Christmas Ship Festival in Seattle for the first time last night, what a great tradition! This incredible shot brought to you by my wife and Google (night sight + photos panorama stitching)
Finally hit my goal of running 1000 miles in 2019 today! When I set the goal early this year I thought it would be a cakewalk, but after a few weeks of slacking off I found myself playing catch-up here at the end.
1
6
I love this criteria, it's what makes contributing to open source so rewarding.
This Post is from an account that no longer exists. Learn more
The slides from my talk at the first Seattle @ApacheBeam meetup are available at s.apache.org/portable-schema…. Thanks to all who attended and I look forward to seeing you again at the next one :)
1
3
10
Got my laptop all done up with @TheASF stickers thanks to @ApacheCon and @BeamSummit :)
1
1
9
I just published the slides for my @BeamSummit talk on using @ApacheArrow in @ApacheBeam at s.apache.org/beam-arrow-vega… and a recording of the talk should be coming soon!
2
11
28
Brian Hulette retweeted
BeamSummit@BeamSummit12 Sep 2019
Kicking off the Beam Summit in Reno 2 today, is @BrianHulette, talking about columnar, cross-language data pipelines and how this can be achieved using @ApacheArrow and @ApacheBeam! #BeamSummit #ACNA19
1
7
Brian Hulette retweeted
Wes McKinney
@wesmckinn4 Sep 2019
Here are materials from my @VLDB2019 workshop (for system developers) on @ApacheArrow Slides: slideshare.net/wesm/apache-a… Notebooks: github.com/wesm/vldb-2019-ap… In the Flight demo, you can see Arrow datasets served at ~1.5GB/s over TCP, it's a speedy one
6
42
1
166
Brian Hulette retweeted
Wes McKinney
@wesmckinn26 Aug 2019
It's been a rare privilege to have Ruby, Python, MATLAB, and R developers all working symbiotically together on a project of interest (i.e. @ApacheArrow) but I hope we see a lot more of this going forward
2
53
1
361
Excited to show the community what I've been working on!
Aizhamal Nurmamat kyzy@iamaijamal16 Aug 2019
#Python or #Java? 🙄It doesn't matter if you use @ApacheBeam portable Schemas!🕺😎 @BrianHulette from @GCPcloud will demonstrate some of the new Beam capabilities in the Python SDK, particularly the ability to use Java’s SqlTransform in Python pipelines! bit.ly/2T6qYr2
2
Load more