Kuhu Shukla (bottom center) and team at the 2017 DataWorks Summit
By Kuhu Shukla
This post first appeared here on the Apache Software Foundation blog as part of ASF’s “Success at Apache” monthly blog series.
As I sit at my desk on a rather frosty morning with my coffee, looking up new JIRAs from the previous day in the Apache Tez project, I feel rather pleased. The latest community release vote is complete, the bug fixes that we so badly needed are in and the new release that we tested out internally on our many thousand strong cluster is looking good. Today I am looking at a new stack trace from a different Apache project process and it is hard to miss how much of the exceptional code I get to look at every day comes from people all around the globe. A contributor leaves a JIRA comment before he goes on to pick up his kid from soccer practice while someone else wakes up to find that her effort on a bug fix for the past two months has finally come to fruition through a binding +1.
Yahoo – which joined AOL, HuffPost, Tumblr, Engadget, and many more brands to form the Verizon subsidiary Oath last year – has been at the frontier of open source adoption and contribution since before I was in high school. So while I have no historical trajectories to share, I do have a story on how I found myself in an epic journey of migrating all of Yahoo jobs from Apache MapReduce to Apache Tez, a then-new DAG based execution engine.
Oath grid infrastructure is through and through driven by Apache technologies be it storage through HDFS, resource management through YARN, job execution frameworks with Tez and user interface engines such as Hive, Hue, Pig, Sqoop, Spark, Storm. Our grid solution is specifically tailored to Oath’s business-critical data pipeline needs using the polymorphic technologies hosted, developed and maintained by the Apache community.
On the third day of my job at Yahoo in 2015, I received a YouTube link on An Introduction to Apache Tez. I watched it carefully trying to keep up with all the questions I had and recognized a few names from my academic readings of Yarn ACM papers. I continued to ramp up on YARN and HDFS, the foundational Apache technologies Oath heavily contributes to even today. For the first few weeks I spent time picking out my favorite (necessary) mailing lists to subscribe to and getting started on setting up on a pseudo-distributed Hadoop cluster. I continued to find my footing with newbie contributions and being ever more careful with whitespaces in my patches. One thing was clear – Tez was the next big thing for us. By the time I could truly call myself a contributor in the Hadoop community nearly 80-90% of the Yahoo jobs were now running with Tez. But just like hiking up the Grand Canyon, the last 20% is where all the pain was. Being a part of the solution to this challenge was a happy prospect and thankfully contributing to Tez became a goal in my next quarter.
The next sprint planning meeting ended with me getting my first major Tez assignment – progress reporting. The progress reporting in Tez was non-existent – “Just needs an API fix,” I thought. Like almost all bugs in this ecosystem, it was not easy. How do you define progress? How is it different for different kinds of outputs in a graph? The questions were many.
I, however, did not have to go far to get answers. The Tez community actively came to a newbie’s rescue, finding answers and posing important questions. I started attending the bi-weekly Tez community sync up calls and asking existing contributors and committers for course correction. Suddenly the team was much bigger, the goals much more chiseled. This was new to anyone like me who came from the networking industry, where the most open part of the code are the RFCs and the implementation details are often hidden. These meetings served as a clean room for our coding ideas and experiments. Ideas were shared, to the extent of which data structure we should pick and what a future user of Tez would take from it. In between the usual status updates and extensive knowledge transfers were made.
Oath uses Apache Pig and Apache Hive extensively and most of the urgent requirements and requests came from Pig and Hive developers and users. Each issue led to a community JIRA and as we started running Tez at Oath scale, new feature ideas and bugs around performance and resource utilization materialized. Every year most of the Hadoop team at Oath travels to the Hadoop Summit where we meet our cohorts from the Apache community and we stand for hours discussing the state of the art and what is next for the project. One such discussion set the course for the next year and a half for me.
We needed an innovative way to shuffle data. Frameworks like MapReduce and Tez have a shuffle phase in their processing lifecycle wherein the data from upstream producers is made available to downstream consumers. Even though Apache Tez was designed with a feature set corresponding to optimization requirements in Pig and Hive, the Shuffle Handler Service was retrofitted from MapReduce at the time of the project’s inception. With several thousands of jobs on our clusters leveraging these features in Tez, the Shuffle Handler Service became a clear performance bottleneck. So as we stood talking about our experience with Tez with our friends from the community, we decided to implement a new Shuffle Handler for Tez. All the conversation points were tracked now through an umbrella JIRA TEZ-3334 and the to-do list was long. I picked a few JIRAs and as I started reading through I realized, this is all new code I get to contribute to and review. There might be a better way to put this, but to be honest it was just a lot of fun! All the whiteboards were full, the team took walks post lunch and discussed how to go about defining the API. Countless hours were spent debugging hangs while fetching data and looking at stack traces and Wireshark captures from our test runs. Six months in and we had the feature on our sandbox clusters. There were moments ranging from sheer frustration to absolute exhilaration with high fives as we continued to address review comments and fixing big and small issues with this evolving feature.
As much as owning your code is valued everywhere in the software community, I would never go on to say “I did this!” In fact, “we did!” It is this strong sense of shared ownership and fluid team structure that makes the open source experience at Apache truly rewarding. This is just one example. A lot of the work that was done in Tez was leveraged by the Hive and Pig community and cross Apache product community interaction made the work ever more interesting and challenging. Triaging and fixing issues with the Tez rollout led us to hit a 100% migration score last year and we also rolled the Tez Shuffle Handler Service out to our research clusters. As of last year we have run around 100 million Tez DAGs with a total of 50 billion tasks over almost 38,000 nodes.
In 2018 as I move on to explore Hadoop 3.0 as our future release, I hope that if someone outside the Apache community is reading this, it will inspire and intrigue them to contribute to a project of their choice. As an astronomy aficionado, going from a newbie Apache contributor to a newbie Apache committer was very much like looking through my telescope － it has endless possibilities and challenges you to be your best.
About the Author:
Kuhu Shukla is a software engineer at Oath and did her Masters in Computer Science at North Carolina State University. She works on the Big Data Platforms team on Apache Tez, YARN and HDFS with a lot of talented Apache PMCs and Committers in Champaign, Illinois. A recent Apache Tez Committer herself she continues to contribute to YARN and HDFS and spoke at the 2017 Dataworks Hadoop Summit on “Tez Shuffle Handler: Shuffling At Scale With Apache Hadoop”. Prior to that she worked on Juniper Networks’ router and switch configuration APIs. She likes to participate in open source conferences and women in tech events. In her spare time she loves singing Indian classical and jazz, laughing, whale watching, hiking and peering through her Dobsonian telescope.
Forgotten Warrior: Native Vet Waits 41 Years For Medals
Theodore Harvey’s a modest man that lives simply. His bed is properly turned out–crisp sheets are stretched tautly across a single frame without a visible wrinkle, though his hands shake with each querulous movement. His magazines,National Geographics for the most part, lie stacked neatly against the windowsill, next to a shadow box celebrating honors won in Vietnam.
Those honors, simple bits of metal and cloth to the outsider, mean more to Harvey than nearly anything else in the room.
Neither young nor old for his years, Harvey looks all of his 78 hard-lived years–nearly a quarter of them spent fighting, training and waiting on foreign soil.
Harvey was 19 when he enlisted in 1954. He fought–valiantly–for 17 years before he was discharged in 1971.
He then waited 41 years and three days to receive decorations he should have received half a lifetime ago.
Around the tables set up in the Mescalero High School Gymnasium, veterans of different wars–Vietnam and Iraq to name the usual suspects–watched, their individual stories and questions writ large in their expressions and movements.
For the young 1Lt. Daniel Hance, recently returned from the sands, the ceremony was a day of honor, glory and well-deserved recognition. Hance’s eyes shone and his hands were steady as he pinned on the Bronze Star.
For Jerry Ligon, commander of VVA 1062, there was a hint of sorrow as he fastened the Purple Heart, a match for his own medal, on Harvey’s coat.
Theodore Harvey is a Native American veteran that lives quietly in the Mescalero Apache Reservation just outside of Ruidoso, and his story is, unfortunately, far from unique.
Native Americans, percentage-wise, serve in greater frequency in the armed forces than any other ethnicity, according to Department of Defense statistics.
An estimated 12,000 Native Americans stepped up in World War I, with that number rising to about 44,000 soldiers in World War II–roughly 1/8 of the population at the time. About 42,000 willingly marched in to Vietnam, only 10 percent conscripts,according to the Naval History and Heritage website.
There are an estimated 190,000 Native American veterans today, according to the DoD.
Yet recognition for these warriors, as well as other critical benefits, lags behind other veterans, many of whom already are struggling to collect their dues.
”AIAN (American Indian/Alaskan Native) veterans have 1.9 times higher odds of being uninsured compared with non-Hispanic white veterans,“ the report states. They also are "significantly more likely to delay care due to not getting timely appointments,” they are unlikely to get through on the phone and frequently have transportation problems.
A mail-order Apache souvenir throwing tomahawk featuring a swastika on its blade, from around 1952. This was made by the Pasadena, California, company Techrite Corporation and was from an ad in the February 1952 issue of Popular Mechanics.
Two original Arizona state route signs. These ran from the late 1920s until around 1940.
SR 67, is the only road that links historic Route 89a with the rim of the Grand Canyon. The Arizona DOT doesn’t own the last miles of this route, but it is still labeled as SR 67.
SR 61 runs through Apache County starting in Show Low, Arizona all the way to the New Mexico border.
You always know when the journey starts, but it’s hard to tell when will it end… #day567 #lego365 #lego #legofan #afol #legostagram #legophotography #instalego #minifigures #toys #bricks #brickcentral #bricknetwork #toygroup_alliance #toyslagram_lego #tosylagram #legography #toyartistry_lego #lego_hub #journey #apache
I recently hit 500 followers and that way more than I ever imagined happening so to celebrate I’m giving away some gorgeous crystals! (be thankful, i almost kept them for myself) :P
Full moon kit with 3 crystals- Moonstone, Apache tear and clear quartz
1 amethyst cluster!!!
RULES:
You must be a follower (new followers welcome of course)
Reblog this photo.
world wide of course
Thank you all! Happy almost Halloween and Blessed be!!!
