Big Fast Data Systems Engineer
7 January 2014, Walmart, Full Time, San Bruno, CA, USA
As a Systems Engineer on our BFD Team (Big Fast Data) Tech Team, you will be working on systems that provide data delivery and processing at Walmart scale for a variety of users. Our full-stack approach and end-to-end team responsibility mean that you will be involved in these systems in many ways, from design and planning to implementation and review, from packaging and deployment to user and developer support. The specific challenges will vary from one system to the next, but expect BFD themes to include large-scale (distributed) systems, data and I/O-intensive processing, and work with and on key data that powers @WalmartLabs. Expect to dive into open source, whether as adoptee or contributor, to take on any project you need to make it do your bidding----or take on new code to push the frontier of what's possible.
Challenges include: robust distributed processing, data-stream computational pipelines, real-time indexing, capacity planning, and performance tuning.
- Design, prototype, and develop high-performance distributed systems that ingest, process, and manage terebytes to petabytes of real-time streams and archival data from, throughout Walmart and beyond. (You know, Walmart, as the largest retailer on the planet: Who were you thinking of when you were reading or writing about "market baskets" or data mining?)
- Be at the forefront of solutions for distributed processing, live-data-stream computation, real-time indexing, capacity planning, and performance tuning.
- Work with and further develop open-source software such as Hadoop, Cassandra, Mupd8, and more.
- Make and impact on a global scale.
The Desired Skill-set:
- BS/MS/PhD in Computer Science, Computer Engineering, CSE, ECE, or related field
- 2-5+ years of Programming experience in C/C++/Java/Perl/Python, including fluency in at least one systems-building language
- Extensive Unix/Linux systems-programming experience
- Expertise in distributed/sclable systems and algorithms with awareness of time and space complexity
- Strong background in process design for reliable systems; strong "big picture" awareness of systems
- Good generalist experience, with ability and willingness to read and write into all layers of a software stack
- Experience working with operations or with deployment, monitoring, and other sustainable operation of software
- Working knowledge of standard tools for optimizing and testing code; plus to have experience building tools/apps, software libraries/frameworks, and/or working with custom-built tools
- Past experience handling large amounts of data on distributed platforms is also a huge plus
- Bachelor's Degree in Computer Science or related field and 6 years experience designing and building scalable ecommerce or distributed applications.