Author: karanparikh

Domum

(domum means home in Latin)

High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads is a fantastic paper. Its primary focus is how to build distributed systems that are both highly available and strongly consistent. This is achieved by building multi-homed systems. As the paper describes them —

Such systems run hot in multiple datacenters all the time, and adaptively move load between datacenters, with the ability to handle outages of any scale completely transparently. [1]

While the paper mostly addresses building multi-homed systems in the context of distributed stream processing systems, the concepts and ideas are general enough that they can be applied to any large scale distributed software system with some modifications.

Before designing a distributed system that is resilient to failures it is paramount to understand what a failure even means in the context of software systems. Section 3 of the paper talks about common failure scenarios and highlights an important fact — partial failures are common, and “are harder to detect, diagnose, and recover from” [1] (compared to total failures). An important takeaway from this section is that when designing a new system (or trying to improve an old/current system) one should always think about what partial failures can occur, and how the system can/would react to it.

The next section motivates the need for multi-homed systems by first talking about singly-homed and failover-based systems. While singly-homed and failover-based systems are common, one typically does not run into multi-homed systems unless one operates at Google-scale (or close to). Building multi-homed systems is hard. But they offer significant benefits over singly-homed and failover-based systems in the face of (partial or total) failure. Google leverages its existing infrastructure, in particular Spanner, to build multi-homed systems with high availability.

Section 5 is the most interesting portion of the paper and talks about the challenges inherent in building multi-homed system. My main takeaway from this section is that it is virtually impossible to build a multi-homed distributed system without a system like Spanner (which is itself a multi-homed system) serving as the foundation — many of Spanner’s features, like global synchronous replication, reads at a particular version, etc. are used to solve the challenges mentioned in this section.

The paper ends with the description of three multi-homed systems at Google: F1/Spanner, Photon, and Mesa. I highly recommend reading the papers for each of these systems as well, as they have a lot more details about how these complex systems were built.

References
[1] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44686.pdf

Quotes: September Edition

I wrote a blog post talking about how I write down lines from books I’m reading. Here are some lines that I loved from the books that I read in September:

(I also just realized that both the books I read in September were about World War 2)

People aren’t supposed to look back. I’m certainly not going to do it anymore. — Slaughterhouse-Five

But sleep would not come. Tears came instead.  — Slaughterhouse-Five

“That’s one thing Earthlings might learn to do, if they tried hard enough: Ignore the awful times, and concentrate on the good ones.” — Slaughterhouse-Five

“Other people’s dreams aren’t very interesting, usually.” — Slaughterhouse-Five

The worst can always be true. — The Nightingale

Inside, the house echoed the voice of a man who wasn’t there. — The Nightingale

I belong to a generation that didn’t expect to be protected from every danger. — The Nightingale

Vianne finished the letter and immediately began reading it again. — The Nightingale

But when he looked at her — and she looked at him — they both knew that there was something worse than kissing the wrong person.
It was wanting to.
— The Nightingale

“Now that you’ve met me, you’ll never be able to forget me. That’s something.” — The Nightingale

I see the surgeon in him suddenly, the man who is used to staring past blood and bone to find what is broken. — The Nightingale

How can I start at the beginning, when all I can think about is the end? — The Nightingale

“Some stories don’t have happy endings. Even love stories. Maybe especially love stories.” — The Nightingale

Goal Tracking: September Edition

At the beginning of the year I published a post outlining what some of my goals for the year were. In the spirit of being transparent, here is the progress I made on them over the course of September –

  1. 100% self availability over the course of the month.
  2. Volunteered for 0 hours  😦
  3. Little to no procrastination. Minus a tiny bit of laziness regarding buying tickets for Hawaii.
  4. Continued being honest and open over the course of September. 
  5. Made a tiny bit of progress on learning Rust. This was mostly just reading blog posts about Rust.
  6. I read 2 books in September — Slaughterhouse-Five (I read this book for the first time in 2012. And I was unimpressed. But re-reading it in 2016, with a new found appreciation and understanding of Kurt Vonnegut, made me realize how brilliant and amazing this book is) and The Nightingale (I LOVED this book! It’s beautifully written, with an engaging story, a haunting backdrop, and well developed characters. It made me cry quite a bit. This is one of my favorite books of all time).
  7. I read 1 research paper — Design patterns for container-based distributed systems. I wrote a post about it as well.
  8. I wrote 4 posts.
  9. I didn’t really play much guitar in September.
  10. (a) (goal achieved)
    (b) (goal achieved)
    (c) (goal achieved)
    (d) (goal achieved)
    (e) (goal achieved)

Papir

(This post is a summary of two papers I have recently read. Papir is the Norwegian word for paper)

Real-Time Twitter Recommendation: Online Motif Detection in Large Dynamic Graphs is a paper that was presented at VLDB 2016. It combines two of my favorite topics, distributed systems and graph theory, into a short (2 pages!) paper. It presents a simplified version of the algorithm that Twitter uses to detect motifs in real-time in a user’s social graph, which is then used to generate recommendations for the user. One thing I liked about this paper is that it presents naive solutions to the problem at hand before diving into the elegant solution that Twitter uses. The paper then presents their solution to the problem, and explains how it works at Twitter scale by graph partitioning, pruning, and offline data structure generation.

Design patterns for container-based distributed systems is a paper by Google that talks about software design patterns that are emerging from software systems that are built around containers. Software like Docker and CoreOS has made working with containers easier, and more and more companies are moving towards a container based ecosystem. Google was one of the first companies to use containers, and this paper contains design and architecture patterns that they have observed in their container based systems. The design patterns presented are grouped under three main categories of which I enjoyed reading about “Multi-node application patterns” the most. This sections deals with design patterns in distributed systems, where each node holds multiple related containers (called “pods” in the paper). It was interesting to read about how distributed system problems like leader election, scatter-gather, etc. can be dealt with language agnostic containers rather than by language specific libraries. I loved this line from the end of the paper, which made me think of containers in an entirely new light:

In all cases, containers provide many of the same benefits as objects in object-oriented systems, such as making it easy to divide implementation among multiple teams and to reuse components in new contexts. In addition, they provide some benefits unique to distributed systems, such as enabling components to be upgraded independently, to be written in a mixture of languages, and for the system a whole to degrade gracefully.

 

Quotes: August Edition

I wrote a blog post talking about how I write down lines from books I’m reading. Here are some lines that I loved from the book that I read in August:

It’s tough to live with people stuck in the past, isn’t it? — Harry Potter and the Cursed Child

The truth is a beautiful and terrible thing, and should therefore be treated with great caution. — Harry Potter and the Cursed Child

I exist to serve. — Harry Potter and the Cursed Child

Thank you for being my light in the darkness. — Harry Potter and the Cursed Child

Goal Tracking: August Edition

At the beginning of the year I published a post outlining what some of my goals for the year were. In the spirit of being transparent, here is the progress I made on them over the course of August –

  1. 65%-75% self availability over the course of the month.
  2. Volunteered for 0 hours  😦
  3. An embarrassing amount of procrastination.
  4. August 2016 was, by far, the most honest and open I’ve ever been in my life. The experiences I’ve had during this month have helped me grow a lot and certain moments are most definitely imprinted on the fabric of my mind.
  5. Made zero progress on learning Rust.
  6. I only managed to read one book in August — Harry Potter and the Cursed Child (a good, but NOT great, book).
  7. I read 2 research papers — Real-Time Twitter Recommendation: Online Motif Detection in Large Dynamic Graphs and High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads (I LOVED this paper. I will be writing a post about it soon).
  8. I wrote 5 posts.
  9. I learned to play 2-3 songs on the guitar.
    (Note to self — I need to learn more non-metal songs. Any time someone asks me to play a song the interaction is always along the lines of:
    “Oh cool! Why don’t you play something?”
    “Sure, but you probably haven’t heard the song.”
    Yes, almost none of my friends listen to metal music)
  10. (a) (goal achieved)
    (b) (goal achieved)
    (c) (goal achieved)
    (d) (goal achieved)
    (e) (goal achieved)

100

I’ve been tracking all the artists I’ve seen live since 2013, and a few weeks ago (at my third Outside Lands festival) this tally reached 100.

The first artist I saw live was Chevelle at the Fillmore in San Francisco, CA on July 8th 2014. I remember standing on the fringes of the mosh pit and being extremely scared that I would be sucked in (I didn’t). I also remember being amazed by how good the band sounded live, and how talented each and every member was. Needless to say I had a fantastic time — I was blown away by the whole experience, and by the time Chevelle came back on stage for their encore I’d already decided that I would try and see as many concerts as possible.

It’s really hard to pick a favorite show of the 100+ I’ve seen. A top 5 list (in no particular order) would probably feature Tame Impala (at Coachella 2016), AC/DC, Metallica, TesseracT, Deafheaven. Not only do Tame Impala sound sublime live (I’m so excited to see them again on September 3rd), but the lights and videos they use during their live sets elevate their music to another level. AC/DC and Metallica hold huge sentimental value for me since they were the bands I grew up listening to, and seeing them live was an emotional experience. TesseracT is my (current) favorite band and the first progressive metal band I enjoyed listening to, and they were awesome live (very excited to see them again with Gojira on October 7th). Plus I got to meet the band before the show which was very cool! Deafheaven was easily the most intense show I’ve been to. Though given the type of music they play this is to be expected.

The bands I’ve seen the most times live are Tennis and Mastodon. The last time I saw Mastodon was when they played the night before my 25th birthday. It was a great birthday gift (yes, I do know it was a day early).

The Chevelle show ignited a love for live music within me that still shines brightly today.

Sunrise

I got to see Haitus Kaiyote at Outside Lands 2016 last weekend. They were one of the best bands I’ve seen live so far. Seeing them live gave me a newfound love and appreciation for their music and I’ve been listening to their songs all of last week. Not only is their music beautiful, but the pictures they paint with their lyrics are gorgeous as well, and some of them seem to appeal to my current state of mind.

 

What I’m currently listening to: 65daysofstatic

No Man’s Sky came out recently. While I don’t intend to play the game (I own an XBox One, which I haven’t really used for the past 3-4 months) I heard very good things about the soundtrack that 65daysofstatic put together for the game and decided to give it a listen. It is sublime. My appreciation for the music grew even more after I learned how the album was created.