On boats and consensus

I finally got done reading the Raft paper. I will be the first to admit that I’m a bit off schedule.

I was introduced to Paxos during my undergraduate studies at UIUC. Three times in fact; the first during the distributed systems class, the second during the cloud computing class, and the third during the advanced distributed systems class. Each time I had the same thought while (re)learning the algorithm: Man, this is pretty hard. It always took me a couple of times of reading the slides (for the course) or the (simplified) paper before I understood the algorithm. And I would forget it in about a month. Not the entire algorithm, but bits and pieces. Important bits and pieces unfortunately.

With Raft I’m hoping that will not be the case. It took me 2 reads of the paper to understand pretty much most of the algorithm. It has relatively few moving pieces (only 3 types RPCs!) which makes it pretty easy to understand. The paper does a great job of explaining the algorithm by first explaining how the system would function without safety constraints, and then modifying the RPCs and algorithm to deal with safety. Even with the safety constraint added the algorithm is quite intuitive. This simplicity of the algorithm leads to good performance as well: in the steady state (no crashes, and assuming the client communicates directly with the leader) the leader only has to wait for a response from a majority of followers in order to respond to a client request. My favorite part of the algorithm is leader election. Using randomized election timeouts is a simple yet effective idea.

I would recommend anyone interested in distributed systems read the Raft paper and watch this video as well.

 

linkedin.github.io

Matthew Shoup and I worked on a new design for linkedin.github.io for LinkedIn’s January hackday, and with some help from Yevgeniy Brikman we launched it on 2/21.

I had a lot of fun working on this small project. This was the first JavaScript project I’ve worked on since I graduated and it took me a few minutes to get into the JavaScript + HTML + CSS development “frame of mind”. Once that initial rustiness wore off though it was a lot of fun to build this website.

Twitter bootstrap is still amazing. The ability to write “debugger” in your JavaScript source file and have the browser pause there so that you can inspect state is fantastic. And each time I use the developer tools in Chrome I am reminded on Bret Victor’s “Inventing on Principle” talk.

“Paper” Review: Distributed systems for fun and profit

Book link: Distributed systems for fun and profit

OK, I will admit it, this is not a paper. I think the best description for it is a mini-textbook on distributed systems.

Over the course of five chapters this book goes over most of the major concepts in distributed systems, including synchronous v.s. asynchronous systems, vector clocks, the CAP theorem, detecting failures, and consensus algorithms. While the book doesn’t go in depth into any one of these topics, it helps lay a fantastic foundation on which you can build upon by reading books and papers (more on this in a bit) on distributed systems and related concepts. If your goal is to understand at a high level what systems like Kafka, Cassandra, or MongoDB do and why they are designed the way they are this book is for you. It will make terms like “eventual consistency” and “quorums” make sense when you read documentation for these systems and will help you design your applications better as well.

I think my favorite aspect about this book is that it introduces readers to newer distributed systems concepts like the CALM theorem, RAFT, CRDTs etc.

Another thing that I loved was that each chapter (and the Appendix) also has links to papers and other resources that go more in depth for each concept presented.

Overall, I would highly recommend reading this. If you’re new to distributed systems this book will help you get started. If you live and breathe distributed systems then this is a good resource to have on hand to brush up/look up topics quickly. And it has such pretty diagrams!

val happy = new Year(2014);

It’s that time of the year again, where people make lists of things they want to achieve in the new year. Here’s mine:

  1. Learn a new programming language: learning Go has been on my radar for a long time now. I like how the language looks and it has a few features that seem really awesome (this is probably my favorite). The first thing I write in it will most probably be an implementation of 2 phase commit, mainly to see how hard/easy it is to write concurrent code. 
  2. Read three research papers a month: I read a lot of research papers for CS 525, and it was something I really enjoyed. I’ll be focusing mainly on distributed systems papers, but I might read some papers on machine learning as well. First few papers on the list: F1 and Raft.
  3. Blog more: I hardly blogged at all in 2013. Hopefully 1 and 2 above will give me more things to write about.
  4. Play guitar for at least 30 minutes everyday.
  5. Work out 6 days a week. And start running again.

Happy new year.

“Goodbye grandpa”

My grandfather passed away today.

I didn’t realize how close I was to him until I heard of his death. That’s when it really hit me.

Some of my earliest memories of India are associated with him. When my family moved to India I remember staying with him and my grandmother. He used to drive me to school. I was still not used to Indian food, a result of which I used to put the food they served us at school in my school bag. What I was not so good at was removing the food from my bag once I’d get back home. He discovered this when he found a trial of ants leading to my bag. I remember him having a good laugh over it.

He taught me a lot about Hinduism. He used to take me the EME temple close to his house whenever possible. He also used to try and tell me snippets from the Ramayana when I was growing up.

Cricket and politics were his passions, and I must admit it was quite fun to see him getting worked up about both. He loved telling people that his son looked like Sourav Ganguly (which I believe he does).

Like me, he loved his food. We were both huge fans of my grandmother’s mutton curry. Kamling, his favorite Chinese restaurant in Mumbai, soon became one of my favorite places to eat as well. And man, did he enjoy his seafood! He tried to explain to me numerous times how to eat bony fish without choking by correctly deboning it, and I hope one day to actually be able to do that.

In later years I became one of his go-to people for all tech related issues, be it how to check his email on his phone, or how to configure his computer. In fact, the last time I visited him before leaving for the U.S. was when I helped him setup Gmail to allow him to reply in Bengali. That was the last time I saw him.

In short, my grandfather was awesome.

Goodbye grandpa. You will be missed terribly.

New Beginnings

I graduated in May after which I went back home to India for a while. I spent two months doing pretty much nothing, which I must say, was awesome. I spent time with my family, hung out with friends, read books, watched some movies, and lost two teeth (a result of which I lost 5 pounds yay!). I also spent some time with Akka, brushed up on my distributed systems knowledge and read one paper. Some of my loftier goals, which included learning Go, reading more than one paper, and open sourcing Gelato, were not achieved unfortunately. All in all though, it was a wonderful Summer.

I came back to the U.S. towards the end of July. I live in Mountain View with my awesome buddies Kevin, Onur, and Sam. I just want to say, I ❤ our house! I’m amazed that we found such a great place to live.

house
My house!

Oh, and if you’re wondering why I moved to Mountain View, it’s because I now work at LinkedIn! I’m a Software Engineer on the Service Infrastructure (SI) team at LinkedIn. As I’ve said earlier, my internship at LinkedIn was fantastic and it feels great to be back. Last Summer I worked on integrating the Play framework into LinkedIn’s existing infrastructure. As a member of the SI team I’m currently working on Rest.li (Rest.li on GitHub. This is also the first open source project I’ve contributed to).

IMG_20130819_191359
LinkedIn Building 2029

Super excited to see what the future holds 🙂

Concurrency, Threads, Actors and Akka

Concurrency is a topic that’s close to my heart. It all started when I was introduced to Pthreads in my CS 241 Systems Programming class at UIUC. Having the ability to write code that could be doing multiple things simultaneously was (and is) pretty awesome. The assignments in that class, which included writing a parallel sorting function, a parallel version of make and a simple HTTP Server, really helped us appreciate the power of concurrency and threading. My love for concurrency has only grown since then and I’ve written concurrent code in both Java and Python.

This Summer I spent some time with Scala Actors and Akka (I got familiar with the concept of actors on Scala 2.8.1 and then upgraded to Scala 2.10.2 for Akka 2.1.4. As mentioned, Scala 2.10 and above will use Akka for actors). I feel that the actor model is a great way to think of and write concurrent programs. By relying only on immutable message passing (Yes, I know that you can pass mutable messages. But please don’t do that.) between actors for communication and coordination I feel that the likelihood of common errors seen in threading based concurrent programs like deadlock, livelock, race conditions etc. are reduced. Moreover, there are certain classes of applications, for example distributed systems applications, that can easily be modeled as a system with entities that communicate by sending messages to each other. Actors would be great for something like this as the actor code would map pretty neatly to the message flows in the system.

Akka builds on top of the actor model and adds a bunch of cool features. Some of my favorite include: supervision and monitoring, being able to refer to any actor using a hierarchical path, FSM support, Netty based remoting, and routers. This is a great example of why I’m a fan of Akka. The author of the post was able to model his problem as an FSM and translate it pretty neatly into code. This example ties in with what I was saying earlier about actors and distributed systems. By decomposing the system into a set of entities and possible messages that can be exchanged between the two the author of the blog post was able to come up with an elegant solution for the problem.

For programmers who’ve only ever used the threading model of concurrency in the past, I say give actors a try! You might be able to come up with a more awesome and well-structured solution to your problem. Akka also has a Java API in case you do not want to use Scala.

Akka resources:

  1. Akka Java and Scala guide (these are amazing)
  2. The Akka Team Blog

So, you want to win a hackathon?

In the past 3 years I’ve taken part in 11 hackathons. Of these, I’ve won 4 and have reached the finals in 2. Here are a few things that I’ve observed that I feel should increase your chances of winning a hackathon:

Cool tech does not always equate to a cool product: Regardless of how awesome your tech is, if you don’t have a great product you’re probably not going to win. Use the clever algorithm you came up with or the sweet new tech stack you’ve invented in a product or application that other people can use and love. For example:

DON’T do this: We solved the P v.s. NP problem and then solved the graph coloring problem.

DO this: We solved the P v.s. NP problem and used that to create an application that determines which of your friends you are most likely to start a company with by finding a solution to the graph coloring problem.

That was a pretty bad example, but hopefully I was able to get my point across.

Respect the demo: The demo is the most important part of a hackathon. This is where you convince the judges, in 2 or 3 minutes, that you spent the last 18 to 24 hours creating a product that is the best thing since sliced bread. Make sure you practice what you’re going to be showing the judges! Having a well thought out demo is key to winning a hackathon. Focus on the main features of your product, why it is awesome and why people would want to use it. Since time is limited try to show the most important and/or impressive features of your application.

Did I mention you should PRACTICE, PRACTICE, PRACTICE your demo?

Don’t be afraid to ask for help: Most hackathons have employees (of the company organizing the hackathon) being present during the event as mentors for the competing teams. Don’t be afraid to ask them for help. These employees are an excellent source of knowledge and you should always reach out to them anytime you’re stuck on a nasty bug, can’t figure out how to use a new technology or library etc. Since hackathons give you limited time to work on your product it makes sense to get rid of roadblocks quickly. Of course, make sure you spend some time trying to solve your problem on your own first before turning to the mentors for help.

Try to make your product look pretty: Having a product that is polished and looks good is key to winning a hackathon in my opinion. Make sure you allocate some time to work on the UI and UX of your product. If your team lacks someone with design skills, or you’re simply running short on time, use something like Twitter Bootstrap or jQuery UI to add polish to your application.

Again, these are just my personal observations and opinions. Adhering to them in no way guarantees you a win.

Testing Async Functions with Jasmine

Sam and I have been using Jasmine as our Javascript testing library for our Software Engineering project, mainly because I’ve used QUnit in the past and wanted to try something new. It also makes the tests in our group seem “uniform” in that Jasmine tests look a lot like RSpec, which is what the Rails team on our project is using. Our application has a RESTful Rails backend and a frontend written in Backbone.

We have a function in our Backbone view that calls a function on our Backbone model. The Backbone model talks to our RESTful backend and based on the response triggers the “success” or “error” callback functions provided by the view. Here is the pattern that we used to test this function:


describe('A Jasmine test', function() {
it('should be able to test async functions', function() {
// variable to check if our function call was successful
var wasSuccessful = null;
// Dummy Backbone model
var testModel = new TestModel();
runs(function() {
// someAsyncFunction is an async function that calls the 'success' function in our params object
// if the call was successful. It calls the 'error' function in our params object otherwise.
testModel.someAsyncFunction({
data: 'test',
success: function() {
wasSuccessful = true;
},
error: function() {
wasSuccessful = false;
}
});
});
var timeout = 2000;
waitsFor(function() {
return wasSuccessful;
}, 'Async call should be a success', timeout);
runs(function() {
expect(wasSuccessful).toBe(true);
});
});
});

view raw

test.js

hosted with ❤ by GitHub