At this year I went to QCon São Paulo, I saw many cool things and many new technologies. At the conference, some speakers talked about how we can make resilient systems. Kolton Andrus (slides) has spoken about of Netflix Toolset, he mentioned Spinnaker, Chaos Kong and Hystrix. I like so much of Hystrix, because the concept behind this mechanism sounds very simple and can be useful in many situations. In this post, I will try to talk about it.
Oh, what is Hystrix? The Netflix describes Hystrix as “[…] a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.”. Behind all advantages that it gives to us, this library has a lot of patterns, however in this post I will write about only one, called of Circuit-Breaker.
Let’s think about circuit-breaker. Let me see… Oh Circuit-Breaker pattern works like a home circuit-breaker, Obvious. But, how it works? You can see it here. In IT world, this mechanism can be applied to some cases to prevent errors or decrease latency. Imagine that you have a call to an external service, in this scenario many things can fail, you can get network error, some server can be out, or something unexpected can occurs. If you want to make your system resilient, you have to assume that your system can fail.
To avoid that problem, you can respond a default response or with cache. Using a circuit-breaker, when something goes wrong N times, the circuit-breaker stops to call failing method and responds only with a fallback response. After a time period, the circuit-breaker try to call a method again, it can works or not.
In this post, I will try to show some examples with the same app that I created at the previous post. You can download it here(please use version 0.0.2). I added Docker only to turn easier the environment setup, but you can use only ruby(>=2.2) and rails(>=5) to run.
I will show only one the stoplight gem, if you search at RubyToolbox site you can found many similar solutions like circuitbox or resilient, through.
Ok, how it works? Well, we will see it together along this post. So, it’s easier, at least for me, understand with examples.
Ps: To run with docker you can use ‘docker-compose up’ and after that ‘docker exec -it <image-name/id> bash’ to enter in bash. In the end we have to type ‘rails c’.
First of all, we have to add a new gem in this project, so the following line solves it.
Ps: Everytime that we add a new gem in Gemfile, we need to run ‘docker-compose build’
At the project folder, we need to type the command below to start our server.
The command above build the image from Dockerfile, starts all dependencies that we added in docker-compose and executes a command specified in docker-compose (in our case ‘bundle exec puma’).
After that, we have to enter in rails console, so type ‘docker exec -it -it bash" and 'rails c'.
The gem Stoplight is very illustrative, it works like a stoplight (Mr. Obvious attacks again! kkk). Please look at the diagram below.
Ps: I tried to extract the flux above from Stoplight code.
Now, let’s play with this gem and, besides that we will try to use it inside our simple ‘blog’ code.
The code above shows how Stoplight works, to create a light we must set an identifier, at this case ‘example’. We have to pass a code block that will be executed, in this case a lambda called ‘function’.
When we have a light, we can run it and the result will be evaluated and also get the light color.
Ok, if code works everytime when called this gem is unuseful, but if we have a code that can fail, we can use it to avoid some error scenarios. Let’s take a look when something goes wrong.
The code above is similar than previous one, we added a function that always fail(at least since we define Blargh class), we also added some customizations like ‘with_fallback’ and ‘with_cool_of_time’.
The function ‘with_fallback’ defines a custom fallback, this code will be evaluated when something goes wrong.
The function ‘with_cool_of_time’ defines a timer to turn to yellow and then the next call will define if light will be green or red again. Don’t be afraid with this, Stoplight has a nice documentation and you can see many examples at project page.
When we have a huge application with multiple instances this code don’t work well because Stoplight uses memory by default to store all light statuses. This gem already has a solution to it, we can use Redis to share lights statuses between the app instances. So, we have to create a file at config/initializers and add the following code into it.
Now, we will use Stoplight in article#show route to responds with a fallback when something goes wrong. So, we have to add the possibility to fail then a code that fail can be useful. We also have to create a command. You can use all code directly on Article.find, but I will use a command to keep same pattern.
Create a file into lib/commands/article folder, this command will be responsible to search an article and to encapsulate a circuit breaker behavior. Add the following lines into created file.
After that, we need to register a new command inside a Blog::Container.
And finally change the code responsible to find Article inside a ArticlesController to call our new Command.
Run ‘docker-compose up’ or ‘rails s’, create a article and see if everything works as expected.
Now we are ready to add the stoplight inside our project. First of all, we have to add the fail possibility. We will use the following code to determine if it will fail or no.
After that, we need to add to code responsible responds by fallback. In our case, when something goes wrong fallback will responds with a fake Article.
And add the code responsible to fail or search Article in database.
When you search for a inexistent Article, the method #find will throw a error.
But you don’t need to responds with a fallback in this case. To around this problem we have to add the code below.
And finally the code to get together all things.
All code will look like this:
Now, we can run our server again, and access a Article#show route. When you get a error the server will respond with a fallback, like the image below.
If you check your log, you will see something like this:
We can add a custom notifier to something that you want when light changes the status from red to green or vice versa.
Bonus: We can add a custom notifier in our project, so take a look at the following example:
We only need to keep the same method signature of notify and add the custom notifier in stoplight configuration. To configure stoplight, we need to add the line below inside config/initializers/stoplight.rb.
When we call the route again until an error occurs. The log will look like this:
Stoplight has a panel to displays all light statuses and to do some actions like lock light status. If you are interested in it, please take a look at the project page.
You can download all code showed from here. That’s all. If you have any doubt please post it in commentaries section.