Distributed Configuration

Software development is about embracing change. One way to achieve it is by keeping the software as configurable as possible. That brings the problem of storing the configuration somewhere. In a microservices based architecture, the stateless application often find themselves the need for a centralized, highly available store for such configurations. We will discuss some approaches to achieve the same.

For the impatient, if you are looking for code and examples, you may skip ahead to the end of this article.

The Beginnings

Let’s say you were the engineer at Awesome Software Inc. who was responsible for developing a feature for the sales team. The feature is to send out mails to your customers periodically for a survey on how your product was doing. You hit google and land on Amazon SES and see the code is right there. You quietly do Ctrl+C and Ctrl+V. Then you thought to be a bit creative. You notice that the FROM is hard coded.

static final String FROM = "sender@example.com";

The software engineer in you did not like the hard coding. You decide to make it configurable. You pull up your application.properties and rename it to application.yml, since all the cool kids use yaml. You gave yourself a pat on the back and even announced in the team meeting that you made it configurable. The sales team was happy, the product manager was happy and your microservice was deployed on production.

app:
  mail:
    from: no-reply@awesomesoftware.inc

Your company Awesome Software was an over night success and lets say it became one of the A in FAANG and suddenly you need to scale your app. Without breaking a sweat you configure multiple nodes of your app in your Kubernetes cluster and just saved the day and you head off to bar to celebrate.

Meanwhile Product Manager noticed that the mails that were sent were going to spam since it had a no-reply and they need to change it(right now). Guess who will be called in the middle of the night!

Can I write yet another app ?

You can create an app with a DB whose sole responsibility is to store centralized application configuration. Now you have created two problems. The app has to be Highly available and Consistent. Not to mention the wiring required in the consumer app to read the config. But wait this problem has been solved and resolved. Lets take some of the existing approaches.

Zookeeper

Zookeeper has been around since dark ages. We know it is a centralized data storage that is HA . Almost every app in the Hadoop ecosystem uses Zookeeper. You may already be using it, passively.

Spring Cloud provides a way of auto configuring to connect to a Zookeeper cluster. Now in this specific context, we need to read the application.yml from the Zookeeper. You need to include org.springframework.cloud:spring-cloud-starter-zookeeper-config as a dependency. Pay special attention to the configs especially enabled flag. Also note that this can be kept in application.yml, unless you have a special need for keeping this in bootstrap.yml.

Spring Cloud enhances Zookeeper to support solution to other problems such as Service Discovery, Client side load balancing etc. Also since you might be already having a zookeeper cluster, it might be a good place to start.

Trade offs

• Zookeeper configuration with spring does not support @RefreshScope. That means you will need to restart your services every time a config changes.

Consul

Hashicorp has been solving some interesting problems in distributed computing and Consul is one of their products. Consul solves multiple problem but specifically the feature that could help us is the Key Value store. It can be treated like a Hashmap in the cloud, if you will.

Consul also provides solution to distributed problems such as Service Discovery and Service Mesh. Consul runs are agent and server. For use cases such as Service Mesh, an agent runs along with application a.k.a Sidecar pattern

Both Consul and Zookeeper provides ACL to secure access.

Spring Cloud Config

If you are married to Spring, you could also check the spring cloud config. Interestingly the configurations are stored in git. Enabling the config server reads from the repo and boots up the application. I personally haven’t tried the option, but worth mentioning here when we talk about the options.

Hands on

For an example in code, I am choosing Consul. I admire Spring Boot Framework Architects in keeping the configurations more or less the same and it enables developers to switch between different implementations pretty quickly.

Code is shared at https://github.com/JayadevRajan/spring-consul

bootstrap.yml

Notice that the application name should not have space.

• Format is ‘yaml’
• profile separator is used to form the url
• data-key is also part of the url in the end.

spring:
  application:
    name: identity-service #no space allowed
  cloud:
    consul:
      enabled: true   #global flag
      host: http://127.0.0.1
      discovery:
        enabled: true #for service discovery
        instance-id: identity-service
        healthCheckPath: /api/health
        register: false
      config:
        failFast: false
        enabled: true
        prefix: microservice/identity-service
        defaultContext: identity-service
        data-key: data
        profileSeparator: '::'
        format: yaml
        watch:
          enabled: true #run time watch


---  #---treats it as a different config file, yet bundled in the same. 
spring:
  cloud:
    consul:
      config:
        failFast: true
        enabled: true
        prefix: microservice/identity-service
        defaultContext: identity-service

  config:
    activate:
      on-profile: dev

Property Bean

The property which we want to capture is expressed as a Java Bean. I used Lombok to simplify some boiler plate code. Other that that it’s a POJO.

@RefreshScope
@NoArgsConstructor
@Component
@Data
@ConfigurationProperties(prefix = "identify-app")
public class UserConfiguration {

    @Data
    public static class User{
        private int sessionTimeout;
        private boolean sendNotification;
    }

    private User user;
}

application-{profile}.yml

The application property which translates into the bean like above.

identify-app:
  user:
    session-timeout: 60
    send-notification: false

Consul Server

If you are testing locally, you may want to run a consul server. We can use a docker container to pull an image. Yes docker is the new maven. And I am lazy like that.

version: '3'

services:

  consul-agent-1: &consul-agent
    image: consul:latest
    networks:
... 

Thats it!! Bring it on.

docker-compose up

When the spring boot application starts up, it first reads the bootstrap.yml and sees that consul is enabled. It then tries to read the properties from the consul end point.

http://127.0.0.1:8500/v1/kv/microservice/identity-service/identity-service::dev/data

You may post the yaml

Make sure you put the properties using your favourite Rest API Editor or using the consul UI Key Value.

There are two KVs corresponding to the application.yml and application-{profile}.yml

Each of those are put under the Data Key.

Go ahead and change session-timeout. The application refreshes itself. No restart needed. Bliss!

It’s worth mentioning that all properties are not refreshable without restart. Spring creates a lot of beans when it starts up. The beans configs— if they read any property from the config — reads it only once and does not read again. So this might need a restart.

The full example can be found at https://github.com/JayadevRajan/spring-consul