Chotot (chotot.vn) which belongs to 701Searchs, is known as one of the most famous company that achieved c2c (Customer to Customer) model in selling stuffs. It’s been 5 years since first launch in 2012, Chotot now has about 10 milion users/months, 1 milion user’s accesses/day, 600 milion views/month and 3000 successful transactions each day.
To be capable of serving this kind of very big users, Chotot has their own tech team that helping developing and scaling product.
They use microservice – an architectural style that structures an application as collection of loosely coupled services, which implement business capabilities. This microservice architect also enables the continous delivery/deployment.
Using API gateway has some benefits:
– Insulates the clients from how the application is partitioned into microservices
– Insulates the clients from the problem of determining the locations of service instances
– Provides the optimal API for each client
– Reduces the number of requests/roundtrips. For example, the API gateway enables clients to retrieve data from multiple services with a single round-trip. Fewer requests also means less overhead and improves the user experience. An API gateway is essential for mobile applications.
– Simplifies the client by moving logic for calling multiple services from the client to API gateway
– Translates from a “standard” public web-friendly API protocol to whatever protocols are used internally
One more thing very important of API gateway brings: It produces the metadata that very helpful for monitoring.
There are plenty of tools to do this. Chotot uses Tyk. Tyk is powerful API gateway enables you to control who accesses your API, when and where and how they access it. Tyk also has recorded detail analytics on how users interact with your API and when things go wrong.
Tyk stores metric data in Redis, and if developers want to collect metric in another backend store, they have to use tyk-pump which is able to move analytics data in Redis to any supported backend. Tyk-pump at this moment only supports serveral data store, and they are not what Chotot needs. So Chotot have to write their own tyk-pump version. They want to build tyk-pump which is able to aggreate metric data to InfluxDB and Statd then visualize them with Grafana as wombo combo.
Unfortunately, Chotot’s stacks based on C/C++ and there is nobody has Go experience before. Chotot needs a hand.
That brings Chotot to the Dwarves Foundation. Tyk is written in Go, which is the language that Dwarves Foundation is highly fluent. We have been 3 years in Go development and also the founder of Golang Vietnam. We proudly say that we have strong background to fulfil Chotot’s mission.
We helped Chotot develop Tyk-pump.
– Develop the tyk-pump tool that can be able to manipulate the metric data from Redis and push into output backend, then display to visualizer.
– Contrubition to the tyk-pump Repo (via Pull Request)
– Setting up all tyk and relevant tools via docker that can be used in local of every engineer.
– Analyzing how InfluxDB and statd manage their own data store.
– Discuss and finalize the schema of metric data needed.
– Develop by forking the tyk-pump and starting coding.
When contributing to an open source repository like Tyk, code quality has to be at high level. Coding style, naming variables and comments have to follow convention. Source code was reviewed carefully by a huge amount of watchers in github.
InfluxDB and statd has their own different format of data store. One is time series database and one is network daemon. We have to handle all cases to make sure input/output as expected.
There are a ton of metric information, but we only need some of them. Categorizing and filtering were what we did to choose helpful ones for building effective dashboard visualization.
Tyk brings many relevant supported tools like dashboard, tyk-cli, tyk-cluster. To use Tyk, setup a runnable environment is a must.
– Go as primary language.
– Node.js for looking up (statsd built on it).
– Docker/Docker-compose perfectly plays a role of development tools.
– Telegraf which is metric agent.
– InfluxDB a time-series data.Grafana for data visualizer.