Building a distributed application from scratch is a really good way to learn a lot about operating systems, networking and, system design in general. In this tutorial, we are going to build a very simple distributed application that can be really useful for beginners getting into distributed system design.
Pre-requisites:
- Must have built at least a basic backend application before in any language
- Basic understanding of the OSI model of networking
Table of Contents:
- Introduction
- Our Own Protocol
- Writing a Basic TCP Server
- Handling Each Connection
- Applying the Operations
- Testing Our App
- Storing to disk
- Maintaining a Log of Operations
- Making Our App Distributed
What are we going to build?
Our app, let’s just call it Mathlog for now, will allow users to send basic mathematical operations like addition, subtraction, multiplication and division and it will keep track of all the operations till now. At any point, users can also get the final result of applying all those operations.
The app itself is really simple, intentionally to keep the focus more on learning about how it operates in a distributed way.
The log of operations on the right shows all the operations that have happened till that point. It will be stored in the disk and allow us to reconstruct the value if our app ever goes down.
Our own Protocol:
Our first step would be building a basic TCP server. The reason is that our app is not going to use HTTP. Instead, we are going to design our own distributed layer 7 protocol (OSI model). And the underlying layer 4 protocol that it will use will be TCP.
Of course, we could have used something like HTTP or gRPC, but what fun would that be?
This is a very used pattern in popular protocols like Redis and HTTP. The Layer 7 protocol is just some rules over the TCP socket.
Our protocol right now is going to be extremely simple. For now, it will just accept 2 kinds of messages from the TCP socket,
operation <operator> <number>\nresult\n
The first kind allows us to perform a new mathematical operation. Each word in the message will be separated by a space. The second kind is to retrieve the final result value from the server.
The end of every message is denoted by a \n. The server then responds with an error or response string with a \n at the end.
We will parse these texts from the TCP socket and then perform the required action. Don’t worry, we will add more stuff and refine our protocol as we go.
Writing a basic TCP server:
I will use Golang to build the app. You can of course use any language to follow along.
This just starts a basic TCP server and handles each new connection in a separate goroutine.
Handling each connection:
The handleConnection function will be called whenever a new client connects to our application. It needs to read the message from the TCP socket and take the required action based on that.
Here is the code,
This function parses the input into separate strings. It then uses a switch case to find out what downstream operation it needs to perform. i.e., a mathematical operation or returning the result. There are also checks if the user sends an invalid message.
There could be errors during conn.Write(). I haven’t handled those here but you can do it if you want.
Applying the operations:
Alright, our app is starting to take shape. Now, we will define the global variable called result that will store the final result of all operations. We will also create the function which applies each operation to the global variable.
The function receives the next 2 words in the input namely the operator and operand, eg: + 5, and applies it to the final result global variable. There is a switch case that determines what operation to perform based on the input.
Testing our app:
Now that our app is functional, we can go about testing it. We cannot just simply use our browser or Postman to test this since we are not using the HTTP protocol.
For Linux/Mac users, you can use telnet to test this app. For Windows users, you can also use telnet, but it doesn’t come pre-installed. You have to manually install telnet first.
To make our lives easier, I have written a small test client using Python that you can use to test this. You can find it HERE.
Simply run the go server and then run this Python script in a new terminal. The rest will be self-explanatory.
If everything is working, we can proceed to the next step which is storing the operation logs.
Maintaining a log of operations:
As we already discussed, we need to keep track of all the operations that are happening so that we can recover them in order if our app goes down.
We will do this by creating an operations.mlog file and storing each operation in a new line.
We have created 2 more global variables, offset, and operations which will keep track of the last applied operation’s offset and a list of all operations till now respectively. Both of these values will be really important later when we make our app distributed.
We then write the operation in the next line of the file.
If you run the test using the Python script above, you will see that a new .mlog file is created in the current directory and it will contain a history of all the operations like,
+ 5
- 5
+ 10Note that, storing each operation in a new line is not a good idea in the real world. In an actual real-world application, the operations.mlog file structure would be greatly optimized to reduce the number of disk reads and disk seeks with a lot of compression applied.
You can read more about how Redis does it here if you are interested. But we will go ahead with this approach since this is pretty simple to understand.
Reading from the file:
We have one more thing left to do. When we stop and start our app again, it should read the operations.mlog file from the disk and re-apply those operations.
We will create a new function which does exactly that
It just reads the file and applies all the operations in the file iteratively. It also stores the operations in an in-memory slice. This will come in handy later.
Making our app distributed:
Now that all the basic functionalities of the app are done, we have a solid foundation to dive deeper into how to make the app distributed in the next part.
To give a sneak peek, we will have a master node and multiple slave nodes. Whenever an operation comes, the master will relay that to the slave nodes too. The slave nodes won’t be able to handle new operations, but they will be able to serve the final result.
Until next time!
