A cloud file sync service, often referred to as a file synchronization or file syncing service, allows users to store and synchronize their files across multiple devices and the cloud. This technology is particularly useful for maintaining up-to-date copies of files across various devices and ensuring data availability from anywhere.
Popular Cloud File Sync Services:
- Dropbox
- Google Drive
- Microsoft OneDrive
- Apple iCloud Drive
Functional Requirements for the Service:
Our file-sharing service will usually have a desktop client. This client needs to sync the changes to the cloud whenever a file is created, edited, or deleted in the folder being synced.
A user must be able to see these changes on their other devices which also have the client installed.
There can be other requirements like authentication, payment for cloud storage, etc. which are worth exploring, but we will keep it short and simple and focus on the main problem of syncing the files.
Non-Functional Requirements:
- The file sync service should have low latency in syncing changes across devices and the cloud to provide a seamless user experience.
- The service should aim for high availability to ensure that users can access their files and sync changes at any time.
- The system should ensure eventual consistency across all devices and the cloud, meaning that changes made on one device will eventually propagate to others. (Eventual consistency is OK!)
Assumptions:
- We will have 100 Million users, Out of which 10 Million are active daily
- Each file will be on average about 1 GB
- Each user will have about 10 files
- There will be 100 edits per user per day
- The metadata like file name, size, last modified date, etc. per file is 10 KB.
A Naive Design:
In this basic approach when a change is made to a file in client 1, the entire file is uploaded to the sync service. The sync service then stores the metadata about the file like last modified time, file size, etc. in the Database and uploads the actual file in some blob storage like AWS S3.
Blob storage is a cloud service for storing and managing unstructured data like images, videos, and documents in a scalable and cost-effective manner.
Then, the sync service alerts the other clients for the same user about the file change. The other clients can then request the file through a separate Download service to update the file.
Bandwidth Usage:
The main issue in this is the bandwidth usage.
Sure, If it’s the first time a file is created, then the entire file will have to be uploaded by client 1 and downloaded by client 2. So if the file size is 1 GB, That will be 2 GB of data usage for the user.
But imagine if the entire file is uploaded and downloaded each time an edit is made. On average there will be 100 edits per day. So,
1 GB x 100 x 2 = 200 GB data transferred per day for a single user!
That’s a lot.
Chunking The Files:
The solution to the high bandwidth usage is breaking the files into chunks of smaller sizes and only syncing those chunks where the change was made.
For example, we could divide the file into 1 MB chunks, and whenever the content of a particular chunk is changed we can sync only those chunks.
1 MB x 100 x 2 = 200 MB data transferred per day for a single user.
This is way less than before.
Of course, we will need to store extra metadata for each chunk and handle the merging of chunks with the original file properly in our clients.
Scaling the API:
Now, we can focus on splitting the services and reducing the coupling between them and scaling up the services.
We will split the Sync service into a Upload Service and a Notification Service.
We will also add a couple of queues Upload Queue and a Notification Queue. This will de-couple all of our services from each other. Since eventual consistency is fine, this approach should be good enough.
For scaling each of the individual services, we can just create more instances of it and add a load balancer in front of them.
Scaling The Database:
The final part is scaling the database. Since, we will be storing the metadata in the database, and it will be updated each time a file chunk is changed, we can expect a lot of reads and writes to the database.
100 edits x 10 Millions users / 86400 seconds => 1150000 IOPS per second per day
To handle these operations, we will need to shard our database since the write operations are also high.
Sharding allows us to distribute our data across multiple database nodes. Sharding is more complex but is also more scalable. We split the data using a particular column name called as a shard key.
While choosing a shard key, there are two main things we need to focus on.
- High Cardinality — It should have a high number of values
- Low Frequency — The frequency of it appearing in queries should be low
For our purposes, we can shard by USER ID and sort by FILE ID column. This will allow us to quickly find all the files associated with a user.
Keep in mind that while this overview provides a comprehensive view of the design considerations, real-world implementation can involve further intricacies and optimizations based on evolving technologies and user needs.
Additional Explorations:
There are several additional aspects and considerations that readers can explore when designing a distributed cloud file sync service like OneDrive. Here are some key areas to delve into:
- Data Deduplication: Investigate methods for deduplicating data across users and versions. This can significantly reduce storage costs by storing unique data chunks.
- Data Compression: Look into data compression techniques to further optimize storage and bandwidth usage, especially for text-based files.
- Conflict Resolution: Explore strategies to handle conflicts when the same file is edited simultaneously on different devices. Implementing conflict resolution mechanisms can prevent data inconsistencies.
- Versioning: Consider providing versioning support, allowing users to access previous versions of their files. This is valuable for data recovery and collaboration scenarios.
