Microservice oriented architecture has obvious advantages over a monolith setup especially in the modern world: clear responsibility and ownership model, easier to develop and maintain, easier to scale and regionalize, smaller blast radius and more. Snap Engineering has adopted the SOA architecture with the utilization of Envoy and Service Mesh. This effort brought flexibility and efficiency to our teams, while providing clear responsibility and separation for our organizational setup.
However, with microservices, things are distributed across multiple fleets, databases, regions and cloud providers. Building a reliable and efficient system that can communicate between services, maintain application states and gracefully deal with outages has become a hard problem. In order to properly orchestrate services and track application states, technologies including queues and databases are often used. Engineers end up spending lots of time implementing state tracking functionality and writing error handling code to keep their system intact and resilient to component failures.
Take Snap’s Asynchronous Ads Reporting as an example. This system is used to help advertisers generate analytics reports asynchronously based on their requirements and notify them by emails when ready.
This involves 3 microservices for the simplicity of the view:
Data Fetching Service: Fetch raw data based on the query, do pre processing and generate formatted files, processing could run for minutes based on the query.
Report Generation Service: Read in generated files, run through data analysis jobs and produce a downloadable report and put it into another user accessible file system.
Notification Service: Send out email notification to the advertiser that requested the report.
It looks easy enough to build this system at first glance, we just need an async API and in the background, call each microservice one by one until the advertiser receives the notification.
But the reality of building such a system and making it run reliably is not as simple. In the real world, anything can happen. Any one of the microservices could go down, or there could be network issues at any point of time. For example, if a report is generated and the notification service goes down, then the advertiser will not be able to receive the notification and the processing in previous steps would have been wasted. In order to solve it, we usually would add retries, timeouts or other error handling logic to make sure we have a reliable system that a short outage won’t void all the previous states.
In other cases, if advertisers send out the request, the API receives it and starts the job, but when a response is being sent back, their network goes down. Then they wouldn’t know whether their request has been received or not, so typically the same request will be resent and thus starting a duplicate job. This will cause issues if the system is not idempotent or jobs are extremely costly to run. To solve this problem, it requires the application status to be tracked, so we will know if there's already a job running for the same request.
We can see from the above discussion that building a reliable “simple” microservices system is not simple. It often requires engineers to think through edge cases and write boilerplate error handling logic in any places where systems are integrated. Beyond error handling, in certain systems like payment or billing systems if we are not able to track the application states, it could mean money loss or other worse consequences.
In Snap, we must have reliable services so that our advertisers can have consistent high quality experiences. At the same time, we want our engineering to be more efficient and able to iterate fast while developing reliable services. We will need a central orchestration solution to help orchestrate our services and track system states across services. Luckily we have found a solution to this, and we will give a brief introduction in the next section and show how Snap Engineering is embracing it.
Temporal is an open source project which helps with service/workflow orchestration problems. Using its own words: