Stateless Orchestration

As the lead developer on a small engineering team, I gathered requirements and worked with adjacent development teams to design and implement a job orchestration service that could manage data mobility across a fleet of remote storage systems.

A single pane of glass

Imagine the system administrator at a large company. They have 20 terabyte-scale data storage systems to manage. They exist in edge locations, or co-located with office spaces across the globe, or in public cloud providers, and probably actually a mix of all of the above. The administrator needs to coordinate data backups and ensure system uptime across all of these systems, while also ensuring that international data privacy laws are respected. Every week they need to check in on each of these systems to ensure their backup process is functioning properly.

Instead of logging in to each system individually, what if there was a single place to dispatch work and observe results while the transport details stay out of the way?

My responsibilities on this project were to implement the centralized orchestrator for communicating with the remote storage systems. Each system exposed an engine that could connect directly to other engines and be commanded to push or pull files from other system’s engines. As part of the requirements generation process, I collected a basic summary of what needed to be built for the orchestrator:

• Stateless orchestration which rebuilds truth from each engine on startup.
• Report and retry job failures depending on the failure reason.
• Responsive to system downtime, ability to re-push commands when the system became available again.

Stateless Orchestration

While the orchestrator would be a huge quality-of-life improvement, local control would remain available. This meant that at any given time, the orchestrator could be out-of-date if it relied on any internal state about the jobs present on an engine. Live retrieval was the only possible way to ensure the orchestrator was consistent with the remote systems. I worked with the engine development team to build a hook / push mechanism to ensure the orchestrator received updates without needing to poll the engines.

I implemented a custom message broker using Python and asyncio to process messages in order to facilitate the multiple interacting systems over the network and handle system downtime. An off-the-shelf solution like Kafka was not possible because the engine could not be modified to support it. I used a long-lived tcp socket to receive hook events and push jobs to each engine.

• Asynchronous, message-based communication between storage systems.
• One socket, multiple channels, unrelated workloads do not block each other.
• Central management of remote system state and job lifecycle.

I decided to slice the orchestrator into two microservices. One service would handle the communication with the remote engines, while the other service would handle the orchestration of jobs. The thought process was that the brains of the orchestrator would always require full context and was therefore not scalable by design, but the number of remote storage devices could range up to hundreds, requiring a variable number of communication service instances.

Lead Developer

As the Lead Developer on a greenfield project, I was responsible for setting code quality standards, configuring build pipelines, and mentoring new teammates. A focus on linting and unit testing ensured clean code that could be continuously deployed for demonstrations to our leadership. One of our fresh hires was assigned to the team and I enjoyed being able to provide advice and demonstrate a high bar for documentation and secure development practices, on top of their strong ability to write efficient code. I used Jenkins to automate the build process, ensuring all quality control steps were enforced before producing a deployable artifact.

Why it failed, and what I learned

The orchestrator was never deployed to real customers. Sadly, the platform it was offered on had not attracted enough business interest to continue justifying development. When the platform was discontinued, my team was championed to incept development on APEX Navigator, which would offer similar data mobility operations as well as public cloud deployment functionalities. My focus on building a secure application also gave me an edge when I applied to be the team’s Security Champion.

If I were to build the orchestrator again, I would have chosen different boundaries when slicing the application implementation into multiple microservices. This was a case of premature optimization, and at the expected scale for the project I now know that a single service could have handled all of the workload and traffic without needing the overhead of communication between multiple microservices. The simpler implementation would have been more efficient and easier to maintain.

This experience taught me the importance of building systems that can adapt to changing environments and priorities. It also reinforced the value of working closely with other teams to ensure that systems are designed to work together seamlessly.