Multicloud Deployment as a Service

I put myself on the bleeding edge of a new cross-tenant cloud service and used what I learned to set the tone at a VP level while owning outcomes and keeping high standards for security and maintainability.

Managed Storage in the Cloud

These days, public cloud providers like AWS and Azure can be tied-in to corporate intranets directly, operating traffic through a secure VPN connection. This allows for seamless integration of cloud services with on-premises infrastructure, enabling organizations to leverage the benefits of cloud computing with direct access to their data. However, latency and bandwidth fees quickly make the case for uploading and co-locating data with the compute services. That kind of data mobility is easy when it is built-in to the storage devices themselves.

Enter Dell APEX Navigator, the direct public cloud analogue for purchasing a managed Dell storage system on-premises- wherein a service technician delivers and installs the appliance in a data center and subsequently manages upgrades and maintenance schedules. Instead of technicians and data centers, we built automations into a cloud-native cross-tenant SaaS platform.

Automated Deployment Platform

At the outset, I was responsible for a small exploratory team of engineers tasked with documenting the installation process for existing Dell storage offerings over multiple series of interviews with product managers. This documentation was crucial for designing the automation workflows for deployment and management of these systems in the cloud.

I went above my expected responsibilities to lead additional discussions with directors, product managers, and architects regarding the integration of Navigator with other Dell management solutions, and the technical limitations of the differing levels of integration. My presentations and breakdown of the problem space were used at the vice-president level to decide on the strategic future of the new platform.

I undertook learning a working level of knowledge in multiple possible options for automating our deployment and management operations:

• Ansible
• CloudFormation
• Temporal
• Terraform

In order to make my recommendations, I interviewed product managers of two different internal technologies we were considering for the telemetry and management stack, I learned a broad set of AWS and Azure concepts, and used that knowledge to develop network diagrams for each option. I presented my evaluations of these options to the architecture and development team, and provided recommendations for my preferred technology combinations based on ease-of-use and business requirements.

Secure Multitenancy

The implications of a central SaaS management platform that can reach into customer's cloud environments and make modifications at both the infrastructure and application layers is awe-inspiring. It was imperative to ensure sufficient security controls were in place to prevent an unauthorized actor operating within the Dell network from accessing credentials and keys that the automated processes used as part of their legitimate operations. Additionally, the same permissions that customers granted to our platform in their cloud environment had to be restricted in order to ensure that one customer could not see data from any other customer.

My responsibility in the code was to implement the security controls which postured a defense in depth approach in order to ensure that customers' credentials were isolated from one another. One aspect was to ensure credentials were encrypted at-rest to prevent exfiltration from raw disk access. One possible solution was to encrypt credentials before storing them in our database, but management of encryption keys was a concern, which rendered this approach moot. The chosen solution was to utilize a dedicated keystore to securely store credentials, whose access was restricted and injected at runtime only to services that required it. Input validation was performed on all access tokens to ensure a request could only operate in the cloud environment for a validated customer.

Security also means maintainability. Across multiple microservices that the team was developing, duplicate code for common patterns was emerging. I made the case to the team that we needed a common library in order to simplify code changes and make the necessary abstractions for common behavior across clouds, and with their buy-in I led that effort. I additionally documented our database and keystore schema, and integrated Liquibase into our code, allowing for the quick understanding and evolution of our persisted data.

Excellence in Execution

I am proud of the team I worked with on this project. Not only were we responsible for the keystone services in the environment, granting and gating access to customer cloud environments, but through our commitment to high standards, zero outages or incidents occurred as a result of our services. The diagrams I created at the beginning remained the definitive model of the project even beyond the first two years of implementation.

On this high note, and sensing changes coming, I took an opportunity to further refine my security skills by applying to the recent opening of Security Champion for the org, and was accepted.