IDP - Why, When and How?
.avif)
To begin with I would like to talk about the acronym IDP.
A few years ago, when I started working full time in the “infrastructure area”, we were talking about INternal DEveloper PPlatform and this refers to the infrastructure and set of tools that automate and facilitate the development, deployment and operation of services and underlying resources, has a focus on automation and integration of all the layers that a product development team needs, including components of CI/CD, IaC, container orchestration, observability, etc.
Over the years we started talking about INternal DEveloper Portal, which is an evolution of an Internal Developer Platform.
According to Gartner, “By 2026, 80% of large software engineering organizations will establish platform engineering teams as internal providers of reusable services, components and tools for application delivery. Platform engineering will ultimately solve the core problem of cooperation between software developers and operators.”
With INternal DEveloper POrtal we began to take a product approach when it comes to the platform, it may or may not include a UI and it facilitates self-service tasks with operating flows designed as a product and based on the internal development and security needs that the company needs.
In order not to get boring, I would like you to stick with the following concepts:
- Self-service
- Standardization
- Security
- Reduce cognitive load
- Platform as a product
Why an IDP?
After the very brief introduction, let's talk about why.
Akua, as is public knowledge, is “The acquiring processing platform for all payment rails in Latam” and developing services in this context requires special care in security without losing speed in development and deployment and having autonomy for the operation/management of associated services and resources.
With this context, we formed the Platform team in Akua to develop the IDP, with an aggressive but motivating challenge ahead, to develop the IDP from the ground up so that the product development team can deploy and manage microservices and associated resources, all without losing sight of the concepts I mentioned above, Self-service, Standardization, Security, Reducing cognitive load, Platform as a product.
Was it a whim? no, I've known Juanjo for a long time and we know from shared experience that having an IDP in a company that needs to be PCI compliant, following the highest levels of security with high availability and scalability is a very valuable benefit.
At Akua we could well have created everything by hand, without pursuing standards, but it's bread for today and hunger for tomorrow, let's think that for some reason the industry created the concept of IaC (Infrastructure as Code) https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac), not just on a whim, is a need that came about almost naturally because of the implications and challenges in managing infrastructure.
When an IDP?
I anticipated something but we did it from minute zero and it was challenging, very challenging to think about:
- What IaC tools are we going to use?
- What are we going to use for CI/CD?
- How do we make sure that our IDP doesn't turn into a monster and is simple and simple to manage and scale?
- Where are Akua's services going to run?
- How are we safe by default?
And a question may appear hanging in the air, why from such an early stage?
Akua needs to have a speed of development Standardized and Safe and maintain it over time, and that without an IDP is very difficult to achieve or cannot be achieved, you could “get out fast” without having an IDP, but I assure you that sooner rather than later everything becomes chaos, for the product and for the management and maintenance of the infrastructure and software.
How?
Let's address the last point, the how.
There are no recipes, sorry, but if you came looking for one, this post is not for you 😂
What I am going to tell you is what our IDP offers to date.
Peeeero before talking about what it offers I will tell you what we use to offer what we offer:
- As an IaC tool, we use Pulumi with Typescript.
- For CI/CD we use Gitlab with private runners to interact with our cloud provider.
- Speaking of cloud provider, we use AWS (Amazon Web Services)
- For the configuration of applications we use Helm and developed a chart that all microservices use.
- Where do our applications run? in Kubernetes
- For Observability we use NewRelic
Ok luispi, so what?
Earlier I talked about cognitive load and the desire for our IDP to remain simple and maintainable over time.
Well, to meet that objective, we developed an internal library (as I mentioned with Pulumi and Typescript) that contains the business and security logic for the registration, removal and modification of platform resources.
To prevent our team of product developers from having to learn an IaC tool in addition to the fact that from the Platform team we want to have the freedom to change course if we need it, management is done from Port (https://www.getport.io/), where each team responsible for the application can create, update and delete resources with a simple form, don't you believe me? I'll leave you a gif
When the application is synchronized, under the hood we run a private Gitlab runner on aws and using our IaC tool (Pulumi) we validate the state of what we want to deploy with what exists in the cloud.
Beneath this “simple flow” there are many hours of design and development that we had and have with the Platform team, as of the date of writing this publication, our IDP can manage:
- As I mentioned microservices in Kubernetes and this includes: deployment, service account, service, ingress, stateful set, cron job, HPA, light cache (?)
- SQS
- SNS
- S3
- DynamoDB
- RDS Aurora
- IAM Role, Security Group, Secret and ECR for the application/microservice
- Multiple environments
- Single and multi-tenant
All these resources taking into account cost optimization and safety.
Before continuing I want to talk about how of the last two items on the list, Multiple environments, Single and Multi Tenant.
Multiple environments
When we talk about multiple environments, we mean that an application can run in different environments in the life cycle until it reaches production.
The approach and solution we gave from our IDP in Akua is to abstract the creation of resources environment by environment, we went to a more abstract concept where you create or model what resources the service will need to solve a business problem and then be able to select in which environment you want to synchronize the previously defined resources, being able to choose between a test and a productive environment.
Here's a short gif:
Single and multi-tenants
Our Platform and our portal are designed and developed to support single/multi-tenant, and not only at the product software level, but also across the entire technological stack that our platform has, and by this I mean:
- Application workloads
- Resources from our cloud provider
- Observability (tracing, logs, metrics)
- etc
And this functionality was developed with respect for some of the mantras that must be followed when creating an IPD Reduce cognitive load, why?. Our product technology team can configure if the application is going to run on 1... n tenants and when deploying it can select environment and tenant, the rest is responsible for our IDP to understand how, when and where to configure all the underlying resources 😎.
Conclusions
Throughout the publication, we try to address in a more or less abstract way everything that surrounds an IDP, from what the acronyms mean to how a Platform team and an IDP can impact your company's business, in our case in Akua 🏼.
Some things I would like to summarize what an IDP can provide you with are:
💸 Cost reduction: By optimizing infrastructure and automating processes, organizations can achieve significant savings in operating and maintenance costs.
⚙️ Greater agility and speed of delivery: The implementation of a robust and flexible platform accelerates the time to market of new applications and functionalities, providing a competitive advantage in a constantly evolving market.
📈 Improved scalability and availability: The microservice-based architecture and container orchestration make it possible to efficiently scale platform resources, ensuring high availability and performance.
🛡 Better security and regulatory compliance: The application of security practices and the use of advanced tools make it possible to protect sensitive data and comply with security and privacy standards.
Are the decisions one-sided? Of course not, to build an IDP if you don't listen to the needs of “customers” everything can be done uphill.
There is still a long way to go, but we are very proud of what we built, are building and are going to build in our IDP.
Until next time 👋🏼