Sign in Subscribe

Lessons Learned from Building Koiner

In this post, I share my lessons learned from my experience building Koiner, including experiments conducted and choices made along the way.

Lessons Learned from Building Koiner

Building Koiner V1 was an invaluable learning experience. It provided a playground for freely experimenting with new software patterns and tools without the pressure of tight deadlines. The insights I gained were instrumental in shaping the B2B SaaS product I was architecting during my day job; I couldn’t have achieved the same level of quality without my work on Koiner. Although juggling the development of both projects simultaneously was very challenging, it ultimately enriched my experience.

Throughout this process, I also gained a deeper understanding of the Koinos architecture, which has greatly benefited my dApp development efforts. I now have a comprehensive grasp of the details of the Koinos blockchain and greater insights into the activities occurring on-chain.

Koiner V1

After dedicating significant effort since early 2022, I successfully launched the first version of Koiner shortly after the Koinos Mainnet went live in November 2022. This milestone marked an important step forward in my journey.

Koiner V1 - Launched in November 2022

Koiner mobile version

In the first quarter of 2023, I released the mobile version of Koiner, which has since been enhanced with additional features over the following months. This continuous improvement reflects my commitment to providing users with an increasingly robust and user-friendly experience as we build towards a more decentralized future.

Koiner mobile version - launched in 2023

V1 Stack

During development, I experimented with various tools and design patterns. This resulted in the following stack:

Backend

Cloud

Front-end

  • VueJS
  • Quasar UI framework
  • Urql GraphQL client
  • jsonforms: JSON schema form builder as foundation for table views
  • Custom modules build on top of jsonforms for Quasar:

Key Learnings from Koiner V1

While overengineering was intentional to explore a more robust architecture for enterprise applications, Koiner didn’t truly require that level of complexity. Here are my key takeaways from building V1:

Premature Scaling: While splitting data into separate databases and microservices and stitching them together with GraphQL Mesh was a good learning experience, it ultimately introduced unnecessary complexity. This complexity not only made the system harder to manage but also slowed down development and deployment cycles.

Overengineering: Although microservices, hexagonal architecture, and CQRS are powerful design patterns, they were overkill for an application with minimal business logic. Simplifying the architecture would have allowed for quicker iterations and a more streamlined development process without sacrificing functionality.

Deployment Challenges: Although Kustomize proved useful for Kubernetes deployments, managing them manually became cumbersome and highly inefficient, especially in a microservices environment where automation and scalability are key. Moving towards a fully automated deployment pipeline could significantly improve efficiency.

Repetitive code: The Koiner GraphQL API built in NestJS involved too much repetitive code, even with reusable modules. It highlighted the need for better optimization to reduce repetition.

Reliability: Debugging indexing errors due to the event-driven setup was not easy to do. Improving the decoupling of indexers and enhancing error logging and handling would lead to greater reliability and ease of troubleshooting.

Monorepo Advantages: A monorepo setup with NX significantly enhanced the integration between modules and microservices. This centralized approach not only simplified dependency management but also streamlined my development process, allowing me to work more efficiently across different components of the project.

Experience in Kubernetes: As someone relatively new to Kubernetes, building Koiner V1 and running Koinos on this platform has provided me with invaluable experience. This hands-on experience has deepened my understanding of Kubernetes best practices and its capabilities in managing complex applications.

Koiner V2

Throughout working on Koiner, I embraced the spirit of experimentation, using each new tool and methodology as an opportunity to refine my approach and improve the overall product. By actively testing different software patterns and integrations, I gained valuable insights that have significantly influenced Koiner’s development.

I’ve dedicated six months to developing Koiner V2 in 2023. In Q1 2024, I shifted my focus for a couple of months to the development of a new dApp. However, I'm now back to working on Koiner, aiming to release this significant update before returning to development of the new dApp.

Goals for Koiner V2

Before diving into Koiner V2, I reflected on the lessons learned from Koiner V1 to define clear goals. These objectives aim to enhance efficiency, adaptability, and user experience of the platform.

Code Efficiency: Minimize boilerplate and repetitive code to improve maintainability and readability. The intention was to simplify the architecture, moving away from layered and hexagonal architecture, while utilizing open-source tools for automatic API generation.

Go Fully Cloud Native & Cloud Agnostic: Transition to a fully cloud-native infrastructure that allows for seamless deployment across different cloud providers. Implement an auto-deploy feature for the Kubernetes cluster with a single click, streamlining the deployment process for ease of use and efficiency. This will not only improve scalability but also reduce downtime during updates.

Leverage Open-Source Tools: Reduce reliance on proprietary software to prevent vendor lock-in, ensuring greater flexibility and control over the technology stack. By utilizing open-source solutions, I aim to foster a more adaptable and community-driven ecosystem that can evolve with changing needs.

Integrations: Enhance Koiner's capabilities through integrations with tools like Supabase, capitalizing on the strengths of Postgres and simplifying the GraphQL schema. This will improve performance and make it easier to extend functionalities within the platform.

Improve Reliability: Develop decoupled indexers that can operate independently and in parallel, increasing resilience and performance. Implement improved error logging and handling mechanisms to facilitate quicker debugging and enhance overall system reliability.

Extendable Architecture: Design Koiner to be easily extendable, allowing for the addition of new modules and integrations with decentralized application (dApp) data without significant rework. This focus on extensibility will make it possible to innovate and expand the Koiner platform efficiently.

Experiments

During the development of the new backend for V2, I experimented with five different tools to enhance the product’s infrastructure and optimize its performance.

Experiment 1: N8N. My first experiment involved integrating N8N for workflow automation, aiming to develop open-source N8N indexing modules for Koinos with the idea in mind of building a decentralized data network similar to TheGraph and SubQuery. However, the complexity of N8N proved to be too high, making debugging challenging and creating unnecessary overhead. While I admire N8N as a product, I found that working directly with NestJS was far more efficient for my workflow. Unfortunately, I had to set aside my vision of building a decentralized data network for the time being.

Experiment 2: Supabase. In my search for simplicity and better tooling for building GraphQL APIs I’ve stumbled upon Supabase, which is an open-source Firebase. After experimenting with it, also for the 1-week challenge we did with a group of developers for Koinos, I’ve decided to fully commit to Postgres by utilizing Supabase for auto-generated APIs and real-time data handling based on the Postgres database schemas. This has streamlined development and enhanced my overall developer experience a lot!

Experiment 3: Pulumi. I started experimenting with Pulumi to automate infrastructure management and streamline deployment processes, integrating it more smoothly into my development workflow. Pulumi enabled me to write the Infrastructure as Code (IaC) in TypeScript, aligning perfectly with the rest of my tech stack. My goal was to build a one-click automated deployment pipeline for the entire Kubernetes cluster. This includes setting up a StackGres database cluster, a Koinos node, and the Koiner backend powered by Supabase, Kong gateway, along with the VueJS frontend. The process wasn’t easy though—self-hosting Supabase on Kubernetes with Kong and StackGres proved to be an incredibly challenging task to get up and running.

Experiment 4: Stackgres. In my pursuit of building a fully cloud-native infrastructure, I explored hosting PostgreSQL database clusters on Kubernetes using Stackgres. This enables me to fully leverage Kubernetes for database management, enhancing scalability, reliability, and operational efficiency. By integrating StackGres with Pulumi for infrastructure as code (IaC), I will be able to automate key database operations such as backups, failovers, and scaling, ensuring seamless management and deployment directly from my IaC configurations.

Experiment 5: TimeScaleDB. While exploring Supabase integrations, I came across TimeScaleDB. Though I'm still working on the integration with Koiner, this will soon power the upcoming Koiner charts.

New Backend Stack for Koiner V2

The new Koiner stack includes some integrations I’m really excited about. It’s amazing to see how mature open-source solutions are reshaping the software landscape and giving developers more leverage!

Here’s a look at the key components:

  • Functional Programming Style: Adopted a flat structure that leverages functional programming principles.
  • NX.dev: Leveraging NX for a powerful monorepo setup that simplifies dependency management across multiple modules.
  • Supabase: Utilizing Supabase as a backend solution to provide seamless integration with PostgreSQL while enabling real-time capabilities.
  • GraphQL Mesh Gateway: Implementing a GraphQL Mesh Gateway to unify various data sources and create a flexible API layer.
  • Kubernetes: Deploying the application on Kubernetes for robust orchestration and management of containerized microservices.
  • Kubernetes Dashboard: Using the Kubernetes Dashboard for enhanced visibility and control over the cluster's operations.
  • Cert-Manager + DNSimple Webhook: Automating SSL certificate management through Cert-Manager in conjunction with DNSimple for seamless domain validation.
  • Pulumi for Continuous Integration (CI): Implementing CI workflows with Pulumi to automate infrastructure provisioning and application deployment.
  • Kong Gateway: Employing Kong as an API gateway to manage and secure API traffic efficiently.
  • Leveraging PostgreSQL + Extensions: Maximizing PostgreSQL’s capabilities through its powerful extensions to enhance data processing and performance.
  • RabbitMQ + Postgres Triggers: Utilizing triggers to manage AMQP queues effectively, enabling responsive and event-driven architecture.
  • Stackgres: Hosting PostgreSQL databases in Kubernetes using Stackgres for enhanced scalability and operational efficiency.
  • TimescaleDB: Integrating TimescaleDB for time-series data management, allowing for time-based queries.

Koiner V2 is launching soon! 🚀In my upcoming posts, I will expand on the roadmap for Koiner and Koiner Pro, detailing the exciting developments ahead.

Stay tuned Koiners!