Ænix.io

Last Thursday in Almaty (Kazakhstan), a room packed with engineers spent three hours migrating VMs off VMware. Not watching a demo. Actually doing it. Timur Tukaev aka @tym83 led the workshop: convert OVA to QCOW2, push images to S3, and boot VMs on Cozystack. Then came the ultimate time-sink: untangling the networking assumptions VMware had quietly baked into those machines over the years. Throw in PostgreSQL and Kafka, wire them together, and keep tweaking the Java app until it finally came alive. Later that day, Timur took the stage again to talk about the future of Kubernetes. Short version: the exciting work has shifted from the cluster itself to what's being built on top of it. The full talk is going up on the Cloud Native Kazakhstan YouTube channel, alongside sessions on hiding the control plane, Kubescape, cross-account Istio on AWS, and a case study on swapping Ceph for Longhorn. Attendees get to keep their Cozystack tenant for 30 days. No overnight resets — this is a real setup to stress-test properly. Plus, it doubles as a stepping stone toward Cozystack administrator certification. Next up: Bishkek (Aug 17), Tashkent (Aug 20), and Astana (Aug 24). Europe in October, the US in November, and a global online session in early September. Links and details here: aenix.io/workshops/vmware-to-cozystack One final takeaway: the migration is the easy part. The real bottleneck is that vCenter and Veeam live in people's fingers, not in the docs. It’s 15 years of muscle memory. Nobody budgets time for unlearning — and that’s exactly what kills a pilot. P.S. Huge thanks to the organizers, the team at QOSI (Qazaq Open Source Initiative) and everyone who came out! Appreciate the invite, your trust, and the incredible vibes. P.P.S. A massive shout-out to Ivan Okhotnikov aka @xor_dev — a rockstar engineer on the Ænix team and an all-around awesome human. Thanks for helping pre-bake the workshop code, pulling those all-nighters, and providing real-time tech support (Google Meets and all) for our attendees while the workshop was live.

We’ve open-sourced another tool we built: keycloak-kms-proxy. Here’s the problem it solves. Keycloak stores user PII, including email addresses, names and custom attributes, as plaintext in its database. Under GDPR, this is personal data and needs proper protection at rest. Storage-level encryption or PostgreSQL-side encryption only gets you so far. The encryption key still lives close to the data, often inside the same PostgreSQL environment. Key rotation is also difficult to handle without downtime. There are unofficial plugins that address this, such as Keycloak PII Data Encryption Provider. But they come with their own trade-offs: tight coupling to specific Keycloak versions, Java-level integration and a static encryption key passed through an environment variable. We took a different approach. keycloak-kms-proxy is a transparent proxy that operates at the PostgreSQL wire-protocol level. It sits between Keycloak and PostgreSQL. Keycloak sees a regular Postgres database, while the proxy encrypts selected columns on write and decrypts them on read. The design follows the same basic principles as Kubernetes KMSv2. Every encrypted value is self-describing ($KKP$...), which means the proxy can safely work with a partially encrypted database while the backfill utility gradually processes existing records. KEK rotation in Vault is fully transparent. The key version is stored alongside each wrapped DEK, so existing data remains decryptable without migrations or downtime. In Cozystack, this is already available as an optional feature in the system Keycloak package. Enable encryption in the Helm chart and the proxy is deployed automatically. Keycloak is then reconfigured to connect through it, with no additional setup required.

📣 This August, the Ænix team is bringing a series of hands-on open source private cloud workshops to four cities across Central Asia: Almaty, Bishkek, Tashkent and Astana. The workshop focuses on moving beyond traditional virtualization and exploring cloud-native infrastructure for teams evaluating alternatives to VMware. ❓ We’ll cover: • how a modern cloud-native private cloud is built; • how VMware components map to an open source stack; • where Kubernetes, KubeVirt, LINSTOR, Cilium and GitOps fit in; • how to migrate virtual machines and introduce self-service infrastructure; • when migration makes sense and when keeping your existing VMware environment is the better choice. Cozystack will be used as a hands-on example of an integrated platform for testing these concepts and migration scenarios in practice. 🗣️ The workshops will be led by Timur Tukaev, COO at Ænix, the team behind Cozystack. 📍 Almaty — August 13, 10:00–14:00 📍 Bishkek — August 17, 14:00–18:00 📍 Tashkent — August 20, 15:00–19:00 📍 Astana — August 24, 14:00–18:30 All workshops are in person and follow the same program. 🔗 Choose your city and register via the link: https://u.to/VpqnIg 🙌 Know someone in Almaty, Bishkek, Tashkent or Astana who might be interested? If you have partners, colleagues or friends working with cloud infrastructure, virtualization or Kubernetes in these cities, please share this post with them.

🚀The Cozystack team has open-sourced Blockstor, a control plane for block storage in Kubernetes: LVM and ZFS as backends, replication over DRBD, and a LINSTOR-compatible REST API. The project lives in the cozystack organization and is developed as part of Cozystack, a platform accepted into the CNCF Sandbox. The license is Apache 2.0. 🔥The main thing that makes it worth a look: it is not a fork, and it is not a wrapper. Blockstor is written from scratch in Go, but it speaks the same REST API as LINSTOR — so all the client tooling you already run keeps working without a single change: the linstor CLI, linstor-csi, piraeus-operator, and the golinstor library. Read the full article in Cozystack blog 👇 https://cozystack.io/blog/2026/08/blockstor-linstor-compatible-storage-for-kubernetes/

Catch you at the 70th Cozystack community meeting today! When: Thursday at 18:00-19:00 (Central European Time, CET) Where: Zoom Link to join in: https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5 This meeting is free and open to everyone! Come and join us! Minutes: https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/edit?tab=t.0#heading=h.ygb3chatmgn2 📢 Agenda and Notes: - Marketplace service in details 🎤 Open Floor: -Test flakiness and platform stability: catching component conflicts earlier, isolating tests from external factors - Discussion of versioning and release process - Release frequency, semver vs calver and - Backport strategy discussion - Design proposal review: open proposals on GitHub https://github.com/cozystack/community/pulls - Viewing backup process events in the web dashboard 🗓️Do not miss Cozystack community meetings: https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2 https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack?view=month 💡Have a feature idea or design proposal for Cozystack? Submit your Pull Request here: https://github.com/cozystack/community/pulls 🚀

Hi everyone, if you'd like to add anything to tomorrow's Cozystack community meeting agenda, feel free to reach out to @vsop78 or simply drop your ideas in the comments When: Tomorrow at 18:00-19:00 (Central European Time, CET) Where: Zoom Link to join in: https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5 This meeting is free and open to everyone! Come and join us!

New Cozystack Highlights: Persistent Public IPs. Today IP addresses in Kubernetes are ephemeral. You create a LoadBalancer service, you get an IP, you delete the service, the IP is gone. Recreate it and you get a different one. For anyone running real workloads on public IPs, that churn is a problem, and Cozystack users have been asking to fix it for a while. The prototype shown at the last community meeting borrows the model everyone already knows from storage: just like a PersistentVolumeClaim binds to a PersistentVolume through a StorageClass, an IPAddressClaim binds to an IPAddress through an IPAddressClass. What that unlocks: — Reserve an IP before attaching it to anything, then bind it to a service via annotation — Delete the service, the IP stays reserved and comes back to the same claim — Move one reserved IP between services without losing it — Contention policies: keep an IP alive after its claim is deleted, or release it back to the pool — A reserved IP can't be stolen by another service The design is deliberately platform-agnostic. The core controller handles the claim-to-address logic; a separate driver talks to the backend, MetalLB today, but the same pattern fits AWS Elastic IPs or anything else, much like a CSI driver for storage. Still a prototype, but the direction is set: IP addresses become a first-class, reservable resource in Cozystack. 🎥 Full demo from the community meeting: https://www.youtube.com/watch?v=cQJyA25Hf6U #Cozystack #Kubernetes #Networking #CloudNative #OpenSource

😎Your infrastructure is already running. 💰But can your customers actually buy cloud services from it? On August 19, Andrei Kvapil, creator of Cozystack, founder of AEnix, will walk through the full path from servers to a sellable public cloud: • VMs, managed Kubernetes, databases and S3 storage • Billing, usage metering and hosting panels • WHMCS and custom integrations • Migration from VMware, OpenStack or a custom stack • A realistic two-week path to your first customers Free live webinar. Questions submitted during registration get priority in the live Q&A. August 19 · 10:00 CET · Online Register even if you cannot attend live. We will send you the recording. Registration is now open!👇 https://aenix.io/webinars/launch-public-cloud/

🚀Cozystack v1.6.0 is now available. The release was published on July 22, 2026, and includes all fixes previously shipped in the v1.5.1, v1.5.2, and v1.5.3 patch releases. ⚙️This release changes several important parts of the platform. Tenant Kubernetes workers now run Talos Linux instead of Ubuntu, tenants can enable OIDC authentication for Kubernetes and Grafana, and a new SecurityGroup API provides a safer interface for managing application network policies. 🧩Cozystack 1.6 also introduces hierarchical resource quotas, completes the etcd-operator v1alpha2 migration, expands Keycloak security and backup options, and makes application deletion consistently reclaim its storage. 👀The upgrade surface is larger than usual. 🧿Operators should review the upgrade section before applying the release. Read the full release information in our social media👇 https://x.com/aenix_io/status/2082778204167868599 or https://www.linkedin.com/pulse/cozystack-16-talos-workers-tenant-sso-security-groups-hierarchical-jkznf

New on the Kubernetes blog: Andrei Kvapil /@kvaps and Timofei Larkin/@lllamnyp (Ænix) break down how the controller-runtime cache actually works — and why your controller isn't crashing the API server even at hundreds of reconciles per second. If you write controllers in Go, this builds the mental model most people are missing: reads come from a local cache, writes go to the API server, and almost every production surprise (memory blowups, stale reads, silent O(n) scans) follows from that one fact. https://kubernetes.io/blog/2026/07/29/controller-runtime-cache-explained/

🎥 Recordings from CozySummit Virtual 2026 What If Every Cozystack Change Became a Commit? | Simon Koudijs, configbutler.ai 00:00 - Speaker intro and Config Butler background 00:47 - First impressions of Cozy Stack 01:19 - Resource history problem: who, when, and why 01:50 - Options for handling unmanaged platform changes 02:18 - Kubernetes audit files and audit webhooks 02:48 - Classic GitOps and its interface tradeoffs 03:41 - Live demo setup on a local Cozy Stack cluster 04:25 - Audience quiz submission demo 05:37 - Quiz submissions as Kubernetes custom resources 06:24 - Capturing platform changes as YAML in Git 06:41 - Classic GitOps vs reverse GitOps 07:13 - Reverse GitOps principles 08:05 - GitOps Reverser implementation overview 08:56 - Watch rules, Git targets, and Git providers 10:03 - Live Git commits from audience submissions 10:55 - Git author vs committer attribution 11:16 - Capturing the reason behind changes 11:29 - Coffee ordering demo intro 12:01 - Application artifacts and configuration state 12:32 - Coffee ordering and admin configuration demo 13:27 - Handling concurrent edits 14:22 - Completing the who, when, and why audit trail 14:50 - Why Cozy Stack fits reverse GitOps well 15:32 - Cozy Stack dashboard identity forwarding demo 16:21 - Capturing high-level resources and Flux changes 17:14 - Watch vs admission webhooks vs audit webhooks 17:43 - Benefits: auditability, GitOps on-ramp, and GUI usage 18:37 - Setup complexity and Kubernetes API limitations 19:10 - Secrets encryption and avoiding inline passwords 19:51 - Feedback request and future ideas 20:27 - Key takeaway: Kubernetes API as record of intent 20:54 - Links and feedback form 21:39 - Q&A begins 21:46 - Why not just snapshot or back up ETCD 22:22 - Server-side apply and field owners question 22:39 - YAML audit trail and resource history 22:52 - Git history length and conflict handling 24:00 - Conflict resolution strategy 24:31 - API interface for viewing Git changes 24:48 - Handling Git outages with Redis cache Enjoy watching and share your thoughts: https://youtu.be/dkyjOq-I7Zs?si=qIwL5IiIU8ZxPArq

Focus on today: how Andrei Kvapil @kvaps, the founder of Ænix and the creator of Cozystack, an open source cloud platform, built aeman, a daily planning board for engineers on top of GitHub Projects. Read the full article in Ænix blog 👇👇👇 https://blog.aenix.io/focus-on-today-how-we-built-aeman-a-daily-planning-board-for-engineers-on-top-of-github-projects-c59da4451b8b

🎥 Cozystack community meeting 2026-23-07 https://youtu.be/vA8ilRMQz0k Cozystack: Free Cloud Platform based on Kubernetes | CNCF Sandbox Project Minutes: https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/ 📢 Agenda and Notes: - Cozystack v1.6.o release - Cozystack preparation for CNCF Incubation - Public IPs as First-Class Resources in Cozystack by Timofei Larkin (lllamnyp) https://github.com/cozystack/community/pull/35 🎤 Open Floor: - How to make Kubernetes cheaper on resources: where node overhead actually lives by Malformed C, maintainer of Periapsis - Marketplace service — discussion 🖖 Share your enhancement and architecture proposals by opening a PR at https://github.com/cozystack/community/pulls 🫵 Join the community: Telegram group t.me/cozystack Slack group https://kubernetes.slack.com/archives/C06L3CPRVN1 (Get invite at https://slack.kubernetes.io) Cozystack resources: https://cozystack.io https://cozystack.io/docs/get-started https://cozystack.io/blog https://github.com/cozystack/cozystack 📅 Subscribe to our community meetings calendar: https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack 🗓️ Add meetings to your calendar: https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2 https://youtu.be/vA8ilRMQz0k

Catch you at the Cozystack community meeting today! When: Thursday at 18:00-19:00 CET Where: Zoom Link to join in: https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5 This meeting is free and open to everyone! Come and join us! Minutes: https://docs.google.com/document/d/1YObSKsQ3Gzi1gSLPlUp-PbSYEeVkoyzY20M_hXKVhq4/edit?tab=t.0#heading=h.ygb3chatmgn2 📢 Agenda and Notes: - Cozystack v1.6.0 release - Public IPs as First-Class Resources in Cozystack by Timofei Larkin (lllamnyp) https://github.com/cozystack/community/pull/35 🎤 Open Floor: - How to make Kubernetes cheaper on resources: where node overhead actually lives by Malformed C, maintainer of Periapsis - Test flakiness and platform stability: catching component conflicts earlier, isolating tests from external factors - Discussion of versioning and release process - Release frequency, semver vs calver and - Backport strategy discussion - Design proposal review: open proposals on GitHub - Viewing backup process events in the web dashboard - Marketplace service — discussion 🗓️Do not miss Cozystack community meetings: https://webcal.prod.itx.linuxfoundation.org/lfx/lfsixxnFWxbvsyEuC2 https://zoom-lfx.platform.linuxfoundation.org/meetings/cozystack?view=month 💡Have a feature idea or design proposal for Cozystack? Submit your Pull Request here: https://github.com/cozystack/community/pulls 🚀

Hi everyone, if you'd like to add anything to tomorrow's Cozystack community meeting agenda, feel free to reach out to @vsop78 or simply drop your ideas in the comments When: Tomorrow at 18:00-19:00 CET Where: Zoom Link to join in: https://zoom-lfx.platform.linuxfoundation.org/meeting/93845795591?password=a263fc60-ea72-41c8-84c8-a00d683ecee5 This meeting is free and open to everyone! Come and join us!

🧿 How LINSTOR and DRBD Keep Your Data Alive When a Server Dies Every bare-metal operator's nightmare: a disk fails at 3 AM. If your storage is node-local (a local-path or hostPath volume pinned to one node), you're restoring from backup and praying it's recent. Ceph gives you replication but demands 3+ dedicated storage nodes, a PhD in CRUSH maps, and constant tuning. Most teams either over-invest in storage infrastructure or under-invest and pay the price in outages. Cozystack uses LINSTOR + DRBD for replicated block storage. DRBD replicates at the block device level — below the filesystem, below the database, below everything. It's been battle-tested in Linux HA setups for over two decades. LINSTOR adds the Kubernetes-native orchestration layer on top. ⚙ How It Works 1. DRBD (Distributed Replicated Block Device) synchronously mirrors block devices between nodes. Every write to Node A is immediately written to Node B before the application gets an acknowledgment. 2. LINSTOR manages DRBD resources as Kubernetes StorageClasses. When a PVC requests storage, LINSTOR creates a DRBD volume, selects replica nodes, and handles the lifecycle. 3. CSI Driver exposes LINSTOR volumes as standard Kubernetes PersistentVolumes. Result: any pod or VM that uses a replicated StorageClass gets synchronous block-level replication across nodes — transparently. 🎛 Setting Up Storage Step 1 — Prepare disks: # Set up LINSTOR CLI alias alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor' # Check what LINSTOR sees linstor node list linstor physical-storage list If disks don't appear, they likely have leftover metadata. Clean them: # Using talm (installation covered in POST 9) # WARNING: never wipe the OS install disk (machine.install.disk) — only data disks. talm -f nodes/node1.yaml wipe disk nvme0n1 nvme1n1 Step 2 — Create storage pools (ZFS): linstor physical-storage create-device-pool \ zfs node1 \ /dev/nvme0n1 /dev/nvme1n1 \ --pool-name data \ --storage-pool data Repeat for each node. Step 3 — Verify: linstor storage-pool list You should see a data pool on each node with available capacity. Step 4 — Use it: Any application that specifies storageClass: replicated now gets DRBD-replicated volumes (three replicas by default). PostgreSQL, MongoDB, VM disks — all of them. # In any application config: values: storageClass: replicated size: 50Gi 🛟 What happens when a node fails: 1. DRBD detects the node is gone 2. The volume remains available on the surviving replica node 3. Kubernetes reschedules the pod to a node that has the replica 4. When the failed node comes back, DRBD automatically resyncs No manual intervention. No restore from backup. No data loss. 📘 Documentation - Disk Preparation: https://cozystack.io/docs/storage/disk-preparation/ - Disk Encryption: https://cozystack.io/docs/storage/disk-encryption/ - DRBD Tuning: https://cozystack.io/docs/storage/drbd-tuning/ - NFS (RWX): https://cozystack.io/docs/storage/nfs/ Join the community: Telegram group t.me/cozystack Slack group https://kubernetes.slack.com/archives/C06L3CPRVN1 (Get invite at https://slack.kubernetes.io) Cozystack resources: https://cozystack.io https://cozystack.io/docs/get-started https://cozystack.io/blog https://github.com/cozystack/cozystack #Cozystack #LINSTOR #DRBD #Kubernetes #CloudNative #BareMetal #PrivateCloud #Storage

🎥 Recordings from CozySummit Virtual 2026 Treating Kubernetes as a Linux Distro: APT-Style Packaging with FluxCD | Andrei Kvapil, CEO, AENIX 00:00 - What Cozystack is and its managed services focus 01:14 - What makes a service truly managed 02:23 - Platform stack foundations: storage, networking, virtualization, and monitoring 03:34 - Cozystack as a tested Kubernetes distribution 04:16 - Monorepo structure and package organization 05:06 - System, apps, and extra charts 06:28 - Helm charts as packages 07:08 - Delivering Cozystack through Flux and OCI artifacts 08:42 - Platform chart and configuration storage 09:25 - Previous bundle-based delivery model 10:18 - Customizing charts and delivering ad hoc fixes 10:52 - Local development workflow challenges 11:32 - Cozy chart tool and faster local testing 12:33 - Show, diff, and apply commands 13:36 - Simple installation and air-gapped delivery goals 14:20 - Cilium, kube-proxy, and the chicken-and-egg problem 15:06 - Flux all-in-one and early bootstrap networking challenges 16:54 - Moving from in-cluster HTTP server to OCI artifacts 17:26 - Limitations of OCI-backed multi-chart Helm repositories 18:50 - Source Watcher and artifact composition 20:04 - Flux controller responsibilities and artifact generators 20:53 - Combining sources into external artifacts 21:45 - Making Cozystack reusable for different customers 22:23 - Package manager idea inspired by Alpine Linux 22:55 - Package sources, packages, and Helm releases 24:04 - Package variants and multi-chart packages 25:06 - Package dependencies and dependency checks 26:30 - Cozy package CLI commands 27:04 - Migration from bundles to packages 28:04 - Platform package variants like Ubuntu Desktop and net install 28:33 - Scaling Cozystack without feature bloat 29:41 - External repositories and vendor-distributed packages 30:35 - Evolution toward an ecosystem model 32:22 - Kubernetes as the new Linux and ecosystem-driven distributions 32:43 - Helpful links and documentation 33:32 - Q&A begins 33:39 - State management and disaster recovery over time 35:07 - MetalLB vs Cilium L2 announcements 36:26 - Secure Boot support 37:18 - Cozystack software development lifecycle 38:25 - Learning resources for Kubernetes and Cozystack Enjoy watching and share your thoughts: https://www.youtube.com/watch?v=vKRZR7gjU40

𝐀𝐫𝐞 𝐲𝐨𝐮 𝐮𝐬𝐢𝐧𝐠 𝐂𝐨𝐳𝐲𝐬𝐭𝐚𝐜𝐤 𝐢𝐧𝐝𝐞𝐩𝐞𝐧𝐝𝐞𝐧𝐭𝐥𝐲? As part of the CNCF Incubation review, the TOC is speaking with Cozystack adopters about their real-world experience. We are especially looking for teams and individual adopters using Cozystack outside a commercial engagement with Ænix. If that is you, 𝐩𝐥𝐞𝐚𝐬𝐞 𝐜𝐨𝐧𝐭𝐚𝐜𝐭 𝐓𝐢𝐦𝐮𝐫 𝐓𝐮𝐤𝐚𝐞𝐯 𝐄-𝐦𝐚𝐢𝐥: timur.tukaev@aenix.io 𝐓𝐞𝐥𝐞𝐠𝐫𝐚𝐦: @tym83 𝐋𝐢𝐧𝐤𝐞𝐝𝐢𝐧: https://www.linkedin.com/in/tym83/ 𝐨𝐫 𝐬𝐞𝐧𝐝 𝐮𝐬 𝐚 𝐝𝐢𝐫𝐞𝐜𝐭 𝐦𝐞𝐬𝐬𝐚𝐠𝐞 𝐡𝐞𝐫𝐞. 𝐓𝐡𝐢𝐬 𝐢𝐬 𝐧𝐨𝐭 𝐚 𝐬𝐚𝐥𝐞𝐬 𝐜𝐚𝐥𝐥. 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰𝐬 𝐜𝐚𝐧 𝐫𝐞𝐦𝐚𝐢𝐧 𝐜𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐭𝐢𝐚𝐥. #Cozystack #CNCF #Kubernetes #CloudNative #OpenSource #CNCFIncubation

Cozystack has started the process of moving from the CNCF Sandbox to CNCF Incubation ✏ We submitted our application in November 2025 and have continued updating it as the project evolved and new releases were published. The application is now under review. 💫 We believe Cozystack has reached the level of maturity expected from an Incubating project. The next steps include due diligence and a detailed review by experienced members of the CNCF Technical Oversight Committee (TOC). 🚀 Reaching this stage matters to us. It would not have happened without the people who use Cozystack, open issues, ask questions, test new releases and contribute to the project in many different ways. Thank you for being part of it! 🙌 We will share updates as the review progresses. You can follow the status of the Cozystack application here: https://github.com/cncf/toc/issues/1916 The full CNCF TOC project review board is available here: https://github.com/orgs/cncf/projects/27/views/9

🎥 Recordings from CozySummit Virtual 2026 Building a Multi-Cloud Service Mesh from the Ground Up with Kilo | Lu Servén Marín - SRE, AuthZed 00:00 - Service mesh and multicluster mesh basics 00:53 - Kubernetes networking as a built-in service mesh 01:43 - Why service mesh functionality should be abstracted 02:29 - Multicluster service mesh gap in Kubernetes 03:02 - Existing service mesh options and limitations 03:38 - What a multicluster service mesh provides 04:02 - Why users run multiple Kubernetes clusters 04:40 - Scope of the talk and what will be built 05:19 - Focus on connectivity and discovery 06:05 - Networking and discovery implementation plan 07:02 - Kubernetes pod-to-pod connectivity rules 08:14 - CNI and how pod networking works 09:50 - Basic CNI configuration with bridge and port map 11:53 - Pod communication through bridge networking 12:59 - Multi-node networking problem 15:01 - CNI providers and route programming 16:05 - Why Kilo is used for multicloud networking 17:46 - Kilo routing modes and overlays 19:13 - Cross-cluster routing with WireGuard 20:04 - Pod-to-service connectivity 20:18 - Kubernetes services and kube-proxy 22:00 - Cluster-to-cluster connectivity problem 23:12 - Solving multicluster networking with Kilo 24:03 - Local service discovery with DNS 25:13 - Remote service discovery with DNS 25:32 - Multicluster Services API overview 26:44 - Service Reflector project 27:21 - Demo setup with two clusters 27:53 - Installing Kilo for cluster networking 28:35 - Deploying a webcam service 29:16 - Testing webcam access with port forwarding 29:49 - Peering clusters with Kilo 30:22 - Accessing remote service through remote DNS 31:04 - Why local DNS discovery fails 31:43 - Creating a mirror service manually 32:22 - Successful local discovery through mirrored service 33:05 - Need for an automatic service reflection control loop 34:16 - GitHub link and project upstreaming 34:44 - Q&A begins 35:01 - NAT traversal and STUN service discussion 35:49 - WireGuard scaling limits 36:43 - Comparison with Kube-OVN and OVN interconnect Enjoy watching and share your thoughts: https://youtu.be/IDJhQ7vHisE?si=CmlAyqGK-dY8hB7W