profile

Mark C Allen

Everyday DevOps - Using Feature Flags to Build a Better AI


Everyday DevOps

Using Feature Flags to Build a Better AI

In my latest post for DevCycle, I dive into how we tackled the tricky job of rolling out new AI models at Helix without risking a big public flop. Instead of launching updates blindly, we used feature flags to test new models with a small group of users first, either using email or device IDs. This let us spot issues early, tweak things quickly, and ensure Helix’s image recognition stayed sharp and accurate. Honestly, feature flags made the whole process way smoother!

Tips & Tricks for the week

I recently encountered a production bug that I couldn't reproduce locally, which was complicated by internal microservices, running memory-intensive machine learning models on GPUs, that weren't exposed through the public API.

To solve this, I used Codezero, where I could access the kubernetes cluster running the microservice from my local machine using just a DNS name.

Installed the Codezero agent using brew.

PrismJS Highlighted Shell Command
brew install c6o/tap/codezero-app

Logged into my Codezero account, created a new teamspace and installed it into my kubernetes cluster.

PrismJS Highlighted Shell Command
helm repo add --force-update codezero https://charts.codezero.io && \
helm install --create-namespace --namespace=codezero \
--set space.name='testone' --set org.id='my org id' \
--set org.apikey='this is a secret' \
--set hub.url='https://hub.codezero.io' codezero codezero/codezero

From the Codezero app, I can consume the service I need by DNS name and curl returns what I need.

PrismJS Highlighted Shell Command
curl http://nval-ltf-model-web.nval-app:8000/
{
  "collections": "http://nval-ltf-model-web.nval-app:8000/collections/",
  "greenlists": "http://nval-ltf-model-web.nval-app:8000/greenlists/"
}

For more, see my blog post: Debugging with Codezero

What this week has in store

Adding AI tools to my chatbot has been great; I can give it functions to perform specific tasks, such as querying my database. This week, I'm going to integrate MCP servers into the mix and see how they work in conjunction with the existing tools.

See you next week!

Mark C Allen

1:1 Chat

Need a Different Perspective

Do you have a problem with your release process? Has Kubernetes got you down? Do you need an outsider's perspective on what's holding up your deployments? If you have 25 minutes, book a time with me.

Mark C Allen

For DevOps engineers. Ideas, concepts, tips & tricks based on my day-to-day experience.

Share this page