Open Source Models Are Making AI Engineering Accessible to All

A couple of weeks ago, I had the opportunity to speak at Hannah Foxwell’s fantastic "AI for the Rest of Us" conference. My talk focused on a topic that has been brewing in my mind over years of working in DevOps, MLOps, and now GenAI: the barriers to AI engineering are rapidly crumbling. With today's tools, if you can navigate an IDE and push some YAML to Git, you're already well-qualified to dive into AI.

A Familiar Pattern

Having witnessed the evolution from DevOps (ClusterHQ) to MLOps (Dotscience), and now deeply involved in GenAI with HelixML, I've noticed a recurring pattern: complex tools get simpler, workflows become standardized, and what once seemed like rocket science transforms into just another engineering task. This democratization of AI is making it accessible to a broader range of developers.

Breaking Down the AI Application

Building an AI application can be broken down into six key components:

1. Models

   • At their core, models are mathematical functions that transform inputs (like text) into outputs (also often text). While they can be complex, they fundamentally turn words into numbers and back again.

2. Prompts

   • Prompts are instructions you give to the model in plain English. Sometimes, you need to be very explicit-think of it like talking to a toddler. The more precise your prompt, the better the output.

3. Knowledge

   • This is your AI’s personal knowledge base. It can include documents, websites, or any other data sources that you want the model to learn from. Curating this knowledge base is crucial for specialized applications.

4. Integrations

   • Connecting your AI to real business systems through APIs is where things get interesting. Whether it's integrating with a CRM, an e-commerce platform, or any other system, these integrations make AI truly powerful in practical scenarios.

5. Tests

   • Testing is non-negotiable for any production application, including AI. You need to ensure your model behaves as expected and doesn't go sideways when deployed. Tools like pytest can be adapted for testing AI applications.

6. Deployment

   • Running your AI on a server involves versioning the entire application, often using YAML files. For example, you can use Flux to manage deployment to Kubernetes (K8s) and HelixML. This ensures that your application is consistently deployed and managed in a production environment.

Tools Making It Easier

The tools available today are making AI engineering more accessible than ever:

• IDEs: Modern IDEs like VSCode have extensive support for AI development, including plugins for model training, debugging, and deployment.
• Version Control: Git remains the backbone of software development. With YAML files, you can version control your entire application, from models to deployments.
• CI/CD Pipelines: Continuous integration and continuous deployment (CI/CD) tools like Jenkins and GitHub Actions are essential for automating the build, test, and deployment processes.
• Open Source Models: Platforms like Hugging Face provide a vast library of pre-trained models that can be fine-tuned for specific tasks. This reduces the barrier to entry significantly.

Practical Example: Deploying an AI Model with Flux

Let’s take a brief look at how you might use Flux to deploy an AI model to Kubernetes:

1. Define Your Application in YAML

   • Create a kustomization.yaml file that defines your application, including all the necessary resources (e.g., deployments, services).

2. Set Up Flux

   • Install Flux on your Kubernetes cluster and configure it to watch your Git repository.

3. Automate Deployment

   • Use Flux’s GitOps approach to automatically deploy changes from your Git repository to your Kubernetes cluster.

Here’s a simplified example of what your kustomization.yaml might look like:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

And the corresponding deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-app
spec:
  replicas: 3
  selector:
    match