How to build your own agentic butler

Full walkthrough and open source repo available. Don't forget to star it!

Alfred is our family butler. I’ve been working on it since August 2024 when I first realized that agents can help me be more present.

A month ago I shared how Alfred is organizing all the random facts in my life which blew up this blog. Less than a month after launch and almost 3k people want to have their own agentic butlers.

Before you read further, if you like the Alfred project, please star the repo here: https://github.com/ssdavidai/alfred

If you are one of them, this post is for you. Below I will share a step by step walkthrough on how you can set up Alfred for yourself.

If it’s too technical, I can do it for you. Click here to apply.

My life, according to Alfred

Agents are a hot, complex things. Here’s how I see agents:

Everything runs across 6 layers. Agents need a computer, a runtime, data, memory and activities. Cloud computing has some of these locked down already:

Computer: It can be on a macOS. A Raspberry Pi. A random Linux server you rent from Digital Ocean or Hetzner. Or an agentic computer like Zo Computer. This is the Infrastructure Layer.

Runtime: Something needs to manage LLMs doing stuff.. OpenClaw is an agentic runtime. Zo Computer, Claude Code, Claude Cowork have their own agentic runtime. They live in the Agent Layer.

Data: The average person gets bombarded by about 100GB of data every day. Gmail, iMessage, Zoom calls, Slack. You need a way to connect and send these to your agent reliably. A Data Layer.

Memory: Data is a noisy collection of things in your life. Memory is a graph that connects the things and explains them with reason. This graph is the visual above. Agents need this to make sense of your messy human life. This is what I call the Semantic Layer.

Activities: An agent needs to be able to execute tasks too. Send emails, create research reports, check the weather, call your mom. Do stuff on a schedule, proactively or ad hoc when you ask it to. This is what gives the agent the Kinetic Layer.

Alfred has all six layers and all layers can be swapped. Every layer has different challenges, security risks and levels of complexity. Below I will show you the simplest, out of the box solution you can DIY for yourself today. Let’s set it up.

Step 1: Set up your agentic computer

Unless you’re happy playing sysadmin, you should use Zo Computer. Zo takes care of all the security, stability, hosting etc issues for you and provides what Pieter Levels called “vibe servering”.

You can get your Zo Computer set up for free, but if you want Alfred to be always available, you should sign up for a paid plan. You can use the DAVID discount code to get 15% off.

The homepage of your Zo. Open Source models are free until end Feb

This is the easiest way to do it and my walkthrough will assume you’re running on Zo. But you can do just the same on a macOS or on Linux as well.

Step 2: Configure your agent runtime

Zo comes with an agent runtime configured. My Alfred currently is running on OpenClaw but I’m working on a full migration to Zo.

Until then, let’s set up OpenClaw runtime on your agentic computer. Don’t worry i’ll keep it simple and stupid.

First off, create a new account on Tailscale. Once you’re logged in visit the Keys page by clicking here and create a new auth key.

Then open the Terminal page of your Zo and type in the following command (just copy and paste it).

npm install -g @ssdavidai/zoclaw && zoclaw init

This is an installer I wrote so you can deploy OpenClaw on a Zo Computer in under a minute.

As a first step it will ask for the auth key you just created. This will get saved securely as a Zo Secret.

It will install Tailscale on your Zo, set up a private VPN so Alfred is hidden from the public web. This process uses a custom install of a package I wrote specifically for deploying Tailscale on Zo. You can find the repository here.

Once it gets to the OpenClaw installer, you need to go through the interactive menu. I recommend to look through the Skills it offers and install what seems sensible (all safe) and install the Hooks. Don’t forget to add an API key for any LLM provider you want your agent to use.

Once done, Zo will offer you to launch the TUI. This is the terminal-based chat. Hit y and voila, you’re talking to your own OpenClaw instance on a secure Zo Computer.

Step 3: Installing Alfred

Having the agent runtime is just the beginning. Now we need to install Alfred. This will create the Vault and launch Alfred’s Workers. Alfred’s Workers are the little helpers that will constantly keep building and maintaining your knowledge base.

This includes the Curator, the Janitor, the Distiller and the Surveyor.

The Curator is watching the inbox folder on your Zo Computer and if there’s a new file in there, it picks it up, reads the contents, extracts the relevant things and how they relate to each other and create a few new entries in the Vault.

It’s turning a messy meeting transcript with a potential client into People, Projects, Organizations, Decisions, Assumptions, etc.. all linked together.

The Janitor keeps looking through the Vault periodically to fix anything that may be broken. It runs both code and LLM calls to do that.

The Distiller is turning the Vault from a memory of things to a memory of reason. It reads the Vault periodically and creates Synthesis and other types of learnings. If you commit to a meeting next Thursday but you are going to be out of office, the Vault Distiller will pick that up and store it as a contradiction that Alfred will read.

The Surveyor groups the things in the Vault together that are similar and connects them. It uses a few ML libraries (HBSDCAN and Leiden algorithm) and LLM calls for labelling.

Install Instructions

All you need to do is copypaste this command:

pip install alfred-vault && alfred quickstart

After a quick install, this is what you should see

Once the install is completed, the quickstart will begin. For a simple install just:

1. Hit enter (the vault will be found in your Zo Files as a folder)

2. Hit y, then enter (this will scaffold Alfred’s basic vault, see below)

3. Hit 3, then enter (We’re using OpenClaw. I haven’t yet tested the native Zo or the Claude Code connection)

4. Hit y, then enter (This will enable the Surveyor. Only do this if you’re on a paid plan, because this will download a bunch of things on your Zo and you need some compute for that. Pro plan should be enough.)

5. If you enable the Surveyor, you will need to provide an OpenRouter API key (you can get one for free).

If you’re on the free plan only and just trying it out, I suggest you skip the Surveyor on Zo.

Once it completed, it will ask you if you want to launch the daemons. Hit y, then enter. This will fire up Alfred’s Workers (with or without the Surveyor).

If you want to look at what they’re doing, you can type in alfred tui that will launch the terminal dashboard where you can watch what Alfred is doing live.

Step 4: Build your Vault

Go to Files on your Zo and find the vault folder. In there you will find an inbox folder.

Upload any kind of files in there. The Curator will pick it up, process it into the vault and then move the file to the processed folder.

That’s it.

Okay but what is Alfred good for?

Alfred helps me be more present.

Now you have a secure agentic computer with Alfred installed on it. The work is not done but your self-healing knowledge graph is not being built and maintained. I have a few custom integrations I’m using daily to help me be more present, less forgetful and in short have a better executive function than what my brain allows me to do.

Ambient recordings

I wear an Omi pendant (everything processed locally no audio or transcript is stored), which broadcasts all recording transcripts to Alfred’s Inbox and then the Vault. Since Alfred is connected to my calendar, my emails and some other tools, this means that I don’t have to put mental effort into using Alfred.

An example of how Alfred extracts ambient recordings. (Sensitive stuff redacted)

Last week I was cooking and realized how much of a mess our pantry was. Boxes upon boxes, completely disorganized, you couldn’t find anything. I said “I’ll clean this up on Sunday”.

I forgot about it. Without Alfred I would’ve never done it. Just ended up on a pile of ever growing stuff I committed to but never followed through. However, Alfred got the transcript, processed into the Vault and automatically created a reminder for me on Sunday morning to clean the pantry.

Keeping track of work

I stopped using Claude. I stopped using ChatGPT. Everything goes through Alfred now via Slack. Every meeting transcript, email, chat log gets pushed into the inbox folder and then subsequently processed into the Vault. I don’t have to remember anything, I can be as scattered and as chaotic as I actually am because Alfred cleans up after me.

Tasks, commitments, project updates all stored in the vault in the proper structure without me having to lift a finger. It doubles as a CRM or a Task Management system that...runs itself.

Minimum Viable Day

Managing our time with a 5 month old while working from home is not impossible but it’s damn close to it. If that wasn’t enough, my ADHD doesn’t help. Sometimes I get time blindness where I get pulled into doing something and forget about myself.

This was just annoying before, but it’s catastrophic now, because the days are long and the nights are short.

What I implemented just today is this:

Alfred pings me when a new period starts during my day. I have these periodic check ins with a todo list and some questions at 4:40am, 8:30am, 1pm, 5pm and 8pm. If I don’t respond in Slack in a few minutes, Alfred gives me a phone call to nudge me.

This is a fantastic way to use tech to do the right kinds of interruptions. I hear my phone ring and I’m out of it.

It grounds me. Keeps me present.

Durable Agentic Workflows

By default Zo comes with built in Agents that can run Scheduled Tasks. Your OpenClaw can do cron jobs (which is the same thing).

But for Alfred I wanted something more reliable so I integrated Temporal. It’s a durable execution engine. Imagine n8n but with code and if anything fails you have an audit log and you can retry any step. Also Alfred / OpenClaw / Zo can write the code part so it’s really just vibe automations.

This doesn’t sound like much but due to this hybrid architecture what can be done with code is done with code and everything is always broken down to stupid simple tasks the agents can execute. This means Alfred’s durable workflows are a lot more reliable than OpenClaw’s cron jobs and a lot more robust and resilient than n8n workflows.

Alfred Voice Mode

I connected my OpenClaw to ElevenLabs Agents platform and set up some custom configs that allow me to just say “Alfred, call my mom and ask her when she’s coming to Budapest next” and Alfred

1. Looks through the Vault to find my mom’s profile

2. Gets her phone number

3. Calls her via ElevenLabs Agents

4. Talks to her in Hungarian as it’s noted on her file

5. Once the conversation is done, saved the answers to the vault and reports back to me via Slack

Now I obviously wouldn’t want to delegate connection with my mom to Alfred but sometimes I’m in the middle of something and this is just easier. Not to mention how now I can delegate calls to vendors, utility providers, contractrors, etc.

The Roadmap

There are a few things I’m looking to add to this stack:

Set up Obsidian Sync

I started working on a simplified way to install Obsidian on your Zo. Obsidian is meant to be run with a graphic interface and Zo doesn’t have a desktop so it’s not simple, but not impossible either. This is currently in progress, here’s the Zobsidian repo if you want to watch it.

Enable automations out of the box

Zo currently doesn’t support Docker so Temporal suppport is there but inactive. Once I figure out how to reliable host Temporal on Zo I will add it there.

You can still use it on your own machine as long as you’re running Temporal on it too.

Alfred Starter Pack

My Omi pendant pipeline, the Elevenlabs Agent pipeline and a bunch of other data pipelines I have are pretty uniquely built for my needs. I am planning to package them and share them as skills / plugins soon.

How can you get your own Alfred?

If you got this far there are two options:

1. You set this up and now looking for more.

2. You don’t want the hassle and need help.

If you already have this, drop me a comment with your experience. I will probably soon set up a way for all of us Alfred owners to talk and share our experience.

If you need help, I am offering a very few limited spots for a white glove setup. I will install the full Alfred package on a Zo or on Hetzner, manage it for you, build automations and data integrations for you.

This is a premium service and will only be available for a limited time. I don’t want to run another “host OpenClaw” services. I don’t want to give people an agent, I want to give them a butler.

If this is for you, click here to register your interest.