Our family butler, Alfred

How I'm engineering presence for our family

It’s 5:30. My silent alarm just went off. It’s a wristband that needs two taps to turn off the alarm, but it rarely works. So I start pounding my wrists in panic while my daugher is sleeping on my left and my wife is sleeping on my right.

I’m lying in bed and the storm of thoughts arrive. My working memory is empty, I have no filter and more importantly zero clue what I need to do today. I need to somehow download the right information into my brain to even get started with the day.

Since I have an underdeveloped executive function in the grey goo in my head that’s not just difficult, it’s paralyzing. I reach for my phone in hopes of finding information that would bring some much needed clarity.

I anxiously scroll through my notes, Slack, emails, calendar events, project management tools to rebuild context for the day, but no matter. I’m already lost because I have now overwhelmed my cortex with too much information.

My brain wants to give up and flee the scene but I use a Balance Phone so I cannot open social media. I’m only half awake but I crawl out of bed and into the living room where I subconsciously reach for my iPhone.

I use it as a camera. It shouldn’t be there, it should be in a tripod in my home office. But somehow it always finds its way downstairs. After a few minutes (15?) of getting my doomscrolling fix I boot up just enough to get to the kitchen and take my meds and drink some water.

It’s not ideal that I have to rely on stimulants to get my day started but it is what it is. I learned to live with it.

---

After the holidays, I came back to work and I realized that my entire context of what I was working on was gone. I was reading through Github commits and Telegram messages, scanning through Linear as if I was reading someone else’s work.

I spent two weeks to get back on track with everything and that’s while taking 30mg Ritalin LA everyday. For the better part of the last 3 decades up until a year ago this was hard enough for me to lose out on six figure deals or kill entire projects.

Meds help me with functioning. It’s still hard but I don’t drop clients, projects or commitments anymore. But the everyday stuff is hard.

Remembering to take out the bins. Or that I promised my sister-in-law that I would fix her drawer. Or that I need to take the meat out of the freezer so I can meal prep tomorrow.

To say that having a family butler that can fill in the gaps is a must is an understatement.

I have accepted the limitations of my monkey brain and I’m happy with them. They unlock my unique creativity and problem solving skills that shape who I am. But that doesn’t mean I get to escape responsibility for being forgetful.

So I decided I will build my way out of it.

---

You need a family butler

According to Household Staff website, butlers are “responsible for maintaining the smooth operation of day-to-day activities,
ensuring the comfort and satisfaction of all household members and guests.”

The job is delegation of attention. That’s not just doing stuff, it’s owning stuff. It’s proactively noticing when something’s wrong. It’s anticipatory.

Daily briefings

I deal with so much information daily that it overwhelms my brain. When I get overwhelmed, I am not present.

Alfred makes it easier to be present by paying attention to the things that overwhelm me.

For now it lives inside our Slack workspace, where Alfred saves me from the morning torture I described above every morning. Instead, I get this:

I don’t have to prompt, I don’t have to trigger anything. I don’t have to use anything, it’s just there. I replaced half an hour of context building plus half an hour of accidental doomscrolling with a prewritten, personalized message.

But it’s not easy to get this right because you need tons of data from different sources. For me this means:

• Health data comes from WHOOP directly

• Appointments and emails come from Google

• Goals are set every Monday, Alfred asks me, grills me and reminds me of them including setting a reward if I hit them.

• Weekly briefs are longer, which include financial data, like account balance, upcoming bills, payments, quick overview of world news, USD/HUF and USD/EUR rates and some reminders from the previous week.

It’s easy to generate hallucinated information or slop so I wanted to make sure Alfred gives me relevant and true information. For this, it has a nightly health check script that tests every single integration to see if they are working with auto-fix if broken.

Knowledge Base

In order for this to work, Alfred has its own knowledge base in an Obsidian Vault. This is inspired by the ontology created by Palantir.

When I built Alfred, I systematically scanned through my Google Drive and my emails to backfill this knowledge base. Then I went ahead and processed all my meeting transcripts (I’ve been recording all my meetings for over a year using Sembly). Then to be really meta I also added conversation history with Alfred itself too.

I also set up a script so every hour the raw conversation history gets exported to the Obsidian Vault, then it gets processed in the background. This keeps the Vault clean and up to date.

Every time Alfred needs to get information it will get it from the Vault.

Alfred is powered by Clawdbot

Our family butler is powered by Clawdbot¹ Moltbot on a Mac Mini hosted remotely by MacStadium in a datacenter in Nevada.

I use Claude Opus 4.5 and Haiku 4.5 to run Alfred (for now).

So what is Moltbot? If you want a deep dive you can watch this video below. It’s a bit too much hype for my liking though.

Stuff Alfred does for us now

Apart from giving me daily and weekly briefings and storing information in an Obsidian Vault, Alfred can now:

• Have access to a shared Apple Note we use as a shopping list and order groceries from Auchan on request.

  

• Helps me deal with the biggest presence killer: Alfred automatically tracks my screentime across all my devices in 15 minute chunks, categorizes them, assigns to projects, clients, writes descriptions and logs them in Solidtime. At first I tried to just track screentime but it became a bit more complex so I will write a separate post about it.

  

• Manage my calendar. Yesterday I got a new keynote opportunity for next week. I asked Alfred to estimate travel time and block the relevant timeslot and add all relevant info from the email to the calendar event

  

• Manage my emails. I’m very aggressive with the idea of “ignore every email unless you must deal with it”. The best way to get inbox zero is to ignore it and ironically it works. Alfred is trained to only raise my attention on emails that absolutely require my attention. Out of ~200 emails I got on Tuesday only one were relevant.

• Create and schedule content. I created a Postiz integration which is connected to my social accounts. I also created a Ghost integration which is connected to my other blog. I’m now testing my ideas on how to create content from braindump that’s not absolutely sloppy.

  

• Manage reminders, todos and tasks. My brain is free to be present. I can just ask Alfred to remind me in 40 minutes to take the focaccia out of the oven.

  

• Keep track of financials. I use Fintable to keep track of our finances. Alfred keeps an eye on the spreadsheet Fintable uses and can query it on demand.

Stuff I’m working on for Alfred

Voice calling

Moltbot’s support for voice call is a bit wonky and slow. I set up Twilio with a phone number and asked Alfred to call my mom and ask her when she is coming to Budapest next. She did get the call but after the first response Alfred switched back to English and lost all context. The audio quality was terrible and the whole experience was slow. I did build ultra-low latency voice agents before so I might look at this later.

It would be nice to have an agent that can do quick phone calls for service providers but more importantly I would want Alfred to give me a phone call so I don’t have to read the stuff he sends.

Home Assistant

Last year I started setting up smart home automation and I also got a Home Assistant Voice Preview Edition. Moltbot has a Home Assistant plugin so I’m hoping to wire these together which would make Alfred infinitely more intelligent about our life.

This would allow me to do things like notify me by talking proactively instead of sending Slack messages. Perfect example would be let’s say I have a meeting in town at 2pm and traffic says I should leave at 1:15pm latest to get there comfortably but I’m still sitting at my laptop at 1pm. So Alfred could just tell me “David, you should leave soon for your meeting, traffic is heavy”.

The other example is doing things like managing a shopping list via Alfred so when I’m cooking and realize we’ve run out of rice, I can just say “Hey Alfred, add rice to the shopping list”.

Then I could have Alfred automatically order whatever is on the shopping list every Sunday night.

I know these are not entirely new ideas. Tools like Alexa have been around for ages. But the level of agency and the level of control I can have over the entire system is unprecedented.

Not to mention that I could finally do what I promised my mom when I was 17, that “one day I will live in a house that greets me when I get home”.

All in all, Alfred now works at the barebones². It does things automatically for me, I can ask it to set up automated scripts, remind me to do stuff without prompting, which is basically the ultimate goal:

To prompt less and live more.

1. It got hastily rebranded after the founder got a nicely worded letter from Anthropic.

2. You will find security warnings about Moltbot. They are right, which is why I put my entire system behind a Tailscale private network called Tailnet.