Ever feel like your digital life is scattered across a hundred different apps, none of which really understand the full picture of who you are? That's exactly why I'm building AM (Allied Mastercomputer) - a privacy-first system that aggregates, processes, and analyzes everything from your devices to create what I call a "second brain."
The Journey So Far
This isn't my first rodeo. AM is the culmination of several prototypes - Pino, Loom, Loom v2, and Loom v3. Each iteration taught me something crucial:
- Pino: Proved passive data collection was feasible
- Loom v1: Showed the power of unified timeseries storage
- Loom v2: Demonstrated real-time processing at scale
- Loom v3: Validated the multi-modal ML pipeline approach
Now AM brings it all together into something actually useful.
What AM Actually Does
At its core, AM collects raw data streams from everywhere: - Your phone's sensors (GPS, accelerometer, heart rate from your watch) - Environmental audio (with consent, of course) - Screenshots and passive photos triggered by motion - App usage, notifications, calendar events - Even your emails and social media activity
But here's where it gets interesting. Instead of just storing this data, AM processes it through ML pipelines to extract meaning:
Raw GPS coordinates → "You're at the gym"
Heart rate spike + calendar → "Stressed during that meeting"
Audio transcript + location → "Conversation with Sarah about the project"
The real magic happens when AM starts finding patterns you didn't even know existed. Like that time it noticed my heart rate always spiked when certain email senders appeared in my inbox. Or when it correlated my poor sleep with late-night coding sessions (shocker, I know).
The Technical Stack (Because I Can't Help Myself)
I went all-in on a microservices architecture:
- gRPC with Protocol Buffers for efficient data streaming (3-10x smaller than JSON!)
- NATS JetStream for reliable message queuing
- TimescaleDB with pgvector for time-series data and ML embeddings
- Go for high-performance services
- Python for ML processors
- River ML for streaming anomaly detection that works from the first data point
The anomaly detection system is my favorite part - it's a 4-tier architecture that not only detects anomalies but explains WHY they happened:
{
"primary_cause": "exercise_onset",
"confidence": 0.92,
"temporal_sequence": [
"10:31 - accelerometer activity increased",
"10:32 - GPS detected movement start",
"10:33 - heart rate began climbing"
]
}
Why I'm Obsessed With This
The Privacy Angle
I'm tired of giving my data to companies that use it to manipulate me. With AM, everything stays local. No cloud dependencies, no tracking, no BS. Your data is yours, processed on your hardware, under your control.
The Memory Problem
Human memory is fallible. I forget important conversations, lose track of insights, and miss patterns in my own behavior. AM is like having perfect recall - searchable, queryable, analyzable.
The Health Connection
My smartwatch tells me my heart rate. My calendar tells me I have meetings. But nothing connects the dots to say "your heart rate spikes 20% during meetings with your boss - maybe address that stress?" AM does.
The Future Vision
Eventually, I want AM to evolve into a true digital twin. Imagine: - An AI agent that knows your patterns well enough to act on your behalf - Predictive health alerts based on subtle changes - A system that learns what makes you productive/happy/healthy - Maybe even consciousness augmentation (yeah, I dream big)
The Hard Parts (Let's Be Real)
Building this isn't easy. Here are the biggest challenges:
Battery Life: Continuous data collection murders phone batteries. Solution? Smart triggers - the IMU detects motion, then activates cameras. Low-power monitoring with burst capture. Target: 2-3 days battery life.
Data Volume: A day of audio/video/sensors = GBs of data. Solution? Compression, selective capture based on novelty, and rolling retention (keep raw data for 30 days, abstractions forever).
Privacy Concerns: Recording everything is creepy. Solution? Opt-in everything, local processing, encryption everywhere, and clear data deletion tools. You own your data, period.
ML Compute: Running NVIDIA Parakeet STT and LLMs isn't cheap. Solution? Quantized models, edge computing where possible, and smart batching. The new open source models are getting good enough for personal use.
The dream? A personal AI that actually knows you - not from surveillance capitalism, but from your own sovereign data. An AI that nudges you toward better habits, catches health issues early, and preserves your memories forever.
Want to Help?
This is open source (AGPL for personal use, commercial licenses available). Check out the GitHub repo if you want to contribute or just follow along.
Fair warning: this is still very much a work in progress. But if you're interested in owning your digital self, building a real "second brain," or just think continuous personal data collection is cool (and not creepy when you control it), come join the journey.
Because let's be honest - if we're going to live in a world where everything is recorded anyway, shouldn't we at least own the recordings?
Contact: am@steele.red | GitHub