A Hundred Years of Family Photos — The Project Continues

Building the roadmap, meeting the AI team, and getting the documentation right before we go any further.

When I published the original post about rescuing our family photo library, I described it as thirty years of memories. That was an understatement. Counting scans of old prints and slides, this collection reaches back nearly a hundred years — photographs of family members as children in the 1940s, all the way forward to last year’s trip to Egypt. The scale of what we’re preserving is larger than I initially let on, and it deserves to be said plainly: this is a century of one family’s life, and getting it right matters.

Since that post, the project has taken a significant step forward — not in deleting more files, but in something arguably more important: building a solid foundation so that the work already done doesn’t unravel, and so the work still ahead can be done safely and confidently.

Meet the Team

I should introduce the collaborators here, because this project genuinely could not have happened the way it did without them — and because they’re not human, which is worth explaining to anyone who hasn’t worked this way before.

The division of labor is straightforward: Claude thinks, plans, and documents. Max executes. Mike approves anything that could cause permanent changes.

What We Did Today

Today’s session was entirely focused on documentation and project governance — the kind of work that isn’t glamorous but determines whether a complex technical project stays on the rails months from now.

We started by reviewing Max’s own summary of the project — a “Statement of Work” that Max had drafted at the start of an earlier session. It was good, but it had gaps: missing were the rules around misfiled duplicates (more on those shortly), the safety rules that protect against accidental data loss, and lessons learned from mistakes made in a previous deduplication run.

From there we reviewed the actual database structure and all of the Python scripts written over the course of this project. Some were from early experimental phases and are now obsolete. Others are current but have gaps that need to be fixed before they can be trusted with live data. A couple of critical scripts don’t exist yet at all.

By the end of the session, five formal documents had been produced:

“Documentation isn’t the opposite of action — it’s what makes action safe when the stakes are a hundred years of family history.”

The Misfiled Duplicate Problem

One thing worth explaining for non-technical readers is the concept of a misfiled duplicate, because it illustrates why human judgment still matters even after most decisions have been automated.

Most duplicates are simple: the same photo exists in a well-organized event folder and in a staging dump, and the right answer is obvious — keep the organized one, delete the dump. Rules handle those automatically.

But a small number of cases are genuinely puzzling. Imagine a photograph from a Mission trip that somehow ended up filed in the House Flood folder as well. Both copies are real; neither location is obviously wrong in the way a staging dump is wrong. This isn’t a duplicate that should be deleted — it’s a filing error that needs a human to resolve.

In the original deduplication run, 63 such cases were identified and reviewed individually. Going forward, a dedicated process catches and flags these separately so they are never accidentally swept up in an automated deletion.

A Technical Wrinkle Worth Mentioning

One of the more interesting challenges this project has surfaced is the relationship between two computing environments: the Linux server where the photos actually live, and the sandboxed container where Max runs his scripts.

Max operates inside a kind of isolated virtual workspace. To give Max access to the actual photo library, the two environments are connected by a special link — essentially a shortcut that makes the server’s photo folder appear inside Max’s workspace. This has worked well overall, but it’s fragile. If the server restarts or the connection is reconfigured, that link can quietly break. If Max then runs a script supposed to move or delete files, the results can be unpredictable.

This risk is now explicitly documented, and a verification step has been added to the start of every work session: before Max does anything involving files, he confirms the link is intact. It’s a small thing — but the kind of small thing that prevents a very bad day with irreplaceable photographs.

What Comes Next

With the documentation in place, the next session can focus on the deduplication work that remains.

The goal remains what it always was: a library where you can find the photograph of your grandfather as a young man, know approximately when it was taken, and trust that you’re looking at the only copy.

We’re closer than ever.

Documentation for this project — including the workflow specification, database reference, and script guide — was produced in collaboration with Claude (Anthropic) and reflects work in progress as of March 2026. Claude and Max operated under Mike’s supervision; human approval was required before any file deletions.

Rescuing Thirty Years of Family Memories

How a chaotic digital photo library of over 116,000 images was sorted, deduplicated, and organized — without losing a single irreplaceable moment.

Somewhere on a hard drive, there are photographs of me as a child in the 1940s. There are photos and videos from a trip to Egypt in 2023, snapshots from Normandy beaches, a grandchild’s first steps, and decades of Christmases. Over seventy years of digital photography and scans, the collection had grown to more than 116,000 files — and it was a mess.

The same photo might exist in three different folders. A picture taken at the Paris Temple in 2015 could be filed under “2015 France,” “2016 France,” and a staging folder called “00 iPhone Dumps” — all at once. Nobody had done anything wrong. This is simply what happens when photos accumulate across phones, cameras, computers, and backup drives over three decades without a consistent system.

This is the story of how we fixed it.

The Scale of the Problem

Before any work began, a complete audit of the library revealed the true scope of the disorder. The collection contained 116,468 individual photo and video files stored across a Linux server. Many files appeared multiple times — not because anyone intended to keep duplicates, but because of how photos naturally accumulate: syncing a phone creates one copy, backing up a laptop creates another, and importing into a new photo app creates a third.

Nearly one in four files was a duplicate. That’s roughly 27,000 photographs and videos taking up space, cluttering searches, and making it harder to find the photos that matter.

“The same photo of a monastery in Crete existed in four different folders simultaneously — none of them labeled with the location.”

Beyond duplicates, the folder organization itself was inconsistent. Some years had their photos neatly organized — 2018 had a folder for France, a folder from our missionary assignment to the Visitors’ Center of the Paris Temple, a folder for San Francisco. Other years had events scattered at the root level with no parent folder. Staging folders with names like “00 iPhone Dumps” and “00 Camera Roll” had accumulated thousands of photos that were never properly filed. One folder, left over from a 2021 attempt to use Adobe Lightroom’s cloud sync, contained photos that had been quietly duplicated across the entire library.

How We Approached It

Rather than manually reviewing 27,000 files — a task that would take weeks — I worked with Claude from Anthropic to build a system to do most of the work automatically, with human judgment applied only where it genuinely mattered.

What GPS Data Revealed

Modern smartphones embed precise GPS coordinates in every photo they take. This turned out to be a powerful verification tool. When the AI flagged uncertainty about whether a photo belonged in “2023 France & Egypt” or “2023 Egypt,” we could simply check: where was the camera when this photo was taken?

Running a geographic check on over 250 disputed photos confirmed that every single one was taken within Egypt’s borders — not France. The AI’s decisions were correct in every case.

This GPS verification process opens up exciting possibilities for the future. That 2014 trip to Crete? The GPS data already reveals clusters of photos taken at Preveli Monastery, near Rethymno, and near Heraklion — places that were visited but never named in the folder structure. A future phase of this project will use geographic clustering to automatically suggest subfolder names based on where the photos were actually taken, discovering the named places within a trip that were never manually labeled.

The Outcome

The deduplication review is now complete. Of the 27,101 duplicate files identified, 26,735 have been approved for deletion — an approval rate of 98.6%. The process took one evening of work, most of it automated.

The next phase will consolidate the folder structure itself — moving root-level event folders under their proper year parents, so that every photo from 2017 lives somewhere inside a “2017” folder rather than scattered at the root level. After that, Phase 3 will use GPS clustering to automatically suggest sub-locations within trip folders.

The goal is simple: a library where you can find the photo of your father as a child, know approximately when it was taken, and trust that you’re looking at the only copy.

What This Means for the Future

Everything built for this library — the duplicate detection, the folder quality rules, the AI review pipeline, the GPS verification — is reusable. The same system will run against a second photo library on a separate drive, and eventually against the full combined archive. The hard work of building and debugging the pipeline is done. Applying it to new collections is now a matter of days, not months.

Thirty years of family history, finally in order.