Migrating to kDrive from Flickr, Apple and Google Photo Clouds

Page content

As I write this my consolidated photo album is being uploaded to kDrive, to serve as an offsite backup but the journey to this point took about two weeks, due in part to experimentation and learning to use various tools.

Tools I used

  • rsync
  • Google Takeout
  • Flickr Export
  • jdupe
  • Gemini
  • Euria
  • Le Chat, by Mistral

Work Flow

The first step is to request your data from Google Photos via the Google Takeout Tool, the Flickr Export tool for flickr, and to download all your photos locally from Apple Photos before disconnecting the local library from iCloud. Disconnecting Photos from iCloud gives you 30 days to realise you made a terrible mistake and fix it.

Export and organise

The next step is to unzip the Google Taekout files in one place, and the flickr export in another place. You want to keep the tree structure created by the zips for the next part.

Exiftool

Exiftools is a command line tool. Google Takeout and Flickr Export may detach metadata from your photos and add them to json files. Exiftools writes the exif data back into your photo files. If you ask Gemini or other AI solution for help it will provide you with the command you need to use. Request a dry run, and get the dry run to write to a text file to double check that it does what you expect.

Keep the zip files as they are. If you make a mistake it’s good to have them on hand. Downloading 50 GB files from Google Takeout takes time.

With Flickr it’s even more critical because Flickr generates 2gb files. I created a script to automatically download my 168 files.

Once you are happy that exiftool is behaving as expected you can run the command for real. Both of these steps take time so let them run in the background.

Google Takeout

Google takeout generates albums in three key ways, by individual names if you used face recognition, event name if you created an album, and by year, automatically. You will have two to three copies of some photos. In some directories you will only find json files.

When exiftool has run you can backup or delete the json files. If you have the zip files, then you’re safe.

Flickr

When I expanded the Flickr zips it created a monoolithic directory with all the photos. I ran exiftools to marry json data with the photos.

Apple Photos

If you want to extract photos from Apple Photos quickly the quickest solution is to right click, show package contents, navigate to originals, and copy photos to another directory. You will need to use exiftool to create a directory where they are sorted by year, month day, and then you can run jdupe and add them to your main library.

Looking for Duplicates and Creating Chronological Libraries

With the data added by Exiftool we can now organise the photos chronologically. The issue is that we have event photos in albums, and the same event photos in the year folder. That’s where jdupe comes in. It allows us to automatically compare photos within a directory before removing the duplicate copies.

Once this is done we can organise all the photos chronologically. This makes comparing photos much easier. It also adds a human accessible way of organising photos by year, month and day.

We repeat this step for Google Takeout and Flickr so that we end up with two clean chronological libraries.

The next step is to run jdupe again. This time we’re comparing Flickr to Google Photos. The reason for this is that in an ideal world we have a perfect mirror, with both libraries being complete. In reality we might have interrupted payment to flickr, or Google photos so we have gaps. That’s why we look for duplicates, before merging unique photos into our main photo library.

Tools such as rsync will help you merge the two libraries into the main library, as well as backup the clean library to a second hard drive on an external hard drive or on another device.

The kDrive migration

If you have not already done so, install the kDrive app and log in. Open the app and navigate to your library’s folder and tell kdrive to sync the folder. It will then start copying the data to your cloud. Now you wait for it to be done.

Cleanup and Looking Forward

Once the main library is synced to kDrive I can delete two photos folders from kDrive and my local machine. I can tell kDrive on my phone to sync to the new library folder on kDrive.

That Synching Feeling

For now:

  • Photosync adds photos to photoprism
  • immich app adds photos to Immich
  • kDrive app uploads to kDrive storage

Photoprism and Immich Watching

Both Photoprism and Immich allow you to watch an import folder(photoprism) or external library (immich). If you set the main library as a watch folder then new photos uploaded to kdrive will be added to the main library, and photoprism and Immich will add them to their own libraries. Unselect the “move” option to keep the chronological library intact.

And Finally

With jdupe, exiftool and rsync you can go from having three photo libraries wittled down to just one. You can then tell kdrive desktop to watch and sync that folder. You can use rsync to mirror the library to two or three other drives and filesystems. I have APFS, APFS (case sensitive) and ext4. I also have an offsite backup via kDrive.