Rsync to just delete files on destination when missing from source

I have this situation where I have a huge number of images (about 50 millions, with 3-4 versions of each one), organized in a nested tree of directories, like images/103/045/475/example-{format}.jpg.

This immense catalog of images is replicated from our internal « master » to a CDN-like box. Sometimes, the replication is out of sync and some images a destroyed on the master but on the slave.

It’s not a surprise that Rsync has the right set of options to deal with this :

rsync --recursive --delete --ignore-existing --existing --prune-empty-dirs --verbose src/ dst/

Let me explain each option.

--recursive will explore the whole directory tree, not just the first level.

--delete will remove files in dst that are not in src.

--ignore-existing will not update any file in dst

--existing will not create any file in dst.

--prune-empty-dirs will remove empty directories in dst, not just deleting files.

--verbose will log what it does.

By not trying to compare the files, it’s much faster, but of course it’s only cleanup, not a real synchronization.

You can also run this a first time with --dry-run to print each action instead of executing them, to verify that Rsync does what you want.

The complete list of options is available in the man page

Cet article, publié dans Informatique, est tagué . Ajoutez ce permalien à vos favoris.

2 commentaires pour Rsync to just delete files on destination when missing from source

  1. immobilier dit :

    extraordinaire post, merci beaucoup.

  2. Max dit :

    Thanks for the explanation.

Laisser un commentaire

Entrez vos coordonnées ci-dessous ou cliquez sur une icône pour vous connecter:

Logo WordPress.com

Vous commentez à l'aide de votre compte WordPress.com. Déconnexion / Changer )

Image Twitter

Vous commentez à l'aide de votre compte Twitter. Déconnexion / Changer )

Photo Facebook

Vous commentez à l'aide de votre compte Facebook. Déconnexion / Changer )

Photo Google+

Vous commentez à l'aide de votre compte Google+. Déconnexion / Changer )

Connexion à %s