[ACCEPTED]-Moving a directory atomically-atomic

Accepted answer
Score: 61

The final solution is combining the symlink- and 3 the rename-approach:

mkdir alpha_real
ln -s alpha_real alpha

# now use "alpha"

mkdir beta_real
ln -s beta_real tmp 

# atomically rename "tmp" to "alpha"
# use -T to actually replace "alpha" instead of moving *into* "alpha"
mv -T tmp alpha

Of course, the application 2 accessing alpha has to be able to deal with 1 symlinks changing in the path.

Score: 21

You can do this if you use symlinks:

Let's 12 say alpha is a symlink to directory alpha_1, and 11 you want to switch the symlink to point 10 to alpha_2. Here's what that looks like 9 before the switch:

$ ls -l
lrwxrwxrwx alpha -> alpha_1
drwxr-xr-x alpha_1
drwxr-xr-x alpha_2

To make alpha refer to 8 alpha_2, use ln -nsf:

$ ln -nsf alpha_2 alpha
$ ls -l
lrwxrwxrwx alpha -> alpha_2
drwxr-xr-x alpha_1
drwxr-xr-x alpha_2

Now you can remove 7 the old directory:

$ rm -rf alpha_1

Note that this is NOT 6 actually a fully atomic operation, but it 5 does happen very quickly since the "ln" command 4 both unlinks and then immediately recreates 3 the symlink. You can verify this behaviour 2 with strace:

$ strace ln -nsf alpha_2 alpha
...
symlink("alpha_2", "alpha")             = -1 EEXIST (File exists)
unlink("alpha")                         = 0
symlink("alpha_2", "alpha")             = 0
...

You can repeat this procedure 1 as desired: e.g. when you have a new version, alpha_3:

$ ln -nsf alpha_3 alpha
$ rm -rf alpha_2
Score: 17

Picking up on David's solution here, which 7 is fully atomic ... the only problem you'd 6 run into is that the -T option for mv is non-POSIX, and 5 so certain POSIX OSes may not support it 4 (FreeBSD, Solaris, etc. ... http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html). With slight 3 modification, this approach can be altered 2 to be fully atomic, and portable across 1 all POSIX OSes:

mkdir -p tmp/real_dir1 tmp/real_dir2
touch tmp/real_dir1/a tmp/real_dir2/a
# start with ./target_dir pointing to tmp/real_dir1
ln -s tmp/real_dir1 target_dir
# create a symlink named target_dir in tmp, pointing to real_dir2
ln -sf tmp/real_dir2 tmp/target_dir
# atomically mv it into ./ replacing ./target_dir
mv tmp/target_dir ./

exaple via: http://axialcorps.wordpress.com/2013/07/03/atomically-replacing-files-and-directories/

Score: 14

Since Linux 3.15, the new renameat2 system call can 14 atomically exchange two paths on the same 13 file system. However, there’s not even a 12 glibc wrapper for it yet, let alone a coreutils 11 way to access it. So it would look something 10 like this:

int dirfd = open(".../base", O_PATH | O_DIRECTORY | O_CLOEXEC);
syscall(SYS_renameat2, dirfd, "alpha", dirfd, "bravo", RENAME_EXCHANGE);
close(dirfd);
system("rm -rf alpha");

(Of course, you should do proper 9 error handling etc. – see this gist for a more sophisticated 8 renameat2 wrapper.)

That said – the symlink solution 7 mentioned by others is both easier and portable, so 6 unless bravo already exists and you must atomically 5 update it, go with the symlink instead.


2020 4 update: a glibc wrapper for this system 3 call is available since glibc 2.28, released 2 2018-08-01 (Debian Stretch, Fedora 29). It’s 1 still not accessible via coreutils, though.

int dirfd = open(".../base", O_PATH | O_DIRECTORY | O_CLOEXEC);
renameat2(dirfd, "alpha", dirfd, "bravo", RENAME_EXCHANGE);
close(dirfd);
system("rm -rf alpha");
Score: 8

Use a separate, guaranteed atomic, operation 7 to act as a semaphore.

So, if the creating 6 and removing a file operations are atomic:

1) create 5 a file called "semaphore".

2) If and only 4 if that is successful (no conflict with 3 existing file), do the operation (either 2 process alpha or move the directory, depending 1 on the process)

3) rm semaphore.

Score: 6

If you mean atomic across both operations, I 15 don't believe so. The closest would be:

mv alpha delta
mv bravo alpha
rm -rf delta

but 14 that would still have a small window where 13 alpha didn't exist.

To minimize the likelihood 12 of anything trying to use alpha while it's 11 not there you could (if you have the authority):

nice --20 ( mv alpha delta ; mv bravo alpha )
rm -rf delta

which 10 will crank up your process priority substantially 9 while the mv operations are happening.

If, as 8 you say in your addendum, there's only one 7 place that checks alpha and it errors if 6 it's not there, you could change that code 5 to not error immediately, but try again 4 in a short time (easily sub-second for two 3 mv operations) - these retries should alleviate 2 any problem unless you're replacing alpha 1 very frequently.

Score: 4

The SQLite documentation section File Locking and Concurrency in SQLite Version 3 has a well-written 4 description of its escalating locking protocol 3 to control concurrent reading, exclusive 2 writing, and rollback after a crash. Some 1 of those ideas apply here.

Score: 4

This should do the trick:

mkdir bravo_dir alpha_dir
ln -s bravo_dir bravo
ln -s alpha_dir alpha
mv -fT bravo alpha

strace mv -fT bravo 2 alpha shows:

rename("bravo", "alpha")

which looks pretty atomic to 1 me.

Score: 1

Even if you were accessing the inodes directly 2 there would still be no way to atomically 1 swap the inode values in user-space.

Score: 1

Worrying about the atomic nature of the 7 operation is meaningless. The thing is, the 6 access to alpha by the other task will not 5 be atomic anyway.

Oddthinking's semaphore 4 approach is the only way to go.

If you can't 3 modify the other task then you'll have to 2 ensure it's not running before doing the 1 replacement.

Score: 0

I don't believe there's any atomic way to 2 do this. Your best bet is to do something 1 like this:

mv alpha delme
mv bravo alpha
rm -rf delme
Score: 0

Something to keep in mind is that if your 5 process has any of the files in alpha open 4 when this move/delete occurs the process 3 will not notice and any data written will 2 be lost when the file is closed and finally 1 removed.

Score: 0

mv and ln can be used for atomic operations. I've 3 used ln(1) to deploy web applications atomically.

The 2 correct way to replace a symlink is with 1 ln -nsf

ln -nsf <target> <link_name>

e.g.

$ mkdir dir1
$ mkdir dir2
$ ln -s dir1 mylink
$ ls -l mylink
lrwxrwxrwx  1 phil phil 4 Nov 16 14:45 mylink -> dir1
$ ln -nsf dir2 mylink
$ ls -l mylink
lrwxrwxrwx  1 phil phil 4 Nov 16 14:46 mylink -> dir2
Score: 0

It is also possible to replace a whole parts 2 of the content at once in some prefix (Z here) using 1 unionfs-fuse:

# mkdir a b c Z
# touch a/1 b/2 c/3
# ln -s a X
# ln -s b Y
# unionfs X=RW:Y=RW Z
# shopt -s globstar
# file **
a:   directory
a/1: empty
b:   directory
b/2: empty
c:   directory
c/3: empty
X:   symbolic link to a
Y:   symbolic link to b
Z:   directory
Z/1: empty
Z/2: empty
# ln -sfn c Y
# file **/*
a:   directory
a/1: empty
b:   directory
b/2: empty
c:   directory
c/3: empty
X:   symbolic link to a
X/1: empty
Y:   symbolic link to c
Y/3: empty
Z:   directory
Z/1: empty
Z/3: empty
# fusermount -u Z
# rm -r a b c X Y Z
Score: 0

mount --bind bravo alpha should do this on Linux

It leaves the contents 11 of alpha hidden but you can bind mount the 10 parent filesystem elsewhere if you want 9 to clean it out.

If the filesystem is NFS 8 exported you would likely need to ensure 7 the export options allowed crossing filesystem 6 boundaries and do the bind mount on the 5 server.

You also need to be careful about 4 processes that have an open handle on the 3 alpha directory or subdirectory (eg cwd).

Other 2 *nix may have similar tricks up their sleeves 1 but it isn't standardised.

More Related questions