[ACCEPTED]-Moving a directory atomically-atomic
The final solution is combining the symlink- and 3 the rename-approach:
mkdir alpha_real ln -s alpha_real alpha # now use "alpha" mkdir beta_real ln -s beta_real tmp # atomically rename "tmp" to "alpha" # use -T to actually replace "alpha" instead of moving *into* "alpha" mv -T tmp alpha
Of course, the application 2 accessing alpha has to be able to deal with 1 symlinks changing in the path.
You can do this if you use symlinks:
Let's 12 say alpha is a symlink to directory alpha_1, and 11 you want to switch the symlink to point 10 to alpha_2. Here's what that looks like 9 before the switch:
$ ls -l lrwxrwxrwx alpha -> alpha_1 drwxr-xr-x alpha_1 drwxr-xr-x alpha_2
To make alpha refer to 8 alpha_2, use ln -nsf:
$ ln -nsf alpha_2 alpha $ ls -l lrwxrwxrwx alpha -> alpha_2 drwxr-xr-x alpha_1 drwxr-xr-x alpha_2
Now you can remove 7 the old directory:
$ rm -rf alpha_1
Note that this is NOT 6 actually a fully atomic operation, but it 5 does happen very quickly since the "ln" command 4 both unlinks and then immediately recreates 3 the symlink. You can verify this behaviour 2 with strace:
$ strace ln -nsf alpha_2 alpha ... symlink("alpha_2", "alpha") = -1 EEXIST (File exists) unlink("alpha") = 0 symlink("alpha_2", "alpha") = 0 ...
You can repeat this procedure 1 as desired: e.g. when you have a new version, alpha_3:
$ ln -nsf alpha_3 alpha $ rm -rf alpha_2
Picking up on David's solution here, which 7 is fully atomic ... the only problem you'd 6 run into is that the
-T option for
mv is non-POSIX, and 5 so certain POSIX OSes may not support it 4 (FreeBSD, Solaris, etc. ... http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html). With slight 3 modification, this approach can be altered 2 to be fully atomic, and portable across 1 all POSIX OSes:
mkdir -p tmp/real_dir1 tmp/real_dir2 touch tmp/real_dir1/a tmp/real_dir2/a # start with ./target_dir pointing to tmp/real_dir1 ln -s tmp/real_dir1 target_dir # create a symlink named target_dir in tmp, pointing to real_dir2 ln -sf tmp/real_dir2 tmp/target_dir # atomically mv it into ./ replacing ./target_dir mv tmp/target_dir ./
Since Linux 3.15, the new
renameat2 system call can 14 atomically exchange two paths on the same 13 file system. However, there’s not even a 12 glibc wrapper for it yet, let alone a coreutils 11 way to access it. So it would look something 10 like this:
int dirfd = open(".../base", O_PATH | O_DIRECTORY | O_CLOEXEC); syscall(SYS_renameat2, dirfd, "alpha", dirfd, "bravo", RENAME_EXCHANGE); close(dirfd); system("rm -rf alpha");
(Of course, you should do proper 9 error handling etc. – see this gist for a more sophisticated 8
That said – the symlink solution 7 mentioned by others is both easier and portable, so 6 unless
bravo already exists and you must atomically 5 update it, go with the symlink instead.
2020 4 update: a glibc wrapper for this system 3 call is available since glibc 2.28, released 2 2018-08-01 (Debian Stretch, Fedora 29). It’s 1 still not accessible via coreutils, though.
int dirfd = open(".../base", O_PATH | O_DIRECTORY | O_CLOEXEC); renameat2(dirfd, "alpha", dirfd, "bravo", RENAME_EXCHANGE); close(dirfd); system("rm -rf alpha");
Use a separate, guaranteed atomic, operation 7 to act as a semaphore.
So, if the creating 6 and removing a file operations are atomic:
1) create 5 a file called "semaphore".
2) If and only 4 if that is successful (no conflict with 3 existing file), do the operation (either 2 process alpha or move the directory, depending 1 on the process)
3) rm semaphore.
If you mean atomic across both operations, I 15 don't believe so. The closest would be:
mv alpha delta mv bravo alpha rm -rf delta
but 14 that would still have a small window where 13 alpha didn't exist.
To minimize the likelihood 12 of anything trying to use alpha while it's 11 not there you could (if you have the authority):
nice --20 ( mv alpha delta ; mv bravo alpha ) rm -rf delta
which 10 will crank up your process priority substantially 9 while the
mv operations are happening.
If, as 8 you say in your addendum, there's only one 7 place that checks alpha and it errors if 6 it's not there, you could change that code 5 to not error immediately, but try again 4 in a short time (easily sub-second for two 3
mv operations) - these retries should alleviate 2 any problem unless you're replacing alpha 1 very frequently.
The SQLite documentation section File Locking and Concurrency in SQLite Version 3 has a well-written 4 description of its escalating locking protocol 3 to control concurrent reading, exclusive 2 writing, and rollback after a crash. Some 1 of those ideas apply here.
This should do the trick:
mkdir bravo_dir alpha_dir ln -s bravo_dir bravo ln -s alpha_dir alpha mv -fT bravo alpha
strace mv -fT bravo 2 alpha shows:
which looks pretty atomic to 1 me.
Even if you were accessing the inodes directly 2 there would still be no way to atomically 1 swap the inode values in user-space.
Worrying about the atomic nature of the 7 operation is meaningless. The thing is, the 6 access to alpha by the other task will not 5 be atomic anyway.
Oddthinking's semaphore 4 approach is the only way to go.
If you can't 3 modify the other task then you'll have to 2 ensure it's not running before doing the 1 replacement.
I don't believe there's any atomic way to 2 do this. Your best bet is to do something 1 like this:
mv alpha delme mv bravo alpha rm -rf delme
Something to keep in mind is that if your 5 process has any of the files in alpha open 4 when this move/delete occurs the process 3 will not notice and any data written will 2 be lost when the file is closed and finally 1 removed.
mv and ln can be used for atomic operations. I've 3 used ln(1) to deploy web applications atomically.
The 2 correct way to replace a symlink is with 1 ln -nsf
ln -nsf <target> <link_name>
$ mkdir dir1 $ mkdir dir2 $ ln -s dir1 mylink $ ls -l mylink lrwxrwxrwx 1 phil phil 4 Nov 16 14:45 mylink -> dir1 $ ln -nsf dir2 mylink $ ls -l mylink lrwxrwxrwx 1 phil phil 4 Nov 16 14:46 mylink -> dir2
It is also possible to replace a whole parts 2 of the content at once in some prefix (
Z here) using 1
# mkdir a b c Z # touch a/1 b/2 c/3 # ln -s a X # ln -s b Y # unionfs X=RW:Y=RW Z # shopt -s globstar # file ** a: directory a/1: empty b: directory b/2: empty c: directory c/3: empty X: symbolic link to a Y: symbolic link to b Z: directory Z/1: empty Z/2: empty # ln -sfn c Y # file **/* a: directory a/1: empty b: directory b/2: empty c: directory c/3: empty X: symbolic link to a X/1: empty Y: symbolic link to c Y/3: empty Z: directory Z/1: empty Z/3: empty # fusermount -u Z # rm -r a b c X Y Z
mount --bind bravo alpha should do this on Linux
It leaves the contents 11 of alpha hidden but you can bind mount the 10 parent filesystem elsewhere if you want 9 to clean it out.
If the filesystem is NFS 8 exported you would likely need to ensure 7 the export options allowed crossing filesystem 6 boundaries and do the bind mount on the 5 server.
You also need to be careful about 4 processes that have an open handle on the 3 alpha directory or subdirectory (eg cwd).
Other 2 *nix may have similar tricks up their sleeves 1 but it isn't standardised.
More Related questions