cryptmount and mount not playing nicely together

Discussion:

(too old to reply)

William Unruh

2023-12-02 20:30:53 UTC

I have two a regular mount point /fastlocal, which is a partition on an
SSD. On top of various items in /fastlocal, I cryptmount a variety of
encrypted partitions. But this is leading to trouble. After mounting the
encrypted partition, I ran mount -a. Now, it should NOT have remounted
/fastlocal, since it was already mounted. But it did. Mount now says
that there are two mounts of fastmount
/dev/sda7 on /fastlocal type ext4 (rw,noatime)
/dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
And now the
cryptmount partitions were buried. Ie, all I find in /fastlocal is the
empty mountpoints for the various cryptmount partitions. I try to
unmount the cryptmount partions but they all tell me that they are in
use and cannot be umounted. I try to unmount /fastlocal, and it tells me
that cannot do so as it is in use. So I am stuck.
If I were not manymiles away from the machine, I could reboot, but
somehow a usb dive got left in the machine, and when I try to reboot, it
tries to reboot from the usbdrive (it is a copy of the Mga8 installation
disk). And my arm is not long enough to reach it. Is there any way out
of this predicament.
And this appears to be a bug in mount and/or cryptmount. Ie, the second
mount of /fastlocal covers up the cryptmount mounts.

William Unruh

2023-12-02 21:46:47 UTC

Permalink

Post by William Unruh
I have two a regular mount point /fastlocal, which is a partition on an
SSD. On top of various items in /fastlocal, I cryptmount a variety of
encrypted partitions. But this is leading to trouble. After mounting the
encrypted partition, I ran mount -a. Now, it should NOT have remounted
/fastlocal, since it was already mounted. But it did. Mount now says
that there are two mounts of fastmount
/dev/sda7 on /fastlocal type ext4 (rw,noatime)
/dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
And now the
cryptmount partitions were buried. Ie, all I find in /fastlocal is the
empty mountpoints for the various cryptmount partitions. I try to
unmount the cryptmount partions but they all tell me that they are in
use and cannot be umounted. I try to unmount /fastlocal, and it tells me
that cannot do so as it is in use. So I am stuck.
If I were not manymiles away from the machine, I could reboot, but
somehow a usb dive got left in the machine, and when I try to reboot, it
tries to reboot from the usbdrive (it is a copy of the Mga8 installation
disk). And my arm is not long enough to reach it. Is there any way out
of this predicament.
And this appears to be a bug in mount and/or cryptmount. Ie, the second
mount of /fastlocal covers up the cryptmount mounts.

It is even stupider than I thought.
When I installed the SSD, a few years ago, Ihad a homedirectory on
/disk9/home/unruh
When I installed the SSD, I installed my home directory onto the SSD
whcih was mounted on /fastlocal.
This worked fine but just in case there were some old references in some
software script to /disk9/home/unruh, I made a soft link to
/fastlocal/unruh. Then sometime, I put in a bind mount from
/fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
by hand after the system had booted up, this did not matter, but if I
did a mount -a, that mount from /fastlocal/unruh on top of
/fastlocal/unruh covered up the cryptmount.
Now the bug in the system meant that if I tried to umount
/fastlocal/unruh, I think it tried to do that first mount, instead of
the last mount. and the cryptmounts sat on top of it. and stopped the
unmount. But the crytpmounts were now covered by the last
/fastlocal/unruh mount and the system was all tied up and unfixable.

Or is there some way I can get out of this mess without rebooting?

jim whitby

2023-12-02 22:42:29 UTC

Permalink

Post by William Unruh

It is even stupider than I thought.
When I installed the SSD, a few years ago, Ihad a homedirectory on
/disk9/home/unruh
When I installed the SSD, I installed my home directory onto the SSD
whcih was mounted on /fastlocal.
This worked fine but just in case there were some old references in some
software script to /disk9/home/unruh, I made a soft link to
/fastlocal/unruh. Then sometime, I put in a bind mount from
/fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
by hand after the system had booted up, this did not matter, but if I
did a mount -a, that mount from /fastlocal/unruh on top of
/fastlocal/unruh covered up the cryptmount.
Now the bug in the system meant that if I tried to umount
/fastlocal/unruh, I think it tried to do that first mount, instead of
the last mount. and the cryptmounts sat on top of it. and stopped the
unmount. But the crytpmounts were now covered by the last
/fastlocal/unruh mount and the system was all tied up and unfixable.
Or is there some way I can get out of this mess without rebooting?

Can you unmount by using using umount /dev/sd???

William Unruh

2023-12-03 15:41:16 UTC

Permalink

Post by jim whitby

Post by William Unruh

It is even stupider than I thought.
When I installed the SSD, a few years ago, Ihad a homedirectory on
/disk9/home/unruh
When I installed the SSD, I installed my home directory onto the SSD
whcih was mounted on /fastlocal.
This worked fine but just in case there were some old references in some
software script to /disk9/home/unruh, I made a soft link to
/fastlocal/unruh. Then sometime, I put in a bind mount from
/fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
by hand after the system had booted up, this did not matter, but if I
did a mount -a, that mount from /fastlocal/unruh on top of
/fastlocal/unruh covered up the cryptmount.
Now the bug in the system meant that if I tried to umount
/fastlocal/unruh, I think it tried to do that first mount, instead of
the last mount. and the cryptmounts sat on top of it. and stopped the
unmount. But the crytpmounts were now covered by the last
/fastlocal/unruh mount and the system was all tied up and unfixable.
Or is there some way I can get out of this mess without rebooting?

Can you unmount by using using umount /dev/sd???

Nope, does not work. Complains "Target is busy".
Not sure what it thinks the target is since in /etc/fstab, the target is
/disk9/home/unruh
which is a softlink to /fastlocal/unruh, which seems to have resulted in
/fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
mounting a directory on top of itself should be illegal and not work.
What is interesting is that other mounts onto directories inside
/fastlocal/unruh get hidden by the mount of the directory on top of
itself, and then make the directory busy.

David W. Hodgins

2023-12-03 19:58:41 UTC

Permalink

Post by William Unruh
Nope, does not work. Complains "Target is busy".
Not sure what it thinks the target is since in /etc/fstab, the target is
/disk9/home/unruh
which is a softlink to /fastlocal/unruh, which seems to have resulted in
/fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
mounting a directory on top of itself should be illegal and not work.
What is interesting is that other mounts onto directories inside
/fastlocal/unruh get hidden by the mount of the directory on top of
itself, and then make the directory busy.

Don't try to unmount the device, as that's used by both mounts (directly or
indirectly). umount the directory used in the second mount, to reveal the
directory and it's contents used in the first mount.

If the second mount is busy, something started after the second mount is using
a directory or file inside of that second mount. It could be something such
as the current directory inside of a terminal in a konsole tab, or a program
such as tracker monitoring the file system for changes to index.

The program lsof can help identify those programs.

Regards, Dave Hodgins

William Unruh

2023-12-04 00:03:19 UTC

Permalink

Post by David W. Hodgins

Don't try to unmount the device, as that's used by both mounts (directly or
indirectly). umount the directory used in the second mount, to reveal the
directory and it's contents used in the first mount.
If the second mount is busy, something started after the second mount is using
a directory or file inside of that second mount. It could be something such
as the current directory inside of a terminal in a konsole tab, or a program
such as tracker monitoring the file system for changes to index.
The program lsof can help identify those programs.

Unfortunately neither lsof nor fuser indicate that anything is using
that directory.
Also it is not mounted on any remote nfs client. findmnt indicates that
the various cryptmounts are mounted, but mount sees none of them and
cryptmount -u says they are not mounted, presumably because they are
hidden under the second mount of /fastlocal/unruh on top of itself.
Presumably it is also because the second mount occured via the link.
(ie it was in fstab
/fastlocal/unruh /disk9/home/unruh none bind,nofail 0 0

Post by David W. Hodgins
Regards, Dave Hodgins

David W. Hodgins

2023-12-02 23:10:53 UTC

Permalink

Mounting something on a directory does hide everything under it, including
prior mounts on that directory.

To unhide those things, unmount the second mount, so in this case
"umount /fastlocal". It will only umount the most recent mount.

Regards, Dave Hodgins

jim whitby

2023-12-04 14:50:49 UTC

Permalink

On 12/3/23 10:41, William Unruh wrote:
<snip>>>>

Post by William Unruh

Post by jim whitby
Can you unmount by using using umount /dev/sd???

Part of man umount
...

-f, --force
Force an unmount (in case of an unreachable NFS system).

Note that this option does not guarantee that umount command
does not hang. It’s strongly
recommended to use absolute paths without symlinks to avoid
unwanted readlink(2) and stat(2) system
calls on unreachable NFS in umount.
...

-l, --lazy
Lazy unmount. Detach the filesystem from the file hierarchy
now, and clean up all references to this
filesystem as soon as it is not busy anymore.

A system reboot would be expected in near future if you’re
going to use this option for network
filesystem or local filesystem with submounts. The
recommended use-case for umount -l is to prevent
hangs on shutdown due to an unreachable network share where
a normal umount will hang due to a
downed server or a network partition. Remounts of the share
will not be possible.

Try the -l first

Jim

William Unruh

2023-12-05 19:22:14 UTC

Permalink

Post by jim whitby
<snip>>>>

Post by William Unruh

Post by jim whitby
Can you unmount by using using umount /dev/sd???

None of these are nfs filesystems. they are all local. I have also found
-f not to work even for nfs mounted filesystems.

Post by jim whitby
...
-l, --lazy
Lazy unmount. Detach the filesystem from the file hierarchy
now, and clean up all references to this
filesystem as soon as it is not busy anymore.

The problem is that I have no idea what it making them busy. In this
case I suspect it is the fact that cryptmount filesystems are mounted on
the first mounting /fastlocal/unruh, and the second one hides all of he
cryptmounts. So I cannot unmount cryptmount partitions, and the system
things that there is only the second mount and the cryptmounts are
making the second one busy.
If I do
findmts
I see only the one mount onto /fastlocal/unruh, not two of them.

I finally "solved" the problem by getting someone who was able to get
into the locked server room to unplug the usb stick from the machine
which was stopping the reboot. Then I could reboot the machine making
sure I had removed the indirect mount onto the /fastlocal/unruh via the
solft link.
(Ie, the second mount was a mount to /disk9/home/unruh which was a
solftlink to /fastlocal/unruh, which the mount commend interpreted as a
mount onto /fastlocal/unruh, even though a mount of that same partition
already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
had lines like
/dev/sdb3 /fastlocal/unruh
and
/dev/sdb3 /disk9/home/unruh bind,....
and disk9/home/unruh was a softlink to /fastlocal/unruh
That second one did not occur during reboot, but did occur after I
mounted the crytmounts onto some interior directories in
/fastlocal/unruh and then ran mount -a
If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount
command would complain that /dev/sdb3 was already mounted onto
/fastlocal/unruh, but did not complain, I guess because I told it to
mount onto the link /disk9/home/unruh. Ie, the error trap failed to
complete the link and realise I was trying a double mount.

Ie, mount should not allow bin mounts onto links, or should test that
the link does not result in a double mount from the same partition onto
the same mountpoint

Post by jim whitby
A system reboot would be expected in near future if you’re
going to use this option for network
filesystem or local filesystem with submounts. The
recommended use-case for umount -l is to prevent
hangs on shutdown due to an unreachable network share where
a normal umount will hang due to a
downed server or a network partition. Remounts of the share
will not be possible.
Try the -l first
Jim

Kirk_Rockstein

2023-12-06 18:04:18 UTC

Permalink

Post by William Unruh

Post by jim whitby
<snip>>>>

Post by William Unruh

Post by jim whitby
Can you unmount by using using umount /dev/sd???

None of these are nfs filesystems. they are all local. I have also found
-f not to work even for nfs mounted filesystems.

Post by jim whitby
...
-l, --lazy
Lazy unmount. Detach the filesystem from the file hierarchy
now, and clean up all references to this
filesystem as soon as it is not busy anymore.

The problem is that I have no idea what it making them busy. In this
case I suspect it is the fact that cryptmount filesystems are mounted on
the first mounting /fastlocal/unruh, and the second one hides all of he
cryptmounts. So I cannot unmount cryptmount partitions, and the system
things that there is only the second mount and the cryptmounts are
making the second one busy.
If I do
findmts
I see only the one mount onto /fastlocal/unruh, not two of them.
I finally "solved" the problem by getting someone who was able to get
into the locked server room to unplug the usb stick from the machine
which was stopping the reboot. Then I could reboot the machine making
sure I had removed the indirect mount onto the /fastlocal/unruh via the
solft link.
(Ie, the second mount was a mount to /disk9/home/unruh which was a
solftlink to /fastlocal/unruh, which the mount commend interpreted as a
mount onto /fastlocal/unruh, even though a mount of that same partition
already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
had lines like
/dev/sdb3 /fastlocal/unruh
and
/dev/sdb3 /disk9/home/unruh bind,....
and disk9/home/unruh was a softlink to /fastlocal/unruh
That second one did not occur during reboot, but did occur after I
mounted the crytmounts onto some interior directories in
/fastlocal/unruh and then ran mount -a
If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount
command would complain that /dev/sdb3 was already mounted onto
/fastlocal/unruh, but did not complain, I guess because I told it to
mount onto the link /disk9/home/unruh. Ie, the error trap failed to
complete the link and realise I was trying a double mount.
Ie, mount should not allow bin mounts onto links, or should test that
the link does not result in a double mount from the same partition onto
the same mountpoint

Maybe Running:
find /proc -name mounts | xargs grep /dev/sdb3
then kill -9 the PID's of found offending processes/mounts.

David W. Hodgins

2023-12-06 22:37:53 UTC

Permalink

Post by Kirk_Rockstein
find /proc -name mounts | xargs grep /dev/sdb3
then kill -9 the PID's of found offending processes/mounts.

That can be shortened to "grep /dev/sdb3 /proc/*/mounts", which shows what
processes have been started after sdb3 was mounted, or mounted it.

It does not help indicate which of those processes have open files within
that mount.

Regards, Dave Hodgins