Troubleshooting MacOS APFS FileVault Permissions De-Sync During High Sierra to Big Sur Update
Fixing a machine on the brink of disaster
February 1, 2021
Tagged: MacOS
MacOS Big Sur Upgrade from High Sierra Failing Due to Mismatched FileVault (+ Fix)
I had a real bitch of a bug recently when trying to update my laptop from High Sierra to Big Sur. The best I can divine about what happened is that, while doing its usual multiple restarts as a new OS is installed, somehow my drive permissions fell out of sync with my normal OpenDirectory login permissions. Because I used FileVault (never again?) on the drive, this meant that when the system was trying to access the Macintosh HD - Data partition using the passed in credentials from booting from Macintosh HD, the drive wouldn’t properly mount and hence I couldn’t boot the OS.
It was also unclear if the system upgrade itself had failed or was still in progress, as when I would login it would act as if the upgrade was going, would show me a progress bar, and then at the end would loop back to the install screen.
I would be able to “log in” in the normal user login screen and the password would be accepted, but then, upon logging in, I’d get the circle with a line through it symbol to indicate the machine couldn’t be booted.
Sometimes instead it would just restart the machine, putting me back at the login screen. But either way I couldn’t truly login. I recognized something was supremely wrong when I tried to boot into recovery mode and try to manually mount the Macintosh HD - Data drive, but it wouldn’t accept the password I literally just used to login to my account. So uh, what?
I tried to debug this issue for about three weeks. I called Apple Support twice, and spent probably ~4 hours total on calls with them to no avail. I tried every diskutil command in terminal I could find, and even tried some weirder things. Below I wanted to outline the typically suggested methods and why they didn’t work for me, and tell you what did work for me at the end. If you’re having this issue and are trying to find the solution, scroll to the bottom of this post! I’m writing all the stuff below in order to attract people to this post that are having the same issue, because it’s a lot of us and I want to get the word out!
Note: I also apparently had used iCloud for FileVault, so I had no personal recovery key! This means that methods like this didn’t work.
UPDATE 2/13/21: Since publishing this I have received a ton of messages from people saying the steps below helped them recover all of their data! Some people have also asked if they can tip me for writing this up, to which I say sure! You can venmo me @kyle-kukshtel or use PayPal and send money to [email protected].
Additionally, because the steps below are somewhat technical, I’ve had people reaching out to ask if I can help them debug this issue with them over the phone/video, to which I say, sure! For $150 I’ll get on a 30 minute call to work through this with you. If we don’t get the machine to a workable state I can refund you via Paypal. If you’re interested in this feel free to reach out to me via email at [email protected] with the subject line “[Big Sur Upgrade Help Request]”. Otherwise, hope the following works for you!
UPDATE 6/25/21: A lot of people seem to be encountering some version of this bug, and the bug itself slightly varies based on the machine you are using. If you’re unable to do the following steps below or want to try other methods as well, some people have reported the steps in the following blog post also worked for them:
https://mrmacintosh.com/big-sur-upgrade-not-enough-hd-space-serious-issue-possible-data-loss/
However, it’s worth saying that this blog post does assume your drive is able to mount, which was not the case in my issue and I had to do the steps below. I’d say try their solution first, and consider the fix below the more “nuclear” option.
What Didn’t Work
Resetting the password
This was one of the most common suggestions I found. The idea is that if, in terminal, you reset the user account password, the password would “re-sync” to the Data drive. I found this not to be the case. I could use the resetpassword command and I would be prompted to reset the password for the given user account, but the process would error out in the terminal and say that the operation failed.
However, the operation clearly didn’t fail because the new password was accepted back at the login screen. So it was changing the password at that level, but the changes weren’t propagating through to the Data partition.
Resetting iCloud password / iCloud login recovery
Knowing that the FileVault key was stored in iCloud for the drive, I saw some forum posts that suggested that resetting the iCloud password would re-sync permissions. This also didn’t work.
I was able to reset the iCloud password, but both it and trying to do an iCloud login recovery via the “Forgot Password?” dialog led to a message saying that “The supplied iCloud was unable to unlock this volume”. Again the same issue - the Macintosh HD “users” weren’t synced to the Data partition.
Using diskutil to unencrypt/unlock the drive
This option is the one that works for most people, but it didn’t work for me. Many people have a similar issue, and often find themselves at one of these two posts:
Unlock or decrypt an encrypted APFS boot drive from the command line
Decrypting an APFS encrypted volume using diskutil on macOS 10.13.2
If you’re like me though, this doesn’t work. Primarily because this line fails:
diskutil apfs unlockVolume /dev/apfs_volume_id_here
When prompted for a passphrase, just like in Disk Utility, no user password is accepted. Therefore, you’re unable to even mount the volume to unlock it or decrypt it. I also tried every password for this. I tried both user account passwords, I tried both iCloud account passwords, etc. Nothing.
Sidenote: WTF is iCloud recovery User?
One thing that came up here when I was walking through the above posts, specifically when running this command:
diskutil apfs listCryptoUsers disk1s1
Was that I would see the local users and their UUIDs, but I would also see “iCloud Recovery User” and “iCloud Recovery Key”. I could find nothing on the internet about what these were for. Apple Support (below) didn’t know, very few posts anywhere mention them, and nothing I found talks about how to use to unlock the drive or anything.
My best guess is that they are a cached version of the “correct” iCloud account to unlock the drive locally and without needing internet access. However, given that I was unable to use either user’s iCloud account to unlock the drive, it’s unclear if these entries were corrupted, or referenced something else entirely.
Booting into recovery / Installing another OS
At one point, even booting into Internet Recovery stopped working. I have no idea why. One day it was working, and the next day it wasn’t with me changing literally nothing.
Feeling like all was lost, I thought to try and basically restart the process by trying to install a different, older OS (Catalina) on the machine just to see what would happen.
I was able to boot a USB installer of Catalina, but the install promptly failed and… booted me into the proper recovery mode! Yay? This meant that, for me to get into recovery now, I had to boot into the Catalina installer, have it fail, and then access recovery. This sucked, but at least I had access to a terminal again.
If I tried to boot into Internet Recovery normally, the machine would act like it was going to properly boot into Internet Recovery, but then would kick me out to normal recovery (no Disk Utility, no Terminal access), where, because the drive wasn’t mounted, all I could do was restart the machine or wipe the disk.
I did find that once I was booted into this recovery, I could remove the USB installer that got me there in the first place in case I needed to hook up an external drive and charger cable (I’m on a 13in, so only 2 ports).
Single User Mode
This was basically a non-starter. I could get to where the terminal was spitting out tons of lines on the screen, but it failed to do anything because the disk wouldn’t mount. I was unable to enter any commands so called this a dead end.
Calling Apple Support
I called Apple Support! I had tried a lot of stuff on my own before this and saw this as a bit of a last ditch effort as I know Apple’s line is mostly “Well you backed up the disk right? Just delete the partition and reinstall the operating system.” This… turned out to basically be the case with them. I even got elevated to a senior tech person, but I really felt like I was telling them about diskutil and its commands/parameters. I don’t want to disparage the tech support, as I’m sure they fix plenty of meaningful issues, but I was most surprised by how their solutions were more or less what I read online, but with the specific difference of them being basically allergic to trying anything not directly supported in GUIs.
I was expecting them to at least walk me through the same diskutil commands I tried above and get the same results, but they didn’t even do that. They basically saw the boot loop, saw that I couldn’t launch and it wouldn’t accept the password, and basically threw up their hands. So I kept looking.
Oh and one other great thing. They said I could send in the computer to be fixed, but the person I talked to said that Apple wouldn’t be likely to be able to fix it and would classify this as a “hardware” bug. They would then charge me $500 for a new disk drive, and send me back the computer wiped. OR they said I could take it to an Apple store, where the person there (very likely) wouldn’t be able to fix it, and then they would send it in to Apple, they would wipe my drive, and send it back.
What surprised me most about this whole exchange with them was what felt like a real lack of creativity in trying to solve the issue. I made it really clear that I was okay, as a last resort, to wipe the drive, but the fact they didn’t even try any more esoteric approaches kind of soured me on my experience with them. I wanted to be helped but they felt reticent to try anything.
Using fdesetup to change FileVault settings on the drive
This is where I got most creative, and what ultimately didn’t work. I read some forum posts around where people were suggesting that, in circumstance similar to mine, they were able to use fdesetup to basically reconfig FileVault or make a new FileVault user and login through that and fix the credentials for the other users once logged in. Sounded good.
Only issue: fdesetup isn’t available on Macintosh HD and is stored in the usr/bin location on the Data drive (for similar reasons I was unable to access any Keychain files). What I ended up doing to get access to fdesetup was mounting a drive that had a backup of another Mac from a while ago (like ~6 years ago), and accessing its fdesetup from the backed-up usr/bin/ location.
I was able then to run fdesetup from the terminal on the broken machine but… couldn’t actually get it to target the drive I had FileVault on. It seemed like the command was only running in the context of the drive it was a part of, and I couldn’t figure out how to use the device command to target a different drive, or if that was even the goal of that flag. Der Flounder also had a post on fdesetup that gave me hope, but it still seemed that, to effectively use this command, the drive already needed to be unlocked. For similar reasons, stuff like sysadminctl were also ineffective.
Closing thoughts before the fix
A big issue here is that, I’m just a lowly single-user with a personal laptop and my use case is unfortunately very very very close to: “someone who stole a laptop from a top-secret government facility and is trying to unencrypt a drive to steal state secrets”. FileVault is as close to military-grade encryption that an average consumer is going to get, and as such is not easily un-encrypted by design. I could even tell on some of the Apple support calls that there was suspicion that I was effectively trying to break into someone else’s laptop.
While this level of encryption is great and warranted for people who work on high-security-needs projects/teams, when this stuff breaks for an average user there are little-to-no-options, you’re effectively screwed. Even for someone like me, a tech-savvy-power-user-programmer whatever, the issue was incredibly hard to research to even find the wrong answers. The fact that this state was induced by a normal good-faith routine upgrade is… bad. Like really bad. Like worst-possible-scenario bad. A “normal” person tries to upgrade their laptop, only to find, sorry! You have to now delete all your data because you’re hopelessly locked out of your machine! Hope you had a backup! Crazy.
What Worked
When the fdesetup stuff was failing, I had given up. I opened up Disk Utility and was about to just wipe the data drive, assuming it was unfixable. But then I remembered some response from some support post where someone was like “I deleted the Macintosh HD partition and the Update partition and it worked!” I’m not even paraphrasing here (really). The post was that single line and I just wrote it off as a red herring, the person not understanding the issue, or just noise from someone who posted a solution to fuck with people. There was no additional info on if they even had the same issue or anything, so I just wrote it off at the time.
But with nothing left to lose I said “sure why not”. I deleted the Update partition and the Macintosh HD partition (not Macintosh HD - Data). The Macintosh HD partition, after deletion, re-created itself. I then tried to mount Macintosh HD - Data and…. it worked! What the fuck?
I had been reticent to delete these partitions before because, in the already fickle system state I was in, I was worried it would further destabilize my ability to even get into recovery mode. But like I said with nothing left to try I said why not, tried it, and it fucking worked.
My best guess at why this “fixes” things is that, when the Macintosh HD partition is deleted, the re-created partition is created using the most up-to-date credentials stored… somewhere? I’m assuming from the Data partition, as the system assumes the two are linked. What’s insane though is that it’s not like I used a different password for the mount. I used the same password for the drive, which would imply that if that password worked it should unlock the drive because presumably it doesn’t have to route the authentication through Macintosh HD then into Data? I have no idea.
Either way, deleting the Macintosh HD partition and the Update partition fixed the issue for me. I was able to then mount the data drive, and then promptly copied all my folders to an external drive I had mounted as well. This went off without a hitch. Computers!
For anyone else having this issue and suffering through all the same dead ends and promising posts from Der Flounder that don’t work for you, I hope this helps! Here’s some additional related posts that may give you other solutions to try before trying this (because deleting Macintosh HD may make it worse for you instead?)
https://iphoneteile.com/16571/Mac-password-not-working/
https://www.reddit.com/r/osx/comments/exte41/os_x_10153_update_trouble_with_startup_disk_and/
https://forums.macrumors.com/threads/error-unlocking-apfs-volume.2089475/
https://www.reddit.com/r/macsysadmin/comments/kj27ar/big_sir_failed_update_filevault_out_of_sync/
And many many more. These are just a few selects. What works for most people in a similar situation is just mounting the drive from terminal. If you’re unable to do that, welcome to my hell and I recommend you try the fix above.
Published on February 1, 2021.
Tagged: MacOS