OneDrive on mac is keeping two copies of all downloaded files

Anonymous
2022-07-14T09:40:57+00:00

Environment:

MacOS: 12.4 (21F79)

Mac filesystem type hosting OneDrive folder: APFS Data Volume: /dev/disk1s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)
OneDrive: 22.131.0619.0001 (Standalone)
OneDrive home account as part of Microsoft 365 Personal subscription
OneDrive folder: ~/OneDrive, symlink to /Users/mikaelkjaerulff/Library/CloudStorage/OneDrive-Personal
OneDrive “Files On-Demand” is off by selecting “Download all OneDrive files now”
OneDrive “File collaboration” is off
OneDrive status is “Your files are synced” and there are no warnings/errors
OneDrive storage: 134 GB used of 1.01 TB (as reported by https://onedrive.live.com/?v=managestorage)

I have discovered that OneDrive keeps 2 local copies of all the synced files.  So instead of taking up 134 GB of disk space it uses 268 GB – that can’t be right!  The (I assume) primary copy is in the OneDrive folder:

~/Library/CloudStorage/OneDrive-Personal/

And then there is the (I assume) secondary (or internal?) copy in:

~/Library/Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/

This is true copies of the files, not hard-linked files – I checked that it is different inodes:

> cd ~/Library

> ls -i CloudStorage/OneDrive-Personal/Gravhund/ActivitiesData.xlsx Group\ Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/ActivitiesData.xlsx
8965664803 CloudStorage/OneDrive-Personal/Gravhund/ActivitiesData.xlsx
8965664807 Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/ActivitiesData.xlsx

There are some differences. If I run a diff on the two locations, like:

> cd ~/Library

> diff -rq CloudStorage/OneDrive-Personal/ Group\ Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive | grep -v .DS_Store > ~/tmp/diff.txt

Note that I exclude Mac Spotlight files (.DS_Store) – Spotlight does not run on the secondary copy. I get:

> cat -v ~/tmp/diff.txt

Only in CloudStorage/OneDrive-Personal/: .Trash

Only in Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-01-14 ?M-^Ersoversigt.PDF
Only in CloudStorage/OneDrive-Personal/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-01-14 A?M-^Jrsoversigt.PDF
Only in Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-02-17 SEAS-NVE - årsopgørelse.pdf
Only in CloudStorage/OneDrive-Personal/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-02-17 SEAS-NVE - a?M-^Jrsopgørelse.pdf
Only in Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-03-22 Novafos - årsregning 2020.pdf
Only in CloudStorage/OneDrive-Personal/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-03-22 Novafos - a?M-^Jrsregning 2020.pdf
Only in Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-12-31 ?M-^Ersoversigt.PDF
Only in CloudStorage/OneDrive-Personal/Ejerforeningen matr. nr. 146 Hareskov By, Værløse/Billag/2021: 2021-12-31 A?M-^Jrsoversigt.PDF
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/IMPO/sed0.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/IMPO/sed0.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/IMPO/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/IMPO/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/KULD/sed0.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/KULD/sed0.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/KULD/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/KULD/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/PROV/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/PROV/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/PROV.KRITIK/sed0.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/PROV.KRITIK/sed0.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/PROV.PRID/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/PROV.PRID/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/RYGI/sed0.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/RYGI/sed0.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/RYGI/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/RYGI/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/RYGS/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/RYGS/sed2.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/UDST/sed1.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/UDST/sed1.txt differ
Files CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/UDST/sed2.txt and Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/UDST/sed2.txt differ
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk: index.htm~
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk: tilmeld.htm~
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk: tilmeld_ver.htm~
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk/traen: lydighed.htm~
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk/traen: ring.htm~
Only in CloudStorage/OneDrive-Personal/Gravhund/WWW/nsdgk/traen: spor.htm~

I see the following main differences:

  1. Symlinks in primary are converted into normal files with content of the symlink in the secondary.
  2. Some file names are converted (like ?M-^Ersoversigt.PDF to årsregning 2020.pdf).
  3. Looks like some emacs backup-files are not copied to the secondary location and not uploaded to the cloud.

(1)

 Symlinks are converted to normal files:

> ls -l "CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/IMPO/sed0.txt" "Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/IMPO/sed0.txt"
lrwxrwxr-x  1 mikaelkjaerulff  staff  18 Jun 17 20:01 CloudStorage/OneDrive-Personal/Gravhund/GravDB/ETL/IMPO/sed0.txt -> ../common/sed0.txt
-rwxr-xr-x@ 1 mikaelkjaerulff  staff  18 Jun 17 20:01 Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/IMPO/sed0.txt
> cat -v "Group Containers/UBF8T346G9.OneDriveStandaloneSuite/OneDrive.noindex/OneDrive/Gravhund/GravDB/ETL/IMPO/sed0.txt"
../common/sed0.txt

I do understand that it often makes little to no sense synchronizing symbolic links to a remote location as-is – they will often point to an invalid location.  But if the symlink is relative (like above), then it might work (if the remote filesystem supports the concept of symbolic links).  The real annoyance is that if you do a fresh download from the OneDrive cloud the symlinks are “lost” – which is the case when you do a reset of OneDrive.  The symlinks are restored as normal files with the content of the symlink.  As far as I can tell there is no easy way to find just these files and convert them back to symlinks.

(2)

Apple’s APFS file system encodes file and directory names using Unicode but is not a “normalizing file system”. I assume the secondary OneDrive copy does some normalization (from NFC to NFD) of file and directory names before uploading to the cloud.  At least that is the only explanation I can come up with.

Ref on APFS and normal forms:

(3)

I don’t understand why emacs backup files are not copied to the secondary and subsequently uploaded to the cloud.  I have a suspicion it could be due to the way emacs create the backup file.  Nevertheless, OneDrive does not pick up the emacs backup files – this must be a bug in OneDrive.

I could understand if OneDrive has the need for a temporary location (to normalize for example) for files and directories before being uploaded to the cloud – but with emphasis on “temporary”.  The secondary copies live forever – that can’t be right.

As a side note: Still OneDrive goes to 100+ % CPU for a minute or so every time I save a file in my OneDrive location.  At least the newest version of OneDrive does not cause kernel panics anymore (fingers crossed, knock on wood…).

What – if anything – can I do to avoid having the secondary copies?

/Mikael

Microsoft 365 and Office | OneDrive | For home | MacOS

Locked Question. This question was migrated from the Microsoft Support Community. You can vote on whether it's helpful, but you can't add comments or replies or follow the question.

0 comments

45 answers

Sort by: Oldest
Most helpful Newest Oldest
  1. Anonymous
    2022-09-28T18:39:11+00:00

    @Dearwolf,

    what I read so far from those advisors is ... simply wrong.

    But I did not follow all replies.

    The problem is, that Apple changed something, so that your "regualar" sync folder for OneDrive needs to be on the system drive, IIRC.

    Of course, normally people have this on an external drive - I for example don't have the required space on the internal drive and also want to spare IO to the internal disk,. as it cannot be replaced!

    Somehow, Microsoft thought it a good idea to circumvent this by a simply horrific contruct:

    (base) tja@mini:/Volumes/XTRMQ/OneDrive_TB1$ ls

total 16

-rw-r--r--@ 1 tja  staff  6148 Sep 18 23:34 .DS_Store

drwxrwxrwx  4 tja  staff   128 Sep 18 23:35 .ODContainer-OneDrive/

lrwxr-xr-x  1 tja  staff    49 Sep 18 23:35 OneDrive@ -> /Users/tja/Library/CloudStorage/OneDrive-Personal

    So, the "real" content seems to be in my Library for the OneDrive CloudStorage for OneDrive, as suggested by the SymLink. But indeed, in the ".ODContained-OneDrive" directory, you can find the real files! (base)  

    tja@mini:/Volumes/XTRMQ/OneDrive_TB1$ du -hs .ODContainer-OneDrive/OneDrive
332G	.ODContainer-OneDrive/OneDrive

    The directory in my Library SEEMS to have the same content, but this is not true:

    (base) tja@mini:~/Library/CloudStorage/OneDrive-Personal$ du -hs .

35G	.

    So, this "only" consumes 35 GB and not 332 GB!
    Even as the number of files is the same.

    This COULD be good new.

    But it is'nt - as I will write in a second post!

    Was this answer helpful?

    2 people found this answer helpful.
    0 comments
  2. Anonymous
    2022-09-28T18:50:20+00:00

    Now comes the horrific part, which you already noticed!

    As it seems, the regular IO INDEED is going to the internal disk first, as hinted by the SymLink into the Library.

    And then, OneDrive just copies the bulk of files to the "regular" synxc location!

    So, BOTH disks get the full amount of IO - the internal disk even double or tripple as much: Once when written to initially, then reading the files while copying them to the external disk and probably again when exchanging the files with those "stubs" that appear to be the full files but are not the real thing (35 instead of 332 GB)!

    So, this contruct does NOT save my internal disk from IO - to the contrary, it may well produce more IO to the disk than when I had the sync location directly on the internal disk.

    And yes, IF your sync location is on the internal system disk, it may well be that Microsoft uses exactly the same mechanism and DOUBLES or TRIPLES the IO to this disk!

    To be honest, this is a legal issue even.

    How I can be sure?

    Now, my Mac Mini M1 is not so old and I installed nearly nothing onto it, always using external disks instead.

    BUT:

    Critical Warning:                   0x00

Available Spare:                    100%

Available Spare Threshold:          99%

Percentage Used:                    0%

Data Units Written:                 33,109,790 [16.9 TB]

Host Write Commands:                269,912,806

Media and Data Integrity Errors:    0

Error Information Log Entries:      0

    YES, my little 512 GB internal disk has nearly 17 TB written to it!!!!!!!!

    Here is the history:

    mini:smart root# grep "Data Units Written" M1_202* | tail -16

M1_2022-02-25T01:54:23.log:Data Units Written:                 7,888,410 [4.03 TB]

M1_2022-02-25T01:54:23.sum:Data Units Written:                 7,888,410 [4.03 TB]

M1_2022-02-25T11:45:43.log:Data Units Written:                 8,631,543 [4.41 TB]

M1_2022-02-25T11:45:43.sum:Data Units Written:                 8,631,543 [4.41 TB]

M1_2022-02-28T18:34:36.log:Data Units Written:                 25,788,338 [13.2 TB]

M1_2022-02-28T18:34:36.sum:Data Units Written:                 25,788,338 [13.2 TB]

M1_2022-05-26T03:14:26.log:Data Units Written:                 28,007,427 [14.3 TB]

M1_2022-05-26T03:14:26.sum:Data Units Written:                 28,007,427 [14.3 TB]

M1_2022-08-28T02:37:18.log:Data Units Written:                 29,979,371 [15.3 TB]

M1_2022-08-28T02:37:18.sum:Data Units Written:                 29,979,371 [15.3 TB]

M1_2022-09-19T21:08:18.log:Data Units Written:                 32,558,298 [16.6 TB]

M1_2022-09-19T21:08:18.sum:Data Units Written:                 32,558,298 [16.6 TB]

M1_2022-09-19T21:15:32.log:Data Units Written:                 32,575,574 [16.6 TB]

M1_2022-09-19T21:15:32.sum:Data Units Written:                 32,575,574 [16.6 TB]

M1_2022-09-28T20:46:34.log:Data Units Written:                 33,109,790 [16.9 TB]

M1_2022-09-28T20:46:34.sum:Data Units Written:                 33,109,790 [16.9 TB]

    At that time, I was syncing and reordering several OneDrive accounts ....

    See the jump between 25. and 28. February 2022.

    Was this answer helpful?

    2 people found this answer helpful.
    0 comments
  3. Anonymous
    2022-09-28T19:03:01+00:00

    P.S. @Dearwolf:

    You can check, which files in the Library really are using disk space - at that moment:

    "du -ks" on the file

    Most will show 0 size, but that that take up the 35 GB at my Mac will show their real size!

    For example:

    (base) tja@mini:~/Library/CloudStorage/OneDrive-Personal/UNENCRYPTED/MEDIA$ du -ks vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

0	vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi

1732	EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    So, one file seems to be empty.

    But in the real location:

      (base) tja@mini:/Volumes/XTRMQ/OneDrive_TB1/.ODContainer-OneDrive/OneDrive/UNENCRYPTED/MEDIA$ du -ks vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

35728	vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi

1732	EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    Was this answer helpful?

    0 comments
  4. Anonymous
    2022-09-28T19:12:01+00:00

    I try to summarize the situation:

    Apple forced OneDrive to reside on the internal disk.

    Microsoft invented some caching mechanism that does exactly that, but while copying or writing data, they read this data again and write it again to the location where the OneDrive sync folder was configured - most probably on an external disk.

    After doing so, the "cache" on the internal disk will be deleted again - only holding this much data.

    And YES, this also means that BOTH disks are involved when just reading data!

    It may well be that part of the data is read from the "cache" on the internal disk, but if the data is not cached anymore, OneDrive reads it from the external disk and then writes it to the cache on the internal disk - probably to read it a second time when delivering to the application.

    Finally, this just means this:

    You can either use Apple or OneDrive from Microsoft.

    Was this answer helpful?

    2 people found this answer helpful.
    0 comments
  5. Anonymous
    2022-09-28T19:13:26+00:00

    And finally, to prove this:

    (base) tja@mini:~/Library/CloudStorage/OneDrive-Personal/UNENCRYPTED/MEDIA$ du -ks vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    0 vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi

    1732 EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    (base) tja@mini:/Volumes/XTRMQ/OneDrive_TB1/.ODContainer-OneDrive/OneDrive/UNENCRYPTED/MEDIA$ du -ks vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    35728 vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi

    1732 EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    Read it from the sync location:

    (base) tja@mini:/Volumes/XTRMQ/OneDrive_TB1/.ODContainer-OneDrive/OneDrive/UNENCRYPTED/MEDIA$ cat /Volumes/XTRMQ/OneDrive_TB1/OneDrive/UNENCRYPTED/MEDIA/vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi > ~/justtoreadthefile

    And then check the size in the "cache" (Library) again:

    (base) tja@mini:~/Library/CloudStorage/OneDrive-Personal/UNENCRYPTED/MEDIA$ du -ks vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    35728 vlc-record-2014-11-06-22h04m41s-MVI_0582.MOV-.avi

    1732 EXTRACTED_JPG/2013-03/IMG_20130304_222011.jpg

    Bammmm! The cache on the internal disk got updated (written to) ...

    Both reading and writing from / to OneDrives uses a caching system that involves both drives - for all IO!

    Of course, this also makes things slower - you are using a second drive as cache.

    Was this answer helpful?

    0 comments