Slashdot Mirror
Spotify Is Writing Massive Amounts of Junk Data To Storage Drives (arstechnica.com)
An anonymous reader quotes a report from Ars Technica: For almost five months -- possibly longer -- the Spotify music streaming app has been assaulting users' storage devices with enough data to potentially take years off their expected lifespans. Reports of tens or in some cases hundreds of gigabytes being written in an hour aren't uncommon, and occasionally the recorded amounts are measured in terabytes. The overload happens even when Spotify is idle and isn't storing any songs locally. The behavior poses an unnecessary burden on users' storage devices, particularly solid state drives, which come with a finite amount of write capacity. Continuously writing hundreds of gigabytes of needless data to a drive every day for months or years on end has the potential to cause an SSD to die years earlier than it otherwise would. And yet, Spotify apps for Windows, Mac, and Linux have engaged in this data assault since at least the middle of June, when multiple users reported the problem in the company's official support forum. Three Ars reporters who ran Spotify on Macs and PCs had no trouble reproducing the problem reported not only in the above-mentioned Spotify forum but also on Reddit, Hacker News, and elsewhere. Typically, the app wrote from 5 to 10 GB of data in less than an hour on Ars reporters' machines, even when the app was idle. Leaving Spotify running for periods longer than a day resulted in amounts as high as 700 GB. According to comments left in the Spotify forum in the past 24 hours, the bug has been fixed in version 1.0.42, which is in the process of being rolled out.
Bandwidth, memory, clock cycles....don't matter. Use more shitty layers of abstraction over layers built into high level languages, then kick it out the door.
...sounds like they are using clients as storage for their streaming service? have any lawyers scoured the TOS for any details?
Problem solved.
This is a non issue on desktops, really.
And if you lug around a laptop to listen to Spotify, I guess you have more important things to do on it than listen to music anyway.
The only device where I save songs offline is my smartphone, for when I'm out and about without 4G coverage.
... for highlighting the potential for damage as news, don't ya think?
Spotify has a side-business in distributed storage?
Can be avoided more by moving pagefiles &/or %temp%/%tmp% & browser caches to a 2nd disk (preferably HDD or a software ramdisk for example that don't have as limited lifecycles on writes).
* I put both of those onto a Gigabyte IRAM 4gb unit SATA 1 hardware RamDisk (DDR-RAM) so I also do not lose speed (even though I keep a Western Digital Raptor 10,000 rpm HDD for backups off of a Promise Ex-8350 128mb ECC ram caching raid sata 1/2 controller (SATA 1/2))
APK
P.S.=> That avoids a LOT of 'excess writes' onto my primary SSD (Intel 530 240gb Flash SSD (SATA 6))... apk
Pagefiles I don't put on software ramdisk (had to clarify that), but on HDD instead & print spool location can be moved too along with APPLICATION temp areas (some apps let you do that such as WinRar for example).
* To get more "on topic": I cannot believe some remote service is doing this... I wonder what for? On a guess - advertising purposes.
APK
P.S.=> Sorry about that - haven't had my coffee yet this a.m. so here I go for that... apk
If you're writing enough to pagefiles, you need more RAM anyway.
If you're writing a lot to temporary areas, you need to stop doing so.
That said, I'm on an SSD machine at the moment that has been running for 6 months, with absolutely no special treatment, imaged from a years-old working PC without changing anything, and it's written 1.5TB. 1TB of that was the initial imaging process.
It's the main workhorse in an IT Office in a school, use for 10+ hours every single day for everything imaginable. Client machines rarely use much.
It has a write-life of 100TB. If it dies, I just hit F12 and re-image cleanly.
At current usage (not including the initial image), I count that as 1TB of write a year, which gives longer the expected lifetime of the PC itself, however far out I am.
There's no need for special treatment, no need to use special SSD transfer software, no need to over-provision, or increase RAM cache or anything else. Just have a PC that isn't slogging itself to death, and slap an SSD in.
Don't expect it to last forever, but you shouldn't need to adjust ANYTHING at all.
And I've done this on all the staff work machines earlier this year - zero failures so far and it has made much more of a performance difference than doubling the amount of RAM. In fact, where machines had motherboards that were limited in RAM, we SSD'd and saw HUGE performance increases better than those clients whose RAM we doubled but are running on traditional hard disks.
At home I have a 1TB EVO 850 and that's the same. Literally imaged byte-for-byte, and is stupendously fast and no need for any software changes whatsoever, and the write numbers are predicting 20+ years of life despite a similar 10+ hours a day of usage.
Don't RELY on it never failing. But they are going to be in warranty (whether that's by number of years, or data written) for the life of your machine, under even heavy usage, unless you're doing something incredibly stupid (like use in NVR, RAID, or similar without buying a high-write-endurance model).
See subject & there's more you can move onto another disk too (print spooler location, app temp areas) https://hardware.slashdot.org/...
* I should've noted that in my original post but I am still "out of it" a bit from sleep...
(Would've posted it sooner than you posted but I have time limits between posts as an AC poster & can only post 5x a day or so typically...)
APK
P.S.=> FINALLY I can have some coffee (lol, I need it today) as I just woke up about a 1/2 hr. ago - brain needs caffeine today... apk
This sounds like some smart software architect to the abstraction of the persistance/storage layer of the Spotify stack too far whilst at the same time storing to much of miniscule datapoints in Spotifys objects. Because once abstracted properly, adding attributes to your objects and the entire stack is trivial.
Think of it:
If your stacks ORM neatly abstracts everything concerning persistance and on the backside syncs on neatly whenever it has the opportunity, all you need is app-side developers and software designers storing every little piece of data they can find and that changes evers millisecond and then you have your bandwidth/load disaster as described.
If something like this is the case with Spotify, which I do strongly suspect, it is a good example that goes to show that you can take clean-room design too far. And that a haphazard duct-tape and chickenwire approach to product development can have significant advantages, as you build around unforseen roadblocks on a daily basis and only add the features really needed.
I see an example of this every day, as I am currently doing WordPress development and building a WordPress pipeline for an agency. Large parts of the WP legacy architecture are an abysmally convoluted mess built by people who shouldn't have been let near a keyboard 15 years ago. But having a non-developer build a production capable demo of a website in WP is significantly faster than starting with an actual UX prototype, which quickly leads our team into real-world problems that we often haven't suspected. And suddenly a proper ORM and cleanroom design would cause hassle at one end or the other.
My to eurocents.
We suffer more in our imagination than in reality. - Seneca
Generally there is no reason to do that, but there are some poorly coded applications that will page memory to disk, even when they don't need to.
Is it just my observation, or are there way too many stupid people in the world?
I use Google Play Music. Not only can it cache songs, you can also upload your own collection. And now that Google has acquired and integrated Songza, their playlists are awesome.
lucm, indeed.
See my original post you replied to for verification & also my "addendum" link that notes more you can move https://hardware.slashdot.org/... off SSD to help avoid 'finite write life' on SSD.
I posted it before you posted YOUR reply actually but like the other poster who noted what you did? You, like myself while I was doing that addendum, must have been composing yours & you did not get to see it in time is all...
ADDITIONALLY: If you don't require them? You can remove NTFS time/date stamps, Win3.x/DOS 'shortnaming', etc. - et al $MFT writes too... I've written all of this (probably more) a decade++ or more ago on other spots online but I thought I'd put it out here as it CAN help vs. this stupidity being done (& against finite SSD write life too of course). This can also help the SSD's in question's performance a little bit too (bonus) by NOT performing those writes.
(Sorry about that but I corrected that 'grammatical' (? don't know what you'd really call that other than not clear enough) error)
APK
P.S.=> It's all 'corrected' & noted long before your reply which I largely agree with (on MOST software ramdisks etc.)... apk
(Sorry about that but I corrected that 'grammatical' (? don't know what you'd really call that other than not clear enough) error - upon reviewing it, I noted it, & had to clarify it in the link above...)
APK
P.S.=> It's all 'corrected' & noted long before your reply which I largely agree with (on MOST software ramdisks etc.)... apk
This is just Appify being an appy app app, unlike LUDDITE software that doesn't even know what an app drive is!
Apps!
I place my pagefile o a "True SSD" as I call it based on DDR Ram (Gigabyte IRAM 4gb SATA 1) vs. software ramdisks (which I also use for browser caches to 'flush them' @ shutdown). I noted that in my original post in fact so see subject...
This post was in the interests of avoiding 'finite write life' on Flash SSD is all for those NOT aware of this possibility to extend the life of their investments in Flash based SSD hardware.
You can STOP writes of this nature also as I've told others as well in NTFS:
Win3x/DOS type 'shortnaming' being done in $MFT
Time/date of access stamping
print spooler location
App level TEMP areas (WinRar as an 'e.g.' allows this)
As well...
APK
P.S.=> This is ALL covered in my other posts to others replying - but, for once, I am GLAD you're 'nitpicking' folks (it shows some here are aware of this possibility to avoid Flash SSD finite write life).
In fact - I noted that I covered this in PC performance guides for Windows I'd done almost 2 decades prior to this post @ other spots online, but those are NOT here so, here is some ideas from them that aid this problem is all... apk
See subject - I place mine on a software unit (or my "True SSD" noted) by registry location alteration: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters by changing the "DataBasePath" parm.
* Helps File seek cycle of Open/Read/Close portion of access by being faster vs. HDD.
Vs. malware too (hosts won't update IP stack vs. change until hosts changes under %WinDir%\system32\drivers\etc
(Functions quite like *NIX shadow pwd file moving it from std. location & protects TRUE hoss location - they can try alter the original but the 'shadow' matters & you have moved it to a FASTER area too)
APK
P.S.=> True hosts never gets 'hurt' assuming malware maker didn't read reg (many don't & it can't be gotten to in usermode via my APK Hosts File Engine 9.0++ SR-4 32/64-bit https://www.google.com/search?... - goes above & beyond Windows ACL/SFP/WFP protection)
Here is a possibly related complaint from almost three years ago.
I don't typically place pagefiles on software ramdisk - I do however gain via NTFS compression on temp ops, print spool, hosts, browser cache though (another point I forgot to note - ugh!) - you can put more of that temporary junk onto them (that also includes app temp ops also like you can do in WinRar).
HENCE why I said "preferably HDD" in my initial post & why I did my "addendum correction" clarification post after my original post.
* Lots of folks are 'nitpicking' my post but that's why I put out my 'addendum/correction' post to CLARITY it's preferable to do paging on HDD vs. software ramdisks (if I don't be SUPER concise, these nitpicks always start (which is good in a way (shows folks are reading my posts), but also bogus imo - especially when they overlook I used the word PREFERABLE on HDD vs. SSD for paging))
(Good point on compression in filesystem usage on your part though - I omitted that - performance guides I've written decades ago have it though should anyone question it)
APK
P.S.=> It's a LOT of detail in this area & I can't even fit it all in 1 post as AC plus admittedly I omitted other areas in my original post (but I do list them in subsequent posts) that gain by this technique for BOTH performance (seek/access + smaller file mass off disk (cpu makes up for decompression cycle) & even security to a degree!
Man, above ALL else - I really, Really, REALLY need that coffee I noted today (but I also can't put all the detail noted into my posts as /. places size/length limits on AC posts)... apk
I recently cleaned up the Windows installation drive with the cleanup tool. 3.27 terabytes of Windows Update-files, according to the tool. That's 13 times the size of the disk where the data should be stored.
Where is this data coming from? And how do obvious bugs like this ever get out of QA ( yes, i know the answer, no testing, and many coders today are sloppy )
See subject - I do for seek speed https://hardware.slashdot.org/... + even security to a degree https://hardware.slashdot.org/... in my other posts that cover even more benefits of using alternate disks (especially software based ramdisk OR even hardware RamDisks as I use too), other areas you can move to other drives to offset finite writes on Flash SDD & gain speed + even security!
* NTFS compression use for speed gains is also noted here https://hardware.slashdot.org/... & why it helps (smaller filemass & cpu speeds offset decompression speed in RAM once files are read up off disks of ANY kind w/ today's disks of most all types).
APK
P.S.=> Why don't YOU guys contribute good ideas here (or don't you have any? I suspect you don't) vs. attempting to razz me by unidentifiable ac posts instead? I'm trying to inform/help, you're not so GROW UP please... apk
Or you could try to maximize the drive life and reuse it until SATA is replaced by something else.
what if?
If you leave browsers up all all the time, they have the same problem. Firefox and Chrome. https://www.grc.com/sn/sn-580....
You can lose something that is loose, so tighten the loose item so you don't lose it.
See subject: It's all I've got for hardware SSD (haven't seen a SATA 2 or better unit & gigabyte doesn't make one), so I "make do" w/ what I have to work with is all for the reasons in my subject & other posts. ... & yes, of course, for offsetting temp, cache, printspool, browser cache writes OFF of flash SSD (or HDD slowing them up too mind you) this article states occurring.
* Combining NTFS compression (for security mostly but also for storage & some speed gains (smaller filemass to read up off disk & today's fast cpu's offset decompression of file once found + read into ram for actual use) with it allows me to place more data onto it also (not in the case of pagefiles though - that's NOT compressible afaik from NTFS4-5 @ least).
(The 'nitpickers' amaze me - I wrote preferably HDD in my initial post vs. SDD, but as usual, I knew I'd be f'd with so I did a clarification post https://hardware.slashdot.org/... of that 'issue' here ontop of that as well vs. nitpicking trolls... especially the usual unidentifiable ac ones that harass me here nigh constantly!)
APK
P.S.=> Would I like say, a DDR-2 or better RAM based one on SATA 6 (or better yet, NVMe, PCIex etc. w/ less bus constriction) as "real RAM" vs. Flash doesn't have this issue in this article as Flash SSD does?
Sure - know of one? TIA if you do... apk
useful programs and operating systems roving the interwebs, forcing themselves to be installed on your device, and doing horrible things to it, with people clamoring for this to happen as it gives them some tiny trinket of pleasure. Wait, isnt that what social media programs are anyway, but in that case its people doing social hacking (peer pressure) to get the programs installed. humans are now merely bots used to infect other humans so they use shitty programs.
users in that thread are reporting that even with the new version the issue persists... "I do have 1.0.42 installed. Still writing stupid data. "
Seems developers don't consider to optimize disk I/O. Recently I saw a live event streamed using firefox (from a not so great website, i guess it uses flash) and it kept my disk 100% all the time. Why should a streaming service write all those video data into disk, can't it just cache in RAM n display n forget the bits?
Such unnecessary disk i/o wears my disk down, increases power use (if I'm on say battery on my laptop) and of course creates a kind of internal DoS as it hogs the disk i/o and rest of processes can't get disk i/o or get delayed -- resulting in a sluggish OS response even to say some file explorers. ie a well behaved app/software should not hog any shared piece of hardware/resource (like disk-io) leading to system instability.
Apps should be benchmarked not only on their memory foot print or CPU usage (like algorithm/big(Oh) s) but also on their external data traffic usage like disk/network i/o.
I regularly watch my disk i/o usage by processes and get rid of any if I suspect they are hitting it unnecessarily hard.
No OS at the time was written in BASIC. That would gave been FAR too slow and used too much memory to be feasible, and all for no benefit. The OS back in those days was written in assembly language. Those old machines did boot up to a BASIC interpreter prompt, but this is no different than Unix which boots up to a terminal with a Bourne (bash nowadays) shell interpreter prompt. The interpreter is not the OS.
Spotify Is Writing Massive Amounts of Junk Data To Storage Drives
Or are they talking about the music files?
It must have been something you assimilated. . . .
I don't & see subject: ...PREFERABLY HDD "funnyman"? I did a clarifying post vs. 'nitpickers' https://hardware.slashdot.org/... , LIKE YOU, to clarify that!
* You can't bury that clarification of mine (vs. ADHD/ADD literal illiterates) OR your general illiteracy (I suspect on purpose) & as far as THIS material goes? I know I've done posts on it while you were in diapers I strongly wager!
APK
P.S.=> Yes, I know - next you'll either downmod me via your alternate sockpuppet accounts OR post so much you bury my stating this (trying to "hide" these facts)... apk
See subject: Answer that question & prove it (good luck under your 'fake name' to hide your fake "ne'er-do-well" life)!
I doubt you can code let alone something as useful & ubiquitous as APK Hosts File Engine 9.0++ SR-4 32/64-bit https://www.google.com/search?...
* LOL, all "your kind" does, is troll under fake names online!
(Lastly - there isn't better hosts files vs. mine - mine's built from most of ALL the best!)
APK
P.S.=> IF you do (under your fake name? LOL, impossible to backup)?
Then I'll challenge you to show you've done more of my partial ONLY small list of favorites I have (that I accomplished LONG before you were out of diapers most likely) & we can "continue to compare notes" on that account!
Watch this be downmoderated, troll posts usually by unidentifiable ac 'hidden' to bury it (weak, lol, everyone sees them here anyway as most browse below -1 easily cheated misused downmod stupidity here))... apk
Just chaff to combat DRM.
That will teach 'em. It is absurd what programmers can do and get away with it, simply because you click on "I agree" on EULA starting with "No warranty"
Yes, it is going to be expensive, but software is getting worse.
It is possible to use Spotify in the web browser... this would stop it from writing too much to your drive.
On linux my /tmp is a tmpfs that lives in RAM, so it never touches the disk (I also have no swap).
Throw in uBlock Origin, and you dont have to listen to the ads either!
maybe im old school but i create m3u8(playlists) files with address of music streams (seperate files for each address) just using vlc but you can use just about any player that can play remotly.
Chrome and Vivaldi also does this, possibly due to some extension I have (Adblock Plus and BankID mostly.)
Three times at-least they have written away 1 TB of data thanks to messing with swap I guess.
I wish there was some information when the load got high / lots of data was written. Need some program for that. (Samsung EVO 850.)
Nobody has mentioned the temp fix yet so
On OS X, Open /Applications/Spotify.app/Contents/MacOS/Spotify in a hex editor.
Search for "VACUUM;" Replace with "xxxxxx;"
Once you apply that fix you can manually vacuum with
"~/Library/Application Support/Spotify/PersistentCache/mercury.db"
download sqlite3 from https://sqlite.org/download.ht...
See subject: Compressed pagefiles = impossible since they are 'raw written', & even if on an NTFS partition, they're almost like their own partition (that can't be compressed).
APK
P.S.=> I've noted it in my other posts since believe you me, IF that was possible? Software based ramdisks on NTFS would make sense to try do - almost like 'doubling' RAM (well, windows compression due to legacy of other cpus isn't the best compression but it works)... apk
See subject: You can't - it's a natural operation like browser caches (almost) & moving pagefiles, caches, temp ops, print spooler OFF the flash based SSD reduces writes on it & thus, increases its life (look @ the point of this article as proof) AND increases performance on the disk you moved them from - why? LESS WORK DONE!
YOU CAN EVEN MOVE EVENTLOGS OFF THAT DISK TOO - that's one I've missed (from their default locations) & iirc, it's possible to move the registry itself off a disk & move it to another!
Removing NTFS access stamps/Win3x-DOS shortnames etc. ala:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"NtfsDisable8dot3NameCreation"=dword:00000000
"Win31FileSystem"=dword:00000000
"Win95TruncatedExtensions"=dword:00000001
Helps increase SSD life as well + increases NTFS filesystem performance (not performing those writes to $MFT).
APK
P.S.=> How the HELL you got upmodded to +4 for your post astounds me (sockpuppets galore self-upmodding imo)... I've been using NT since version 3.5 & KNOW this all works (as well as programming + network adminstrating it)... apk
In fact I doubt any of you guys do so your options is HDD or software ramdisk & yes I have multiple pagefiles (2gb on IRAM & 512mb on a WD 10,000 rpm Raptor driven off of a Promise Ex-8350 128mb ECC ram caching raid sata 1/2 controller)
* Windows can use multiple pagefiles...
APK
P.S.=> ... & "There ya go"... apk
You try to be legal and buy your music and videos, but there are always things fucking you around. Warnings on dvds about piracy. You just cant press play. Music services injecting ads, and killing of disk space...bizarre limitation on how long tv and movies stay on streaming services. Ads even AFTER you pay, and then they advertise ad free services that you pay extra for.. AND STILL GET ADS.
Piracy will continue while its more convenient to do so.
See subject: The Gigabyte IRAM != software ramdisk or HDD fool - it's a ramdisk card with 4gb RAM on it & yes you can have multiple pagefiles (primary is on IRAM here).
APK
P.S.=> You lose/fail - but that's just what "your kind" does, lol... apk
See subject: So, learn to accept your SELF-defeat w/ grace stupid -> https://hardware.slashdot.org/... and learn the language also dumbass!
* What I love the best, is that the NEXT TIME you troll me w/ your stupid bullshit I will not only KNOCK YOU FLAT OUT AGAIN as usual vs. "your kind" (see subject) but I can toss this right back @ you, again... lol!
APK
P.S.=> Thanks for being SO stupid... apk
See subject & https://hardware.slashdot.org/... everyone (have good laugh).
* I don't take FAKE NAME ONLINE using do nothing "ne'er-do-wells" like YOU seriously @ all - "Your kind"?
They're just (& yes, I've gotta say it as per my inimitable style) "too, Too, TOO EASY - just '2ez'" to BLOW AWAY w/ YOUR OWN MISTAKES shown in the link above...
(Especially when they don't accomplish squat, get shot the F down easily for their fuckups (see link above, hahaha) & can't ADMIT THEY SCREWED UP (like you CLEARLY did)).
APK
P.S.=> Poor lil' douche lucm... lol! apk
I don't take FAKE NAME ONLINE
Yes you did just that in another thread, pretending to be someone else and linking back to this thread. Unfortunately your unique way to express yourself betrayed you. Next time try to write full sentences and don't constantly refer to the titles of your posts if you want to conceal your identity. The fact that you're probably one of the only persons on Slashdot who frequently posts links to other comments also was an obvious tell.
lucm, indeed.
See subject: You use a delusional FAKE name online & that's the proof - got proof of YOUR accusations? No - you lose.
* Is "lucm" even YOUR INITIALS? Again, no - More proof of your "issues" (to put it lightly) as you're operating under a DELUSIONAL FAKE NAME online... lol!
I post by AC (I don't need the cookies & javascript you do so I am not trackable AND I go faster minus them - you also operate inefficiently) but ID myself by my initials in my posts (everyone here knows that).
APK
P.S.=> Your BIGGEST LOSS was your MASSIVE LAUGHABLE SCREWUP here -> https://hardware.slashdot.org/... which I pointed out RIGHT in that link (classic, hilarious, & priceless), lol... thank you for being SO stupid! apk
You're no longer fooling me with your torrent of babble and bragging. Skate around it as much ss you want, but twice in this thread you've been caught lying, and now you've also been caught pretending to be someone else in other threads while waging your little vengeful campaign.
You're not merely the excentric techie people assume you are. You're a dishonest, scheming individual that just happens to have a hard time expressing himself succinctly and clearly. I'm disappointed, it's like finding out that the joyful greeter I see every week at the department store is a convicted sex offender.
lucm, indeed.
See subject: You FAIL then as you did here HUGELY https://hardware.slashdot.org/... (what a blunderer you are!)
* No 'small wonder' you use a FAKE NAME online (to cover your FAKE LIFE being an online "ne'er-do-well" do-nothing KNOW nothing mere puny troll...)
APK
P.S.=> You blew it twice & the proof's above - it's no lie, it's easily verified that YOU FAILED TWICE, lol (UNLIKE your accusations you can't prove)... apk
See subject: I am a liar, you can see it https://hardware.slashdot.org/... (what a bad person I am!)
* Slashdot I ask for your FORGIVENESS (to those who have LOVE in their heart and KNOW everyone can make MISTAKES...)
APK
P.S.=> I blew it twice & the proof's above - I lied, it's easily verified that I FAILED TWICE, lol (UNLIKE you who were right)... also it's OBVIOUS my method for identifying myself is NOT FOOLPROOF is it... apk
See subject: That's the "best ya got" after YOU BLEW IT here lucm (you're butthurt proving you know you blew it) https://hardware.slashdot.org/...
APK
P.S.=> Unbelievable - what a butthurt little punk you are lucm... apk
See subject: I sold my IRAM, I have a MBP now. Sold BY APPLE (which does not has RAM because they use GORILLA GLASS from Qualcom @ 2.33GHZ per minute).
so YOU LOSE, my MBP != IRAM and again you BLEW IT while I use iTunes for AUDIO and VIDEO processing w/GORILLA GLASS.
APK
P.S.=> IOS on MBP does NOT need pagefile because of the USB-C motherboard, if you new COMPUTERS you would know that. lol... apk