Slashdot Mirror
Large-Scale Mac Deployment?
UncleRage writes "I've been asked to research and ultimately recommend a deployment procedure for Macs across a rather large network. I'm not a stranger to OS X; however, the last time I worked on deployment NetRestore was still king of the mountain. Considering the current options, what methodologies do admins adhere to? Given the current selection of tools available, what would you recommend when planning, prototyping, and rolling out a robust, modular deployment scenario? For the record, I'm not asking for a spoon-fed solution; I'm more interested in a discussion concerning the current tools and what may (or may not) have worked for you. There are a lot of options available for modular system deployment... what are your opinions?"
that is a whole lot of gay to be rolling out
Is there even such a thing in this world? Folks like to disparage Windows, but it really is the only OS built for very large enterprises. Linux solutions don't really compare to Windows solutions - there, I said it...
I have had great success out of both DeployStudio (http://deploystudio.com/) and LanREV (http://www.lanrev.com) in K-12 schools with 200+ machines.
Man, I'd update your resume because they won't need you anymore. Or, insist that some MS products are still around because of ... of...email ...no...um...well, that's your problem.
It's NOT me! It's the meds! I'm on 1000mg of Fukitol.
Check out the following:
http://www.macenterprise.org/
http://www.deploystudio.com/Home.html
http://rsug.itd.umich.edu/software/radmind/
Guess what? It would be you, not the Macs. I'd have fired you for wasting the time needed to tear a display apart instead of sending it to the manufacturer to be repaired.
.. of OS X server? It doesn't require client access licenses like Windows server versions do, and many of the services seem tailored to providing the best administration possible for an OS X network. I don't have any personal experience, but that's the first place I'd look if I had to admin an entirely OS X network.
You have two choices in general on the Mac side:
-- UNIX-y utilities, usually on the command line and a bit crufty in places, but free and nicely configurable
-- Mac-type utilities with marvelous interfaces that will probably set you back a nice chunk of change
When I was in the business, we used Carbon Copy Cloner, but g4u, Remote Desktop 3, or just plain old rsync are all pretty good bets depending on what type of imaging you're planning to do. CCC actually has one foot in both of the two camps I just described.
Of course, I even remember the crusty old days of Assimilator.
....fucking Apple Cinema Display
Damn! Is there a video? I tried googling "apple cinema display fucking" and "apple cinema display porn" and nothing.
So, what was it fucking? The DVD drive? or the USB port?
It's NOT me! It's the meds! I'm on 1000mg of Fukitol.
If you post on slashdot a question on the best way to deploy lots of Macs, all you'll get is trollish comments from pre-pubs.
Really. It's the car equivalent on asking how to adjust the stock Caliber SRT4 wastegates on a Honda Civic SI site.
For real answers, check out System X. The hardware FAQ and history links will provide lots of useful info.
For initial deployment, Deploy Studio: http://www.deploystudio.com/
For authentication and settings management, use OpenDirectory.
For ongoing control and user support, use Remote Desktop (from Apple).
For a more advanced option, try Radmind to keep the Macs in sync: http://rsug.itd.umich.edu/software/radmind/
I don't know anything about their deployment procedure, but here at Virginia Tech the Math Emporium has over 500 macs set up for student access. The courses I've had there have been boring, but the actual setup of the place is pretty neat.
And the masses cried out, "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0!"
Taking it apart yourself is worse than paying somebody else $400/hr to take it apart for you?
Check out the Mac management software from JAMF software. It pretty much covers it all, from package management to image deployment to remote desktop to inventory. Used in many mac-based school districts and Universities.
First we build and test a good image on a machine for a couple of weeks.
Then we either use that image,if it was correct the first time, or build a new one from it if it required touching up.
We use Apple's free Disk Utility which comes free with all macs.
We then get about 10 - 15 firewire drives and copy that image on them. (You have to make sure the drives are bootable, you can actually deploy that same image onto the drive itself.)
Then we line up 10-15 machines and use again the Disk Utility to image them.
Depending on the size of the image, just about the time you have the next 10-15 unboxed and set up (very easy to do since they're all all-in-ones), the first batch is ready.
Works for us, but then again, our schedule is flexible and we can afford a couple of days of leisurely imaging.
Oh, yeah, and if you do have an image you can also work with Apple, they'll preload it on for you.
If you don't know what AltaVista is (was), get off my lawn.
Here is an excellent resource (at least last time I checked and it has been awhile, they used to be called macosxlabs.org). http://www.macenterprise.org/ As far as tools, the built in tools are very good. A third party tool that can be very useful for bootable drive images is Carbon Copy Cloner. When you say large, do you mean hundreds or thousands, or less? It will definitely change things for you. I think that you will be surprised by both the ease of the transition, and the things that should be easy that are not. Really I don't know how we can help you unless you have specific areas where you are interested in learning solutions (and I don't say that to be a jerk, I'll try to answer questions where I can). How many servers? Directory Server? File Sharing? Exchange Server/POP/IMAP? Calendaring? Centralized home directories? Budget per user? Of course there are cool things that cost money and are not really needed, and hard things that are cheap but work well once set up etc. I would help more, but I don't know where to start... take a look at the link above, and ask questions as you get a better idea of he scope
I like you, developed deployment for a mac based network (600 or so macs) back when command line ASR and netrestore were the best options. However, we also upgraded our deployment methods as Apple incorporated some of the technologies we used (cloning and automatic install options) into their server software. Today that particular piece of software is very well polished and does the job extremely well. The last time we did an installation (a few years ago) we used custom netboot images with automatic install options for different types of computers (lab, classroom, etc.) based on mac address. At the time we used a third party unix package manager or OS X called Radmind, but it proved to be more trouble than it was worth. However, Apple Remote desktop's package management and monitoring work very well and lets your do most of the upgrade install tasks you need to. In the end, the only per-machine work was setting up the machine to boot from the network by default.
Also, if you have the bandwidth, you can centralize your OS installs as server based images that are never installed on the thin clients. If you get it to work, it makes upgrades and deployment very easy.
If you want to discuss some of the problems we faced and our solutions, please feel free to contact me.
Science will save us. The question is, will it destroy us first?
400 dollars an hour?! What are you using? Lawyers? How does that work?
1) Monitor breaks
2) Sue Apple
3) Free monitor?
For justice, we must go to Don Corleone
I used to run a network with hundreds of apples with radmind. We installed the initial images with NetRestore (multicast for the larger influxes), and upon reboot, the computers would download their radmind certificate from LDAP and install all of the software that it needed.
It takes more up front time to set up and configure radmind, but it works wonderfully for almost anything you want to do.
We use DeployStudio, a freeware project http://www.deploystudio.com/ . Support for DS is pretty from the community, or you can buy training, but if you want to go with a vendor product JAMF Casper suite makes a great product, that we did not think was outrageously expensive.
Apple has a robust remote installation suite with OS X Server, which is darn cheap compared to most other commercial offerings.
10.6 includes a first party version of NetRestore (full system image deployment, similar to Ghost or Flash Archive on Solaris), but most people deploying across a large number of systems should roll their own images with packaged based tools like DeployStudio or InstaDMG:
http://www.deploystudio.com/
http://code.google.com/p/instadmg/
Some other good sites for finding info:
http://www.afp548.com/
http://www.macenterprise.org/
Try asking this on serverfault.com. Lots of advice can be found there.
It's been mentioned a couple of times, but mostly with -1 scores, so it's easy to miss: Radmind. It's a very powerful deployment tool with a totally transparent mechanism so you can tweak it to do *exactly* what you want in terms of mucking with files on the disk. I've seen people complain about it being hard to use, but I thought it was pretty straightforward -- install an app, run the change detector, tweak as desired (if at all), build an app image, deploy.
http://rsug.itd.umich.edu/software/radmind/
Stupid post.
2 would never happen and would cost WAY more than 400 bucks in time alone.
Get Applecare and it's covered for 3 years. Ship it back to Apple while they fix it. That's what we do.
- Zav - Imagine a Beowulf cluster of insensitive clods...
Why on earth is this being asked on Slashdot? Head to afp548.com and macenterprise.org (particularly its mailing list). You'll find info on InstaDMG, DeployStudio, even Radmind.
This is a Big Mac deployment? Sounds like a job for my tummy!
Apple Software Restore, which comes "in the box". We set up a base machine, populate the /System/Library/User Templates/English.lproj/ and then make a disk image to our fileserver using ASR. Then, boot new machines in Target Disk Mode and deploy the image using your workstation.
We could probably come up with something clever using a boot partition, but this works fine for us. If you want to get fine-grained, have a look at Radmind but keep in mind that Adobe apps will thwart your every attempt to manage them at that level.
All of the above are Free/free. We handle patching using Apple Remote Desktop (not free, but well worth the money). You can also configure your machines to authenticate against an Active Directory (like we do); if you're willing to modify your schema, you can even manage your installation from your MMC snap-ins like you can with Windows boxen.
Open Directory
By centralizing information about users and network resources, directory services provide the infrastructure required for managing users, groups, and computers on your network. Directory services can benefit organizations with as few as 10 people and are essential for enterprise networks that have thousands of users. Deploying a directory server helps reduce administrative costs, improve security, and provide users with a more productive computing experience.
Remote Desktop
Apple Remote Desktop is the best way to manage the Mac computers on your network. Distribute software, provide real-time online help to end users, create detailed software and hardware reports, and automate routine management tasks -- all without leaving your desk. Featuring Automator actions, Remote Spotlight search, and a new Dashboard widget*, Apple Remote Desktop 3 makes your job easier than ever.
* You'll notice Open Directory has no Dashboard widget. It's because it isn't uniquely Apple and therefore isn't polished to a blinding shine.
You're likely to get some laptops in addition to desktops. Get yourself a large room, a dozen or more firewire cables, power strips together. Before the machines arrive, use a macbook pro or macbook (a laptop) to develop your base image. Install all software on it that is going to be on most of the machines. Test thoroughly. Be sure all your remote access is tested. (ARD/SSH)
Use netrestore to create the base image. When the computers arrive, copy the base image to a group of laptops, with netrestore app. The number varies depending on how many computers you are going to be imaging, the size of your base image, and how much help you have. 8-12 is typical if only one person is going to be restoring.
First thing you should do with machines out of the box is label them, have labels made up in advance. Then set them all up imaging over firewire, just get an assembly line going. You CAN do netrestore over the network, but it's been my experience it's less reliable. (machines randomly fail to restore, sometimes entire groups fail at an annoying 99% etc) Firewire is usually faster anyway since your fileserver or switch is very unlikely to be able to keep up with imaging a dozen at once. FW800 imaging is an amazing thing.
Once machines are imaged, there should be a folder of scripts sitting on each machine's local admin acct, one for each group of machines. The script will prompt for computer name and run. When run it will rename the computer and delete all the apps that should not be on that particular image. This can also be done by running the script remotely over apple remote desktop. If you don't have ARD, *get it now*. It will save you incredible amounts of time. Using this removal script method adds only a few minutes of time per image but you're doing them in parallel so its negligible, and saves you the major headache of managing a half dozen different base images.
As long as you made the image on a laptop, it should have full hardware support for the camera etc. Different images are required for PPC, but fortunately that's not a headache you have to worry about. (I did, PAIN)
Boot camp adds a level of complexity, requiring you to partition the hard drives before restoring to them, and then using something like Ghost or Acronis. One person can image between 40-80 machines in 8 hrs depending on how things go. Helps to have grunts to do the minor things like unpacking and delivery to stations. Find some carts so you can move machines several at a time. Inform the cleaning staff that you're going to have a mountain of packing material to dispose of. Keep 1 box for every 20 machines in case you need to box them up to send to a repair shop down the road.
If you insist on using netrestore over the network, be sure you have multicast enabled on the switches. It doesn't like crossing subnets but can be made to work.
I work for the Department of Redundancy Department.
If your installation is big enough, you could probably get some good advice from...an Apple technical sales rep.
"Ever replace a backlight in a fucking Apple Cinema Display? That's 3 layers (and a thousand assorted screws and layers of tape)"
Sounds like replacing a backlight in every LCD monitor that has ever existed.
Oh, the fail.
I worked at a school district for some time with a significant Mac deployment. We used Mike Bombich's software extensively, and especially for deployment, his NetBoot utility.
It does take a little bit of configuration on the server-side to start, but it looks like some other posters have already linked to tutorials for setting that up. MB has a utility to create a net-bootable-image that can used to image that machine with your choice of disk images (we had different images for different architectures, and different software packages), or can be automated to pick an image automatically.
His NetBoot software also has the ability to run a shell script to complete configuration settings that may need to be done on a per-machine basis (setting the computer network name for example).
For running updates, and modifying settings after the initial imaging, Apple's remote desktop is actually very useful. Although the feature set is limited, it DOES allow for the execution of shell commands from the Remote Desktop interface, which makes upgrading or changing settings on a large number of machines fairly easy.
Ship it back to Apple while they deny that it's a manufacturing defect, but agree to repair it out of the goodness of their heart.
That's what the rest of the world does.
The hardware is more reliable than most OEMs unless you got burned by iMac G5s with bad caps, but that wasn't really Apple's fault. A lot of OEMs got hit by those damned caps.
You should have just mailed in the damn Cinema Display. Service providers (at least non-Apple owned providers) can't replace anything on them but the power brick these days. Just mail it in and let the repair depot monkeys figure it out. I would never want to replace an LCD backlight (which isn't exactly a user-accessible part on ANY display) if it could ever be helped.
I managed a deployment of roughly 800 Macs across the campus of a large university using Radmind. I've also managed the campus Linux, Solaris and OpenBSD kerberos servers, web servers and file servers with the same software. Radmind's learning curve is a little steeper at first, but it's one of the most flexible deployment options out there once you get the hang of it.
Radmind's not really a competitor with tools like NetRestore. When used correctly, NetRestore is great for total reimaging of deployed hardware: nothing beats a block-copy installation for speed. Where NetRestore falls down is when dealing with deployment entropy. After imaging, the machine is in an unknown state ("post-image"), and the only way to be sure all machines are in the same state is to blow away the entire disk and reimage, usually at a cost of gigabytes of bandwidth per machine.
This is where Radmind excels. It's basically a tripwire with software deployment and roll back, all based on the differences between what should be installed and what's actually on the disk. The core utility, fsdiff, looks at all files and directories designated as managed by the administrator and generates a list of differences. You can capture those changes as a loadset and upload them to the Radmind server for deployment to other machines, or you can undo any changes detected by fsdiff and restore the client to a known good state.
The great thing about this method of management is that there's minimal bandwidth used. If fsdiff detects no changes on the filesystem, there's no reason to download anything: your system is in a known good state. On the other hand, it makes deploying Apple's system and security updates pretty damn easy. Grab the updater from Apple's website, install, and run the Radmind tools to capture the changes. Store the changes on the server, add the new loadset to your machines' profile (command file), and let your clients pull down the changes.
The Radmind community is very helpful. Most questions to the mailing list (hosted on SF.net, Google groups mirror here) are answered very quickly, and people are eager to share details about local setups and scripting solutions. A typical setup for a Radmind-managed Mac OS X client usually involves a few possible methods for initiating updates, most of which involve iHook as the UI:
Since we relied on students to help run our labs, we also deployed a special, unprivileged local user account, whom the students could log in as. This also triggered a Radmind update. And of course you can trigger updates over ssh (which works well in combination with something like pdsh).
We combined Radmind with NetBoot for rapid, consistent deployments. Once the hardware was in place and on the network, we netbooted, used ASR to install a minimum and relatively recent system, and let Radmind bring everything up to date, including per-host license files and location specific software.
Radmind's not perfect. It manages at the file level. If you want something to manage, say, config files on a line-by-line basis, Radmind isn't going to fit the bill (yet). Generally speaking, though, Radmind manages Mac OS X with ease. Once you've got Radmind managing your Macs, you'll find you have a lot of extra time to do interesting things instead of troubleshooting problems brought on by stale deployments.
The Radmind wiki is a decent place to start looking. Good luck.
Rule 35: If no porn is found at the moment, it will be made.
Source
Yes.
2007 Shuttle PC, dead after one year (just out of warantee)
Custom PC tower, 5 years, finally fails to make it past post last week.
2006 Mac Mini - still rocking on.
Most of our corporate machines are towers or standard desktops, internals never upgraded since purchase. A fleet of 2009 minis would be fine for these, and iMacs for reception (or senior managers).
Savings: no AV software, easier deployment of apps and policies, dont require MS Active directory or client CALs to manage them - however, not knowing month to month what hardware is going to come available from Apple would suck. Windows apps could be easily delivered using citrix or teminal server for those that need it.
Ever tried to manage 100 notebooks and backup personal data? Howabout encryption software - finally available with bitlocker if you get Vista Pro or premium - but then system folders encrypted too, a pain to manage. I liek just the encrypted home folders - which can also be mounted from an OS X server - and replicated for laptops.
Also how about common accessories like power adapters for 100 laptops and a single OS image that will work for everything?
If you can break the MS monopoly then there are savings to be made up to a certain scale.
However I will admit managing more than 1000 of these puppies could be challenging and I havent seen much that would help except maybe Zenworks from Novell - but then eDirectory is not cheap, but again savings from requiring fewer people to manage everything and fewer servers required.
For a bulk deployment I'd also look at splitting home off from the boot drive, and have a spare boot image with minimum required apps on every Mac, and script an RSync to keep it fresh from a single image.
A new PowerBook hmm? It's clear you're well informed on the subject of Apple computers, given that a "new" PowerBook has to be at LEAST 3 years old at this point.
I do this for the federal government, after coming from a university environment where I grew up with the Mac from the bad ol' days of the late 90s through Apple's phoneix-like rise from those ashes into the titan it is now. Truth be told, not much has changed.
For mass deployments, I'm about to look into Casper, but NOTHING I've seen or heard about beats netboot/netrestore -- and mind you, I live and breathe Mac. I use PCs to manage Remedy tickets, and that's it. The ability to create a master image, upload it to a server, restart a machine with the n key pressed and have it image itself was and is nothing short of magical, and it's the deployment solution I'm moving toward for the portion of the Treasury Department network I control (if I die, money will cease to be printed). Unless Casper can top that, netinstall + n is still my deployment solution of choice, and one that the folks where I used to work are still trying to replicate three years later. There's nothing faster or more foolproof.
Prototyping is just as easy. I deal with everything from banknote designers (pull out a bill. Isn't it pretty? My designers drew all that stuff on their Macs) to executive management, and though they use their machines differently, they all have the same baseline needs -- a rock solid configuration that's hardened to IT Security's exacting (if evolving) standards, and Office to handle collaboration. My base image is a hardened installation of Leopard with fully-patched Office. That's standard across all machines. This base image is what I run in regular user mode on my personal production machine so I will know firsthand exactly what the user experiences from day to day. I customize the default user environment on the standard image to suit _my_ tastes and allow the users to tweak and refine that environment as they see fit. I learned years ago that this is the best approach for standardizing a user's desktop because I know how to work around the various quirks of OS X that can become annoying after using it for an extended period of time, and they usually haven't been on Macs long enough to have figured these things out. The more experienced of my newest users typically bristle at this since to a person they always think their approach/way of configuring the Finder/desktop is THE way to have their machines work, but I usually don't hear a peep from them after a week or two of working in my environment. The biggest compliment to me is when I cease to get trouble tickets from my bitchiest users because they find that I've already anticipated and addressed their most obvious complaints in the standard image.
On top of the standard image, I install applications specific to the machine's role. The designers, for instance, get Adobe CS 4 and additional design-focused applications such as Quark and a font manager. My video people get Final Cut Studio. My engravers get the same package as the designers. My method of choice for deploying to these disparate groups lately has been to install the specialized applications on the standard image and create secondary images applicable to specific groups. Banknote design machines, for example, have their own special image and the video production machines have an image all their own. This simplifies things mightily because all I have to know when I want to deploy a new workstation (or repair a broken one) is where it's going. Oh, this is a replacement banknote machine? Put the banknote image on it. Copy the _user folder_ -- and nothing else -- from the old machine, create an account on the new machine, point it at the old user folder, and voila. Completely new hardware, and the user has no idea anything's changed. I've upgraded users from Tiger-running G5s to Leopard-running 8 core Mac Pros, and the only difference they noticed was the machine was "a lot faster." And the Apple menu's a different color. That's the power of Mac OS X.
Security, as I'm sure you well know, is not an issue on the Mac, but given the sensitivity of what my users do, I
So here you go. Far too much conceptual information about a process I suspect almost no one here knows beyond the few that already mentioned it. Enjoy.
So the best I can do is telling you how I do it for about 400 Mac's, and the tools I use. I basically use two OS-X 10.6 servers that host NetBoot images and Radmind, and then Apple Remote Desktop (ARD) on a client to control events occurring on all the clients be they booted locally or NetBooted.
I'll also be up front, if you are not computer savvy, and don't want to be, do not touch Radmind with the idea of using it to deploy anything beyond software to an already existing deployment. Stick with an image based package. If however you are computer savvy, can get around a command line, and need to support an unlimited number of *nix machines, especially in a lab, Radmind is an incredibly strong tool.
I solely use Radmind for both OS deployment and software updates because it's a delta based package and tripwire system which you don't need to rebuild over time unless an administrator makes horrible mistakes without a backup. If I really needed an image, I would have Radmind generate that build for me and then use 10.5/10.6's NetBoot/NetInstall creation tool on the results.
I do not use NetRestore, NetInstall, or any other deployment tools for OS-X. It is a waste of time to constantly rebuild and maintain various images over time vs a delta based deployment system, especially when I'm the only one supporting the image. It may take *slightly* longer to deploy than a sector based image, but the amount of effort placed on the administrator in the long term significantly decreases. Sure, learning Radmind might take a whole lot of time and effort, but the more random and variously configured machines you need to support are, the more attractive it becomes to spend time learning how to use it beyond a software package deployment tool. Heck, the right people behind it could probably support thousands of *nix servers without much of any effort.
You can also reverse the use of Radmind over time to maintain just software packages by making a negative transcript targeting just ".". If you do that, and make sure clients don't see the overall OS level packages, you can update software only without updating the OS at its core.
So radmind has a set of tools that come with it, and I'm only going to mention the most critical of them. One scans a computer for changes. Two other apps takes that scan and either uses it to upload data to a server, or to use the knowledge on the server to 'cause' changes to the client. Another downloads the command lists from the server, and those command lists have knowledge of all the "package" transcripts that actually define almost every file on the computer. Using them all in combination in scripts by someone that knows how to manipulate the results are what can make Radmind powerful.
Up front there are negatives and positives about Radmind:
Negatives:
It can be very complicated.
A lot of the documentation is poor, though it's better today than when I started using it.
Simple mistakes in a transcript can suddenly prevent the client-side app from functioning. Discovering why can sometimes be very difficult. (especially if it's a nested command file level issue that only gives you "Input/Output error" when lapply crashes.)
It only supports network compression, which frankly is worthless. No file-based compression during capture.
Almost any error in a delta file will break process of updating/deploying machines. It really requires you have someone learn it in and out.
The default method of deploying images to massive numbers of machines that may need different builds is unwieldy. There are ways around some of this.
The GUI console in OS-X once you have several hundred transcripts is annoying to use, and creating and using subfolders for transcripts or command files will seriously screw your deployment life up.
It has no GUI on anything except OS
By your own admission it *WAS* a hidden cost to Macs. Now that you *CAN* find them 3rd party, you're whining about the past.
Its still a hidden cost, its just less now.
Plus the whole selling argument Apple makes for getting a Mac is to avoid stupid technical hassles. This is a stupid technical hassle that wastes tons of time -- that's a cost too. I can't count how often Mac users have to go scurrying about because they forgot the adapter in their car or office or at home. Nor can I count how often I've huddled around some dimwits 13" or 15" screen to watch a presentation in a conference room with a projector sitting right next to it.
So you're convinced that hanging on to connectors created 10 or more years ago on laptops is a good engineering design call?
They are a good design call until more people than not don't NEED it.
Here's some light reading on the topic for ya.
I have nothing against displayport. I have nothing against the progress it represents. You seem to think I somehow dislike displayport or progress in general. That couldn't be further from the truth. All 3 monitors on my desk are hooked up via DVI. And my newest one supports both displayport and hdmi as well, so it should be forward compatible with my next video card too.
But it ALSO has a VGA port, which has proven useful on many occasions. And its good to have that legacy option, because despite the fact that its 'obsolete' its still MASSIVELY IN USE. And that's on a stationery device that never goes anywhere, where having an adapter or two isn't actually inconvenient, nor apt to be left behind or misplaced. Virtually all monitors and projectors you encounter right now take VGA and will have a VGA cable hanging off them ready to plug into your laptop... so yes that is the most sensible port to put on the laptop.
If they want to add displayport too, that's awesome.
Oh yeah, that article ends with three or four advertisements for places that sell cables... cheaper than Apple's.
Glad to see you are coming around to my original argument then. That Apple grossly overcharges for them.