Slashdot Mirror
Ask Slashdot: Moving From *nix To Windows Automation?
Zubinix writes "I have a background in doing automation in a Unix/Linux environment using scripting languages such as perl and bash shell, as well as ssh for remote scripting. My next project will be in the Windows environment so what approach and methodology is best for developing, say, the automation required for a test system? I don't want to use things like Cygwin, as I need to integrate with Windows applications such as Exchange and Sharepoint. Is there a list of should and should not dos when it comes to Windows automation?"
Your answer to everything is WMI. Just start command prompt, type :> wmic And voila, you are ready
And why not? Powershell is perfect for the job.
There are bunches of tools out of the box to automate things in Windows - the scripting host (supports js/vb), power shell, and wmi - or a combination of things. You can open a wmi interface in vb for instance.
When I was working in Software Quality Assurance we had a lot of luck with Mercury Quick Test Pro and Test Batch Runner. They have a solid recording interface than can be coded manually (in VBScript.Net but what can you do?). Integrated fabulously with C# .Net code for doing black-box and grey-box testing. I'd also suggest Symantec Ghost for setting up test systems.
-- Adam McCormick
So, been down this EXACT road. I was doing Linux automation, was so effective was brought in to do Windows too - yea!
Anyway, tried the cygwin route... It went OK, but just not quite it.
Went the vbscript route for a while (pure hell), but could work with wmi and had the windows objects available.
Soon, I started writing console apps in C# to make the trickier stuff happen. The .Net framework made it all so easy.
My final set of tools came to be powershell (access to the .net framework, you can do almost anything), Systernals psexec (for running processes on remote machines), and basic vbscript .bat. I had it set up with a web interface so I could enter a dos command into a web interface and point it a machine. It would build the bat and run it on the remote machine and return the standard out. This allowed me to add IIS sites and app pools, install com components, install apps, run msunit tests, and basically do whatever I wanted to any machine on the domain. Took me a quarter to build, but worked well. I've moved on in the company, but my replacement is still using it.
ymmv
Day 1) Walk in the door, optimistic about what can be done with this "Enterprise" platform.
Day 2) Walk in the door, with a headache, hoping to find an answer for how to manage what were simple tasks under *nix.
Day 3) Walk in the door. Sit down at your desk. Plant your head firmly on the keyboard and cry.
Day 4) Walk in the door, rip your soul out of your chest, stomp on it, and throw it in the nearest recycle bin. Sit down at your desk, and wonder why in 4 days you can't find a valid answer to automation that was so simple under *nix.
Day 5) Walk in the door. Sit down at your desk, and think about how miserable you are now that you're working on a Windows-only network. Leave 2 hours early, and drink away your pain at the nearest bar.
The longer it goes on, the worse the pain gets, until you realize that you have a stash of cheap liquor and pot in your desk drawer, and you use more of both in one day than an entire fraternity use in a hard partying weekend.
I do have some answers for parts of it. Powershell is part of the answer, but far from complete, unless you like virtually every command you type returning worthless 6 line responses. Cygwin may solve some of the problems, but not all of them. ActivePerl may solve some problems. In the end, you will realize that everything is a mouse click away, and those mouse clicks are the only way to do it. Prepare to spend the rest of your life in remote desktop connections, and putting more miles on your mouse than you ever did playing first person shooters.
Serious? Seriousness is well above my pay grade.
http://stackoverflow.com/ has experts that go there just to help with questions like this.
Amazingly enough, that's something I'm doing right now. (no, I'm not the original author). I smoothed over the rough spots as much as possible, and now we're doing the Linux migration. It's a long, slow process to change a decade of deep rooted Windows-isms, but it will happen soon enough. The only Windows that will be left will be the workstations for those who choose not to switch over to Linux. In a web based enterprise, there's no excuse for being locked into any platform.
Serious? Seriousness is well above my pay grade.
Stay away from VBScript, its very out of date at this time. Best work with Powershell, it has deep integration into Active Directory and Exchange (as well as OS, WMI etc). Its quite simple to get a script up and running, and its widely supported. It comes built into the OS in Windows 2008 onwards, for 2003 its available as a download.
Why would you switch to Linux - an inferior, security ridden, and an OS which can't stay up longer than a month?!
It's a hobbyist OS at best and doesn't survive in a real commercial environment.
Don't get your hopes up with Sharepoint. This is quite possibly the single worst piece of crap ware I've encountered in the past 2 years.
Hey I'm a smiley kind of guy and I have to promote Microsoft products to undermine *nix and expand our market share, but I'm afraid I'm running out of ways to do it.
I wondered if the ever so friendly and clever Slashdot crowd can think of imaginative ways to carry out this task? Please help me. Remember; I think you're wonderful.
Powershell. The only tool that knows how to talk to all the different frameworks in Windows is Powershell. No other tool can talk to .NET, COM, WMI, native APIs (via P/Invoke), and external stdio based tools. If you can't do the automation you want using something in one of the above frameworks, you've got bigger problems than finding a good automation tool.
Since the test guy usually has to be a part time sysadmin too, you should be aware of these tools:
System update readiness tool: http://support.microsoft.com/kb/947821/en-us
WMI diagnostic utility: http://www.microsoft.com/downloads/en/details.aspx?familyid=d7ba3cd6-18d1-4d05-b11e-4c64192ae97d&displaylang=en
gplogview: http://www.microsoft.com/downloads/en/confirmation.aspx?familyId=BCFB1955-CA1D-4F00-9CFF-6F541BAD4563
Windows SDK (including debugging tools for windows): http://www.microsoft.com/downloads/en/details.aspx?FamilyID=35AEDA01-421D-4BA5-B44B-543DC8C33A20
ollydbg: http://www.ollydbg.de/
sysinternals suite: http://technet.microsoft.com/en-us/sysinternals/bb842062
Windows Management Framework: http://support.microsoft.com/kb/968929
WDK: http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=36a2630f-5d56-43b5-b996-7633f2ec14ff
WAIK: http://www.microsoft.com/downloads/en/details.aspx?familyid=696DD665-9F76-4177-A811-39C26D3B3B34&displaylang=en
Windows 7 SP1 WAIK supplement: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=0AEE2B4B-494B-4ADC-B174-33BC62F02C5D
If XP is involved, check out Windows SteadyState. It's like deepfreeze, if you've ever used that. qemu is also a great way to boot test machines and capture output at scale; using CoW disks you can have fresh machines every time you boot regardless if the test machines are XP or not.
-- thalakan
I don't agree. Powershell is actually very powerful as it can extend or be extended by the .Net framework. It is also very flexible, which is very convenient for systems automation.
Big enterprise schedulers, such as Tidal, have built-in support for Powershell and many enterprise storage solutions, such as Compellent, also have built-in support. Also VMWare has the very impressive PowerCLI, which is basically a series of extensions for Powershell that can automate almost everything in VirtualCenter.
lucm, indeed.
Essentially, for newer versions of Exchange and SharePoint, Powershell is the only scripting option, and is excellent. For older versions, you don't have a lot of options, but you can probably call COM APIs using PowerShell as well, but the effort is a lot higher. The APIs exposed by Exchange (e.g.: MAPI) are hideous. SharePoint can be managed via direct SQL database queries from anything, with some care.
For Windows servers in general, stick to PowerShell. It's by far the best shell/scripting language by a wide margin. There's a learning curve, but it's absolutely worth it. To get set up to an equivalent level to Bash+SSH:
- Install PS2 + WinRM2 on pre-2008 R2 servers. It's not installed by default, and 2008 comes with PS1. There's a hotfix that updates PowerShell and all associated components to v2 on XP, 2003, Vista, and 2008. .ps1 script files from running by default is a shit workaround for not having an execute bit, and is totally useless. Just turn it off.
- Enable unsigned scripts to run. Preventing
- Enable WinRM. This is the equivalent of SSH, but has to be enabled on both servers (which is obvious), and the clients (for no discernible reason).
- Enable CredSSP. This improves WinRM by allowing delegated credentials to be used remotely. Unlike SSH, WinRM is an RPC protocol, and due to limitations of Windows authentication, the remote server cannot use network resources with the client's credentials (one hop security). The workaround is credentials delegation. This requires a server-side setting, and two client-side settings.
- Install plugins. There are about a dozen that ship with Windows 2008 R2, a plugin for Exchange 2007 and 2010, and a plugin for Sharepoint. To set up the AD plugin on pre-2008 R2 domains, there's a module that has to be deployed, it's basically a web service that the PS plugin uses. The SQL plugin is an optional download that is particularly handy for SharePoint.
That seems like a lot, but can be done in seconds from the command-line, and even better, can be done completely automatically using Group Policy. For new server deployments, I ask for my servers to be created under an AD container, and I just set those options up in a common policy configured on that container, and then everything just works.
The equivalent of a remote SSH session in: Enter-PSSession 'computername'
However, it gets better: You can create and store sessions in variables by creating them using New-PSSession. You can pipe objects into remote sessions and invoke commands on them, or arrays of them. In other words, a single line of code lets you invoke commands in bulk across farms of servers.
For example, as a Citrix XenApp admin, I often have to update machine policy across a farm. This is just two lines with PowerShell:
$xaservers = Get-XAServer | Select -ExpandProperty ServerName
Invoke-Command -CN $xaservers { gpupdate }
That will query the list of XenApp servers, extract the server names into an array, and invoke "gpupdate" in parallel across all of them using a temporary secure shell session.
There might be equivalent capabilities on Linux, but nothing on the Windows platform comes close to what you can do with PowerShell.
All Windows automation that you might want to do is exposed through a COM interface and there is no programming language targeting the Windows platform that doesn't include support for COM
What about languages that aren't so much targeting Windows but just barely support it?
On a personal note, when decided to switch from command scripts to a programming language, I was afraid that it would be hard. But it turned out that most things are actually easier (if at times a bit more tedious) and it's a lot harder to shoot yourself in the foot in most of them. At the very least you'll never get errors caused by the unpredictable script syntax again.
BASH is basically turing complete. Also, "[...]at times more tedious[...]" is the issue. I do 95% of my work in a Windows environment and a lot of times I long for the simplicity of *nix. Most of the time in Windows I feel like I'm fighting a huge and complex system, trying desperately to make it behave in a sane and rational way in order to produce the desired output. In the *nix world at least it's just my code I'm battling with, not the entire OS and ecosystem. But hey, maybe some people actually like a world where all systems communicate using XML and where character encoding is "enicode" yet in practice can be pretty much anything as long as the marketing division can claim it's all unicode because technically the operating system uses unicode internally...
Greylisting is to SMTP as NAT is to IPv4
Oddly enough, I am not convinced. What the heck does it mean that PS can extend the .Net framework, and itself be extended by .Net? Also, for a program to have "built-in support" for PS doesn't make much sense at all because PS is a shell environment. Quite honestly, your post just seems like a bunch of buzzwords. To be more convincing, I would like to see some example commands showcasing the power and flexibility of PS.
Those cli utilities are 30 plus years old. Any sysadmin worth his salt will know them inside and out. How is the syntax any more arcane than what is spit out by ps? And, er, last I checked the ease with which completely unrelated utilities can be chained is the point!
The soylentnews experiment has been a dismal failure.
Those cli utilities are 30 plus years old. Any sysadmin worth his salt will know them inside and out. How is the syntax any more arcane than what is spit out by ps? And, er, last I checked the ease with which completely unrelated utilities can be chained is the point!
"Arcane" is a term that is only ever used to describe a non-Microsoft OS, sort of like the way the word "upstate" is only ever used to describe a region of the state of New York.
It is shorthand for "I don't know anything about it, therefore something is wrong with it."
It is a miracle that curiosity survives formal education. - Einstein
You bring up a good point, but I'm going to address this to the OP.
As previously noted by the other commenter, Powershell is useful if your environment is full of Vista computers or newer or if you have more control over the environment so Powershell can be installed (there is an XP installer). It's major draw is the usefulness to systems admins that want to do real-time execution.
Otherwise, you're looking at VBScript or batch scripts. Neither are nowhere near as elegant or smooth as some other languages, but tapping WMI and other Windows objects is actually really simple using VBScript. There are loads of examples here. Also, a Google search in the form of "vbscript %thing%" will usually get you pointed in the right direction.
Do yourself a huge favor, though - get a decent editor. While Windows has a simple notepad app, there is no context highlighting, in-line completion, or other helpful tools for looking at script code. Personally, I prefer PrimalScript because it's useful for other things like SQL, C, Perl, etc (including PowerShell). It also has a built-in debugging engine. However, that app can be expensive. One good free alternative is Notepad++ with the appropriate plug-ins for the files you want to create/edit. There are plenty of others out there, so grab whatever you feel works best for you.
If you're planning on using a database and are familiar with MySQL, you should feel nearly at home with Microsoft's version of SQL. There are some minor syntax differences (for example, the update query is slightly different, if my memory is correct) but it shouldn't take long to get used to them.
Lastly, ideally, you're going to want to an account that is a local administrator on all of the systems you will be touching. Yes, you can do some serious damage with such an account and is typically not the way of things in the *nix world, but many aspects of a Windows system are inaccessible unless you have an Administrator level account - especially remote access to certain things like the registry and WMI. If getting a local administrator account on all the systems you're responsible for isn't an option, have someone that does have one (or a domain admin account) on speed-dial in case your scripts fail due to permissions errors.
Good luck!
My sources are unreliable, but their information is fascinating. -- Ashleigh Brilliant
I'm a *nix guy, and I do have to say Powershell is pretty sweet.
Here's an example of something extending Powershell. VMWare released a module for PowerShell that allows for control of VMWare env using Powershell.
#Load VMWare snap-in for powershell
LoadSnapin -PSSnapinName "VMware.VimAutomation.Core"
#Create VM from template and then start it
$myNewVMName = "NewVM_01"
$myTemplate = Get-Template "TemplateName"
$strDestinationHost = "ESX01"
$myNewVM = New-VM -Name $myNewVMName -Template $myTemplate -VMHost (Get-VMHost $strDestinationHost)
Start-VM $myNewVM
VMWare created a really awesome extension to Powershell, that allows for all sorts of inheritance and piping. Microsoft created a rather poorly implemented Active Directory extension for Powershell (can't pipe or inherit on things I would expect to be able to)... MS could have used the VMWare example to make a better AD extension.
Let me know when it's as ubiquitous as bash or csh. As far as I know, it's only pre-installed on Windows Server 2008 or greater.
Okay. It's as ubiquitous as bash or csh. It ships out of the box with Windows Vista, 7, Server 2008 and 2008 R2, and for XP it is included in Windows Update so any computer properly receiving it's updates will have it.
bash...
Probably the hardest thing to learn in the *nix scripting world is sed.
I wrote a menuing system for a Xenix minicomputer back in the early 1990s in straight sh, never having touched it or the Unix tool set before, in a couple of hours. And I can tell you I ain't no genius.
Try do anything vaguely useful in Powershell without prior knowledge in a couple of hours. It is a gawdawful horror story, a good example of the insane overkill that Microsoft applies to simple problems. It keeps the MCSE's employed with the bizarre range of esoteric and overly-complicated solutions, but when you're just trying to move some data around from tool to tool or piping some output through a regex evaluator on its way to a SQL RDBMS, you end up going "What the fuck is wrong with those people in Redmond?"
The world's burning. Moped Jesus spotted on I50. Details at 11.
This is false. Powershell is available as an addon for Win XP and Win 2k3. The poster above said preinstalled fo a reason. Any decent Window environment should have a software distribution system which makes the update for PS 2.0 a non issue.
When in Rome, do as the Romans do.
He was asking about Windows server administration, and PowerShell is the right way to do that.
For most new Microsoft server products, the script bindings are only available for PowerShell (in the form of snapins or modules). Many third-party vendors are releasing products that are PowerShell only. It is also the only shell that can call all mainstream APIs (COM, WMI, WinRM, and .NET).
Perl specifically is one of the worst possible options for Windows scripting. It is a text processing scripting language designed for Linux, where most applications and even system APIs are text based. Windows mostly uses object-oriented binary APIs. Very few Microsoft server products have text-based command-line tools that could be automated with Perl.
Suggesting Perl for Windows automation is a bit like trying to script Linux with VBScript. Does that sound like a good idea to you?
I can do the same thing in 3 lines of code using only SSH and virsh in our KVM environment (1 line to set sh as shell, 1 to run SSH and use virsh to clone / create a VM, and 1 to start it).
And I can do that with 1 line of PowerShell code (PowerShell has a consistent way to short parameters, use positional parameters and pipe full objects like virtual machines):
new-vm ESX01 -name NewVM_01 -templ TemplateName | start-vm
Your point? The OP was asking what to use to set up and administer test systems in a *Windows* environment. PowerShell is a valid option for that.
Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*
I'm a programmer-slash-sysadmin, and I've taken a stab at writing both command-line tools using C/C++, .NET, and these days with PowerShell, so I might be able to answer your question.
When I write complex software, I usually provide some command-line tools for the admins to do things like import data, export logs, or whatever. This is a royal pain in the ass, but I do it because it's useful. I always end up spending something like 80% of the time on annoying stuff like handling command-line parameters, input validation, and error messages. Of course, because whatever I come up with is new and different, I have to write doco for it, train staff, and so on.
Now, with PowerShell, instead of having to write a whole new program to implement a command, I can simply extend a class from a framework library. The PS framework does almost everything automagically: handle the pipeline, input parameters, parameter validation, parameter sets, optional parameters, help, tab-complete, wildcard matching, output formatting, etc...
To give you an idea, it is possible to write a PowerShell command in C# (.NET) with several parameters and complex output that does a useful task in about 50 lines of rather trivial code. The equivalent C program would be thousands of lines.
More importantly, the resulting PowerShell command will be orthogonal, consistent, discoverable, and embeddable
By orthogonal, I mean things like: every PS command that handles wildcards does it with the same shared library, which is trivial to use. Hence, I can do things like: Get-VM "prod_win_[a-k]*" which is a VMware snapin, and the behavior will be exactly the same as if I had called a Citrix XenApp snapin to lists farm servers with: Get-XAServer "prod_win_[a-k]*". Similarly, the output of commands is structured data. You can say: Get-VM | Export-CSV or Get-XAServer | Export-Csv and get similar results. Compare with legacy command line tools, which often implement such formatting internally, and badly. I just had to work on some Novell systems, for example, which export invalid XML files because the utility doesn't escape ampersands! Of course, our example 50 line command will get all of that. Export to CSV? You have it! Export to XML? Done! Quickly, tell me how to export CSV formatted data from the following 3 Linux command-line tools: 'ls', 'apt-get', and 'ps'.
By consistent, I mean that parameters are always specified with "-param", never "/param", or "--param", or "-abcdgh" where each letter does something that you can only determine by reading a five page document. Similarly, Microsoft has established a strict naming system for developers, so that instead of "retrieve-foo", "ask-foo", "query-foo", "request-foo", "list-foo", "foo-list", "fooenum", or god knows what else, the only acceptable standard is "get-foo". VMware has "Get-VM", Citrix has "Get-XAServer", Microsoft has "Get-Process", etc... no guessing! There are standards for command names, parameter names, and coding conventions. E.g.: Verb Naming Rules, and Cmdlet Development Guidelines.
By discoverable, I mean that if I write a little 50-line utility with a bunch of parameters, an administrator at the command line can simply press 'tab' and have both the command and the parameter names automatically completed. The help is automatically generated from my 50 lines of code. GUI script wizards can load a bunch of metadata about each parameter to enable drag & drop script development.
By embeddable, I mean that even a 50-line utility can be called not just from the shell, but from within a hosting application in the same process. Instead of having to handle text-based streams, PS commands take .NET objects, and return .NET objects. There's no guessw
How exactly do you see Windows updates for all packages again? Oh, right, it's not a feature. Yawn.
Ignorance can be cured by Google (or bing). It took me 30 seconds to find out that Windows Update does indeed have a COM API which can be readily used from PowerShell. Here a way to list *all* available packages from Microsoft Update:
PS C:\Windows\system32> $mus = new-object -com "Microsoft.Update.Session"
PS C:\Windows\system32> $updates = $mus.CreateUpdateSearcher().Search("") | select -exp updates
PS C:\Windows\system32> $updates|select title
Title
-----
Latvian Language Pack - Windows 7 Service Pack 1 (KB2483139)
Slovenian Language Pack - Windows 7 Service Pack 1 (KB2483139)
( other language packs )
Definition Update for Microsoft Security Essentials - KB2310138 (Definition 1.103.1197.0)
Note how the list was produced by selecting just the title. The updates returned by the update searcher are actually objects which not only expose a lot of extra information such as disk/cpu/memory capacity recommendations, severity indicators from the security response team, knowledge base article references, license text etc, but also allows you to directly proceed to install by piping them.
Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*
And maybe you want to hone up on your writing skills? You wrote
How exactly do you see Windows updates for all packages again?
You wrote Windows with a capital W. If you meant to hit on the fact that there's no central repository for 3rd party packages you should have left out the Windows part, like
How exactly do you see updates for all packages again?
See? not too difficult.
Anyway, having a central repository has very little to do with automating a test system. For automating setting up a test system having the packages available will do just nicely. Pathetic that you must move the goalposts to keep the illusion that Windows is no good.
Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*
The point was the grandstanding, the comment "I do have to say Powershell is pretty sweet" implied something interesting or insightful was going on.
There is. PowerShell is a refreshing new take on shell scripting. Granted, it is best fit for Windows because Windows is already pretty much object oriented from a system point of view. But at the same time traditional shells would struggle with incorporating OO concepts. PowerShell does away with the need for sed and awk, has a built-in extremely powerful regex engine and has modern constructs like structured exceptions, script blocks, closures, integrated help (no not just man pages) and a lot of other interesting stuff like the common -whatif and -confirm parameters, remote sessions and remote jobs, fan-out remoting etc.
You may not like the fact that Windows now sports a shell which surpasses bash, ksh and zsh in many ways, but that doesn't mean that it is not interesting. It is for a good many of us.
Half of the people in this thread make it sound like this is such a big deal, and it's still a pile of steaming crap.
Yeah, whatever. You are entitled to your opinion. But please educate yourself on the subject.
It's like I've stumbled into some bizarro-world edition of Slashdot where getting a limited command shell to function under Windows is some sort of nirvana. You still have a (very very very) limited shell with little of the functionality of shells created 20+ years ago.
Translation: "Help, someone in here has something good to say about Windows and nobody helps me with bashing them. Anything coming from Windows cannot possibly bring anything new. I refuse to look at it. If it is any good, it must have existed for 20+ years in Unix".
Grow up, please. PowerShell is a full-featured command shell and surpasses bash, ksh and zsh in several areas. You may still prefer *sh shells, but on Windows there is an interesting new take on the concept.
Or, to put it simply, "there is nothing to see here". What you think of as the Rapture is a meh moment at best.
Translation: "I don't want to hear about anything which could challenge what I already know as the truth."
Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*