Slashdot Mirror
Ask Slashdot: Moving From *nix To Windows Automation?
Zubinix writes "I have a background in doing automation in a Unix/Linux environment using scripting languages such as perl and bash shell, as well as ssh for remote scripting. My next project will be in the Windows environment so what approach and methodology is best for developing, say, the automation required for a test system? I don't want to use things like Cygwin, as I need to integrate with Windows applications such as Exchange and Sharepoint. Is there a list of should and should not dos when it comes to Windows automation?"
Just don't. [/sarcasm]
"Be particularly skeptical when presented with evidence confirming what you already believe." -
that and banging your head on your keyboard until the world makes sense again.
This is my sig. There are many like it but this one is mine.
Dont:
Windows Automation
'nuff said
Your answer to everything is WMI. Just start command prompt, type :> wmic And voila, you are ready
Your best bet is to learn Windows Powershell (ships by default with win7): http://msdn.microsoft.com/en-us/library/aa973757(v=vs.85).aspx
Its scripting language is built on .NET and provides access to all of the power and flexibility available in the .NET framework. You can also write your own commandlets/plugins in C#/VB.NET to extend Powershell's capabilities.
not too many options - Windows PowerShell, that's about it.
Is it?
> what approach and methodology is best for developing, say, the automation required for a test system?
Install Linux in a VM, and do all the automation from there.
Then you can eventually migrate to removing the Windows machine and running Linux directly on the hardware.
Hate to say it, but you should probably learn VBScript. There's a VBScript interpreter is built into Windows, and it can interact with COM. It might be a pain, but it will suit your needs.
There's no -1 for "I don't get it."
Windows automation now-a-days == PowerShell. It is actually powerfull and pretty well thought through.
You already mentioned using Perl in your *NIX environments so you could do the same with your Windows boxes too. You could also use Python. Powershell is probably your best option.
There are bunches of tools out of the box to automate things in Windows - the scripting host (supports js/vb), power shell, and wmi - or a combination of things. You can open a wmi interface in vb for instance.
Migrate the Windows machines to *nix...
Oh...wait...NM :)
I'm out - that's the only option I could come up with.
When I was working in Software Quality Assurance we had a lot of luck with Mercury Quick Test Pro and Test Batch Runner. They have a solid recording interface than can be coded manually (in VBScript.Net but what can you do?). Integrated fabulously with C# .Net code for doing black-box and grey-box testing. I'd also suggest Symantec Ghost for setting up test systems.
-- Adam McCormick
autoit or powershell are about as good as it gets for windows automation.
What I read:
I have a background doing things in a logical way, in a stable environment that actually works, but thanks to some incompetent beancounters who have traded the company cow for some buzzword-compliant magic beans, my next project will be building an igloo in the ninth circle of hell.
But for the sake of supporting 'Ask Slashdot':
DO:
DO NOT:
To understand recursion, you must first understand recursion.
C++ is much faster than all that stuff, I've used it for years - it's a synch! Top Tip!
The purpose of existence is to make money.
Powershell is probably your best bet.
So, been down this EXACT road. I was doing Linux automation, was so effective was brought in to do Windows too - yea!
Anyway, tried the cygwin route... It went OK, but just not quite it.
Went the vbscript route for a while (pure hell), but could work with wmi and had the windows objects available.
Soon, I started writing console apps in C# to make the trickier stuff happen. The .Net framework made it all so easy.
My final set of tools came to be powershell (access to the .net framework, you can do almost anything), Systernals psexec (for running processes on remote machines), and basic vbscript .bat. I had it set up with a web interface so I could enter a dos command into a web interface and point it a machine. It would build the bat and run it on the remote machine and return the standard out. This allowed me to add IIS sites and app pools, install com components, install apps, run msunit tests, and basically do whatever I wanted to any machine on the domain. Took me a quarter to build, but worked well. I've moved on in the company, but my replacement is still using it.
ymmv
I see what you did there.
"Nine times out of ten, starting a fire is not the best way to solve the problem." - my wife
Do my job for me! Waaa!
If you really want to do an outstanding job, bring all the experience you have with Unix/Linux, but surround yourself with all the windows technology. I've found in the past that when I switch different environments and try to bring the same technology to the new environment, I miss out on all the advantages (and yes disadvantages) of the technology. I've been working in Unix since the 80's and Linux a long time too, and there was a time when you could not even get me to look at a windows environment. But those days are long long gone. The .Net environment is really very good. I think you'll enjoy it, if you give it a chance. Good Luck!
By far your best option. It can take a while to learn though. SharePoint and Exchange use powershell as their main interface for scripting.
You have really only 2 options....
Learn and tolerate powershell...
buy a nice pistol, put it in your mouth and pull the trigger.....
Both have the same amount of pain and misery attached to them.
Do not look at laser with remaining good eye.
http://stackoverflow.com/ has experts that go there just to help with questions like this.
VBScript is included with any version of Windows you're likely to be working with, is mature, and stable. That having been said, it has the boneheaded pre-.NET Visual Basic syntax, so you may hate yourself for choosing it.
PowerShell is either included with or available as an add-on for most versions of Windows you're likely to be working with. It has a much nicer syntax that is inspired by several Unix/Linux scripting languages, and can make use of .NET assemblies, which is *very* powerful. However, my experience with it was that it wasn't 100% ready for primetime. I've written hundreds of VBScripts, but before I'd hit ten PowerShell scripts, I ran across a nasty bug related to one of the wildcard syntaces (is that even a word?) that the language supports - if I tried to use a for loop to iterate through a list of directories, and any of the directory names included square brackets, I was basically out of luck. There had been a workaround in older versions of PS, but not in the one I was using. Maybe MS eventually fixed this, but if so it literally took years.
In an ideal world, I'd recommend PowerShell, because it can do a lot more, and typically with less script code. But I play it safe by sticking with VBScript, at least until the issues with PS are worked out.
"...always new atoms but always doing the same dance, remembering what the dance was yesterday." -Richard Feynman
This has been haunting me for over 30 years !!
http://www.imdb.com/name/nm0528822/bio
That is all.
Since you already knows how to do it on a *nix computer I advice you to:
1. Buy a Slackware Linux 13.37 CD
2. Install Slackware on your windows computer.
3. Do as you did when doing automation on a Linux computer.
4. Profit from your old knowledge !
Never change whats perfect.
Just saying it like it are.
She is the singer of "We'll Meet Again" at the end of the movie.
I would have said Python alone, which has some pretty good cross-platform support, but IronPython also has access to the entire .NET platform.
You can pick up a free copy ("Developer Edition") of ColdFusion from Adobe. Then code CF pages that call cfspreadsheet or cfsharepoint. (If you're careful to avoid coding directly to Windows (drive letters, backslashes, etc), the spreadsheet code you write will also work with Unix equivalents via POI.) Then use ColdFusion Administrator > Debugging & Logging > Scheduled Tasks to schedule those pages. (That's the CF equivalent of cron.)
Can't beat the price.
I've been down this road. vbscript is more than a little annoying, but you can count on it being there, and it knows about COM (which you'll also enjoy). Good luck.
There's nothing fundamentally really that different from a management standpoint. The more you'll dive in the more your going to find that differences boil down to syntax. Your going to need to test your scripts, your going to need to have peer review, your going to need to use change control, maintenance windows and so on.
There is absolutely no reason for your process to be any different on the Windows side than the Unix side, and if it is than your process needs to be rebuilt. Process should always be tool agnostic and your operating system is simply a tool. If you know your best practices than the only thing you need to figure out is the syntax (tool, language etc) to do what you want to do.
If your simply asking what tools you should use to do scripting or deployment, than you need to look at tools like Altiris, SCCM and so on. If your looking for tools for packaging applications than you would at tools like Wise Packaging Studio. What you really haven't done is explained your need very well. Are you trying to manipulate certain things about exchange with a script? Are you trying to export or import SQL database data with your script?
If you want to run a script on just a few systems than you can schedule the server to run scripts manually and you just need to supply the scripts. If your looking to deploy something on a consistent basis than a combination of GPO's and Active Directory can work quite well. If your looking to automate the distribution of a series of scripts based on certain characteristics of your systems than it would be hard to beat Altiris. If you want to run custom queries or actions than WMI based scripting can do some pretty neat things.
Feel free to message me offline, where I work I'm responsible for the management of both Windows and Mac based systems and do this for a living on a daily basis.
Unless 100% of the applications that you need to integrate into your automation are driven by CLI or some kind of well-defined API, you'll be out of luck.
Windows isn't designed for automation. It's designed for moronic pointing and clicking.
Parity: What to do when the weekend comes.
"I am a one trick pony who cannot learn to do things in other operating systems because my ego gets in the way."
Go elsewhere to seek advice. I'd suggest stackoverflow.com
Got into Autoit on a recent project. Definitely a good implementation. Did what I wanted completely. I was calling Autoit into and out of Maya (3D) scripts, chaining it all together. Was able to wait and timeout on specific Maya exporter dialogs and trap contingencies. There was a little funny business with different ways of accessing the windows controls, but not too hard, just ganked code from some demos.
Use a decent automation engine, will save you years of headaches and maintenance. There are pros and cons though.
Ones such as BMC Bladelogic or HP Server Automation (opsware) are probably the best out there (enterprise-grade).
Be forewarned: they are not cheap. Best for enterprises 100+ systems. The more, the better ROI.
Stay away from VBScript, its very out of date at this time. Best work with Powershell, it has deep integration into Active Directory and Exchange (as well as OS, WMI etc). Its quite simple to get a script up and running, and its widely supported. It comes built into the OS in Windows 2008 onwards, for 2003 its available as a download.
One possible way is OLE automatisation, but be prepared of the horrors of it.
With the win32api module, can do Windowsy stuff, and you can transfer all your python skills from your Unix days. You do have Python skills don't you? Well, get some.
There's a native port of Python so no need for cygwin.
Don't get your hopes up with Sharepoint. This is quite possibly the single worst piece of crap ware I've encountered in the past 2 years.
Hey I'm a smiley kind of guy and I have to promote Microsoft products to undermine *nix and expand our market share, but I'm afraid I'm running out of ways to do it.
I wondered if the ever so friendly and clever Slashdot crowd can think of imaginative ways to carry out this task? Please help me. Remember; I think you're wonderful.
Lets look at the situation. You have the Unix shell (C, Korn, Bourne, Bash, etc), designed over long periods to offer universal functionality and complete environmental control. You can control processes in detail. You can integrate seamlessly and universally with every application on the box, and are given more control and more options than the graphical user interface. The shell scripting languages offer logic and control structures consistent with Turing Complete languages. There is nothing equivalent, or even close in the windows world. There are some 3rd party applications that may attempt to do scripting, but they are all after the fact. Windows was never designed as a 'back end' system. Their offerings offer limited functionality, since its all after the fact. Part of the issue is that its almost universal that every application that has a graphical user interface in Linux or Unix, also has a non-graphical interface integrated and designed as part of the application. Many windows applications don't have that functionality (all user control over the application is graphical, all output is graphical, except to very specific parts areas of the operating system, there is rarely any back-end command and control that a scripting language can do anything with). Shorter answer: you are going from a richly endowed system to a very sparse one. You are already in trouble.
Dos Batch Scripts (.bat & .cmd) SETLOCAL is your friend.
AutoIT (script gui programs!)
http://www.autoitscript.com/site/autoit/
Unix Utilities (deploy if you can)
http://unxutils.sourceforge.net/
Also remember that in dos: /i "thing I want to look for" *filenames*
find
works a lot like a weak version of grep.
Also look into windows powershell (I have not used it but the internet says it's awesome so it must be true.)
Also a lot of windows programs have command line options or programs you can leverage. You can build an entrire AD domain from the command line if you use the appropriate methods.
TURN BACK
(Sign you see in UK&Ireland when you get off the ferry on the wrong side - they drive on the left)
Powershell. The only tool that knows how to talk to all the different frameworks in Windows is Powershell. No other tool can talk to .NET, COM, WMI, native APIs (via P/Invoke), and external stdio based tools. If you can't do the automation you want using something in one of the above frameworks, you've got bigger problems than finding a good automation tool.
Since the test guy usually has to be a part time sysadmin too, you should be aware of these tools:
System update readiness tool: http://support.microsoft.com/kb/947821/en-us
WMI diagnostic utility: http://www.microsoft.com/downloads/en/details.aspx?familyid=d7ba3cd6-18d1-4d05-b11e-4c64192ae97d&displaylang=en
gplogview: http://www.microsoft.com/downloads/en/confirmation.aspx?familyId=BCFB1955-CA1D-4F00-9CFF-6F541BAD4563
Windows SDK (including debugging tools for windows): http://www.microsoft.com/downloads/en/details.aspx?FamilyID=35AEDA01-421D-4BA5-B44B-543DC8C33A20
ollydbg: http://www.ollydbg.de/
sysinternals suite: http://technet.microsoft.com/en-us/sysinternals/bb842062
Windows Management Framework: http://support.microsoft.com/kb/968929
WDK: http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=36a2630f-5d56-43b5-b996-7633f2ec14ff
WAIK: http://www.microsoft.com/downloads/en/details.aspx?familyid=696DD665-9F76-4177-A811-39C26D3B3B34&displaylang=en
Windows 7 SP1 WAIK supplement: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=0AEE2B4B-494B-4ADC-B174-33BC62F02C5D
If XP is involved, check out Windows SteadyState. It's like deepfreeze, if you've ever used that. qemu is also a great way to boot test machines and capture output at scale; using CoW disks you can have fresh machines every time you boot regardless if the test machines are XP or not.
-- thalakan
I tend to do this king of stuff (automated test harnesses) with Python (equipped with necessary platform-specific libraries). Naturally, this is Windows so COM and WMI are your friends. Note that IPython makes for an excellent shell for trying stuff out. I tend to go back and forth between Linux and Windows and Python definitely is a transferable skill.
I would like to die like my grandfather did - sleeping. And not screaming in terror, like his passengers.
Haven't used it myself, but I've seen it mentioned in regards to unit testing on windows - http://www.autoitscript.com/site/autoit/
Did I miss this in the comments or has no one suggested it? (perhaps it's not really geared for what you're doing)
Why not give Opalis a try? It's actually really easy. PowerShell is more powerful, but it's not exactly intuitive.
http://www.microsoft.com/systemcenter/en/us/opalis.aspx
Good luck.
Are you talking about 2008 and Windows 7? Or do you need to go back to Windows 2000, XP, or even 2003? There are significant differences in the tools available based on the version (for example, powershell...)
Have a look at Opalis, which has been acquired by Microsoft and is now part of the System Center suite. It is a very interesting runbook automation environment with plugins for most Windows-related tasks. It is graphical and pretty user-friendly, and if something cannot be done natively you can extend it with Powershell.
lucm, indeed.
Whatever you do, expect it to break on you because of: ...
1. Microsoft update
2. Adobe update.
3. Java update.
4. A field overflow deep within Windows.
5. Random licensing failure.
6. Some random app crashes.
7. Someone on the network updated their version of Word, causing your database access to fail.
8. Hackers.
9. Virii
And, it will probably takes weeks to find the problem and then get it working again, for a little while.
Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
http://www.csscript.net/
Is there a list of should and should not dos when it comes to Windows automation?
Yes.
1. Do not do Windows automation.
That is all...
That is all.
VBScript and "jscript", along with the Windows Script Host, are designed for COM automation. Most of what you need in that case can be found in the script56.chm file.
http://www.microsoft.com/downloads/details.aspx?FamilyId=01592C48-207D-4BE1-8A76-1C4099D7BBB9
Perl is not COM-oriented, of course, but can be adapted if you prefer. .Net is not primarily COM, so it doesn't make much sense unless you already know .Net and like it. Some have suggested WMI. WMI is not script. It's a set of classes that can be used through script -- typically used with VBScript. For most things other than hardware information, WMI is just a clunky, slow wrapper, while the syntax is ugly and unintuitive.
I've written complex tests (1000's of lines) in perl which run on Windows, Linux, AIX, and Solaris. In my case I'm both Unit testing software changes, and regression testing before submitting into fhe code base.
A lot of the tests at our company are written in perl because most of the perl code works on all the platforms we ship our product on. Most of our core product is written in C.
so that you can give yourself a pre-frontal lobotomy to bring yourself down to Windows level...
AutoIt is brilliant, I've had a lot of fun with it. Some people like AutoHotKey, but I can't get used to the syntax. I'm dabbling with MacroMonkey which looks like it can do a lot of the same stuff with a better language (Lua) but not as well-developed a library and nowhere near the same community as AutoIt (although there are some very abrasive personalities on the forum).
This is a more complex problem than what scripting language you are going to use. Automating things is about job management, process management, signals, connecting streams and terminals, setting device modes, filesystem permissions, modifying network settings, and many other things. Unix is designed in a way that lets you change almost every property of the system in numerous ways, following general principles of its architecture. It is a very logical and consistent system.
The problem is that Windows lacks such an modular, abstract foundation. It is a much more arbitrary and inflexible system, it is not designed for putting different pieces of it together in different ways for automation.
For example, on Unix you have numerous small utilities that work together nicely by piping the output of one utility into the input of another one. Windows is really bad at doing such things, and the output format of most of its utilities is not easily machine parseable.
I think, the question is not: How do I automate Windows? The question should be: What system should I use, which one is good at automation?
And the answer is definately Unix, not Windows.
Try REXX http://en.wikipedia.org/wiki/REXX
Powershell can be installed on XP/Server 2003 and Vista/2008 and Powershell V2 comes by default in Windows 7 and is a built in optional feature in Server 2008 R2, so it is available on all modern Windows OS'.
There is pretty much nothing you can't do in Powershell. It has an innovative object pipeline system and excellent syntax. The learning curve is high but what powerful programming language doesn't have a high learning curve?
Want to construct a 'magic packet' and send it over the network? No problem: http://powershell.com/cs/blogs/tips/archive/2011/02/24/sending-magic-packet.aspx
Want to automate SharePoint? http://www.u2u.info/Blogs/karine/Lists/Posts/Post.aspx?ID=9
Want to automate Exchange? http://technet.microsoft.com/en-us/library/bb124413.aspx
Want to read a binary file? Get-Content -Path C:\binary.file -Encoding Byte
Want to connect to a web service? http://technet.microsoft.com/en-us/library/dd315258.aspx
Want to query a SQL database? http://thepowershellguy.com/blogs/posh/archive/2008/02/28/listing-all-the-databases-from-a-sql-server-from-powershell.aspx
Want to construct a GUI? http://technet.microsoft.com/en-us/library/ff730941.aspx
Want to convert a bitmap to a GIF? http://blogs.technet.com/b/heyscriptingguy/archive/2010/07/05/hey-scripting-guy-how-can-i-use-windows-powershell-to-convert-graphics-files-to-different-file-formats.aspx
Want help from the Powershell community? http://www.powergui.org/forumindex.jspa?categoryID=55
Want to search a repositiy of pre-made scripts? http://poshcode.org/
Need help with a script? http://www.powergui.org/forum.jspa?forumID=165
And the list goes on and on and on and on....
I think I might look for another job...
Essentially, for newer versions of Exchange and SharePoint, Powershell is the only scripting option, and is excellent. For older versions, you don't have a lot of options, but you can probably call COM APIs using PowerShell as well, but the effort is a lot higher. The APIs exposed by Exchange (e.g.: MAPI) are hideous. SharePoint can be managed via direct SQL database queries from anything, with some care.
For Windows servers in general, stick to PowerShell. It's by far the best shell/scripting language by a wide margin. There's a learning curve, but it's absolutely worth it. To get set up to an equivalent level to Bash+SSH:
- Install PS2 + WinRM2 on pre-2008 R2 servers. It's not installed by default, and 2008 comes with PS1. There's a hotfix that updates PowerShell and all associated components to v2 on XP, 2003, Vista, and 2008. .ps1 script files from running by default is a shit workaround for not having an execute bit, and is totally useless. Just turn it off.
- Enable unsigned scripts to run. Preventing
- Enable WinRM. This is the equivalent of SSH, but has to be enabled on both servers (which is obvious), and the clients (for no discernible reason).
- Enable CredSSP. This improves WinRM by allowing delegated credentials to be used remotely. Unlike SSH, WinRM is an RPC protocol, and due to limitations of Windows authentication, the remote server cannot use network resources with the client's credentials (one hop security). The workaround is credentials delegation. This requires a server-side setting, and two client-side settings.
- Install plugins. There are about a dozen that ship with Windows 2008 R2, a plugin for Exchange 2007 and 2010, and a plugin for Sharepoint. To set up the AD plugin on pre-2008 R2 domains, there's a module that has to be deployed, it's basically a web service that the PS plugin uses. The SQL plugin is an optional download that is particularly handy for SharePoint.
That seems like a lot, but can be done in seconds from the command-line, and even better, can be done completely automatically using Group Policy. For new server deployments, I ask for my servers to be created under an AD container, and I just set those options up in a common policy configured on that container, and then everything just works.
The equivalent of a remote SSH session in: Enter-PSSession 'computername'
However, it gets better: You can create and store sessions in variables by creating them using New-PSSession. You can pipe objects into remote sessions and invoke commands on them, or arrays of them. In other words, a single line of code lets you invoke commands in bulk across farms of servers.
For example, as a Citrix XenApp admin, I often have to update machine policy across a farm. This is just two lines with PowerShell:
$xaservers = Get-XAServer | Select -ExpandProperty ServerName
Invoke-Command -CN $xaservers { gpupdate }
That will query the list of XenApp servers, extract the server names into an array, and invoke "gpupdate" in parallel across all of them using a temporary secure shell session.
There might be equivalent capabilities on Linux, but nothing on the Windows platform comes close to what you can do with PowerShell.
There are a number of responses above with varying degrees of M$-enlightenment (thalakan's being the most professional); however, it's not entirely true that Windows was designed exclusively for point-and-click administration. That's only true of the GUI shell. Windows was *designed* to be administered by *compiled* code. Preferably C++, which is the only thing that can deal directly with the shitty disaster that is the Win32 codebase without making things worse. Everything else is a shim over the Win32 nightmare, which is still the "core" of the operating system. So, everyone saying "you're just fucked" is in some sense accurate, albeit not precisely correct. The whole OS should have been refactored starting in 2003, when Microsoft pretended to be interested in security. It wasn't, so here we are.
win32api provides COM interfaces to just about everything on Windows. But you will need WMI too so make sure to get Tim Golden's WMI module. And if you need to do anything with .NET, there is IronPython: same language, just different vm underneath.
If you do use WSH (Windows Scripting Host) do not be tempted by VBScript. Too many dead ends and limitations. Use only JSCRIPT which is basically a slightly older version of Javascript. Use Javascript and WSH instead of batch files to glue things together.
MS Exchange tells you what to do with it in the name - swap it for something that works. That said you are stuck with it so the way to deal with this moving target (new release announcement - something that should have been there in the first beta and we've pretended it's been there all along now works!) is to stick to where it's behaviour is bolted down to standards outside of Microsoft and you'll know a patch or upgrade won't break the part you are talking to. Sharepoint is almost as is somebody finally decided to do a decent FTP replacement but was driven to drugs halfway through - with that you'll have no choice but to chase a moving target.
As for the language - the developers where I work have been stung by VB changing into vastly incompatible things to the point where there are still production systems running Win98 (although part of that is hardware compatibility with A/D converter cards). They moved to python to get away from needing a pile of different VB environments. That's one choice but there are plenty of other cross platform choices. If something has libraries that do most of what you want already that's probably the language to go for.
Fanboys that have never touched MS Exchange apart from fetching mail from Outlook will probably flame me for the first line. Instead take a deep breath and google for the new feature list of each of the last few versions of MS Exchange which will give you an idea of how much was missing from the previous one. Getting a proper backup you could use for a bare metal restore or even full mailbox recovery on MS Exchange used to be impossible without shutting the entire creaking structure down for the duration of the backup. It tries to do everything so there is no single thing that is like it - but as far as SMTP goes there are probably well over a dozen better solutions.
Once you have it you are stuck with it and have to change the way your organisation works to fit in with the way MS Exchange operates. When that has happened there is no going back without breaking a lot of things.
Since you mention testing, I thought I should at least mention NAnt. I run a number of build servers under CruiseControl.Net that makes calls to NAnt. The NAnt scripts make calls to thing like NUnit, MSBuild, devenv and FxCop. This automates our continuous integration and release builds. There are other tools that can be thrown in like NCover. I also do automated Flex builds, but that is under Hudson. This is on Windows Server 2003 and 2008 in 32 and 64 bit versions.
As others have mentioned, PowerShell is very helpful as a replacement for your other scripting tools. Personally, I still use Perl.
It's not so bad if you make use of the Sharepoint Client Object Model that comes with SP 2010.
Use the Javascript version (and JQuery) to do whatever you want on an SP page, and use the .NET version of the Client Object Model to write IronPython scripts that integrate it with other systems.
run away as far as you can, don't waste your time with that system! have you been demoted or something?
Most Microsoft products from the OS itself to server software to office apps expose COM Automation interfaces. COM has been the standard for extension and automation on Windows for a long time. NET is creeping up there but COM interfaces are always guaranteed to be available from Windows Explorer to IE to Excel to Exchange. Much of the time the .NET interfaces are just wrappers around COM interfaces., You can use any language that can bind to COM objects - Python for instance with the win32 modules, or PHP. Activestate also provides COM binding libs with their Perl and other scripting language distros. As for language choice, well funny enough I always found Visual Basic 6 to be the fastest and easiest way to work with COM.
Muppet Show > Monty Python
Is there a list of should and should not dos when it comes to Windows automation?"
Am i the only one who had to look twice?
And is the end of the world near, or did the random quote generator resume only to come up with like 5 pages of craziness?
DBD::WMI
Win32::OLE
Win32::WMIC
Win32::GuiTest
Win32::TieRegistry
Win32::Unicode
Win32::API
Inline::C
Perl::Dist::WiX
Win32::MSI::HighLevel
http://search.cpan.org/search?query=win32&mode=dist
http://roth.net/
http://books.perl.org/books
Perl Books - Book: Perl for System Administration
http://www.perl.com/
http://strawberryperl.com/
http://www.cava.co.uk/citrusperl/strawberry.html
google:// site:http://perlmonks.org/
Since I still have to support Windows XP systems, I use what will work with them. I also don't like to have to install any additional run-times. You can accomplish almost anything with a combination of Group Policy and Visual Basic scripts. I would also highly recommended AutoIt, which i find to be a very helpful tool, especially when trying to automate anything that has a GUI. Finally I would note that what you used to MAN, now just look up on MSDN. I just use google as follows: "msdn [terms]".
If you're comfortable with Perl, it's definitely something you can bring with you to the Windows world. You can use Strawberry Perl with or without Cygwin. If you need to deploy code, you can use PAR::Packer's pp utility to wrap your script, its included modules, and the perl interpreter into a standalone executable.
Perl can do a ridiculous amount of stuff when you start using modules from CPAN. The Windows world is very GUI heavy, but a lot of things can be done using CPAN modules. Writing to Word docs. Controlling VMWare. Networking with just about every protocol you can think of. Creating GUI applications for others to use. There's also a nifty little module called Win32::GuiTest which will let you programmatically control GUI programs that don't provide a command line interface.
Good luck!
For those things where clicking is unavoidable, MJT Macro Scheduler not only allows you to automate it, but to run in client/server mode, so you can have clients that take macroing commands from a central server. This Enterprise version isn't exactly cheap, but it can be a great way to get around the most aggravating repetitive tasks for which scripting doesn't work.
For normal Windows stuff (including COM/OLE/whatevertheheckitscallednow automation) you'll definitely want Mark Hammond's py2in32.
Even so, you'll be scratching your head and reinventing a few wheels if you want to do automation of arbitrary preexisting programs.
Last time I did any Windows automation (which was a long time ago because I stay as far away from windows as possible), I used the PyWinAuto module, which lets you interact with other programs quite handily from inside your Python script.
Been there, been asked to to that. In my case, management had their head up their ass.
In a s/w project, part of the platform architecture decision is: What resources do we have available to support each option? By the time the Windows/Exchange/Sharepoint decision was made, the resources available to support development and maintenance should already have been identified. So some genius picked the platform and then realized that all they had was a *NIX guy?
You don't decide to field a football team (yes, but what kind of football, you ask) and then realize that all you have are basketball players.
In my case, I offered to distill the requirements down to a platform independent set and build the system the way I knew how. They could go find some Windows wizards and give their implementation a try. The Windows version was never completed.
Have gnu, will travel.
Best Practices aren't just an abstract outline of how to solve a problem, they also incorporate some level of practical detail as to how to go about solving the problem. Best Practices don't necessarily (or should) include implementation details, but they must be at least aware of how the implementation differences impact the choice of solutions and methodologies.
In my case, I help run the development build system for Oracle's JDK and the associated OpenJDK project. We have Linux, Solaris, Windows, and Mac to deal with, across multiple CPU architectures. While the conceptual design of the system started out being simple, we've found that the quirks of each OS require a rather significantly different approach to solving the same problem on each platform. What works on Solaris doesn't work well under Windows, et al. This kind of thing is common in multi-platform environments, where the most efficient method of doing someone on one platform can be severely sub-optimal other places.
Back to the Question: my answer is that you should likely give up on the techniques you found worked well under *nix. You want to use the Native Windows tools. Powershell and a native Python implementation work very well for basic stuff, but you'll likely want to crack open a copy of Visual Studio and actually do some .Net development for anything rather complex. I've particularly found that several common assumptions under *nix aren't true in Windows (e.g. process manipulation/management is very different), and you really need to move into the Windows mindset to get a good design and implementation going.
There are always four sides to every story: your side, their side, the truth, and what really happened.
Try it. The PyWin32 mailing list is also very helpful.
> so what approach and methodology is best for developing, say, the automation required for a test system?
Suicide?
RUN
RUN like the wind
Get as far away as you can get.
Why
Ruby has for years come with a wrapper library for Win32OLE.DLL, which in turn allows you to automate any program that acts as an OLE client, like IE, Excel, etc.
As long as the program has OLE hooks, you can control whatever functions it makes available via OLE. Another tool that is often used with Ruby in the Windows environment is AutoIt!, a Windows scripting tool, which itself (I believe) operates through the OLE interface.
Application called "Scheduled Tasked" (usually in Program --> Accessories --> System Tools) is equivalent to setting Cron Jobs.
Group Policy (Active Directory and Appropriate package must be installed) can distribute and run your scripts (say for users at login).
Backup frequently, being sure to include System States.
I have had similar problem. I cant say its good or bad but I adpated myself to use batch programming but I heavily use Gnuwin32 ports and UnxUtils that give most of the Unix commands as exe http://gnuwin32.sourceforge.net/ http://sourceforge.net/projects/unxutils/ Happy Hacker
PowerShell is your friend.
I don't know why you think that Cygwin cannot deal with Windows type stuff like Exchange and Sharepoint. You're a little vague as to what you need to do with them. I use Cygwin while on Windows and find it can do just about all I want to do.
There's a product called PowerWF Studio from Devfarm software that combines the best of building block "drag and drop" workflow development with an excellent Powershell editor. Definitely worth a look for this type of situation.
FD: I don't work for the company, but I happen to be friends with the owner. I'd recommend looking at it regardless.
I'm about 2/3 Windows and 1/3 Linux sysadmin. I've done perl, bash, DOS batch, vbscript, Powershell (etc). I honestly feel much more power over my enterprise with the Windows tools than I do Linux. It's probably a combination of a couple things:
Active Directory for single-sign-on (no authentication) admin access to all my boxes, by just opening one command prompt / Powershell window with Run-As and my admin credentials. So in one window I can do anything to any of our 1500 Windows servers in my enterprise. I don't have to mess with sending alternate credentials to each server, or accepting/caching SSH keys, or any of that.
While archaic at times, the structured availability of all the WMI classes is SO powerful. While similar in ways to the /proc system, I find a lot more value. And you're not grepping config files, where unexpected content or typos will mess you up.
(Oversimplified) one-liner Powershell to get the BIOS versions for all your servers? $servers just needs to be an array of computernames. Can be populated from anywhere (a database, a text file, or Active Directory query)
gwmi win32_bios -computer $servers | select __SERVER, ReleaseDate | sort ReleaseDate ..love it.
Quick Test Pro really is a good product, but it's really expensive too. If you want a free (if not Free) alternative, look into AutoIt [autoitscript.com]. It has a SciTe editor that comes with it, which can help with setting up automation by recording. (Never use recording, on any GUI automation system, without at least looking over the script; but it gives you a nice baseline to work from.)
(T>t && O(n)--) == sqrt(666)
There is a free limited version of the TakeCommand (just console, no gui, and no script interpreters) but including their very nice extentions to the bach language
I may be out of fashion, but love to script windows in rexx, and the win32 ibm code base its now free and open source
For frecuently needed windows specific, recurrent, and platform idiosincratic tasks, nothing beats the ease of use and flexibility of AutoIT (or AutoHotKey... both freeware)
Using the Console2 project in sourceforge you can have a free tabbed terminal window
Or just buy the full TakeCommand (some people here use radmin instead)
Having done this myself (a bunch of Windows servers that were kept around because they were needed for a particular proprietary app), the right answer that works is actually Cygwin. With sshd on every box. It's the only halfway sane way.
If you're not allowed to install Cygwin, you're working with one hand tied behind your back. Get your resume in order.
The important thing here: Keep a local Cygwin repo, in which you copy all those downloaded files to an accessible place on your network. It is less important that the Cygwin installation be up to date than that it be consistent. You can have DLL Hell with multiple versions of cygwin.dll happening to be present on a box ...
And not surprising.
First, get into VBScript AND VBA. I think SFU 3.5 runs on Vista/7, and Powershell is just very useful. But VBA and VBS will et you do plenty.
Trying to use Java and Javascript will frustrate in many ways, but might be helpful.
It sounds like you are trying to develop test utilities for your development. VBS/VBA are very helpful here. You could develop an Excel sheet and use VBS to create SMTP messages and drive mail to an Exchange server pretty easily. And use it to pass messages to your various platforms, including Sharepoint if someone has been listening to Microsoft consultants again. It happens even in the best organizations, don'te be embarassed.
I work with a creative guy that uses VBS to create messages for a test financial system. He's got it down so he can create 100,000+ messages that fire all errors and encompass all desired scenarios, with unique sample data if desired or a standard data set. These include ASCII, EBCDIC, packed binary/bcd/decimal/whatever- the-programmers-are-calling-this-week, BLOBs, hex, and encrypted strings. When the developers tried to do this themselves, it took them 4-5 manhours for one test, and they test 20-30 times a day at some points. He satisfies their demands in 5-10 minutes, submits via HTTPS PUT, and then parses the results and sends this back to them all in a half hour. They loathe him, because he demonstrates their futile efforts to solve the problem the same day they declared the problem solved, usually before they can schedule the celebratory luncheon. Fun to watch, since it's not my group involved...
Oh, and he's porting this to a Java app so it can be given out to the development teams and deprive them of another excuse to drag out the process. There is nothing about this development process that offshoring hasn't made substantially worse by every measure. Well, maybe the celebratory luncheons are more fun...
deleting the extra space after periods so i can stay relevant, yeah.
The most important part is error correction. There are two primary methods in the windows automation world.
In the Gibbs method, you smack the low-paid butt monkey on the back of the head sharply when he does it wrong.
In the Hill method, you give the butt monkey several quick slaps to the top of the head. This works best with older balding butt monkeys.
For the more advanced, there's also the Blackadder method where you devise a cunning plan to avoid being forced to go catch bullets in the first place. That one can work for a while, but tends to fail in the end.
Comment removed based on user account deletion
On second tought, if you have the money:
- the best native scripting alternative in the same range (from win95 to win7) probably will be the polar engineering vba
- and again, in the same support range (from win95 to win7) by far the best option for custom binaries its Delphi (7 will do) or maybe powerbasic if you only care for api and com integration and dont need rad
Autoit, free and can compile it's scripts to exe if you want. IT can even interact with GUI only probrams unattetnded
I will just had the freeware servers freeSSHd and freeFTPd for a perfect scripted remote administrated windows on the cheap
http://www.freesshd.com/
Or for the cygwin based mobassh for a more linux feeling
http://mobassh.mobatek.net/
As soon as you mentioned Exchange and Sharepoint, you're fucked ... the only way you could make it worse it to mention Project. Crap software written by large crowds of overpaid geeks who don't respect their company anyways. Get a grip. If you're 'next project' is M$ then don't come whining to the crowd. Suck up and handle the shit in your britches
well, let's review. You've done automation with Unix before ... check ... So I'm guessing your not a complete moron. Now you are switching to Windows. Wow!!! I don't think any of us has done that. Another project on a different OS?!?! you didn't!! ... There's no *way* you're doing projects on one OS and then switching to another!!! Just wow!! ... oh ok, all kidding and bs aside. If you're for real, do a google search and figure it out. If you're just a fucking shill then ... ... ... just sayin
How about Perl or ReXX instead of Powershell? What would slashdotters see as their advantages and disadvantages compared to Powershell?
There is likely to be a reasonable selection of earlier work available in both, even for Windows.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
...so you can write scripts that depend on the format and have them work in 2032.
Just saying.
-- Terry
I would like to subscribe to your newsletter.
Try the veal.
-- Terry
I highly recommend writing IronPython scripts and having them regular run using the Windows Task Scheduler. This has worked remarkably well for us.
I got laughed at the first time I suggested it, but now my IronPython scripts are well in demand for such tasks.
Y
cmd.exe has the most horrifying syntax and evaluation model imaginable. if you need anything more than a few lines of code, ditch it
See: http://www.opsi.org/ (opsi is an open source Client Management System for Windows clients and is based on Linux servers.)
If there's any language you should learn for automation on windows, it's Python. I've worked as a build engineer in several companies and python has always been my rock in the windows environment. My background was awk and bash scripting but once I got a taste of Python, I never looked back.
What automation tool is best for you depends, of course, a lot on what you need to do.
I've used Winbatch for years now. For basic stuff it is straightforward and easy to learn. But it still has the capabilities of performing sophisticated and complicated automated work.
With the right licence you can deploy compiled scripts to as many computers you like. I have dozens of scripts I use in this manner every day.
we make production lines and production line modules of all sort and we have pretty much everything run on windows, i have mostly patched up all my testing systems based on NI softwares, teststand + nidaq + NI488.2 + some .net code and you can get your system up and running in no time and trouble. getting tens of devices working together, controlling IO-s and measurement equipment over serial, GPIB, ethernet, USB or whatever is easy with these tools. NI Teststand is specialy crafted for testing systems and its great at it. in telecom devices testing its pretty much industry standard. some early attempts i made based on labview, very easy to make but result is not very good and maintainability is horrorshow
and start preparing for interviews. That is the only option that will let you keep your sanity.
I am just now using Python for an automation project.
It's a great glue to hold things together. With built in modules and pyserial, you've got IO covered. With pygame, you've got joystick and graphics support. Numpy handles your matrix inversions and vector calculus. Glade and Pygtk lets you create nifty GUIs. And Matplotlib handles the outputs.
Great plus: if you still decide to migrate to *nix, you won't have to rewrite a thing.
Best, Jonathan
I've been happy with a combination of CMD plus Jscript/COM/WMI integration but avoid VBscript due to the lack of robust error checking whereas Jscript has try/catch(ex) and is a more robust and familiar language. There aren't as many examples in Jscript but you can always convert from VBscript.
Powershell is a little like Perl and is the new standard where it can be found.
Search technet and msdn for things from The Scripting Guy and that should lead you to most of the useful references.
I also found MSI, via WiX, a useful way to distribute changes and you can build an MSI installer that doesn't actually install, I.e. It doesn't register software but it can run Jscript and deliver binary files.
Using Jscript in HTA is a great way to make web-gui-based admin tools.
WTF is wrong with the Slashdot UI? Comment threads are impossible to follow. Click to expand a comment and the screen jumps around to anywhere but the thread you are trying to follow. It's been doing this for what seems like a few weeks now and it has become tiresome in the extreme.
Windows wasn't designed for automation/server tasks. I predict lots of pain in your future.
- I've got bad karma because I won't parrot everyone else's opinion
I wish Microsoft employees would stop posting questions on how to do their job on /.
If you know Perl, use it. There are lots of Win32 modules to help manage Windows. Additionally, you can compile using perl2exe and run your script on boxes that don't have Perl installed. Otherwise, learn Powershell.The specialized functions for managing Windows can't be beat for quick script creation.
Maybe this fits your needs: http://channel9.msdn.com/Blogs/kmcgrath/Introduction-to-Creating-Coded-UI-Tests-with-Visual-Studio-2010
Utilities
Pay attention to the suggestions above from folks that already recommended good utility suites such as SysInternals, NirSoft, AutoIt!, etc. since they pretty much covered most of them.
Look into GnuWin32 for native Linux utilities such as sed, awk, grep, coreutils, textutils, and all the other utilities that you'd find on a Linux system for your windows servers. The only disadvantage is that you have to include two of the large multi-language libs that all exe from GnuWin32 depend on. (Use Depends to find out which ones.)
PowerShell .NET Framework (New-Object -TypeName System.Namespace... or directly with [System.Namespace...]), COM+ Automation Objects with (New-Object -ComObject ProdID), or execute utilities directly using their names or command prompt commands using cmd.exe /c.
I would recommended this now as a better language with more modern features over Command Prompt (Batch) and you can leverage it to execute all the other utilities and commands within and capture then process their stdin/err output simply. As the parent poster already mentioned that from within PS you can access a lot of the API's directly for automating windows such as Windows Management Interface (WMI) to query a lot of info but do very little,
Within PowerShell you have Invoke-Command and Enter-PSSession to simulate command line access similar to SSH as long as the remote server has WinRM (Windows Remote Management) interface installed and configured. You also have Job control and create by using commands with the -AsJob parameter or Start-Job commands. Some of the syntax in PS is borrowed from Bash and the pipeline ideaology of using output from one command and passing it to another as input is highly leveraged in PS and improved upon since it is object oriented allowing you to manipulate and filter out individual properties within those objects when they get passed down the pipeline giving you easier manipulation power over text and objects than you had with piping to sed, awk, or grep in your own scripts. (I still do quite a log of Batch scripting and piping to sed, grep, awk that I have on my system from the GnuWin32 utilities that I've installed that I find better and simpler to use directly than Cygwin.)
Many of the newer Microsoft and major applications include direct PowerShell modules to allow control of them and I've successfully used these modules for VMware, Citrix, SQL, Exchange, Sharepoint.
COM+ Automation
Other times I've directly accessed COM+ Automation object interfaces that are available from other software exe and dll libraries by discovering their ProgIDs with OLEView COM Automation listing or by directly interrogating the exe and dll files using Visual Studio Object Browser. Once you find the ProgID then you can look at the methods, properties, events, and other members that are exposed from within those objects and determine if you can instantiate or access instances of those objects to connect to these applications and control them using those object members. I've been able to discover a few applications that have these interfaces that are not very well publicised or documented by the vendors and use those to programmatically automate some third-party applications. PowerShell makes that easy as long as you don't have to retrieve or pass complex binary structures, otherwise you'd have to go to full C, C++, or C#.
64-to-32-bit
One thing about PowerShell is that you now have to be aware of 64-to-32-bit issues with Windows and the Windows/SysWow64 (32-bit windows apps and libraries on a 64-bit system) and Wow6432Node (32-bit Registry mapped on a 64-bit system).
There is also a bit of a caveat to be aware of when interfacing with COM+ objects across the 64-to-32-bit barrier on the same system since the 32-bit COM+ objects registered on a 64-bit system are only accessible from with a 32-bit instance of PowerShell x86 and remotely these 32-bit COM+ objects are only accessible in a 32
Believe me, coming from a Unix background you will go bat shit insane trying to figure out how to do the simplest of tasks on Windows. I have 8 years of Windows development experience and seriously contemplate leaving my job whenever I have maintain Windows code.
In my experience, python is a wonderful solution for these sorts of problems. With it's windows modules you can script using COM, manipulate the registry, access local and remote databases easily. Using the ssh protocol you can use standard deployment tools like fabric. Finally, you are using python so your scripts are likely to be easy to read and maintain. I create a standard local, private python installation for the purpose.
It's a native Win32 port of tcsh, with a bunch of the *nix tools you like. The downside is a) it's csh and b) you have to either pay $350 for a shell or pirate it.
The poster specifically mentioned exchange and sharepoint. Microsoft has created special PowerShell extensions just for those products. Not only can it leverage WMI, VBS and the rest of the windows alphabet soup, but powershell is a hell of a lot more secure out of the box (i.e. no Win server will execute powershell code unless the execution policy is set and the code is trusted within that scope).
PowerShell is a very, very concise language. Most of my VBs scripts could be compressed from fifty or 100 lines down to 3 lines or so. It's completely different syntax than anything else, but overall it's where the industry is headed.
I highly recommend you pick up a book or two on it, it's not hard to learn, but 9 times out of 10, you are trying to do the exact same thing someone else has already done very elegantly.
For enterprise stuff, make sure you are using code signing and ideally scoping your execution policy to accept only signed code from your company.
As far as scheduling, Windows Task Scheduler = Cron (ok, so maybe not 100% equal, but that's what us Win guys use to kick off scripts).
Also, you are probably in the wrong place to ask for Windows Enterprise advice, just an FYI.
Doesn't python work on windows? I don't know about integrating with sharepoint or outlook though them programs is bad.
Of your sixteen uses of the word 'your' (possesive), it was only used six times correctly. The other ten times, you meant 'you're' (contraction of 'you are'). The content of your post seemed intelligent, but your delivery makes it seem as if you're uneducated, or uncaring/lazy in your writing.
I'm not sure what's scarier.
The fact that this guy is on essentially a *nix site with knowledgeable *nix users to ask for help but he's asking for Windows help....
or
the fact that a lot of the people posting answers to his question are obviously M$ knowledgeable and posting answers to his questions on a *nix site....
. .. ... /. has been invaded by the M$ Borg
I'm good with numbers -
If it's not in the .net framework, you can easily utilize c/c++ libs, also inserting jsp/html/vb is just as easy. And if you need more raw manipulation the Win32 libs via c/c++ work just as well.
I've created many automated applications in C# and have yet to find anything I cannot do...