Slashdot Mirror
Specifications of Intuit's .QFX Format?
mad.frog asks: "I recently upgraded my ancient version of Quicken to the latest (Macintosh) version, with the intention of being able to download my credit card transactions directly from my bank into Quicken, rather than entering all that stuff by hand. As it turns out, almost no banks support doing this for non-Windows platforms (not surprisingly, Intuit doesn't point this out on the package). But here's the weird part: the information downloaded is just an xml-like text file (.QFX). Anyone know how (or why) they would make such a generic file platform-specific -- what business advantage does Intuit (or my bank) have in restricting how I use this information? Also, does anyone happen to know details of this (apparently undocumented, Intuit-specific) format so that I can hack mine into submission and use this data anyway, even if it's not on my bank's Platform Of Choice?"
OTOH, it would be in violation of the DMCA, not to mention stealing from Intuit and your bank, so maybe you better not confess to it here.
I believe that Quicken has some sort of API for adding transactions. I haven't used it in the past several years, but I know that around '98 or so, they had some sort of pluggable system for connecting to various internet banking systems. Perhaps you should stat by talking to Intuit.
My bank's wouldn't import right. They weren't using windows linefeeds.... they didn't set the cleared flags, either. What use is that?
la lalala! Perl to the rescue.
I did an export, examined the code, and wrote a little script to fix the downloaded file into what I needed (replace linefeeds, add the cleared flag).
Here is my code; it might be good for starters. Not sure what the new version needs.
------------
print "I fix QIF files!\n\nEnter Filename: ";
chomp($filein=<STDIN>);
print "Working on $filein.\n";
open (FLN, "<$filein");
@raw=<FLN>;
close FLN;
open (FLO, ">Fixed_$filein");
foreach (@raw)
{
chomp $raw[$a];
print FLO "$raw[$a]\n";
$test=(substr $raw[$a], 0, 1);
if ($test eq "D")
{print FLO "C*\n"; }
$a++;
}
close FLO;
print "Finished!\nhit return and close window.";
<STDIN>;
-----
"You may all go to hell and I will go to Texas"
Sen. Davy Crocket to US Congress, Nov. 1, 1835
hack mine into submission
You trust your electronic banking to an unsupported hack you're going to whip up?
You're braver than I am...
May we never see th
I have a problem with Quick 2003 for the Mac. Citibank says to use a PIN to access my data. I have set up an account at the Citi online website, but *neither* the PIN or my Citi online password access the data. What gives? Does anybody else know which of these two to use? One of them worked at one time, but now nothing!!
what business advantage does Intuit (or my bank) have in restricting
The key word there is support. Just because something isn't supported, that doesn't mean it wont work. Have you tried it?
Very often this is simply done to reduce the load on the company's tech support line. (Either Intuit or the bank or both). Its hard enough to get competent folks to man your support lines, without having to train them on things they will only encounter once in a hundred calls. So they save some money by going lowest-common-denominator.
This probably isn't it, but it wouldn't hurt for you to open the file and change the cr to crlf since that is a distinction b/t dos and bsd.
Intuit is a bastard of a company.
... fire up Quicken ... click Import:
I happen to use Quicken 2001 on a Mac.
When I switched banks at the beginning of the year, I asked if they supported Quicken.
They said, "Yes. You can download your transactions into Quicken."
I didn't dig any further since, my assumption was, once I get the file downloaded, Quicken should just import the data. Well, that's not quite the case.
I go online and download the a "Web Connect" (QFX) file for my account and it gets saved to my desktop. I flip over to Quicken and say "Import Web Connect". Quicken opens the file, proceed to connect to the internet and reports back "Quicken is currently unable to verify the financial institution information for this download. Please try again later."
Hmmm... that's strange. Why did it go online? Why didn't it just import the file?
After a few days of trying different things and talking to the bank, I decided to break down and call Intuit Tech Support ($1.95/minute). After being on hold for awhile and talking to a technician, I am told "Your bank doesn't support Macs". The rest of the conversation was along the lines:
Me: "What do you mean?"
Tech: "You can't download transactions into Quicken from your bank because they don't support Macs."
Me: "That doesn't make sense. My bank is not the issue. I have the QFX file - which is the same file that I would get if I were on a PC - Quicken is just refusing to import it."
Tech: "That's because your bank doesn't support Macs."
Me: "I already have the file from the bank; Quicken just needs to read it in."
Tech: "Quicken needs a QFX file formatted for Macs. Your bank would need to pay to have different servers for Macs."
At this point, I knew I was screwed. So, I started thinking, what information can Quicken be sending when it goes online during an import? The OS, probably. But, what else? I fired up BBEdit and opened the QFX file. They probably wouldn't send account or transaction info - leaving mainly the following parameters:
ORG (Bank name)
FID (Some ID internal to Intuit)
INTU.BID (Same as FID)
BANKID (Bank Routing Number)
So, for kicks, I called up a buddy of mine who has an account at another bank that has "Mac Access" and asked him to download a QFX file and give me those parameters.
A quick BBedit
Viola! The file imported with no issues. All the online connection does is connect back to Intuit and ask "Has this bank paid for Mac support?" - if the answer is "no", the import is stopped.
Now, since I don't use any Quicken Online Banking features, I can't vouch for how it affects those (I would expect them to fail since the routing numbers are wrong). But, as a pure import facility, there are no issues doing it this way.... which happens to be the only feature that most people I know use (importing vs. Quicken Online Banking).
So, Intuit is going out of their way to make pay for "Mac Support" when it doesn't cost them a cent more. Sure, development of Quicken for Mac might cost more, but that's why it's double the price of the PC version. We pay for certain functions in Quicken - importing transactions is one and to prevent us from doing so because they want more money from our banks just doesn't seem right.
There's no reason why you have to use the QFX format for this. Most lending institutions will support the QIF format, which can be imported quite nicely into Quicken 2003 for Mac OS X. (I know, because I did it just last night. :) I don't know what the QFX format does, but I suspect that it allows Quicken to actually interact with your credit card, allowing payments to be made from within Quicken. QIF, on the other hand, merely provides a record of transaction that must be manually imported into Quicken each time.
Now, I can't speak for Quicken 2003, because I haven't tried it, and because Intuit's site doesn't go into details...but, Intuit does have a guide on how to get QFX files working on the Mac. Have you tried that?
-Waldo Jaquith
I have several years of financial data on Quicken which I found tedious to use. But Quicken lets you export the data in the (now not-supported) QIF format, which turns out to be a fairly straightforward text file, which then can be read into some other custom-made application. (I am in the process of loading this into a simple homemade MS ACCESS application).
You can get the spec here (but you must register first).
Intuity's Developer Network let's you download both the QuickBooks SDK, as well as their QuickBase API. Here it is:
http://developer.intuit.com/
The QuickBase API evidently supports VB, Perl, and Java, so you have a platform independent option available.
While I can't definately state that this will help you, in my own experience, I've been able to get at/do everything I need to in QuickBooks using these products.
Good luck!
Open Financial eXchange is a format invented by Intuit, M$, and somebody else. QFX is Intuit's variant that I *think* extends OFX slightly in ways interesting to Intuit.
QIF is an old format that I believe banks are interested in dropping in favor of the more flexible OFX format.
Innovision is one company that sells an OFX server to banks.
"Anyone know how (or why) they would make such a generic file platform-specific -- what business advantage does Intuit (or my bank) have in restricting how I use this information?"
.. well .. that's great, but I think we can get a real business advantage by keeping all of that stuff secret and not using it. Let's just ship the Windows version."
.. they might be working on it. My guess is that the smart thing to do is develop Windows stuff first, make 99% of your users happy, and develop other stuff later.
Yeah, I can see the business meeting now:
Intuit Programmer: "OK. We have written software that works on all possible platforms. It's all debugged and ready to go."
Intuit Business Guy: "Really? It runs perfectly on all platforms, and you're done with the whole debugging cycle? Didn't that cost a whole lot more than just writing code for the one platform that 99% of our users use?"
Intuit Programmer: "No. The magical code elves wrote all of the extra code for us. The code gnomes tested it. We paid them in fairy dust; no actual money involved."
Intuit Buisness Guy: "Hmmn
The question you should be asking is "how much of a business advantage would Intuit gain from investing the resources to develop this stuff for multiple platforms?" My guess is that it's just not worth it to them at this point. But who knows
The QFX file format is a standard implementation of OFX. For those not in the know, OFX (Open Financial eXchange) is an XML standard for financial data exchange. This is supported by a number of third party financial software providers (including my glorious employer). If your bank doesn't have OFX support, then you're pretty much up a creek. QIF, Intuit's older data format, is pretty much dead now. I don't know if they still support it, but from what I've been told support is sunsetting rapidly.
I don't know who told you that no banks support QFX/OFX for non-windows platforms, it is a platform independent standard. Unless, of course, your bank purchased a windows/IE specific solution. I know our software does work on mac versions of Quicken (I wrote it), and if they follow the intuit example code it is trivial to write.
The OFX standard, including the DTD, can be found here
If memory serves (which it may not), doesn't Microsoft own Intuit?
While QFX was an early standard, it is now only supported by Quicken and legacy versions of Money. Years ago OFX was started to act as a sort of standards body for downloadable transactions.
I believe that GNUCash has OFX support in development and there is a LGPL OFXLib on Freshmeat or on SourceForge. You could also go to www.ofx.net to get the specs (though you will need to sign up).
For me Quicken has lost most of its advantages over the years. I started using it back in '95 and at the time I thought it was the greatest thing ever. If your bank supported it (and I choose banks based on this) you could download all your transactions and launch your bill pay, and you could even mail out checks which saved me a lot of postage (supposedly the bank backend did EFT's instead of printed checks, and thats now they made the money).
Nearly eight years later, nobody offers free Quicken access (prices range from typical $5.95 a month to a staggering $26.95 that Bank Of America wants for Quicken access) and it doesn't matter if your using billpay or not (HSBC does allow you to download transactions for free, but thats atypical though hardly unexpected for a world class bank like them). I can still download my credit card transactions automatically without a fee, but for my unsupported credit cards and for all my checking and savings accounts I have to manually log into the web sites and download the transactions, then manually import them into Quicken which is still easier then entering in everything by hand but not by much.
At one point I had a web based transaction manager I wrote which would screen-scrape transactions from a couple of the banks I used in liu of Quicken (or QIF/QFX downloads because I knew they would be the next to go). I was seriously considering dumping Quicken and switching over to this, then one day someone exploited a security hole in Bind and erased my hard drive. =( I really haven't had the heart yet to sit down and rewrite the application.
Anyhow, what I am wanting to impress upon readers are the issues that I hate about online banking as well as import of files from a bank - to save time from hand entering transactions.
On the surface, the idea that you could import these transactions, or call them all up in a web browser, sounds good at first glance. Saves you the typing, no need for writing down the transactions, etc, right? But that is where the problem lies.
When you write down your transactions, or hand enter them (from receipts) - at the end of the month when you balance your account your version had better match up with the bank's version - if it doesn't then there is a problem, and the job is to figure out if it is on thier end, or your end.
Say you go to a restaurant and order dinner, and pay for it with your debit card. You enter the transaction at home that night. A month later you receive your bank statement, and during your account balancing session you notice that on the same night are two transactions for the same amount at that restaurant - but you know you only had dinner once, based on your accounting entry (this is not really a made up story - I have had it happen to me several times, at different restaurants).
Had you been importing the account transactions, or been looking at the "web-statement" online, you might very well miss the problem (there is a way around this, kinda a "reverse-balancing" system - whereby you compare the statement to your collection of receipts, to see if all the receipts are accounted for and only used once - but this system has flaws, mainly due to needing to read each line, possibly skipping lines, and other issues) - but you would never know you lost some money...
It really is bad accounting practice - I doubt banks and businesses would use such a system - so why should individuals? I tend to think this issue isn't brought up much because these mistakes, since they aren't as easily caught by individuals, actually cause more money to be earned by the banks and businesses (especially if the businesses and customers bank at the same bank - the money never leaves the bank, so bank doesn't care, and the business has made double money off of one transaction - why should they care?).
Good accounting practices are to keep a separate ledger of your own, and don't keep two separate running copies of the ledger (to avoid double entry errors). Then, at the end of the month compare your ledger to the bank's.
This is one thing I like about the older CheckFree software (and the only thing keeping me on Windows) - you had basically a check ledger for each of your accounts, and you could enter in transactions - at the end of the month you enter starting and ending amounts, and check off each transaction as you see it on your statement. If you have the same number of transactions with the same amounts, the total at the end comes out to be $0.00 - indicating the account it balanced with no descrepancies. It has save me many times.
I tried to go with on-line service with BofA (where I bank), but it doesn't offer that kind of feature, nor does it have the option to pay online to merchants who don't take EFT payments through them (whereas CheckFree will cut a paper check and mail it). I have yet to find a solution under Linux that will work for me (GNUcash would be it if it had EFT support, but without it I would need to use some other electronic payment system, and also do double entry, which breaks one of the main good accounting practices). If anyone knows of one, let me know (I have considered running the CheckFree software under Wine, as well)...
Reason is the Path to God - Anon
Careful! You just posted a security altering device that may fall under the jurisdiction of the DMCA. Perpare for an appropriate long term prison sentence.
I believe the QFX format is for the most part OFX (with one proprietary tag thrown in). The tag I believe is 00002. Information on OFX can be found here: http://www.ofx.net/ofx/default.asp. If your bank supports MS OFX than adding the Intuit bastardization tag should get you up and running the MS implementation from my experiece follows the spec.
http://libofx.sourceforge.net/
You're doing unfiltered transformations - sed or awk is just as good if not better.
Perl is for doing all sorts of fancy things.
Awk is for applying rules (unlimited on what they can do) based on regex matching.
Sed is for linear transformations based on regex matching.
Yours is a linear transformation and can be done in awk or sed which are basically file filters.
sed "s/^D/C/" file | sed "s//\n/" > file.out
Where is and is CR.
Your mileage may vary and you may have to use a different comparator.
Awk syntax is very similar to C in an interpreted environment, and can be done on the command line as well, or alternatively in a file. It's what C might have been like if the C interpreter had ever taken off.
Sed and Awk are very powerful for regular expression matching. Perl to me is better used as a decision making tool than a transformation tool.
Sed and awk also allow for back-references in the regexes if memory serves.
My 2 cents.