09 November 2008

Crunching Numbers

During the last couple of day, I spent my free time trying to sort out the family finances; big (good) changes are on the horizon, we need to plan a bit better from now on, and it's amazing how "bad" expenses stick out straight away when you aggregate them, instead of relying just on rough day-by-day cash-flow estimates ("how much money is still in my account?").

I tried to use KMyMoney for this sort of thing several times in the past, but I was always inevitably thwarted by the effort required to copy records one-by-one from online bank statements, because the sort of bank accounts I use don't allow any desktop clients to automatically pull data. So this time I thought I'd fix it once and for all, and set down to write a few scripts to do that, albeit in a somehow indirect way.

KMyMoney can natively import transactions in the legacy (and wildly non-standard) QIF format, while OFX requires a plugin (why? No idea). Unfortunately, on my Debian Etch, the bloody plugin somehow never gets installed correctly, but hey, maybe one day it will, and OFX is the way of the future anyway (it's XML-based and much more exactly specified than the old plaintext-based, informal QIF). So I wrote a Python script to convert the HTML or CSV produced by my online accounts to OFX, then passed the output to ofx2qif, a handy script included in the libofx-dev package (at least in the 0.8.2 version I'm using). The result is ready to be imported in KMyMoney. Slightly cumbersome, but it does the trick. I need to add a bit more intelligence to the scripts, to speed up the categorisation effort that follows (which is the whole point of the exercise), e.g. "LINK xxxxxx" payees should all be set to "cash machine" etc, but it's already working fairly well.

The effect was startling; finally, all my expenses are tracked and I can properly budget and forecast. (...How the hell I'm spending so much on mobile-phone calls??)

I'm actually slightly pissed off that cash transactions are now so opaque; I've no idea why £50 were withdrawn from an ATM on that January day (even though I'm sure it made sense when I first checked the statement 6+ months ago), but I know for a fact that those £37.68 from last December were for a delicious Japanese dinner, as I paid for it with my debit-card.
In a way, this goes against the "classic" principle that "by using plastic, you never really know how much money you don't have" (so you tend to spend more). I still believe in that principle, and I'm slightly baffled by the evidence.

I'd strongly recommend this sort of exercise to everyone, anyway. You don't need to use a dedicated program like KMyMoney (even though it helps), Excel might be enough, as long as you can easily add transactions from your online account (via CSV or cut&paste).

Labels: creditcards, foreverythingelsetherespython, GeekDiary, kmymoney, personal, personal finances

posted by GiacomoL @ 12:17 PM   2 comments links to this post

11 October 2008

Notes on Google Finance API

I recently started dabbling in shares (yeah, I know, I like swimming against the flow), and so I looked around for good stock-price trackers for KDE3. To my surprise, I couldn't find anything apart from the usual KMyMoney (which I like and use, but it's not really something you can keep open all day on your desktop). I had already put my stock info on Google Finance, and lo, the service has a GData API, so I decided to write a small script in Python to retrieve my portfolio and the daily variations.

The good thing about GData APIs is that they are all the same at a basic level, so even when client libraries don't explicitly expose new features (like in this case, as Finance is quite a new service), you can still use the APIs to get easy authentication and feed-parsing. So I went and downloaded the official (and excellent) gdata-python-client package, which is dead-easy to use:

from gdata.service import GDataService

client = GDataService()
client.email = 'your_email@gmail.com'
client.password = 'your_password'
client.service = 'finance'

client.ProgrammaticLogin()

baseURL = "http://finance.google.com/finance/feeds/%(email)s/" % {'email':client.email}

# to get all data in one go:
#  bigFeed = client.GetFeed(baseURL + "portfolios?positions=true&returns=true")
# and then positionsFeed is included in each Entry
#  positionsFeed = gdata.GDataFeedFromString(bigfeed.entry[0].FindExtensions('feedLink')[0].children[0].ToString())

# ... but I really just want the first portfolio
positionsFeed = client.GetFeed(baseURL + "portfolios/1/positions?returns=true")

for entry in positionsFeed.entry:
 details = entry.FindExtensions('symbol')[0] 
 symbol = details.attributes['symbol']
 name = details.attributes['fullName']
 
 data = entry.FindExtensions('positionData')[0]
 totalReturn = round(float(data.attributes['returnOverall']) * 100,2)
 gainPerc = round(float(data.attributes['gainPercentage']) * 100,2)
 
 print name + " (" + symbol + ") Return: " + str(totalReturn) + "% - Gain: " + str(gainPerc) + "%" 

One should really make an applet or something for KDE, but I'm leaving for Japan in about six hours so I can't be bothered...

Labels: foreverythingelsetherespython, GeekDiary, Google, GoogleFinance

posted by GiacomoL @ 9:31 PM   0 comments links to this post

05 October 2008

Horrible Hack to get Python 2.6 on Debian or Kubuntu

I wanted to try out the newly-released 2.6 version of our beloved Python, but unfortunately Debian didn't have a package for it yet (and it still doesn't). I wasn't too afraid of screwing up my laptop, as it's probably going to be formatted very soon anyway, and I didn't want to mess around with deb build scripts, so this is what I've done:

1. got the official source distribution, untarred and cd in the resulting dir Python2.6
2. got some additional packages: apt-get install tk8.4-dev libgdbm-dev libdb-dev libreadline-dev libsqlite3-dev libncurses5-dev (and possibly a few others)
3. ./configure --prefix=/usr --enable-ipv6
4. make
5. checkinstall -D --pkgname=python2.6 --pkgversion=2.6 --maintainer=g.lacava@gmail.com --inspect --backup=yes --install=no make altinstall
   This command allowed me to review the package contents and remove what I didn't need, which is basically everything outside the "python2.6" directories and which might already exist on my system (so I didn't want to overwrite it).
   I took out the lines /usr/bin/pydoc, /usr/bin/idle and /usr/share/man/man1/python.1
   UPDATE: when checkinstall asks if you want to create a default set of docs, say "yes", or you might get an error about ranlib further down (see comments).
6. installed the produced .deb package
7. copied back pydoc and idle (from the build directory) and /usr/share/man/man1/python.1 (from the Misc directory), all with "2.6" appended. I then set up alternatives with update-alternatives --install symlink name alternative priority (mainly in order to "redebianize" my impure karma); UPDATE: well, using alternatives (a 100% Debian solution which works perfectly well for loads of other multi-version script engines) will break your system, because some developers absolutely must reinvent the wheel every 5 minutes and then proudly announce that bugs won't be fixed. The stupidity of it all is staggering.

First impressions: 2.6 seems fast as hell. I don't know if this is due to the custom compilation though, rather than improvements in the runtime.

Labels: Debian, foreverythingelsetherespython, GeekDiary, python

posted by GiacomoL @ 2:14 PM   13 comments links to this post

26 August 2008

GrEstimator - the web version

I finally bit the bullet and put online an interactive version of my GrEstimator script (which will soon see a new release, by the way).

The GrEstimator Web Service is currently very basic. You provide an email, a GoodReads ID and a shelf (or tag) you want to estimate, and the system will email the result (expressed in a currency of your choice, calculated with exchange rates from WebserviceX's Currency Converter).

The tool is beta ("almost alpha" really), so be gentle and let me know if it dies on you :)

Known issues:

• you have to provide a numerical GoodReads ID, which is the one appearing at the end of the URL when you look up a user (e.g. "http://www.goodreads.com/user/show/1383164"). I've asked to be authorized to look up an ID by providing an email, and I'm waiting for the response; once I'm allowed, you'll be able to just provide the email you use with GR.
• you cannot estimate more than 200 books on a shelf; this is a limitation of the GoodReads API.
• books not listed on Amazon will be ignored.
• it currently spawns a thread for each estimate. I have to implement a system of queues to limit the amount of threads running at any given time, just in (the remote) case the service becomes popular.
• you cannot choose the output currency, it's USD only. This will be fixed soon with a new option. Fixed.
• the result is based on average prices. I'll soon add an option to say if you want that or rather the maximum potential price (which really tends to be funny). Fixed.
• the service does go through all the Amazon locales (.com, .co.uk, .de, .fr, .ca, .jp in this order) but only if item lookup fails on the previous locale. This means that, if I find a book on .com marked as unavailable, I will still consider it as "found" and won't repeat the lookup on a different locale. I actually just realized this as I was writing the post, it will be fixed very soon. Fixed.

• producing a "blog badge".
• pulling prices from somewhere else than Amazon.
• an "update" feature of some sort would be nice.

Labels: books, foreverythingelsetherespython, GeekDiary, goodreads, grest

posted by GiacomoL @ 9:49 AM   0 comments links to this post

21 August 2008

How much is your bookshelf worth?

Just for fun, I wrote a little python script that will pull a feed of books from GoodReads and calculate their total worth according to Amazon.

Requirements:

• Python 2.5
• an Amazon Web Services Developer Key (get one from the AWS site)
• a GoodReads account

The script allows for shelf-specific filtering and supports different Amazon locales, with output configurable to be in any currency. Due to a limitation in the GoodReads API, it will estimate only the first 200 books.

You can see it in action using the web-based version

Download: GR_Estimator 1.1

• Release 1.1
  • Added support for multiple currencies
  • fixed a few bugs
• Release 1.0
  • Initial release

Labels: amazon, books, foreverythingelsetherespython, GeekDiary, goodreads, grest

posted by GiacomoL @ 6:40 PM   0 comments links to this post

19 August 2008

A lil' script

Since my itch is now scratched, I might as well make the code available: GenBooks 1.0 is the little script I used to generate my list of books by pulling my "favourites" shelf on GoodReads. It works with python templates and even embeds your Amazon Associate ID. Requires Python 2.5 (because it uses the new ETree module) and the Python Imaging Library 1.1.x. There is no license, it's all public domain. Have fun.

Labels: books, foreverythingelsetherespython, GeekDiary, goodreads

posted by GiacomoL @ 8:13 PM   4 comments links to this post

My favourite books...

... are now listed here. I generated the list while playing around with the GoodReads API and the Python Imaging Library. I'm trying to think of something else to build with the GR API... any suggestions?

Labels: books, foreverythingelsetherespython, GeekDiary, goodreads, personal

posted by GiacomoL @ 4:26 PM   0 comments links to this post

14 August 2008

Export your books from aNobii to GoodReads

Labels: anobii, foreverythingelsetherespython, GeekDiary, goodreads, librarything

posted by GiacomoL @ 8:00 AM   3 comments links to this post

16 May 2008

for everything else...

• Do you need to convert WMA files to MP3?
  Just use the free PyMedia library.
• Do you need to upload these files to an FTP?
  The ftplib module is included in the standard Python library.
• Do you need to schedule this upload?
  The standard Python library gives you the sched module.
• Do you need to persuade some people to pay you a (minimal) amount of money for the (minimal) effort of writing the script?
  I'm afraid you'll have to do that yourself...

There are things a computer cannot do.
For everything else, there's Python

Labels: foreverythingelsetherespython, ftplib, GeekDiary, pymedia, python, sched

posted by GiacomoL @ 11:18 AM   2 comments links to this post

08 April 2008

Google App Engine

Google App Engine. Write applications in Python using a WSGI-compatible application framework, then host them on Google’s highly scalable infrastructure.

"Holy s**t" was Ryan Tomayko's comment, and my first thought as well. Google is realising the promises of WSGI, and what a sight it is.
On one hand, this is fantastic; it made me think of "J2EE done right". On the other hand, if you use GAE, you are handing your entire infrastructure and data (!) over to Google, which might not be the smartest move for lots of companies (any FTSE100 for example, and probably any NASDAQ-listed as well).

Anyway, Python rocks.

Labels: django, foreverythingelsetherespython, GeekDiary, Google, python, WSGI

posted by GiacomoL @ 12:04 PM   0 comments links to this post