Dev-Picayune

Okay, very weird problem today. I am not sure what to make of it… when using the latest version of MySQLdb (1.2.2.final.0) which is the version in Ubuntu 7.10 Gutsy, I am getting an “incomplete format” error when performing an executemany. My variables are presented as a list of dictionaries as I’m using named/mapped variables in the SQL statement. Using the exact same code with MySQLdb (1.2.1.final.2) is fine (the version in Ubuntu Feisty 7.04).

So for completeness sake, I have documented the scenario like this:

On a MySQL server, you just need a simple little table:

CREATE TABLE `test_table` (
`id` int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`test_col_1` varchar(20) NOT NULL,
`test_col_2` varchar(20) NOT NULL,
PRIMARY KEY  (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=latin1

Then the Python code (obviously you’ll need to set the connection stuff appropriately):

import MySQLdb
 
print "-----------------------------------------------------"
print MySQLdb.version_info
print "-----------------------------------------------------"
 
sqlStringItems = """INSERT INTO test_table
(test_col_1,test_col_2)
VALUES
(%(test1)s, %(test2)s);"""
 
dtlDict = {'test1':'test value 1','test2':'test value 2'}
 
con = MySQLdb.connect(host="localhost",user="myuser",passwd="mypassword",db="test")
cur = con.cursor()
exitval = cur.executemany(sqlStringItems,[dtlDict])
 
print "exit value from executemany= " + str(exitval)
cur.close()

version (1, 2, 2, ‘final’, 0) yields:

ValueError: incomplete format

Whereas version (1, 2, 1, ‘final’, 2) works fine.

Also, doing the same thing, but with execute instead of executemany works fine (and of course taking the dict out of the list). So this seems to be purely an issue with executemany. Looking at the MySQLdb code, it’s obvious that between the two versions it was pretty radically changed.

I guess I need to report this as a bug, but I’m just finding it hard to believe that I didn’t do something wrong.

Update (2008-06-03)
Back in January, I reported this issue (1874176). Then in late April, Raphael Guillet submitted a possible fix. I haven’t spent much time looking at the right vs. wrong of the fix, but I have tried it and it appears to work. So for those looking, here’s the possible fix which as near as I can tell has still not been cut into the official version.

The change is in the cursors.py file. So you have to hunt that down wherever it is stored in your particular OS version and/or install.

Comment out lines 201 and 202:

        #e = m.end(1)
        #qv = m.group(1)

And then add right after that:

        e=len(query)
        qv = query[p:e]

I’ll try to continue to monitor this.

So just as a bookmark to myself, I wanted to document what all I was working on from a technical perspective.

On the work front, I’ve recently deployed a Django app that folks are just totally loving. It’s actually a hacked together group of scripts to import data and export data to two different legacy systems. In the middle, I use the Django admin stuff to allow folks to maintain the data, and scripts at the beginning and end of the day handle import and export duties.

I finally started hacking around on WordPress themes. I’ve not gotten very far but hopefully in the next week or so, I’ll have new designs for both sites.

Then, with regard to PyCon, I submitted some talk proposals. Not sure I should have bothered, but it seemed like the thing to do. Now I’m just curious to see if any of my proposals are accepted. One of the proposals is for a project I’ve only begun really gathering the pieces for. I’ve resolved that whether the talk is approved or not, I’m going to work away at the project and then I’ll either have a talk or I’ll have a lightning talk to present. I’ll be blogging my way through my progress here, so should have some interesting stuff to post.

I’ve finally gotten tired of my poor little blog getting attacked by spammers. Actually, it was just one in particular and I know I could’ve done some IP blocking or added to the protection scheme on my comments, but rather than working more on Pumpkinvine, I’ve now switched to WordPress. And while it seems like I’m giving up, it’s really so I have the chance to catch my breath while I use all the WordPress plugins to handle the spam for me. But for reference, I ought to package up my most recent version of Pumpkinvine and make it available for folks although it’s really a pretty rough collection of PHP scripts that approximate the same style as blosxom. So why bother creating another system? But it seemed like a good idea at the time. Probably the neatest part of Pumpkinvine was going to be the switchable back-ends so you could choose between text files, SQLite, MySQL or something else by just changing a single setting. Of course, the only back-end I had done was the text files-based core.

Ultimately, I was tired of messing with PHP. I’d like to go with more Python. Django appears to the fastest way to get to having a working blog, but lack of personal free time, and lack of desire to fight with FastCGI and Dreamhost to get up and running, kind of dampened my spirits. So I’ll settle for my clever Python script that I wrote to convert from Pumpkinvine to WordPress and some time that I’ll need to devote to learning how to create WordPress themes.

(now updated with some pointless Python code)

This is perfect for this little blog concerned with trivial and small things. I noticed that my Peanuts calendar on my desk is essentially a reproduction of the 1990 calendar. Obviously, they all have to be reproductions because Charles Schulz died in 2000. So looking more closely, I realized they’d chosen 1990 primarily because the dates and days of 1990 match 2007. So a quick hack with the ‘cal’ program under linux and I was off and searching for what will most likely be next year’s calendar. Since 2008 is a leap year and obviously 1991 is not, I knew that wasn’t going to work.

First I generated target calendar with:

cal 2008 > cal2008.txt

Then, I just manually (although I could have written a script) looped through a few known leap years and found that 1980 seems to match perfectly as well.

cal 1980|diff - cal2008.txt

Obviously, what we’re looking for is a diff of only the first line, the year. Side note: the selection of 1952 (also a match), would seem to be fairly unlikely.

For 2009, there are a bunch of choices including: 1953, 1959, 1970, 1981, 1987 and 1998.

Why am I posted this? I don’t know other than it was fun to use ‘cal’ and ‘diff’ to anticipate which Peanuts calendar would be next.

UPDATE

I couldn’t resist doing a Python solution to this little problem. I know this isn’t as efficient as someone else could make it, but it was entertaining for me to do it. It took about 5 or 6 minutes to write and ran correctly the first time (matching my previous results):

import sys 
import datetime
def matchCals(inputYear,minYear,maxYear):
    sDayOfWeek = datetime.date(inputYear,1,1).weekday()
    eDayOfWeek = datetime.date(inputYear,12,31).weekday()
    for testYear in range(minYear,maxYear+1):
        if (sDayOfWeek == datetime.date(testYear,1,1).weekday() and eDayOfWeek == datetime.date(testYear,12,31).weekday()):
            print "Match Found... %s = %s" % (str(testYear) , str(inputYear))
 
if __name__ == "__main__":
    matchCals(int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3]))

Here I just check the day of the week on the first day of the year, and the last day of the year. If both of those fall on the same day, I assume the calendars for those years are the same. And since we’re dealing with a limited set of dates (valid complete Peanuts calendars running roughly 1951 to 1999, it handles a min and max year for input as well. So to run it for 2008, you just type in:

python ./caltest.py 2008 1951 1999

Of course, there is no error checking… so if you enter a string or an invalid year… that’s your own problem. I just wanted to see how quick I could do it. Obviously, I’ve waisted more of my time… and more of your time.

Brandon and I were exchanging emails the other day about the state of Python IDE’s and specifically about IDE support for GUI’s. My thought was that while nearly every distribution of Python has Tkinter with it, it’s just so impotent that despite being included it’s not really the standard GUI library for Python. Whether it’s wxPython, or PyGTK or something else, there’s plenty of debate on GUI frameworks and Python. That has certainly hurt the Python community a little if what you’re looking for a great editor IDE combined with a great GUI layout tool.

So what does this have to do with web frameworks? I’d say that unlike the GUI thing, we’re probably better off in a sense because Python retains it’s own identity. Unlike Ruby, where one can barely think or mention the name Ruby without also saying “Rails”. It’s like saying Marco without saying Polo. Ruby has (in my opinion) lost a fair amount of identity because of Rails. Certainly, there are plenty of projects in the Ruby community that by their nature will start to reverse that (one that immediately comes to mind is IronRuby). But just imagine for a moment if Python was just known for one web framework only. What if it was “Python turning TurboGears” or “Python on Pylons” or “Twisted Python (aka Python in a Knot)” or even “Python was his Django”. Python would lose some of its identity. So, for those who like to revel in the diversity of web solutions for Python, enjoy! And for those who wish the community would just pick one framework, they can just go to Zope.

If you’re using Linux at all and need ODBC access to data sources, you’ve basically got two choices for Python library access, mxODBC and pyodbc. And really, until recently, pyodbc wasn’t really strongly considered a Linux option. In fact, even today, it’s not very Linux friendly to get installed (or at least I didn’t think it was).

I was needing to connect to our IBM iSeries database (AS/400 for most of us who aren’t newbies) from a Linux box. I managed to download and eventually install the IBM Access Client under Ubuntu and I had configured unixODBC stuff to the point that I felt I had a valid DSN setup. It was time to try to connect from Python. Since I work for a non-profit organization, I knew that eGenix had always treated non-profits differently and allowed the mxODBC library to be used for free. But after downloading and installing the latest version of the mxODBC stuff, it was apparent that I needed a license key. Thinking that non-profits were still getting a break, I wrote to their sales department inquiring about licenses for a non-profit organization. This is what I got back:

thank you for your enquiry about mxODBC. Unfortunately mxODBC is no longer free for non-profit organisations. The new release of mxODBC continues several years’ improvements. The change in licensing was necessary because less than 0.1% of all downloads of mxODBC led to purchases and we are sure that mxOBDC is currently being used in commercial environments without a licence. Furthermore, we know that mxODBC is often in use in environments with commercial products such as Oracle. We have, therefore, unfortunately had to introduce a compulsory licensing scheme and have also reviewed our licensing policy. Because our product is extremely competitively priced we cannot afford to give away a great many licences particularly outside of Germany as we cannot offset the donation against tax. We also have to consider the administrative overhead involved in verifying non-profit status. It is for these reasons that we have decided to stop providing mxODBC for free.

Fortunately, I had already decided to hack away on getting pyodbc to run while I was waiting for eGenix to respond. Point is, if you’re willing to keep installing headers and such, you should eventually be able to get pyodbc to compile and work under Linux. The big one for me was that I found that version 2.0.35 was the only version I got get to work with my Python 2.5 installation (not sure about other versions). Doing the sudo python setup.PY install failed everytime with 2.0.36, but 2.0.35 was fine.

As for eGenix, I am sure the mxODBC is a fine product. Their mx libraries have long been a part of the overall Python landscape and I can’t help but think that if my requirements were for a trusted ODBC connection that was continuously used in a production environment, mxODBC might be the best option. But I also can’t help but think that pyodbc and the overall FOSS-spirit of the Python community might eventually do them (eGenix) in. It’s obvious that if the pyodbc project continues to mature under its current model that it will become the way to do ODBC with Python. It’s not that I have anything against folks making money. I am, after all, a capitalist. But part of the fun of a free market is watching how the software market continues to change with competition from free software.

As a side note, I think it is interesting that the IBM client access software, while not open source, is now free.

Technorati Tags: python, odbc, pyodbc, mxODBC

Thanks to a ‘Security Breach’ at DreamHost, my 2+ sites were hijacked. Obviously have things running again, but this issue points out several things. FTP is not very secure (as if we didn’t know that). Running PHP, while convenient because everyone runs it, is also a dangerous because the bad guys just wrote a script that edited any ‘index.php’ files out there on the accounts they managed to breach. If I’d been running some funky cool django stuff, there’d have been no index.php to hijack.

I found myself needing to do a fairly complicated thing. Well, it’s not complicated, but for a DOS batch file, it seemed complicated. The problem I was facing: to insert some “official” DOS command stuff into a .bat file that is run on all computers in our entire organization. I have no idea how many machines were talking about here but I would guess the number to be in the 5,000+ range or higher. Specifically, this is our official domain login script. Yikes. I don’t like having anything to do with something that gets touched this often. One screwup in there and it’s not a good thing. The problem is that we have a special group of machines (about 130) that are not registered on the domain. Now we need them on the domain but we don’t want to run some of the programs that our “Software Taliban” uses for monitoring machines. All of these special machines have a common naming scheme which is the letter S followed by a 7-digit number. So my task was insert some DOS .bat stuff in the existing .bat file that will run without external dependencies (other than whatever is already guaranteed to be there). All I knew for sure was that the environment variable COMPUTERNAME would be filled with the name of the computer and that my special machines were running 2K or XP and that either way, the OS environment variable would be filled with “Windows_NT”.

I also wanted to make sure I didn’t catch machines that were named “Snoopy” or anything like that. So for lack of a better way, I chose the following as my DOS string substitution method:

SET CN=%COMPUTERNAME% 
SET CN=%CN:0=x% 
SET CN=%CN:1=x% 
SET CN=%CN:2=x% 
SET CN=%CN:3=x% 
SET CN=%CN:4=x% 
SET CN=%CN:5=x% 
SET CN=%CN:6=x% 
SET CN=%CN:7=x% 
SET CN=%CN:8=x% 
SET CN=%CN:9=x% 
IF /i "%CN%"=="Sxxxxxx" GOTO END

The magic here is the funky %variable:y=x% stuff. Essentially, it’s string replacement where any ‘y’ is replaced with an ‘x’. This then normalizes the name into something I can compare against. Doing it this way also guarantees that I am comparing the length as well. So while it isn’t pretty, I was pleasantly surprised to find it was possible to do in DOS. There is also a method for getting a substring using a similar syntax: %variable:~s,l% where s is the starting position (1-based) and l is the length of the sub-string. Talk about obscure. But then again, how often do you have to stick to plain old DOS .bat files?

Technorati Tags: DOS, .bat files

Since there seems to be virtually nothing Twisted or Nevow related going on at PyCon, I thought (last minute I know) that I would at least try to organize a BoF meeting (most likely pretty informal) for folks into Twisted and Nevow. So if you are interested, please add your name to the wiki page for the Twisted BoF. If there’s at least 3 or 4 folks, I’ll try to setup a time, even if it’s a dinner or lunch or something. Or if someone else wants to suggest a time that would be great as well.

Technorati Tags: pycon2007, python, twisted, nevow