This post originated from an RSS feed registered with Python Buzz
by Ben Last.
Original Post: The Shells, The Shells, Esmerelda!
Feed Title: The Law Of Unintended Consequences
Feed URL: http://benlast.livejournal.com/data/rss
Feed Description: The Law Of Unintended Consequences
You know, given his thpeeth impedimenth, you'd have thought Quasimodo would have found a girlfriend with an easier name, maybe a Betty or an Anna (probably not a Thuthan). Anyway, that has nothing to do with the subject of today's content-free babblings, which is the crossover point between shells and environments.
Jeremy Zawodny writes on using the MySQL client like a shell, according to this Forever Geek post. Now, having rattled on about using interactive Python as an environment for working on MySQL a while back, I thought this could be interesting, until I found that it was just a short blog on aliasing the "cd" and "ls" commands to "use" and "show tables". Sort of useful, I guess.
But it got me thinking about shells. Rather than attempt to gradually retrofit mysql with the basic commands that a shell has, why not take a programmable environment like, oh, I dunno, let me take something at random, say, Python and extend that to hook better into MySQL, or whatever other environment you happen to be in.
There are a few Python shells around, such as IPython and PyCrust or the extremely bashalike PySH (which I like to pronounce to rhyme with fish, just because I can). I took a look at each one, and found that the whole subject is packed with Interesting Questions.
First off, what's a Python Shell? To some people, it's an interactive Python environment. To others, it's something more like bash or cmd, an environment for interacting with an operating system, but which has a degree of programmability (in Python, naturally). My definition would be the latter - I like command-line shells. To me, bashis Unix, in the sense that the underlying OS is rather less important than the pipe/small commands/vi nature of the environment, which is why I like cygwin.
Second in the list of Interesting Things To Ponder, it seems to me, is that of the context within which commands are executed. For instance, all OS shells that I have ever used have a notion of a current working directory, whether this is set explicitly by cd or as some sort of default value for command expansion, as in VMS's set def. The mysql client has a notion of the current database which is arguably similar; also it has a connection to a given database server, specified at startup. These are its context, thus to integrate MySQL into a Python shell, we need to extend the "current context" to include both the server and the current database.
But context-dependence is a double-edged sword. As well as the working context within which Stuff Happens, there's the deep and complex question of how a given line of input is interpreted, when there are multiple contexts in which the various elements of it can be assigned semantics.
But that's putting it too formally. What I did next was to write some commands that might work in my ideal shell. First off, I want the ability to do something like:
for x in `ls *.py`: cp x x.replace('.py',.py.sav')
For now, ignore the mix of bash and Python; I want to be able to use ls to get me a list of names and use that as the sequence over which to iterate a for loop. Then I want to be able to apply Python string operations to the variables to do clever stuff with names, and I want the operating system commands, like cp to be invoked for me.
Here's another:
for a in "Table1 Table2".split():
rows = select * from a
print "%s has %d rows" % (a,len(rows))
After all, the original article that sparked this off was about extending MySQL, so why shouldn't I be able to mix in SQL commands as well as OS stuff?
One way to do this is to try adding new keywords to Python, but that way madness lies; also that way involves C programming, and I do this sort of stuff for fun, so why drop into another language? No, there had to be a way to do this in pure Python.
Have you ever had a problem work like a catalyst in your head? You know the way that grit acts on an oyster, provoking the formation of a pearl, or a seed crystal dropped into a solution can kick off the rapid growth of a bigger structure? This was like that - over two nights I lost sleep whilst my brain tickled away at the problem. I dreamed in code, something that hasn't happened in a long time. Finally, after two days of nibbling away at the issue without going near a keyboard, I thought I had a breakthrough. Take a look at the second example above. How does the interpreter know that the "select" is a SQL expression? That the "*" is to be passed to the SQL interpreter, where a Unix-esque shell would glob it in place. I decided that the solution had two parts. The first is disambiguation: the context in which words are to be interpreted has to be more explicit. The second is that the shell should be a preprocessor - it would rewrite the input lines so that the Python interpreter always executes pure Python.
The next question - what was an appropriate syntax to use? I chose to use the backtick, "`", for these reason:
It's used in Unix shells for a roughly similar purpose - to bracket a command that's to be executed in a subshell and its output captured, rather like that first example command above.
It's used in Python only as syntactic sugar - one can replace `a` with str(a), so a Python in which the meaning of `` has been overridden loses nothing.
Thus the shell can take any sequence of characters within backticks and rewrite it to be legal Python that will have the desired effect. Let's go back to that example:
for x in `ls *.py`
It's more-or-less equivalent to:
for x in os.popen("ls *.py")[1].readlines():
Python that has the desired effect.
However, I also want to be able to mix in SQL commands, so the shell has to be able to support multiple different contexts in which stuff can be understood. I also want to be able to add new contexts as I feel like it; choosing a new alternative to backticks for each one would get difficult very fast, so I chose to stay with backticks and disambiguate some other way.
Finally, I chose a name for the shell: Quasi. This is because:
It let me make terrible jokes; when asked how I thought of it, I could reply 'I had a hunch...'
I liked this title of this entry too much not to make the link
Nobody will ever be sure how to pronounce it - does the A rhyme with "day" and the I with "eye", or does the whole rhyme with "Ozzy"?
And after a weekend during which I wrote altogether too much code to be healthy, version 0.1 exists. You can download the source, if you really want to, here (16k ZIP file). Be aware that the file will change as I do more stuff to it.
So what can it do? Here are some examples of commands that really work:
>>>#Get a list of all Python files in the current directory and copy them to backup versions.
>>>for x in `os ls *.py`: `cp $x $(x.replace('.py','.py.sav'))`
>>>#connect to a MySQL server. The keyword "sql" at the start of the line
>>>#is syntactic sugar that lets us leave off the `` ticks.
>>>sql connect host=myHost, user=MyUser, passwd=MyPassword, db=myDb
>>>#do a query and capture the results
>>>table = 'Sample'
>>>rows = `sql select * from $table`
>>>rows
(('Some', 'All'), ('Here', 'There'), ('A', 'B'))
There are three contexts:
The SQL context, marked by the keyword "sql" at the start of the line or after the opening backtick (or the character "&", so `& select * from Sample` is legal).
The OS context, marked by the keyword "os". OS commands return their output (captured from popen) as a list of strings.
The shell context, marked by the keyword "shell" or "!". Shell commands return their status, but not their output, so you can do `shell vi $x` to edit the file whose name is in x and it'll work.
There's an extra trick for the OS context - if you put a variable at the start, before a "|" (pipe), then that variable will be printed to the stdin of the process. It handles lists appropriately. Variable substitution is done with the syntax $ (where the end of the name is recognized sensibly) or $(), where everything between the parentheses is evaluated by Python. So the following are all legal:
Those of a curious disposition may like to type "trace 3" as their first command - you can see the Python that your command expands to.
Next entry - how the damn thing works, and musings on the interesting nature of shells that have built-in interpreters... code that executes in two different namespaces.
Previous Topic
