Running Pylons on top of PyPy
The next episode of the "Running Real Applications on Top of PyPy" series:
Yesterday, we spend some time with Philip Jenvey on tweaking Pylons and PyPy to cooperate with each other. While doing this we found some pretty obscure details, but in general things went well.
After resolving some issues, we can now run all (72) Pylons tests on
top of pypy-c compiled with the following command:
translate.py --gc=hybrid --thread targetpypystandalone --faassen --allworkingmodules --oldstyle
and run some example application. Here is the obligatory screenshot (of course it might be fake, as usual with screenshots). Note: I broke application on purpose to showcase cool debugger, default screen is just boring:
Please note that we run example application without DB access, since
we need some more work to get SQLAlchemy run on top of pypy-c together with
pysqlite-ctypes. Just one example of an obscure details that sqlalchemy is
relying on in the test suite:
class A(object):
locals()[42] = 98
Update:This is only about new-style classes.
This works on CPython and doesn't on PyPy.
Cheers,
fijal
Comments
Very good to see this work! This is a a good thing to be trying and hearing it makes me happy.
We're busy working on making Zope 3 run on Jython, which should get make some of our C level dependencies optional. These make a port to PyPy harder as well. Zope 3 libraries have umpteen thousands of tests that can be run, so that should give one some coverage. The libraries come packaged separately too.
The trickiest part would be those bits that depend on the ZODB. Porting the ZODB to PyPy should allow new possibilities, but it'll be hard too, I imagine.
Hi Martijn,
in fact having zope3 work with pypy would be very nice. i discussed a bit with Phillip and he suggested to first get zope.interface and zope.component to work, then zope.proxy/zope.security. IIRC my first try with zope.interface yielded 3 failures out of 231 tests. I had to hack the test runner a bit to not rely on GC details - i guess that your work for Jython might imply that as well. What is the best way to follow your Jython work, btw?
best & cheers,
holger
The Jython project is a summer of code project. Georgy Berdyshev is the student and is sending messages to jython-dev.
Here was a recent status report:
https://sourceforge.net/mailarchive/forum.php?thread_name=ee8eb53d0806082009g5aec43dbn3da1f35b751cba70%40mail.gmail.com&forum_name=jython-dev
I see that the hyperlink to Georgy's report just now got eaten by the comment software. Here it is again, hopefully working this time.
Georgy Berdyshev is lurking in the #pypy channelg (gberdyshev or similar), FWIW.
Let's see who entered that line:
4361 pje # This proves SA can handle a class with non-string dict keys
4361 pje locals()[42] = 99 # Don't remove this line!
pje ? Yes. That PJE. Complete with "don't remove this!"....we'll have to see what mr. guru was up to with that one. This test is also only present in the 0.5 branch which hasn't had alpha releases yet.
Would love to hear some other examples of "obscure details" the test suite is relying upon...my guess would be extremely few or none besides this one example.
It tests that SQLAlchemy isn't depending on class dictionaries containing only string keys.
Unfortunately, this makes the test then depend on the ability to have non-string keys in the class dictionary. ;-)
The test is to ensure that SQLAlchemy will be able to map objects whose *classes* have AddOns defined.
By the way, as of PEP 3115, the locals() of a class can be an arbitrary object, so making compile-time assumptions about what *can't* be done with a class' locals() is probably not a good idea.
Also, as of every existing version of Python>=2.2, a metaclass may add non-dictionary keys to the class dictionary during class.__new__. So, it has never been a valid assumption that class __dict__ keys *must* be strings. If PyPy is relying on that, it is already broken, IMO.
PJE: What you say is mostly beside the point. PyPy has no problem at all with non-string keys in (old-style) class dicts. The point is more that locals() cannot be used to assign things to this dictionary, see the docs:
"locals()
Update and return a dictionary representing the current local symbol table. Warning: The contents of this dictionary should not be modified; changes may not affect the values of local variables used by the interpreter."
Well, if you plan on supporting, say, Zope or Twisted, you'll need to support modifying class-body frame locals.
There really isn't any point to optimizing them, not only due to PEP 3115, but also due to pre-3115 metaclasses. (And just the fact that most programs don't execute a lot of class suites in tight loops...)
Zope does things like:
frame = sys.getframe(1)
frame.f_locals['foo'] = bar
It does this to make zope.interface.implements() work, among other things. This allows you to the following:
# IFoo is actually an instance, not a
# class
class IFoo(zope.interface.Interface):
pass
class Myclass:
# stuffs information in the class
zope.interface.implements(IFoo)
The martian library (which Grok uses) actually generates this into its directive construct.
Some of this stuff could become class decorators in the future, I imagine, but we're stuck supporting this future for the forseeable future as well.
I didn't "generates this", but "generalizes this". I think PJE's PEAK library also has stuff for this ("class advisors").