Sprint topics @ PyCon

The sprint represents a rare opportunity for a bunch of us to be in a
room together and really hack away at stuff. I think that going after
little bugs in Trac would miss the point. Though we certainly have
little bugs that need fixing, those are easy enough for people to
knock off on their own. I'd like to focus the sprint on the items that
are harder and can benefit from more discussion and focused attention.
Here's my list:

Docudo
template search paths (to make it easy to customize an app you install)
Kid error display
Kid performance
Deployment
Running multiple TurboGears applications in-process
Running multiple WSGI apps in-process (including looking at the Paste options)
FastData
Widget browser

Note that I don't expect us to tackle all of these, or necessarily
finish many of them. The main point is to build a good foundation so
that at least some people are clear on the direction and the approach
and we've had a chance to experiment a bit.

What appears on your wish list? Which ones above are important to you?

(By the way, I put Docudo up there because I think it's important for
the ongoing documentation effort, and I think that once the project is
kickstarted people are more likely to contribute.)

Kevin

--
Kevin Dangoor
Author of the Zesty News RSS newsreader

email: k...@blazingthings.com
company: http://www.BlazingThings.com
blog: http://www.BlueSkyOnMars.com

Pray forgive the length here - I've been brewing on some of these for a
while. Other note: maybe best to reply using "Pycon: <subject>" rather
than replying to this whole thread?

In my opinion, the primary competition for TurboGears is not php - it's
the web frameworks like phpnuke, drupal and Joomla!/Mambo. The nice
thing about these is that you're running right out of the box (no
programming) and it's very easy to write applications that fit into
these frameworks.

Where I'm mostly going toward is having the ability for there to be a
few different "turbogears gui frameworks" out there (maybe oriented
towards different audiences), but having applications run on any of
these. So, Kevin already mentioned deployment using eggs. To go to
the "next step" we need a few more things which are more ...
introspective in nature. For instance:

1) Configuration.

Right now, we have a python config file. The only issue with it is
that there's no way to create a form from the config file that a user
can edit at instantiation time. So, there would need to be a way to
add a FormEncode/DataGrid item in there for configuration. There also
needs a way for the application to store state on a per instance basis
(I've got these 5 blogs, each configured differently).

2) Permissions.

Yes, we have Identity. The problem is that everyone will have a
different concept of what "admin" or "user" is. I really liked how
Zope2 did their permissions: you put the name of the action as the
permission. So "Edit Blog" would be the permission, not "Needs Admin".
The thing Zope2 did badly was was giving the users one single
obnoxious page to do the mapping for all applications at one time.
That can be fixed at the framework level. Some frameworks could take
the "user/group/role/permission" approach; others might take a more
simple "user/permissions" - all depending on the target audience.

3) Services

Some applications will end up becoming services that other applications
could use. TG should be in the middle here, defining the interface,
but allowing multiple applications to serve the need. For instance,
picture a "News" application, which shows a summary on the main page,
and allows a review/workflow for publication. You can easily see how a
blog system might want to "publish" to the News service. Other ideas
for services would be a search service (here's the url and the text:
index it). Another might be a "user comment" service.

Oh, I'll be at pycon as well for the whole time. Just haven't put my
name up.

Cheers,
Gary Godfrey
Austin, TX

I think the core of your email here is misguided. Turbogears is not
trying to be a CMS (as phpnuke, drupal, Mambo, etc are), but a web
framework. Turbogears itself is not designed to be "working out of
the box". Turbogears *applications* however are a different matter.

Lee

We'll see how big the thread gets. The length of the message is quite
fine! Nice to have some detail of what you're thinking.

I disagree, for the moment. Those packages started life as CMSes in
various forms, and have opened up more over time to application
development. TurboGears is heading the other direction. The real focus
is "application development". I don't think there will ever be a
TurboGears version that comes with something like a Drupal "node" out
of the box. I *do*, however, want it to be easy to install pieces that
give you a blog, forums, wikis, etc...

OK, so you may have the same notion. The TurboGears core handles the
core bits for these "gui" frameworks.

The python-based config files are extremely flexible (since they're
Python!). Ultimately, it can contain whatever kind of configuration
information is needed by the various parts of the system.

I think that having forms that the user can edit might fit more in one
of these "gui framework" layers. I would like to see the Widget
Browser in the Toolbox expand to give you more things to do with
widgets and forms.

Yeah, this would be one of those things that falls into the "multiple
TurboGears apps in one process" item. As of today, CherryPy supports
doing separate config for each app, which is a bonus.

Generally speaking, anything you can do to avoid storing state on the
server is better for your application. That said, if there was a
"namespaced" version of sessionfilter that kept session data separate
for each app, that would do the trick.

I believe that identity lets you specify by either role *or*
permission. So, you could define that all "admins" can do something,
or you can define that everyone with the "access-this-thingy"
permission can do it.

I know that this is a big Zope 3 feature. I'd like to see this kind of
thing grow out of real usage rather than "wouldn't it be cool if?"
sort of discussion.

That may be a sign of the different focuses: though Zope was original
built as an application framework, it really gained prominence as a
CMS. Your example even sounds like something from a CMS. I'm not
saying that's a bad place for TurboGears to go. I'm just pointing out
that TurboGears' place in the world is much more focused on
database-driven applications than content management.

I do know that there are CMS projects in the works, though, so these
kinds of things will come up soon enough.

Excellent! See you there!

And thanks for the feedback!

Kevin

I vote for some Kid love (it deserves it :-)).

Ciao
Michele

As someone who's written an entire application for a client in TG
already with a 0.8 feature set, I'm much more interested in stability
of the core features and the magic of the 1.0 release number than
multiapp support or helpers like a Widget browser. (Hell, I don't have
Widgets to browse!)

So I second dealing with Kid issues and other long-standing requests
over new features.

--Liza

How long is the Sprint going to be?

Thanks
Alvin

I can certainly appreciate that. The widget browser would definitely
be far down on the prioirty queue. Multiapp support is pretty core,
though, to being able to reuse applications that you or others
create...

Once TG 0.9 releases start happening, we're on the path to 1.0. It's
possible that some of the groundwork laid at the sprint will only land
in TurboGears after 1.0.

David Stanek's been doing a great job with Kid, and it'll be nice to
bring some others up to speed with it to help out and fix the
outstanding requests there.

Kevin

> Scalability

Sneaky. I almost didn't notice this one, since the rest came from my list.

What does "scalability" mean to you? better caching? some kind of
clustering support? faster raw performance? easier to lay out a 1,000
module project?

I had thought of adding "standard caching" to my list after I sent it.
I think that's an important one.

4 days, with people there for varying amounts of those 4 days.

Kevin

But Liza's point about strengthening the core first is a good one.

--
Mike Orr <slugg...@gmail.com>
(m...@oz.net address is semi-reliable)

This is the kind of big-ticket item that I'd like to talk about at the
Sprint... to at least knock around some ideas, if nothing else. It's
also worth noting the presence of CherryPaste and additional new work
that has gone on with CP's WSGI support. There are now a number of
potential ways to get access to WSGI middleware and other
applications, and those are worth considering.

At this point, TurboGears has diverged a bit from CherryPy's
configuration system. TurboGears' config is now contained in mildly
special Python modules. I'd like to stick with this config system
going forward, because it provides some capabilities that people have
wanted beyond purely static config files.

Kevin

Kid Error Display
Nested Forms
Caching Framework
db -> model code generation
model -> form code generation

For the kid error display, simply having the option of displaying the
locals for every level of the stack would make debugging kid errors so
much simpler.

I have an ongoing quest for nested forms/container widgets. The forms
rewrite gives me a better base to build on, but I need to get around
to doing it.

The caching is something I need to look into for my own work. I
currently either use a dict or a pickled dict to memoize a function, I
expire based on the mtime of the pickle by deleting the dict and
recreating it. It's cheap but I'd like to have something a bit more
elegant.

For the generation stuff, I tend to customize my models/forms fairly
heavily, so I can't just use the auto functionality provided by
sqlobject and fastdata but I'd also rather not have to write the
boilerplate and just skip to the customization. Not a high priority,
but it'd be nice.

**** API Documentation is my number one request.

DHH wrote a great piece on the topic:
http://www.loudthinking.com/arc/000479.html

I wouldn't underestimate caching. I'd also imagine that if you got
some profiling running on your app, you'd be able to squeeze more out
of a single box.

That you shall have.

Kevin

I should be around for the sprints, but I haven't decided what to do
yet. I'd be interested in working on combining TG and WSGI a bit more
intimately, especially if there are other people interested in working
on that. I think a good goal for a sprint is not necessarily just the
code written at the sprint, but helping to introduce people to new
things and make people comfortable with code bases that might otherwise
seem intimidating.

I'd also be particularly interested in a WSGI-level caching middleware.
This would be based primarily on HTTP style cache controls, almost
like a caching proxy, except in WSGI and in-process, so there's a lot of
extra controls and interfaces you can put on it. I know there's several
other people outside of TG that would also be interested in this on the
WSGI level. The HTTP level is also where you can insulate Kid's
performance problems -- any caching before the template won't have
anywhere near as much effect as caching the rendered page.

It might be nice to look into merging some of the changes I made in the
paste-error-reporting branch into Kid. But to use that in TurboGears
requires using Paste's exception formatter. But if you want to move to
paste.evalexception (which you should want to do), then you'd be doing
that anyway. That also gives you access to view local variables.

I also have a stealth SQLObject project that would be fun for a sprint
(green field), but I haven't really committed to that project myself
yet, and it wouldn't really help TG get to a stable base any faster
(unless the project was wildly successful). There's also lots of
SQLObject stuff that is more maintenance-related, which would certainly
be useful to do -- things like straightening out the cache situation,
transactions, eliminating the naming conventions, polishing
sqlobject-admin's rough edges.

--
Ian Bicking / ia...@colorstudy.com / http://blog.ianbicking.org

Local variables, yay!

What's up with TG's uninformative tracebacks? Is that something
deep-seated or is it something we can fix at the sprint? It's the
only thing that has made me wonder whether to switch back to Quixote.
Three different errors during web requests; three uninformative
tracebacks. I posted one of them to the list a couple days ago
(subject: Database / config problem). Here's the middle of it:

File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r663-py2.4.egg/turbogears/controllers.py",
line 184, in expose
func, tg_format, html, fragment, *args, **kw)
File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r663-py2.4.egg/turbogears/database.py",
line 193, in run_with_transaction
retval = func(*args, **kw)
File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r663-py2.4.egg/turbogears/controllers.py",
line 196, in _execute_func
output = errorhandling.try_call(func, self, *args, **kw)
File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r663-py2.4.egg/turbogears/errorhandling.py",
line 64, in try_call
output = dispatch_error(func, self, error, *args, **kw)
File "<string>", line 5, in dispatch_error
File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r663-py2.4.egg/turbogears/errorhandling.py",
line 27, in _default_error_handler
return getattr(self, error_source.__name__ )(*args, **kw)

The "expose" and "run_with_transaction" lines look like mundane
request dispatching. I assume 'func' is the exposed method or
thereabouts. Then suddenly it switches to 'dispatch_error' and goes
through a bunch of boring error-handling functions I don't care about,
and the part I *do* care about that caused the error is shown nowhere.
Maybe it's buried in one of those "<string>" lines if it uses eval().
When I did the same operations under "tg-admin shell", I got a much
more informative traceback that showed exactly what the error was and
where.

Mike, I think this is related to the new error handler mechanism, I
think (and hope) it will be fixed by the introduction of a
exception_handler decorator that shouldn't get into your way unless
you're using it to mange exceptions:

http://tinyurl.com/7r8pk

OTOH I also hope that during PyCon TG will adopt (for CP and Kid) Ian's
eval exception, I tried it with RhubarbTart and it's really useful, and
a must have feature for 0.9/1.0 IMHO.

Ciao
Michele

Simon

Right, I should've updated or mentioned the version. I was using 663.

Thanks
Alvin

Wonderful! Meaningful tracebacks!!!

File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r743-py2.4.egg/turbogears/database.py",

File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r743-py2.4.egg/turbogears/controllers.py",
line 217, in _execute_func

File "/usr/local/lib/python2.4/TurboGears-0.9a0dev_r743-py2.4.egg/turbogears/errorhandling.py",
line 61, in try_call
output = func(self, *args, **kw)
File "/home/mso/svn/hazweb/Cameo-Search-Demo/cameosd/controllers.py",
line 45, in browse
select = Select([Chemical.q.id, Name.q.name], where=where,
NameError: global name 'Select' is not defined

Hooray, hooray, hooray. Now, about those local variables....

Thanks Simon, great work. :-)

Ciao
Michele

Once this ticket:

http://www.cherrypy.org/ticket/461

finds it's way into CherryPy it should be possible to use the
paste.evalexception middleware right?

Ciao
Michele

--Tracy

Right now it should be fairly easy already, either using CherryPaste, or
just putting it in place manually so long as you are only running
locally (where the app is mounted at the root, and the
SCRIPT_NAME/PATH_INFO issues won't bite you).

--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org

All that said, I don't know how much help I can be at the sprint. But
if all I can do to help is to make a coffee run, count me in! I'm very
much looking forward to meeting every one, and contributing in whatever
way I can.

Gary Godfrey's comment struck an interesting chord with me when he
mentioned PHP. The reason I even looked at Python in the first place
was that our company did an "Enterprise" level project in PHP, and we
felt PHP wasn't up to the task. So, I was looking for something that
would provide a similarly friendly development experience to PHP, but
be much more robust. When I then stumbled upon the TG site, I knew I
had found a winner. Further, when I saw the Kid template syntax was
"<?python ... ?>", I knew I could sell it to the other developers at
the company. They wouldn't have to step too far outside of their
comfort zone.

My point is that I think TG specifically can act as the sales pitch to
PHP development community to entice them to try Python. What's missing
is the friendly web site. If you have used the PHP site, you know how
easy it is to search for and find the documentation you need, official
on top with user comments below. TG is such a powerful combaination of
tools that it's choice is almost a no-brainer. But, I think people will
be annoyed by the fact thay then need to browse the TG site for some
documentation, then go to the Kid site for some more, and SQLObject for
even more. I'd love to see a central repository for all of that
knowledge, complete with user comments about how best to apply that
knowledge in TG-specific projects. I know that would be a daunting task
because each of these things is a stand alone project with it's own
documentation.

Anyway, just my $0.02.

-Arthur McLean

I can't speak for Kevin but I'd say just being there is a
contribution, especially the first day when people are deciding tasks,
and there may be a discussion of ideas regarding which way to go (e.g,
Paste integration, Docudo's design, etc). Who knows where you might
find your niche or might learn something? If there is a documentation
task force, we'll certainly be looking for newbie input regarding what
you wished it contained, what you don't understand, what would help
sell TG and Python to your company, etc. All that helps tailor the
docs to a real-world audience.

There's also the sprint's impact on you. Every PyCon I've gone to has
given me a direction for the following year, something that became the
theme of the year. Once it was Webware. Last year it was WSGI and
framework integration, which indirectly led me to TurboGears later.
Once I did the Docutils sprint as a complete newbie to it. I was
interested in text processing but had seldom used Docutils (ReST).
But I learned a lot and was even able to contribute a utility function
to the source.

Your greatest contribution may not even be at the sprint, but
something the following year based on what you learned at the sprint.

The PHP online documentation format kicks ass, and I don't even like PHP.

Cheers.
Ronald

________________________________
Ronald Jaramillo
mail: ronald AT checkandshare DOT com
blog: http://www.checkandshare.com/blog