The configuration syntax in the new version of zc.buildout is extended with two operators += and -= to allow adjusting options in an extended profile. This is very helpful. You don't need anymore to overwrite an option in the base configuration when all you want is to remove/add an entry.

here is an example:

# base.cfg file
[buildout]

parts =
part1
part2

eggs =
eggs1
eggs2
eggs3

To add new part and remove egg2 in an extended configuration, we do:

# newconfig.cfg file

[buildout]
extends = base.cfg

parts +=
part3

eggs -=
eggs2

Running buildout, the result will be as if you have run it with this configuration file:

[buildout]
parts =
part1
part2
part3

eggs =
egg1
eggs3


That's it.
If you want to add new egg to all your profiles all you have to do is to add it to the base.cfg.
This may save someone's time.

If you get an error like this when running ./bin/buildout:

bash: bin/buildout: /home/mustapha/workspace/collective.buildbot/test/parts/buildslave/customer.pr: bad interpreter: Permission denied

Pay attention: the full path to the used python is truncated here.

that means that the shebang line (the first line in the bin/buildout script starting with #!) is too long. The maximum length of a shebang line is system dependent. On my Ubuntu the maximum length is 80. This table lists the maximum length of the shebang line per OS
Now one can add arguments to the command controlled by supervisor. You have to give all arguments between [] like this: [arg1 arg2 arg3]. This way one can use ctl scripts generated by other recipes and give them the appropriate argument to make them run in foreground. An example:

[buildout]
parts =
instance1
supervisor

[instance1]

...

[supervisor]
recipe = collective.recipe.supervisor
port = 9001
user = mustapha
password = secret
serverurl = http://supervisor.mustap.com

programs =
10 instance1 ${buildout:bin-directory}/instance1 [fg] ${instance1:location}
10 instance1 ${instance1:location}/bin/runzope ${instance1:location}
20 myprog /mypath/bin/myprog [-n 100] /mypath



For more info: http://pypi.python.org/pypi/collective.recipe.supervisor

UPDATE: After a discussion with Chris Mc Donough, I updated this post. For zope instances and zeo instances, please use the runzope and runzeo scripts instead of instance and zeo
The collective.recipe.supervisor is buildout recipe that helps with the install and configuration of supervisor.

How to use it: http://pypi.python.org/pypi/collective.recipe.supervisor

Let's create a project called plone.app.myproject:
$ mkdir plone.app.myproject 
$ cd !$
$ git svn init -s http://server.com/!$
The last command assumes that plone.app.myproject/trunk, plone.app.myproject/branches and plone.app.myproject/tags already exist in the svn repository.
$ git svn fetch
$ git add .
$ git commit -m 'initial import'
$ git svn dcommit
That's it. At this stage the master branch is reflecting the remote trunk.
When developing zope/plone eggs with buildout I have to edit the buildout configuration file ( in 3 places ) every time I create/delete/rename a development egg in the src directory or in other development directories (sometime I have more than one).


I have to add/delete/rename the egg in the eggs option of the [buildout] and then add/delete/rename the egg path in the develop option of the [buildout] and in the end add/delete/rename the zcml option of the zope [instance] or in the configure.zcml file of my policy package. This is too much specially when the speed is set to development mode. I need a less boring way to develop.


Solution


buildout.eggtractor is a buildout extension that scans the src directory or a list of directories I give for eggs and picks them up automatically. So no more editing of the buildout's configuration file.


When buildout.eggtractor finds an egg in the scanned directory it:


  • adds the egg to the eggs option of all zope instance parts or to a set of given parts

  • adds the egg's path in the develop option of the [buildout]

  • If tractor-autoload-zcml is not set to false, scans the egg folder for configure.zcml, meta.zcml and overrides.zcml and adds the appropriate zcml entries to the zcml option of the zope instance parts or to a set of given parts.


This steps are done on the fly when running buildout. So I can add/delete/rename an egg and it will be picked up.


NOTE: The extension does not write to the buildout's configuration file.


buildout.eggtractor options


tractor-src-directory: A set of directories to scan for development eggs. Defaults to the src directory of the buildout.

tractor-target-parts: A set of parts to update their eggs option with eggs found in the tractor-src-directory. Defaults to zope instance parts if any.

tractor-autoload-zcml(boolean): Update the zcml option of tractor-target-parts with the eggs found in tractor-src-directory. Defaults to true

tractor-zcml-top: A set of eggs to load their zcml files first. Defaults to an empty set.

How to use it


Using buildout.eggtractor is very simple. As said, it is a buildout extension. All I have to do is to declare it in the extensions option:

[buildout]
parts =

extensions = buildout.eggtractor

That's all. buildout.eggtractor will scan the src directory and do its job every time I run the buildout command.


When I have other directories I want to scan I just add an tractor-src-directory option in the [buildout] and add my directories there:

[buildout]
parts =

extensions = buildout.eggtractor

tractor-src-directory =
dev-src1
dev-src2
src

In a few cases when the priority of loading zcml files matters. I add the egg to be loaded first in the tractor-zcml-top option in the [buildout]:

[buildout]
parts =

extensions = buildout.eggtractor

tractor-src-directory =
dev-src1
dev-src2
src

tractor-zcml-top =
plone.app.mypackage1


If I want to add the eggs found in the development directories to the eggs option of a given set of parts, I add a tractor-target-parts option and add the parts there:

[buildout]
parts = instance1 instance2 instance3

extensions = buildout.eggtractor

tractor-target-parts = instance1 instance3

This way only instance1 and instance3 will be updated.


If I have already other way to include the zcml files (ie: z3c.autoinclude) and don't want eggtractor to generate the zcml slugs, I add an tractor-autoload-zcml option and set it to false


In most cases you will only need to add buildout.eggtractor to the extensions option of the [buildout] without any extra configuration options.

LIMITATION


The extension assumes that the egg name reflects its file system structure example: if the egg name is com.mustap.www the extension assumes that the file system structure is one of the following::

  • com.mustap.www/src/com/mustap/www

  • com.mustap.www/com/mustap/www


This is where the extension looks for configure.zcml, meta.zcml and overrides.zcml files.


If the egg name does not reflect the file system structure, the extension will ignore it.

XXX: I guess walking through the directory is better than this assumption.
In my case this is not a limitation as I choose my egg names that way.

URL

You can get it from http://pypi.python.org/pypi/buildout.eggtractor

THANKS

Martin, Wichert and Hanno for the help and feedbacks
My colleague Jakob aka bartholdy discovered this one today. The instance is running more than one year without stop.
Software used:

Plone 2.5.1,
CMF-1.6.2,
Zope (Zope 2.9.4-final, python 2.4.3, freebsd5),
Five 1.3.7,
Python 2.4.3 (#1, Oct 3 2006, 11:03:02) [GCC 3.4.2 [FreeBSD] 20040728],
PIL 1.1.5


That's great.
I tried Tarek's PloneSoftwareCenter branch today. The good news is: IT IS WORKING. Bravo Tarek. If you don't already know it, the branch is about adding PYPI features to PSC.

Used software for test:

  • Plone 3.0.4

  • DataGridField 1.6.0-beta2

  • PloneHelpCenter 1.5.0-beta-5

  • PloneSoftwareCenter (Tarek's branch named pypi)

You need a .pypirc file in your home directory with the username/password for an existing plone user:

[server-login]
username: mustap
password: secret


Now one can easily upload a package to PSC:

$ python2.4 setup.py bdist_egg upload \
--repository=http://localhost:8080/site1/software-center

running bdist_egg
running egg_info
writing requirements to test.egg-info/requires.txt
writing test.egg-info/PKG-INFO
writing namespace_packages to test.egg-info/namespace_packages.txt
writing top-level names to test.egg-info/top_level.txt
writing dependency_links to test.egg-info/dependency_links.txt
writing entry points to test.egg-info/entry_points.txt
reading manifest file 'test.egg-info/SOURCES.txt'
writing manifest file 'test.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
running build_py
creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/plone

... [many output] ...

copying test.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
creating 'dist/test-0.1dev-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
running upload
Submitting dist/test-0.1dev-py2.4.egg to http://localhost:8080/site1/software-center
Server response (200): OK


Great !

and here is how it looks like in plone:


One can use easy_install to install the uploded package:

$ easy_install \
-f http://localhost:8080/site1/software-center/simple test

Searching for test
Reading http://localhost:8080/site1/software-center/simple
Best match: test 0.1dev
Downloading http://localhost:8080/site1/software-center/test/releases/0.1dev/test-0.1dev-py2.4.egg
Processing test-0.1dev-py2.4.egg
creating /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages/test-0.1dev-py2.4.egg
Extracting test-0.1dev-py2.4.egg to /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages
Adding test 0.1dev to easy-install.pth file

Installed /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages/test-0.1dev-py2.4.egg
Processing dependencies for test
Finished processing dependencies for test


This is really a good step forward to make PSC pypi-compatible.

Try it.
We can add a 'versions' section in buildout.cfg to indicate the version of used recipes.

ie: buildout.cfg for myProject-1.0

[buildout]

parts =
zope2
instance

versions = myProject-1.0

[myProject-1.0]
plone.recipe.zope2install = 0.2
plone.recipe.zope2instance = 0.8

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.9.7/Zope-2.9.7-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
debug-mode = off
zodb-cache-size = 70000

This way, buildout will use plone.recipe.zope2install version 0.2 and plone.recipe.zope2instance version 0.8. For other stuff like setuptools it will "pick" the best distribution. We can tell buildout to generate an error when it "Picks" a version:

[buildout]

parts =
zope2
instance

versions = myProject-1.0
allow-picked-versions = false

[myProject-1.0]
plone.recipe.zope2install = 0.2
plone.recipe.zope2instance = 0.8

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.9.7/Zope-2.9.7-final.tgz

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
debug-mode = off
zodb-cache-size = 70000
I had a good discussion (by email) with Erik Forsberg about zopeskel.localcommands and how the "insert" stuff works. The discussion started by reporting a setuptools problem with zopeskel.localcommands. Setuptools fails with 'SyntaxError' if a template contains a .py file with cheetah variables.

He came with a suggestion to add '_insert' in the end of file names that should be treated in insert-mode so setuptools will not try to compile them (in case of .py files). I think it is a good suggestion.

So here is how it works now:

The '_insert' in the end of file names is required for .py files and optional for other kind of files. For zopeskel.localcommands there is no difference between "configure.zcml_insert" and "configure.zcml". Both will be treated in insert-mode. In the case of .py files if you don't add '_insert' in the end of their names, setuptools will fail with 'SyntaxError' when installing ZopeSkel. That's normal because of the template-variables.

For readability I recommend using the '_insert' syntax for files that their content should be inserted in existing project files.

As example here is the structure of the "portlet" template as is now:
 portlets/
+portlet_filename+.pt_tmpl
+portlet_filename+.py_tmpl

configure.zcml_insert
profiles/
default/
portlets.xml_insert
tests/
base_+portlet_filename+.py_tmpl

test_+portlet_filename+portlet.py_tmpl


(README.txt updated)
ZopeSkel is a great tool for generating plone projects structure from paste templates.
You can install it with:
$ easy_install \
http://svn.plone.org/svn/collective/ZopeSkel/trunk#egg=ZopeSkel-dev
You can list the available templates:
$ paster create --list-templates
Available templates:
archetype: A Plone project that uses Archetypes
basic_namespace: A project with a namespace package
basic_package: A basic setuptools-enabled package
basic_zope: A Zope project
nested_namespace: A project with two nested namespaces.
paste_deploy: A web application deployed through paste.deploy
plone: A Plone project
plone2.5_buildout: A buildout for Plone 2.5 projects
plone2.5_theme: A Theme for Plone 2.5
plone2_theme: A Theme Product for Plone 2.1 & Plone 2.5
plone3_buildout: A buildout for Plone 3 projects
plone3_portlet: A Plone 3 portlet
plone3_theme: A Theme for Plone 3.0
plone_app: A Plone App project
You can start a new project:
$ paster create -t archetypes myproject
...
You have to answer many questions and in the end you will get an archetype based skeleton to start your project.

Now what will you do to add a new plone content type, Zope 3 utility, ...etc ?
You have to create the new files by hand and copy/paste code from other projects. This is not the best approach to the problem. Martin Aspeli comes with the idea of implementing a paster local commands/templates so one can just add plone content types by running paster local commands inside the project directory.

zopeskel.localcommands is one implementation of such idea.

How to use zopeskel.localcommands

To try it, create a test environment with virtualenv:
$ virtualenv myenv
$ cd myenv
Install my ZopeSkel branch:
$ ./bin/easy_install \
http://svn.plone.org/svn/collective/ZopeSkel/branches/mustap#egg=ZopeSkel-dev
You can check the available templates:
$ ./bin/paster create --list-templates
Nothing special :)
check now the available commands:
$ ./bin/paster --help
...
Commands:
create Create the file layout for a Python distribution
help Display help
make-config Install a package and create a fresh config file/directory
points Show information about entry points
serve Serve the described application
setup-app Setup an application, given a config file
Nothing special too :)
Now create an archetype project as described above, cd to the project folder and
check the available commands:
$ cd myproject
myproject$ ../bin/paster --help
...
Commands:
create Create the file layout for a Python distribution
grep Search project for symbol
help Display help
make-config Install a package and create a fresh config file/directory
points Show information about entry points
serve Serve the described application
setup-app Setup an application, given a config file

ZopeSkel local commands:
addcontent Add plone content types to your project
You get a new section called "ZopeSkel local commands" with one command called "addcontent". This new section is only available if paster detects that your project is "addcontent" aware (more about this later) . To see the list of available templates from this command:
myproject$ ../bin/paster addcontent --list
Available templates:
portlet: A portlet skeleton
To add a portlet to your project, run the following command from anywhere inside your project (you don't need to be in the project's root folder):
myproject$ ../bin/paster addcontent portlet
Enter portlet_name (Portlet name (human readable)) ['Example portlet']: My Portlet
Enter portlet_type_name (Portlet type name (should not contain spaces)) ['ExamplePortlet']: MyPortlet
Enter description (Portlet description) )['']: My Portlet
Recursing into portlets
Copying +portlet_filename+.pt_tmpl to /home/mustapha/Projects/pylonsenv/test/plone/example/portlets/myportlet.pt
Copying +portlet_filename+.py_tmpl to /home/mustapha/Projects/pylonsenv/test/plone/example/portlets/myportlet.py
Recursing into profiles
Recursing into default
Recursing into tests
Copying +portlet_filename+_base.py_tmpl to /home/mustapha/Projects/pylonsenv/test/plone/example/tests/myportlet_base.py
Copying test_+portlet_filename+.py_tmpl to /home/mustapha/Projects/pylonsenv/test/plone/example/tests/test_myportlet.py
You get new files and if you check the configure.zcml in the portlets folder and the portlet.xml in the profiles/default folder, you will see that they are updated too.

How to add new templates ?

The python part is very similar to what you know from ZopeSkel. The only difference is that your template class must inherit from ZopeSkelLocalTemplate.

The template structure on the file system is the same as what you know from ZopeSkel with one difference: if your project has a file with the same name as a template file, the "addcontent" command will try to operate in insert-mode.

To make things easy, let's take Martin's plone 3 portlet template and transform it to zopeskel.localcommands template so you can add as many portlets as you want to your project. Here is the files and the structure we use:
 portlets/
+portlet_filename+.pt_tmpl
+portlet_filename+.py_tmpl
configure.zcml_insert
profiles/
default/
portlets.xml_insert
tests/
base_+portlet_filename+.py_tmpl
test_+portlet_filename+portlet.py_tmpl
The files that end with "_tmpl" will be handled as normal ZopeSkel templates.
But the new thing here is the files that end with "_insert". The content of these files will be inserted in the correspondent files of the destination project. As example here is the content of the file portlet.xml_insert:

#<?xml version="1.0"?>
# <!-- This file is used to register new types of
# portlets. It can also be used to register
# completely new column types. See CMFPlone's
# version of this file for more information.
# -->
# <portlets>
# <!-- -*- extra stuff goes here -*- -->

<portlet
addview="${dotted_name}.${portlet_type_name}"
title="${portlet_name}"
description="${description}"
/>

#</portlets>
Here, some notes are needed:

1. If your project contains a file named "profile/default/portlets.xml", only the lines not starting with "#" will be inserted

2. If your project doesn't contain the file "profile/default/portlets.xml", the "#"'s in the beginning of lines will be removed and a profile/default/portlets.xml" file will be created with the hole content. So it is a good idea when creating a new local template to think about how the file will look like if it has to be created from scratch. Look at the zcmlmeta template as example.

Now, if you are asking how "zopeskel.localcommands" knows where (in the destination file) to insert the content of the source file, keep reading.

NOTE: The "_insert" in the end of the name of the template file is required for .py files and optional for other kind of files. For zopeskel.localcommands there is no difference between configure.zcml_insert" and "configure.zcml". Both will be treated in insert-mode. In the case of .py files if you don't add "_insert" in the end of their name, setuptools will fail with 'SyntaxError' when installing ZopeSkel. That's normal because of the template variables in the file. For readability I recommand using the '_insert' syntax.


Now, take a look to the python part. Here is the Portlet class:
from zopeskel import var
from zopeskel.localcommands import ZopeSkelLocalTemplate

class MyPortlet(ZopeSkelLocalTemplate):
"""
A plone 3 portlet skeleton
"""
_template_dir = 'templates/portlet'
summary = "A Plone 3 portlet"
use_cheetah = True

vars = []
vars.append(var('portlet_name',
'Portlet name (human readable)',
default="Example portlet"))

vars.append(var('portlet_type_name',
'Portlet type name (should not contain spaces)',
default="ExamplePortlet"))

vars.append(var('description',
'Portlet description',
default=""))

def pre(self, command, output_dir, vars):
"""
you can use package_namespace, package_namespace2,
package and package_dotted_name of the parent
package here. you get them for free in the vars
argument. ie:
vars['package_namespace'] contains 'plone'
vars['package_namespace2'] contains 'test' or ''
vars['package'] contains 'example'
vars['package_dotted_name'] contains 'plone.test.example'
"""

vars['portlet_filename'] = \
vars['portlet_type_name'].lower()

vars['dotted_name'] = "%s.portlets" % \
vars['package_dotted_name']
Three things to say here:
1. Your template class has to inherit from ZopeSkelLocalTemplate
2. You can get the parent project namespaces and package name for free in the vars argument as you see in the "pre" method.
3. The ZopeSkelLocalTemplate class defines an attribute named "marker_name" like this:
    marker_name = "extra stuff goes here"
Your can override it in your template class if you want. The purpose of this marker_name is to tell "addcontent" command where to insert the content if it has to insert something in a file. The correspondent file must include a line containing " -*- marker_name -*-" in our case:
-*- extra stuff goes here -*-
if you look in portlet.xml of your project you will find a line containing:
<!-- -*- extra stuff goes here -*- -->
If you want to know more about marker_name look at insert_into_file method of paste.script.command.Command.

OK, now we have the file system structure and the template class. We need just one more thing: add an egg entry point for our template class to make it available for the "addcontent" command. As I said before it is very similar to how you add a normal ZopeSkel template. The difference is that for normal ZopeSkel templates the entry point is added under [paste.paste_create_template] section, but zopeskel.localcommands adds a new kind of entry points named [zopeskel.zopeskel_sub_template] and our templates have to be added under that section. If you look in the setup.py file of ZopeSkel to the entry_points argument, you will find:
      [zopeskel.zopeskel_sub_template]
portlet = zopeskel.localcommands.templates:MyPortlet

Enable "addcontent" in other ZopeSkel templates

To enable "addcontent" command in your current project add a line with "ZopeSkel" to your paster_plugins.txt file of your egg-info folder.

For the moment the "addcontent" command is only enabled for ZopeSkel's archetype template. You can enable "addcontent" command for other ZopeSkel templates by addinng an "egg_plugins" attribute to the template class like this:
   egg_plugins = ['ZopeSkel']

With ZopeSkel you can easily generate a setuptools-ready skeleton for your plone product. To install it:
$ easy_install \
http://svn.plone.org/svn/collective/ZopeSkel/trunk#egg=ZopeSkel-dev
Tip1:

I faced a problem after generating my skeleton with:
$ paster create -t plone_app myploneapp
After I added some python files to the product, I run the following command to see how the package will be built:
$ python setup.py build
I checked the 'build' folder. All the files I added are not there :( !!!. Setuptools didn't pick my files.
To get all files in the built, you have to do one of these:

  • edit the SOURCES.txt file in the egg-info folder and add the full path to the files you added.

  • setuptools is an SVN friendly tool. Put your package in an svn repository and check it out. Your working copy will be setuptools-friendly. When you add some files to your package, run:
$ svn add path_to_your_files/
and then:
$ python setup.py build
look in the 'build' folder. You have a complete package without missing files and the SOURCES.txt file is updated.

Tip2:

When generating a plone product skeleton with paster you can tell it to create an svn repository with the standard trunk/ tags/ branches/ hierarchy.
$ paster create -t plone_app myploneapp \
--svn-repository=http://your.svn.server.org/path/to/your/project
This will create:

  • The 'myploneapp' product in the current folder

  • myploneapp/trunk, myploneapp/tags and myploneapp/branches under http://your.svn.server.org/path/to/your/project repository


if you get "No egg-info directory found" error, just update your setuptools to the latest version. This is a known bug in versions prior to 0.6c7. If you cannot or you don't want to update setuptools for some reasons, do:
myploneapp$ svn ci -m 'I have to check in' 
and then
myploneapp$ python setup.py egg_info
this should fix the problem.

Your friends can install your product by running:
$ easy_install \ 
http://your.svn.server.org/path/to/your/project/myploneapp/trunk

Tip3:

Vidar wrote a very good tool to generate plone3 product skeleton from wsl files. To install it:
$  easy_install \
http://svn.plone.org/svn/collective/genesis.wsl/trunk
You will get a new paster template named wsl_plone. Take the Vidar's example from his page and save it as myploneapp.wsl and run:
$ paster create -t wsl_plone myploneapp wsl=myploneapp.wsl \
--svn-repository=http://your.svn.server/your_path/
$ svn ci myploneapp/
to get an svn working copy of your plone3 product.

Problem:

You need to setup Zope behind Apache with SSL and you need to access some/all of the CGI environment variables set by the mod_ssl from within Plone. How to do it ?

To setup Zope behind Apache with SSL is not the hard part. I'll give anyway an example of setting an apache virtualhost with SSL.

Apache doesn't forward the mod_ssl CGI environement variables to Zope. Why ? Because Zope doesn't support SSL until now.

When you setup apache with SSL as proxy for your Plone site, it (apache) receives HTTPS-requests from the outside but it sends HTTP-requests to Zope. That's why you don't get the SSL headers through to the proxied Plone site.

» Read More

I'd like it to be my code. But that is not the case .-)
This article at Dr. Dobb's Journal is a good reading. It is from 2005. It explains what beautiful code is.

link: http://www.ddj.com/dept/architect/184407802
Inspired by this caching recipe and to continue with the previous post, I think it will be more readable, clean and reusable if we use a decorator to cache methods inside a tool:

class myTool(...):

@cachedmethod
def getMyDataFromDB(*args, **kw):

data = fetch_mydata_from_db()

return data
Decorators make magic easy.

The cachedmethod decorator would be as follow (The code is not tested):

» Read More