Must@p Zone

Adjusting options in buildout

Tue, 10 Jun 2008 19:15:47 +0200

The configuration syntax in the new version of zc.buildout is extended with two operators += and -= to allow adjusting options in an extended profile. This is very helpful. You don't need anymore to overwrite an option in the base configuration when all you want is to remove/add an entry.

here is an example:



# base.cfg file

[buildout]



parts =

    part1

    part2



eggs =

    eggs1

    eggs2

    eggs3

To add new part and remove egg2 in an extended configuration, we do:



# newconfig.cfg file



[buildout]

extends = base.cfg



parts +=

    part3



eggs -=

    eggs2

Running buildout, the result will be as if you have run it with this configuration file:



[buildout]

parts =

    part1

    part2

    part3



eggs =

    egg1

    eggs3

That's it.
If you want to add new egg to all your profiles all you have to do is to add it to the base.cfg.

Shebang max. length

Sun, 25 May 2008 15:11:26 +0200

This may save someone's time.

If you get an error like this when running ./bin/buildout:

bash: bin/buildout: /home/mustapha/workspace/collective.buildbot/test/parts/buildslave/customer.pr: bad interpreter: Permission denied

Pay attention: the full path to the used python is truncated here.

that means that the shebang line (the first line in the bin/buildout script starting with #!) is too long. The maximum length of a shebang line is system dependent. On my Ubuntu the maximum length is 80. This table lists the maximum length of the shebang line per OS

collective.recipe.supervisor: now accepts arguments

Fri, 23 May 2008 15:44:45 +0200

Now one can add arguments to the command controlled by supervisor. You have to give all arguments between [] like this: [arg1 arg2 arg3]. ~~This way one can use ctl scripts generated by other recipes and give them the appropriate argument to make them run in foreground~~. An example:



[buildout]

parts = 

   instance1 

   supervisor



[instance1]



...



[supervisor]

recipe = collective.recipe.supervisor

port = 9001

user = mustapha

password = secret

serverurl = http://supervisor.mustap.com



programs =

 10 instance1 ${buildout:bin-directory}/instance1 [fg] ${instance1:location}

 10 instance1 ${instance1:location}/bin/runzope ${instance1:location}

 20 myprog /mypath/bin/myprog [-n 100] /mypath

For more info: http://pypi.python.org/pypi/collective.recipe.supervisor

UPDATE: After a discussion with Chris Mc Donough, I updated this post. For zope instances and zeo instances, please use the runzope and runzeo scripts instead of instance and zeo

A buildout recipe to install supervisor

Thu, 22 May 2008 08:43:33 +0200

The collective.recipe.supervisor is buildout recipe that helps with the install and configuration of supervisor.

How to use it: http://pypi.python.org/pypi/collective.recipe.supervisor

self.note = svn import with git

Wed, 7 May 2008 21:00:09 +0200

Let's create a project called plone.app.myproject:

$ mkdir plone.app.myproject 

$ cd !$

$ git svn init -s http://server.com/!$

The last command assumes that plone.app.myproject/trunk, plone.app.myproject/branches and plone.app.myproject/tags already exist in the svn repository.

$ git svn fetch

$ git add .

$ git commit -m 'initial import'

$ git svn dcommit

That's it. At this stage the master branch is reflecting the remote trunk.

Autoload eggs from buildout's development directories

Wed, 30 Apr 2008 19:32:47 +0200

When developing zope/plone eggs with buildout I have to edit the buildout configuration file ( in 3 places ) every time I create/delete/rename a development egg in the src directory or in other development directories (sometime I have more than one).

I have to add/delete/rename the egg in the eggs option of the [buildout] and then add/delete/rename the egg path in the develop option of the [buildout] and in the end add/delete/rename the zcml option of the zope [instance] or in the configure.zcml file of my policy package. This is too much specially when the speed is set to development mode. I need a less boring way to develop.

Solution

buildout.eggtractor is a buildout extension that scans the src directory or a list of directories I give for eggs and picks them up automatically. So no more editing of the buildout's configuration file.

When buildout.eggtractor finds an egg in the scanned directory it:

adds the egg to the eggs option of all zope instance parts or to a set of given parts

adds the egg's path in the develop option of the [buildout]

If tractor-autoload-zcml is not set to false, scans the egg folder for configure.zcml, meta.zcml and overrides.zcml and adds the appropriate zcml entries to the zcml option of the zope instance parts or to a set of given parts.

This steps are done on the fly when running buildout. So I can add/delete/rename an egg and it will be picked up.

NOTE: The extension does not write to the buildout's configuration file.

buildout.eggtractor options

tractor-src-directory: A set of directories to scan for development eggs. Defaults to the src directory of the buildout.

tractor-target-parts: A set of parts to update their eggs option with eggs found in the tractor-src-directory. Defaults to zope instance parts if any.

tractor-autoload-zcml(boolean): Update the zcml option of tractor-target-parts with the eggs found in tractor-src-directory. Defaults to true

tractor-zcml-top: A set of eggs to load their zcml files first. Defaults to an empty set.

How to use it

Using buildout.eggtractor is very simple. As said, it is a buildout extension. All I have to do is to declare it in the extensions option:



  [buildout]

  parts =

  

  extensions = buildout.eggtractor

That's all. buildout.eggtractor will scan the src directory and do its job every time I run the buildout command.

When I have other directories I want to scan I just add an tractor-src-directory option in the [buildout] and add my directories there:



  [buildout]

  parts =

  

  extensions = buildout.eggtractor

  

  tractor-src-directory = 

                        dev-src1 

                        dev-src2

                        src

In a few cases when the priority of loading zcml files matters. I add the egg to be loaded first in the tractor-zcml-top option in the [buildout]:



  [buildout]

  parts =

  

  extensions = buildout.eggtractor

  

  tractor-src-directory = 

            dev-src1 

            dev-src2

            src

            

  tractor-zcml-top = 

            plone.app.mypackage1

If I want to add the eggs found in the development directories to the eggs option of a given set of parts, I add a tractor-target-parts option and add the parts there:



  [buildout]

  parts = instance1 instance2 instance3

  

  extensions = buildout.eggtractor

  

  tractor-target-parts = instance1 instance3

This way only instance1 and instance3 will be updated.

If I have already other way to include the zcml files (ie: z3c.autoinclude) and don't want eggtractor to generate the zcml slugs, I add an tractor-autoload-zcml option and set it to false

In most cases you will only need to add buildout.eggtractor to the extensions option of the [buildout] without any extra configuration options.

LIMITATION

The extension assumes that the egg name reflects its file system structure example: if the egg name is com.mustap.www the extension assumes that the file system structure is one of the following::

com.mustap.www/src/com/mustap/www

com.mustap.www/com/mustap/www

This is where the extension looks for configure.zcml, meta.zcml and overrides.zcml files.

If the egg name does not reflect the file system structure, the extension will ignore it.

XXX: I guess walking through the directory is better than this assumption.
In my case this is not a limitation as I choose my egg names that way.

URL

You can get it from http://pypi.python.org/pypi/buildout.eggtractor

THANKS

Martin, Wichert and Hanno for the help and feedbacks

Running more than one year without stop

Fri, 22 Feb 2008 10:07:27 +0100

My colleague Jakob aka bartholdy discovered this one today. The instance is running more than one year without stop.
Software used:



Plone 2.5.1,

CMF-1.6.2,

Zope (Zope 2.9.4-final, python 2.4.3, freebsd5),

Five 1.3.7,

Python 2.4.3 (#1, Oct 3 2006, 11:03:02) [GCC 3.4.2 [FreeBSD] 20040728],

PIL 1.1.5

That's great.

PYPI in PSC

Thu, 27 Dec 2007 22:08:28 +0100

I tried Tarek's PloneSoftwareCenter branch today. The good news is: IT IS WORKING. Bravo Tarek. If you don't already know it, the branch is about adding PYPI features to PSC.

Used software for test:

Plone 3.0.4

DataGridField 1.6.0-beta2

PloneHelpCenter 1.5.0-beta-5

PloneSoftwareCenter (Tarek's branch named pypi)

You need a .pypirc file in your home directory with the username/password for an existing plone user:



[server-login]

username: mustap

password: secret

Now one can easily upload a package to PSC:



$ python2.4 setup.py bdist_egg upload \

  --repository=http://localhost:8080/site1/software-center

running bdist_egg

running egg_info

writing requirements to test.egg-info/requires.txt

writing test.egg-info/PKG-INFO

writing namespace_packages to test.egg-info/namespace_packages.txt

writing top-level names to test.egg-info/top_level.txt

writing dependency_links to test.egg-info/dependency_links.txt

writing entry points to test.egg-info/entry_points.txt

reading manifest file 'test.egg-info/SOURCES.txt'

writing manifest file 'test.egg-info/SOURCES.txt'

installing library code to build/bdist.linux-i686/egg

running install_lib

running build_py

creating build/bdist.linux-i686/egg

creating build/bdist.linux-i686/egg/plone



... [many output] ...



copying test.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO

creating 'dist/test-0.1dev-py2.4.egg' and adding 'build/bdist.linux-i686/egg' to it

removing 'build/bdist.linux-i686/egg' (and everything under it)

running upload

Submitting dist/test-0.1dev-py2.4.egg to http://localhost:8080/site1/software-center

Server response (200): OK

Great !

and here is how it looks like in plone:

One can use easy_install to install the uploded package:



$ easy_install \

    -f http://localhost:8080/site1/software-center/simple test

Searching for test

Reading http://localhost:8080/site1/software-center/simple

Best match: test 0.1dev

Downloading http://localhost:8080/site1/software-center/test/releases/0.1dev/test-0.1dev-py2.4.egg

Processing test-0.1dev-py2.4.egg

creating /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages/test-0.1dev-py2.4.egg

Extracting test-0.1dev-py2.4.egg to /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages

Adding test 0.1dev to easy-install.pth file



Installed /home/mustapha/Projects/pylonsenv/lib/python2.4/site-packages/test-0.1dev-py2.4.egg

Processing dependencies for test

Finished processing dependencies for test

This is really a good step forward to make PSC pypi-compatible.

Try it.

Note.self = Specifying recipe version in buildout

Sun, 25 Nov 2007 15:55:35 +0100

We can add a 'versions' section in buildout.cfg to indicate the version of used recipes.

ie: buildout.cfg for myProject-1.0



[buildout]



parts =

    zope2

    instance



versions = myProject-1.0



[myProject-1.0]

plone.recipe.zope2install = 0.2

plone.recipe.zope2instance = 0.8



[zope2]

recipe = plone.recipe.zope2install

url = http://www.zope.org/Products/Zope/2.9.7/Zope-2.9.7-final.tgz



[instance]

recipe = plone.recipe.zope2instance

zope2-location = ${zope2:location}

user = admin:admin

debug-mode = off

zodb-cache-size = 70000

This way, buildout will use plone.recipe.zope2install version 0.2 and plone.recipe.zope2instance version 0.8. For other stuff like setuptools it will "pick" the best distribution. We can tell buildout to generate an error when it "Picks" a version:



[buildout]



parts =

    zope2

    instance



versions = myProject-1.0

allow-picked-versions = false



[myProject-1.0]

plone.recipe.zope2install = 0.2

plone.recipe.zope2instance = 0.8



[zope2]

recipe = plone.recipe.zope2install

url = http://www.zope.org/Products/Zope/2.9.7/Zope-2.9.7-final.tgz



[instance]

recipe = plone.recipe.zope2instance

zope2-location = ${zope2:location}

user = admin:admin

debug-mode = off

zodb-cache-size = 70000

ZopeSkel local commands - Insert mode

Sun, 11 Nov 2007 23:09:05 +0100

I had a good discussion (by email) with Erik Forsberg about zopeskel.localcommands and how the "insert" stuff works. The discussion started by reporting a setuptools problem with zopeskel.localcommands. Setuptools fails with 'SyntaxError' if a template contains a .py file with cheetah variables.

He came with a suggestion to add '_insert' in the end of file names that should be treated in insert-mode so setuptools will not try to compile them (in case of .py files). I think it is a good suggestion.

So here is how it works now:

The '_insert' in the end of file names is required for .py files and optional for other kind of files. For zopeskel.localcommands there is no difference between "configure.zcml_insert" and "configure.zcml". Both will be treated in insert-mode. In the case of .py files if you don't add '_insert' in the end of their names, setuptools will fail with 'SyntaxError' when installing ZopeSkel. That's normal because of the template-variables.

For readability I recommend using the '_insert' syntax for files that their content should be inserted in existing project files.

As example here is the structure of the "portlet" template as is now:

 portlets/

          +portlet_filename+.pt_tmpl

          +portlet_filename+.py_tmpl



          configure.zcml_insert

 profiles/

         default/

                 portlets.xml_insert

 tests/

       base_+portlet_filename+.py_tmpl



       test_+portlet_filename+portlet.py_tmpl

(README.txt updated)