Plone Testing Tools And Techniques

Plone Testing Tools and Techniques
An Overview

Presented By: Jordan Baker, Scryent
Plone Conference 2009, Budapest

Image CC Encyclopedia Britannica

Wednesday, October 28, 2009

Goals of this Talk

• practical and relevant usage of testing in Plone, not exhaustive

• give you the conﬁdence to start testing, or incorporate it more fully into your
development process

• give you some hands-on examples of using testing

• provide a good review of the topic for more experienced developers

• provide some quality resources as reference


About Me (Jordan Baker)

• Using Plone since version 1.0

• Senior Consultant/Founder of Scryent, a Plone service provider based in
Toronto, Canada

• Over the years I have gone from writing no tests, to writing lots of tests, to
writing tests ﬁrst, to getting 100% code coverage and beyond

• Quality code and enhanced developer productivity are some of my passions


Slides and examples

• These slides are available on Slideshare so you can follow along:

http://tinyurl.com/pttt-slides

• Also the Plone example used in this talk is available from bitbucket. To
checkout a copy and build:

$ hg clone https://hexsprite@bitbucket.org/hexsprite/pttt/
$ cd pttt
$ python2.4 bootstrap.py && bin/buildout


When to test?

• When writing production code. Testing reduces defects as measured by
reported bugs.

• Prototyping doesn't always require tests, as there is indeed some overhead to
writing and maintaining tests.

• Safe as long as you remember to throw away your prototypes.


Testing Results

• improves code quality, reduced defects

• provides more maintainable developer documentation

• creates more loosely coupled code which can offer maintenance and re-use
improvements

• “Integrating Unit Testing into a Software Development Team's Process”:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.5798&rep=rep1&type=pdf


Why Test

• produce more robust code faster

• tests passing is proof that you delivered what you were contracted to
produce

• acceptance tests can map directly to your software requirements

• business measurement help prevent scope creep

• finding bugs early makes them cheaper to fix

• refactor with confidence


Test First!

• writing tests ﬁrst forces you to get clear about requirements up front

• facilitates a conversation with your product owners

• generally results in more modular and reusable code

• manual testing is slow and error-prone! The longer you put off writing tests
the less efﬁcient you are.


Test Levels in the Testing Pyramid

functional,
system

integration

unit


Test-Code Cycle


More tests is not always better

• Tests are software too and must be maintained.

• Too many tests makes running your entire test suite slow.

• Remember the pyramid. Just like the food pyramid you want to focus on
having lots of tests from the lower levels which run quickly, and fewer from
the top of the pyramid.

• Some people are opportunistic testers. They will write a minimal test for the
main path of their code and then only write additional tests if regressions are
found.


Unit Tests

• Answer the question: Is the code doing the right thing? Written from
developer’s perspective

• Tests implementation of a sliver of behaviour

• Fast

• Doesn't rely on any external resources (so use mocks or fakes for external
resources)

• Have lots of these, for every important variation of behaviour

• Python’s unittest module - http://docs.python.org/library/unittest.html


When to Unit Test

• as often as possible when able to test components in isolation.

• if you depend on a few outside components, think about using a Mock object
to represent them

• if you rely on having a full Plone site conﬁgured, then you are a better off with
an integration test


Unit Testing Example

class TestFizzBuzz(unittest.TestCase):
"""Unit test for the FizzBuzzWriter
"""

def setUp(self):
self.fizzbuzz = FizzBuzzWriter()

def test_numbers_divisable_by_three(self):
self.failUnlessEqual(self.fizzbuzz.write_number(3), 'FIZZ')

• setUp called before each test method

• tearDown called after each test method

• methods beginning with test* are automatically added


Unit Testing Boilerplate

• Tests are defined in a package named tests. Typically this should be created
a subdirectory tests in your package. Don’t forget an empty __init__.py!

• Create a seperate test file for each component you are testing, eg.
test_fizzbuzz.py

• Only files starting with test_ are picked up by Zope’s test runner

• Required to find all tests and make them available to the test runner

def test_suite():
return unittest.defaultTestLoader.loadTestsFromName(__name__)


Test methods

• Base class unittest.TestCase deﬁnes several assertion methods to use:

• failIf(expr[, msg])

• failUnless(expr[, msg])

• failUnlessEqual(ﬁrst, second[, msg])

• failUnlessRaises(exception, callable, ...)

• there’s also equivalent methods that start with assert...()

• and more, see docs


FizzBuzz implementation

class FizzBuzzWriter(object):
def write_number(self, number):
if number % 3 == 0 and number % 5 == 0:
return 'FIZZ BUZZ'
if number % 3 == 0:
return 'FIZZ'
if number % 5 == 0:
return 'BUZZ'
return str(number)

def write_numbers(self, count):
return ( self.write_number(number) for number in range(1, count + 1) )


Using Zope's testrunner

• TIP: run only the test you need to run:

$ bin/instance test -s your.module -t test_something

• By using -s you can specify the module to look for tests.

• If you want to run only one particular test method then use -t to specify which
one.

• Also to run only unit tests (as opposed to integration and functional tests) use
the -u option:

$ bin/instance test -s your.module -u


Running the Tests

$ bin/instance test -s pttt.fizzbuzz
Running tests at level 1
Running unit tests:
Running:
.....


Sample Applications of Unit Testing in Plone

• View Classes: instantiate your view class, using mock objects as necessary
for request and context

• Test validators for all corner cases, eg. phone number validator

• Simple Event handlers


Additional Unit Testing Resources

• ofﬁcial unittest docs: http://docs.python.org/library/unittest.html

• http://plone.org/documentation/tutorial/best-practices/unit-testing

• http://www.slideshare.net/Quintagroup/intro-to-testing-in-zope-plone-
presentation

• http://trizpug.org/Members/tbryan/unittest_talk


Integration Tests

• Plone is a complex system with many moving parts

• integration tests will be necessary to confirm that components are properly
registered and work together

• When to use

• when testing integration of various components and the flow in and out of
your code. eg. cataloging, relationships, object creation, security, workflow

• Verifying that view was registered correctly and works in context of the
Plone site


Integration Testing with PloneTestCase

• Builds on top of ZopeTestCase and PortalTestCase.

• All tests happen in a sandbox. Zope cleans up after you and it doesn’t affect
any of your on-disk ZODBs.

• They are by their very nature quite slow. So write them as unit tests if
possible. Essentially a full Plone site is created to run your tests in.


Plone Integration Test Example

• Because users love to run FizzBuzz so often they want to generate a Page of
Fizz Buzz results.

• Let’s introduce a new view (using ﬁve.grok) which when given an appropriate
number and a document id will generate a page with the results.


Writing an Integration Test

class FizzBuzzIntegration(PloneTestCase):
layer = ptc_layer

def test_fizzbuzz_contents(self):
view = self.folder.restrictedTraverse('@@fizzbuzz')
view.generate_document(5)

# document with id fizzbuzz exists
self.failUnless('fizzbuzz' in self.folder)

text = self.folder.fizzbuzz.getText()

# verify contents are correct (Document converted to HTML for us)
self.failUnlessEqual(text, '1 2 FIZZ 4 BUZZ')

def test_fizzbuzz_anonymous_cant_generate(self):
self.logout()
view = self.folder.restrictedTraverse('@@fizzbuzz')
self.assertRaises(Unauthorized, view.generate_document, 5)

Quick review of PloneTestCase’s API

• self.loginAsPortalOwner()

• .login(user)

• .logout()

• .setRoles([roles])

• .portal

• .folder

• PloneTestCase.default_user


collective.testcaselayer and PloneTestCase

• collective.testcaselayer, developed by Ross Patterson, provides layer base
classes for Zope, CMF and Plone development.

• Layers allow expensive setup of ﬁxtures, such as a Plone site, to be shared
by many tests.

• Simpliﬁes repetitive Plone testing boilerplate code while providing a clean
way to extend.

• Also of interest is the ability to create ZODB sandboxes for each layer so that
sibling layers are isolated from each other.


Installing collective.testcaselayer using testing
extras

• When to use: when adding dependencies that are only required for testing

• examples: collective.testcaselayer, mocker library, zc.testbrowser

• dependencies are registered as setuptools extras so they can be optionally
included

• you should have a separate development buildout conﬁguration that speciﬁes
the extras


Adding collective.testcaselayer

• Use the extra_requires functionality of setuptools so that production usage
won’t require collective.testcaselayer

•
from setuptools import setup, find_packages
...
tests_require = ['collective.testcaselayer']
...
setup(name='pttt.fizzbuzz',
...
install_requires=[
'setuptools',
# -*- Extra requirements: -*-
],
tests_require=tests_require,
extras_require={'tests': tests_require},
...
entry_points="""


Enabling tests extras in your buildout

• Find the eggs section in your buildout part and add:

...
eggs +=
pttt.fizzbuzz [tests]
...


Autoinclude your ZCML

• Plone 3.3 includes support via z3c.autoinclude to automatically load ZCML in
dependent packages using a setuptools entry point.

• For Plone 3.2 you have to add z3c.autoinclude to your buildout and
something like this somewhere in your buildout ZCML:

<includePlugins package="plone" file="configure.zcml" />

• Here’s how we’d register ours in our packages’ setup.py:

entry_points="""
[z3c.autoinclude.plugin]
target = plone
“"",

Generating the Fizzbuzz document

class FizzBuzz(grok.View):
grok.context(ATFolder)
grok.require('zope2.View')

def generate_document(self, count):
writer = FizzBuzzWriter()
text = 'n'.join(writer.write_numbers(count))
self.context.invokeFactory('Document', id='fizzbuzz', title="Fizzbuzz
results", text=text)


Running the test

$ bin/instance test -s pttt.fizzbuzz -t test_fizzbuzz_contents -t
test_fizzbuzz_anonymous_cant_generate


Make it faster with roadrunner

• Roadrunner is a tool that preloads Plone to make running your integration
tests much much faster. http://pypi.python.org/pypi/roadrunner

• Due to interactions with z3c.autoinclude it will currently only reload your tests
without repeating the Plone site setup.

• Its easy to use too. Just add a roadrunner part to your buildout.cfg like so:
[roadrunner]
recipe = roadrunner:plone
packages-under-test = pttt.*
• Roadrunner accepts the same command line arguments as Zope testrunner.
So we can run our tests like this:

$ bin/roadrunner -s pttt.ﬁzzbuzz


Plone Integration Testing Resources

• Programmatically Manipulate Plone: http://plonemanual.twinapex.ﬁ/content/
index.html

• PloneTestCase Tutorial - PloneConf 2004: http://www.zope.org/Members/
shh/Tutorial

• http://plone.org/documentation/tutorial/testing/integration-doctests-using-
plonetestcase

• PloneTestCase: http://plone.org/products/plonetestcase


Doctests

• tests that look like interactive Python sessions

• When to use

• API Documentation, verify examples work

• its a matter of taste, some people prefer unittests especially when there
are a lot of variations

• they are especially good for functional tests where the prose explains the
acceptance test and the code veriﬁes that it is working


Doctest boilerplate

import unittest
import doctest

from Testing import ZopeTestCase as ztc

from example.tests.tests import base

def test_suite():
"""This sets up a test suite that actually runs the tests in the class
above
"""
return unittest.TestSuite([

ztc.ZopeDocFileSuite(
'tests/functional.txt', package='example.tests',
test_class=base.ExampleFunctionalTestCase,
optionflags=doctest.REPORT_ONLY_FIRST_FAILURE |
doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS),

])


A Simple Doctest

First we'll import and instantiate our FizzBuzzWriter class

>>> from pttt.fizzbuzz.fizzbuzz import FizzBuzzWriter
>>> writer = FizzBuzzWriter()

Let's write out single numbers

>>> writer.write_number(1)
'1'
'FIZZ'
'FIZZ BUZZ'

Finally let's write a sequence of numbers

>>> list(writer.write_numbers(15))
['1', '2', 'FIZZ', '4', 'BUZZ', 'FIZZ', '7', '8', 'FIZZ', 'BUZZ',
'11', 'FIZZ', '13', '14', 'FIZZ BUZZ']


Functional and System Tests

• End to end tests

• Typically black-box tests that act as an end-user would and are written from
end user’s point of view. eg. click the button labeled “Login” vs. call the login
method

• inherently brittle as the UI often changes

• Use zope.testbrowser, twill, Selenium

• typically written as doctests but don’t need to be


Functional Tests

• When to use:

• blackbox testing

• automated acceptance testing

• Smoke or face-saving tests - exercise most important paths of business
value

• When not to use:

• testing every corner case. use unit tests for that


About zope.testbrowser

• testbrowser implements a Python scriptable browser that simulates
accessing your Plone site.

• Full docs for zope.testbrowser: http://pypi.python.org/pypi/zope.testbrowser

• Note that using the testbrowser in Plone we should use the Five version
which wraps it for use in a Zope 2 environment:

from Products.Five.testbrowser import Browser

• Not Javascript aware. If you need to test DHTML, Javascript, Ajax et al then
you will need to use something like Selenium.


Using zope.testbrowser

• Don’t hammer Plone.org ;)

from zope.testbrowser.browser import Browser
# this version of testbrowser works with external resources
# for testing your own apps using internal Browser class:
# from Products.Five.testbrowser import Browser

browser = Browser('http://plone.org')
browser.getControl(name='SearchableText').value = 'Testing'
browser.getForm(id='search').submit()
assert browser.isHtml
assert 'Untested Code is Broken Code' in browser.contents


Plone Functional testbrowser API basics
• Open your test Plone site: browser.open('http://nohost/plone/')

• Click a link: browser.getLink(link_text).click()

• Click the ﬁrst link when multiple have the same name:
browser.getLink(link_text, index=0).click()

• Get URL: browser.url

• Reload: browser.reload()

• Change form value: browser.getControl(input_name).value = ‘Whatever’

• Submit form: browser.getForm(id)


Two ways to login

• One is to use Plone’s login mechanism, arguably more black-box since it
should simulate what your users will actually do.

>>> browser.open(portal_url)
>>> browser.getControl(name='__ac_name').value = portal_owner
>>> browser.getControl(name='__ac_password').value =
default_password
>>> browser.getControl(name='submit').click()

• Alternatively you could add an auth header:

>>> browser.addHeader('Authorization', 'Basic root:secret')


Functional Testing Example

• Let’s take a look at an existing functional testing example from Plone’s source
code from plone/app/workﬂow/tests/sharingpage.txt:

• A couple things about this

• browser.handleErrors = False means you’ll get proper tracebacks
instead of cryptic 500 Errors

• demonstrates pure black box testing, no assumptions of internal state


Functional Testing Resources

• http://plone.org/documentation/tutorial/testing/functional-tests

• Face Saving Tests: http://gojko.net/2007/09/25/effective-user-interface-
testing


System Tests using twill

• You can write system tests using testbrowser as well

• Twill provides a simple domain speciﬁc language (DSL) for performing web
tests. It also provides a Python API.

• When to use

• when doing whole system testing that includes caching proxy, multiple
web servers, backend and database

• testing live systems

• deployment testing


Installing twill in your buildout

• twill is on pypi, simply use zc.recipe.egg to install its scripts:

[twill]
recipe = zc.recipe.egg
eggs =
twill

• Also make sure to add the part to your [buildout] parts section.


Simple Twill Script

go http://plone.org
formvalue searchform SearchableText testing
submit

# assert that we found the following text
find 'Untested Code is Broken Code'

# assert that we didn't find this text:
notfind 'Testing sucks'


Running twill interactively
$ bin/twill-sh

-= Welcome to twill! =-

current page: *empty page*
>> go http://plone.org
==> at http://plone.org
current page: http://plone.org
>> formvalue searchform SearchableText testing
current page: http://plone.org
>> submit
current page: http://plone.org/search?SearchableText=testing
>>
>> # assert that we found the following text
>> find 'Untested Code is Broken Code'
>>
>> # assert that we didn't find this text:
>> notfind 'Testing sucks'


Writing a System Test

• Verify your SSL conﬁguration:

# TEST: join form redirects to https
go http://mysite.com
follow "Join"

# test that the resulting URL is https://
url "^https:"
url "/join_form$"
find "Registration Form"

# fill in registration form and register
fv 2 fullname "Test User"
fv 2 email "test@tester.com"
submit
find "You have been registered."
fv 2 form.submitted 1
submit
find "You are now logged in"

# home page is http
url "^http://"


Mock and Fake objects

• When to use:

• unittest code that has external dependencies such as databases or
network connections

• depends on external components


Basic Mocks

• Let’s say you have a test method that expects an object to be provided which
provides an absolute_url method in order to return a calculated URL:

• Without using an Integration test, how could you do this? Simply create a
simple Python class that implements only the methods you need.

• This can get tiresome if you have to create many of these mocks, or if you
want to make some assertions about what is called on those mocks with
what arguments.


Basic Mocking Example

def calc_url(context):
return context.absolute_url() + "/calculated_url"

class MockDocument:
def absolute_url(self):
return "http://plone/some_document"

def test_calc_url(self):
mock = MockDocument()
self.failUnlessEqual(calc_url(mock), "http://plone/some_document/calculated_url")


Mocker

• Python library for creating Mock objects and managing expectations we have
for those Mock objects.

• Other more advanced features as well such as proxies, patching and proxy
replacement.

• Unittest integration

• Martin Aspeli took this further and created plone.mocktestcase which
provides some helper methods that are speciﬁc to Zope and Plone
development.

http://pypi.python.org/pypi/plone.mocktestcase


A Mocking Example

• Writing an interface to an external network service

• Using the Component Architecture to create an abstract interface for a
remote network service and using MockTestCase’s mock_utility to make this
available to application code under test:

In the
network_service = self.mocker.mock()
self.mock_utility(self.video_provider, INetworkService)


Mocking Resources

• http://www.martinaspeli.net/articles/mock-testing-with-mocker-and-
plone.mocktestcase


Coverage

• answers the question: are my tests covering all the code in my application?

• for production code I aim for 100%. Why? anything less than 100% can hide
defects.

• 100% code coverage is considered minimum standard *

• does NOT guarantee the code has no bugs

• * - http://www.plope.com/Members/chrism/a_hundred_percent


Coverage won’t catch this
• This pseudo-code example would give 100% code coverage. What if I
passed 0 to myfunc?

def test_myfunc():
myfunc(100)

def myfunc(n):
return 100 / n


Generating coverage report

• Zope’s testrunner comes with a builtin coverage report tool, invoke like this:

$ bin/instance test -s pttt.ﬁzzbuzz --coverage=./coverage

• When calculating coverage you should know that code runs much slower. So
grab a coffee, tea or water and catch up on your xkcd.

• The results are in:

Ran 5 tests with 0 failures and 0 errors in 0.005 seconds.
Tearing down left over layers:
Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
lines cov% module (path)
16 100% pttt.fizzbuzz.fizzbuzz (/Users/jbb/co/pttt/src/pttt.fizzbuzz/
pttt/fizzbuzz/fizzbuzz.py)


HTML Coverage report using z3c.coverage

• Produces nice looking reports that you can discuss with your team:


HTML Coverage report using z3c.coverage

• Add this part to buildout:
[z3c-coverage]
recipe = zc.recipe.egg
eggs = z3c.coverage
extra-paths = ${zope2:location}/lib/python

• Don’t forget to add z3c-coverage to [buildout] parts =

• Generate reports:

$ bin/coverage ./coverage ./coverage/reports


Coverage Resources

• http://plone.org/documentation/tutorial/testing/code-coverage


Pdb-fu (debugging)

• Getting test failures from Zope Testrunner? Use -D to drop you into Pdb using
post-mortem debugging

• Use pdb’s jump command to re-run certain lines of codes. Watch out for
side-effects.

• Special variable __return__ is set. Use “continue” to go to the end of the
method and then you can easily examine the return result.

• Breakpoint variables. Looping through 1000 items, set a breakpoint with an
expression.

• Missing source code? This could help, add to your buildout.cfg:
[buildout]
unzip = true


General Testing Tips

• Write at the lowest level possible. This will ensure that your test will run as
quickly as possible which becomes important when you have hundreds or
thousands of tests.

• Take your time, write out what you’re trying to do in your commends ﬁrst
before writing actual code:

def test_whatever(self):
# verify document was created
# ensure that notification was sent


General Testing Tips

• Broken tests that lie around for a long time because they are minor issues
that no one can be bothered to ﬁx can mask larger bugs due to too much
noise and not enough signal. 0 failing tests (and no cheating!)

• Hard to test something? Step back, try to break it up into easier to test
pieces.


Some Further Resources

• Plone Testing tutorial: http://plone.org/documentation/tutorial/testing/

• plone.reload: http://pypi.python.org/pypi/plone.reload

• http://diveintopython.org/unit_testing/index.html

• example tests of all sorts, mostly still valid: http://svn.plone.org/svn/
collective/examples/example.tests/trunk/


Open Spaces and Sprinting

• For all of those who want to have open discussions about Testing in Plone
we’ll have an Open Space on Friday.

• Also I’ll be sprinting on bringing roadrunner up to spec with Plone 4, Dexterity
and Zope 2.12. Interested? Drop me a line.


Thanks

• Your Feedback is appreciated: jbb@scryent.com

• Find me on twitter: @hexsprite

• Good luck and happy testing!


Thanks

• Photos used licensed under Creative Commons 2.0:

• http://www.ﬂickr.com/photos/rupertuk/2102548824/

• http://www.ﬂickr.com/photos/kacey/2626513873


Plone Testing Tools And Techniques

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Plone Testing Tools And Techniques (20)

Recently uploaded (20)

Plone Testing Tools And Techniques