Chandler 0.7
This tutorial will step you through the creation of an extension for Chandler. Chandler extensions are usually called “parcels”. The Feeds parcel that we use as an example is an RSS reader that integrates with the existing data model and allows the user to treat RSS data like other data in Chandler. This tutorial will guide you step-by-step through the process of writing such an extension.
The first thing that we need to do is to get a copy of Chandler. Probably the easiest way to do this is to follow the instructions for downloading prebuild binaries at the BuildingChandler wiki page.
When developing for Chandler you often need to pass parameters
to Chandler on startup. To make this easy, you should run
Chandler.py from the command prompt using the
RunPython script that is
located in the release subdirectory. Here are a few
parameters that will be useful while developing our parcel:
-c, --create-W-P <directory>For example, to start Chandler with a fresh repository that is stored in the current directory, with the web server running, use all three parameters.
On Linux or Macintosh type:
$ ./release/RunPython Chandler.py --create -W -P .
On Windows, when using cygwin, type:
$ ./release/RunPython.bat Chandler.py--create -W -P .
If you’re not using cygwin type:
$ release\RunPython.bat Chandler.py --create -W -P .
Note: The web server can also be started from the "Test" menu, using the "Activate built-in web server" menu item.
Note: If you are getting an Internal Error message you may need to add the --catch=never parameter.
The first thing to do is to uninstall the native version of the feeds parcel. This is accomplished by running the following commands in the command prompt:
$ cd projects/Chandler-FeedsPlugin/ $ PYTHONPATH=../../ ../../release/RunPython setup.py develop -u --install-dir=../../ $ cd ../../
Then we need to create an appropriate directory structure for a new project.
In this example we are going to make a new Chandler-FeedsPlugin directory
directly to the user’s home directory but the location of this directory is by no
means fixed and you can place it freely to your directory tree.
$ mkdir ~/Chandler-FeedsPlugin $ cd ~/Chandler-FeedsPlugin
Chandler parcels are built and distributed as
Python Eggs (see also PEAK website).
Throughout this tutorial we assume that you already have a working copy of Python and Egg setuptools installed
on your computer. The next step is to make a project setup and description file that is used to install the parcel
to Chandler. Place the following code in setup.py file.
from setuptools import setup
setup(
name = "Chandler-FeedsPlugin",
version = "0.3",
description = "Simple RSS/Atom feed support for Chandler",
author = "OSAF",
test_suite = "feeds.tests",
packages = ["feeds"],
include_package_data = True,
entry_points = {
"chandler.parcels": ["RSS/Atom Feeds = feeds"]
},
classifiers = ["Development Status :: 3 - Alpha",
"Environment :: Plugins",
"Framework :: Chandler",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python",
"Topic :: Office/Business :: Groupware"],
#long_description = open('README.txt').read(),
)
Then we need to create a Python package that is called feeds by making a directory feeds
and creating an empty feeds/__init__.py file inside it. Here is a list of shell commands
that accomplishes just that (Windows users need to adjust accordingly):
$ mkdir feeds $ cat > feeds/__init__.py ^D
We then need to get a copy of FeedParser which can be downloaded from http://www.feedparser.com or can be copied from the already working Chandler-FeedsParcel directory:
$ cp ...chandler/projects/Chandler-FeedsPlugin/feeds/feedparser.py feeds/.
The last thing to do is to run our setup script to install the parcel to Chandler. Here is how to do that:
$ ...chandler/release/RunPython setup.py develop
If everything worked properly and you did not see any error/warning messages,
you should now have a working project that is ready for development. When you are
done, you can restore the original configuration by running make install
in the Chandler's root directory.
Chandler stores all of its data objects in a Repository (see ), which is an object oriented database. When an object is created in the Repository then certain attributes of the object will persist even if Chandler is shut down and restarted. Instead of storing data in structured tables like a traditional database would do, the Repository keeps all of its objects in a "soup". Objects can be retrieved by making queries to this soup, or simply by following references from one object to another. Any type of attribute can be stored in the Repository including complex data structures like dictionaries, lists, and references to other objects. In all other respects these Repository objects behave just like Python objects.
You may be curious and excited to jump right in and write some code but in the world of Chandler data is king, and the sooner you define your data types the more easily you can build a user interface around it. Start thinking about the units of information that your extension will manage. This will help develop your schema. A schema is a description of your data and how the different data items relate to each other. Don’t worry, you can change your data types later, but you can’t develop any meaningful application without at least some scaffolding.
Here are some questions that may help you get started:
When the schema is well defined, then Chandler will be able to easily aggregate data from our parcel to other parts of the application.
For the Feeds parcel these are the two key pieces of data:
Let’s take a look at some example code that should be placed in feeds/channels.py.
__all__ = ["FeedChannel", "FeedItem"]
__parcel__ = "feeds"
import time, logging, urllib
from datetime import datetime
from PyICU import ICUtzinfo
from osaf.pim.calendar.TimeZone import convertToICUtzinfo, formatTime
from dateutil.parser import parse as dateutil_parse
from application import schema
from util import indexes
from osaf import pim
from i18n import MessageFactory
from twisted.web import client
from twisted.internet import reactor
from osaf.startup import PeriodicTask, fork_item
_ = MessageFactory("Chandler-FeedsPlugin")
logger = logging.getLogger(__name__)
FETCH_FAILED = 0
FETCH_NOCHANGE = 1
FETCH_UPDATED = 2
class FeedChannel(pim.ListCollection):
"""
This class implements a feed channel collection that is visualized
in the sidebar.
"""
#
# FeedChannel repository interface
#
link = schema.One(schema.URL)
category = schema.One(schema.Text)
author = schema.One(schema.Text)
date = schema.One(schema.DateTime)
url = schema.One(schema.URL)
etag = schema.One(schema.Text)
lastModified = schema.One(schema.DateTime)
copyright = schema.One(schema.Text)
language = schema.One(schema.Text)
ignoreContentChanges = schema.One(schema.Boolean, initialValue=False)
isEstablished = schema.One(schema.Boolean, initialValue=False)
isPreviousUpdateSuccessful = schema.One(schema.Boolean, initialValue=True)
logItem = schema.One(initialValue=None)
schema.addClouds(
sharing = schema.Cloud(
literal = [author, copyright, link, url]
)
)
def __setup__(self, *args, **kw):
"""
This method initializes a feed channel.
"""
pass
def refresh(self, callback=None):
"""
This method updates the feed channel content.
"""
pass
The link, category, author, date,
etc. attributes are all basic attributes using simple types that are defined in the schema
module. The refresh() method is like any other Python method.
Behind the scenes: If you are very familiar with
Python, you may notice that Chandler-specific attributes are defined as class
attributes, but in practice they are actually used as instance attributes.
If you have an instance of class FeedChannel that is called fc then
fc.author refers specifically to an instance attribute that stores strings.
Here the attributes are all defined using schema.One, which indicates
that there is just one of them per an instance of the class. You can also use
schema.Many, schema.Sequence, or schema.Mapping
to define Python sets, lists, and dicts, respectively.
You should notice that the FeedChannel class inherits from pim.ListCollection class.
This class is a simple Chandler-based list of items. We use it
to represent the fact that a feed contains zero or more
feed items. Also note that we have not declared anything about the type of
data that this feed can contain. Chandler’s collections can contain
any type of item and each entry in Chandler’s Sidebar is a collection of
some kind.
Now lets look at the definition of a news item in feeds/channels.py:
.
.
.
class FeedItem(pim.ContentItem):
"""
This class implements a feed channel item that is visualized
in the summary and detail views.
"""
#
# FeedItem repository interface
#
link = schema.One(schema.URL, initialValue=None)
category = schema.One(schema.Text, indexed=True)
author = schema.One(schema.Text, indexed=True)
date = schema.One(schema.DateTime)
channel = schema.One(FeedChannel)
content = schema.One(schema.Lob, indexed=True)
updated = schema.One(schema.Boolean)
@apply
def body():
def fget(self):
return self.content
def fset(self, value):
self.content = value
return property(fget, fset)
schema.addClouds(
sharing = schema.Cloud(
literal = [link, category, author, date]
)
)
def __setup__(self, *args, **kw):
"""
This method initializes a new feed item.
"""
pass
The definition of class FeedItem is very similar to that of class
FeedChannel. The most important
distinction here is that class FeedItem inherits from class pim.ContentItem,
which is the base class for all user-visible data in Chandler.
Now that we have defined our two data types, we complete our parcel by placing the
following lines in feeds/__init__.py file.
def installParcel(parcel, oldName=None):
"""
This function installs the RSS feed parcel.
"""
pass
When Chandler creates a new or refreshes an existing Repository it looks at all of the installed parcels and includes any data definitions that it can find.
Now that we have defined the data types that our Feeds parcel
is going to manage, we take a look at how they are integrated into Chandler's user
interface. There are many ways to do this but
the simplest and potentially richest way is to work within the
frameworks of Sidebar, Summary View, and Detail View.
These are the key parts of the Chandler’s graphical user interface:
RSS’s two level hierarchy lends itself well to Chandler’s existing user
interface. Feeds (FeedChannel) contain Feed Items (FeedItem),
which again have details like a body, url, author, and so forth.
Note: A Collection is just that: a generic collection that can contain any kind of Item, including mail messages, calendar events, and RSS news items. An Item on the other hand can be a member of multiple Collections. Because Chandler’s user interface understands these basic constructs, we can create a new Collection for each feed and Chandler will do its best to display the members of those collections correctly.
We need to create a new menu item so that the user can enter a URL
for a new RSS feed. Then we can create a new Collection
that contains the items from that feed.
For this purpose we use the Event system to receive a notification that our menu
item has been clicked, which in turn will run some Python code that generates a new
FeedChannel. Let’s take a look at example code that should be placed in
feeds/blocks.py file.
import logging, wx
from osaf.views import detail
from application import schema, dialogs
import osaf.framework.blocks.Block as Block
from channels import FeedChannel, FeedItem
from i18n import MessageFactory
from osaf import messages
from osaf.pim.structs import SizeType, RectType
_ = MessageFactory("Chandler-FeedsPlugin")
logger = logging.getLogger(__name__)
class AddFeedCollectionEvent(Block.AddToSidebarEvent):
"""
This class implements an event for adding a new collection to the sidebar.
"""
def onNewItem(self):
"""
This method is invoked when the user clicks on "New Feed Channel"
menu item.
"""
pass
def installParcel(parcel, oldVersion=None):
"""
This function defines the feed parcel detail view layout.
"""
detail = schema.ns("osaf.views.detail", parcel)
blocks = schema.ns("osaf.framework.blocks", parcel)
main = schema.ns("osaf.views.main", parcel)
feeds = schema.ns("feeds", parcel)
# Create an AddFeedCollectionEvent that adds an RSS collection to the sidebar.
addFeedCollectionEvent = AddFeedCollectionEvent.update(
parcel, "addFeedCollectionEvent",
blockName = "addFeedCollectionEvent")
# Add a separator to the "Collection" menu ...
blocks.MenuItem.update(parcel, "feedsParcelSeparator",
blockName = "feedsParcelSeparator",
menuItemKind = "Separator",
parentBlock = main.CollectionMenu)
# ... and, below it, a menu item to subscribe to a RSS feed.
blocks.MenuItem.update(parcel, "newFeedChannel",
blockName = "newFeedChannel",
title = _(u"New Feed Channel"),
event = addFeedCollectionEvent,
eventsForNamedLookup = [addFeedCollectionEvent],
parentBlock = main.CollectionMenu,
)
We also need to update our __init__.py to contain a
reference to this code:
def installParcel(parcel, oldName=None):
"""
This function installs the RSS feed parcel.
"""
from application import schema
# load our blocks parcel, too
schema.synchronize(parcel.itsView, "feeds.blocks")
The installParcel() function is called whenever Chandler
starts with a fresh Repository. You’ll need to use --create
every time when you change anything in installParcel().
There are two key concepts that are used here:
First, the main variable refers to a specific area
of the Repository using schema.ns(), called a "parcel". Parcels are persistent containers for Python
objects in the Repository. In this case main.CollectionMenu refers to
the object in the “osaf.views.main” parcel, named “CollectionMenu”. You
can use the web-based Repository Viewer to view the items in this particular parcel here: http://localhost:1888/repo/parcels/osaf/views/main.
You can also navigate to this directory by clicking on the “//parcels”
link in the top of the Repository Viewer at
http://localhost:1888/repo/.
Note: Don’t worry about the second
parameter (parcel.itsView) to schema.ns just yet. This view is a
connection to the Repository. You’ll learn more about repository views later.
Second, we’re not necessarily creating a new MenuItem.
Instead, we’re updating one named “NewFeedChannel” that may
already be in the Repository. If it doesn’t exist only then is a new object
created. The name “NewFeedChannel” uniquely identifies it within the
parcel object that is passed to installParcel. The parcel’s name is more or
less the same as the current module’s name. In this case, the parcel’s
name is “feeds” because it exists in the parcels/feeds directory. The
named parameters to update() will be mapped directly to
attributes on the MenuItem.
There is a connection between the use of the update()
method and schema.ns(). The first parameter passed to
schema.ns() above is the name of a parcel. Any
attributes of the main variable actually reference the parcel named “osaf.views.main” above. Another module
could refer to the “NewFeedChannel” MenuItem like this:
feeds = schema.ns("feeds", view)
feedMenuItem = feeds.NewFeedChannel
To see if everything is working start Chandler with a fresh repository, and run the web server:
$ ./release/RunPython Chandler.py --create -W
Now use your favorite web browser to go to http://localhost:1888/repo/. On the page that is displayed, you should see an area like this:
- feeds: FeedChannel, FeedItem, UpdateTask
- blocks: AddFeedCollectionEvent, FeedItemDetail
How is a menu item connected to the parent menu?
Each parameter to update() sets an attribute in that object.
The parentBlock attribute is used to attach NewFeedsChannel
to the CollectionMenu. This is done with Bidirectional References. The
schema for MenuItem and Menu are defined such that if a
Menu is a parent of a MenuItem, then the MenuItem
is automatically a child of the Menu. Hence, when CollectionMenu
is assigned to NewFeedsChannel’s parentBlock then
NewFeedsChannel is also automatically added to the CollectionMenu’s
childrenBlocks and the Repository will maintain the integrity of this relationship.
We are going to see more uses of bidirectional references later in this tutorial.
Note: The parentBlock/childrenBlocks
relationship is true for all elements in the user interface.
Recall from the above that FeedChannel inherited from ListCollection.
When we create a new FeedChannel collection object we also need to add it to
Chandler’s Sidebar so that the user can see it. Creating a FeedChannel object is just like
creating any other object in Python, except that you can also pass attribute values along
in the constructor and they will be set automatically. Let’s take a look at example code:
class AddFeedCollectionEvent(Block.AddToSidebarEvent):
"""
This class implements an event for adding a new collection to the sidebar.
"""
def onNewItem (self):
"""
This method is invoked when the user clicks on "New Feed Channel"
menu item.
"""
def calledInMainThread(channelUUID, success):
"""
This callback is invoked once the feed has been processed. If all
is okay, success will be True, otherwise False.
"""
self.itsView.refresh(notify=False)
channel = self.itsView.findUUID(channelUUID)
if not channel.isEstablished and not success:
# request a new URL from the user.
url = dialogs.Util.promptUser(
_(u"The provided URL seems to be invalid"),
_(u"Enter a URL for the RSS Channel"),
defaultValue = unicode(channel.url))
url = str(url)
if url != None:
try:
# try to recreate the channel...
channel.displayName = url
channel.url = channel.getAttributeAspect("url", "type").makeValue(url)
channel.isEstablished = False
channel.isPreviousUpdateSuccessful = True
channel.logItem = None
channel.itsView.commit()
# ... and then try to update its contents.
channel.refresh(callback=calledInTwisted)
except:
# unable to recreate the feed channel.
wx.MessageBox (_(u"Could not create channel for %(url)s\nCheck the URL and try again.") % {"url": url},
_(u"New Channel Error"),
parent=wx.GetApp().mainFrame)
def calledInTwisted(channelUUID, success):
"""
This callback is what we really pass to twisted, and it will
invoke the calledInMainThread method in the main thread.
"""
wx.GetApp().PostAsyncEvent(calledInMainThread, channelUUID, success)
# get a URL from the user, ...
url = dialogs.Util.promptUser(
_(u"New Channel"),
_(u"Enter a URL for the RSS Channel"),
defaultValue = u"http://")
if url == None:
return None
url = str(url)
# ... and then try to create a new channel.
try:
# create a new feed channel...
self.itsView.refresh(notify=False)
channel = FeedChannel(itsView=self.itsView)
channel.displayName = url
channel.url = channel.getAttributeAspect("url", "type").makeValue(url)
self.itsView.commit()
# ... and then update its contents.
channel.refresh(callback=calledInTwisted)
except:
# unable to create a new feed channel.
wx.MessageBox (_(u"Could not create channel for %(url)s\nCheck the URL and try again.") % {"url": url},
_(u"New Channel Error"),
parent=wx.GetApp().mainFrame)
return None
# return succesfully
return channel
Here we have a class AddFeedCollectionEvent and in it onNewItem() method.
After creating the channel we do not need to do anything else; the AddFeedCollectionEvent
automatically adds the returned collection to Chandler’s Sidebar. In case of an error we let the
AddFeedCollectionEvent know about it by returning value None.
Note: In this example we use embedded functions
calledInMainThread and calledInTwisted to handle failures
but for the sake of clarity we can ignore them for now.
The Summary View displays three attributes from the items: who, about,
and date. These attributes are usually not true repository attributes but instead
repository attribute redirection is used to forward their references to other attributes in the
class. To define this redirection schema.Descriptor() is used. Note the summary view will display
0 items until we include the refresh functionality much later in the tutorial.
class FeedItem(pim.ContentItem):
.
.
.
@apply
def body():
def fget(self):
return self.content
def fset(self, value):
self.content = value
return property(fget, fset)
This code in FeedItem handles the accessing and changing of the content of the feed.
The Detail View is divided into rows but is a more complex user interface than just a set of
rows. Generally there is one row per attribute, though more complex displays are possible.
Typically a row consists of a label that is followed by the value of an attribute. Every important
attribute should be displayed to the user. Each attribute may have a special way to display
itself or it may require special behavior when it is edited. To display a particular item, we need
to use user interface elements that are known as Blocks. Blocks are persistent Python objects that
wrap wxWidgets classes. The Detail View layout is defined via a list of these Blocks and the
attributes that they will display. Let’s take a look in feeds/blocks.py
file and place the following code in function installParcel():
def installParcel(parcel, oldVersion=None):
"""
This function defines the feed parcel detail view layout.
"""
.
.
.
# The hierarchy of UI elements for the FeedItem detail view
feedItemRootBlocks = [
# The markup bar
detail.MarkupBar,
detail.makeSpacer(parcel, height=6, position=0.01).install(parcel),
# Author area
detail.makeArea(parcel, "AuthorArea",
position=0.19,
childBlocks = [
detail.makeLabel(parcel, _(u"author"), borderTop=2),
detail.makeSpacer(parcel, width=8),
#field("AuthorAttribute", title=u"author"),
detail.makeEditor(parcel, "author",
viewAttribute=u"author",
border=RectType(0,2,2,2),
readOnly=True),
]
).install(parcel),
# Category
detail.makeArea(parcel, "CategoryArea",
position=0.2,
childBlocks = [
detail.makeLabel(parcel, _(u"category"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "category",
viewAttribute=u"category",
border=RectType(0,2,2,2),
readOnly=True),
]
).install(parcel),
# URL
detail.makeArea(parcel, "LinkArea",
position=0.3,
childBlocks = [
detail.makeLabel(parcel, _(u"link"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "link",
viewAttribute=u"link",
border=RectType(0,2,2,2),
readOnly=True),
],
).install(parcel),
# Date area
detail.makeArea(parcel, "DateArea",
position=0.4,
childBlocks = [
detail.makeLabel(parcel, _(u"date"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "date",
viewAttribute=u"date",
border=RectType(0,2,2,2),
readOnly=True,
stretchFactor=0.0,
size=SizeType(90, -1)),
],
).install(parcel),
detail.makeSpacer(parcel, height=7, position=0.8999).install(parcel),
]
# The BranchSubtree ties the blocks to our FeedItem’s Kind.
detail.makeSubtree(parcel, FeedItem, feedItemRootBlocks)
Here we can see four elements in feedItemRootBlocks. The explanation for these
elements reads as follows:
detail.makeArea(parcel, "AuthorArea", ...) author attribute. detail.makeArea(parcel, "CategoryArea", ...) category attribute. detail.makeArea(parcel, "LinkArea", ...) link attribute. detail.makeArea(parcel, "DateArea", ...) date attribute. You should notice that each of the entries in feedItemRootBlocks
has a position attribute that is between 0 and 1. This position attribute
is used to sort the widgets in the display. A position value 0 represents the top of the Detail View
whereas position value 1 represents the bottom. This means that if another Detail View widget
has position value 0.25 then it will appear between AuthorArea and LinkArea.
Finally, we need to register these Detail View blocks with the Detail View itself. Once again, a helper function takes care of this for us:
detail.makeSubtree() BranchSubtree, which is an annotation, that lists the blocks that will appear in the detail view when displaying one of its Items.We call this function with our collection item class and our list of blocks:
def installParcel(parcel, oldVersion=None):
.
.
.
# The BranchSubtree ties the blocks to our FeedItem’s Kind.
detail.makeSubtree(parcel, FeedItem, feedItemRootBlocks)
Most of the items in the childrenBlock lists should be relatively self explanatory.
Calls to makeLabel() and makeSpacer() methdos simply generate a label and
a spacer Block to the left of the actual Detail View. A call to makeEditor() generates a
Block that is called Attribute Editor. An Attribute Editor is a special type of Block that
dynamically creates the correct widget to display and/or edit the given attribute, even if that
attribute is of complex data type like a datetime. The Attribute Editors can be modified on a
case-by-case basis to allow use of read-only blocks or specialized displays for certain types.
It is not yet possible to customize the actual widgets that Attribute Editors use but that is
planned for the future versions of Chandler.
For the most basic attributes Attribute Editors are an appropriate mechanism to display attribute data. If you however need a more complex display beyond that what is already provided you can write your own Block that inherits from a wxWidgets or from an existing Block.
The Feeds parcel needs a way to display the body part of an article. We are going to make use
of an already existing HTMLDetailArea Block to provide a friendly display with proportioned text.
The HTMLDetailArea class is a Block that is able to render documents that are written in HTML.
To make use of it we derive new Block from HTMLDetailArea and implement method
getHTMLText(), which is called when it is time to display a feed item content on screen.
class FeedItemDetail(detail.HTMLDetailArea):
"""
This class implements a block for visualizing feed item content.
"""
def getHTMLText(self, item):
"""
This method renders the feed item content as HTML.
"""
# check that we have a valid feed item.
if item == item.itsView:
return
if item is not None:
displayName = getattr(item, 'displayName', None)
if displayName is None:
displayName =u"<" + messages.UNTITLED + u">"
# make the html
HTMLText = u"<html><body>\n\n"
link = getattr(item, "link", None)
if link:
HTMLText = HTMLText + u"<a href=\"%s\">" % (link)
HTMLText = HTMLText + u"<h5>%s</h5>" % (displayName)
if link:
HTMLText = HTMLText + u"</a>\n"
content = getattr(item, "content", None)
if content:
content = content.getReader().read()
else:
content = displayName
#desc = desc.replace("<", "<").replace(">", ">")
HTMLText = HTMLText + u"<p>" + content + u"</p>\n\n"
if link:
HTMLText = HTMLText + u"<br><a href=\"" + unicode(item.link) + u"\">" + _(u"more...") + u"</a>"
HTMLText = HTMLText + "</body></html>\n"
return HTMLText
To make the new Block visible we just have to make sure that an instance of it is included in
the feedItemBlocks:
def installParcel(parcel, oldVersion=None):
.
.
.
).install(parcel),
detail.makeSpacer(parcel, height=7, position=0.8999).install(parcel),
FeedItemDetail.update(parcel, "ItemBodyArea",
position=0.9,
blockName="articletext",
size=SizeType(100,50),
minimumSize=SizeType(100,50),
),
]
# The BranchSubtree ties the blocks to our FeedItem"s Kind.
detail.makeSubtree(parcel, FeedItem, feedItemRootBlocks)
This ensures that it is part of the BranchSubtree that we built earlier.
To make our RSS reader more user friendly let’s give it the ability to periodically
fetch news stories without that any action is required from the user. Chandler leverages
the Twisted network library for task management. For the purposes of our example parcel
we are going to use a PeriodicTask instance to regularly call the refresh()
method in all instances of class FeedChannel. In channels.py file add the following class:
class UpdateTask(PeriodicTask):
"""
This class implements a periodic task that checks and reads new feeds
on 30 minutes interval.
"""
# target is periodic task
def getTarget(self):
return self
# target is already constructed as self
def __call__(self, periodicTask):
return self
# create a view called Feeds and keep it around 500 items in size
def fork(self):
return fork_item(self, name="Feeds", pruneSize=500, notify=False)
# target implementation
def run(self):
"""
This method implements a periodic task for updating feeds.
"""
# update our view
view = self.itsView
view.refresh(notify=False)
# call refresh on all followed feed channels
for channel in FeedChannel.iterItems(view):
channel.refresh()
# return true to keep the timer running
return True
Here the UpdateTask implements the basic interface of a
PeriodicTask: the run() method is called on a regular interval.
A repository view is an interface to the objects in the Repository. As mentioned above, it is sort of a connection to the Repository. However a view is also a way to accumulate a set of changes to the Repository without worrying about changes interfering from other code.
All changes made to the Repository are made through a view. Each view maintains its own
set of changes that are written back to the central Repository when the commit()
method is called. Changes that have been committed from other views are brought into the current
view by using the refresh() method.
The Repository is tightly coupled with the Python language. All repository object
instances have an implicit itsView attribute that refers to the view where
changes are made. When an object’s attributes are read or written, the changes
happen through the object’s itsView view. This is in contrast to other
common patterns found in other object databases, such as passing a “view”
parameter along as a parameter to any read/write methods.
Chandler’s user interface runs on one thread, and all repository operations
on this "UI Thread" share the same view. The UpdateTask may be running
on a background thread, and it would be unfortunate if a user changed something in the user
interface at the same time that same data is being updated in a background thread. Chandler
will ensure that if this task is running on a separate thread then it also gets its own view.
In this case the view is never shared outside of the current class. This allows the feed updating
code make a series of updates without worrying about changes that may happen in another view. When
all the updates have happened, then the results are committed back to the Repository.
The UpdateTask needs to find all instances of class FeedChannel
and call their refresh() method. The method iterItems() is a class method
on all classes that are inherited from class Item. It finds all instances of a given class
that are in Repository. The view parameter to method iterItems() is necessary because classes
themselves don’t exist in any specific view.
The last step in writing a periodic task is to make it persistent using repository.
For this purpose PeriodicTask class is used to persist information about
how and when a task should be run. We’ll simply create an instance of PeriodicTask
in installParcel() in feeds/__init__.py as we did with the blocks earlier.
from channels import UpdateTask
def installParcel(parcel, oldName=None):
"""
This function installs the RSS feed parcel.
"""
from application import schema
from osaf import startup
from datetime import timedelta
# start a periodic task
UpdateTask.update(parcel, "updateTask",
run_at_startup=True,
interval=timedelta(minutes=30))
# load our blocks parcel, too
schema.synchronize(parcel.itsView, "feeds.blocks")
During startup Chandler uses PeriodicTask.iterItems() method to find all instances of
PeriodicTask class, just like our UpdateTask does to find all instances
of FeedChannel class. It will then register each task with Twisted, which will handle
the instantiation and running of our task.
In this final chapter we are going to implement the real application logic, which has been secondary
from the objectives of this tutorial but which is essential to fill our channels with real news. We are
going to use a Python feedparser library that is available at
http://www.feedparser.org/. The feed download will occur in
FeedChannel’s refresh() method, which first calls download()
method and then depending on the success of the download either feedFetchSuccess(info, callback=None)
or feedFetchFailed(failure, callback=None) methods. We’ll need to first download the
feed information, and then create a new FeedItem objects for any items that are not already in
the FeedChannel. Without further delay let’s take a look at the example source code
that should be placed in feeds/channels.py file.
.
.
.
def date_parse(s):
"""Parse using dateutil’s parse, then convert to ICUtzinfo timezones."""
return convertToICUtzinfo(dateutil_parse(s))
def setAttribute(self, data, attr, newattr=None):
"""
This function sets a given attribute overriding the name with newattr.
"""
if not newattr:
newattr = attr
value = data.get(attr)
if value:
type = self.getAttributeAspect(newattr, "type", None)
if type is not None:
value = type.makeValue(value)
setattr(self, newattr, value)
def setAttributes(self, data, attributes):
"""
This function sets a group, which can be a dictionary or a list, of attributes.
"""
if isinstance(attributes, dict):
for attr, newattr in attributes.iteritems():
setAttribute(self, data, attr, newattr=newattr)
elif isinstance(attributes, list):
for attr in attributes:
setAttribute(self, data, attr)
class ConditionalHTTPClientFactory(client.HTTPClientFactory):
"""
This class implements HTTP network access services for retrieving RSS feeds.
"""
def __init__(self, url, lastModified=None, etag=None, method="GET",
postdata=None, headers=None, agent="Chandler", timeout=0,
cookies=None, followRedirect=1):
"""
This method initializes a HTTP conduict.
"""
# optimize our server access by using the "Last-Modified" and
# "ETag" fields of the HTTP request header.
if lastModified or etag:
if headers is None:
headers = { }
if lastModified:
headers["if-modified-since"] = lastModified
if etag:
headers["if-none-match"] = etag
# initialize a HTTP conduict...
client.HTTPClientFactory.__init__(self, url, method=method,
postdata=postdata, headers=headers, agent=agent, timeout=timeout,
cookies=cookies, followRedirect=followRedirect)
# ... and set a callback handler for failures.
self.deferred.addCallback(
lambda data: (data, self.status, self.response_headers)
)
def noPage(self, reason):
"""
This method implements a callback for situations when an RSS feed
could not be retrieved.
"""
if self.status == "304":
client.HTTPClientFactory.page(self, "")
else:
client.HTTPClientFactory.noPage(self, reason)
class FeedChannel(pim.ListCollection):
"""
This class implements a feed channel collection that is visualized
in the sidebar.
"""
#
# FeedChannel repository interface
#
link = schema.One(schema.URL)
category = schema.One(schema.Text)
author = schema.One(schema.Text)
date = schema.One(schema.DateTime)
url = schema.One(schema.URL)
etag = schema.One(schema.Text)
lastModified = schema.One(schema.DateTime)
copyright = schema.One(schema.Text)
language = schema.One(schema.Text)
ignoreContentChanges = schema.One(schema.Boolean, initialValue=False)
isEstablished = schema.One(schema.Boolean, initialValue=False)
isPreviousUpdateSuccessful = schema.One(schema.Boolean, initialValue=True)
logItem = schema.One(initialValue=None)
schema.addClouds(
sharing = schema.Cloud(
literal = [author, copyright, link, url]
)
)
feedparser = None
def __setup__(self, *args, **kw):
self.addIndex("link", "value", attribute="link")
def refresh(self, callback=None):
"""
This method updates a feed channel content.
"""
# Make sure we have the feedsView copy of the channel item
feedsView = self.itsView
feedsView.refresh(notify=False)
item = feedsView.findUUID(self.itsUUID)
return item.download().addCallback(item.feedFetchSuccess, callback).addErrback(item.feedFetchFailed, callback)
def download(self):
"""
This method uses a HTTP conduict to download an RSS channel feed.
"""
url = str(self.url)
etag = str(getattr(self, "etag", None))
lastModified = getattr(self, "lastModified", None)
if lastModified:
lastModified = lastModified.strftime("%a, %d %b %Y %H:%M:%S %Z")
(scheme, host, port, path) = client._parse(url)
scheme = str(scheme)
host = str(host)
path = str(path)
factory = ConditionalHTTPClientFactory(url=url,
lastModified=lastModified, etag=etag, timeout=60)
reactor.connectTCP(host, port, factory)
return factory.deferred
def feedFetchSuccess(self, info, callback=None):
"""
This method implements a callback for succesful RSS feed downloads.
"""
(data, status, headers) = info
# getattr returns a unicode object which needs to be converted to
# bytes for logging
channel = getattr(self, "displayName", None)
if channel is None:
channel = str(self.url)
else:
channel = channel.encode("ascii", "replace")
if not data:
# Page hasn"t changed (304)
logger.info("Channel has not changed: %s" % channel)
return FETCH_NOCHANGE
logger.info("Channel downloaded: %s" % channel)
# set etag
etag = headers.get("etag", None)
if etag:
self.etag = etag[0]
# set lastModified
lastModified = headers.get("last-modified", None)
if lastModified:
self.lastModified = date_parse(lastModified[0])
count = self.parse(data)
if count:
logger.info("...added %d FeedItems" % count)
self.isEstablished = True
self.isPreviousUpdateSuccessful = True
self.logItem = None
self.itsView.commit()
if callback:
callback(self.itsUUID, True)
return FETCH_UPDATED
def feedFetchFailed(self, failure, callback=None):
"""
This method implements a callback for failed RSS feed downloads.
"""
# getattr returns a unicode object which needs to be converted to
# bytes for logging
channel = getattr(self, "displayName", None)
if channel is None:
channel = str(self.url)
else:
channel = channel.encode("ascii", "replace")
logger.error("Failed to update channel: %s; Reason: %s",
channel, failure.getErrorMessage())
if self.isEstablished:
if self.isPreviousUpdateSuccessful:
self.isPreviousUpdateSuccessful = False
item = FeedItem(itsView=self.itsView)
item.displayName = _(u"Feed channel is unreachable")
item.author = _(u"Chandler Feeds Parcel")
item.category = _(u"Internal")
item.date = datetime.now(ICUtzinfo.default)
item.content = view.createLob(_(u"This feed channel is currently unreachable"))
self.addFeedItem(item)
self.logItem = item
self.itsView.commit()
else:
if self.logItem:
self.logItem.content = view.createLob(u"This feed channel has been unreachable from " + unicode(formatTime(self.logItem.date)) + u" to " + unicode(formatTime(datetime.now(ICUtzinfo.default))))
self.itsView.commit()
if callback:
callback(self.itsUUID, False)
return FETCH_FAILED
def parse(self, rawData):
"""
This method uses an external library method to parse the RSS feed content
and then fills the channel attributes.
"""
if self.feedparser is None:
import feedparser
FeedChannel.feedparser = feedparser
data = self.feedparser.parse(rawData)
# For fun, keep the latest copy of the feed inside the channel item
self.body = unicode(rawData, "utf-8")
return self.fillAttributes(data)
def fillAttributes(self, data):
"""
"""
# Map some external attribute names to internal attribute names:
attrs = {"title":"displayName", "description":"body"}
setAttributes(self, data["channel"], attrs)
# These attribute names don"t need remapping:
attrs = ["link", "copyright", "category", "language"]
setAttributes(self, data["channel"], attrs)
date = data["channel"].get("date")
if date:
self.date = date_parse(str(date))
# parse feed items.
return self._parseItems(data["items"])
def addFeedItem(self, feedItem):
"""
Add a single item, and add it to any listening collections.
"""
feedItem.channel = self
self.add(feedItem)
def _parseItems(self, items):
"""
This method parses all the news items in the RSS feed.
"""
view = self.itsView
count = 0
for newItem in items:
# Convert date to datetime object
if getattr(newItem, "date_parsed", None):
try:
# date_parsed is a tuple of 9 integers, like gmtime( )
# returns...
# date_parsed seems to always be converted to GMT, so
# let's make a datetime object using values from
# date_parsed, coupled with a GMT tzinfo...
kwds = dict(tzinfo=ICUtzinfo.getInstance('UTC'))
itemDate = datetime(*newItem.date_parsed[:5], **kwds)
# logger.debug("%s, %s, %s" % \
# (newItem.date, newItem.date_parsed, itemDate))
newItem.date = itemDate
except:
logger.exception("Could not get date: %s (%s)" % \
(newItem.date, newItem.date_parsed))
newItem.date = None
# Get the item content, using the "content" attribute first,
# falling back to what"s in"description"
content = newItem.get("content")
if content:
content = content[0]["value"]
else:
content = newItem.get("description")
title = newItem.get("title")
matchingItem = None
link = getattr(newItem, "link", None)
if link:
# Find all FeedItems that have this link
matchingItem = indexes.valueLookup(self, "link", "link", link)
# If no matching items (based on link), it"s new
# If matching item, if title or description have changed,
# update the item and mark it unread
if matchingItem is None:
feedItem = FeedItem(itsView=view)
feedItem.refresh(newItem)
self.addFeedItem(feedItem)
logger.debug("Added new item: %s", title)
count += 1
else:
# A FeedItem exists within this Channel that has the
# same link. @@@MOR For now I am only going to allow one
# FeedItem at a time (per Channel) to link to the same place,
# since it seems like that gets the behavior we want.
oldTitle = matchingItem.displayName
titleDifferent = (oldTitle != title)
# If no date in the item, just consider it a matching date;
# otherwise do compare datestamps:
dateDifferent = False
haveFeedDate = "date" in newItem
if haveFeedDate:
if matchingItem.date != newItem.date:
dateDifferent = True
if not self.ignoreContentChanges:
oldContent = matchingItem.content.getReader().read()
contentDifferent = (oldContent != content)
else:
contentDifferent = False
if contentDifferent or titleDifferent or dateDifferent:
matchingItem.refresh(newItem)
if matchingItem.read:
matchingItem.updated = True
matchingItem.read = False
msg = "Updated item: %s (content %s, title %s, date %s)"
logger.debug(msg, title, contentDifferent, titleDifferent,
dateDifferent)
return count
def markAllItemsRead(self):
"""
This method marks all items in this feed channel as read.
"""
for item in self:
item.read = True
@schema.observer(author)
def onAuthorChange(self, op, attr):
self.updateDisplayWho(op, attr)
def addDisplayWhos(self, whos):
super(FeedChannel, self).addDisplayWhos(whos)
author = getattr(self, 'author', None)
if author is not None:
whos.append((10, author, 'author'))
class FeedItem(pim.ContentItem):
"""
This class implements a feed channel item that is visualized
in the summary and detail views.
"""
#
# FeedItem repository interface
#
link = schema.One(schema.URL, initialValue=None)
category = schema.One(schema.Text, indexed=True)
author = schema.One(schema.Text, indexed=True)
date = schema.One(schema.DateTime)
channel = schema.One(FeedChannel)
content = schema.One(schema.Lob, indexed=True)
updated = schema.One(schema.Boolean)
@apply
def body():
def fget(self):
return self.content
def fset(self, value):
self.content = value
return property(fget, fset)
schema.addClouds(
sharing = schema.Cloud(
literal = [link, category, author, date]
)
)
schema.initialValues(
displayName = lambda self: _(u"No Title")
)
def _compareLink(self, other):
"""
This method compares two feed items.
"""
return cmp(str(self.link).lower(), str(other.link).lower())
def refresh(self, data):
"""
This method updates a feed item content.
"""
# fill in the item
attrs = {"title":"displayName"}
setAttributes(self, data, attrs)
attrs = ["link", "category", "author"]
# @@@MOR attrs = ["creator", "link", "category"]
setAttributes(self, data, attrs)
content = data.get("content")
# Use the "content" info first, falling back to what"s in "description"
if content:
content = content[0]["value"]
else:
content = data.get("description")
if content:
self.content = self.getAttributeAspect("content", "type").makeValue(content, indexed=True)
if "date" in data:
self.date = date_parse(str(data.date))
else:
# No date was available in the feed, so assign it "now"
self.date = datetime.now(ICUtzinfo.default)
@schema.observer(author)
def onAuthorChange(self, op, attr):
self.updateDisplayWho(op, attr)
def addDisplayWhos(self, whos):
super(FeedItem, self).addDisplayWhos(whos)
author = getattr(self, 'author', None)
if author is not None:
whos.append((10, author, 'author'))
In this tutorial, you have learned how to:
The next step is to spend some time developing your own data types and application behavior. This tutorial will get you some basic functionality for your data but Chandler’s true potential comes to light when you begin to explore some of its more advanced capabilities. When items have been properly defined, many of these capabilities “just work” for your new class.
# Copyright (c) 2003-2007 Open Source Applications Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from channels import UpdateTask
def installParcel(parcel, oldName=None):
"""
This function installs the RSS feed parcel.
"""
from application import schema
from osaf import startup
from datetime import timedelta
# start a periodic task
UpdateTask.update(parcel, "updateTask",
run_at_startup=True,
interval=timedelta(minutes=30))
# load our blocks parcel, too
schema.synchronize(parcel.itsView, "feeds.blocks")
# Copyright (c) 2003-2007 Open Source Applications Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging, wx
from osaf.views import detail
from application import schema, dialogs
import osaf.framework.blocks.Block as Block
from channels import FeedChannel, FeedItem
from i18n import MessageFactory
from osaf import messages
from osaf.pim.structs import SizeType, RectType
_ = MessageFactory("Chandler-FeedsPlugin")
logger = logging.getLogger(__name__)
class FeedItemDetail(detail.HTMLDetailArea):
"""
This class implements a block for visualizing feed item content.
"""
def getHTMLText(self, item):
"""
This method renders the feed item content as HTML.
"""
# check that we have a valid feed item.
if item == item.itsView:
return
if item is not None:
displayName = getattr(item, 'displayName', None)
if displayName is None:
displayName =u"<" + messages.UNTITLED + u">"
# make the html
HTMLText = u"<html><body>\n\n"
link = getattr(item, "link", None)
if link:
HTMLText = HTMLText + u"<a href=\"%s\">" % (link)
HTMLText = HTMLText + u"<h5>%s</h5>" % (displayName)
if link:
HTMLText = HTMLText + u"</a>\n"
content = getattr(item, "content", None)
if content:
content = content.getReader().read()
else:
content = displayName
#desc = desc.replace("<", "<").replace(">", ">")
HTMLText = HTMLText + u"<p>" + content + u"</p>\n\n"
if link:
HTMLText = HTMLText + u"<br><a href=\"" + unicode(item.link) + u"\">" + _(u"more...") + u"</a>"
HTMLText = HTMLText + "</body></html>\n"
return HTMLText
class AddFeedCollectionEvent(Block.AddToSidebarEvent):
"""
This class implements an event for adding a new collection to the sidebar.
"""
def onNewItem (self):
"""
This method is invoked when the user clicks on "New Feed Channel"
menu item.
"""
def calledInMainThread(channelUUID, success):
"""
This callback is invoked once the feed has been processed. If all
is okay, success will be True, otherwise False.
"""
self.itsView.refresh(notify=False)
channel = self.itsView.findUUID(channelUUID)
if not channel.isEstablished and not success:
# request a new URL from the user.
url = dialogs.Util.promptUser(
_(u"The provided URL seems to be invalid"),
_(u"Enter a URL for the RSS Channel"),
defaultValue = unicode(channel.url))
url = str(url)
if url != None:
try:
# try to recreate the channel...
channel.displayName = url
channel.url = channel.getAttributeAspect("url", "type").makeValue(url)
channel.isEstablished = False
channel.isPreviousUpdateSuccessful = True
channel.logItem = None
channel.itsView.commit()
# ... and then try to update its contents.
channel.refresh(callback=calledInTwisted)
except:
# unable to recreate the feed channel.
wx.MessageBox (_(u"Could not create channel for %(url)s\nCheck the URL and try again.") % {"url": url},
_(u"New Channel Error"),
parent=wx.GetApp().mainFrame)
def calledInTwisted(channelUUID, success):
"""
This callback is what we really pass to twisted, and it will
invoke the calledInMainThread method in the main thread.
"""
wx.GetApp().PostAsyncEvent(calledInMainThread, channelUUID, success)
# get a URL from the user, ...
url = dialogs.Util.promptUser(
_(u"New Channel"),
_(u"Enter a URL for the RSS Channel"),
defaultValue = u"http://")
if url == None:
return None
url = str(url)
# ... and then try to create a new channel.
try:
# create a new feed channel...
self.itsView.refresh(notify=False)
channel = FeedChannel(itsView=self.itsView)
channel.displayName = url
channel.url = channel.getAttributeAspect("url", "type").makeValue(url)
self.itsView.commit()
# ... and then update its contents.
channel.refresh(callback=calledInTwisted)
except:
# unable to create a new feed channel.
wx.MessageBox (_(u"Could not create channel for %(url)s\nCheck the URL and try again.") % {"url": url},
_(u"New Channel Error"),
parent=wx.GetApp().mainFrame)
return None
# return succesfully
return channel
def installParcel(parcel, oldVersion=None):
"""
This function defines the feed parcel detail view layout.
"""
detail = schema.ns("osaf.views.detail", parcel)
blocks = schema.ns("osaf.framework.blocks", parcel)
main = schema.ns("osaf.views.main", parcel)
feeds = schema.ns("feeds", parcel)
# Create an AddFeedCollectionEvent that adds an RSS collection to the sidebar.
addFeedCollectionEvent = AddFeedCollectionEvent.update(
parcel, "addFeedCollectionEvent",
blockName = "addFeedCollectionEvent")
# Add a separator to the "Collection" menu ...
feedsMenu = blocks.Menu.update(parcel, "feedsDemoMenu",
blockName = "feedsDemoMenu",
title = _(u'Feeds'),
helpString = _(u'RSS reader'),
childrenBlocks = [ ],
parentBlock = main.ExperimentalMenu)
# ... and, below it, a menu item to subscribe to a RSS feed.
blocks.MenuItem.update(parcel, "newFeedChannel",
blockName = "newFeedChannel",
title = _(u"&Create new feed channel..."),
event = addFeedCollectionEvent,
eventsForNamedLookup = [addFeedCollectionEvent],
parentBlock = feedsMenu,
)
# The hierarchy of UI elements for the FeedItem detail view
feedItemRootBlocks = [
# The markup bar
detail.MarkupBar,
detail.makeSpacer(parcel, height=6, position=0.01).install(parcel),
# Author area
detail.makeArea(parcel, "AuthorArea",
position=0.19,
childBlocks = [
detail.makeLabel(parcel, _(u"author"), borderTop=2),
detail.makeSpacer(parcel, width=8),
#field("AuthorAttribute", title=u"author"),
detail.makeEditor(parcel, "author",
viewAttribute=u"author",
border=RectType(0,2,2,2),
readOnly=True),
]
).install(parcel),
# Category
detail.makeArea(parcel, "CategoryArea",
position=0.2,
childBlocks = [
detail.makeLabel(parcel, _(u"category"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "category",
viewAttribute=u"category",
border=RectType(0,2,2,2),
readOnly=True),
]
).install(parcel),
# URL
detail.makeArea(parcel, "LinkArea",
position=0.3,
childBlocks = [
detail.makeLabel(parcel, _(u"link"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "link",
viewAttribute=u"link",
border=RectType(0,2,2,2),
readOnly=True),
],
).install(parcel),
# Date area
detail.makeArea(parcel, "DateArea",
position=0.4,
childBlocks = [
detail.makeLabel(parcel, _(u"date"), borderTop=2),
detail.makeSpacer(parcel, width=8),
detail.makeEditor(parcel, "date",
viewAttribute=u"date",
border=RectType(0,2,2,2),
readOnly=True,
stretchFactor=0.0,
size=SizeType(90, -1)),
],
).install(parcel),
detail.makeSpacer(parcel, height=7, position=0.8999).install(parcel),
FeedItemDetail.update(parcel, "ItemBodyArea",
position=0.9,
blockName="articletext",
size=SizeType(100,50),
minimumSize=SizeType(100,50),
),
]
# The BranchSubtree ties the blocks to our FeedItem"s Kind.
detail.makeSubtree(parcel, FeedItem, feedItemRootBlocks)
# Copyright (c) 2003-2007 Open Source Applications Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
__all__ = ["FeedChannel", "FeedItem"]
__parcel__ = "feeds"
import time, logging, urllib
from datetime import datetime
from PyICU import ICUtzinfo
from osaf.pim.calendar.TimeZone import convertToICUtzinfo, formatTime
from dateutil.parser import parse as dateutil_parse
from application import schema
from util import indexes
from osaf import pim
from i18n import MessageFactory
from twisted.web import client
from twisted.internet import reactor
from osaf.startup import PeriodicTask, fork_item
_ = MessageFactory("Chandler-FeedsPlugin")
logger = logging.getLogger(__name__)
FETCH_FAILED = 0
FETCH_NOCHANGE = 1
FETCH_UPDATED = 2
def date_parse(s):
"""Parse using dateutil's parse, then convert to ICUtzinfo timezones."""
return convertToICUtzinfo(dateutil_parse(s))
class UpdateTask(PeriodicTask):
"""
This class implements a periodic task that checks and reads new feeds
on 30 minutes interval.
"""
# target is periodic task
def getTarget(self):
return self
# target is already constructed as self
def __call__(self, periodicTask):
return self
# create a view called Feeds and keep it around 500 items in size
def fork(self):
return fork_item(self, name="Feeds", pruneSize=500, notify=False)
# target implementation
def run(self):
"""
This method implements a periodic task for updating feeds.
"""
# update our view
view = self.itsView
view.refresh(notify=False)
# call refresh on all followed feed channels
for channel in FeedChannel.iterItems(view):
channel.refresh()
# return true to keep the timer running
return True
def setAttribute(self, data, attr, newattr=None):
"""
This function sets a given attribute overriding the name with newattr.
"""
if not newattr:
newattr = attr
value = data.get(attr)
if value:
type = self.getAttributeAspect(newattr, "type", None)
if type is not None:
value = type.makeValue(value)
setattr(self, newattr, value)
def setAttributes(self, data, attributes):
"""
This function sets a group, which can be a dictionary or a list, of attributes.
"""
if isinstance(attributes, dict):
for attr, newattr in attributes.iteritems():
setAttribute(self, data, attr, newattr=newattr)
elif isinstance(attributes, list):
for attr in attributes:
setAttribute(self, data, attr)
class ConditionalHTTPClientFactory(client.HTTPClientFactory):
"""
This class implements HTTP network access services for retrieving RSS feeds.
"""
def __init__(self, url, lastModified=None, etag=None, method="GET",
postdata=None, headers=None, agent="Chandler", timeout=0,
cookies=None, followRedirect=1):
"""
This method initializes a HTTP conduict.
"""
# optimize our server access by using the "Last-Modified" and
# "ETag" fields of the HTTP request header.
if lastModified or etag:
if headers is None:
headers = { }
if lastModified:
headers["if-modified-since"] = lastModified
if etag:
headers["if-none-match"] = etag
# initialize a HTTP conduict...
client.HTTPClientFactory.__init__(self, url, method=method,
postdata=postdata, headers=headers, agent=agent, timeout=timeout,
cookies=cookies, followRedirect=followRedirect)
# ... and set a callback handler for failures.
self.deferred.addCallback(
lambda data: (data, self.status, self.response_headers)
)
def noPage(self, reason):
"""
This method implements a callback for situations when an RSS feed
could not be retrieved.
"""
if self.status == "304":
client.HTTPClientFactory.page(self, "")
else:
client.HTTPClientFactory.noPage(self, reason)
class FeedChannel(pim.ListCollection):
"""
This class implements a feed channel collection that is visualized
in the sidebar.
"""
#
# FeedChannel repository interface
#
link = schema.One(schema.URL)
category = schema.One(schema.Text)
author = schema.One(schema.Text)
date = schema.One(schema.DateTime)
url = schema.One(schema.URL)
etag = schema.One(schema.Text)
lastModified = schema.One(schema.DateTime)
copyright = schema.One(schema.Text)
language = schema.One(schema.Text)
ignoreContentChanges = schema.One(schema.Boolean, initialValue=False)
isEstablished = schema.One(schema.Boolean, initialValue=False)
isPreviousUpdateSuccessful = schema.One(schema.Boolean, initialValue=True)
logItem = schema.One(initialValue=None)
schema.addClouds(
sharing = schema.Cloud(
literal = [author, copyright, link, url]
)
)
feedparser = None
def __setup__(self, *args, **kw):
self.addIndex("link", "value", attribute="link")
def refresh(self, callback=None):
"""
This method updates a feed channel content.
"""
# Make sure we have the feedsView copy of the channel item
feedsView = self.itsView
feedsView.refresh(notify=False)
item = feedsView.findUUID(self.itsUUID)
return item.download().addCallback(item.feedFetchSuccess, callback).addErrback(item.feedFetchFailed, callback)
def download(self):
"""
This method uses a HTTP conduict to download an RSS channel feed.
"""
url = str(self.url)
etag = str(getattr(self, "etag", None))
lastModified = getattr(self, "lastModified", None)
if lastModified:
lastModified = lastModified.strftime("%a, %d %b %Y %H:%M:%S %Z")
(scheme, host, port, path) = client._parse(url)
scheme = str(scheme)
host = str(host)
path = str(path)
factory = ConditionalHTTPClientFactory(url=url,
lastModified=lastModified, etag=etag, timeout=60)
reactor.connectTCP(host, port, factory)
return factory.deferred
def feedFetchSuccess(self, info, callback=None):
"""
This method implements a callback for succesful RSS feed downloads.
"""
(data, status, headers) = info
# getattr returns a unicode object which needs to be converted to
# bytes for logging
channel = getattr(self, "displayName", None)
if channel is None:
channel = str(self.url)
else:
channel = channel.encode("ascii", "replace")
if not data:
# Page hasn"t changed (304)
logger.info("Channel has not changed: %s" % channel)
return FETCH_NOCHANGE
logger.info("Channel downloaded: %s" % channel)
# set etag
etag = headers.get("etag", None)
if etag:
self.etag = etag[0]
# set lastModified
lastModified = headers.get("last-modified", None)
if lastModified:
self.lastModified = date_parse(lastModified[0])
count = self.parse(data)
if count:
logger.info("...added %d FeedItems" % count)
self.isEstablished = True
self.isPreviousUpdateSuccessful = True
self.logItem = None
self.itsView.commit()
if callback:
callback(self.itsUUID, True)
return FETCH_UPDATED
def feedFetchFailed(self, failure, callback=None):
"""
This method implements a callback for failed RSS feed downloads.
"""
# getattr returns a unicode object which needs to be converted to
# bytes for logging
channel = getattr(self, "displayName", None)
if channel is None:
channel = str(self.url)
else:
channel = channel.encode("ascii", "replace")
logger.error("Failed to update channel: %s; Reason: %s",
channel, failure.getErrorMessage())
if self.isEstablished:
if self.isPreviousUpdateSuccessful:
self.isPreviousUpdateSuccessful = False
item = FeedItem(itsView=self.itsView)
item.displayName = _(u"Feed channel is unreachable")
item.author = _(u"Chandler Feeds Parcel")
item.category = _(u"Internal")
item.date = datetime.now(ICUtzinfo.default)
item.content = view.createLob(_(u"This feed channel is currently unreachable"))
self.addFeedItem(item)
self.logItem = item
self.itsView.commit()
else:
if self.logItem:
self.logItem.content = view.createLob(u"This feed channel has been unreachable from " + unicode(formatTime(self.logItem.date)) + u" to " + unicode(formatTime(datetime.now(ICUtzinfo.default))))
self.itsView.commit()
if callback:
callback(self.itsUUID, False)
return FETCH_FAILED
def parse(self, rawData):
"""
This method uses an external library method to parse the RSS feed content
and then fills the channel attributes.
"""
if self.feedparser is None:
import feedparser
FeedChannel.feedparser = feedparser
data = self.feedparser.parse(rawData)
# For fun, keep the latest copy of the feed inside the channel item
self.body = unicode(rawData, "utf-8")
return self.fillAttributes(data)
def fillAttributes(self, data):
"""
"""
# Map some external attribute names to internal attribute names:
attrs = {"title":"displayName", "description":"body"}
setAttributes(self, data["channel"], attrs)
# These attribute names don"t need remapping:
attrs = ["link", "copyright", "category", "language"]
setAttributes(self, data["channel"], attrs)
date = data["channel"].get("date")
if date:
self.date = date_parse(str(date))
# parse feed items.
return self._parseItems(data["items"])
def addFeedItem(self, feedItem):
"""
Add a single item, and add it to any listening collections.
"""
feedItem.channel = self
self.add(feedItem)
def _parseItems(self, items):
"""
This method parses all the news items in the RSS feed.
"""
view = self.itsView
count = 0
for newItem in items:
# Convert date to datetime object
if getattr(newItem, "date_parsed", None):
try:
# date_parsed is a tuple of 9 integers, like gmtime( )
# returns...
# date_parsed seems to always be converted to GMT, so
# let's make a datetime object using values from
# date_parsed, coupled with a GMT tzinfo...
kwds = dict(tzinfo=ICUtzinfo.getInstance('UTC'))
itemDate = datetime(*newItem.date_parsed[:5], **kwds)
# logger.debug("%s, %s, %s" % \
# (newItem.date, newItem.date_parsed, itemDate))
newItem.date = itemDate
except:
logger.exception("Could not get date: %s (%s)" % \
(newItem.date, newItem.date_parsed))
newItem.date = None
# Get the item content, using the "content" attribute first,
# falling back to what"s in"description"
content = newItem.get("content")
if content:
content = content[0]["value"]
else:
content = newItem.get("description")
title = newItem.get("title")
matchingItem = None
link = getattr(newItem, "link", None)
if link:
# Find all FeedItems that have this link
matchingItem = indexes.valueLookup(self, "link", "link", link)
# If no matching items (based on link), it"s new
# If matching item, if title or description have changed,
# update the item and mark it unread
if matchingItem is None:
feedItem = FeedItem(itsView=view)
feedItem.refresh(newItem)
self.addFeedItem(feedItem)
logger.debug("Added new item: %s", title)
count += 1
else:
# A FeedItem exists within this Channel that has the
# same link. @@@MOR For now I am only going to allow one
# FeedItem at a time (per Channel) to link to the same place,
# since it seems like that gets the behavior we want.
oldTitle = matchingItem.displayName
titleDifferent = (oldTitle != title)
# If no date in the item, just consider it a matching date;
# otherwise do compare datestamps:
dateDifferent = False
haveFeedDate = "date" in newItem
if haveFeedDate:
if matchingItem.date != newItem.date:
dateDifferent = True
if not self.ignoreContentChanges:
oldContent = matchingItem.content.getReader().read()
contentDifferent = (oldContent != content)
else:
contentDifferent = False
if contentDifferent or titleDifferent or dateDifferent:
matchingItem.refresh(newItem)
if matchingItem.read:
matchingItem.updated = True
matchingItem.read = False
msg = "Updated item: %s (content %s, title %s, date %s)"
logger.debug(msg, title, contentDifferent, titleDifferent,
dateDifferent)
return count
def markAllItemsRead(self):
"""
This method marks all items in this feed channel as read.
"""
for item in self:
item.read = True
@schema.observer(author)
def onAuthorChange(self, op, attr):
self.updateDisplayWho(op, attr)
def addDisplayWhos(self, whos):
super(FeedChannel, self).addDisplayWhos(whos)
author = getattr(self, 'author', None)
if author is not None:
whos.append((10, author, 'author'))
class FeedItem(pim.ContentItem):
"""
This class implements a feed channel item that is visualized
in the summary and detail views.
"""
#
# FeedItem repository interface
#
link = schema.One(schema.URL, initialValue=None)
category = schema.One(schema.Text, indexed=True)
author = schema.One(schema.Text, indexed=True)
date = schema.One(schema.DateTime)
channel = schema.One(FeedChannel)
content = schema.One(schema.Lob, indexed=True)
updated = schema.One(schema.Boolean)
@apply
def body():
def fget(self):
return self.content
def fset(self, value):
self.content = value
return property(fget, fset)
schema.addClouds(
sharing = schema.Cloud(
literal = [link, category, author, date]
)
)
schema.initialValues(
displayName = lambda self: _(u"No Title")
)
def _compareLink(self, other):
"""
This method compares two feed items.
"""
return cmp(str(self.link).lower(), str(other.link).lower())
def refresh(self, data):
"""
This method updates a feed item content.
"""
# fill in the item
attrs = {"title":"displayName"}
setAttributes(self, data, attrs)
attrs = ["link", "category", "author"]
# @@@MOR attrs = ["creator", "link", "category"]
setAttributes(self, data, attrs)
content = data.get("content")
# Use the "content" info first, falling back to what"s in "description"
if content:
content = content[0]["value"]
else:
content = data.get("description")
if content:
self.content = self.getAttributeAspect("content", "type").makeValue(content, indexed=True)
if "date" in data:
self.date = date_parse(str(data.date))
else:
# No date was available in the feed, so assign it "now"
self.date = datetime.now(ICUtzinfo.default)
@schema.observer(author)
def onAuthorChange(self, op, attr):
self.updateDisplayWho(op, attr)
def addDisplayWhos(self, whos):
super(FeedItem, self).addDisplayWhos(whos)
author = getattr(self, 'author', None)
if author is not None:
whos.append((10, author, 'author'))