pyMicrodata Package

pyMicrodata Package

This module implements the microdata->RDF algorithm, as documented by the U{W3C Semantic Web Interest Group Note<http://www.w3.org/TR/2012/NOTE-microdata-rdf-20120308/>}.

The module can be used via a stand-alone script (an example is part of the distribution) or bound to a CGI script as a Web Service. An example CGI script is also added to the distribution. Both the local script and the distribution may have to be adapted to local circumstances.

(Simple) Usage

From a Python file, expecting a Turtle output::

from pyMicrodata import pyMicrodata print pyMicrodata().rdf_from_source(‘filename’)

Other output formats are also possible. E.g., to produce RDF/XML output, one could use::

from pyMicrodata import pyMicrodata print pyMicrodata().rdf_from_source(‘filename’, outputFormat=’pretty-xml’)

It is also possible to embed an RDFa processing. Eg, using::

from pyMicrodata import pyMicrodata graph = pyMicrodata().graph_from_source(‘filename’)

returns an RDFLib.Graph object instead of a serialization thereof. See the the description of the L{pyMicrodata class<pyMicrodata.pyMicrodata>} for further possible entry points details.

There is also, as part of this module, a L{separate entry for CGI calls<processURI>}.

Return formats

By default, the output format for the graph is RDF/XML. At present, the following formats are also available (with the corresponding key to be used in the package entry points):

• “xml”: U{RDF/XML<http://www.w3.org/TR/rdf-syntax-grammar/>}
• “turtle”: U{Turtle<http://www.w3.org/TR/turtle/>} (default)
• “nt”: U{N-triple<http://www.w3.org/TR/rdf-testcases/#ntriples>}
• “json”: U{JSON-LD<http://json-ld.org/spec/latest/json-ld-syntax/>}

@summary: Microdata parser (distiller) @requires: Python version 2.5 or up @requires: U{RDFLib<http://rdflib.net>} @requires: U{html5lib<http://code.google.com/p/html5lib/>} for the HTML5 parsing; note possible dependecies on Python’s version on the project’s web site @organization: U{World Wide Web Consortium<http://www.w3.org>} @author: U{Ivan Herman<http://www.w3.org/People/Ivan/>} @license: This software is available for use under the U{W3C® SOFTWARE NOTICE AND LICENSE<href=”http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231“>} @copyright: W3C

exception rdflib.plugins.parsers.pyMicrodata.HTTPError(http_msg, http_code)

Bases: rdflib.plugins.parsers.pyMicrodata.MicrodataError

Raised when HTTP problems are detected. It does not add any new functionality to the Exception class.

__init__(http_msg, http_code)

__module__ = 'rdflib.plugins.parsers.pyMicrodata'

exception rdflib.plugins.parsers.pyMicrodata.MicrodataError(msg)

Bases: exceptions.Exception

Superclass exceptions representing error conditions defined by the RDFa 1.1 specification. It does not add any new functionality to the Exception class.

__init__(msg)

__module__ = 'rdflib.plugins.parsers.pyMicrodata'

__weakref__

list of weak references to the object (if defined)

rdflib.plugins.parsers.pyMicrodata.processURI(uri, outputFormat, form)

The standard processing of a microdata uri options in a form, ie, as an entry point from a CGI call.

The call accepts extra form options (eg, HTTP GET options) as follows:

@param uri: URI to access. Note that the “text:” and “uploaded:” values are treated separately; the former is for textual intput (in which case a StringIO is used to get the data) and the latter is for uploaded file, where the form gives access to the file directly. @param outputFormat: serialization formats, as understood by RDFLib. Note that though “turtle” is a possible parameter value, some versions of the RDFLib turtle generation does funny (though legal) things with namespaces, defining unusual and unwanted prefixes... @param form: extra call options (from the CGI call) to set up the local options (if any) @type form: cgi FieldStorage instance @return: serialized graph @rtype: string

class rdflib.plugins.parsers.pyMicrodata.pyMicrodata(base='', vocab_expansion=False, vocab_cache=True)

Main processing class for the distiller @ivar base: the base value for processing @ivar http_status: HTTP Status, to be returned when the package is used via a CGI entry. Initially set to 200, may be modified by exception handlers

__init__(base='', vocab_expansion=False, vocab_cache=True)

@keyword base: URI for the default “base” value (usually the URI of the file to be processed) @keyword vocab_expansion: whether vocab expansion should be performed or not @type vocab_expansion: Boolean @keyword vocab_cache: if vocabulary expansion is done, then perform caching of the vocabulary data @type vocab_cache: Boolean

__module__ = 'rdflib.plugins.parsers.pyMicrodata'

graph_from_DOM(dom, graph=None)

Extract the RDF Graph from a DOM tree. @param dom: a DOM Node element, the top level entry node for the whole tree (to make it clear, a dom.documentElement is used to initiate processing) @keyword graph: an RDF Graph (if None, than a new one is created) @type graph: rdflib Graph instance. If None, a new one is created. @return: an RDF Graph @rtype: rdflib Graph instance

graph_from_source(name, graph=None, rdfOutput=False)

Extract an RDF graph from an microdata source. The source is parsed, the RDF extracted, and the RDF Graph is returned. This is a front-end to the L{pyMicrodata.graph_from_DOM} method.

@param name: a URI, a file name, or a file-like object @return: an RDF Graph @rtype: rdflib Graph instance

rdf_from_source(name, outputFormat='pretty-xml', rdfOutput=False)

Extract and RDF graph from an RDFa source and serialize it in one graph. The source is parsed, the RDF extracted, and serialization is done in the specified format. @param name: a URI, a file name, or a file-like object @keyword outputFormat: serialization format. Can be one of “turtle”, “n3”, “xml”, “pretty-xml”, “nt”. “xml” and “pretty-xml”, as well as “turtle” and “n3” are synonyms. @return: a serialized RDF Graph @rtype: string

rdf_from_sources(names, outputFormat='pretty-xml', rdfOutput=False)

Extract and RDF graph from a list of RDFa sources and serialize them in one graph. The sources are parsed, the RDF extracted, and serialization is done in the specified format. @param names: list of sources, each can be a URI, a file name, or a file-like object @keyword outputFormat: serialization format. Can be one of “turtle”, “n3”, “xml”, “pretty-xml”, “nt”. “xml” and “pretty-xml”, as well as “turtle” and “n3” are synonyms. @return: a serialized RDF Graph @rtype: string

microdata Module

The core of the Microdata->RDF conversion, a more or less verbatim implementation of the U{W3C IG Note<http://www.w3.org/TR/microdata-rdf/>}. Because the implementation was also used to check the note itself, it tries to be fairly close to the text.

@organization: U{World Wide Web Consortium<http://www.w3.org>} @author: U{Ivan Herman<a href=”http://www.w3.org/People/Ivan/“>} @license: This software is available for use under the U{W3C® SOFTWARE NOTICE AND LICENSE<href=”http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231“>}

class rdflib.plugins.parsers.pyMicrodata.microdata.Evaluation_Context

Evaluation context structure. See Section 4.1 of the U{W3C IG Note<http://www.w3.org/TR/microdata-rdf/>}for the details.

@ivar current_type : an absolute URL for the current type, used when an item does not contain an item type @ivar memory: mapping from items to RDF subjects @type memory: dictionary @ivar current_name: an absolute URL for the in-scope name, used for generating URIs for properties of items without an item type @ivar current_vocabulary: an absolute URL for the current vocabulary, from the registry

__init__()

__module__ = 'rdflib.plugins.parsers.pyMicrodata.microdata'

__str__()

get_memory(item)

Get the memory content (ie, RDF subject) for ‘item’, or None if not stored yet @param item: an ‘item’, in microdata terminology @type item: DOM Element Node @return: None, or an RDF Subject (URIRef or BNode)

new_copy(itype)

During the generation algorithm a new copy of the current context has to be done with a new current type.

At the moment, the content of memory is copied, ie, a fresh dictionary is created and the content copied over. Not clear whether that is necessary, though, maybe a simple reference is enough... @param itype : an absolute URL for the current type @return: a new evaluation context instance

set_memory(item, subject)

Set the memory content, ie, the subject, for ‘item’. @param item: an ‘item’, in microdata terminology @type item: DOM Element Node @param subject: RDF Subject @type subject: URIRef or Blank Node

class rdflib.plugins.parsers.pyMicrodata.microdata.Microdata(document, base=None)

This class encapsulates methods that are defined by the U{microdata spec<http://dev.w3.org/html5/md/Overview.html>}, as opposed to the RDF conversion note.

@ivar document: top of the DOM tree, as returned by the HTML5 parser @ivar base: the base URI of the Dom tree, either set from the outside or via a @base element

__init__(document, base=None)

@param document: top of the DOM tree, as returned by the HTML5 parser @param base: the base URI of the Dom tree, either set from the outside or via a @base element

__module__ = 'rdflib.plugins.parsers.pyMicrodata.microdata'

getElementById(id)

This is a method defined for DOM 2 HTML, but the HTML5 parser does not seem to define it. Oh well... @param id: value of an @id attribute to look for @return: array of nodes whose @id attribute matches C{id} (formally, there should be only one...)

get_item_properties(item)

Collect the item’s properties, ie, all DOM descendent nodes with @itemprop until the subtree hits another @itemscope. @itemrefs are also added at this point.

@param item: current item @type item: DOM Node @return: array of items, ie, DOM Nodes

get_top_level_items()

A top level item is and element that has the @itemscope set, but no @itemtype. They have to be collected in pre-order and depth-first fashion.

@return: list of items (ie, DOM Nodes)

class rdflib.plugins.parsers.pyMicrodata.microdata.MicrodataConversion(document, graph, base=None, vocab_expansion=False, vocab_cache=True)

Bases: rdflib.plugins.parsers.pyMicrodata.microdata.Microdata

Top level class encapsulating the conversion algorithms as described in the W3C note.

@ivar graph: an RDF graph; an RDFLib Graph @type graph: RDFLib Graph @ivar document: top of the DOM tree, as returned by the HTML5 parser @ivar ns_md: the Namespace for the microdata vocabulary @ivar base: the base of the Dom tree, either set from the outside or via a @base element

__init__(document, graph, base=None, vocab_expansion=False, vocab_cache=True)

@param graph: an RDF graph; an RDFLib Graph @type graph: RDFLib Graph @param document: top of the DOM tree, as returned by the HTML5 parser @keyword base: the base of the Dom tree, either set from the outside or via a @base element @keyword vocab_expansion: whether vocab expansion should be performed or not @type vocab_expansion: Boolean @keyword vocab_cache: if vocabulary expansion is done, then perform caching of the vocabulary data @type vocab_cache: Boolean

__module__ = 'rdflib.plugins.parsers.pyMicrodata.microdata'

convert()

Top level entry to convert and generate all the triples. It finds the top level items, and generates triples for each of them; additionally, it generates a top level entry point to the items from base in the form of an RDF list.

generate_predicate_URI(name, context)

Generate a full URI for a predicate, using the type, the vocabulary, etc.

For details of this entry, see Section 4.4 @param name: name of the property, ie, what appears in @itemprop @param context: an instance of an evaluation context @type context: L{Evaluation_Context}

generate_property_values(subject, predicate, objects, context)

Generate the property values for a specific subject and predicate. The context should specify whether the objects should be added in an RDF list or each triples individually.

@param subject: RDF subject @type subject: RDFLib Node (URIRef or blank node) @param predicate: RDF predicate @type predicate: RDFLib URIRef @param objects: RDF objects @type objects: list of RDFLib nodes (URIRefs, Blank Nodes, or literals) @param context: evaluation context @type context: L{Evaluation_Context}

generate_triples(item, context)

Generate the triples for a specific item. See the W3C Note for the details.

@param item: the DOM Node for the specific item @type item: DOM Node @param context: an instance of an evaluation context @type context: L{Evaluation_Context} @return: a URIRef or a BNode for the (RDF) subject

get_property_value(node, context)

Generate an RDF object, ie, the value of a property. Note that if this element contains an @itemscope, then a recursive call to L{MicrodataConversion.generate_triples} is done and the return value of that method (ie, the subject for the corresponding item) is return as an object.

Otherwise, either URIRefs are created for <a>, <img>, etc, elements, or a Literal; the latter gets a time-related type for the <time> element.

@param node: the DOM Node for which the property values should be generated @type node: DOM Node @param context: an instance of an evaluation context @type context: L{Evaluation_Context} @return: an RDF resource (URIRef, BNode, or Literal)

class rdflib.plugins.parsers.pyMicrodata.microdata.PropertySchemes

__module__ = 'rdflib.plugins.parsers.pyMicrodata.microdata'

contextual = 'contextual'

vocabulary = 'vocabulary'

class rdflib.plugins.parsers.pyMicrodata.microdata.ValueMethod

__module__ = 'rdflib.plugins.parsers.pyMicrodata.microdata'

list = 'list'

unordered = 'unordered'

registry Module

Hardcoded version of the current microdata->RDF registry. There is also a local registry to include some test cases. Finally, there is a local dictionary for prefix mapping for the registry items; these are the preferred prefixes for those vocabularies, and are used to make the output nicer.

@organization: U{World Wide Web Consortium<http://www.w3.org>} @author: U{Ivan Herman<a href=”http://www.w3.org/People/Ivan/“>} @license: This software is available for use under the U{W3C® SOFTWARE NOTICE AND LICENSE<href=”http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231“>}

utils Module

Various utilities for pyMicrodata

@organization: U{World Wide Web Consortium<http://www.w3.org>} @author: U{Ivan Herman<a href=”http://www.w3.org/People/Ivan/“>} @license: This software is available for use under the U{W3C® SOFTWARE NOTICE AND LICENSE<href=”http://www.w3.org/Consortium/Legal/2002/copyright-software-20021231“>}

class rdflib.plugins.parsers.pyMicrodata.utils.URIOpener(name)

A wrapper around the urllib2 method to open a resource. Beyond accessing the data itself, the class sets the content location. The class also adds an accept header to the outgoing request, namely text/html and application/xhtml+xml (unless set explicitly by the caller).

@ivar data: the real data, ie, a file-like object @ivar headers: the return headers as sent back by the server @ivar location: the real location of the data (ie, after possible redirection and content negotiation)

CONTENT_LOCATION = 'Content-Location'

__init__(name)

@param name: URL to be opened @keyword additional_headers: additional HTTP request headers to be added to the call

__module__ = 'rdflib.plugins.parsers.pyMicrodata.utils'

rdflib.plugins.parsers.pyMicrodata.utils.fragment_escape(name)

rdflib.plugins.parsers.pyMicrodata.utils.generate_RDF_collection(graph, vals)

Generate an RDF List from vals, returns the head of the list @param graph: RDF graph @type graph: RDFLib Graph @param vals: array of RDF Resources @return: head of the List (an RDF Resource)

rdflib.plugins.parsers.pyMicrodata.utils.generate_URI(base, v)

Generate an (absolute) URI; if val is a fragment, then using it with base, otherwise just return the value @param base: Absolute URI for base @param v: relative or absolute URI

rdflib.plugins.parsers.pyMicrodata.utils.get_Literal(Pnode)

Get (recursively) the full text from a DOM Node.

@param Pnode: DOM Node @return: string

rdflib.plugins.parsers.pyMicrodata.utils.get_lang(node)

rdflib.plugins.parsers.pyMicrodata.utils.get_lang_from_hierarchy(document, node)

rdflib.plugins.parsers.pyMicrodata.utils.get_time_type(string)

Check whether the string abides to one of the accepted time related datatypes, and returns that one if yes @param string: the attribute value to be checked @return : a datatype URI or None

rdflib.plugins.parsers.pyMicrodata.utils.is_absolute_URI(uri)