API Documentation

Providers

class micawber.providers.Provider(endpoint, **kwargs)

The Provider object is responsible for retrieving metadata about a given URL. It implements a method called request(), which takes a URL and any parameters, which it sends off to an endpoint. The endpoint should return a JSON dictionary containing metadata about the resource, which is returned to the caller.

Parameters:

• endpoint – the API endpoint which should return information about requested links
• kwargs – any additional url parameters to send to the endpoint on each request, used for providing defaults. An example use-case might be for providing an API key on each request.

request(url, **extra_params)

Retrieve information about the given url. By default, will make a HTTP GET request to the endpoint. The url will be sent to the endpoint, along with any parameters specified in the extra_params and those parameters specified when the class was instantiated.

Will raise a ProviderException in the event the URL is not accessible or the API times out.

Parameters:

• url – URL to retrieve metadata for
• extra_params – additional parameters to pass to the endpoint, for example a maxwidth or an API key.

Return type:

a dictionary of JSON data

class micawber.providers.ProviderRegistry([cache=None])

A registry for encapsulating a group of Provider instances. It has optional caching support.

Handles matching regular expressions to providers. URLs are sent to the registry via its request() method, it checks to see if it has a provider that matches the URL, and if so, requests the metadata from the provider instance.

Parameters:cache – the cache simply needs to implement two methods, .get(key) and .set(key, value).

register(regex, provider)

Register the provider with the following regex.

Example:

registry = ProviderRegistry()
registry.register(
    'http://\S*.youtu(\.be|be\.com)/watch\S*',
    Provider('http://www.youtube.com/oembed'),
)

Parameters:

• regex – a regex for matching URLs of a given type
• provider – a Provider instance

request(url, **extra_params)

Retrieve information about the given url if it matches a regex in the instance’s registry. If no provider matches the URL, a ProviderException is thrown, otherwise the URL and parameters are dispatched to the matching provider’s Provider.request() method.

If a cache was specified, the resulting metadata will be cached.

Parameters:

• url – URL to retrieve metadata for
• extra_params – additional parameters to pass to the endpoint, for example a maxwidth or an API key.

Return type:

a dictionary of JSON data

micawber.providers.bootstrap_basic([cache=None])

Create a ProviderRegistry and register some basic providers, including youtube, flickr, vimeo.

Parameters:cache – an object that implements simple get and set Return type:a ProviderRegistry with a handful of providers registered

micawber.providers.bootstrap_embedly([cache=None[, **kwargs]])

Create a ProviderRegistry and register as many providers as are supported by embed.ly. Valid services are fetched from http://api.embed.ly/1/services/python and parsed then registered.

Parameters:

• cache – an object that implements simple get and set
• kwargs – any default keyword arguments to use with providers, useful for specifying your API key

Return type:

a ProviderRegistry with support for embed.ly

# if you have an API key, you can specify that here
pr = bootstrap_embedly(key='my-embedly-key')
pr.request('http://www.youtube.com/watch?v=54XHDUOHuzU')

micawber.providers.bootstrap_noembed([cache=None[, **kwargs]])

Create a ProviderRegistry and register as many providers as are supported by noembed.com. Valid services are fetched from http://noembed.com/providers and parsed then registered.

Parameters:

• cache – an object that implements simple get and set
• kwargs – any default keyword arguments to use with providers, useful for passing the nowrap option to noembed.

Return type:

a ProviderRegistry with support for noembed

# if you have an API key, you can specify that here
pr = bootstrap_noembed(nowrap=1)
pr.request('http://www.youtube.com/watch?v=54XHDUOHuzU')

Parsers

Functions for parsing text and HTML

micawber.parsers.parse_text_full(text, providers[, urlize_all=True[, handler=full_handler[, **params]]])

Parse a block of text, converting all links by passing them to the given handler. Links contained within a block of text (i.e. not on their own line) will be handled as well.

Example input and output:

IN: 'this is a pic http://example.com/some-pic/'
OUT: 'this is a pic <a href="http://example.com/some-pic/"><img src="http://example.com/media/some-pic.jpg" /></a>'

Parameters:

• text – a string to parse
• providers – a ProviderRegistry instance
• urlize_all – whether to convert all urls irrespective of whether a provider exists
• handler – function to use to convert metadata back into a string representation
• params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

micawber.parsers.parse_text(text, providers[, urlize_all=True[, handler=full_handler[, block_handler=inline_handler[, **params]]]])

Very similar to the above parse_text_full() except URLs on their own line are rendered using the given handler, whereas URLs within blocks of text are passed to the block_handler. The default behavior renders full content for URLs on their own line (e.g. a flash player), whereas URLs within text are rendered simply as links so as not to disrupt the flow of text.

Parameters:

• text – a string to parse
• providers – a ProviderRegistry instance
• urlize_all – whether to convert all urls irrespective of whether a provider exists
• handler – function to use to convert links found on their own line
• block_handler – function to use to convert links found within blocks of text
• params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

micawber.parsers.parse_html(html, providers[, urlize_all=True[, handler=full_handler[, block_handler=inline_handler[, **params]]]])

Parse HTML intelligently, rendering items on their own within block elements as full content (e.g. a flash player), whereas URLs within text are passed to the block_handler which by default will render a simple link. Also worth noting is that URLs that are already enclosed within a <a> tag are skipped over.

Note

requires BeautifulSoup or beautifulsoup4

Parameters:

• html – a string of HTML to parse
• providers – a ProviderRegistry instance
• urlize_all – whether to convert all urls irrespective of whether a provider exists
• handler – function to use to convert links found on their own within a block element
• block_handler – function to use to convert links found within blocks of text
• params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

Functions for extracting rich content from text and HTML

micawber.parsers.extract(text, providers, **params)

Extract all URLs from a block of text, and additionally get any metadata for URLs we have providers for.

Parameters:

• text – a string to parse
• providers – a ProviderRegistry instance
• params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

Return type:

returns a 2-tuple containing a list of all URLs and a dictionary keyed by URL containing any metadata. If a provider was not found for a URL it is not listed in the dictionary.

micawber.parsers.extract_html(html, providers, **params)

Extract all URLs from an HTML string, and additionally get any metadata for URLs we have providers for. Same as extract() but for HTML.

Note

URLs within <a> tags will not be included.

Parameters:

• html – a string to parse
• providers – a ProviderRegistry instance
• params – any additional parameters to use when requesting metadata, i.e. a maxwidth or maxheight.

Return type:

returns a 2-tuple containing a list of all URLs and a dictionary keyed by URL containing any metadata. If a provider was not found for a URL it is not listed in the dictionary.

Cache

class micawber.cache.Cache

A reference implementation for the cache interface used by the ProviderRegistry.

get(key)

Retrieve the key from the cache or None if not present

set(key, value)

Set the cache key key to the given value.

class micawber.cache.PickleCache([filename='cache.db'])

A cache that uses pickle to store data.

Note

To use this cache class be sure to call load() when initializing your cache and save() before your app terminates to persist cached data.

load()

Load the pickled data into memory

save()

Store the internal cache to an external file

class micawber.cache.RedisCache([namespace='micawber'[, **conn]])

A cache that uses Redis to store data

Note

requires the redis-py library, pip install redis

Parameters:

• namespace – prefix for cache keys
• conn – keyword arguments to pass when initializing redis connection