User Documentation

Documentation that is useful for everybody.

Contents:

Getting Started (Command Line)

The settings file

Field definitions

Search syntax

Search templates and search_aliases

Moving Data between Collections

lesana derivatives

Front-ends

Getting Started (Command Line)

lesana can be used from the command line through the lesana command; for more details run lesana help.

Many commands will try to open some file in an editor: they will attempt to use, in this order, $EDITOR, sensible-editor or as a fallback vi, which should be installed on any POSIX-like system.

To start a new collection, create a directory and run lesana init into it:

mkdir $DIRECTORY
cd $DIRECTORY
lesana init

It will create the basic file structure of a lesana collection, including a settings.yaml skeleton and it will initialize a git repository (use --no-git to skip this part and ignore all further git commands).

It will then open settings.yaml in an editor: fill in your values for the available variables, and define your list of fields; see The settings file for details. Then save and exit, and you are now ready to commit the configuration for your new collection, as the changes have already been added to git:

git commit -m 'Collection settings'

An empty collection is not very interesting: let us start adding new entries:

lesana new

It will again open an editor on a skeleton of entry where you can fill in the values. When you close the editor it will print the entry id, that you can use e.g. to edit again the same entry:

lesana edit $ENTRY_ID

After you’ve added a few entries, you can now search for some word that you entered in one of the indexed fields:

lesana search some words

this will also print the entry ids of matching items, so that you can open them with lesana edit. See Search syntax for more details on the search syntax.

If you’re using git, entries will be autoadded to the staging area, but you need to commit them, so that you can choose how often you do so.

Search results are limited by default to 12 matches; to get all results for your query you can use the option --all. This is especially useful when passing the results to a template:

lesana search --template templates/my_report.html --all \
    <some search terms> \
    > some_search_terms-report.html

will generate an html file based on the jinja2 template templates/my_report.html with all the entries found for those search terms.

If later on you want to clone the repository elsewhere (using regular git commands) you can use git init in the new repository to install the hooks to manage updating the local cache every time the repository is updated via git, and then run lesana index to prepare the first version of the cache.

The settings file

The file settings.yaml defines the properties of a collection.

It is a yaml file with a dict of properties and their values.

name:

the human readable name of the collection.

lang:

the language of the collection; valid values are listed in the xapian stemmer [https://xapian.org/docs/apidoc/html/classXapian_1_1Stem.html] documentation and are usually either the English name or the two letter ISO639 code of a language.

entry_label:

a jinja2 template used to show an entry in the interface; beside the entry fields two useful variables are eid for the full entry ID and short_id for the short version.

default_sort:

a list of field names (possibly prefixed by + or -) that are used by default to sort results of searches in the collection. The fields must be marked as sortable in their definition, see below.

search_aliases:

a dict of <name>: <search snippet> which can be used in a query template. For more details see Search templates and search_aliases

fields:

The list of fields used by the collection, as described below.

Field definitions

name:

a name for the field (computer readable: keeping it lowercase alphabetic ascii is probably safer).

type:

the type of the field: valid fields are listed in lesana.types module (see the name property for each field)

index:

whether this field should be indexed: valid values are free for fields that should be available in the free text search and field for fields that should only be available by specifying the field name in the search.

sortable:

boolean; whether this field is sortable. Sortable fields enable sorting the results and search by ranges, but having too many sortable fields make the search more resurce intensive.

help:

a description for the field; this is e.g. added to new entries as a comment.

default:

the default value to use when creating an entry.

prefix:

the optional term prefix used inside xapian: if you don’t know what this means you should avoid using this, otherwise see Term Prefixes [https://xapian.org/docs/omega/termprefixes.html] on the xapian documentation for details.

Some field types may add other custom properties.

list properties

list:

the type of the entries in the list; note that neither lists of non uniform values nor lists of lists are supported (if you need those you can use the yaml generic type, or write your own derivative with an additional type).

integer properties

auto:

automatic manipulation of the field contents.

The value of increment will autoincrement the value at every update.

The reference command-line client will run this update before editing an entry, allowing further changes from the user; a command line user can then decide to abort this change through the usual git commands.

Other clients may decide to use a different workflow.

increment:

the amount by which an auto: increment field is incremented (negative values are of course allowed). Default is 1.

decimal properties

precision:

if this property is set, every value in this field will get rounded to the given number of decimals.

With this property it is possible to store decimal values as YAML floats instead of strings.

date and datetime properties

auto:

automatic manipulation of the field contents.

The following values are supported.

creation

autofill the field at creation time with the current UTC time (datetime) or local zone day (date).

update

autofill the field when it is updated with the current UTC time (datetime) or local zone day (date).

The reference command line client will run this update before editing an entry, allowing further changes from the user; a command line user can then decide to abort this change through the usual git commands.

Other clients may decide to use a different workflow.

values

The string, text, list and numeric types can have a property value with a list of valid values for that field.

An empty value is always allowed.

For the list type, each element of the list is checked, not the whole list.

Search syntax

Searches in lesana use the human readable query string format defined by xapian.

The simplest search is just a list of terms: e.g. searching for thing object will find entries where either thing or object is present in one of the fields with free indexing.

It is also possible to specify that a term must be in one specific field: the syntax for this is the name of the field follwed by : and the term, e.g. name:object will search for entries with the term object in the name field.

Search queries can of course include the usual logical operators AND, OR and NOT.

More modifiers are available; see the Query Parser [https://getting-started-with-xapian.readthedocs.io/en/latest/concepts/search/queryparser.html] documentation from xapian for details.

Search templates and search_aliases

In some contexts, search queries are rendered as jinja2 templates with the contents of the search_aliases property as set in settings.yaml.

The values of those search aliases should be valid search snippets with the syntax documented above; it’s usually a good idea to wrap them in parenthesis, so that they are easier to use in complex queries; e.g.:

my_alias: '(name:object OR name:thing)'

can correctly be used in a query like:

{{ my_alias }} AND description:shiny

Moving Data between Collections

Entries can be exported from a lesana collection to another using the lesana export command and a jinja2 template.

The template should generate a yaml file that is a valid lesana entry for the destination collection and it can use the fields of the starting collection as variables. The variable entry is also available and gives complete access to the entry of the original collection, so fields with names that aren’t valid jinja templates can be accessed as entry.data[<field-name>].

E.g. to convert between a collection with fields name, short-desc, desc to a collection with fields name, description one could use the following template:

name: {{ name }}
description: |
    {{ entry.data.[short-desc] }}

    {{ desc | indent(width=4, first=False) }}

From the origin collection you can then run the command:

lesana export <path/to/destination/collection> <path/to/template>

to export all entries.

You can also export just a subset of entries by adding a xapian query with the parameter --query; you can test the search using:

lesana search --all <some search terms>

and then actually run the export with:

lesana search --query '<some search terms>' <destination> <template>

note that in this second command the spaces in the search query have to be protected from the shell.

lesana derivatives

Front-ends

Collector

Collector [https://git.sr.ht/~fabrixxm/Collector] is a Gtk3 app to manage collection inventories throught yaml files, which also works on GNU/Linux mobile devices.

linkopedia

linkopedia [https://git.sr.ht/~fabrixxm/linkopedia] is a read-only web interface to Lesana collections.

Developer Documentation

Documentation that is useful for developers who are using lesana as a library.

Contents:

Promises

Semantic versioning

Collection format stability

Disposable cache

Promises

Semantic versioning

This project uses semver [http://semver.org/].

Collection format stability

Future versions of lesana will be able to read collections written by older versions.

Older versions in the same mayor release will also be able to work concurrently on the same repository.

If in the future a change of formats will be required, conversions scripts will be written in a way that will make them as stable as possibile, and will have enought test data to keep them maintained for the time being.

Disposable cache

Contrary to the yaml files, the xapian cache is considered disposable: from time to time there may be a need to delete the cache and reindex everything, either because of an upgrade or to perform repository mainteinance.

Of course, effort will be made to reduce the need for this so that it only happens sporadically, but it will probably never completely disappear.

Contributor Documentation

Documentation that is useful for contributors of lesana.

Contents:

Release procedure

Release procedure

Check that the version number in setup.py and in docs/source/conf.py is correct.

Check that the changelog is up to date.

Tag the version you are preparing:

$ git tag -s v$VERSION
$ git push
$ git push --tags

for the tag content use something like:

Version $VERSION

* contents
* of the relevant
* changelog

Generate the distribution files:

$ python3 -m build

Upload

$ twine upload -s dist/*

Send the release announce to:

valhalla/lesana-announce@lists.sr.ht, ~valhalla/lesana-discuss@lists.sr.ht

Close the bugs marked as pending_release on https://todo.sr.ht/~valhalla/lesana.

Man Pages

Contents:

lesana-edit

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-export

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-index

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-init

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-new

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-rm

SYNOPSIS

DESCRIPTION

OPTIONS

lesana

SYNOPSIS

DESCRIPTION

OPTIONS

COMMANDS

TEXT EDITOR

lesana-search

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-get-values

SYNOPSIS

DESCRIPTION

OPTIONS

lesana-show

SYNOPSIS

DESCRIPTION

OPTIONS

TEMPLATES

lesana-edit

SYNOPSIS

lesana edit [–help] [–collection <collection>] [–no-git] <entry>

DESCRIPTION

Lesana edit will open an existing entry (specified by id or partial id) in an editor, so that it can be changed.

If the collection is configured to use git, after the editor has been closed, it will add the file to the git staging area, unless --no-git is given.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--no-git

Don’t add the new entry to git.

lesana-export

SYNOPSIS

lesana export [-h] [–collection COLLECTION] [–query QUERY]

destination template

DESCRIPTION

Lesana export converts entries from one lesana collection to another, using a jinja2 template.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--query QUERY, -q QUERY

Xapian query to search in the collection

lesana-index

SYNOPSIS

lesana index [–help] [–collection COLLECTION] [files [files …]]

DESCRIPTION

Lesana index adds some entries to the xapian cache, listed by filename (by default all of the files found in the items directory).

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--reset

Delete the existing xapian cache before indexing.

lesana-init

SYNOPSIS

lesana init [–help] [–collection <collection>] [–no-git]

DESCRIPTION

lesana init initializes a new lesana collection.

It will create the directory (if it does not exist) and, unless --no-git is specified it will initialize it as a git repository, create a .gitignore file with some relevant contents and add hooks to update the local cache when the files are changed via git.

It will then create a skeleton settings.yaml file and open it in an editor to start configuring the collection.

When leaving the editor, again unless --no-git is used, it will add this file to the git staging area, but not commit it.

It is safe to run this command on an existing repository, e.g. to install the hooks on a new clone, but it will overwrite the hooks themselves even if they have been changed by the user.

OPTIONS

--help, -h

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The directory where the collection will be initialized. Default is .

--no-git

Do not use git in the current collection.

lesana-new

SYNOPSIS

lesana new [–help] [–collection <collection>] [–no-git]

DESCRIPTION

Lesana new creates a new lesana entry.

It will create an empty entry and open an editor so that it can be filled.

If the collection is configured to use git, after the editor has been closed, it will add the file to the git staging area, unless --no-git is given.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--no-git

Don’t add the new entry to git.

lesana-rm

SYNOPSIS

lesana rm [-h] [–collection COLLECTION] entries [entries …]

DESCRIPTION

Lesana rm removes an entry from the collection, removing both the file and the cached entry.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

lesana

SYNOPSIS

lesana [–help] <command>

DESCRIPTION

lesana is a tool to organize collections of various kinds. It is designed to have a data storage / serialization format that is friendly to git and other VCSs, but decent performances.

To reach this aim it uses yaml [http://yaml.org/] as its serialization format, which is easy to store in a VCS, share between people and synchronize between different computers, but it also keeps an index of this data in a local xapian [https://xapian.org/] database in order to allow for fast searches.

lesana supports collections of any kind, as long as their entries can be described with a mostly flat dictionary of fields of the types described in the documentation file field_types.

Some example collection schemas are provided, but one big strength of lesana is the ability to customize your collection with custom fields either by simply writing a personalized settings.yaml.

OPTIONS

-h, --help

Prints an help message and exits.

COMMANDS

new(1)

Creates a new entry.

edit(1)

Edits an existing entry.

show(1)

Shows an existing entry.

index(1)

Index some entries in the xapian cache.

search(1)

Searches for entries in the xapian cache.

export(1)

Exports entries from one lesana collection to another

init(1)

Initialize a new lesana collection

rm(1)

Removes an entry.

TEXT EDITOR

Many lesana subcommands will try to open files in a text editor chosen as follows:

first, the value of $EDITOR is tried

then the command sensible-editor, as available under e.g. Debian and its derivatives

lastly, it will try to fallback to vi, which should be available under any posix system.

lesana-search

SYNOPSIS

lesana search [–help] [–collection COLLECTION] [–template TEMPLATE]

[–offset OFFSET] [–pagesize PAGESIZE] [–all] [–expand-query-template] [–sort FIELD1 [–sort FIELD2 …]] [query [query …]]

DESCRIPTION

Lesana search allows one to make searches in the collection and render the results.

The section Search syntax in the full documentation describes the query syntax in more detail; it is available online at https://lesana.trueelena.org/user/search.html or it may be installed on your system (e.g. in Debian and derivatives it will be at /usr/share/doc/lesana/html/user/search.html).

By default entries are printed according to the entry_label from the settings.yaml file, but they can be rendered according to a jinja2 template.

If no query is specified, it will default to '*', i.e. search all entries: thus lesana search --all will print all entries, while just lesana search will print the first 12 entries, possibly according to the relevant sorting options.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--template TEMPLATE, -t TEMPLATE

Template to use when displaying results

--offset OFFSET

.

--pagesize PAGESIZE

.

--all

Return all available results

--sort

Sort the results by a sortable field.

This option can be added multiple times; prefix the name of the field with - to reverse the results (e.g. --sort='-date').

--expand-query-template

Render search_aliases in the query as a jinja2 template

lesana-get-values

SYNOPSIS

lesana search [–help] [–collection COLLECTION] [–template TEMPLATE]

–field FIELD [query [query …]]

DESCRIPTION

Lesana get-values will list all values found in a field and the number of entries where that value has been found.

A template can be specified to format the results.

Extracting the values from a sortable field is significantly more efficient than doing so from a non-sortable field, but adding too many sortable fields can make general searches and indexing slower.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--template TEMPLATE, -t TEMPLATE

Template to use when displaying results

--field

Name of the desired field.

lesana-show

SYNOPSIS

lesana show [–help] [–collection COLLECTION] [–template TEMPLATE] <entry>

DESCRIPTION

lesana show will print an entry (specified by id or partial id) to stdout.

A template can be specified with --template <template> to pretty print entries.

OPTIONS

-h, --help

Prints an help message and exits.

--collection COLLECTION, -c COLLECTION

The collection to work on. Default is .

--template TEMPLATE, -t TEMPLATE

Use the specified template to display results.

TEMPLATES

The templates used by lesana show are jinja2 templates.

The entry fields are available as variables, and the full entry is available as the variable entry and can be used to give access to fields with names that aren’t valid jinja2 variables e.g. as entry.data[<field-name>].

lesana reference

lesana package

Subpackages

lesana.data package

Submodules

lesana.collection module

Collection

Entry

TemplatingError

lesana.command module

Command

Edit

Export

GetValues

Index

Init

Lesana

New

Remove

Search

Show

Update

lesana.templating module

Environment

to_yaml()

lesana.types module

LesanaBoolean

LesanaDate

LesanaDatetime

LesanaDecimal

LesanaFile

LesanaFloat

LesanaGeo

LesanaInt

LesanaList

LesanaString

LesanaText

LesanaTimestamp

LesanaType

LesanaURL

LesanaValueError

LesanaYAML

lesana package

Subpackages

lesana.data package

Submodules

lesana.collection module

Collection

Collection.PARSER_FLAGS

Collection.entries_from_short_eid()

Collection.entry_from_eid()

Collection.entry_from_rendered_template()

Collection.get_all_documents()

Collection.get_all_search_results()

Collection.get_field_values()

Collection.get_search_results()

Collection.get_template()

Collection.git_add_files()

Collection.indexed_fields

Collection.init()

Collection.remove_entries()

Collection.remove_file()

Collection.render_query_template()

Collection.save_entries()

Collection.start_search()

Collection.update_cache()

Collection.update_field()

Entry

Entry.auto()

Entry.empty_data()

Entry.get_data()

Entry.idterm

Entry.render()

Entry.short_id

Entry.validate()

Entry.yaml_data

TemplatingError

lesana.command module

Command

Command.add_arguments()

Edit

Edit.arguments

Edit.main()

Export

Export.arguments

Export.main()

GetValues

GetValues.arguments

GetValues.main()

GetValues.name

Index

Index.arguments

Index.main()

Init

Init.arguments

Init.main()

Lesana

Lesana.commands

New

New.arguments

New.main()

Remove

Remove.arguments

Remove.main()

Remove.name

Search

Search.arguments

Search.main()

Show

Show.arguments

Show.main()

Update

Update.arguments

Update.main()

lesana.templating module

Environment

to_yaml()

lesana.types module

LesanaBoolean

LesanaBoolean.empty()

LesanaBoolean.load()

LesanaBoolean.name

LesanaDate

LesanaDate.auto()

LesanaDate.empty()

LesanaDate.load()

LesanaDate.name

LesanaDatetime

LesanaDatetime.auto()

LesanaDatetime.empty()

LesanaDatetime.load()

LesanaDatetime.name

LesanaDecimal

LesanaDecimal.empty()

LesanaDecimal.load()

LesanaDecimal.name

LesanaFile

LesanaFile.name

LesanaFloat

LesanaFloat.empty()

LesanaFloat.load()

LesanaFloat.name

LesanaGeo

LesanaGeo.load()

LesanaGeo.name

LesanaInt

LesanaInt.auto()

LesanaInt.empty()

LesanaInt.load()

LesanaInt.name

LesanaList

LesanaList.empty()

LesanaList.index()

LesanaList.load()

LesanaList.name

LesanaString

LesanaString.empty()

LesanaString.load()

LesanaString.name

LesanaText

LesanaText.name

LesanaTimestamp

LesanaTimestamp.empty()

LesanaTimestamp.load()

LesanaTimestamp.name

LesanaType

LesanaType.allowed_value()

LesanaType.auto()

LesanaType.empty()

LesanaType.index()

LesanaType.load()

LesanaURL

LesanaURL.name

LesanaValueError

LesanaYAML

LesanaYAML.empty()

LesanaYAML.load()

LesanaYAML.name

lesana.data package

lesana.collection module

class lesana.collection.Collection(directory=None, itemdir='items')

Bases: object

PARSER_FLAGS = 23

entries_from_short_eid(seid)

entry_from_eid(eid)

entry_from_rendered_template(template, data)

get_all_documents()

Yield all documents in the collection.

Note that the results can’t be sorted, even if the collection has a default_sort; if you need sorted values you need to use a regular search with a query of ‘*’

get_all_search_results()

get_field_values(field, querystring='*')

get_search_results(offset=0, pagesize=12)

get_template(template_fname, searchpath='.')

git_add_files(files=[])

property indexed_fields

classmethod init(directory=None, git_enabled=True, edit_file=None, settings={})

Initialize a lesana repository

directory defaults to . if git_enabled is True, git support is enabled and if possible a git repository is initalized. edit_file is a syncronous function that runs on a filename (possibly opening the file in an editor) and should manage its own errors.

remove_entries(eids)

remove_file(fname)

render_query_template(query)

Render a query template, filling it with search_aliases.

save_entries(entries=[])

start_search(querystring, sort_by=None)

Prepare a search for querystring.

update_cache(fnames=None, reset=False)

Update the xapian db with the data in files.

fnames is a list of basenames of files in self.itemdir.

If no files have been passed, add everything.

if reset the existing xapian db is deleted before indexing

Return the number of files that have been added to the cache.

update_field(query, field, value)

class lesana.collection.Entry(collection, data={}, fname=None)

Bases: object

auto()

Update all fields of this entry, as required by the field settings.

This is called by the reference client before an edit, so that the user can make further changes.

Note that the stored file is not changed: if you need it you need to save the entry yourself.

empty_data()

get_data()

property idterm

render(template, searchpath='.')

property short_id

validate()

property yaml_data

exception lesana.collection.TemplatingError

Bases: Exception

Raised when there are errors rendering a jinja template

lesana.command module

class lesana.command.Command(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

add_arguments(parser: ArgumentParser)

Add argparse arguments to an existing parser.

Override this method to add arguments to a subcommand.

class lesana.command.Edit(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command, ExternalEditorMixin

Edit a lesana entry

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--no-git'], {'help': "Don't add the new entry to git", 'action': 'store_false', 'dest': 'git'}), (['eid'], {'help': 'eid of an entry to edit'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Export(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Export entries to a different collection

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--query', '-q'], {'help': 'Xapian query to search in the collection'}), (['destination'], {'help': 'The collection to export entries to'}), (['template'], {'help': 'Template to convert entries'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.GetValues(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

List all values for one field, with entry counts.

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--field', '-f'], {'help': 'Name of the field', 'required': True}), (['--template', '-t'], {'help': 'Template to use when displaying results'}), (['query'], {'help': 'Xapian query to limit the count search "                       + "in the collection', 'nargs': '*', 'default': '*'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

name: Optional[str] = 'get-values'

The name used to call this subcommand from the command line.

If this property is none, the default is the name of the class set to lowercase.

class lesana.command.Index(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Index entries in a lesana collection

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--reset'], {'action': 'store_true', 'help': 'Delete the existing index and reindex from scratch.'}), (['files'], {'help': 'List of files to index (default: everything)', 'default': None, 'nargs': '*'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Init(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command, ExternalEditorMixin

Initialize a lesana collection

arguments = [(['--collection', '-c'], {'help': 'The directory to work on (default .)', 'default': '.'}), (['--no-git'], {'help': 'Skip setting up git in this directory', 'action': 'store_false', 'dest': 'git'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Lesana

Bases: MainCommand

Manage collections

commands: Iterable['Command'] = (<lesana.command.New object>, <lesana.command.Edit object>, <lesana.command.Show object>, <lesana.command.Index object>, <lesana.command.Search object>, <lesana.command.GetValues object>, <lesana.command.Update object>, <lesana.command.Export object>, <lesana.command.Init object>, <lesana.command.Remove object>)

The subcommands: a tuple of Command subclasses.

class lesana.command.New(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command, ExternalEditorMixin

Create a new entry

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--no-git'], {'help': "Don't add the new entry to git", 'action': 'store_false', 'dest': 'git'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Remove(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Remove an entry from a collection

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['entries'], {'help': 'List of entries to remove', 'nargs': '+'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

name: Optional[str] = 'rm'

The name used to call this subcommand from the command line.

If this property is none, the default is the name of the class set to lowercase.

class lesana.command.Search(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Search for entries

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--template', '-t'], {'help': 'Template to use when displaying results'}), (['--offset'], {'type': <class 'int'>}), (['--pagesize'], {'type': <class 'int'>}), (['--all'], {'action': 'store_true', 'help': 'Return all available results'}), (['--sort'], {'action': 'append', 'help': 'Sort results by a sortable field'}), (['--expand-query-template', '-e'], {'action': 'store_true', 'help': 'Render search_aliases in the query as a jinja2 template'}), (['query'], {'help': 'Xapian query to search in the collection', 'nargs': '*', 'default': '*'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Show(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Show a lesana entry

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--template', '-t'], {'help': 'Use the specified template to display results.'}), (['eid'], {'help': 'eid of an entry to edit'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

class lesana.command.Update(collection_class=<class 'lesana.collection.Collection'>, entry_class=<class 'lesana.collection.Entry'>)

Bases: Command

Update a field in multiple entries

arguments = [(['--collection', '-c'], {'help': 'The collection to work on (default .)'}), (['--field', '-f'], {'help': 'The field to change'}), (['--value', '-t'], {'help': 'The value to set'}), (['query'], {'help': 'Xapian query to search in the collection', 'nargs': '+'})]

main()

Main code of this subcommand.

Override this method to implement the actual program.

lesana.templating module

Custom jinja2 filters and other templating helpers

class lesana.templating.Environment(*args, **kwargs)

Bases: Environment

A customized jinja2 environment that includes our filters.

lesana.templating.to_yaml(data)

Return the yaml representation of data.

lesana.types module

Type checkers for lesana fields.

Warning: this part of the code is still in flux and it may change significantly in a future release.

class lesana.types.LesanaBoolean(field, types, value_index=None)

Bases: LesanaType

A boolean value

empty()

load(data)

name = 'boolean'

class lesana.types.LesanaDate(field, types, value_index=None)

Bases: LesanaType

A date

auto(value)

Return an updated value.

If the field settings auto is update return the current date, otherwise the old value.

empty()

load(data)

name = 'date'

class lesana.types.LesanaDatetime(field, types, value_index=None)

Bases: LesanaType

A datetime

auto(value)

Return an updated value.

If the field settings auto is update return the current datetime, otherwise the old value.

empty()

load(data)

name = 'datetime'

class lesana.types.LesanaDecimal(field, types, value_index=None)

Bases: LesanaType

A fixed point number

Because of a limitation of the yaml format, these should be stored quoted as a string, to avoid being loaded back as floats.

Alternatively, the property precision can be used to force all values to be rounded to that number of decimals.

empty()

load(data)

name = 'decimal'

class lesana.types.LesanaFile(field, types, value_index=None)

Bases: LesanaString

A path to a local file.

Relative paths are assumed to be relative to the base lesana directory (i.e. where .lesana lives)

name = 'file'

class lesana.types.LesanaFloat(field, types, value_index=None)

Bases: LesanaType

A floating point number

empty()

load(data)

name = 'float'

class lesana.types.LesanaGeo(field, types, value_index=None)

Bases: LesanaString

A Geo URI

load(data)

name = 'geo'

class lesana.types.LesanaInt(field, types, value_index=None)

Bases: LesanaType

An integer number

auto(value)

Return an updated value.

If the field settings auto is increment return the value incremented by the value of the field setting increment (default 1).

empty()

load(data)

name = 'integer'

class lesana.types.LesanaList(field, types, value_index=None)

Bases: LesanaType

A list of other values

empty()

index(doc, indexer, value)

Index a value for this field type.

Override this for types that need any kind of special treatment to be indexed.

See LesanaList for an idea on how to do so.

load(data)

name = 'list'

class lesana.types.LesanaString(field, types, value_index=None)

Bases: LesanaType

A string of unicode text

empty()

load(data)

name = 'string'

class lesana.types.LesanaText(field, types, value_index=None)

Bases: LesanaString

A longer block of unicode text

name = 'text'

class lesana.types.LesanaTimestamp(field, types, value_index=None)

Bases: LesanaType

A unix timestamp, assumed to be UTC

empty()

load(data)

name = 'timestamp'

class lesana.types.LesanaType(field, types, value_index=None)

Bases: object

Base class for lesana field types.

allowed_value(value)

Check whether a value is allowed in this field.

Return the value itself or raise LesanaValueError if the value isn’t valid.

auto(value)

Return an updated value, as appropriate for the field.

Default is to return the value itself, but types can use their configuration to e.g. increment a numerical value or return the current date(time).

empty()

index(doc, indexer, value)

Index a value for this field type.

Override this for types that need any kind of special treatment to be indexed.

See LesanaList for an idea on how to do so.

load(data)

class lesana.types.LesanaURL(field, types, value_index=None)

Bases: LesanaString

An URL

name = 'url'

exception lesana.types.LesanaValueError

Bases: ValueError

Raised in case of validation errors.

class lesana.types.LesanaYAML(field, types, value_index=None)

Bases: LesanaType

Free YAML contents (no structure is enforced)

empty()

load(data)

name = 'yaml'

Python Module Index

l

| l |
- | lesana |
|     lesana.collection |
|     lesana.command |
|     lesana.data |
|     lesana.templating |
|     lesana.types |

Index

A | C | E | G | I | L | M | N | P | R | S | T | U | V | Y

A

add_arguments() (lesana.command.Command method)

allowed_value() (lesana.types.LesanaType method)

arguments (lesana.command.Edit attribute)

(lesana.command.Export attribute)

(lesana.command.GetValues attribute)

(lesana.command.Index attribute)

(lesana.command.Init attribute)

(lesana.command.New attribute)

(lesana.command.Remove attribute)

(lesana.command.Search attribute)

(lesana.command.Show attribute)

(lesana.command.Update attribute)

auto() (lesana.collection.Entry method)

(lesana.types.LesanaDate method)

(lesana.types.LesanaDatetime method)

(lesana.types.LesanaInt method)

(lesana.types.LesanaType method)

C

Collection (class in lesana.collection)

Command (class in lesana.command)

commands (lesana.command.Lesana attribute)

E

Edit (class in lesana.command)

empty() (lesana.types.LesanaBoolean method)

(lesana.types.LesanaDate method)

(lesana.types.LesanaDatetime method)

(lesana.types.LesanaDecimal method)

(lesana.types.LesanaFloat method)

(lesana.types.LesanaInt method)

(lesana.types.LesanaList method)

(lesana.types.LesanaString method)

(lesana.types.LesanaTimestamp method)

(lesana.types.LesanaType method)

(lesana.types.LesanaYAML method)

empty_data() (lesana.collection.Entry method)

entries_from_short_eid() (lesana.collection.Collection method)

Entry (class in lesana.collection)

entry_from_eid() (lesana.collection.Collection method)

entry_from_rendered_template() (lesana.collection.Collection method)

Environment (class in lesana.templating)

Export (class in lesana.command)

G

get_all_documents() (lesana.collection.Collection method)

get_all_search_results() (lesana.collection.Collection method)

get_data() (lesana.collection.Entry method)

get_field_values() (lesana.collection.Collection method)

get_search_results() (lesana.collection.Collection method)

get_template() (lesana.collection.Collection method)

GetValues (class in lesana.command)

git_add_files() (lesana.collection.Collection method)

I

idterm (lesana.collection.Entry property)

Index (class in lesana.command)

index() (lesana.types.LesanaList method)

(lesana.types.LesanaType method)

indexed_fields (lesana.collection.Collection property)

Init (class in lesana.command)

init() (lesana.collection.Collection class method)

L

lesana

module

Lesana (class in lesana.command)

lesana.collection

module

lesana.command

module

lesana.data

module

lesana.templating

module

lesana.types

module

LesanaBoolean (class in lesana.types)

LesanaDate (class in lesana.types)

LesanaDatetime (class in lesana.types)

LesanaDecimal (class in lesana.types)

LesanaFile (class in lesana.types)

LesanaFloat (class in lesana.types)

LesanaGeo (class in lesana.types)

LesanaInt (class in lesana.types)

LesanaList (class in lesana.types)

LesanaString (class in lesana.types)

LesanaText (class in lesana.types)

LesanaTimestamp (class in lesana.types)

LesanaType (class in lesana.types)

LesanaURL (class in lesana.types)

LesanaValueError

LesanaYAML (class in lesana.types)

load() (lesana.types.LesanaBoolean method)

(lesana.types.LesanaDate method)

(lesana.types.LesanaDatetime method)

(lesana.types.LesanaDecimal method)

(lesana.types.LesanaFloat method)

(lesana.types.LesanaGeo method)

(lesana.types.LesanaInt method)

(lesana.types.LesanaList method)

(lesana.types.LesanaString method)

(lesana.types.LesanaTimestamp method)

(lesana.types.LesanaType method)

(lesana.types.LesanaYAML method)

M

main() (lesana.command.Edit method)

(lesana.command.Export method)

(lesana.command.GetValues method)

(lesana.command.Index method)

(lesana.command.Init method)

(lesana.command.New method)

(lesana.command.Remove method)

(lesana.command.Search method)

(lesana.command.Show method)

(lesana.command.Update method)

module

lesana

lesana.collection

lesana.command

lesana.data

lesana.templating

lesana.types

N

name (lesana.command.GetValues attribute)

(lesana.command.Remove attribute)

(lesana.types.LesanaBoolean attribute)

(lesana.types.LesanaDate attribute)

(lesana.types.LesanaDatetime attribute)

(lesana.types.LesanaDecimal attribute)

(lesana.types.LesanaFile attribute)

(lesana.types.LesanaFloat attribute)

(lesana.types.LesanaGeo attribute)

(lesana.types.LesanaInt attribute)

(lesana.types.LesanaList attribute)

(lesana.types.LesanaString attribute)

(lesana.types.LesanaText attribute)

(lesana.types.LesanaTimestamp attribute)

(lesana.types.LesanaURL attribute)

(lesana.types.LesanaYAML attribute)

New (class in lesana.command)

P

PARSER_FLAGS (lesana.collection.Collection attribute)

R

Remove (class in lesana.command)

remove_entries() (lesana.collection.Collection method)

remove_file() (lesana.collection.Collection method)

render() (lesana.collection.Entry method)

render_query_template() (lesana.collection.Collection method)

S

save_entries() (lesana.collection.Collection method)

Search (class in lesana.command)

short_id (lesana.collection.Entry property)

Show (class in lesana.command)

start_search() (lesana.collection.Collection method)

T

TemplatingError

to_yaml() (in module lesana.templating)

U

Update (class in lesana.command)

update_cache() (lesana.collection.Collection method)

update_field() (lesana.collection.Collection method)

V

validate() (lesana.collection.Entry method)

Y

yaml_data (lesana.collection.Entry property)

Getting Started (Command Line)

lesana can be used from the command line through the lesana command; for more details run lesana help.

Many commands will try to open some file in an editor: they will attempt to use, in this order, $EDITOR, sensible-editor or as a fallback vi, which should be installed on any POSIX-like system.

To start a new collection, create a directory and run lesana init into it:

mkdir $DIRECTORY
cd $DIRECTORY
lesana init

It will create the basic file structure of a lesana collection, including a settings.yaml skeleton and it will initialize a git repository (use --no-git to skip this part and ignore all further git commands).

It will then open settings.yaml in an editor: fill in your list of fields and all other data, save and exit. You are now ready to commit the configuration for your new collection:

git commit -m 'Collection settings'

An empty collection is not very interesting: let us start adding new entries:

lesana new

It will again open an editor on a skeleton of entry where you can fill in the values. When you close the editor it will print the entry id, that you can use e.g. to edit again the same entry:

lesana edit $ENTRY_ID

After you’ve added a few entries, you can now search for some word that you entered in one of the indexed fields:

lesana search some words

this will also print the entry ids of matching items, so that you can open them with lesana edit.

If you’re using git, entries will be autoadded to the staging area, but you need to commit them, so that you can choose how often you do so.

Search results are limited by default to 12 matches; to get all results for your query you can use the option --all. This is especially useful when passing the results to a template:

lesana search --template templates/my_report.html --all \
    some search terms \
    > some_search_terms-report.html

will generate an html file based on the jinja2 template templates/my_report.html with all the entries found for those search terms.

Moving Data between Collections

Entries can be exported from a lesana collection to another using the lesana export command and a jinja2 template.

The template should generate a yaml file that is a valid lesana entry for the destination collection and it can use the fields of the starting collection as variables. The variable entry is also available and gives complete access to the entry of the original collection, so fields with names that aren’t valid jinja templates can be accessed as entry.data[<field-name>].

E.g. to convert between a collection with fields name, short-desc, desc to a collection with fields name, description one could use the following template:

name: {{ name }}
description: |
    {{ entry.data.[short-desc] }}

    {{ desc | indent(width=4, first=False) }}

From the origin collection you can then run the command:

lesana export <path/to/destination/collection> <path/to/template>

to export all entries.

You can also export just a subset of entries by adding a xapian query with the parameter --query; you can test the search using:

lesana search --all <some search terms>

and then actually run the export with:

lesana search --query '<some search terms>' <destination> <template>

note that in this second command the spaces in the search query have to be protected from the shell.

Promises

Semantic versioning

This project uses semver [http://semver.org/].

Collection format stability

Future versions of lesana will be able to read collections written by older versions.

Older versions in the same mayor release will also be able to work concurrently on the same repository.

If in the future a change of formats will be required, conversions scripts will be written in a way that will make them as stable as possibile, and will have enought test data to keep them maintained for the time being.

Disposable cache

Contrary to the yaml files, the xapian cache is considered disposable: from time to time there may be a need to delete the cache and reindex everything, either because of an upgrade or to perform repository mainteinance.

Of course, effort will be made to reduce the need for this so that it only happens sporadically, but it will probably never completely disappear.

All modules for which code is available

lesana.collection

lesana.command

lesana.templating

lesana.types

Source code for lesana.collection

import collections
import io
import logging
import os
import shutil
import uuid

import jinja2
import ruamel.yaml
import xapian
from pkg_resources import resource_filename, resource_string

from . import templating, types

logger = logging.getLogger(__name__)

try:
    import git

    git_available = True
except ImportError:
    git_available = False

[docs]class Entry(object):
    def __init__(self, collection, data={}, fname=None):
        self.collection = collection
        self.data = data or self.empty_data()
        self.fname = fname
        self.eid = self.data.get('eid', None)
        if not self.eid:
            if self.fname:
                self.eid, ext = os.path.splitext(os.path.basename(self.fname))
            else:
                self.eid = uuid.uuid4().hex
        if not self.fname:
            self.fname = self.eid + '.yaml'

    def __str__(self):
        label = self.collection.settings.get('entry_label', None)
        if label:
            t = jinja2.Template(label)
            return t.render(**self.get_data())
        else:
            return self.eid

[docs]    def get_data(self):
        d = self.data.copy()
        d['eid'] = self.eid
        d['fname'] = self.fname
        d['short_id'] = self.short_id
        return d

[docs]    def empty_data(self):
        data = self.collection.yaml.load("{}")
        for name, field in self.collection.fields.items():
            if field.field.get('default', None):
                data[name] = field.field['default']
            else:
                data[name] = field.empty()
            if field.field.get('help', None) is not None:
                comment = "{name} ({type}): {help}\n".format(**field.field)
                try:
                    data.yaml_set_comment_before_after_key(
                        key=name,
                        before=comment,
                        indent=0
                    )
                except AttributeError:
                    logger.warning(
                        "Not adding comments because they are not"
                        "supported by the yaml loader."
                    )
            valid_values = field.field.get('values', [])
            if valid_values:
                comment = "{name} ({type}): {valid_values}".format(
                    valid_values="|".join(valid_values),
                    **field.field
                )
                try:
                    data.yaml_set_comment_before_after_key(
                        key=name,
                        before=comment,
                        indent=0
                    )
                except AttributeError:
                    logger.warning(
                        "Not adding comments because they are not"
                        "supported by the yaml loader."
                    )
        return data

    @property
    def yaml_data(self):
        to_dump = self.data.copy()
        # Decimal fields can't be represented by
        # ruamel.yaml.RoundTripDumper, but transforming them to strings
        # should be enough for all cases that we need.
        for field in self.collection.settings['fields']:
            if field['type'] == 'decimal':
                v = to_dump.get(field['name'], '')
                if v is not None:
                    to_dump[field['name']] = str(v)

        s_io = io.StringIO()
        self.collection.yaml.dump(to_dump, s_io)
        return s_io.getvalue()

    @property
    def idterm(self):
        return "Q" + self.eid

    @property
    def short_id(self):
        return self.eid[:8]

[docs]    def validate(self):
        errors = []
        valid = True
        for name, field in self.collection.fields.items():
            value = self.data.get(name, None)
            try:
                self.data[name] = field.load(value)
            except types.LesanaValueError as e:
                valid = False
                errors.append(
                    {
                        'field': name,
                        'error': e,
                    }
                )
        return valid, errors

[docs]    def render(self, template, searchpath='.'):
        jtemplate = self.collection.get_template(template, searchpath)
        try:
            return jtemplate.render(entry=self)
        except jinja2.exceptions.TemplateSyntaxError as e:
            raise TemplatingError('Template Syntax Error: ' + str(e))

[docs]    def auto(self):
        """
        Update all fields of this entry, as required by the field settings.

        This is called by the reference client before an edit, so that
        the user can make further changes.

        Note that the stored file is not changed: if you need it you
        need to save the entry yourself.
        """
        for name, field in self.collection.fields.items():
            self.data[name] = field.auto(self.data.get(name, None))

[docs]class Collection(object):
    """
    """

    PARSER_FLAGS = (
        xapian.QueryParser.FLAG_BOOLEAN
        | xapian.QueryParser.FLAG_PHRASE  # noqa: W503
        | xapian.QueryParser.FLAG_LOVEHATE  # noqa: W503
        | xapian.QueryParser.FLAG_WILDCARD  # noqa: W503
    )

    def __init__(self, directory=None, itemdir='items'):
        self.basedir = directory or os.getcwd()
        self.itemdir = os.path.join(self.basedir, itemdir)
        self.yaml = ruamel.yaml.YAML()
        self.yaml.preserve_quotes = True
        self.yaml.typ = 'rt'
        try:
            with open(os.path.join(self.basedir, 'settings.yaml')) as fp:
                self.settings = self.yaml.load(fp)
        except FileNotFoundError:
            self.settings = self.yaml.load("{}")
        self.fields = self._load_field_types()
        os.makedirs(os.path.join(self.basedir, '.lesana'), exist_ok=True)
        if 'lang' in self.settings:
            try:
                self.stemmer = xapian.Stem(self.settings['lang'])
            except xapian.InvalidArgumentError:
                logger.warning(
                    "Invalid language %s, in settings.yaml: using english.",
                    self.settings['lang'],
                )
                self.stemmer = xapian.Stem('english')
        else:
            self.stemmer = xapian.Stem('english')
        self._enquire = None
        self.entry_class = Entry

    def _get_subsubclasses(self, cls):
        for c in cls.__subclasses__():
            yield c
            yield from self._get_subsubclasses(c)

    def _load_field_types(self):
        type_loaders = {}
        for t in self._get_subsubclasses(types.LesanaType):
            type_loaders[t.name] = t
        fields = {}
        for i, field in enumerate(self.settings.get('fields', [])):
            try:
                fields[field['name']] = type_loaders[field['type']](
                    field,
                    type_loaders,
                    # value slot 0 is used to store the filename, and we
                    # reserve a few more slots just in case they are
                    # needed by lesana or some derivative
                    value_index=i + 16,
                )
            except KeyError:
                # unknown fields are treated as if they were
                # (unvalidated) generic YAML to support working with
                # collections based on lesana derivatives
                logger.warning(
                    "Unknown field type %s in field %s",
                    field['type'],
                    field['name'],
                )
                fields[field['name']] = types.LesanaYAML(field, type_loaders)
        return fields

    def _index_file(self, fname, cache):
        with open(os.path.join(self.itemdir, fname)) as fp:
            data = self.yaml.load(fp)
        entry = self.entry_class(self, data, fname)
        valid, errors = entry.validate()
        if not valid:
            logger.warning(
                "Not indexing {fname}: invalid data".format(fname=fname)
            )
            return False, errors

        doc = xapian.Document()
        self.indexer.set_document(doc)

        for field, loader in self.fields.items():
            loader.index(doc, self.indexer, entry.data.get(field))

        doc.set_data(entry.yaml_data)
        doc.add_boolean_term(entry.idterm)
        doc.add_value(0, entry.fname.encode('utf-8'))

        cache.replace_document(entry.idterm, doc)
        return True, []

    @property
    def indexed_fields(self):
        fields = []
        for field in self.settings['fields']:
            if field.get('index', '') in ['free', 'field']:
                prefix = field.get('prefix', 'X' + field['name'].upper())
                fields.append(
                    {
                        'prefix': prefix,
                        'name': field['name'],
                        'free_search': field['index'] == 'free',
                        'multi': field['type'] in ['list'],
                    }
                )
        return fields

[docs]    def update_cache(self, fnames=None, reset=False):
        """
        Update the xapian db with the data in files.

        ``fnames`` is a list of *basenames* of files in ``self.itemdir``.

        If no files have been passed, add everything.

        if ``reset`` the existing xapian db is deleted before indexing

        Return the number of files that have been added to the cache.
        """
        if reset:
            shutil.rmtree(os.path.join(self.basedir, '.lesana'))
        os.makedirs(os.path.join(self.basedir, '.lesana'), exist_ok=True)
        cache = xapian.WritableDatabase(
            os.path.join(self.basedir, '.lesana/xapian'),
            xapian.DB_CREATE_OR_OPEN,
        )
        self.indexer = xapian.TermGenerator()
        self.indexer.set_stemmer(self.stemmer)
        if not fnames:
            try:
                fnames = os.listdir(self.itemdir)
            except FileNotFoundError:
                logger.warning(
                    "No such file or directory: {}, not updating cache".format(
                        self.itemdir
                    )
                )
                return 0
        updated = 0
        for fname in fnames:
            try:
                valid, errors = self._index_file(fname, cache)
            except IOError as e:
                logger.warning(
                    "Could not load file {}: {}".format(fname, str(e))
                )
            else:
                if valid:
                    updated += 1
                else:
                    logger.warning(
                        "File {fname} could not be indexed: {errors}".format(
                            fname=fname, errors=errors
                        )
                    )
        return updated

[docs]    def save_entries(self, entries=[]):
        for e in entries:
            complete_name = os.path.join(self.itemdir, e.fname)
            with open(complete_name, 'w') as fp:
                fp.write(e.yaml_data)

[docs]    def git_add_files(self, files=[]):
        if not git_available:
            logger.warning(
                "python3-git not available, could not initalise "
                + "the git repository."  # noqa: W503
            )
            return False
        if not self.settings.get('git', False):
            logger.info("This collection is configured not to use git")
            return False
        try:
            repo = git.Repo(self.basedir, search_parent_directories=True)
        except git.exc.InvalidGitRepositoryError:
            logger.warning(
                "Could not find a git repository in {}".format(self.basedir)
            )
            return False
        repo.index.add(files)
        return True

    def _get_cache(self):
        try:
            cache = xapian.Database(
                os.path.join(self.basedir, '.lesana/xapian'),
            )
        except xapian.DatabaseOpeningError:
            logger.info("No database found, indexing entries.")
            self.update_cache()
            cache = xapian.Database(
                os.path.join(self.basedir, '.lesana/xapian'),
            )
        return cache

[docs]    def render_query_template(self, query):
        """
        Render a query template, filling it with search_aliases.
        """
        t = jinja2.Template(query)
        return t.render(**self.settings.get('search_aliases', {}))

[docs]    def start_search(self, querystring, sort_by=None):
        """
        Prepare a search for querystring.
        """
        cache = self._get_cache()
        queryparser = xapian.QueryParser()
        queryparser.set_stemmer(self.stemmer)
        queryparser.set_database(cache)

        for field in self.indexed_fields:
            queryparser.add_prefix(field['name'], field['prefix'])

        if querystring == '*':
            query = xapian.Query.MatchAll
        else:
            query = queryparser.parse_query(querystring, self.PARSER_FLAGS)

        self._enquire = xapian.Enquire(cache)
        self._enquire.set_query(query)

        if not sort_by and self.settings.get('default_sort', False):
            sort_by = self.settings['default_sort']

        if sort_by:
            keymaker = xapian.MultiValueKeyMaker()
            for k in sort_by:
                if k.startswith('+'):
                    reverse = False
                    slot = self.fields[k[1:]].value_index
                elif k.startswith('-'):
                    reverse = True
                    slot = self.fields[k[1:]].value_index
                else:
                    reverse = False
                    slot = self.fields[k].value_index
                keymaker.add_value(slot, reverse)
            self._enquire.set_sort_by_key_then_relevance(keymaker, False)

[docs]    def get_search_results(self, offset=0, pagesize=12):
        if not self._enquire:
            return
        for match in self._enquire.get_mset(offset, pagesize):
            yield self._match_to_entry(match)

[docs]    def get_all_search_results(self):
        if not self._enquire:
            return
        offset = 0
        pagesize = 100
        while True:
            mset = self._enquire.get_mset(offset, pagesize)
            if mset.size() == 0:
                break
            for match in mset:
                yield self._match_to_entry(match)
            offset += pagesize

[docs]    def get_all_documents(self):
        """
        Yield all documents in the collection.

        Note that the results can't be sorted, even if the collection
        has a default_sort; if you need sorted values you need to use a
        regular search with a query of '*'
        """
        cache = self._get_cache()
        postlist = cache.postlist("")
        for post in postlist:
            doc = cache.get_document(post.docid)
            yield self._doc_to_entry(doc)

[docs]    def get_field_values(self, field, querystring='*'):
        field = self.fields[field]
        if field.field.get('sortable', False):
            self.start_search(querystring)

            spy = xapian.ValueCountMatchSpy(field.value_index)
            self._enquire.add_matchspy(spy)

            cache = self._get_cache()
            self._enquire.get_mset(0, cache.get_doccount())

            for v in spy.values():
                yield {
                    'value': v.term,
                    'frequency': v.termfreq,
                }

        else:
            logger.info(
                "Trying to get the list of values for a non sortable field."
            )
            logger.info(
                "This is going to be pretty inefficient."
            )
            if field.field['type'] == 'list':
                values = []
                for e in self.get_all_documents():
                    values.extend(e.data[field.field['name']])
            else:
                values = (
                    e.data[field.field['name']]
                    for e in self.get_all_documents()
                )
            logger.info("Values are %s", str(values))

            counter = collections.Counter(values)
            for v in counter.most_common():
                yield {
                    'value': v[0],
                    'frequency': v[1],
                }

    def _match_to_entry(self, match):
        return self._doc_to_entry(match.document)

    def _doc_to_entry(self, doc):
        fname = doc.get_value(0).decode('utf-8')
        data = self.yaml.load(doc.get_data())
        entry = self.entry_class(self, data=data, fname=fname,)
        return entry

[docs]    def entry_from_eid(self, eid):
        cache = self._get_cache()
        postlist = cache.postlist('Q' + eid)
        for pitem in postlist:
            return self._doc_to_entry(cache.get_document(pitem.docid))
        return None

[docs]    def entries_from_short_eid(self, seid):
        # It would be better to search for partial UIDs inside xapian,
        # but I still can't find a way to do it, so this is a workable
        # workaround on repos where the eids are stored in the
        # filenames.
        potential_eids = [
            os.path.splitext(f)[0]
            for f in os.listdir(self.itemdir)
            if f.startswith(seid)
        ]
        return [self.entry_from_eid(u) for u in potential_eids if u]

[docs]    def remove_entries(self, eids):
        cache = xapian.WritableDatabase(
            os.path.join(self.basedir, '.lesana/xapian'),
            xapian.DB_CREATE_OR_OPEN,
        )
        for eid in eids:
            for entry in self.entries_from_short_eid(eid):
                if entry is not None:
                    cache.delete_document(entry.idterm)
                    self.remove_file(entry.fname)
                else:
                    logger.warning("Not removing {}: no such entry".format(
                        eid
                    ))
        cache.commit()
        cache.close()

[docs]    def remove_file(self, fname):
        f_path = os.path.join(self.itemdir, fname)
        if git_available and self.settings.get('git', False):
            try:
                repo = git.Repo(self.basedir, search_parent_directories=True)
            except git.exc.InvalidGitRepositoryError:
                logger.warning(
                    "Could not find a git repository in {}".format(
                        self.basedir
                    )
                )
                return False
            repo.index.remove([f_path])
        os.remove(f_path)

[docs]    def update_field(self, query, field, value):
        self.start_search(query)
        changed = []
        for e in self.get_all_search_results():
            e.data[field] = value
            changed.append(e)
        self.save_entries(changed)
        self.git_add_files(
            [os.path.join(self.itemdir, e.fname) for e in changed]
        )
        self.update_cache([e.fname for e in changed])

[docs]    def get_template(self, template_fname, searchpath='.'):
        env = templating.Environment(
            loader=jinja2.FileSystemLoader(
                searchpath=searchpath, followlinks=True,
            ),
            # TODO: add autoescaping settings
        )
        try:
            template = env.get_template(template_fname)
        except jinja2.exceptions.TemplateNotFound as e:
            raise TemplatingError('Could not find template ' + str(e))
        return template

[docs]    def entry_from_rendered_template(self, template, data):
        try:
            template = self.get_template(template)
            rendered = template.render(**data)
        except jinja2.exceptions.TemplateSyntaxError as e:
            raise TemplatingError(e)
        try:
            data = self.yaml.load(rendered)
        except ruamel.yaml.YAMLError as e:
            logger.warning(
                "The following data failed to load as YAML: \n{}".format(
                    rendered
                )

            )
            raise TemplatingError(e)
        entry = self.entry_class(self, data=data)
        self.save_entries([entry])
        return entry

[docs]    @classmethod
    def init(
        cls, directory=None, git_enabled=True, edit_file=None, settings={}
    ):
        """
        Initialize a lesana repository

        directory defaults to .
        if git_enabled is True, git support is enabled and if possible a git
        repository is initalized.
        edit_file is a syncronous function that runs on a filename
        (possibly opening the file in an editor) and should manage its
        own errors.
        """
        c_dir = os.path.abspath(directory or '.')
        os.makedirs(c_dir, exist_ok=True)
        if git_enabled:
            # Try to initalize a git repo
            if git_available:
                repo = git.Repo.init(c_dir, bare=False)
            else:
                logger.warning(
                    "python3-git not available, could not initalise "
                    + "the git repository."  # noqa: W503
                )
                repo = None
            # Add .lesana directory to .gitignore and add it to the
            # staging
            lesana_ignored = False
            try:
                with open(os.path.join(c_dir, '.gitignore'), 'r') as fp:
                    for line in fp:
                        if '.lesana' in line:
                            lesana_ignored = True
                            continue
            except FileNotFoundError:
                pass
            if not lesana_ignored:
                with open(os.path.join(c_dir, '.gitignore'), 'a') as fp:
                    fp.write('#Added by lesana init\n.lesana')
                if repo:
                    repo.index.add(['.gitignore'])

            # Add post-checkout and post-merge hook
            hook_source = resource_filename('lesana', 'data/post-checkout')
            hooks_dir = os.path.join(
                c_dir,
                '.git',
                'hooks',
            )
            checkout_hook = os.path.join(hooks_dir, 'post-checkout')
            merge_hook = os.path.join(hooks_dir, 'post-merge')

            shutil.copy(hook_source, checkout_hook)
            if not os.path.islink(merge_hook):
                os.symlink(checkout_hook, merge_hook)
        # If it doesn't exist, create a skeleton of settings.yaml file
        # then open settings.yaml for editing
        filepath = os.path.join(c_dir, 'settings.yaml')
        if not os.path.exists(filepath):
            skel = resource_string('lesana', 'data/settings.yaml').decode(
                'utf-8'
            )
            yaml = ruamel.yaml.YAML()
            skel_dict = yaml.load(skel)
            skel_dict['git'] = git_enabled
            skel_dict.update(settings)
            with open(filepath, 'w') as fp:
                yaml.dump(skel_dict, stream=fp)
        if edit_file:
            edit_file(filepath)
        if git_enabled and repo:
            repo.index.add(['settings.yaml'])
        coll = cls(c_dir)
        os.makedirs(os.path.join(coll.basedir, coll.itemdir), exist_ok=True)
        return coll

[docs]class TemplatingError(Exception):
    """
    Raised when there are errors rendering a jinja template
    """

Source code for lesana.command

import argparse
import logging
import os
import subprocess
import sys

try:
    import argcomplete
except ImportError:
    argcomplete = False

from . import Collection, Entry, TemplatingError

logger = logging.getLogger(__name__)

def _get_first_docstring_line(obj):
    try:
        return obj.__doc__.split('\n')[1].strip()
    except (AttributeError, IndexError):
        return None

[docs]class MainCommand:
    commands = ()

    def _main(self, args):
        self.parser.print_help()

[docs]    def main(self):
        desc = _get_first_docstring_line(self)
        self.parser = argparse.ArgumentParser(description=desc)
        self.parser.add_argument('--verbose', '-v',
                                 action='store_true',
                                 help="Display debug messages")
        self.parser.set_defaults(func=self._main)
        self.subparsers = self.parser.add_subparsers()
        for name, sub in self.commands:
            sub_help = _get_first_docstring_line(sub)
            s_parser = self.subparsers.add_parser(
                name,
                help=sub_help,
                description=sub.__doc__,
            )
            for arg in sub.arguments:
                s_parser.add_argument(*arg[0], **arg[1])
            s_parser.set_defaults(func=sub._main)
        if argcomplete:
            argcomplete.autocomplete(self.parser)
        self.args = self.parser.parse_args()

        if self.args.verbose:
            logging.getLogger('lesana').setLevel(logging.DEBUG)

        self.args.func(self.args)

[docs]class Command:
    def __init__(self, collection_class=Collection, entry_class=Entry):
        self.collection_class = collection_class
        self.entry_class = entry_class

    def _main(self, args):
        self.args = args
        self.main()

[docs]    def edit_file_in_external_editor(self, filepath):
        # First we try to use $EDITOR
        editor = os.environ.get('EDITOR')
        if editor:
            try:
                subprocess.call([editor, filepath])
            except FileNotFoundError as e:
                if editor in str(e):
                    logger.info(
                        'Could not open file {} with $EDITOR (currently {})'
                        .format(
                            filepath, editor
                        )
                    )
                else:
                    logger.warning("Could not open file {}".format(filepath))
                    return False
            else:
                return True
        # then we try to use sensible-editor (which should be available on
        # debian and derivatives)
        try:
            subprocess.call(['sensible-editor', filepath])
        except FileNotFoundError as e:
            if 'sensible-editor' in e.strerror:
                logger.debug(
                    "Could not open file {} with editor: sensible-editor"
                    .format(filepath)
                )
            else:
                logger.warning("Could not open file {}".format(filepath))
                return False
        else:
            return True
        # and finally we fallback to vi, because ed is the standard editor,
        # but that would be way too cruel, and vi is also in posix
        try:
            subprocess.call(['vi', filepath])
        except FileNotFoundError as e:
            if 'vi' in e.strerror:
                logger.warning(
                    "Could not open file {} with any known editor".format(
                        filepath
                    )
                )
                return False
            else:
                logger.warning("Could not open file {}".format(filepath))
                return False
        else:
            return True

[docs]class New(Command):
    """
    Create a new entry
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--no-git'],
            dict(
                help="Don't add the new entry to git",
                action="store_false",
                dest='git',
            ),
        ),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        new_entry = self.entry_class(collection)
        collection.save_entries([new_entry])
        filepath = os.path.join(collection.itemdir, new_entry.fname)
        if self.edit_file_in_external_editor(filepath):
            collection.update_cache([filepath])
            if self.args.git:
                collection.git_add_files([filepath])
        saved_entry = collection.entry_from_eid(new_entry.eid)
        print(saved_entry)

[docs]class Edit(Command):
    """
    Edit a lesana entry
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--no-git'],
            dict(
                help="Don't add the new entry to git",
                action="store_false",
                dest='git',
            ),
        ),
        (['eid'], dict(help='eid of an entry to edit',)),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        entries = collection.entries_from_short_eid(self.args.eid)
        if len(entries) > 1:
            return "{} is not an unique eid".format(self.args.eid)
        if not entries:
            return "Could not find an entry with eid starting with: {}".format(
                self.args.eid
            )
        entry = entries[0]
        # update the entry before editing it
        entry.auto()
        collection.save_entries([entry])
        # and then edit the updated file
        filepath = os.path.join(collection.itemdir, entry.fname)
        if self.edit_file_in_external_editor(filepath):
            collection.update_cache([filepath])
            if self.args.git:
                collection.git_add_files([filepath])
        saved_entry = collection.entry_from_eid(entry.eid)
        print(saved_entry)

[docs]class Show(Command):
    """
    Show a lesana entry
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--template', '-t'],
            dict(help='Use the specified template to display results.',),
        ),
        (['eid'], dict(help='eid of an entry to edit',)),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        entries = collection.entries_from_short_eid(self.args.eid)
        if len(entries) > 1:
            return "{} is not an unique eid".format(self.args.eid)
        if not entries:
            return "Could not find an entry with eid starting with: {}".format(
                self.args.eid
            )
        entry = entries[0]
        if self.args.template:
            try:
                print(entry.render(self.args.template))
            except TemplatingError as e:
                logger.error("{}".format(e))
                sys.exit(1)
        else:
            print(entry.yaml_data)

[docs]class Index(Command):
    """
    Index entries in a lesana collection
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--reset'],
            dict(
                action='store_true',
                help='Delete the existing index and reindex from scratch.',
            ),
        ),
        (
            ['files'],
            dict(
                help='List of files to index (default: everything)',
                default=None,
                nargs='*',
            ),
        ),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        if self.args.files:
            files = (os.path.basename(f) for f in self.args.files)
        else:
            files = None
        indexed = collection.update_cache(
            fnames=files,
            reset=self.args.reset
        )
        print("Found and indexed {} entries".format(indexed))

[docs]class Search(Command):
    """
    Search for entries
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--template', '-t'],
            dict(help='Template to use when displaying results',),
        ),
        (['--offset'], dict(type=int,)),
        (['--pagesize'], dict(type=int,)),
        (
            ['--all'],
            dict(action='store_true', help='Return all available results'),
        ),
        (
            ['--sort'],
            dict(action='append', help='Sort results by a sortable field'),
        ),
        (
            ['--expand-query-template', '-e'],
            {
                'action': 'store_true',
                'help':
                    'Render search_aliases in the query as a jinja2 template',
            },
        ),
        (
            ['query'],
            {
                'help': 'Xapian query to search in the collection',
                'nargs': '*',
                'default': '*',
            },
        ),
    ]

[docs]    def main(self):
        # TODO: implement "searching" for everything
        if self.args.offset:
            logger.warning(
                "offset exposes an internal knob and MAY BE REMOVED "
                + "from a future release of lesana"  # noqa: W503
            )
        if self.args.pagesize:
            logger.warning(
                "pagesize exposes an internal knob and MAY BE REMOVED "
                + "from a future release of lesana"  # noqa: W503
            )
        offset = self.args.offset or 0
        pagesize = self.args.pagesize or 12
        collection = self.collection_class(self.args.collection)
        query = self.args.query
        if self.args.expand_query_template:
            query = collection.render_query_template(query)
        # sorted results require a less efficient full search rather
        # than being able to use the list of all documents.
        if query == ['*'] and not (
            self.args.sort
            or getattr(collection.settings, 'default_sort', False)
        ):
            results = collection.get_all_documents()
        else:
            collection.start_search(
                ' '.join(query),
                sort_by=self.args.sort
            )
            if self.args.all:
                results = collection.get_all_search_results()
            else:
                results = collection.get_search_results(offset, pagesize)
        if self.args.template:
            try:
                template = collection.get_template(self.args.template)
                print(template.render(entries=results))
            except TemplatingError as e:
                logger.error("{}".format(e))
                sys.exit(1)
        else:
            for entry in results:
                print("{entry}".format(entry=entry,))

[docs]class GetValues(Command):
    """
    List all values for one field, with entry counts.
    """
    arguments = [
        (
            ['--collection', '-c'],
            {
                'help': 'The collection to work on (default .)'
            },
        ),
        (
            ['--field', '-f'],
            {
                'help': 'Name of the field',
                'required': True,
            },
        ),
        (
            ['--template', '-t'],
            {
                'help': 'Template to use when displaying results',
            },
        ),
        (
            ['query'],
            {
                'help': 'Xapian query to limit the count search " \
                      + "in the collection',
                'nargs': '*',
                'default': '*'
            },
        ),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        counts = collection.get_field_values(
            self.args.field,
            ' '.join(self.args.query)
        )
        if self.args.template:
            try:
                template = collection.get_template(self.args.template)
                print(template.render(counts=counts))
            except TemplatingError as e:
                logger.error("{}".format(e))
                sys.exit(1)
        else:
            for v in counts:
                print("{value}: {count}".format(
                    value=v['value'],
                    count=v['frequency']
                ))

[docs]class Export(Command):
    """
    Export entries to a different collection
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)'),
        ),
        (
            ['--query', '-q'],
            dict(help='Xapian query to search in the collection',),
        ),
        (['destination'], dict(help='The collection to export entries to')),
        (['template'], dict(help='Template to convert entries',)),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        destination = self.collection_class(self.args.destination)
        if not self.args.query:
            results = collection.get_all_documents()
        else:
            collection.start_search(' '.join(self.args.query))
            results = collection.get_all_search_results()
        for entry in results:
            data = {
                "entry": entry
            }
            data.update(entry.data)
            try:
                destination.entry_from_rendered_template(
                    self.args.template,
                    data
                )
            except TemplatingError as e:
                logger.error("Error converting entry: {}".format(entry))
                logger.error("{}".format(e))
                sys.exit(1)

[docs]class Init(Command):
    """
    Initialize a lesana collection
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The directory to work on (default .)', default='.'),
        ),
        (
            ['--no-git'],
            dict(
                help='Skip setting up git in this directory',
                action="store_false",
                dest='git',
            ),
        ),
    ]

[docs]    def main(self):
        self.collection_class.init(
            self.args.collection,
            git_enabled=self.args.git,
            edit_file=self.edit_file_in_external_editor,
        )

[docs]class Remove(Command):
    """
    Remove an entry from a collection
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)',),
        ),
        (['entries'], dict(help='List of entries to remove', nargs='+',)),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        collection.remove_entries(eids=self.args.entries)

[docs]class Update(Command):
    """
    Update a field in multiple entries
    """
    arguments = [
        (
            ['--collection', '-c'],
            dict(help='The collection to work on (default .)',),
        ),
        (['--field', '-f'], dict(help='The field to change',)),
        (['--value', '-t'], dict(help='The value to set',)),
        (
            ['query'],
            dict(help='Xapian query to search in the collection', nargs='+'),
        ),
    ]

[docs]    def main(self):
        collection = self.collection_class(self.args.collection)
        collection.update_field(
            ' '.join(self.args.query),
            field=self.args.field,
            value=self.args.value,
        )

Source code for lesana.templating

"""
Custom jinja2 filters and other templating helpers
"""
import decimal
import io

import jinja2
import ruamel.yaml

[docs]class Environment(jinja2.Environment):
    """
    A customized jinja2 environment that includes our filters.
    """

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.filters['to_yaml'] = to_yaml

[docs]def to_yaml(data):
    """
    Return the yaml representation of data.
    """
    if isinstance(data, str):
        if len(data) > 75 or "\n" in data:
            try:
                data = ruamel.yaml.scalarstring.LiteralScalarString(
                    data + "\n"
                )
            except AttributeError:
                data = ruamel.yaml.scalarstring.PreservedScalarString(
                    data + "\n"
                )
    elif isinstance(data, decimal.Decimal):
        data = str(data)
    elif data is None:
        return 'null'
    yaml = ruamel.yaml.YAML()
    s_io = io.StringIO()
    yaml.dump({'data': data}, s_io)
    res = s_io.getvalue()
    res = res.lstrip('{data:').lstrip().strip('...\n').strip()
    return res

Source code for lesana.types

"""
Type checkers for lesana fields.

Warning: this part of the code is still in flux and it may change
significantly in a future release.
"""
import datetime
import decimal
import logging

import dateutil.parser
import xapian

logger = logging.getLogger(__name__)

[docs]class LesanaType:
    """
    Base class for lesana field types.
    """
    def __init__(self, field, types, value_index=None):
        self.field = field
        self.value_index = value_index

[docs]    def load(self, data):
        raise NotImplementedError

[docs]    def empty(self):
        raise NotImplementedError

[docs]    def auto(self, value):
        """
        Return an updated value, as appropriate for the field.

        Default is to return the value itself, but types can use their
        configuration to e.g. increment a numerical value or return the
        current date(time).
        """
        return value

[docs]    def allowed_value(self, value):
        """
        Check whether a value is allowed in this field.

        Return the value itself or raise LesanaValueError if the value
        isn't valid.
        """
        if not value:
            return value
        valid_values = self.field.get('values')
        if not valid_values:
            return value
        if value in valid_values:
            return value
        raise LesanaValueError("Value {} is not allowed in field {}".format(
            value,
            self.field.get('name')
        ))

    def _to_index_text(self, value):
        """
        Prepare a value for indexing.
        """
        return str(value)

    def _to_value(self, value):
        """
        Prepare a value for indexing in a value slot
        """
        return str(value)

[docs]    def index(self, doc, indexer, value):
        """
        Index a value for this field type.

        Override this for types that need any kind of special treatment
        to be indexed.

        See LesanaList for an idea on how to do so.
        """
        to_index = self.field.get('index', False)
        if not to_index:
            return
        if not value:
            logger.debug(
                "Not indexing empty value {}".format(value)
            )
            return
        prefix = self.field.get('prefix', 'X' + self.field['name'].upper())
        indexer.index_text(self._to_index_text(value), 1, prefix)
        if to_index == 'free':
            indexer.index_text(self._to_index_text(value))
            indexer.increase_termpos()
        if self.field.get('sortable', False):
            if self.value_index and self.value_index >= 16:
                doc.add_value(self.value_index, self._to_value(value))
            else:
                logger.debug(
                    "Index values up to 15 are reserved for internal use"
                )

[docs]class LesanaString(LesanaType):
    """
    A string of unicode text
    """
    name = 'string'

[docs]    def load(self, data):
        if not data:
            return data
        return self.allowed_value(str(data))

[docs]    def empty(self):
        return ""

[docs]class LesanaText(LesanaString):
    """
    A longer block of unicode text
    """
    name = 'text'

[docs]class LesanaInt(LesanaType):
    """
    An integer number
    """
    name = "integer"

[docs]    def load(self, data):
        if not data:
            return data
        try:
            return self.allowed_value(int(data))
        except ValueError:
            raise LesanaValueError(
                "Invalid value for integer field: {}".format(data)
            )

[docs]    def empty(self):
        return 0

    def _to_index_text(self, value):
        """
        Prepare a value for indexing.
        """
        return str(value)

    def _to_value(self, value):
        """
        Prepare a value for indexing in a value slot
        """
        return xapian.sortable_serialise(value)

[docs]    def auto(self, value):
        """
        Return an updated value.

        If the field settings ``auto`` is ``increment`` return the value
        incremented by the value of the field setting ``increment``
        (default 1).

        """
        if self.field.get('auto', False) == 'increment':
            increment = self.field.get('increment', 1)
            if int(increment) == increment:
                return value + increment
            else:
                logger.warning(
                    "Invalid configuration value for increment in field %s: "
                    + "%s",
                    self.field['name'],
                    increment,
                )

        return value

[docs]class LesanaFloat(LesanaType):
    """
    A floating point number
    """
    name = "float"

[docs]    def load(self, data):
        if not data:
            return data
        try:
            return self.allowed_value(float(data))
        except ValueError:
            raise LesanaValueError(
                "Invalid value for float field: {}".format(data)
            )

[docs]    def empty(self):
        return 0.0

[docs]class LesanaDecimal(LesanaType):
    """
    A fixed point number

    Because of a limitation of the yaml format, these should be stored
    quoted as a string, to avoid being loaded back as floats.

    Alternatively, the property ``precision`` can be used to force all
    values to be rounded to that number of decimals.
    """
    name = "decimal"

[docs]    def load(self, data):
        if not data:
            return data
        try:
            value = decimal.Decimal(data)
        except decimal.InvalidOperation:
            raise LesanaValueError(
                "Invalid value for decimal field: {}".format(data)
            )
        precision = self.field.get('precision')
        if precision:
            value = round(value, precision)
        return self.allowed_value(value)

[docs]    def empty(self):
        return decimal.Decimal(0)

[docs]class LesanaTimestamp(LesanaType):
    """
    A unix timestamp, assumed to be UTC
    """
    name = "timestamp"

[docs]    def load(self, data):
        if not data:
            return data
        if isinstance(data, datetime.datetime):
            return data
        try:
            return datetime.datetime.fromtimestamp(
                int(data),
                datetime.timezone.utc,
            )
        except (TypeError, ValueError):
            raise LesanaValueError(
                "Invalid value for timestamp field: {}".format(data)
            )

[docs]    def empty(self):
        return None

[docs]class LesanaDatetime(LesanaType):
    """
    A datetime
    """
    name = "datetime"

[docs]    def load(self, data):
        if not data:
            return data
        if isinstance(data, datetime.datetime):
            return data
        if isinstance(data, datetime.date):
            return datetime.datetime(data.year, data.month, data.day)
        # compatibility with dateutil before 2.8
        ParserError = getattr(dateutil.parser, 'ParserError', ValueError)
        try:
            return dateutil.parser.parse(data)
        except ParserError:
            raise LesanaValueError(
                "Invalid value for datetime field: {}".format(data)
            )

[docs]    def empty(self):
        if self.field.get('auto', False) in ('creation', 'update'):
            return datetime.datetime.now(datetime.timezone.utc)

        return None

[docs]    def auto(self, value):
        """
        Return an updated value.

        If the field settings ``auto`` is ``update`` return the current
        datetime, otherwise the old value.

        """
        if self.field.get('auto', False) == 'update':
            return datetime.datetime.now(datetime.timezone.utc)

        return value

[docs]class LesanaDate(LesanaType):
    """
    A date
    """
    name = "date"

[docs]    def load(self, data):
        if not data:
            return data
        if isinstance(data, datetime.date):
            return data
        # compatibility with dateutil before 2.8
        ParserError = getattr(dateutil.parser, 'ParserError', ValueError)
        try:
            return dateutil.parser.parse(data).date()
        except ParserError:
            raise LesanaValueError(
                "Invalid value for date field: {}".format(data)
            )

[docs]    def empty(self):
        if self.field.get('auto', False) in ('creation', 'update'):
            return datetime.date.today()

        return None

[docs]    def auto(self, value):
        """
        Return an updated value.

        If the field settings ``auto`` is ``update`` return the current
        date, otherwise the old value.

        """
        if self.field.get('auto', False) == 'update':
            return datetime.date.today()

        return value

[docs]class LesanaBoolean(LesanaType):
    """
    A boolean value
    """
    name = 'boolean'

[docs]    def load(self, data):
        if not data:
            return data
        if isinstance(data, bool):
            return data
        else:
            raise LesanaValueError(
                "Invalid value for boolean field: {}".format(data)
            )

[docs]    def empty(self):
        return None

[docs]class LesanaFile(LesanaString):
    """
    A path to a local file.

    Relative paths are assumed to be relative to the base lesana
    directory (i.e. where .lesana lives)
    """
    name = 'file'

[docs]class LesanaURL(LesanaString):
    """
    An URL
    """
    name = 'url'

[docs]class LesanaGeo(LesanaString):
    """
    A Geo URI
    """
    name = 'geo'

[docs]    def load(self, data):
        data = super().load(data)
        if data and not data.startswith("geo:"):
            raise LesanaValueError("{} does not look like a geo URI".format(
                data
            ))
        return data

[docs]class LesanaYAML(LesanaType):
    """
    Free YAML contents (no structure is enforced)
    """
    name = 'yaml'

[docs]    def load(self, data):
        return data

[docs]    def empty(self):
        return None

[docs]class LesanaList(LesanaType):
    """
    A list of other values
    """

    name = 'list'

    def __init__(self, field, types, value_index=None):
        super().__init__(field, types, value_index)
        try:
            self.sub_type = types[field['list']](field, types)
        except KeyError:
            logger.warning(
                "Unknown field type %s in field %s",
                field['type'],
                field['name'],
            )
            self.sub_type = types['yaml'](field, types)

[docs]    def load(self, data):
        if data is None:
            # empty for this type means an empty list
            return []
        try:
            return [self.allowed_value(self.sub_type.load(x)) for x in data]
        except TypeError:
            raise LesanaValueError(
                "Invalid value for list field: {}".format(data)
            )

[docs]    def empty(self):
        return []

[docs]    def index(self, doc, indexer, value):
        for v in value:
            self.sub_type.index(doc, indexer, v)

[docs]class LesanaValueError(ValueError):
    """
    Raised in case of validation errors.
    """