beanahead 0.3

Description:

beanahead 0.3

beanahead

Beanahead administers future transactions for beancount ledgers.
It let's you:

Generate regular expected transactions.
Define ad hoc expected transactions.
Reconcile expected transactions against imported transactions.

It's useful to:

Forecast account balances based on expected income and payments.
Add transactions between updating accounts.

Installation
$ pip install beanahead
The only direct dependencies are beancount, beangulp and pandas (pandas could be made optional, see #1).
Beanahead is compatible with both beancount v2 and v3.
Really briefly, how it works

Expected transactions are defined on separate ledgers.
Newly imported entries are reconciled against expected transactions before being included to the main ledger (beanahead does not need to touch your main ledger).

Briefly, how it works

Regular Expected Transactions (electicity bill, rent, etc) are defined by including a single transaction to a dedicated beancount file, by convention 'rx_def.beancount'.
The addrx command is used to populate a Regular Expected Transactions ledger with transactions. This ledger, by convention 'rx.beancount', is 'included' to your main .beancount ledger.
ad hoc Expected Transactions are added to the Expected Transactions ledger, by convention 'x.beancount'. This ledger is also 'included' to your main .beancount ledger.
The recon comamnd offers a cli to reconcile newly imported transactions with transactions on the expected transactions ledgers.

Newly imported transactions are updated to reflect any missing narration, tags, meta and 'other side' postings defined on the corresponding expected transactions.
Reconciled expected transactions are removed from their respective ledgers.



In keeping with the beancount spirit, beanahead provides all its functionality via a CLI (the underlying functions are available within the codebase should you prefer). There are five commands, all subcommands of beanahead.
$ beanahead --help
usage: beanahead [-h] [--version] {make,addrx,recon,exp,inject} ...

subcommands:
{make,addrx,recon,exp,inject}
make make a new beanahead file.
addrx add Regular Expected Transactions.
recon reconcile new transactions.
exp administer expired expected transactions.
inject inject new transactions.


:information_source: Wherever this README shows the output from a --help option, only an abridged version of the output is shown. Run the command at the command line for the full help, e.g. beanahead --help. For subcommands the help includes the documentation of the underlying function.

Using beanahead
Index

Making beanahead files
Regular Expected Transactions

Defining regular transactions

freq
Postings
roll
final


Updating definitions
Adding regular transactions


ad hoc transactions
Defining the payee
Reconciling

Matching
Updating


Injection
Expired expected transactions
Worth remembering
Developers
Alternative packages
beancount recommendations
Licence

Making beanahead files
First off, you'll need to use the make command to create a new .beancount file or three.

:information_source: You'll probably want these files to be in the same directory as your exisitng main ledger. The following examples assume you want to create the files in the current working directory. Pass the --dirpath (or -d) option to specify a different directory.

If you want to include Regular Expected Transactions you'll need to create a definitions file and a dedicated ledger....
$ beanahead make rx_def -f rx_def
$ beanahead make rx -f rx

...and add the following line towards the top of your main ledger.
include "rx.beancount"

If you want to include ad hoc Expected Transactions you'll need to create a separate dedicated ledger...
$ beanahead make x -f x

...and add the following line towards the top of your main ledger.
include "x.beancount"

So, if you want to include both regular and ad hoc expected transactions then you should have created three new .beancount files and added two 'include' lines to top of your main ledger.

:information_source: The -f option provides for defining the filename (make will add the .beancount extension). If -f is not passed then default names will be used which are as those explicitly passed in the examples.


:information_source: The quoted file in the 'include' lines should reflect the path to the expected transactions ledger from the directory in which your main ledger is located. The above examples assume that the expected transactions ledgers have the default filenames and are located in the same directory as the main ledger.

The examples/new_empty_files folder includes a sample of each of the above newly created files.

Regular Expected Transaction Definitions file
Regular Expected Transactions Ledger
Expected Transactions Ledger

NB If you're not using the default account root names (e.g. 'Assets', 'Income', 'Expenses' etc) then you'll also need to use the optional --main argument to provide the path to a ledger from which beanahead can read the customised name options. Example:
$ beanahead make x -f x --main my_ledger

Regular Expected Transactions
Regular expected transactions are defined on the Regular Expected Transaction Definitions file. The addrx command can then be used to populate the Regular Expected Transactions Ledger with transactions generated from these definitions.
Defining regular transactions
A new regular transaction can be defined at any time by adding a single transaction to the definitions file (the 'initial definition'). The date of this transaction will serve as the anchor off which future transactions are evaluated.
The following initial definition would generate regular transactions on the 5th of every month, with the first generated transaction dated 2022-10-05.
2022-10-05 * "EDISON" "Electricity, monthly fixed tarrif"
freq: "m"
Assets:US:BofA:Checking -65.00 USD
Expenses:Home:Electricity

Each initial definition must:

have a unique payee.
include the freq meta field to define the transaction frequency.
include at least two postings of which at least one should be to a balance sheet account ("Assets" or "Liabilities").

Initial definitions can optionally include:

the roll meta field to define if transactions falling on weekends should roll forwards.
the final meta field to define the last transaction date.
additional postings.
custom meta fields.
tags

rx_defs.beancount offers an example of a new definitions file before any transactions have been generated. The initial definitions there cover a variety of circumstances, based loosely on selected sampling of beancount's example.beancount ledger.
freq
The freq meta field is used to specify the frequency with which regular transactions should be generated.
A simple frequency can be specified as "w", "m" and "y", respectively for weekly, monthly and yearly. The unit can be prefixed with a value to specify a multiple, for example "2w" for fortnightly or "3m" for quarterly.
Alternatively, the frequency can be specified with a pandas frequency. For example "BAS-MAR" defines the frequency as the first business day of every March.
Postings
Each definition must include a posting to an account which the regular transactions will appear on the statements of. This can be an "Assets" account (for example, for Direct Debits) or a "Liabilities" account (for example, for regular charges to a credit card). If the amount is variable then just stick in an estimate or the amount that you wish to budget for.
At least one posting to an account on the 'other side' must be included (e.g. to an "Expenses" account). Any number of other postings can be included.
When the regular expected transactions are later reconciled with transactions imported from your statements, the recon command will update the imported transactions with these 'other-side' postings (see reconciling).

:information_source: If the transaction is balanced with only one 'other-side' posting then you're better off not including an amount for it. (By including an amount, if the imported transaction has a different amount then you'll need to manually amend the posting on the updated imported transaction.)


:information_source: If the transaction is split between various 'other-side' postings then it will be necessary to define an amount for at least all but one of these. In this case, if the imported transaction value differs from the expected transaction value then you may want to revise the amounts of those postings for which estimates were included.

Roll
By default, any generated transaction that would be dated on a weekend will be rolled forward to the following Monday. This behaviour can be overriden by the 'roll' meta field.
2022-11-13 * "ETrade Transfer" "Transfering accumulated savings to other account"
freq: "3m"
roll: FALSE
Assets:US:BofA:Checking -4000 USD
Assets:US:ETrade:Cash

The above initial definition is dated 2022-11-13, which is a Sunday. By specifying roll as FALSE the first transaction generated will be dated 2022-11-13 regardless that this is a Sunday. Thereafter transactions will be defined on the 13th of each month regardless of whether these dates represent weekdays or weekends.

:warning: the roll field's value must be in captials and NOT quoted.


:information_source: initial definitions should always be dated on the 'usual' payment day even if that falls on a weekend. For example...
2022-10-16 * "Verizon" "Telecoms, monthly variable"
freq: "m"
Assets:US:BofA:Checking -55.00 USD
Expenses:Home:Phone

This initial definition is dated 2022-10-16 which is a Sunday. The first generated transaction will be automatcially rolled forward to 2022-10-17. All transactions thereafter will be dated as the 16th of each month whenever the 16th is a weekday or otherwise rolled forward to the following Monday.

Final
The 'final' meta field can be used to define a final transaction date. No transactions will be generated that would be dated later than this date (as evaluated prior to any rolling).
2022-10-31 * "Chase" "Chase Hire Purchase"
freq: "BME"
final: 2022-11-30
Liabilities:US:Chase:HirePurchase 322.00 USD
Assets:US:BofA:Checking


:warning: the final field's value should NOT be quoted.

A definition will be automaticaly removed from the definitions file after any final transaction has been generated.
Updating definitions
The addrx command updates the definitions file whenever the ledger is populated with new transactions:

any definition for which a new transaction was generated will be updated to reflect the transaction that would immediately follow the last transaction that was added to the ledger.
Defintions are grouped by balance sheet account* and type of account(s) of the 'other side', for example 'Income', 'Expenses' etc. Each group is introduced with a title row. Within each group definitions are sorted by payee. (*The balance sheet account is assumed as the first "Assets" or "Liabilities" account defined in the postings.)


:information_source: New definitions can be added at any time anywhwere under the line ;; Enter definitions after this line... - they'll find their way to the corresponding section when the file is next updated.

rx_def_updated.beancount is the updated rx_def.beancount file after adding regular transactions to the ledger through to 2022-12-31. :warning: Notice that all comments are lost when a definitions file is updated.
Adding regular transactions
The addrx command is used to populate a Regular Expected Transactions Ledger with transactions evaluated from a definitions file.
$ beanahead addrx --help
usage: beanahead addrx [-h] [-e] defs rx-ledger main-ledger

positional arguments:
defs path to Regular Expected Transactions Definition file.
rx-ledger path to Regular Expected Transactions Ledger file.
main-ledger path to main Ledger file.

optional arguments:
-h, --help show this help message and exit
-e , --end date to which to create new transactions, iso format,e.g. '2020-09-30'. Default 2023-01-11.

For example...
$ beanahead addrx rx_def rx ledger -e 2022-12-31

The above command:

Gets the definitions from the file rx_def.beancount in the current working directory.
For each definition, generates transactions from the defintion date through to '2022-12-31' (both inclusive).
Adds transactions to the rx.beancount ledger.
Updates the rx_def.beancount file (as Updating definitions).
Verifies that the main ledger, ledger.beancount, loads without errors. The path to the main ledger is passed as the third positional argument (this ledger should be the main ledger to which the 'insert "rx.beancount"' line was added). addrx requires this file only to verify that no errors have arisen as a result of introducing the new transactions to the Regular Expected Transactions Ledger.

In the event the main ledger loads with errors then changes made to the definitions file and Regular Expected Transactions Ledger are reverted and advices printed.



If the command is executed as above with the files in the examples/defs folder then the empty rx ledger there will be populated with transactions. The rx ledger would end up as rx_updated.beancount whilst the definitions file would be updated as rx_def_updated.beancount.
ad hoc transactions
Creating ad hoc expected transactions is as simple as adding transactions to an Expected Transactions Ledger created via $ beanahead make x <filename>. The x.beancount file offers an example (again, loosely based on selected sampling of beancount's example.beancount ledger).
Jotting down ad hoc transactions is useful to record transaction details 'in the moment' when you have in mind the 'other-side' postings and maybe know the narration or tag that you might forget by the time you next get round to downloading statements and updating your main ledger.
Transactions can be listed on the ledger in any order. Whenever a transaction on the ledger is reconciled with an imported transaction (extracted from a statement) the ledger is rewritten and any remaining transactions are reordered.

:warning: Any comments will be lost whenever an Expected Transactions Ledger is rewritten.

Defining the payee
The reconciliation of imported transactions with expected transactions (reconciling) can be greatly aided by judiciously naming the expected transactions payee.
Beanahead will treat each 'word' defined in the expected payee as a separate string. An expected payee will match with the payee of any new transaction that includes any of those strings. So:

Do not include short words that represent common syllables.
For example, "Top of the World" will match with "The corner shop", "Another Day" and "Super Offers".
Use few unambiguous words. For example "Top World". Even just "Top" might be a better choice.


:information_source: The transaction that ends up on your main ledger will have the payee as defined on the statement, not "Top"!
:information_source: matches are case-insensitive

Reconciling
The recon command provides for reconciling imported transactions with expected transactions.
$ beanahead recon --help
usage: beanahead recon [-h] [-o OUTPUT] [-k] [-r] new ledgers [ledgers ...]

positional arguments:
new path to new transactions beancount file. Can be
absolute or relative to the current working directory.
ledgers paths to one or more Regular Expected Transactions
Ledgers against which to reconcile incoming
transactions.

optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
path to which to write updated new entries. Can be
absolute or relative to the current working directory.
By default will overwrite 'incoming'.
-k, --keep, --no-remove
flag to not remove reconciled transactions from ledgers.
(By default reconciled transactions will be removed.)
-r, --reverse flag to write updated entries in descending order. By
default will be written in ascending order.

For example...
$ beanahead recon extraction rx x -o injection

The above command:

Gets beancount entries from the file extraction.beancount located in the current working directory. This file should contain only new entries as returned by the beancount extract command and destined for your beancount ledger.
Gets the expected transactions from the ledger files rx.beancount and x.beancount, also located in the current working directory.
Matches expected transactions with new transactions.

The user is requested to confirm or reject each propsosed match.
Where an expected transaction matches more than one new transaction, the user is requested to select which of the new transactions to match (or none).


Updates matched new transactions with information gleaned from the corresponding expected transaction.
Outputs the updated entries. By default overwrites the file passed as the new argument (extraction.beancount above). Alternatively, a path can be passed to the --output option, as in this case where output will be written to injection.beancount in the current working directory.

Entries are by default sorted in ascending order. Pass the -r flag or --reverse option to sort in descending order.


Removes reconciled expected transactions from their respective ledgers. (The -k flag can be passed to 'keep' the expected transactions.)

If the command is executed as above with the files in the examples/recon folder and the matches are confirmed then:

an injection.beancount file will be created that looks like this.
reconciled transactions will be removed from rx.beancount, such that it ends up like this.
reconciled transactions will be removed from x.beancount, such that it ends up like this.

Matching
Beanahead matches expected with imported transactions based on:

payee (see Defining the payee)
similarity in accounts
closeness of dates
closeness of amounts

As a bare minimum, matches require that the new and expected transactions:

are dated within 5 days of each other
include a posting to the same Asset account
either have matching payee or the imported transaction amount is no more than 2% different from the expected

If you want a better insight into how matches are evaluated, check out the reconcile module to get under-the-bonnet.
Updating
Beanahead will update the following fields of matched imported transactions to include any values specifed for the corresponding expected transaction. Any existing values on the imported transaction will not be overwritten.

narration
tags (excluding #rx_txn and #x_txn)
meta (excluding beanahead meta fields such as 'final', 'roll' etc)
postings

postings on the expected transactions will be added to the imported transaction if the imported transaction does not otherwise include a posting to the corresponding account.
if the imported and expected transactions include postings to the same account and only the expected transaction defines a number, the imported transaction's posting will be updated to reflect the value as defined on the expected transaction.



Injection
The output from recon can be copied directly into your main ledger. If you're happy to append the full contents 'as is' to the end of your ledger then the inject command will do it for you.
$ beanahead inject --help
usage: beanahead inject [-h] injection ledger

positional arguments:
injection path to beancount file containing the new entries to be
injected. Can be absolute or relative to the current
working directory.
ledger path to beancount ledger to which new entires are to
be appended. Can be absolute or relative to the current
working directory.

So...
$ beanahead inject injection my_ledger

...would append the updated entires in the injection.beancount file to the end of the my_ledger.beancount file (both files in the current working directory).
Expired expected transactions
Now that your main ledger has been updated with the new entries it'll be necessary to bean-check it to see if all's well. Chances are you'll have to enter some manual postings to balance some transactions.
It's also possible that your balance checks are failing because some expected transactions were included in the new entries although weren't matched (and so now are duplicated), or simply didn't come in (that credit you were waiting for). Beanahead can't chase your debtors but the exp command can at least deal with any expired expected transactions.
$ beanahead exp --help
usage: beanahead exp [-h] ledgers [ledgers ...]

positional arguments:
ledgers paths to one or more Regular Expected Transactions Ledgers
against which to administer expired transactions.

For example:
$ beanahead exp rx x

The above command:

Gets the transactions on the expected transactions ledger files 'rx.beancount' and 'x.beancount' (both in the current working directory).
Offers the user the following options for each transaction that is dated prior to 'today'.

Move transaction forwards to 'tomorrow'.
Move transaction forwards to a user-defined date.
Remove transaction from ledger.
Leave transaction as is.


Rewrites the expected transactions ledgers (if applicable) to reflect the requested changes.

With a bit of luck and perhaps a tweak or two to your ledger, your bean-check should now be checking out.

:information_source: An alternative to using exp is to manually redate / remove transactions on the expected transactions ledgers.

Worth remembering

:warning: Whenever an expected transactions ledger or the regular expected transaction definition files are updated the entries are resorted and the file is overwritten - anything that is not a directive (e.g. comments) will be lost.

Options
Print to stderr
If you are employing a workflow that directs output through sys.stdout then this will merge with Beanahead's own output to this stream. To avoid such conflicts beanahead provides for directing its output to sys.stderr. This can be set from the command line by preceeding any subcommand with --print_stderr, for example...
$ beanahead --print_stderr exp rx x

The same effect can be achieved with the flag -p.
$ beanahead -p exp rx x

If using the underlying functions directly in the codebase then the print stream can be set via the following methods of the beanahead.config module:

set_print_stderr()
set_print_stdout()

Custom account root names
If you're not using the default account root names (e.g. 'Assets', 'Income', 'Expenses' etc) then beanahead will need to know the values that you are using.
The subcommands of the cli that require these names get them from a ledger ('addrx' will read them from the main ledger which is a required argument, 'make' will read them from any ledger passed to its optional --main argument).
If using the underlying functions directly in the codebase then the names can be set either by passing a dictionary to config.set_account_root_names or by passing a ledger that contains these options to utils.set_account_root_names.
The currently set names can be inspected via config.get_account_root_names() and reset to the default values withconfig.reset_account_root_names().
Alternative packages
The beancount community offers a considerable array of add-on packages, many of which are well-rated and maintained. Below I've noted those I know of with functionality that includes some of what beanahead offers. Which package you're likely to find most useful will come down to your specific circumstances and requirements - horses for courses.

beancount-import - an importer interface. Functionality provides for adding expected transactions directly to the main ledger and later merging these with imported transactions via a web-based UI. It requires implementing the importer interface and doesn't directly provide for regular expected transactions. But, if that import interface works for you then you'll probably want to be using beancount-import. (If you need the regular trasactions functionality provided by beanahead, just use beanahead to generate the transactions, copy them over to your ledger and let beancount-import handle the subsequent reconcilation.)

If you can recommend any other alternative package please raise a PR to add it here.
beancount recommendations
If you don't already, it worth trying out a beancount syntax-highlighter extension. Have a look at the 'Editor Support' section of the Awesome Beancount repo to see if there's one available for your prefered editor.
More broadly, awesome-beancount is a great refrence for all things beancount. FWIW, these are my most-awesome of awesome-beancount:

Siddharnt Goel's blog post uses beancount examples to explain Plain Text Accounting and double-entry accounting. A great introduction to these concepts.
The beancount mailing list is the place to look for answers and raise questions.
reds-rants offers a comprehensive set of blogs on everything importing. It's tremendous.
smart_importer offers import hooks that include one for auto-completion of postings based on machine-learning trained on past entries. Smart stuff indeed.
beancount-import is an importer interface. If it works for you to be locked into an importer interface, this is a great one to be locked into.

License
MIT License