GitLocker: The Coding Marketplace

Description:

rstdoc 1.8.2

See background and documentation.
Many companies use DOCX
and thus produce an information barrier.
Working with text is more integrated in the (software) development process.
A final format can be DOCX, but, at least during development, text is better.
Sphinx
is an extension of Docutils
used for many (software) projects,
but it does not support creation of DOCX files, which certain companies demand.
Pandoc
does support DOCX, but does not support the Sphinx extensions,
hence :ref: and the like cannot be used.
This python package supports working with
RST
as documentation format without depending on Sphinx.

link RST documents using
substitutions
(generated in _links_xxx.r?st)
create a .tags file to jump around in an editor that support
ctags
RST handling with python: reformat/create RST tables
post-process Pandoc’s conversion from DOCX to RST
pre-process Pandoc’s conversion from RST to DOCX
Support in building with WAF (or Makefile)

expand
SimpleTemplate
template files .stpl
graphics files (.tikz, .svg, .dot, .uml, .eps or .stpl thereof, and .pyg)
are converted to .png
and placed into ./_images or <updir>/_images, if there, else into current directory.
a gen file specifies how RST should be generated from source code files (see dcx.py)

The conventions used are shown

by the example produced via rstdcx --rest/--stpl/--over/--ipdt smplfldr
by the documentation sources that can be found at
https://github.com/rstdoc/rstdoc/tree/master/doc

pip install rstdoc installs:

Module
CLI Script
Description

dcx
rstdcx, rstdoc
create .tags, labels and links

fromdocx
rstfromdocx
Convert DOCX to RST using Pandoc

listtable
rstlisttable
Convert RST grid tables to list-tables

untable
rstuntable
Converts certain list-tables to paragraphs

reflow
rstreflow
Reflow paragraphs and tables

reimg
rstreimg
Rename images referenced in the RST file

retable
rstretable
Transforms list tables to grid tables

rstdcx
restructuredText sources are split into two types of files:
main files considered by Sphinx, and included files.
Which of .rest or .rst is main or included is determined
by source_suffix in a <root>/conf.py
or opposite to the extension of the included _links_sphinx.r?st file:

if you have .. include:: /_links_sphinx.rest,
then the main file extension is .rst

rstdoc creates documentation (PDF, HTML, DOCX)
from restructuredText (.rst, .rest) using either

Pandoc
Sphinx
Docutils
configurable

rstdoc and rstdcx command line tools call dcx.py.
which

creates .tags to jump around with the editor
handles .stpl files
processes gen files (see examples produced by –rest)
creates links files (_links_docx.r?st, _links_sphinx.r?st, …)
forwards known files to either Pandoc, Sphinx or Docutils

See example at the end of dcx.py.
It is supposed to be used with a build tool.
make and waf examples are included.

Initialize example tree (add --rstrest to make .rst main and .rest included files):
$ ./dcx.py --rest repo #repo/doc/{sy,ra,sr,dd,tp}.rest files OR
$ ./dcx.py --stpl repo #repo/doc/{sy,ra,sr,dd,tp}.rest.stpl files
$ ./dcx.py --ipdt repo #repo/pdt/AAA/{i,p,d,t}.rest.stpl files
$ ./dcx.py --over repo #.rest all over

Only create .tags and _links_xxx.r?st:
$ cd repo
$ rstdoc

Create the docs (and .tags and _links_xxx.r?st) with make:
$ make html #OR
$ make epub #OR
$ make latex #OR
$ make docx #OR
$ make pdf
The latter two are done by Pandoc, the others by Sphinx.

Create the docs (and .tags and _links_xxx.r?st) with
waf:
Instead of using make one can load dcx.py (rstdoc.dcx) in
waf.
waf also considers all recursively included files,
such that a change in any of them results in a rebuild.
All files can have an additional .stpl extension to use
SimpleTemplate.
You can't use 'macro parameter character #' in math modeYou can't use 'macro parameter character #' in math mode waf –docs docx,sphinx_html,rst_odt
$ #or you provide –docs during configure to always compile the docs

rst_xxx: via
rst2xxx.py
sphinx_xxx: via Sphinx and
just xxx: via Pandoc.

The following image language files should be parallel to the .r?st files.
They are automatically converted to .png
and placed into ./_images or <updir>/_images or else parallel to the file.

.tikz or .tikz.stpl.
This needs LaTex.
.svg or .svg.stpl
.dot or .dot.stpl
This needs graphviz.

.uml or .uml.stpl
This needs plantuml .
Provide either

plantuml.bat with e.g. java -jar "%~dp0plantuml.jar" %* or
plantuml sh script with
java -jar `dirname $BASH_SOURCE`/plantuml.jar "$@"

.eps or .eps.stpl embedded postscript files.
This needs inkscape.

.pyg contains python code that produces a graphic.
If the python code defines a to_svg or a save_to_png function,
then that is used, to create a png.
Else the following is tried

pyx.canvas.canvas from the
pyx library or
cairocffi.Surface from
cairocffi
matplotlib.
If matplotlib.pyplot.get_fignums()>1
the figures result in <name><fignum>.png

The same code or the file names can be used in a .r?st.stpl file
with pngembed() or dcx.svgembed() to embed in html output.
{{!svgembed("egpyx.pyg",outinfo)}}
<%
ansvg=svgembed('''
from svgwrite import cm, mm, drawing
d=drawing.Drawing(viewBox=('0 0 300 300'))
d.add(d.circle(center=(2*cm, 2*cm), r='1cm', stroke='blue', stroke_width=9))
'''.splitlines(),outinfo)
%>
{{!ansvg}}

Conventions
Files

main files and included files: .rest, .rst or vice versa.
.txt are for literally included files (use :literal: option).
templates separately rendered : *.rest.stpl and *.rst.stpl
template included: *.rst.tpl
Template lookup is done in
. and .. with respect to the current file.

with %include('some.rst.tpl', param="test") with optional parameters
with %globals().update(include('utility.rst.tpl'))
if it contains only definitions

Links

.. _`id`: are reST targets.
reST targets should not be template-generated.
The template files should have a higher or equal number of targets
than the generated file,
in order for tags to jump to the template original.
If one wants to generate reST targets,
then this should better happen in a previous step,
e.g. with gen files mentioned above.
References use replacement
substitutions:
|id|.
If you want an overview of the linking (traceability),
add .. include:: _traceability_file.rst
to index.rest or another .rest parallel to it.
It is there in the example project, to include it in tests.
_traceability_file.{svg,png,rst} are all in the same directory.

Link files are created in link roots, which are folders where the first main file
(.rest or .rst) is encoutered during depth-first traversal.
Non-overlapping link root paths produce separately linked file sets.
.. include:: /_links_sphinx.r?st, with the one initial /
instead of a relative or absolute path,
will automatically search upward for the _links_xxx.r?st file
(_sphinx is replaced by what is needed by the wanted target when the docs are generated).
Sphinx conf.py is augmented by configuration for Pandoc and Docutils.
It should be where the input file is, or better at the project root
to be usable with waf.
See the example project created with --rest/stpl/ipdt/over
and the sources of the documentation of
rstdoc.

rstdcx CLI
rstdcx is the same as rstdoc.
Without parameters: creates |substitution| links and .tags ctags for reST targets.
With two or three parameters: process file or dir to out file or dir
through Pandoc, Sphinx, Docutils (third parameter):

html, docx, odt, pdf, … uses Pandoc.
rst_html, rst_odt, rst_pdf, … uses
rst2html, …
sphinx_html, sphinx_pdf, … uses Sphinx.
Sphinx provides a nice entry point via the
sphinx bootstrap theme.

4th parameter onward become python defines usable in .stpl files.
Pdf output needs latex. Else you can make odt or docx and use

win: swriter.exe --headless --convert-to pdf Untitled1.odt
linux: lowriter --headless --convert-to pdf Untitled1.odt

Inkscape (.eps, .svg), Dot (.dot), Planuml (.uml), latex (.tex,.tikz)
are converted to .png into ./_images or <updir>/_images or ‘.’.
Any of the files can be a SimpleTemplate template (xxx.yyy.stpl).
Configuration is in conf.py or ../conf.py.
rstdoc --stpl|--rest|--ipdt|-over create sample project trees.
--stpl with .rest.stpl template files,
--rest with only a doc folder with .rest files,
--ipdt with inform-plan-do-test enhancement cycles
--over with .rest files all over the project tree including symbolic links

Examples
Example folders (see wscript and Makefile there):
rstdoc --rest <folder> [--rstrest]
rstdoc --stpl <folder> [--rstrest]
rstdoc --ipdt <folder> [--rstrest]
rstdoc --over <folder> [--rstrest]
Use --rstrest to produce .rst for the main file,
as .rest is not recognized by github/gitlab,
who also don’t support file inclusion,
so no need for two extension anyway.
Examples usages with the files generated by rstdoc --stpl tmp:
cd tmp/doc
rstdcx #expand .stpl and produce .tag and _links_xxx files

#expand stpl and append substitutions (for simple expansion use ``stpl <file> .``)
rstdcx dd.rest.stpl - rest # expand to stdout, appending dd.html substitutions, to pipe to Pandoc
rstdcx dd.rest.stpl - html. # as before
rstdcx dd.rest.stpl - docx. # expand to stdout, appending dd.docx substitutions, to pipe to Pandoc
rstdcx dd.rest.stpl - newname.docx. # expand template, appending substitutions for target newname.docx
rstdcx dd.rest.stpl - html # expand to stdout, already process through Pandoc to produce html on stdout
rstdcx dd.rest.stpl # as before
rstdcx sy.rest.stpl - rst_html # expand template, already process through Docutils to produce html on stdout
stpl sy.rest.stpl | rstdcx - - sy.html. # appending sy.html substitutions, e.g. to pipe to Pandoc
stpl dd.rest.stpl | rstdcx - - dd.html # appending tp.html substitutions and produce html on stdout via Pandoc
rstdcx dd.rest.stpl dd.rest # expand into dd.rest, appending substitutions for target dd.html
rstdcx dd.rest.stpl dd.html html # expand template, process through Pandoc to produce dd.html
rstdcx dd.rest.stpl dd.html # as before
rstdcx dd.rest.stpl dd.html rst_html # expand template, already process through Docutils to produce dd.html
rstdcx dd.rest.stpl dd.docx # expand template, process through Pandoc to produce dd.docx
rstdcx dd.rest.stpl dd.odt pandoc # expand template, process through Pandoc to produce dd.odt
rstdcx dd.rest.stpl dd.odt # as before
rstdcx dd.rest.stpl dd.odt rst_odt # expand template, process through Docutils to produce dd.odt
rstdcx dd.rest.stpl dd.odt rst # as before
rstdcx . build html # convert current dir to build output dir using pandoc
rstdcx . build sphinx_html # ... using sphinx (if no index.rest, every file separately)

#Sphinx is not file-oriented
#but with rstdcx you need to provide the files to give Sphinx ``master_doc`` (normally: index.rest)
#Directly from ``.stpl`` does not work with Sphinx
rstdcx index.rest ../build/index.html sphinx_html # via Sphinx the output directory must be different

#convert the graphics and place the into _images or <updir>/_images
#if no _images directory exists they will be placed into the same directory
rstdcx egcairo.pyg
rstdcx egdot.dot.stpl
rstdcx egeps.eps
rstdcx egother.pyg
rstdcx egplt.pyg
rstdcx egpygal.pyg
rstdcx egpyx.pyg
rstdcx egsvg.svg.stpl
rstdcx egtikz.tikz
rstdcx egtikz1.tikz
rstdcx eguml.uml

#Convert graphics to a png (even if _images directory exists):
rstdcx eguml.uml eguml.png

#Files to other files:

rstdoc dd.rest.stpl dd.rest
rstdoc dd.rest.stpl dd.html html
rstdoc dd.rest.stpl dd.html
rstdoc sr.rest.stpl sr.html rst_html
rstdoc dd.rest.stpl dd.docx
rstdoc dd.rest.stpl dd.odt pandoc
rstdoc dd.rest.stpl dd.odt
rstdoc sr.rest.stpl sr.odt rst_odt
rstdoc sr.rest.stpl sr.odt rst
rstdoc index.rest build/index.html sphinx_html

#Directories to other directories with out info:

rstdoc . build html
rstdoc . build sphinx_html
Grep with python re in .py, .rst, .rest, .stpl, .tpl:
rstdoc --pygrep inline
Grep for keyword lines containing ‘png’:
rstdoc --kw png
Default keyword lines:
.. {{{kw1,kw2
.. {kw1,kw2}
{{_ID3('kw1 kw2')}}
%__ID3('kw1 kw2')
:ID3: kw1 kw2

rstfromdocx

rstfromdocx: shell command
fromdocx: rstdoc module

Convert DOCX to RST in a subfolder of current dir, named after the DOCX file.
It also creates conf.py, index.py and Makefile
and copies dcx.py into the folder.
See rstdcx for format conventions for the RST.
There are options to post-process through:
--listtable (--join can be provided)
--untable
--reflow (--sentence True, --join 0)
--reimg
rstfromdocx -lurg combines all of these.
To convert more DOCX documents into the same
RST documentation folder, proceed like this:

rename/copy the original DOCX to the name you want for the .rest file
run rstfromdocx -lurg doc1.docx; instead of -lurg use your own options
check the output in the doc1 subfolder
repeat the previous 2 steps with the next DOCX files
create a new folder, e.g. doc
merge all other folders into that new folder

fromdocx.docx_rst_5 creates 5 different rst files with different postprocessing.
See rstreflow for an alternative proceeding.

rstlisttable

rstlisttable: shell command
listable: rstdoc module

Convert RST grid tables to list-tables.

Convert grid tables in a file to list-tables. The result is output to stdout:
$ listtable.py input.rst

Convert several files:
$ listtable.py input1.rst input2.rst
$ listtable.py *.rst

Use pipe (but cat might not keep the encoding):
$ cat in.rst | listtable.py - | untable.py - > out.rst

Options

-j, --join
e.g.002. Join method per column: 0=””.join; 1=” “.join; 2=”\n”.join

rstuntable

rstuntable: shell command
untable: rstdoc module

Convert tables of following format to paragraphs with an ID.
The ‘-’ in ID is removed and the ID is made lower case.
Bold is removed.

ID-XY-00
text goes here

ID-XY-01
text again goes here

If the first entry does contain no word chars or spaces between words,
then the table stays. For a different behavior replace paragraph23.
A file produced from a docx using pandoc or fromdocx.py will
need a pre-processing via rstlisttable to convert grid tables to list-table tables.
This is done in one step with rstfromdocx -lu doc.rst.

rstreflow

rstreflow: shell command
reflow: rstdoc module

Reflow tables and paragraphs in a rst document produced from a docx.
Post-process a docx in this order:
rstfromdocx doc.docx
rstlisttable doc/doc.rst > doc/tmp.rst
rstuntable doc/tmp.rst > doc/tmp1.rst
rstreflow doc/tmp1.rst > doc/tmp2.rst
rstreimg doc/tmp2.rst > doc/tmp3.rst
rm doc/doc.rst
mv doc/tmp3.rst doc/doc.rst
rm doc/tmp*
Check the intermediate results.
Else one can also do inplace:
rstfromdocx doc.docx
rstlisttable -i doc/doc.rst
rstuntable -i doc/doc.rst
rstreflow -i doc/doc.rst
rstreimg -i doc/doc.rst

Note
DOCX to RST using Pandoc
rstfromdocx -lurg doc.rst converts a docx to RST and
does all the post-processing in one step.
It is adviced, though, to compare the output with the original and do some manual
corrections here and there.

rstreimg

rstreimg: shell command
reimg: rstdoc module

Reimg renames the images in the rst file and the files themselves.
It uses part of the document name and a number as new names.
This is useful, if more RST documents converted from DOCX
should be combined in one directory and
the names of the images have no meaning (image13,…).

rstretable

rstretable: shell command
retable: rstdoc module

Transforms list tables to grid tables.
This file also contains the code from
the Vim plugin
vim-rst-tables-py3,
plus some little fixes.
rstdoc is used by the Vim plugin
vim_py3_rst
, which replaces
vim-rst-tables-py3.