kicad/Documentation/s-expressions.txt

S-Expression Support in Kicad
============================================================================
Author:  Dick Hollenbeck
Date:    Jan 2011


An s-expression is a text stream or string, in the same vain as XML, consisting
of a sequence of elements. Each element is either an atom or list. An atom
corresponds to a string, while a list corresponds to an s-expression. The
following grammar represents our definition of an s-expression:

sexpr   ::= ( sx )
sx      ::= atom sxtail | sexpr sxtail | NULL
sxtail  ::= sx | NULL
atom    :: quoted | value
quoted  :: "ws_string"
value   :: nws_string

An atom can either be a quoted string, which is a string containing whitespace
surrounded by double quotes, or a non-whitespace string that does not require
surrounding quotes.

The s-expression syntax used in Kicad uses two quoting/syntax strategies, given
by the needs of the Specctra DSN specification and of our own non-specctra
needs. The Specctra DSN specification is not very clear with regard to quoting
and on top of that there is Freerouter's interpretation, which would actually
supercede anything in the Specctra DSN spec anyway, due to a desire to be
compatible with Freerouter.

We have our own needs, which go beyond those of the Specctra DSN spec, so we
support the two syntaxes or quoting protocols for quoted atoms:

1) Specctra quoting protocol (specctraMode)
2) Kicad quoting protocol    (non-specctraMode)

We can control our own destiny better by having a separately defined mode for
non Specctra DSN files.

To summarize, in specctraMode Freerouter dictates to us what we need to do. In
non-specctraMode, which can be thought of as Kicad mode, we have our own quoting
protocol and can make changes without breaking the specctraMode.

There needs to be agreement between how a file is saved, and how a file is read
back in, in either mode, to fulfill the round-tripping requirements. A file
written using one mode may not necessarily be readable using the other mode,
although it might be.  Just don't count on it.


In Kicad mode:

OUTPUTFORMATTER::Quoted() is the tool to wrap s-expression atoms.
DSNLEXER::NexTok() is basically the inverse function, and reads tokens back in.
These two must agree, so that what is written out comes back in un-altered.

The decision to wrap the string or not is left to the Quoted() function. If the
string is wrapped, it will also escape internal double quotes, \n's and \r's.
Any null string is wrapped in quotes, and so is any string which starts with
'#', so that it is not confused with an s-expression comment.


Kicad S-expression Syntax and Quoting Protocol (non-specctraMode):
==================================================================

*) Some atoms are considered keywords, and constitute a grammar superimposed on
the s-expressions.  All keywords are ASCII and lowercase. International characters
are not to be used here.

*) All Kicad s-expression files are saved using a UTF8 encoding and should
support any international characters in the atoms which are not keywords.

*) DSNLEXER::NextTok() requires that any token be on a single line of input. If
you want to save a multi-line string, Quoted() will automatically escape the \n
or \r for you and put the output on a single line.  It should round-trip fine.

*) There can be escape sequences in a quoted string only. Escape sequences allow
foreign tools to generate byte patterns in the input stream. C style 2 byte hex
codes are supported, and so are 3 byte octal escape sequences. See
DSNLEXER::NextTok() for the full list of escape sequences, by searching file
dsnlexer.cpp for the string "ESCAPE SEQUENCES". Any use of the escape mechanism
must still produce UTF-8 encoded text after the escape handling is applied.

*) Just because an escape sequence is supported on input, does not mean that
OUTPUTFORMATTER::Quoted() must generate such an escape sequence for output. For
example, having true tabs in the s-expression file is OK. So that will not be
escaped on output. Other similar cases exist.

*) Backslash is the escape byte.