namespace SCH { /** @mainpage This file describes the design of a new Distributed Library System for KiCad's EESCHEMA. Many of the concepts can be adapted with modest modification to PCBNEW also, in the future. @author Dick Hollenbeck <dick@softplc.com> @date October 2010 - January 2011 @section intr_sec Introduction Schematic <b>parts</b> are frequently needed to complete a circuit design schematic. Computer data entry of parts can be a rate limiting step in the design of an overall PCB. Having ready made access to all needed parts in a design significantly improves the productivity of a circuit designer. Sharing parts within an organization is one step in the right direction, but there is opportunity to share across organizational boundaries to improve productivity even more. Using a part that someone else in another organization has already entered into the computer can eliminate the first data input process for that part. The more complicated the part and the board, the larger the positive impact on productivity because the larger the time savings. <p> Sharing parts within an organization is best done by directly accessing a known internal source for those parts, say on a company network. Sharing parts across organizational boundaries is best done using the Internet in real-time. Having the ability to search for a part based on arbitrary search criteria can speed up the pace at which new parts are found and used. <p> Electronic component manufacturers need and look for ways to differentiate their products from their competitors. With this Distributed Library System facility in KiCad, one way for manufacturers to differentiate themselves and their parts is to publish a part library on the Internet and save their customers the work of doing the data entry of the part into the KiCad design system. <p> Maintaining a comprehensive part library is a fairly labor intensive activity. New parts come into the market everyday. By being able to publish a superior library on the Internet, it may be possible to make a for profit business out of doing this. The KiCad eco-system would benefit should this happen, and there could even be competition between such businesses. Or there can be library specializations or niches. <p> Often a found part is close to what is needed but not exactly what is needed. This Distributed Library System design incorporates the concept of part inheritance using a part description language called <b>Sweet</b>. Sweet is based on s-expression syntax. Inheritance is the ability to incrementally change an existing part without completely re-designing it. It is sometimes easier to modify an existing part than it is to create the new part entirely from scratch. <p> This Distributed Library System design will have the capability to significantly benefit the KiCad eco-system, and that should mean expanding the numbers of users and contributors to the project, and hopefully making for a better KiCad tool-set for all. @section definitions Definitions Only new terms or changes in the definition of terms are given here. <dl> <dt>S-Expression</dt><dd>This is a syntactical textual envelop in the same vain as XML. It may be used to express any number of domain specific grammars. It uses parentheses to indicate the start and end of an element. A domain specific grammar is a set of rules that dictate what keywords may be used, and in what context. A grammar also establishes the allowed places and types of constants. There can be any number of grammars which all use s-expressions to hold the individual elements within the grammar. A grammar is at a higher level than s-expressions, in the same way that a sentence will have grammatical rules which are at a higher level than the rules used to spell words. Technically, grammars nest within grammars. So once you are inside a grammatical element, it will have its own set of rules as to which nested elements it may hold, and once you enter one of those nested elements, then that nested element's grammar pertains, etc.<p> In the case of the grammar for a part, the grammar itself is being given the name Sweet. The name does not extend to the grammar for the schematic, only the part grammar.</dd> <dt>Schematic</dt><dd>This consists of one or more sheets and will be different in three ways from existing schematics. <ul> <li>All sheets will be in one file, thus the entire schematic is in one file. <li>The schematic file will have its own s-expression grammar. <li> There will be a <b>parts list</b> within the schematic, and within the parts list will be <b>all</b> the parts for the schematic. yes <b>all</b> of them. See class PARTS_LIST.</ul> Within the sheets of the schematic will be components.</dd> <dt>Component</dt><dd>A component is an instantiated part. The keyword for component is (comp). A component does not have any of its own properties other than: <ul> <li>rerence designator <li>part pointer or reference into the parts list <li>location <li>rotation <li>stuff i.e. DNS or do stuff the part <li>visual textual effect overrides </ul> Note that the (comp) may not have any properties or fields of its own, and that it may not exist without a corresponding part in the parts_list. A reason for this is to ensure that a BOM can be made simply from the parts_list.</dd> <dt>Component, again for good measure.</dt><dd>A component is an instantiation of a part. A component exists within a schematic which has a parts list containing the part from which the component is instantiated. A component has a unique reference designator, part ref, its own location, orientation, stuff/DNS, and text attributes but <b>not</b> its own text fields/strings (other than reference designator). The part which is instantiated must exist in the parts list of the same schematic.</dd> <dt>Inheritance</dt><dd>Is the ability to mimic form and function from another entity. In our case we use it only for parts. One part may "inherit from" or "extend" another single part.</dd> <dt>Part</dt><dd>A part is a symbolic schematic circuit element found within an EESCHEMA library (or within a parts list). It is re-usable and may be instantiated more than once within a schematic. For it to be instantiated, it must be copied or inherited into the parts list of the instantiating schematic. If inherited into the parts list, then only a concise reference is needed into the originating library. If instead it is copied into the parts list, then the part is fully autonomous and need have no reference to its original copy.</dd> <dt>Parts List</dt><dd>A parts list, keyword (parts_list), is an entirely new construct. It exists within a schematic and is the complete set of parts used within a particular schematic. Each schematic has exactly one parts list contained within it. A parts list is also a library source and a library sink for the current schematic. A parts list in any schematic may also be a library source for any other schematic, but not a library sink. The parts list construct makes it almost wholly unnecessary to write to other types of library sinks.</dd> <dt>Library</dt><dd>A library is no longer a file. It is a memory cache of parts, consistent with the normal definition of memory cache. Each library is backed up with a <b>library source</b>. In rare cases, some libraries may also have a <b>library sink</b>.</dd> <dt>Library Source</dt><dd>A library source is an abstract read only repository of parts. The repository itself might exist on the moon. The difference between a library source and a library sink is that a source is a readable entity.</dd> <dt>Library Sink</dt><dd>A library sink is an abstract place that parts can be written to for future reading. The difference between a library source and a library sink is that a library sink is a writable entity.</dd> <dt>Symbol</dt><dd>The term "symbol" is not used in a specific way in this document. There is no symbol in any of the grammars, so use of it on the developers list will not be understood without explanation. Of course it is possible to have multiple parts all extend a common base part, and you can think of the base part as having most if not all the graphical lines for any derivatives. But we do not need to use the term symbol to describe that relationship, the term "part" is sufficient.</dd> <dt>LPID</dt><dd>This stands for "Logical Part ID", and is a reference to any part within any known library. The term "logical" is used because the contained library name is logical, not a full library name. The LPID consists of 3 main portions: logical library name, part name, and revision number.</dd> <dt>Library Table</dt><dd>This is a lookup table that maps a logical library name (i.e. a short name) into a fully specified library name and library type. An applicable library table consists of rows from (at least) two sources:<ol> <li>A schematic resident library table. <li>A personal library table. </ol> These rows from the two sources are conceptually concatonated (although they may not be physically concatonated in the implementation, TBD). The schematic resident rows take presedence over the personal library table if there are logical library names duplicately defined. (Or we will simply ask that any remote (i.e. public) libraries use uppercase first letters in logical names, TBD.) <p> Eventually there will be an external publicly available internet based logical library table also, but this will need to be glued down at a hard coded URL that we have control over. The internet based library table allows us to advertise remote libraries without having to issue an update to KiCad.</dd> <dt>Query Language</dt><dd>This is a means of searching for something that is contained within a container. Since some library sources are remote, it is important to be able to ask the library source for a part that matches some criteria, for performance reasons.</dd> </dl> @section changes Required Changes In order fulfill the vision embodied by this Distributed Library System design, it will be necessary to change many APIs and file formats within EESCHEMA. In fact, the entire schematic file format will be new, based on s-expressions, the schematic grammar, and the Sweet language for parts. Here are some of the changes required: <ul> <li> All sheets which make up a schematic will go into a single s-expression file. The multiple sheet support will still exist, but all the sheets for a single schematic are all in a single file. <li> A "library" is a collection of "parts". The unit of retrieval from a library is a part as a textual string in the Sweet language. Sweet is a particular "grammar" expressed in s-expression form, and can be used to fully describe parts. Because EESCHEMA does not actually see a "library file", (remember, EESCHEMA can only ask for a part), the actual file format for a library is no longer pertinent nor visible to the core of EESCHEMA. The unit of retrieval from the API is the part, so EESCHEMA gets an entire part s-expression and must then parse it as a RAM resident Sweet string. <li>EESCHEMA knows of no library files, instead there is a library API which abstracts the actual part storage strategy used within any library implementation. The API can be implemented by anyone wanting to provide a library under a given storage strategy. The API provides a storage strategy abstraction in classes LIB_SOURCE and LIB_SINK. The actual storage strategy used in any particular library implementation is not technically part of the conceptual core of EESCHEMA. This is an important concept to grasp. Eventually the library implementations may be jetisoned into a plug-in structure, but initially they are statically linked into EESCHEMA. Should the plug-in strategy ever get done, the boundary of the plug-in interface will remain the C++ library API as given here (mostly in class LIB_SOURCE). The only reason to introduce a plug-in design is to allow proprietary closed source library implementations, and this could eventually come about if a part vendor wanted to provide one for the KiCad project. If a Texas Instruments type of company wants to maintain a KiCad library, we will be positioned to accommodate them. Until then, the LIB_SOURCE implementations can be statically linked into EESCHEMA and there is no conceptual disruption either way. <li> Most library implementations are read only. There are only two library types that are writable, the "dir" type, and the "parts list". All other types are read only from the perspective of the API. Stuffing those read only libraries and maintaining them will be done using the normal update mechanisms pertinent to the library's respective type of repository. The most common place to do incremental enhancements to a part before using it is not in the external library, but now in the parts list with this new design. <li> The design will support classical clipboard usage. The part in the Sweet language can be placed onto the clipboard for use by other applications and instances of EESCHEMA. Eventually larger blocks of components may also be supported on the clipboard, since the Sweet language allows these blocks to be descributed textually in a very readable fashion. (Clipboard support beyond part manipulation is not currently in this revision of the design however, it can be a future separate enhancement. Perhaps someday complete sheets may be passed through the clipboard.) <li> The cumulative set of required changes are significant, and are tantamount to saying that EESCHEMA will need its part handling foundations re-written. A conversion program will convert everything over to the new architecture. The conversion program can simplify things by simply putting all schematic parts into a parts list within each schematic. <li> An Internet connection is required to use some of the library sources. It will be possible to omit these library sources and run KiCad by doing a configuration change. Eventually, some library sources will spring up and will not technically be part of the KiCad project, so they will remain remote, but fully usable to those with an internet connection and permission from the library source's owner. <li>By far, even as radical as the distributed library concept is, complete with remote access, the most significant conceptual change is the introduction of the <b>parts list</b>. This is a special library that exists in a schematic, and is the complete record of all parts used within that same schematic. It is impossible to put a component into a schematic without that component's part first existing within the parts list of that schematic. <li> Because of inheritance, multi-body-form parts, alternate body styles, see also references, and other needs, it is necessary to have a <b>part reference mechanism</b>. A component has to reference a part, the one it "is". A part has to be able to reference another part, either in the same library or elsewhere. Enter the Logical Part ID, or LPID to serve this need. An LPID consists of a logical library name, a part name, and an optional revision. It is used to reference parts, from anywhere. If the reference is from a sheet's component, then the logical library name of the LPID is not needed. Why? Well if you've been paying attention you know. A comp can only be based on a part that exists within the parts_list of the same schematic in which it resides. Likewise, a part within any library that references another part in that <b>same</b> library will also omit the logical library name from the LPID, and it must omit it. Why? Well because it makes renaming the library easier, for one. Two, the logical library name is only a lookup key into a "library table". The library table maps the logical library name into an actual library source [and sink, iff writable]. See LIB_SOURCE and LIB_SINK. <p> In the case of the component referencing the part that it "is", there is no revision number allowed in the LPID. This is because that reference is to the part in the parts list, and the parts list only holds a single revision of any part (where "revision" is what you understand from version control systems). <li> There needs to be a new query language designed and each library source needs to support it. See LIB_SOURCE::FindParts() for a brief description of one. </ul> @section philosophy Design Philosophies <p> Class names are chosen to be as concise as possible. Separate namespaces can be used should these same class names be needed in both EESCHEMA and PCBNEW (later). However, this design does not yet address PCBNEW. Suggested namespaces are SCH for EESCHEMA, and PCB for PCBNEW. <p> Since most if not all the APIs deal with file or non-volatile storage, only 8 bit string types are used. For international strings, UTF-8 is used, and that is what is currently in use within the KiCad file storage formats. <p> The typedef <b>STRINGS</b> is used frequently as a holder for multiple std::strings. After some research, I chose std::dequeue<STRING> to hold a list of STRINGs. I thought it best when considering overall speed, memory fragmentation, memory efficiency, and speed of insertion and expansion. <p> A part description language is introduced called <b>(Sweet)</b>. It supports inheritance and its syntax is based on s-expressions. <p> Since a part can be based on another part using inheritance, it is important to understand the idea of library dependencies. A part in one library can be dependent on another part in another library, or on another part in the same library as itself. There are several library sources, some far away and some very close to the schematic. The closest library to the schematic is the <b>(parts list)</b> class PARTS_LIST. Circular dependencies are not allowed. All dependencies must be resolvable in a straight forward way. This means that a part in a remote library cannot be dependent on any part which is not always resolvable. <p> NRVO described: http://msdn.microsoft.com/en-us/library/ms364057%28VS.80%29.aspx Even with NRVO provided by most C++ compilers, I don't see it being as lean as having class LIB keep expanded members STRING fetch and STRINGS vfetch for the aResults values. But at the topmost API, client convenience is worth a minor sacrifice in speed, so the topmost API does return these complex string objects for convenience. So there is a different strategy underneath the hood than what is used on the hood ornament. When aResults pointer is passed as an argument, I won't refer to this as 'returning' a value, but rather 'fetching' a result to distinguish between the two strategies. <p> Functions named as lookupSomething() or LookupSomething() are to be understood as "find and load if needed". This is different than a simple find operation only, in that an implied load operation is also done, but only if needed. @section architecture Architecture This document set later shows some <b>library sources</b> derived from class LIB_SOURCE. A library source is the backing to a library. The class name for a library in the new design is LIB. <p> <IMG SRC="../drawing.png" ALT="You should see design architecture here"> */ /** * @defgroup string_types STRING Types * Provide some string types for use within the API. */ /** * @defgroup exception_types Exception Types * Provide some exception types for use within the API. */ /** * Class HTTP_LIB_SOURCE * implements a LIB_SOURCE to access a remote document root repository using http protocol. */ class HTTP_LIB_SOURCE : public LIB_SOURCE { friend class LIB_TABLE; ///< constructor the LIB uses these functions. protected: /** * Constructor ( const STRING& aHttpURL ) * sets up a LIB_SOURCE using aHttpURI which points to a subversion * repository. * @see LIB_TABLE::LookupPart() * * @param aHttpURL is a full URL of a document root repo directory. Example might * be "http://kicad.org/libs" * * @param aOptions is the options string from the library table. It can be any * string that the LIB_SOURCE can parse and understand, perhaps using comma separation * between options. Options and their syntax is LIB_SOURCE implementation specific. */ HTTP_LIB_SOURCE( const STRING& aHttpURL, const STRING& aOptions ) throws( IO_ERROR ); }; /** * Class SVN_LIB_SOURCE * implements a LIB_SOURCE to access a [remote or local] subversion repository * using subversion client protocol. */ class SVN_LIB_SOURCE : public LIB_SOURCE { friend class LIB_TABLE; ///< constructor the LIB uses these functions. protected: /** * Constructor SVN_LIB_SOURCE( const STRING& aSvnURL ) * sets up a LIB_SOURCE using aSvnURI which points to a subversion * repository. * @see LIB_TABLE::LookupPart(). * * @param aSvnURL is a full URL of a subversion repo directory. Example might * be "svn://kicad.org/repos/library/trunk" * * @param aOptions is the options string from the library table. It can be any * string that the LIB_SOURCE can parse and understand, perhaps using comma separation * between options. Options and their syntax is LIB_SOURCE implementation specific. */ SVN_LIB_SOURCE( const STRING& aSvnURL, const STRING& aOptions ) throws( IO_ERROR ); }; /** * Class SCHEMATIC_LIB_SOURCE * implements a LIB_SOURCE in by reading a parts list from schematic file * unrelated to the schematic currently being edited. */ class SCHEMATIC_LIB_SOURCE : public LIB_SOURCE { friend class LIB_TABLE; ///< constructor the LIB uses these functions. protected: /** * Constructor SCHEMATIC_LIB_SOURCE( const STRING& aSchematicFile ) * sets up a LIB_SOURCE using aSchematicFile which is a full path and filename * for a schematic not related to the schematic being editing in * this EESCHEMA session. * @see LIB_TABLE::LookupPart(). * * @param aSchematicFile is a full path and filename. Example: * "/home/user/kicadproject/design.sch" * * @param aOptions is the options string from the library table. It can be any * string that the LIB_SOURCE can parse and understand, perhaps using comma separation * between options. Options and their syntax is LIB_SOURCE implementation specific. */ SCHEMATIC_LIB_SOURCE( const STRING& aSchematicFile, const STRING& aOptions ) throws( IO_ERROR ); }; /** * Class PARTS_LIST * is a LIB which resides in a SCHEMATIC, and it is a table model for a * spread sheet both. When columns are added or removed to/from the spreadsheet, * this is adding or removing fields/properties to/from ALL the contained PARTS. */ class PARTS_LIST : public LIB { public: /** * Function GetModel * returns a spreadsheet table model that allows both reading and writing to * rows in a spreadsheet. The UI holds the actual screen widgets, but * this is the table model, i.e. the PARTS_LIST is. */ SPREADSHEET_TABLE_MODEL* GetModel(); }; } // namespace SCH /// @todo remove endsWithRev() in favor of EndsWithRev(), find home for it. /// @todo add simple unit test infrastructure. /// @todo review: http://www.sfml-dev.org/tutorials/1.2/graphics-wxwidgets.php // EOF