Common conversion support code accepts double precision input data for
analog packets. Use the 'double' data type in the CSV input module to
feed sample data to the session bus.
The Saleae Logic exported files (Logic2 digital format) don't contain a
samplerate, so users need to specify the value. For values smaller than
the samplerate that was used during the capture undersampling will take
place. An implementation detail of the input module could result in
incorrect timing of sample values in the session feed. In extreme cases
none of the periods between signal edges qualified for submission. In
that case no sample data was sent to the sigrok session at all.
$ sigrok-cli -i digital_1.bin -I saleae:samplerate=1000
Keep the very timestamp at hand when the last sample data was submitted.
Only advance that timestamp when more sample data was sent. This avoids
the accumulation of timing errors for undersampling scenarios, and does
forward undersampled input data when the user provided sample period has
passed.
This fixes bug #1600.
The previous implementation of the cleanup() routine in the saleae input
module kept user specified options, but lost the previously created list
of sigrok channels. Keep it.
Also make sure that reset() voids the previous copy after grabbing its
value. To not unexpectedly release resources which still get referenced.
This shall unbreak file-reload.
Only return OK from the format match routine when either of the tested
conditions reliably matched. Return an error in all other cases. This
avoids that the Saleae module is "winning a contest" due to even the
weakest condition, and then is not able to handle the input file.
Start the implementation of an input module which covers Saleae Logic's
export files. CSV and VCD are handled by other modules, this one accepts
binary exports for Logic1 digital data (every sample, and when changed),
Logic1 analog data, Logic2 digital data, and Logic2 analog data.
The newer file format versions contain header information and can get
auto-detected, the older formats require a user spec. Some of the file
formats lack essential information in the file content, thus require
another user spec (samplerate for digital data is an example).
The .logicdata file format is unknown, and is not supported. The .sal
format could get added later, but requires local file I/O in the input
module, which current common infrastructure does not provide.
Rephrase the default value for the 'skip' option and the detection of a
user specified value. This is tricky because: Sample numbers are kept in
64bit values. Skip and downsample are fed to formulae so we want them
both to be unsigned. Yet absence of a user spec as well as possible user
values 0 and positive must be told apart. Use all-ones for the default
of "-1" which translates to "first timestamp", users need not be able to
specify that negative value.
Make sure to only downsample 'skip' values when the user specified some.
Which avoids the undesired insertion of huge idle gaps at the start of
the capture. An earlier implementation had to check for -1, this recent
version uses an unsigned number in combination with a boolean flag to
achieve this.
Reword some diagnostics messages, and print the samples count between
timestamps while we are here. Add a check for successful text to number
conversion of timestamp values.
How to reproduce:
$ pulseview -i file.vcd
$ pulseview -i file.vcd -I vcd:downsample=5
$ pulseview -i file.vcd -I vcd:skip=111381600
Example file:
$timescale 1 ns $end
$scope module top $end
$var wire 1 ! d1 $end
$upscope $end
$enddefinitions $end
#111381815
0!
#111381905
1!
#111381990
0!
#111382075
Extend and rework the VCD input module: accept more data types, improve
usability, fix known issues.
Add support for bit vectors (arbitrary width), multi-bit integer values
(absolute 64bit width limit, internal limitation to single precision),
and floating point numbers ('real' in VCD, single precision in sigrok).
Unfortunately sigrok neither has concepts of multi-bit logic channels
nor IEEE-1364 stdlogic values, the input module maps input data to
strict boolean and multiple logic channels. A vector's channels are
named and grouped to reflect their relation. VCD 'integer' types are
mapped to sigrok analog channels. Add support for scoped signal names,
and the re-use of one VCD signal name for multiple variables.
Rework file and text handling. Only skip pointless UTF-8 BOMs before
file content (not between sections). Handle lack of line termination at
the end of the input file. Process individual lines of input chunks,
avoid glib calls when they'd result in malloc pressure, and severely
degrade performance. Avoid expensive string operations in hot loops.
Rearrange the order of parse steps, to simplify maintenance and review:
end of section, new section, timestamp, data values, unsupported. Flush
previously queued values in the absence of a final timestamp. Unbreak
$comment sections in the data part. Apply stricter checks to input data,
and propagate errors. Avoid silent operation (weak warnings can go
unnoticed) which yields results that are unexpected to users. Unbreak
the combination of 'downsample' with 'skip' and 'compress'. Reduce noise
when users limit the number of channels while the input file contains
more data (keep a list of consciously ignored channels). Do warn or
error out for serious and unexpected conditions.
Address minor issues. Use common support for datafeed submission. Keep
user specified options across file re-load. Fixup data type nits, move
complex code blocks into separate routines. Sort the groups of routines,
put helpers first and concentrate public routines at the bottom. Extend
the builtin help text. Update comments, update the copyright for the
non-trivial changes.
Fixes bug #776 by adding support for bit vectors.
Fixes bug #1476 by flushing most recently received sample data.
Input modules often find themselves in the situation where sample data
was received and could be sent to the session bus, but submission should
get deferred to reduce the number of send calls and provide larger data
chunks in these calls. Introduce common support code for buffered sample
data submission (both logic and analog), provide a simple alloc, submit,
flush, and free API.
The input/binary module chops raw input data into chunks and sends these
to the session feed. The total size of input chunks got aligned to the
unit size, the session feed output didn't. Make sure to align session
packets with the input data's unit size, too.
This fixes bug #1582.
The @brief keyword is not needed since JAVADOC_AUTOBRIEF is enabled in the
Doxygen configuration. Remove a remaining "DIAG" prefix in a debug
message, the output already is filtered according to the log level, and
prefixed by the module name.
Don't clobber the user provided samplerate (specified by input module
options). This allows to re-determine the samplerate from a potentially
changed file on disk upon reload.
The list of previously created channels is kept across file reloads. We
also need to keep (or re-create) the list of channels that are used for
datafeed submission of analog data.
Release more allocated resources in the .cleanup() routine, and do reset
internal state such that a .reset() thus .cleanup() then .receive() call
sequence will work. This code path is taken for re-import of files (see
bug #1241, CSV was affected).
Slightly unobfuscate the "end of current input chunk" marker in the data
processing loop. Make the variable's identifier reflect that it's not a
temporary, but instead something worth keeping around until needed again.
Unbreak the calculation of line numbers in those situations where input
chunks (including previously accumulated unprocessed data) happens to
start with a line termination. This covers input files which start with
empty lines, as well as environments with mutli-byte line termination
sequences (CR/LF) and arbitrary distribution of bytes across chunks.
This fixes bug #968.
Accept when there is no line termination in the current input chunk. We
cannot assume that calling applications always provide file content in
large enough chunks to span complete lines. And any arbitrary chunk size
which applications happen to use can get exceeded by input files (e.g.
for generated files with wide data or long comments).
Mention the required synchronization of default option values and format
match logic in a prominent location where options are discussed.
Update the TODO list for the CSV input module. Mixed signal handling got
fixed by rearranging channel creation. Samplerate can get determined
from timestamp columns already. The double data type for analog data
remains.
The previous implementation incompletely handled arbitrary data type
oders in mixed signal input files. Rearrange the logic such that all
format specs get parsed first, then all channel creation details get
determined, then all channels get created. It appears to be essential
that all logic channels get created first, resulting in index numbers
starting at 0 and addressing the correct position in bitfields. The
analog channels get created after all logic channels exist. Adjacent
number ranges for channel types also results in more readable logic in
other locations.
This was tested with -I csv:column_formats=t,2a,l,a,x4,a,-,b3 example
data, and works as expected.
Implement .format_match() support in the CSV input module. Simple
multi-column files will automatically load without an "-I csv" spec.
Non-default option values still do require the module selection before
options can get passed to the module.
Try to balance a compact format and completeness/accuracy of content for
builtin help texts for the CSV input module's options. Assume a technical
audience (this is signal analysis software after all).
Rename a few internal identifiers which help organize the list of options
and their help texts. Too short names became obscure.
Accept 't' format specs for timestamp columns. Automatically derive the
samplerate from input data when timestamps are available and the user
did not provide a rate. Stick with a simple approach for robustness
since automatic detection is easy to override when it fails. This
feature is mostly about convenience.
The previous implementation open coded type checks by comparing an enum
value against specific constants. Which was especially ugly since there
are multiple types which all are logic, and future column types neither
get ignored nor have channels associated with them.
Improve readability and robustness by adding helpers which check classes
(ignore, logic, analog) instead of the multitude of -/x/o/b|l/a variants
within the classes.
Also comment on the order of channel creation, and how to improve it.
Expand the developer comment that's inline in the source file. There is
enough room to explain things. Try to come up with a "one-line" help text
that is precise yet compact, and does inform the user of available format
choices including modifiers. Chances are that longer descriptions start
reducing the usefulness of the help or the visibility of options when
users are in a hurry. Those who care can access the manual.
Mark more options as obsolete, and mention more default values in the
builtin help text. Also tweak a comment on getting channel names from
header lines.
Support for mixed signal CSV input data is desirable and should be
possible. The current implementation just happens to not fully cope with
arbitrary mixes of data types in columns yet. Add a quick workaround,
but also a TODO item to properly address the topic later.
Extend the CSV input module which was strictly limited to logic data so
far. Add support for analog data types. Implement the 'a' column format,
and feed analog data to the session bus.
This implementation feeds data of individual analog channels to the
session bus in separate packets each. This approach was found to work
most reliably, not all recipients support the submission of multiple
samples for multiple channels in a single packet.
A fixed 'digits' value is used. This needs to get addressed later.
Local experiments suggest that the 'double' data type for analog data
can result in erroneous visual presentation (observed with sigrok-cli).
Use 'float' for now, until the issue is understood and got fixed.
Support for double is prepared internally and is easily enabled.
Address several minor nits. Eliminate unneeded variables. Update text to
number conversion comments including wildcard handling. Remove empty
lines in init() which used to spill out a set of lines which all do the
same thing (evaluate a set of options) and shall belong together.
Move the creation of logic channels to the location where formats fields
get iterated, and column processing details get derived. This reduces a
lot of redundancy, and simplifies the addition of more data formats.
Update the list of TODO items at the top of the CSV input module's
source. Text line handling (counting line numbers) got fixed. Adding
support for analog channels was prepared, as are timestamp columns.
Rename the CSV input module's option keywords. To better reflect their
purpose, and for consistency across the rather complex set of options
and how they interact. Rearrange the list of options (not that the order
matters from the outside, but it's good to have during maintenance).
Update builtin help texts which will show up in applications, as well as
the source code comments which discuss these options in greater detail.
Would be nice to have a "general" help text for input modules which is
not tied to one single option, to provide an overview or use examples.
Arrange the option keys, short and long help texts such that the source
better reflects the applications' screen layout. To better support
future maintenance, again.
Consistently separate multi-work keywords for improved readability.
Prefer underscores over dashes for consistency with common keys in
shared infrastructure in other project sources (device options, MQ
items, etc).
Extend the "column-formats" option support in the CSV input module to
also support wildcards and automatic channel count detection. Move the
format string interpretation to the location where the first data line
or the optional header line are seen. Map the simple options (single
column number and channel count, or first column number and optional
channel count) to a format string, to unify internal code paths. Remove
code paths for the previous specific yet limited scenarios.
Rephrase the condition which keeps executing the "initial receive"
phase. The text line termination sequence gets derived from the first
complete text line, but other essential information is only gathered
later, potentially after skipping a large (user specified) amount of
input data. Keep checking for this essential setup data until data or
the header actually were seen, before regular processing of input data
starts.
Extend the CSV input module, introduce support for the "column-formats="
option. This syntax can express the previous single- and multi-column
semantics, as well as any arbitrary order of to-get-ignored, and single-
and multi-bit columns in several formats.
The previous "simple" keywords for single and multi column modes still
are in place, it's yet to get determined whether to axe them. Depends on
whether users can handle the format strings for these simple cases.
The previous implementation allowed CSV input files to use any line
termination in either CR only, LF only, or CR/LF format. The first EOL
was searched for and was recorded, but then was not used. Instead any of
CR or LF were considered a line termination. "Raw data processing" still
was correct, but line numbers in diagnostics were way off, and optional
features like skipping first N lines were not effective. Fix that.
Source code inspection suggests the "startline" feature did not work at
all. The user provided number was used in the initial optional search
for the header line (to get signal names) or auto-determination of the
number of columns. But then was not used when the file actually got
processed.
Reduce "state" in the CSV input module's context. Stick with variables
that are local to routines when knowledge of details need not be global.
Really base the processing of a column's input text on the column's
processing information which was gathered in the setup phase.
Rename few identifiers, to explicitly refer to logic channels (the only
currently supported data type of the CSV input module). Cease feeding
logic data to the session bus when there are no logic channels at all
(currently not really an option). Prepare for simpler dispatching of
parse routines should more data types get added in a future version.
Reduce some "clutter" (overly fragmented stuff that should go together
since it forms logical groups and is not really standalone). Address a
few more minor style nits (sizeof() redundancy, "seemingly inverse"
string comparison phrases).
Improve the code paths which determine logic channels' names from an
optional CSV file header line. Strip optional quotes from the column's
input text (re-use a SCPI helper routine for that). Also use the channel
name for multi-bit fields, append [0] etc suffixes in that case. Comment
on the manipulation of input data, which is acceptable since that very
data won't get processed another time in another code path.
Rephrase the CSV input module's implementation such that generic support
to "process a column" becomes available. All columns of an input file's
text line get inspected, a column can either get ignored, or converted
to logic data. A future version can then remove the current limitations
of single- and multi-column modes (either one single multi-bit cell, or
multiple single-bit cells which must be adjacent).
Combine the bin/oct/hex parse routines into one routine which handles up
to four bits per input number digit with common logic. Availability of
more data than channels (according to user specs) is not fatal.
Drop the counter intuitive "first-channel" option, use "first-column"
instead. Warn when comment leader and column separator are identical
(was silent before, may be unexpected). Extend diagnostics and address
minor readability nits, update comments. Rephrase logic channel name
assignment.
Use simple scalar options to derive generic processing details: Either
'single-column' and 'numchannels' are required, with an optional
'format' spec (resulting in single-column mode). Or 'first-column' with
an optional 'numchannels' (multi-column mode with fixed format, using
all available columns by default). The default is multi-column mode with
one logic channel per column and spanning all columns on a text line.
Don't clobber the value of the user provided 'header' option. Use a
separate flag to track whether the header line was seen before, or
needs to get skipped when it passes by.
Move the communication of the samplerate meta packet to the very spot
where logic sample data gets sent. This allows to optionally determine
late the samplerate, potentially from input data instead of user specs.
Move the helper routines which arrange for the data feed to an earlier
spot, so that references resolve without forward declarations. Rename
routines to reflect that they deal with logic data.
Slightly unobfuscate column text to logic data conversion, and reduce
redundancy. Move sample data preset to a central location.
Rephrase error messages, provide stronger hints as to why the input text
for a conversion was considered invalid.
The previous implementation assumed that in multi-column mode each cell
communicates exactly one bit of input (a logic channel). But only the
first character got tested. Tighten the check, to cover the whole input
text. This rejects fully invalid input, as well as increases robustness
since multi-bit input like "100" was mistaken as a value of 1 before.
Add documentation to the bin/hex/oct text parse routines, and move the
bin/hex/oct dispatcher to the location where its invoked routines are.
Stick with a TODO comment for parse_line() to reduce the diff size.
The parse_line() routine is rather complex, optionally accepts an upper
limit for the number of columns, but unconditionally assumes a first one
and drops preceeding fields. The rather generic n and k identifiers are
not helpful.
Use the 'seen' and 'taken' names instead which better reflect what's
actually happening. Remove empty lines which used to tear apart groups
of instructions which are strictly related. This organization neither
was helpful during maintenance.
Accept when comments are indented, trim the whitespace from text lines
after stripping off the comment. This avoids the processing of lines
which actually are empty, and improves robustness (avoids errors for a
non-fatal situation). Also results in more appropriate diagnostics at
higher log levels.
The CSV input module has grown so many options, that counting them by
hand became tedious and error prone. Eliminate the magic numbers in the
associated code paths.
This also has the side effect that the set is easy to re-order just by
adjusting the enum, no other code is affected. Help text and default
values is much easier to verify and adjust with the symbolic references.
[ see 'git diff --word-diff' for the essence of the change ]
Use size_t for things that get counted: column indices, channel numbers
(line numbers already used size_t). De-anonymize an enum to avoid 'int'
where it gets referenced. Adjust printf(3) format strings. Get unsigned
values from option lookups (stick with 32bits, should be acceptable for
spreadsheet columns and channel counts).
Address other minor nits while we are here: Also terminate the last item
in an enum declaration. Add a doxygen comment for parse_line(). Rename a
parameter to achieve tabular doc text layout.
Rephrase the #include statements in the CSV input module. "config" is
not a system header but is provided by the application source code.
Separate the config and system and application groups (their order is
essential). Alpha-sort the files within their group for simplified
maintenance.
Gerhard Sittig
Uwe Hermann
Andreas Sandberg
Soeren Apel