Home Reference Source Repository

CanSat Data Capture library

Greenkeeper badge Dependency Status Build Status Build status Codecov Status Codacy Badge Commitizen friendly

A library that handles all CanSat ground station-related duties.

Installation

$ npm install @cansat-icarus/capture-lib

If you need just part of project please open an issue. This project could use some splitting.

Features

  • Modular code base for easier coding and maintenance.
  • Base64 packet encoding that minimizes packet size, diminishing the time window where interference may appear.
  • CRC32 checksums sent with every packet to ensure no bad data is mistakenly registered in a ground station.
  • Scores every packet based on the CRC checksum and simple heuristics (eg. a temperature sensor shouldn't suddenly report a temperature 30ºC higher than the previous packet few seconds ago)
  • Automagically connects to a backend server through a WebSocket.
  • Backend data(DB) replication Station->backend
  • Remote control capabilities (exposed to the backend) exposing: a REPL with access to the wrapper/station, CLI access to the host machine (something like tty.js) and possibly a VNC-like full-screen control of the host. (not yet at least)

Documentation

API documentation is available here, built using ESDoc by Travis CI. If you want to build it locally, you can by running npm run docs at the root of the repository. Output will be inside a folder named docs. An overview of each module/component is available in this file.

Included components

Serial

A node-serialport abstraction, allowing to change the port path without recreating the instance and keeps track of state changes (open/close/...).

Base64 Packet Encoding

Base64 is an established standard for encoding data using only alphanumeric characters, '/', '+', and '='. It's easy to manually spot interference and it translates to a reduction to ~19.7% of the original packet size. It's efficient, it's quick (quicker than serializing all data to strings like it's usual) and it works.

Parser

Generic class for binary data parsing. No assumptions about the packet, just a helper interface for constructing an object from raw data. Along with it, a preconfigured instance ready for telemetry, info and settings packets from the CanSat. Re-implemented as IcarusParser to include our mission's packet fields.

Classifier

Not an AI. An algorithm that calculates each packet's score: 60% for the CRC checksum, 40% for the heuristics. The station score reflects reception quality in the recent past (The nth most recent packet accounts for at least 1/(n+1) of the station score).

Data Handler

A function that decodes the data, parses it, gives it a score, and puts it in the db. Just a high-level utility piping everything together.

Backend

Handles the WebSocket (Socket.IO) connected to the backend. Automatically attempts connection, reports metrics (station score, received packet count...), enables the replicator (copies local db to the backend asap, updating it live or as soon as the connection returns) and allows for remote control of the station.

Station

Brings all the other modules together into one class. Wrappers should only need to access the station class, creating an instance and switching one implementation of a module (ex: DB, Backend) for another. A common place for triggering actions, changing settings and listening to events.

Building, testing and all the workflow things

Source code uses ES2017 and some experimental syntax. So, until no supported nodejs versions remain that cannot run our code, babel is used for transpiling the source.

My advice: change things and when you're done, lint, run unit tests, generate and check the new documentation (if changed), check test coverage (make sure it does not drop). And while you're fixing bugs (a unit test failing), run test:watch for help. For always-running automatic linting, use a xo plugin in your editor.

$ yarn run clean
$ npm run clean       # remove and recreate output directory (dist)

$ yarn run lint
$ npm run lint        # lint source using xo

$ yarn run build
$ npm run build       # transpile source code into ./dist

$ yarn run build:watch
$ npm run build:watch # transpile source code into ./dist (and watch for changes in code). Does not work in PowerShell/CMD.

$ yarn run docs
$ npm run docs        # generate documentation

$ yarn run coverage
$ npm run coverage    # run unit tests and generate code coverage report

$ yarn run test
$ npm run test        # run unit tests (with code coverage) and run the linter

$ yarn run test:watch
$ npm run test:watch  # run unit tests and the linter everytime the source changes

Contributing

Just fork the project, do what you want (while trying to follow the advice below), and send a pull request. Or file an issue if your a bit lost... For contacting us, you have links to social media profiles and email in our website.

Some advice for a good pull request:

  • If you're adding new features, write unit tests for them.
  • Run the unit tests before committing.
  • If the tests fail, and you're doing it on purpose, you better have a good explanation! Having a valid reason, change the unit tests to what they should be.
  • Besides writing unit tests, make sure to document any new features and changes to normal behavior.
  • Short, informative commit messages are good. See here why they matter and how to write good ones. Commitizen can help you stick to our commit message style.

License

MIT © 2016 André Breda

References

References

Class Summary

Static Public Class Summary
public

Handles connection with the backend.
public

Calculates packet and station scores.
public

A wrapper over the ExponentialBackoffStrategy from the backoff module.
public

The parser for Team Icarus' CanSat.
public

Parser creation helper class.
public

Handles replication between a local and remote PouchDB database.
public

A node-serialport wrapper that keeps track of port state and allows from on-the-fly port path/name changes (automatically attaches/detaches event listeners and keeps config).
public

Handles everything a Station should.

Function Summary

Static Public Function Summary
public

Converts raw DS18B20 temperatures to ºC.
public

LIS331HH_24G(raw: *): *

public

MMA7361_6G(raw: *): *

public

Converts raw MPX4115A temperatures to hPa.
public

analogToMV(analogReadResult: *): *

public

createLogger(name: String, db: PouchDB): Bunyan

Creates a configured bunyan logger.
public

A handler for incoming Serial#data.
public

getBit(b: *, i: *): Boolean

Gets one bit out of a byte/bytes.
public

getDB(name: String): PouchDB

Gets a local database.
public

getRemoteDB(name: String, username: String, password: String): PouchDB

Gets a remote database.
public

gpsAltitude(raw: *): *

public

Convert raw coordinate object ({deg, billionths}) into an actual latitude or longitude coordinate.
public

gpsCourse(raw: *): *

public

Processes the GPS boolean flags that are crammed in one byte.
public

gpsSpeed(raw: *): *

public

Lists available ports in the system with SerialPort.list.
public

normalizePath(path: ...*): *

public

Creates and returns a "byteDelimiter" parser.
public

passEvent(sourceEmitter: EventEmitter, destEmitter: EventEmitter, sourceEvent: String, destEvent: String)

Forwards an event from one EventEmitter to another.
Static Private Function Summary
private

minmaxH(min: Number, max: Number): Function

Heuristic creation helper.
private

variationH(variation: Number): Function

Heuristic creation helper.

Variable Summary

Static Public Variable Summary
public

Array of CanSat strings: informational messages.
public

Array of CanSat strings: module names.
public

A prefix for capture-lib files (DBs, logs).
public

T-Minus Transceiver Product ID
public

T-Minus Transceiver Vendor ID
Static Private Variable Summary
private

tmBuffer: Buffer

'Transceiver MODE' Buffer for future comparison.

Change Log

Change Log

All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.

Unreleased

v4.0.0 - 2017-05-01

Tested configurations

Basic packet reception was tested and is working. Backend connection and replication is also working (required a few bugfixes included in this release).

Breaking

  • Acceleration unit is now G (was m/s^2). It was deemed more fitting for us. (1ca064957fe633c5e2fe047fa9a40c4748469b82)
  • Dropped support for NodeJS v4 and Electron v1.4. (92a509734042c5da72ecdd667de3301919c50de1)

Added

  • New replicator state that removes undefined behavior between backoff. (758ed04068bc8771da586b270609f448e9073917)

Fixed

  • Backoff now has a minimum delay of 1s and a maximum delay of 60s to prevent deadlocks. (c04503cca805511b251d0169315ad31e1f313a82, df99e95430764d4c4d70c7c76545b24bd7b77ec8)
  • When replication fails, all event listeners for the task are removed right after backing off to prevent duplicate backoff calls. (3176cc9f77b3f5ca6971c1b4a6e92fb2cc89e19a)
  • Limited PouchDB replicator batching to prevent deadlocks. (ba804a692702001cc78ad2b5842b109703df8d55)
  • Replicator is no longer marked as "connecting" while backing off. (758ed04068bc8771da586b270609f448e9073917)
  • Prevent log replicator from going in an infinite loop: do not log pause or active events unless coming from a different state or an error occurs. (dcb4953477659910b27a851a98742254e7d4b748)

Changed

  • The station name is now included in the log DB path to split logs of different missions/stations. (a6e8038573da00bf72b85c99dc85c7564903c204)
  • The PouchDB replication task is now dereferenced while backing off, allowing GC to get rid of it and saving memory. (071d0b1731c669d19d3c18819f206c3bdcf3f6c1)
  • Replicators are now also stopped when disconnecting from the backend to save memory. It's possible they slow down the app too much and the station operator may decide that they mustn't run. (ac06773cba8891c69785656d024e66a6decd6514)
  • Log childIds are now shorter because they no longer begin with "station." (which is redundant, we know they are part of the station). (59bb9e9763bf1f8a94c72b6e2e890a0e102fc613)

v3.0.1 - 2017-04-23

Disabled a failing test for the build to pass. It's intended, we corrected the conversions according to the datasheet for the pressure sensor.

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

v3.0.0 - 2017-04-23

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Breaking

  • Switch 2nd accelerometer to a MMA7361 (a68d0080c43b51e3fa34efe21b847eb28dd55104)

Added

  • Value conversions for the MMA7361 accelerometer. (bc9782da39d1bb9826ba53142c217047c3d0c14b)

Fixed

  • Value conversions for the MPX4115A pressure sensor according to datasheet. (b2e3e028f516e057dd9a3739c431532b18c0fc62)

v2.1.0 - 2017-04-22

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Added

  • Save log files in global.pathPrefix directory. (a5b738e9ebfdd9eeed1a8e8731efe0a2c771caa5)

v2.0.4 - 2017-04-21

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Fixes

  • Split packets or \r\n. (4c21354a61fa4fefa86ba7769de6ca65dde7da28)

v2.0.3 - 2017-04-21

See changelog for 2.0.2, some changes went unreleased.

v2.0.2 - 2017-04-21

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Fixes

  • Save essential fields in bad packets. (c26ea8de50ac1af0a878b02acd076be84bb51079)

v2.0.1 - 2017-04-21

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Changed

  • Packet parser instances are now isolated. One is created per packet. (765166181a81a628493b13201de4fb2cd6ee0b79)

Fixes

  • Packet parsing no longer crashes because of too small packets. (ccc1b5645f1ff954ff8bda8fb901503f88a25f16)

v2.0.0 - 2017-04-20

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Changed

  • BREAKING: Packet encoding algorithm changed from quasi-binary to base64. It gives the same savings, and our transceivers actually work with it. (3006aca0c65bbd9241e405b1f775a8098b475140)

v1.0.3 - 2017-04-14

No changes to the code. Just changing the package name to be scoped (in @cansat-icarus). Last patch version containing nothing. I hope.

v1.0.2 - 2017-04-14

No changes to the code. Just removing the private mark of our package.json so we can publish this in npm.

v1.0.1 - 2017-04-14

Changes to CI configuration, not to the code itself.

v1.0.0 - 2017-04-14

Tested configurations

Basic packet reception was tested and is working. Backend-related code will be tested in the next few weeks.

Added

  • Backend connection logic and data/log db replication. (4eb02e79aa9bae70df6e5860bdc91b3f8831712d, 1bfb9fac2912718488e7d45c12d0d2078bf45ed3)
  • API method: listPorts (in ./serial.js), a wrapper around SerialPort.list that extracts hardware IDs in Windows. (905624d2e6e2f499f71df4102d5b52b208d3a5ad)
  • API method: Station#getAvailablePorts, a wrapper around listPorts that marks and prioritizes T-Minus transceivers in its output. (3ee5de49d789d072a54ce35714dbbaa53cc6d3ed)
  • babel-runtime, to prevent global scope pollution by babel-polyfill. (2a6659b2de1dcb3fc5cfd286418d3efb9c647dae)
  • Logging with bunyan. (eb8399574df2107d09877ff61c7ff1b864d2c36f)
  • API method: Station#cleanup, calls Serial#close and Backend#cleanup; returns a Promise. (f41fcaca18b43ce60c7ed018eea02b6077c931ef)
  • Serial now emits a "pathChange" event, whenever Serial#_path is changed. (c17248085ba3654583dc1274e45a79603c69534c)
  • API method: encode (in quasi-binary.js), a quasi-binary encoder. (33d3e6bd0ec982f25ae00a4b22eef4e3883c7265)
  • BREAKING: Missing fields from CanSat packets (unix timestamp, gps data, acceleration). (89a116c6ca4a52eb3071815fbaf0fa4985e0ad42)

Changed

  • BREAKING: "Serial" event names have changed for coherency with the rest of the codebase: (0a371e32f9374e7282c7fbba83c18d96bde7a8b6)
    • disconnected_forced -> disconnect_force
    • disconnected -> disconnect
    • closed -> close
  • BREAKING: Serial#_state is now initially set to "close" (instead of disconnect), to reflect the fact that no serialports were opened yet. (0a371e32f9374e7282c7fbba83c18d96bde7a8b6)
  • BREAKING: rename field in control packet parser for coherency: module.enabled -> module.enable (97823d08c89247c838b1eb4e63bc501313505b08)
  • BREAKING: Station constructor now requires a name (a6a9b01ca1658d65baabcb0ce8a775bd4c4318a9)
  • CanSat information packets message strings were shortened (88bd67253a973e9789b61c955f69347be7eab4c9)
  • yarn is now used for (deterministic) dependency management. (f1c5f07bf88753ffbc7ea69fa54a6c4758e7fc01)
  • "Packet saved" log messages are now log level trace. (e54530bcc0893e55315dc8b5fd28ffece5e8b743)
  • We now keep 40 log files (instead of 10). (57edbb9d55cc20b641c0f52e280f6f1ca9ed2de6)

Fixes

  • Remove unused (but ignored, so things would never work) parameter in the minmaxH packet heuristic (silent fix included in f7b0d2b98448b5ec193659f31439f1bb325d406a)
  • Serial#_destroyPort no longer tries to destroy ports that do not exist. (d93e342d7543f8f3c571049ebfb10162ad2632cd)
  • Serial.listPorts no longer logs (it's static, therefore it had no access to the logger). (7e97b537d4beb0ebb5159cb87ca07a7fbaaba31f)
  • Don't pass objects with circular references to PouchDB when saving error logs (a42201e4bd52e15ac237b187a33323edddff484a, b8d83db0dc4340f6af5ca4bbd044a57ac1ebcbba)
  • packet.raw now includes the CRC, allowing the field to really be the full raw packet as it was sent. (ab2e1e1d74f1fcc3b92d3907367c9906d453c488)
  • packet._id is now a string, like PouchDB requires. (89a116c6ca4a52eb3071815fbaf0fa4985e0ad42)
  • Write a few bytes to the transceiver to make it work after opening the SerialPort. (b152050a5fa8ab43bb3e90e5a03900fce59ca6a2)

v0.3.0 - 2016-08-14

Tested configurations

None. UNTESTED AND UNSTABLE. DO NOT USE! Until test protocol is finalized, version 1.* (stable) will not be reached. Unit tests are setup and passing.

Added

  • Unit tests for everything implemented (except for the quasi-binary decoder, those already existed).

Changes

Too many to list, all untracked so far. Until v1.* arrives, breaking changes may go undocumented. If for some (probably bad) reason you're using sub-v1 versions, check the few commits that took place between v0.2.1 and v0.3.

v0.2.1 - 2016-08-12

Tested configurations

None. UNTESTED AND UNSTABLE. DO NOT USE! Until unit tests and test protocol is finalized, version 1.* (stable) will not be reached.

Changed

  • Data handler now makes station emit an event on every packet received.
  • Bugfix: Quasi-binary unit tests now work (quasi-binary:error:* were expecting errors to be returned, not throwed).
  • Bugfix: Classifier#classifyPacket was ignoring the argument updateStationClassification.

v0.2.0 - 2016-08-12

Tested configurations

None. UNTESTED AND UNSTABLE. DO NOT USE! Until unit tests and test protocol is finalized, version 1.* (stable) will not be reached.

Added

  • Packet and Station classification algorithm (Classifier).

v0.1.0 - 2016-08-11

Tested configurations

None. UNTESTED AND UNSTABLE. DO NOT USE! Until unit tests and test protocol is finalized, version 1.* (stable) will not be reached.

Added

  • Add local database (PouchDB over LevelDown) for packet storage and replication.
  • Implement Station, the class to rule them all, literally. The only interaction a wrapper should have is with this class.
  • Implement quasi-binary decoder.
  • Implement Serial, a node-serialport wrapper that allows path changes and keeps track of port state.
  • Implement Parser, a general purpose parser meant to be implemented and fitted to anyone's needs.
  • Implement IcarusParser, the parser for our team (extends Parser).
  • Add sensor unit conversion.
  • Add string tables for raw CanSat values.