Skip to main content

I18 InterPulse multithreading

I18 Interblock multithreading

On a single computer, whether it be a server instance, dedicated application, or a browser page - multithreading and isolation are required for speed and security. They appear to be achievable via the same mechanism.

Specification​

The solution should:

  1. Allow blockmaking and crypto signing to occur concurrently on multiple threads, performing no work on the main thread, leaving this idle for things like UI and coordination.
  2. Terminate cleanly on idle - as soon as the threads have no further work to do, they should terminate.
  3. Be isomorphic in that the same lightweight threading mechanism is used on the server as in the browser
  4. Be compatible with Deno, at least architecturally
  5. Allow multithreading to run different instances of libp2p, to share the network and encryption load around

The solution must:

  1. Load worker code from IPFS blocks, allowing dev supplied covenants to be loaded and run.
  2. Support a single instance of IPFS being used between multiple engine threads
  3. Survive a thread being totally taken over by user code but still not able to gain any escalations

Solution​

This proposal is for using the threads library as a means of creating threads as needed on both nodejs and the browser. BlobWorker would be used to load up code from user covenants and run in isolation with resource allocation limits. This library has good control of memory limits, and can be forcibly terminated if cpu usage is too high.

Covenants would need to be loaded up using a wrapper function so that hooks can be applied. All IPFS queries from covenants should be done by action, to allow a security policy to be applied to each worker in a safe way, so that even if the shim is broken into, the worker isolation is the only protection we need.

In the browser, web workers would be used for the isolation, and on nodejs worker_threads would be used.

In lambda functions, this is the same mechanism that would be used for isolation and for extracting the maximum possible usage out of each lambda function. How to schedule lambda functions and other nodes is a separate problem, and should be done using blockchains to model each lambda function or compute node.

There is a pool.settled() function which resolves when all threads have completed, however the way pierce is constructed now, there should be no independent chains running that do not ultimately feed back to (and therefore finish before) pierce resolves.

Transferrable objects should be used to move blocks between the ipfs worker and the engine workers. The Uint8Arrays that Blocks use should be natively transferrable between threads with very little overhead.

SharedWorker IPFS​

To run in multiple tabs on the browser, the ipfs instance must run in a SharedWorker. This thread would then be reponsible for spawning multiple worker threads to do pulsemaking, as well as threads for doing isolated running if required.