# Proposal: FUDGE - Core library for local integration-tests in Substrate
[toc]

**Proponent:** **12wWLUd5qMzLFGqBsMnHLVFeTuYJwuo5ygMAxuSywrBX1XSF**
**Date:** 17.10.2022
**Requested DOT:** ($172,500) ~27,080 DOT
**Short description:** FUDGE (FUlly Decoupled Generic Environment for substrate-based chains) provides a simple and generic way to interact with and manipulate the database of a substrate-based blockchain. This includes building and importing blocks locally without any kind of network or consensus, as also changing existing state and finalizing it without the need of forging a new block.
**Contact:** Frederik Gartenmeister ([e-mail])

## Context of the Proposal
This grant is proposed by Centrifuge. A team made of experienced Substrate builders and a well established project in the Polkadot/Kusama ecosystem.

We have already received and successfully delivered grants: we developed a go-based rpc library for interacting with Substrate nodes ([GSRPC](https://github.com/centrifuge/go-substrate-rpc-client)) as one of the first on chain treasury grants in the Polkadot treasury. We also built an early bridge together with ChainSafe in 2019 which was funded by a web3 foundation grant. Furthermore, the Centrifuge chain is actively pushing xcm integrations in the Polkadot/Kusama ecosystem. All team members have worked extensively on launching Centrifuge and Altair launching the respective parachains. Various team members have presented talks on polkadot related events (sub0, Amsterdot, etc.).

The library itself benefits all teams building on substrate, as it makes no assumptions about a specific chain setup. Furthermore, the need for a tool FUDGE was named during the Polkadot Technical Parachain Summit. Parachain teams signaled the need for tools that can mutate an existing state to fork a chain and that allow to build blocks independently of block times was named. FUDGE already provides these capabilities.

* https://github.com/paritytech/substrate/issues/12442
* https://forum.polkadot.network/t/parachain-technical-summit-next-steps/51

In the context of this proposal, we will focus in the capabilities FUDGE provides for integration testing, as theses tests include the above mentioned features and well shows the capabilities of the library.

### Team Members
The team working on this composed of

* Frederik Gartenmeister (Rust developer, FUDGE project Lead)
* Jeroen Offerijns (CTO of Centrifuge)
* Branan Riley (Rust developer)
* Miguel Hervás (Rust developer)
* Nuno Alexandre (Rust Developer)

## Problem Statement
Interacting with a substrate client, database and runtime is inherently complicated and complex. Which is natural for complex systems. To the best of our knowledge there currently does not exist a simple way to run analytics on a database, mutate a database, test-runtime apis, test block-building locally or in general "debug" the inner workings of Substrate.

FUDGE is a core library that is not bound to specific use-case. The following focuses on its capabilities for integration-tests, which is the most obvious use-case. There are currently two levels of testing Parachains mostly use:

* Pallet based unit-tests
* Test networks for integration-tests

While pallet based unit-tests are suitable for ensuring the closed logic of a pallet works as intended, these tests typically use a different setup compared to the usage of the pallet in the runtime. The types often differ and externally provided functionality is often mocked. The trade-off made here is that setting up a complex test environment is not worth the gained test coverage. Test networks on the other side actually test runtimes with their live-system setup. But they have a high maintenance overhead, are not suitable for fast iterations and it is hard to setup CI processes with them.

Local integration tests should be a third level of testing in Substrate, but are currently impossible to realize. Hereby, a tool for integration testing must provide the following capabilities:
* Removing the need for a network interaction
* Usage of the actual Runtime
* Warp time arbitrary
* Build and import blocks correctly but without consensus
* Expose the state of a chain at any block

With theses features it will be possible to define test-cases that check complete code-paths and run theses tests in a reasonable amount of time. The possibility of running tests on an already existing state of a live-network would be an additional plus.

## Solution

FUDGE is a core library for the above mentioned things. The general idea is to create access to the core traits of Substrate that are concerned with client, database and runtime-api logic. Amongs others, this allows to test runtime-apis, to analyze and manipulate database states and to build and import blocks locally, without any network interaction. The library will help other projects to have safer runtime upgrades, better testing capabilites and more possibilites to build tooling around their chain.
In contrast to other to other tools for testing, like `try-runtime` and `zombinet`, FUDGE makes almost no assumptions what a user wants to do to a database. While `try-runtime` is especially useful as an integral part of a nodes service implementation, FUDGE works independent of the service and allows users to build blocks with different inherents and digests, access state without using runtime APIs. `Zombinet` on the other hand is tailored to enhance testing capabilites in "real" network setups and does not aim to solve the problem of local integration tests. But FUDGE and `zombinet` are able to work well together in cases where a given chains state should be manipulate for testing and then be started in a `zombinet` test network.

The library itself exposes three different *Builders*, a stand-alone-, a relay-chain- and a parachain-builder. At the inside, the builders contain a standard Substrate client, a standard Substrate transaction-pool and a standard Substrate database, like a normal Substrate node. But the builders expose multiple methods that allow to interact with theses elements directly.

### Access State
Any existing state of the underlying database can be accessed via an externalities provided environment.

let builder = StandaloneBuilder::<Block, RtApi, Cidp, Dp, HF>::new(...);

// Access latest state
let latest_block_number = builder.with_state(|| {
    frame_system::Pallet::<Runtime>::block_number()
});

// Access any state, if present
let block_number_at = builder.with_state_at(BlockId::Hash("HASH"), || {
    frame_system::Pallet::<Runtime>::block_number()
});

The interface is deliberately designed to be similar to the interface currently used in the pallet unit-tests. Allowing developers to reuse existing code and providing a used interface.

### Mutate Latest State
The latest state of the underlying database can be arbitrarily mutated, when accessed via the mutable version.

let mut builder = StandaloneBuilder::<Block, RtApi, Cidp, Dp, HF>::new(...);

// Mutate latest state and direcly finalize it again automatically
builder.with_mut_state(|| {
    // Actually transfer funds (omitting arguments here)
    pallet_balances::Pallet::<Runtime>::transfer(...)
});

After the code in the closure inserted new storage elements, the storage changes will be drained from the cache and the latest block will be finalized containing this state root. Blocks can be produced normally on top of the mutated block. This allows to take-over (fork) an existing chain by replacing block production authorities and then restarting the network.

### Block Building and Importing
A new block is simply build and imported by the following code. Developers have the possibility to add extrinsics to the underlying transaction pool. These must be properly signed and are included using the normal authorship logc.

let mut builder = StandaloneBuilder::<Block, RtApi, Cidp, Dp, HF>::new(...);

// Dummy extrinsic, that needs to be build properly somewhere
let some_xt: UncheckedExtrinsic<> = build_xt();

builder.append_extrinsic(some_xt);

// Includes extrinsic if authorship accepts it
builder.build_block();

// Import newly build block as last and finalized block
builder.import_block();

### Blocks for Parachains
Especially for parachain teams, it is important to build blocks in companion with the relay-chain. For this reason FUDGE exposes a macro `companion` to set up a local relay-chain-parachain-network.

const PARA_ID: u32 = 2002u32;

#[fudge::companion]
struct TestEnv {
	#[fudge::parachain(PARA_ID)]
	parachain: ParachainBuilder<PBlock, PRtApi, PCidp, PDp, ()>,
	#[fudge::parachain(2000u32)]
	sibling: ParachainBuilder<PBlock, PRtApi, PCidp, PDp, ()>,
	#[fudge::relaychain]
	relay: RelaychainBuilder<RBlock, RRtApi, RRuntime, RCidp, RDp, ()>,
}

fn main () {
    // Create env, needs to pass builders as arguments
    let env = TestEnv::new(...);
    
    // Progress the relay-chain (2 block) and the parachains (1 block)
    env.evolve();
    
    env.with_state(Chain::Para(PARA_ID), || {
        // latest state of chain parachain
    });
    
    env.with_state(Chain::Relay, || {
        // latest state of relay-chain    
    })
}

Currently, this logic is reduced to force the heads of parachain on the relay-chain side. But the PoC for realizing a normal update of the parachain head on the relay-chain side was already implemented and will allow to test XCMP locally.

---
By omitting the need to start a real network, FUDGE drastically reduces setup times, otherwise needed when starting a local network. The Centrifuge-chain is currently using the library to run integration tests that simulate real-block-production. Developers can also define custom timestamps, allowing them to warp time in order to test states in the future. The audience for FUDGE are the core-developers of the teams building on Substrate. In general, the current need to start a full environment (i.e. a normal blockchain-node of the respective chain) in order to test upgrades and other core-logic is a huge overhead with respect to time. Furthermore, analytics and insights of what is happening are reduced to overseeing logs and events, while FUDGE allows debugging and easy replay by starting the test again.

### Resources
[Integration tests of Centrifuge](https://github.com/centrifuge/centrifuge-chain/tree/main/runtime/integration-tests)
[FUDGE](https://github.com/centrifuge/fudge)

## Breakdown Deliverables
We have already delivered a PoC of the library that proves the working of the idea and we will continue the development to meet a production-grade standard for the library, and also provide missing core-features. Hence, this grant is applied partially in retrospect.

### Milestone 1 - PoC [[Already Delivered](https://github.com/centrifuge/fudge/pull/6)]

* 1.1. Prove that it is possible to access a chains state, mutate the given storage and then re-finalize the block.
* 1.2. Ensure, that the chain is not broken (from this point onwards).
* 1.3. Access a chains state at any block
* 1.4. Build and import a local block
* Standalone-chain
* Relay-chain
* Parachain
* 1.5. Provide a macro to allow a simple relay-chain parachain setup
* 1.6. Use a transaction-pool to locally submit extrinsics

*Estimated hours: 400*
*Estimated budget: $60,000*

### Milestone 2 - Stabilization
This milestone stabilizes the current PoC to a point where it can be used without expecting work-in-progress like behaviour.

The code will contain proper error-handling and error-propagation. The code base is cleaned up from its PoC state and there exists a CI/CD-pipeline. Furthermore, there exist trace logging and logs between chains can be easily differentiated.

#### Deliverables
* 2.1. CI/CD-Pipeline is set-up
* 2.2. Block production improvements
https://github.com/centrifuge/fudge/issues/13
* 2.3. Proper error handling
https://github.com/centrifuge/fudge/issues/7
* 2.4. Proper logging
https://github.com/centrifuge/fudge/issues/26

*Estimated hours: 150*
*Estimated budget: $22,500*

### Milestone 3 - XCMP
This milestone enables parachains and relay-chains to use XCMP locally. This will be realized by building the correct inherents, which subsequently updates a parachains head, signals processed XCM's and enables the transfer of new XCM's.
With this milestone reached, teams will be able to create complex XCM test cases with multiple chains.

* Implementing XCMP-capabilites
https://github.com/centrifuge/fudge/issues/15

*Estimated hours: 300*
*Estimated budget: $45,000*

### Milestone 4 - Tests and documentation
This milestone provides extensive unit and integration-tests und the library side. Furthermore, the documentation, in code and in the repo is up-to-date and allows to easily get started with the library.

* Unit-tests
* Integration-tests
* Documentation
https://github.com/centrifuge/fudge/issues/20

*Estimated hours: 200*
*Estimated budget: $30,000*

### Continous Compatability
As fudge works upon core apis of both polkadot and substrate, a high work load for staying compatible is expected till those stabilize. The work includes among others:
* Staying up-to-date with the latest polkadot version
* Adapting to changes of the core apis (e.g. changes in relay-chain interface on the cumulus side)
* Adapting logic to handle new `StateVersion::V1`
* Adapting to asynchronous backing changes

This work is not part of the milestones as it includes no new features, but rather ensures compatability between FUDGE and the polkadot ecosystem. As such it can not be seen as maintenace. It is rather parallel work which is done besides the fulfillment of the milestones and hence, will be submitted alongside the milestones.

*Estimated hours: 100*
*Estimated budget: $15,000*

## Budget Summary
This table provides a cost breakdown based on the milestones and deliverables above.

| | | | Estimates |
| -------- | -------- |----|--- |
| **MS** | **Deliverable** | **Hours**| **Budget** |
| 1. | PoC| 400 | $60,000 |
||**Subtotal Milestone 1**| **400**| **$60,000**|
| 2.1. | CI/CD-pipeline | 15 | $2,250|
| 2.2. |Block production improvements |50| $7,500 |
|2.3.|Proper error handling|50|$7,500|
|2.4.|Proper logging|35|$5,250|
||**Subtotal Milestone 2**|**150**|**$22,500**|
|3.1.|Implementing XCMP-capabilities|300|$45,000|
||**Subtotal Milestone 3**|**300**|**$45,000**|
|4.1.|Unit tests|80|$12,000|
|4.2.|Integration tests|80|$12,000|
|4.3.|Documentation| 40| $6,000|
||**Subtotal Milestone 4**|**200**|**$30,000**|
||Continous Compatability|100|$15,000|
||**Totals**|**1100**|**$172,500**|

## Payment Conditions
The grant is proposed to the *Polkadot Treasury* and as such, if granted, will be payed in DOT.
At the time of writing the total grant amount of *$172,500* result in a total grant amount of *27,080 DOT*. The on-chain submission will be split into two. First the grant is applied for the first four Milestones (M1 - M4) with an overall value of *$157,500* or *24,643 DOT*. The second submission will include the *Continous Compatability* with an overall value of *$15,000* or *2,437 DOT*. The time of the submission is either after the milestones are delivered or in between them, if the work needed for compatability increases more than expected.

The full deliverable will be found on [https://github.com/centrifuge/fudge/](https://github.com/centrifuge/fudge/).