History of the Terra <> Sifchain IBC Connection
Introduction
As many of you may have noticed, there has been a lot of confusion and questions regarding the Sifchain <> Terra saga.
In this technical article we aim to provide clear explanations on the reasons why:
- Import and Export transactions between Sifchain and Terra stopped working
- There were Sifchain <>Terra transactions stuck in the relayer
- It required considerable engineering effort to address these things
In order to understand the above points, we need to get into a nitty-gritty summary of IBC.
IBC overview
IBC is a cutting-edge technology, and is unique in its kind. With the release of CosmosSDK .43, IBC allowed blockchains built with the CosmosSDK to easily integrate with each other. Through IBC, users finally had the possibility to swap assets across different blockchains built with the CosmosSDK. This is huge, as it implies that a blockchain built with the CosmosSDK and already connected to a non-Cosmos blockchain through its own bridge, could “de facto” fully integrate Cosmos and the external ecosystem.
To give an example:
- Before IBC was released, Sifchain had developed Peggy to bridge assets between Cosmos and Ethereum.
- When IBC was delivered, every blockchain that connected with Sifchain through IBC could take advantage of the Sifchain <> Ethereum bridge called Peggy.
- Every blockchain connected to Sifchain via IBC could leverage Peggy to handle Ethereum assets on its network, without the need of having an Ethereum bridge itself
This allowed achieving an important milestone for the vision of a fully interconnected Web 3.0
Because of the novelty & newness of IBC, those who have been in the “front line” of using it (like Sifchain), have a greater probability of identifying previously undiscovered ‘edge case’ scenarios. This isn’t a bad thing as it allows the underlying technology to be continuously improved, enhanced, and new mechanisms to prevent issues are introduced. Just to give an example, there are several reasons why a relayer may stop forwarding packets (running out of funds to relay packets, client timeouts..), and this implies that it is not uncommon for a relayer to go down.
In summary, IBC opened up an exciting set of possibilities, giving the opportunities to networks like Sifchain to be in the front line leveraging it. However, when adopting newly released tech, it is inevitable to encounter unforeseen scenarios.
Now, brace yourself, and let’s get right into what happened:
Timeline
- The Terra/Sifchain relayer stopped updating the Terra IBC client on Sifchain (1st of November 2021). The reason why it stopped updating is simple: the relayer went down due to enormous amounts of traffic. Because of the instability, the Terra client expired.The only way to migrate that expired client was to migrate the client as per the Cosmos docs.
- This caused a problem, since not only were transfer request packets not relayed (tokens appeared to be stuck), but the consensus state of the Terra IBC client was also not being updated. As the trusting period of the Terra client on Sifchain was only 16hrs, this caused the client to timeout and expire.
- The Terra IBC client on Sifchain received no updates for an amount of time which exceeded the trusting period (57600s = 16hrs), at which point the Client officially expired. At this point no packets could be sent to/from the client without a fork of Sifchain.
- The Cosmos team had preempted such events and provided an update-client governance proposal (this implements the fork discussed in point 3 above). This allows IBC client consensus-states to be updated with the consensus state of a substitute (non-expired) client (with some constraints on the substitute client). This is only available for ibc-go >= v1.1.0. At this point in time, we were not on ibc-go-v2.
- This is when we raised the Magnolia release, which included an upgrade to ibc-go-v2
- After this proposal went through successfully, a new client was created (along with a new channel and connection):
hermes create channel sifchain-1 columbus-5 — port-a transfer — port-b transfer -o unordered
(In hindsight, Hermes could have also been used to directly create a client hermes create client columbus-5 sifchain-1)
7. An attempt was made to submit a proposal to update the expired client to the new client:
sifnoded tx gov submit-proposal update-client 07-tendermint-42 07-tendermint-79…
However, this failed:
raw_log: ‘failed to execute message; message index: 0: client cannot be updated with
proposal: unable to update light client: invalid proposal content’
8. At this point, it appeared that the update to ibc-go-v2 had pruned consensus states of expired clients. The lookup on the CLI also showed that the consensus state for the client of interest had been pruned:
sifnoded q ibc client consensus-states 07-tendermint-42 — node http://rpc.sifchain.finance:80
consensus_states: []
The Sifchain team then raised an issue with the ibc-go team here.
The reasoning behind the above complication is because of the fact that the update-client proposal can ONLY be done on clients which have expired. This implies that a verification is made on whether the consensus state is expired. However, for IBC clients that had already expired prior to upgrading to ibc-go >= v1.1.0, their consensus-states were (automatically) pruned since these clients were deemed to have no use anymore.
9. A fix was provided, whereby Client’s with missing consensus states were classified as ‘Expired’. This was reasonable since only clients that had expired could have their consensus states pruned, and so all clients with missing consensus states must have expired.
10. A new version of Sifnoded was released with this fix (a fork of ibc-go was used to speed up the process) and when accepted by a governance vote it became active at block height 5,870,400.
11. On March 9th, a new channel was created between Sifchain and Terra, as the 07-tendermint-79 client had expired:
hermes create channel sifchain-1 columbus-5 — port-a transfer — port-b transfer -o unordered
For the client upgrade to succeed (in the next step), the parameters of the clients created with this command needed to match the original client parameters. This required the following Hermes chain config:
[[chains]]
id = ‘columbus-5’
.
.
clock_drift = ‘1s’
max_block_time = ‘3s’
trusting_period = ‘57600s’
trust_threshold = { numerator = ‘1’, denominator = ‘3’ }
[[chains]]
id = ‘sifchain-1’
.
.
clock_drift = ‘1s’
max_block_time = ‘3s’
trusting_period = ‘14days’
trust_threshold = { numerator = ‘1’, denominator = ‘3’ }
The reason why clock_drift=1s and max_block_time=3s was set is because the max_clock_drift of the original clients was equal to three seconds. Since max_clock_drift is calculated as src_chain_config.clock_drift + dst_chain_config.clock_drift + dst_chain_config.max_block_time in the Hermes source code, we set clock_drift=1s and max_block_time=3s in the Hermes config for both chains. The clock_drift and max_block_time parameters don’t seem to make it onto the on-chain config so the hope is they are used solely to calculate the max_clock_drift and as such there are no downsides to setting them to these somewhat arbitrary values.
The parameters returned from the create channel command were as follows:
Success: Channel {
ordering: Unordered,
a_side: ChannelSide {
chain: ProdChainHandle {
chain_id: ChainId {
id: “sifchain-1”,
version: 1,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
“07-tendermint-90”,
),
connection_id: ConnectionId(
“connection-54”,
),
port_id: PortId(
“transfer”,
),
channel_id: Some(
ChannelId(
“channel-47”,
),
),
version: None,
},
b_side: ChannelSide {
chain: ProdChainHandle {
chain_id: ChainId {
id: “columbus-5”,
version: 5,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
“07-tendermint-64”,
),
connection_id: ConnectionId(
“connection-49”,
),
port_id: PortId(
“transfer”,
),
channel_id: Some(
ChannelId(
“channel-31”,
),
),
version: None,
},
connection_delay: 0ns,
}
12. On March 9th, a Hermes relayer was also set up between 07-tendermint-90 on Sifchain and 07-tendermint-64 on Terra. The latest_height could be checked manually:
sifnoded query ibc client state 07-tendermint-90 — node http://rpc.sifchain.finance:80
terrad query ibc client state 07-tendermint-64 — node http://public-node.terra.dev:26657
And updated manually:
hermes update client sifchain-1 07-tendermint-90
hermes update client columbus-5 07-tendermint-64
13. The very same day, a proposal was submitted to upgrade the client:
sifnoded tx gov submit-proposal update-client 07-tendermint-42 07-tendermint-90 \
— from my_key \
— keyring-backend test \
— node tcp://rpc.sifchain.finance:80 \
— title “v0.11.0-update-terra-ibc-client” \
— description “Proposal to update the Terra IBC client. Necessary to unlock stuck transfers.” \
— chain-id sifchain-1 \
— deposit 10000000000000000000rowan \
— fees 100000000000000000rowan \
— broadcast-mode block \
-y
See https://www.mintscan.io/sifchain/proposals/31
14. On March 13th the proposal was accepted, and the 07-tendermint-42 became active again:
sifnoded q ibc client status 07-tendermint-90 — node http://rpc.sifchain.finance:80
status: Active
15. Hermes was used to flush the stuck TXs:
hermes tx raw packet-recv columbus-5 sifchain-1 transfer channel-18
The stuck Terra tokens were returned to their senders! :D
16. A relayer was set up to keep 07-tendermint-42 active. The relayer between 07-tendermint-90 and 07-tendermint-64 can probably be stopped and those clients left to expire.
Next Steps
Updating the Sif client on the Terra chain
Updating the Terra Client on Sifchain with the latest Terra state will not re-establish the connection between Terra and Sif. This is because the Sif client on the Terra chain has also expired. It will however be (technically) easy to update the client with a governance proposal on the Terra chain in the same way the Terra client on Sifchain was updated. This can only be done once the Terra chain has upgraded to a later version of ibc-go. Once the proposal passes, we’ll be close to reinstating Sifchain <> Terra transactions.
Increasing the trusting period
The trusting period of the Terra client on Sifchain is only 16 hrs. This means that if the client does not receive an update for more than 16 hrs it will expire, tokens will be stuck and a governance proposal will be necessary to update the client. The relayer is scheduled to update the client every 10 hrs, meaning that if the relayer goes down we may only have 6 hrs to notice and react. Ideally we should increase the trusting period to 14 days. The Sif client on the Terra chain has a 14 day trusting period, so no update is required for that client.
Increasing the trusting period cannot be achieved via a simple transaction (this would allow anyone to adjust the value causing client’s to expire). Instead it will require a governance proposal. There doesn’t appear to be a ready made governance proposal (as there is for updating a client’s consensus state) and Sifchain engineers have reached out to the ibc-go team to discuss adding a governance proposal to ibc-go that would allow the trusting_period to be changed. A successful governance proposal would be the most straightforward scenario, as without a governance proposal, the only options would be:
- Modify the Sifnoded go code — given that the trusting period is a client parameter (i.e. it is not a property of the other chain though it does depend loosely on the properties of the other chain), and is set by the relayer at the time the connection is setup, it should in principle be possible to change the go code in such a way that the value gets overwritten without breaking the connection. You can have a further look at the difference between client and on-chain parameters here
- Create a new connection, potentially leading to significant ramifications (e.g. without code changes it would probably need new denominations to be defined)
Restore the client
A proposal has been submitted to Terra to un-expire the sif client https://station.terra.money/proposal/775. Once it passes, the sif<>terra connection will be back up and running (although additional work will be needed to enable imports and exports in the Sifchain UI).
Conclusions
The whole Sifchain <> Terra saga has been an incredible learning opportunity for Sifchain engineers, as it deeply strengthened their knowledge of the inner workings of IBC. This is an essential element in order to avoid similar issues in the future, as it allows for higher visibility over IBC edge cases. In addition to this, the knowledge that was gained from this experience helped the whole IBC protocol to go a step forward towards maturity, as other communities can grow from our learnings and enhancements can be made.
Get in Touch with Sifchain
Official Channels
Media & content creators, contact us at communications@sifchain.finance.
