Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions protocols/aave/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,14 @@

## Utilization

Github actions run hourly and send telegram message if there is a market with utilization above `96%`. [Python script code](https://github.com/yearn/monitoring/blob/main/aave/main.py).
Automation on our VPS runs hourly and sends a telegram message if there is a market with utilization above `99%`. [Python script code](https://github.com/yearn/monitoring/blob/main/aave/main.py).

## Governance

[Internal timelock monitoring](../timelock/README.md) for queueing proposal to Aave Governance contract.
Proposal can be executed immediately because the cooldown period is [set to 0](https://etherscan.io/address/0x9aee0b04504cef83a65ac3f0e838d0593bcb2bc7#readProxyContract#F3). After the proposal is executed, the payload is queued to Payload Controller which has min execution delay [set to 1 day](https://etherscan.io/address/0xdabad81af85554e9ae636395611c58f7ec1aaec5#readProxyContract#F6).
Every payload that is executed on any network is the first queue to proposal on the mainnet. Monitoring just the proposal queue on the mainnet enables to get notification for future updates on all networks.

Additionally, Github actions bot runs every hour and fetches queued proposals using The Graph data: https://github.com/yearn/monitoring/blob/main/aave/proposals.py
Additionally, automation on our VPS runs every hour and fetches queued proposals using Aave's governance cache API.

Monitoring Safe multisigs of Protocol emergency Guardian and Governance emergency Guardian. Link with [address](https://app.aave.com/governance/v3/proposal/?proposalId=184) and [explanation](https://governance.aave.com/t/arfc-renewal-of-aave-guardian-2024/17523).

## External Monitoring

ChaosLabs has a public Telegram group with alerts for Aave V3: https://community.chaoslabs.xyz/aave/risk/alerts
31 changes: 31 additions & 0 deletions protocols/aave/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,44 @@
"0xf939E0A03FB07F59A73314E73794Be0E57ac1b4E",
"crvUSD",
),
(
"0x32a6268f9Ba3642Dda7892aDd74f1D34469A4259",
"0xdC035D45d973E3EC169d2276DDab16f1e407384F",
"USDS",
),
(
"0xBdfa7b7893081B35Fb54027489e2Bc7A38275129",
"0xCd5fE23C85820F7B72D0926FC9b05b43E359b7ee",
"weETH",
),
(
"0x5c647cE0Ae10658ec44FA4E11A51c96e94efd1Dd",
"0xcbB7C0000aB88B473b1f5aFd9ef808440eed33Bf",
"cbBTC",
),
(
"0x5Ee5bf7ae06D1Be5997A1A72006FE6C607eC6DE8",
"0x2260FAC5E5542a773Aa44fBCfeDf7C193bc2C599",
"WBTC",
),
(
"0x10Ac93971cdb1F5c778144084242374473c350Da",
"0x18084fbA666a33d37592fA2633fD49a74DD93a88",
"tBTC",
),
(
"0x65906988ADEe75306021C417a1A3458040239602",
"0x8236a87084f8B84306f72007F36F2618A5634494",
"LBTC",
),
],
}

THRESHOLD_UR = 0.99


def print_stuff(chain_name: str, token_name: str, ur: float) -> None:
logger.debug(f"Chain: {chain_name}, Token: {token_name}, UR: {ur}")
if ur > THRESHOLD_UR:
message = f"**BEEP BOP**\n💎 Market asset: {token_name}\n📊 Utilization rate: {ur:.2%}\n🌐 Chain: {chain_name}"
send_alert(Alert(AlertSeverity.MEDIUM, message, PROTOCOL))
Expand Down
27 changes: 8 additions & 19 deletions protocols/ethena/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,19 @@ Ethena is a synthetic dollar protocol built on Ethereum that provides a crypto-n

## Monitoring

The script [`ethena/ethena.py`](ethena.py) runs daily via GitHub Actions to sanity-check that **USDe remains fully backed** and that the public data feeds are fresh and internally consistent. Telegram messages are sent if some values are out of the expected range.
The script [`ethena/ethena.py`](ethena.py) runs daily via our VPS automation to sanity-check that **USDe remains fully backed** and that the public data feeds are fresh and internally consistent. Telegram messages are sent if some values are out of the expected range.

### Data Sources - Chaos Labs
### Data Source - Ethena Transparency API

1. **Attestation**
`GET https://api.chaoslabs.xyz/v1/attestation/ethena`
The primary backing check uses Ethena's own transparency API (`app.ethena.fi`). This API was previously blocked for GitHub Actions IPs, so a Chaos Labs / Oracle Security Proof-of-Reserve endpoint was used instead. That endpoint has since been decommissioned (returns HTTP 503), and Chainlink's USDe Proof of Reserves (Ethena's [PoR launch](https://ethena.fi/blog/usde-proof-of-reserves-launch) with Chainlink, Chaos Labs, LlamaRisk and Harris & Trotter) is not published as a public on-chain feed we can query. Since monitoring now runs on our VPS, Ethena's transparency API is reachable and is used directly.

2. **Attestation Freshness**
If attestation is older than 1 day, skip the check.

3. **Attestation Consistency**
- Check if USDe is fully backed
- Check if only approved assets are used
- Check if delta neutral strategy is maintained
- Check if signature is valid

4. **Attestation Metrics**
- Backing Ratio: `backingAssetsUsdValue / totalSupply`
- Reserve Buffer: `backingAssetsAndReserveFundUsdValue - totalSupply`
- Last Update: `timestamp`
1. **Supply**: `GET /api/solvency/token-supply?symbol=USDe`
2. **Collateral**: `GET /api/positions/current/collateral?latest=true`
3. **Backing Ratio**: `totalBackingAssetsInUsd / supply` — alert CRITICAL if `< 1`. USDe targets ~1:1 collateral backing with a separate reserve fund as the buffer, so the collateral-only ratio sits just above 1.0 in normal operation.

### Data Sources - LlamaRisk

> NOTE: LlamaRisk data is not reliable, so we use Chaos Labs data instead.
> NOTE: LlamaRisk data is not reliable, so it is currently disabled (`llama_risk_check`).

#### Off-Chain

Expand All @@ -41,7 +30,7 @@ Data used is provided by Ethena on [transparency page](https://app.ethena.fi/das
2. **LlamaRisk Dashboard**
`GET https://api.llamarisk.com/protocols/ethena/overview/all/?format=json`

> NOTE: Ethena data is not available when running on Github Actions, so we use LlamaRisk data only.
> NOTE: This LlamaRisk cross-check section is currently disabled (`llama_risk_check`). The note that Ethena data was unavailable applied to the old GitHub Actions setup; on our VPS the Ethena transparency API is reachable and is the primary source (see above).

#### On-Chain

Expand Down
149 changes: 27 additions & 122 deletions protocols/ethena/ethena.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,10 @@
PROTOCOL = "ethena"
logger = get_logger(PROTOCOL)

# NOTE: ethena cannot be used because it blocked for Github Actions IP
# Ethena transparency API endpoints
# Ethena transparency API endpoints (usable from our VPS; were previously blocked for GitHub Actions IPs)
SUPPLY_URL = "https://app.ethena.fi/api/solvency/token-supply?symbol=USDe"
COLLATERAL_URL = "https://app.ethena.fi/api/positions/current/collateral?latest=true"
LLAMARISK_URL = "https://api.llamarisk.com/protocols/ethena/overview/all/?format=json"
CHAOS_LABS_URL = "https://history.oraclesecurity.org/por/attestations?protocol=ethena"

USDE_ADDRESS = "0x4c9EDD5852cd905f086C759E8383e09bff1E68B3"
SUSDE_ADDRESS = "0x9D39A5DE30e57443BfF2A8307A4256c8797A3497"
Expand Down Expand Up @@ -86,7 +84,10 @@ def is_stale_timestamp(ts: str, max_age_hours: int = 3) -> bool:
dt = _parse_timestamp(ts)
if dt is None:
return True
return dt < datetime.utcnow() - timedelta(hours=max_age_hours)
# _parse_timestamp returns naive datetimes, so compare against a naive UTC "now"
# (datetime.utcnow() is deprecated).
now_utc = datetime.now(timezone.utc).replace(tzinfo=None)
return dt < now_utc - timedelta(hours=max_age_hours)


def get_usde_supply() -> float | None:
Expand Down Expand Up @@ -312,147 +313,51 @@ def llama_risk_check():
send_alert(Alert(AlertSeverity.MEDIUM, message, PROTOCOL))


@dataclass
class ChaosLabsAttestation:
timestamp: str
backing_assets_usd_value: float
backing_assets_and_reserve_fund_usd_value: float
backing_assets_exceeds_usde_supply: bool
approved_assets_only: bool
delta_neutral: bool
total_supply: float
signature: str | None


def ethena_native_backing_check() -> bool:
"""Fallback backing-ratio check using Ethena's own transparency API.

Used when the Chaos Labs / Oracle Security attestation endpoint is unreachable
(e.g. provider outage). Only the quantitative backing ratio is checked here; the
Chaos-specific flags (delta-neutral, approved-assets-only, signed attestation) have
no equivalent on this source and are skipped.

Returns:
True if backing data was available and the check ran, False if Ethena's API
could not be reached either (so the caller can escalate to a hard alert).
def ethena_backing_check() -> None:
"""Check that USDe remains fully backed using Ethena's transparency API.

This is the primary backing check. Ethena's transparency API (app.ethena.fi) is
usable now that monitoring runs on our VPS — it was previously blocked for GitHub
Actions IPs, which is why a Chaos Labs / Oracle Security PoR endpoint was used
instead. That endpoint has since been decommissioned (returns 503), and Chainlink's
USDe PoR is not published as a public on-chain feed, so we rely on Ethena's own
transparency data.

Alerts CRITICAL when collateral no longer covers supply (ratio < 1). USDe targets
~1:1 collateral backing with a SEPARATE reserve fund as the buffer, so the
collateral-only ratio sits just above 1.0 in normal operation — applying
COLLATERAL_RATIO_TRIGGER (which assumes the reserve fund is included) here would
false-positive.
"""
supply = get_usde_supply()
collateral = get_total_collateral_usd()
if supply is None or collateral is None or supply == 0:
return False
send_error_message("⚠️ ETHENA: Failed to fetch backing data from Ethena transparency API", PROTOCOL)
return

# Mirror the Chaos Labs check's semantics: alert only when collateral no longer covers
# supply (ratio < 1). USDe targets ~1:1 collateral backing with a SEPARATE reserve fund as
# the buffer, so collateral-only ratios sit just above 1.0 in normal operation — applying
# COLLATERAL_RATIO_TRIGGER (which assumes reserve fund is included) here would false-positive.
backing_ratio = collateral / supply
if backing_ratio < 1:
send_alert(
Alert(
AlertSeverity.CRITICAL,
f"🚨 USDe NOT FULLY BACKED (Ethena API fallback)!\n"
f"🚨 USDe NOT FULLY BACKED!\n"
f"Backing Assets: ${collateral:,.2f}\nTotal Supply: {supply:,.2f}\n"
f"Backing Ratio: {backing_ratio:.4f} ({backing_ratio * 100 - 100:+.2f}%)",
PROTOCOL,
)
)

logger.info(
"Ethena native API fallback – backing: $%s | supply: %s | ratio: %s",
"Ethena transparency API – backing: $%s | supply: %s | ratio: %s",
f"{collateral:,.2f}",
f"{supply:,.2f}",
f"{backing_ratio:.4f}",
)
return True


def chaos_labs_check():
data = fetch_json(CHAOS_LABS_URL)
if not data or not isinstance(data, list) or len(data) == 0:
# Oracle Security / Chaos Labs PoR endpoint is unreachable (e.g. provider outage).
# Don't page on every run for an upstream outage we can't fix: log it and fall back
# to Ethena's own transparency API for the critical backing-ratio check. Only escalate
# to a hard alert if that fallback is also unavailable — meaning we have no backing
# data from any source. The Chaos-specific flags (delta-neutral, approved-assets,
# signature) are unavoidably skipped until the attestation endpoint recovers.
logger.warning("Chaos Labs attestation endpoint unavailable; falling back to Ethena native API")
if not ethena_native_backing_check():
send_error_message(
"⚠️ ETHENA: Failed to fetch Chaos Labs attestation data and Ethena API fallback", PROTOCOL
)
return

# Get the latest attestation (last item in the list)
latest_attestation_raw = data[-1]

try:
attestation = ChaosLabsAttestation(
timestamp=latest_attestation_raw["timestamp"],
backing_assets_usd_value=latest_attestation_raw["backingAssetsUsdValue"],
backing_assets_and_reserve_fund_usd_value=latest_attestation_raw["backingAssetsAndReserveFundUsdValue"],
backing_assets_exceeds_usde_supply=latest_attestation_raw["backingAssetsUsdValueExceedsUsdeSupply"],
approved_assets_only=latest_attestation_raw["approvedAssetsOnly"],
delta_neutral=latest_attestation_raw["deltaNeutral"],
total_supply=latest_attestation_raw["totalSupply"],
signature=latest_attestation_raw.get("signature"),
)
except KeyError as e:
send_error_message(f"⚠️ ETHENA: Missing field in Chaos Labs data: {e}", PROTOCOL)
return

attestation_time = datetime.fromisoformat(attestation.timestamp.replace("Z", "+00:00"))
if datetime.now(timezone.utc) - attestation_time > timedelta(days=1):
logger.warning("Attestation from Chaos Labs is older than 1 day: %s. Skipping check.", attestation_time)
return

# Check if USDe is fully backed
backing_ratio = attestation.backing_assets_usd_value / attestation.total_supply
if not attestation.backing_assets_exceeds_usde_supply:
send_alert(
Alert(
AlertSeverity.CRITICAL,
f"🚨 USDe NOT FULLY BACKED!\nBacking Assets: ${attestation.backing_assets_usd_value:,.2f}\nTotal Supply: ${attestation.total_supply:,.2f}\nBacking Ratio: {backing_ratio:.4f} ({backing_ratio * 100 - 100:+.2f}%)",
PROTOCOL,
)
)
# Cross-check with Chaos Labs flag (for data consistency)
if not attestation.backing_assets_exceeds_usde_supply and backing_ratio >= 1:
send_alert(
Alert(
AlertSeverity.HIGH,
f"⚠️ Data inconsistency: Chaos Labs flag says not backed but ratio shows backed. Ratio: {backing_ratio:.4f} ({backing_ratio * 100 - 100:+.2f}%)",
PROTOCOL,
)
)

# Check if only approved assets are used
if not attestation.approved_assets_only:
send_alert(Alert(AlertSeverity.MEDIUM, "⚠️ Non-approved assets detected in backing!", PROTOCOL))

# Check if delta neutral strategy is maintained
if not attestation.delta_neutral:
send_alert(Alert(AlertSeverity.MEDIUM, "⚠️ Delta neutral strategy not maintained!", PROTOCOL))

# Check signature validity (missing signature could indicate issues)
if attestation.signature is None:
send_alert(
Alert(AlertSeverity.MEDIUM, "⚠️ Attestation signature missing - verification may be incomplete", PROTOCOL)
)
# Calculate and report backing metrics for transparency
backing_ratio = attestation.backing_assets_usd_value / attestation.total_supply
reserve_buffer = attestation.backing_assets_and_reserve_fund_usd_value - attestation.total_supply
logger.info(
"Attestation from Chaos Labs: %s\nBacking Ratio: %s (%s%%)\nReserve Buffer: $%s",
attestation.timestamp,
f"{backing_ratio:.4f}",
f"{backing_ratio * 100:,.2f}",
f"{reserve_buffer:,.2f}",
)


if __name__ == "__main__":
from utils.runner import run_with_alert

# NOTE: skip using LlamaRisk data because it is not reliable
# llama_risk_check()
run_with_alert(chaos_labs_check, PROTOCOL)
# NOTE: LlamaRisk data (llama_risk_check) is not reliable and the former Chaos Labs /
# Oracle Security PoR endpoint is decommissioned; use Ethena's transparency API directly.
run_with_alert(ethena_backing_check, PROTOCOL)