Workshop: Hands-on Evaluation of ERC-20 Tokens

ERCx evaluates ERC (Ethereum Request for Comments) token implementation, either deployed or using their source code. In this workshop, we focus on the ERC-20 standard and the comprehensive test suite (100+ tests) of ERCx. The objectives of this workshops are to:

Get familiar with ERCx and how it can be used to evaluate existing ERC-20 tokens and test undeployed implementations of ERC-20 tokens using source code;
Use ERCx comprehensive test suite to locate bugs in an ERC-20 token implementation;
Review the available interfaces offered by ERCx.

During the workshop, we will conduct the following hands-on activities:

Test your favorite token. In this activity, participants will select some of the tokens they are familiar with and run the ERC-20 test suite on them. Participants will learn how to read and interpret the generated reports.
Find the bug in some ERC-20 tokens. In this activity, participants will be presented with buggy source code implementations of ERC-20 tokens. Participants will exploit the reports generated by ERCx to find the bugs, fix them, and ensure the patched implementations are correct.
Test a list of tokens. In this activity, participants will use ERCx open API to test lists of tokens by automatically querying ERCx reports. Participants will learn how to create their own list of tokens and how to query reports in batches.

Test your favorite tokens

In this activity, we perform evaluation of tokens and read through their generated reports. For this, we use ERCx website. For the purposes of this, we will need some token addresses. As examples, we provide some addresses of well-known tokens below on mainnet:

Tether USD (USDT): 0xdAC17F958D2ee523a2206206994597C13D831ec7
BNB (BNB): 0xB8c77482e45F1F44dE1745F52C74426C631bDD52

However, you can choose any address from mainnet and testnets Goerli or Sepolia.

Addresses can be found on Etherscan.io:

For Mainnet: https://etherscan.io/tokens
For Goerli: https://goerli.etherscan.io/
For Sepolia: https://sepolia.etherscan.io/

Using addresses, we can now get evaluation reports. For this, we go to the website and input the address into the address field, make sure to select ERC-20 and network Mainnet.

Note, if you have selected a common token, it is likely that a report (generated by a previous user) exists for that token. In such a case, you will be offered with the possibility to consult directly the previous report. By clicking on the Test button, it will generate a fresh evaluation conducted on the latest deployed version of the contract and its latest state.

Once the report is generated, we land on a dedicated page, the report page. The report consists of several parts:

The first and topmost part consists in a summarizing view of the report with all essential pieces of information to get a quick overview about the token.
The first info concerns whether the token is behind a proxy address. Being behind a proxy allows the token owner to change the implementation of the contract. Below that is a list of functions that are described in the ERC-20 token standard: name, symbol, …, allowance. A checkmark indicates that the token implements such function. Next is a summary of the test results, organized by level. Levels structure tests according to the nature of the property being tested. ABI tests check for the signature of the function inside the contract (including checks on function's state mutability and its output). Tests of level Minimal (resp. Recommended) check for requirements expressed as MUST (resp. SHOULD) in the token standard. Tests of level Desirable check for security properties that were deemed important for ensuring the security of the tokens. Tests of level AddOn Features include tests involving functions not present in the standard but which are common extension (e.g., minting, burning, pausing etc). For the Tether USD, no addon test was executed since it does not contain such. Finally, Fingerprint tests are check on implementation choices made by the contract such as supporting an infinite approval pattern. These are available by clicking on the blue banner. Those tests display in the second part of the report which provides a focus on the results.
The second part of the report page provides a tabulated presentation of the results where one can focus and get the details about the test results in each level.
The first tab is a Detailed report in the JSON format that provides an easy-to-store representation of the report. The detailed report contains the result of each test as well as additional information such as the possible counter-example found and possible error message. Following up are tabs for each testing level. Next is a presentation of the test results using a per-topic scores according to the functions of the contract involved in the test. For instance, a test involving giving some allowance to a user and performing a transferFrom will count for categories approval and transferFrom. Finally, the last tab provides the log generated by our tool.
The third part consists of general pieces of information about the token: its full name, symbol, number of decimal, networks, and total supply. In addition, the standard(s) field indicate the standards witch which the token conforms (by implementing the required functions).
The fourth part is the Application Binary Interface extracted for this token, which can be useful for implementing further analysis on the token.

Let us come back to the second part of the report and in particular the ABI and Desirable tab, for the considered example (USDT). Examining the ABI tab, we can notice that four tests are failing. These are listed below and appear first in the list of tests.

As can be seen, four functions are not conforming to the signatures prescribed by the standard: decimals, transfer, transferFrom and approve. Those functions do not respect the return type prescribed by the standard.

Looking further at the Desirable tests, we can see that 10 test failed. This sheds light on some undesirable behavior of USDT, in particular the possibility of transferring tokens to the zero address and the possibility of taking fees.

We can proceed similarly for another well-used token, BNB and similarly make several findings:

ABI (the following points can be inferred from the “Test Logs” tab of the token report):
- Many of the tests named in category ABI failed as the contract ABI generated during the creation of BNB (which happened a while back) does not contain the field “stateMutability” for every function. Inclusion of state mutability for every function in the contract ABI was only introduced some time after BNB was out in the market.
- The Approval event is not found in the contract ABI of BNB (one can verify by looking through the BNB contract).
Minimal and Desirable:
- Most tests concerning zero amount such as “testZeroTransferToOthersPossible” failed as many of their functions such as “transfer” reject zero value for transfer/approve _value as seen in line 83 of the contract code. However, in EIP-20 standard, it is clearly stated that “Transfers of 0 values MUST be treated as normal transfers and fire the Transfer event.”.

Similarly to what we described for USDT and BNB, analyze your favorite token and get a thorough scan of its properties!

Find the bug

In this activity, the objective is to use ERCx and its test suite to locate the bug introduced in some simple implementation of an ERC-20 token.

We consider the following buggy implementations of an ERC-20 obtained by mutating a simple correct implementation.

For each of the above contracts:

Copy paste the code in the code form of the website or using the VS Code plugin extension.
Examine the report and the failing test(s).
Locate the bug in the contract by identifying the missing precondition or fixing the code.
Run the test suite again to make sure the fix is effective.

Note: We are building a more systematic approach to this exercise in order to have a map from failing tests to concerned function in the code so as to help bug localization. Stay tuned.

Test a list of tokens

In this activity, the objective is to use the Open API to extend ERCx so that it can evaluate a list of tokens. For this, we will make use of the endpoints described in the API page.

A Python Wrapper

To help in using the Open API, we provide a Python wrapper for this API. The wrapper is a set of Python methods that allows to easily call the API endpoints. The wrapper can be retrieved by cloning the following GitHub project:

https://github.com/runtimeverification/ercx-api-wrapper

We describe the provided methods. Methods are organized into categories.

Tokens and their Evaluations

`def get_token_info(self, network: str, address: str)`	Get the information on a token from its network and address.
`def get_token_report(self, network: str, address: str)`	Get the latest report on a token from its network and address.
`def get_token_evaluations(self, network: str, address: str, level: str)`	Get the latest evaluations of a token by test level from its network and address.
`def get_token_test_evaluation(self, network: str, address: str, name: str)`	Get the latest evaluation of the token by test name.

Property Tests

def get_property_tests(self, level: str = '')

The list of tests is available here.

Users

`def get_my_info(self)`	Get the information about the logged-in user.
`def get_my_token_lists(self)`	Get the token lists of the authenticated user.
`def get_shared_token_lists(self)`	Get the token lists shared with the authenticated user.
`def get_bookmarked_tokens(self)`	Get the bookmarked tokens of the authenticated user.
`def get_bookmarked_tokens_count(self)`	Get the bookmarked tokens of the authenticated user.

Token Lists

`def get_token_list_info(self, list_id: str)`	Get the information of a token list by its id.
`def get_tokens_of_list(self, list_id: str)`	Get the tokens of a token list by its id.
`def get_users_of_list(self, list_id: str)`	Get the users of a token list by its id.
`def get_tokens_count_of_list(self, list_id: str)`	Get the count of tokens in a token list by its id.
`def create_token_list(self, name: str, description: str = '')`	Create a token list. Returns the id of the token list.
`def add_token_to_token_list(self, address: str, network: str, token_list: str)`	Add a token to a token list. Returns true if adding was successful.
`def remove_token_from_token_list(self, address: str, network: str, token_list: str)`	Add a token to a token list. Returns true if adding was successful.
`def share_token_list_with_user(self, user_id: str, permission: str, token_list_id: str)`	Add a user to a token list with some permission.
`def unshare_token_list_with_user(self, user_id: str, token_list_id: str)`	Remove a user from a token list.

Proposed Activities

Using the above endpoints in the wrapper, we suggest you implement the following services:

An evaluator for a list of tokens.
Build a list evaluator for your own list of tokens. For this, build a list of tokens to get the reports for the tokens in your list.
A token ranker.
Storing the results obtained with the previous activity, you can consider ranking the tokens according to the number of passing tests, identifying conformant vs non-conformant token with respect to the ERC-20 token standard.

Note: We will soon propose the feature of running the test suite on lists of tokens directly from the web interface. However, the activity should illustrate how to build services and internal evaluations for tokens using the API.