Running Psyche on-chain

To build the Solana programs, you’ll need a handful of Solana tools installed. See the setup if you’re not using Nix. If you’re using Nix, make sure you are in the development environment by running nix develop.

To start, you’ll need to create a Solana wallet to fund your transactions.

solana-keygen new

By default, the keypair will be generated in ~/.config/solana/id.json.

Run on a local validator (localnet)

To quickly test decentralized training, you can spin up a Solana validator locally and fund your Solana wallet with fake tokens to make transactions. To set up a new training run with this tool, in a new terminal run the following command:

just dev setup-solana-localnet-test-run run_id=<RUN_ID>

This will:

  • Set up a solana-test-validator
  • Deploy all the required programs (Coordinator and Authorizer)
  • Create a local run with the name <RUN_ID>. If no run name is provided, the name test will be used by default. The run ID should not exceed 32 characters; it will be truncated if it exceeds this limit.

Then, in another terminal, run a client to train the test model and join the run with the name <RUN_ID>.

just dev start-training-localnet-client run_id=<RUN_ID>

This will start a run to train a 1.1B parameter model with all the parallelism features enabled. This Psyche client will use a temporary private key, which will be generated and deleted automatically when running the command above. If you want to inspect these keys, they will be stored in ~/.config/solana/solana-keys. To run it with a specific private key, you can run the same command while adding the WALLET_FILE environment variable:

WALLET_FILE=/path/to/wallet.json just dev start-training-localnet-client run_id=<RUN_ID>

For a more lightweight run to avoid OOM errors, or just to use less of your hardware (we see you, 8 GB VRAM cards!), there’s also:

just dev setup-solana-localnet-light-test-run
just dev start-training-localnet-light-client

This will train a 12M parameter model, which should fit on most GPUs.

To spin up another client and join the run, you can run the same command as before:

just dev start-training-localnet-client run_id=<RUN_ID>

or

just dev start-training-localnet-light-client run_id=<RUN_ID>

This will create a new temporary Solana keypair in ~/.config/solana/solana-keys, which will be removed when the client is stopped, so you can spawn as many clients as you want.

Run on Solana Devnet

You’ll need to fund your wallet to make transactions on Devnet. You can request an airdrop from the Solana Foundation of up to 10 devnet SOL every 8 hours. To get your public key, run:

solana-keygen pubkey <PATH_TO_KEYPAIR>

If no path to a keypair is provided, it will use the default keypair located at ~/.config/solana/id.json. Paste the resulting key into the airdrop website to receive tokens.

You can then follow the same steps for deploying the programs, creating a run, and training as on localnet, but using the following just commands:

just dev setup-solana-devnet-test-run
just dev start-training-devnet-client

along with the -light variants:

just dev setup-solana-devnet-light-test-run
just dev start-training-devnet-light-client

Remember to set the WALLET_FILE environment variable to the path of your Solana keypair file when running the training commands, since this will be the wallet holding the devnet funds.

These commands work almost the same as the localnet ones, but they use the public Solana Devnet RPC endpoint (https://api.devnet.solana.com). Also, for all programs (Coordinator, Authorizer, and Treasurer), we need to generate new program IDs—basically the “addresses” where the contracts will be deployed—since the current IDs are used by the Psyche team for development and can’t be overridden. More details on how to update program IDs can be found in the changing contracts section.

Creating a permissioned run

All the commands and setups above use a permissionless run by default. In most testing cases, this is fine, since you can join any number of clients without restrictions. If a permissioned run is needed, you’ll have to take a few extra steps.

You have the same variants as before, but with the permissioned option enabled. These commands won’t create the permissionless authorization and will instead allow you to create the required authorizations manually. The commands are:

# Localnet
just dev setup-solana-localnet-permissioned-test-run
just dev setup-solana-localnet-permissioned-light-test-run
just dev setup-solana-localnet-permissioned-test-run-treasurer
just dev setup-solana-localnet-permissioned-light-test-run-treasurer

# Devnet
just dev setup-solana-devnet-permissioned-test-run
just dev setup-solana-devnet-permissioned-light-test-run
just dev setup-solana-devnet-permissioned-test-run-treasurer
just dev setup-solana-devnet-permissioned-light-test-run-treasurer

depending on your needs.

You can then create an authorization manually by specifying who grants the authorization (the run owner) and who receives it. Run:

cargo run --release --bin psyche-solana-client -- \
    join-authorization-create \
    --rpc [RPC] \
    --wallet-private-key-path [JOIN_AUTHORITY_KEYPAIR_FILE] \
    --authorizer [USER_MASTER_PUBKEY]

Here, the --wallet-private-key-path is the path to the Solana KeyPair that will handle authorization to join and the --authorizer is the pubkey of the account that will receive the authorization. To get the pubkey of a KeyPair file you can use the solana-keygen pubkey <FILE> command.

You can then join any authorized client by running the training commands described above, adding the authorized key as an environment variable, for example:

AUTHORIZER=<GRANTEE_PUBKEY> just dev start-training-localnet-light-client

Running a run with rewards

There’s another program that adds a new layer to the Psyche run called the Treasurer. When this program is deployed, it adds a rewards layer on top of the Coordinator, calculating how much of a specific token each client receives for their training time. This contract isn’t required to test a run, but it adds reward functionality if you want to test it. You can find a more in-depth explanation in the rewards section.

To test this, all the commands mentioned above also have variants that include the Treasurer, such as:

# Localnet
just dev setup-solana-localnet-test-run-treasurer
just dev setup-solana-localnet-light-test-run-treasurer

# Devnet
just dev setup-solana-devnet-test-run-treasurer
just dev setup-solana-devnet-light-test-run-treasurer

These commands deploy the Treasurer alongside the other contracts, create a new test token using the SPL Token tool on the selected network, and top up the run with rewards and collateral for clients that train for more than one epoch.

All these commands also have permissioned variants.

Recovering dev tokens

Most devnet tokens are used to deploy the various contracts. You can reclaim these tokens once you’ve finished testing, which is useful since the Solana devnet faucet is limited. Run:

just dev close-dev-programs

This will close all deployed accounts on devnet and return the tokens to the wallet used for deployment. Be aware that this is an irreversible action: once a program is closed, you can’t reuse the same program ID and must generate a new one.

Psyche decentralized client reference

All the commands above use the same psyche-solana-client package with specific parameters for quick local testing, but it supports many different configurations to test various scenarios.

Here’s a summary of the available commands and options:

Command-line options # Command-Line Help for `psyche-solana-client`

This document contains the help content for the psyche-solana-client command-line program.

Command Overview:

psyche-solana-client

Usage: psyche-solana-client <COMMAND>

Subcommands:
  • show-static-p2p-identity —
  • create-static-p2p-identity —
  • train —
  • predownload —

psyche-solana-client show-static-p2p-identity

Usage: psyche-solana-client show-static-p2p-identity [IDENTITY_SECRET_KEY_PATH]

Arguments:
  • <IDENTITY_SECRET_KEY_PATH>

psyche-solana-client create-static-p2p-identity

Usage: psyche-solana-client create-static-p2p-identity <SAVE_PATH>

Arguments:
  • <SAVE_PATH>

psyche-solana-client train

Usage: psyche-solana-client train [OPTIONS] --run-id <RUN_ID>

Options:

  • --rpc <RPC>

    Default value: http://127.0.0.1:8899

  • --ws-rpc <WS_RPC>

    Default value: ws://127.0.0.1:8900

  • -w, --wallet-private-key-path <WALLET_PRIVATE_KEY_PATH>

  • -i, --identity-secret-key-path <IDENTITY_SECRET_KEY_PATH> — Path to the clients secret key. Create a new random one running openssl rand 32 > secret.key. If not provided a random one will be generated

  • --bind-p2p-port <BIND_P2P_PORT> — Sets the port for the client's P2P network participation. If not provided, a random port will be chosen

  • --bind-p2p-interface <BIND_P2P_INTERFACE> — Sets the network interface for the client's P2P network participation. If not provided, will bind to all interfaces

  • --iroh-relay <IROH_RELAY> — What relays to use - public n0 or the private Psyche ones

    Default value: psyche

  • --iroh-discovery <IROH_DISCOVERY> — What discovery to use - public n0 or local

    Default value: n0

  • --logs <LOGS> — Sets clients logs interface tui: Enables a terminal-based graphical interface for monitoring analytics. console: standard logs json: standard logs with json format

    Default value: tui

    Possible values: tui, console, json, none

  • --oltp-auth-header <OLTP_AUTH_HEADER> — An auth header string for an opentelemetry endpoint. Used for both logging and metrics

  • --oltp-metrics-url <OLTP_METRICS_URL> — A URL for sending opentelemetry metrics. probably ends in /v1/metrics

  • --oltp-tracing-url <OLTP_TRACING_URL> — A URL for sending opentelemetry traces. probably ends in /v1/traces

  • --oltp-logs-url <OLTP_LOGS_URL> — A URL for sending opentelemetry logs. probably ends in /v1/logs

  • --oltp-report-interval <OLTP_REPORT_INTERVAL> — how often to report metrics thru opentelemetry

    Default value: 60.0

  • --metrics-local-port <METRICS_LOCAL_PORT> — If present, output some metrics & stats via this TCP port in JSON format. Useful for debugging or local integration

  • --run-id <RUN_ID> — A unique identifier for the training run. This ID allows the client to join a specific active run

  • --data-parallelism <DATA_PARALLELISM>

    Default value: 1

  • --tensor-parallelism <TENSOR_PARALLELISM>

    Default value: 1

  • --micro-batch-size <MICRO_BATCH_SIZE>

    Default value: 1

  • --write-gradients-dir <WRITE_GRADIENTS_DIR> — If provided, every shared gradient this client sees will be written to this directory

  • --eval-tasks <EVAL_TASKS>

  • --eval-seed <EVAL_SEED>

    Default value: 42

  • --eval-task-max-docs <EVAL_TASK_MAX_DOCS>

  • --prompt-task

  • --checkpoint-dir <CHECKPOINT_DIR> — If provided, every model parameters update will be save in this directory after each epoch

  • --hub-repo <HUB_REPO> — Path to the Hugging Face repository containing model data and configuration

  • --hub-max-concurrent-downloads <HUB_MAX_CONCURRENT_DOWNLOADS>

    Default value: 3

  • --wandb-project <WANDB_PROJECT>

  • --wandb-run <WANDB_RUN>

  • --wandb-group <WANDB_GROUP>

  • --wandb-entity <WANDB_ENTITY>

  • --write-log <WRITE_LOG>

  • --optim-stats-steps <OPTIM_STATS_STEPS>

  • --grad-accum-in-fp32

    Default value: false

  • --dummy-training-delay-secs <DUMMY_TRAINING_DELAY_SECS>

  • --max-concurrent-parameter-requests <MAX_CONCURRENT_PARAMETER_REQUESTS>

    Default value: 4

  • --max-concurrent-downloads <MAX_CONCURRENT_DOWNLOADS>

    Default value: 4

  • --device <DEVICE> — Device(s) to use: auto, cpu, mps, cuda, cuda:N, cuda:X,Y,Z

    Default value: auto

  • --sidecar-port <SIDECAR_PORT>

  • --delete-old-steps

    Default value: true

  • --keep-steps <KEEP_STEPS>

    Default value: 3

  • --rpc-2 <RPC_2>

    Default value: ``

  • --ws-rpc-2 <WS_RPC_2>

    Default value: ``

  • --rpc-3 <RPC_3>

    Default value: ``

  • --ws-rpc-3 <WS_RPC_3>

    Default value: ``

  • --authorizer <AUTHORIZER>

psyche-solana-client predownload

Usage: psyche-solana-client predownload [OPTIONS] --run-id <RUN_ID>

Options:

  • --rpc <RPC>

    Default value: http://127.0.0.1:8899

  • --ws-rpc <WS_RPC>

    Default value: ws://127.0.0.1:8900

  • -r, --run-id <RUN_ID>

  • --model

  • --eval-tasks <EVAL_TASKS>

  • --hub-max-concurrent-downloads <HUB_MAX_CONCURRENT_DOWNLOADS>

    Default value: 3

This document was generated automatically by clap-markdown.

Changing contracts

Psyche uses two main accounts deployed to Solana—the Coordinator and the Authorizer—and one optional account, the Treasurer. If you’re developing changes that modify the on-chain account layout, deploying an updated Coordinator program will likely break existing runs that already have coordinator accounts instantiated.

Because of this, changes to on-chain data structures require deploying a new Coordinator program under a new Program ID to avoid breaking existing runs.

To do this yourself, you’ll need to generate a new Program ID (and keypair).

To deploy a program to devnet or localnet with a new program keypair, regenerate its devnet/localnet keypair file (which is checked into the repo).

For the Solana Coordinator, run:

solana-keygen new -o architectures/decentralized/solana-coordinator/target/deploy/psyche_solana_coordinator-keypair.json -f

You can view the newly generated program ID with:

solana-keygen pubkey architectures/decentralized/solana-coordinator/target/deploy/psyche_solana_coordinator-keypair.json

Make sure to update the declare_id value with the new key before deploying the updated contracts, either manually or using anchor keys sync in the appropriate project folder.

If you want to push these changes to the repo, you’ll need to use git add -f, since these files are normally .gitignored.