Running Psyche on-chain

To build the Solana programs, you'll need a handful of Solana tools installed. See the setup if you're not using Nix.

To start, you'll need to create a Solana wallet to fund your transactions.

solana-keygen new

Run on a local validator (localnet)

In a new terminal, run the following command to:

setup a solana-test-validator
Deploy all the required programs
Create a local run with name <RUN_ID>. If no run name is provided, the name test will be used by default.

just setup-solana-localnet-test-run run_id=<RUN_ID>

Then, in another terminal, run a client to train the test model and joining the run with name RUN_ID. If no run name is provided, the name test will be used by default.

just start-training-localnet-client run_id=<RUN_ID>

This will start a run to train a 1.1b parameter model with all the parallelism features enabled. For a more lightweight run to avoid OOM errors, or just to use your hardware less, (we see you 8gb VRAM cards!) there's also:

just setup-solana-localnet-light-test-run
just start-training-localnet-light-client

By default the client will use the private key generated by solana-keygen new above (located by default in ~/.config/solana/id.json).

To spin up another client and join the run we'll have to create another keypair using:

solana-keygen new --outfile <PATH_TO_NEW_KEYPAIR>

and run the same just command but using the new created keypair:

WALLET_FILE=<PATH_TO_NEW_KEYPAIR> just start-training-localnet-client run_id=<RUN_ID>

or:

WALLET_FILE=<PATH_TO_NEW_KEYPAIR> just start-training-localnet-light-client run_id=<RUN_ID>

Run on Solana's Devnet

You'll need to fund your wallet to make transactions on Devnet. You can request an airdrop from the Solana foundation of up to 10 devnet sol every 8 hours. Simply run

solana-keygen pubkey

and paste the resulting key into the airdrop website.

You can then use the same steps for deploying the programs, creating a run, and training on localnet above, but using following just commands:

just setup-solana-devnet-test-run
just start-training-devnet-client

alongside the -light variants

just setup-solana-devnet-light-test-run
just start-training-devnet-light-client

Regenerating program keypairs

If you're developing things that change the structure of the program's accounts layout, deploying an update to the coordinator program will likely cause breakage with existing runs that have coordinator accounts already instantiated.

Any programs, including the Psyche website's indexer, will fail to read the content of the on-chain data if you use a new IDL with an old in-memory layout.

Therefore, changes to the data structures that end up on-chain will require a deployment of a new coordinator program under a new ProgramID to prevent breakage of existing runs.

In order to do this by yourself, you'll need to generate a new ProgramID (and keypair).

To deploy a program to devnet or localnet with a new program keypair, regenerate its devnet/localnet keypair file (checked into the repo!)

For the solana coordinator, that would be:

solana-keygen new -o architectures/decentralized/solana-coordinator/target/deploy/psyche_solana_coordinator-keypair.json -f

You can see the newly generated program ID by running

solana-keygen pubkey architectures/decentralized/solana-coordinator/target/deploy/psyche_solana_coordinator-keypair.json

Make sure to then update the declare_id's content with the new keys before deploying the new development contracts, either manually or with anchor keys sync in the appropriate project folder.

if you want to push these changes to the repo, you'll need to use git add -f, since they're normally .gitignored.