Store data

Introduction to setting up, storing, and retrieving using the Filecoin network

Start storing your data on the Filecoin network. This section covers packaging your data, importing it into your local Lotus lite-node, finding a storage provider through the Filecoin Plus miner registry, creating a storage deal, and then waiting for the deal to complete. There’s a lot to do, so let’s dive in!

Things to note #

As you’re going through this section, make a note of the following variables:

Prepare your data #

For this tutorial, we’re going to create a dummy 5GB file full of random data and store it on the Filecoin network.

1. Move into your home folder:

   cd ~
   

2. Create a 5GB block of random data to serve as our payload:

   dd if=/dev/urandom of=5gb-filecoin-payload.bin bs=1M count=5200
   

   This process will take up to 60 seconds to create a dummy file.

We now have our payload file ready to be stored using the Filecoin network.

Add data to Lotus #

We need to tell our Lotus lite-node which file we want to store using Filecoin.

1. Import the payload into the lotus daemon using the import command:

   lotus client import 5gb-filecoin-payload.bin
   

   Lotus creates a directed acyclic graph (DAG) based off the payload. This process takes a few minutes. Once it’s complete, Lotus will output the payload CID.

   Import 3, Root bafykb...
   

   This process takes up to 60 seconds.

2. Make a note of the CID bafykb.... This is your Data CID. We’ll use it in an upcoming section.

Importing custom DAGs #

Advanced IPLD users may want to import custom DAGs into Lotus (you may skip this section if that is not you).

The CAR file format allows to serialize any IPLD-DAG (i.e. a IPLD-CBOR). Custom IPLD-DAGs should be encoded in a well-known format (like CBOR) as otherwise Lotus will not know how to interpret them.

If you built your own CAR file, make sure to import it directly with the --car flag.

Files bigger than a sector #

If your file is larger than a sector for the Filecoin network in use, you will need to split your file into multiple parts first.

Storage miners will specify which size(s) they’re offering so you can select the best option for you. Smaller sectors are faster, while larger sectors are more cost-effective.

Now that Lotus has imported your file, we can create a deal with a Filecoin storage provider to store our data!

Filecoin Plus Registry #

The Filecoin Plus Registry is a collection of geographically diverse storage providers that are willing to accept low-cost or free storage deals from users. The more storage providers that offer storage in different parts of the world, the faster we can work toward Filecoin’s underlying mission to store humanity’s most important information. It can help you compare storage providers based on their location, pricing and data size limitations, and also their reputation based on their historical performance.

Let’s find a couple of storage providers to store our data.

1. Go to Filecoin Plus Registry website.

2. Using the table, find a couple of storage providers that suit your needs. Try to find storage providers that are geographically close to you, minimum file size is lower than 5 GiB, and charge 0 FIL for verified deals.

   

3. Once you find suitable storage providers, you can check more detail info about it by clicking the Arrow next to its reputation score.

   

4. Make sure to write down the IDs of the storage providers you want to use. We’ll be referring to these IDs in the next section.

Filecoin Plus Registry only represents a small portion of the entire Filecoin mining community, you can also use other Filecoin reputation systems like FilRep to check more storage provider metrics, like storage power in the network, reachability and overall success rate.

Now that you’ve found your storage providers, you can move onto creating a storage deal!

Create a deal #

To complete this section, you need the Data CID you received after running lotus client import and the IDs of the storage providers you want to use.

1. Start the interactive deal process:

   lotus client deal
   

   The interactive deal assistant will now ask you some questions.

1. Specify the CID of the payload you want to backup on Filecoin. This is the CID that you got from running lotus client import ~/5gb-filecoin-payload.bin:

   Data CID (from lotus client import): bafykbz...
   

2. Wait for Lotus to finish calculating the size of your payload. Lotus calculates this size by counting the individual bits in your payload to ensure that the size is accurate.

   .. calculating data size
   

   The duration of this process depends on the size of your file and the specification of your Lotus node. In tests, Lotus took around 20 minutes file of a ~7.5GB file with a 4-core CPU and 8GB RAM. These specifications are common for most end-user laptops.

3. Enter the number of days you want to keep this file on Filecoin. The minimum is 180 days:

   Deal duration (days): 180
   

4. Tell Lotus whether or not this is a Filecoin Plus deal. Since you signed up to Filecoin Plus in an earlier step, select yes here:

   Make this a verified deal? (yes/no): yes
   

5. Enter the miner IDs from the previous section with an empty space separating the two IDs:

   Miner Addresses (f0.. f0..), none to find: f01000 f01001
   

6. Confirm your transaction by entering yes:

   -----
   Proposing from f136b5uqa73jni2rr745d3nek4uw6qiy6b6zmmvcq
           Balance: 2 FIL
   
   Piece size: 8GiB (Payload size: 7.445GiB)
   Duration: 7200h0m0s
   Total price: ~0 FIL (0 FIL per epoch)
   Verified: true
   
   Accept (yes/no): yes
   

7. Lotus will returns two Deal CIDs:

   .. executing
   Deal (f01000) CID: bafyreict2zhkbwy2arri3jgthk2jyznck47umvpqis3hc5oclvskwpteau
   Deal (f01001) CID: bafeauyreict2zhkbwy2arri3jgthk2jyznck47umvpqis3hc5oclvskwpt
   

8. Take a note of the deal CIDs baf....

Securing a deal #

Given the network’s current speed and stability, users may find that individual deals with miners fail unexpectedly. For this reason, we suggest making up to 10 deals for each CAR file you want to store. While this may seem a bit over-kill, it’s a simple way to increase the chances of a successful deal and your data being stored. This work-around will become less and less necessary as the network matures.

Check the deal status #

Once the data has been sent to the storage clients, the storage deals can take up to 24 hours to complete. You can check the progress of your deals.

1. List successful and pending deals by using the lotus client list-deals command:

   lotus client list-deals --show-failed
   

1. You can check the progress of any data transfers by running lotus client list-transfers:

   lotus client list-transfers
   

   This command will output something like:

   Sending Channels
   ID                   Status   Sending To   Root Cid     Initiated?  Transferred  Voucher
   1620782601911586915  Ongoing  ...KPFTTwY7  ...zyd3kapm  Y           224.1MiB     ...bqhcidjmajbelhlxfqry3d7qlu3tvar45a"}}
   
   Receiving Channels
   ...
   

   If the output of lotus client list-transfers is empty, then your transfer has finished:

   lotus client list-transfers
   

   Sending Channels
   
   
   Receiving Channels
   
   
   

Deal states #

Because of the complex nature of Lotus and the Filecoin network, deals can be in one of many different states.

Processing states #

The following table is the list of states that a deal should enter, assuming there are no errors. This list is in chronological order, from when the deal is first created to when it has completed successfully:

Error states #

The following deal states mean there was a failure somewhere along the line, in alphabetical order:

Informational states #

The following deal states are informational, and do not mean that a deal has failed. This list is in alphabetical order:

These states come from the Lotus project GitHub repository.

Edit this page on GitHub