> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/treeverse/dvc/llms.txt
> Use this file to discover all available pages before exploring further.
# DVC Files Format
> Understanding .dvc files, dvc.yaml, and dvc.lock structure
DVC uses several file formats to track data, define pipelines, and lock reproducible states. This guide explains the structure and purpose of each file type.
## File Types Overview
Single-stage files for tracking data
Multi-stage pipeline definitions
Lock file for reproducibility
## .dvc Files (Single-Stage Files)
`.dvc` files are used to track individual data files or directories. They're created with `dvc add` or when defining single-stage operations.
### Basic Structure
A typical `.dvc` file contains output metadata:
```yaml theme={null}
outs:
- md5: a304afb96060aad90176268345e10355
size: 37891850
path: model.pkl
```
### Complete Schema
List of output files or directories tracked by this .dvc file
Path to the file or directory
MD5 checksum of the file or directory
Size in bytes
Number of files (for directories)
Whether to cache this output
Keep output file between runs
Specific remote to use for this output
Whether to push this output to remote storage
List of dependencies (for single-stage files with commands)
Command to execute (for single-stage files)
Working directory for the command
MD5 checksum of the stage definition
Whether the stage is frozen (won't be re-executed)
Always consider this stage as changed
Custom metadata for the stage
Description of the stage
### Examples
```yaml theme={null}
outs:
- md5: 3d1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d
size: 1024000
path: data/dataset.csv
```
```yaml theme={null}
outs:
- md5: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6.dir
size: 50000000
nfiles: 1000
path: data/images
```
Directory checksums end with `.dir` and represent a hash of all files within.
```yaml theme={null}
cmd: python preprocess.py
deps:
- path: raw_data.csv
md5: 5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c
size: 2048000
outs:
- md5: a304afb96060aad90176268345e10355
size: 1536000
path: processed_data.csv
md5: 9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e
```
```yaml theme={null}
outs:
- md5: e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2
size: 5000000
path: large_model.pkl
remote: s3-large-files
push: true
```
```yaml theme={null}
outs:
- md5: 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
size: 1024
path: metrics.json
cache: false
```
Setting `cache: false` is useful for small files like metrics that don't need caching.
## dvc.yaml (Pipeline Files)
`dvc.yaml` files define multi-stage pipelines with dependencies, parameters, and outputs.
### Basic Structure
```yaml theme={null}
stages:
prepare:
cmd: python prepare.py
deps:
- raw_data.csv
outs:
- prepared_data.csv
train:
cmd: python train.py
deps:
- prepared_data.csv
- train.py
params:
- lr
- epochs
outs:
- model.pkl
metrics:
- metrics.json:
cache: false
```
### Complete Schema
Dictionary of pipeline stages, where keys are stage names
Command to execute. Can be a string or list of commands
Working directory for the command (relative to dvc.yaml location)
List of dependency file paths
List of parameters from params.yaml. Can be:
* Simple strings: `["lr", "epochs"]`
* Custom file: `[{"config.yaml": ["model.type"]}]`
List of output files. Can be paths or objects with options
List of metric files (automatically set `cache: false`)
List of plot files with optional configuration
Prevent stage from running
Always run this stage
Custom metadata (preserved but not used by DVC)
Human-readable description
Iterate over items to create multiple stage instances
Define parameter matrix for stage variations
Variables that can be referenced in the pipeline using `${var}`
Global parameter files to track
Global metric files
Global plot definitions
Model registry artifacts
Dataset definitions
### Advanced Examples
```yaml theme={null}
stages:
train:
cmd: python train.py
deps:
- data/train.csv
params:
- model.architecture
- training.epochs
outs:
- model.pkl:
desc: "Trained XGBoost model"
remote: s3-models
metrics:
- metrics.json:
cache: false
plots:
- plots/training_loss.csv:
x: epoch
y: loss
title: "Training Loss"
```
```yaml theme={null}
stages:
build:
cmd:
- echo "Building model..."
- python build.py
- echo "Build complete"
outs:
- model/
```
```yaml theme={null}
stages:
process:
foreach:
- train
- test
- val
do:
cmd: python process.py ${item}
deps:
- raw/${item}.csv
outs:
- processed/${item}.csv
```
This creates three stages: `process@train`, `process@test`, and `process@val`.
```yaml theme={null}
stages:
train:
matrix:
lr: [0.001, 0.01, 0.1]
optimizer: [adam, sgd]
cmd: python train.py --lr ${item.lr} --opt ${item.optimizer}
outs:
- models/${item.lr}-${item.optimizer}.pkl
```
```yaml theme={null}
vars:
- data_dir: /mnt/data
- model_name: xgboost_v2
stages:
train:
cmd: python train.py --data ${data_dir} --name ${model_name}
deps:
- ${data_dir}/train.csv
outs:
- models/${model_name}.pkl
```
```yaml theme={null}
stages:
train:
wdir: ../experiments
cmd: python train.py
deps:
- ../data/dataset.csv
outs:
- model.pkl
```
Dependencies and outputs are relative to the dvc.yaml location, not the working directory.
## dvc.lock (Lock Files)
`dvc.lock` is automatically generated and should not be edited manually. It ensures reproducibility by recording exact states.
### Structure
```yaml theme={null}
schema: '2.0'
stages:
train:
cmd: python train.py
deps:
- path: data/train.csv
md5: a304afb96060aad90176268345e10355
size: 1536000
- path: train.py
md5: 5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c
size: 4096
params:
params.yaml:
lr: 0.001
epochs: 100
outs:
- path: model.pkl
md5: e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2
size: 5000000
```
### Schema Fields
Lock file schema version (currently "2.0")
Locked state of each stage
Exact command that was executed
Dependencies with checksums
Each dependency includes:
* `path`: File path
* `md5`: Checksum
* `size`: File size in bytes
Parameter files with exact values used
Format: `{"params.yaml": {"lr": 0.001}}`
Outputs with checksums and metadata
Each output includes:
* `path`: File path
* `md5`: Checksum (`.dir` suffix for directories)
* `size`: Size in bytes
* `nfiles`: File count (for directories)
* `files`: File listing (when tracked with `--with-files`)
Locked dataset states
### Lock File Features
DVC uses the lock file to determine if a stage needs to be re-executed:
* If dependencies or parameters change, the stage runs again
* If the lock file matches current state, the stage is skipped
Always commit `dvc.lock` to version control. It's essential for reproducibility
and collaboration.
## File Naming Conventions
* `data.csv.dvc`
* `model.pkl.dvc`
* `images.dvc`
* `any_name.dvc`
* `dvc.yaml` (standard)
* `dvc.lock` (auto-generated)
* Custom: `pipeline.yaml` ❌
* Custom: `train.dvc.yaml` ❌
Pipeline files must be named exactly `dvc.yaml`. The `.dvc` extension is only
for single-stage tracking files.
## Best Practices
Always track these files:
* `.dvc` files
* `dvc.yaml`
* `dvc.lock`
* `params.yaml`
Never track:
* Actual data files
* Cache directories
* `.dvc/config.local`
Good:
```yaml theme={null}
stages:
preprocess_data:
train_model:
evaluate_model:
```
Bad:
```yaml theme={null}
stages:
step1:
step2:
step3:
```
```yaml theme={null}
stages:
train:
desc: |
Train XGBoost model using preprocessed data.
Outputs model.pkl and training metrics.
cmd: python train.py
```
```yaml theme={null}
stages:
train:
params:
- model.type
- model.hyperparameters
- training.epochs
- training.batch_size
```
```yaml theme={null}
stages:
train:
meta:
author: data-science-team
model_version: v2.1
experiment_id: exp-2024-001
```
## Related Commands
```bash theme={null}
# Create .dvc file
dvc add data/dataset.csv
# Create pipeline stage
dvc stage add -n train -d data.csv -o model.pkl python train.py
# Run pipeline and update dvc.lock
dvc repro
# Validate dvc.yaml syntax
dvc dag
# Show pipeline structure
dvc dag --md
```
## Next Steps
Learn about DVC configuration files
Configure remote storage backends