One of the main ideas of doit and other build-tools is to check if the tasks/targets are up-to-date. In case there is no modification in the dependencies and the targets already exist, it skip the task execution to save time, as it would produce the same output from the previous run.
i.e. In a compilation task the source file is a file_dep, the object file is a target.
def task_compile():
return {'actions': ["cc -c main.c"],
'file_dep': ["main.c", "defs.h"],
'targets': ["main.o"]
}
doit automatically keeps track of file dependencies. It saves the signature (MD5) of the dependencies every time the task is completed successfully.
So if there are no modifications to the dependencies and you run doit again. The execution of the task’s actions is skipped.
$ doit
. compile
$ doit
-- compile
Note the -- (2 dashes, one space) on the command output on the second time it is executed. It means, this task was up-to-date and not executed.
Different from most build-tools dependencies are on tasks not on targets. So doit can take advantage of the “execute only if not up-to-date” feature even for tasks that not define targets.
Lets say you work with a dynamic language (python in this example). You don’t need to compile anything but you probably wants to apply a lint-like tool (PyChecker) to your source code files. You can define the source code as a dependency to the task.
def task_checker():
return {'actions': ["pychecker sample.py"],
'file_dep': ["sample.py"]}
$ doit
. checker
$ doit
-- checker
Note the -- again to indicate the execution was skipped.
Traditional build-tools can only handle files as “dependencies”. doit has several ways to check for dependencies, those will be introduced later.
Targets can be any file path (a file or folder). If a target doesn’t exist the task will be executed. There is no limitation on the number of targets a task may define.
Lets take the compilation example again.
def task_compile():
return {'actions': ["cc -c main.c"],
'file_dep': ["main.c", "defs.h"],
'targets': ["main.o"]
}
If there are no changes in the dependency the task execution is skipped. But if the target is removed the task is also executed again. But only if does not exist. If the target is modified but the dependencies do not change the task is not executed again.
$ doit
. compile
$ doit
-- compile
$ rm main.o
$ doit
. compile
$ echo xxx > main.o
$ doit
-- compile
If your tasks interact in a way where the target (output) of one task is a file_dep (input) of another task, doit will make sure your tasks are executed in the correct order.
def task_modify():
return {'actions': ["echo bar > foo.txt"],
'file_dep': ["foo.txt"],
}
def task_create():
return {'actions': ["touch foo.txt"],
'targets': ["foo.txt"]
}
$ doit
. create
. modify
In some cases you can not determine if a task is “up-to-date” only based on some input files, the input could come from a database or an extrernal process. doit defines a “result-dependency” to deal with these cases without need to create an intermediate file with the reulsts of the process.
i.e. Suppose you want to send an email everytime you run doit on a bazaar repository that contains a new revision number.
def task_version():
return {'actions': ['bzr revno']}
def task_send_email():
return {'actions': ['echo "TODO: send an email"'],
'result_dep': ['version']}
Note the result_dep with the name of the task (version). doit will keep track of the output of the task version and will execute send_email only when the bazaar repository has a new version since last time doit was executed.
The “result” from the dependent task compared between different runs is given by its last action. The content for python-action is the value of the returned string or dict. For cmd-actions is the output send to stdout plus stderr.
Tasks can save computed values by returning a dictionary on it’s python-actions. The values must be JSON encodable.
These values can be used on uptodate and getargs. Check those sections for examples.
doit also supports checking if a task is up-to-date directly by the parameter uptodate.
This can be used in cases where you need to some kind of calculation to determine if the task is up-to-date or not. i.e. you check if a value in a database is smaller than a certain value.
uptodate is a list where each element can be True, False, None or a callable.
- False indicates that the task is NOT up-to-date
- True indicates that the task is up-to-date
- None values will just be ignored. This is used when the value is dinamically calculated
Note
uptodate value True does not override others up-to-date checks. It is one more way to check if task is not up-to-date.
i.e. if uptodate==True but a file_dep changes the task is still considered not up-to-date.
uptodate elements can also be a callable that returns False, True or None. This callable must take at least two positional parameters task and values. The callable can also be represented by a tuple (callable, args, kwargs).
- task parameter will give you access to task object. So you have access to its metadata and opportunity to modifiy the task itself!
- values is a dictionary with the computed values saved in the last successful execution of the task.
def fake_get_value_from_db():
return 5
def check_outdated(task, values):
total = fake_get_value_from_db()
return total > 10
def task_put_more_stuff_in_db():
def put_stuff(): pass
return {'actions': [put_stuff],
'uptodate': [check_outdated],
}
Here is a simple example on how to use the positional parameters of uptodate.
Sometimes there is no dependency for a task but you do not want to execute it all the time. If “run_once” the task will not be executed again after the first successful run. This is mostly used together with targets.
Suppose you need to download something from internet. There is no dependency though you do not want to download it many times.
def run_once(task, values):
def save_executed():
return {'run-once': True}
task.insert_action(save_executed)
return values.get('run-once', False)
First a new action is added to task, this action will just save a computed value for the task. So whenever this task is executed this task value ‘run-once’ will be saved. The ‘values’ parameter contains the dict with the values saved from last successful execution of the task. run_once will just check if this task was executed looking for the run-once in `values.
Since “run_once” is a common pattern it is included in doit.tools module.
from doit.tools import run_once
def task_get_pylogo():
url = "http://python.org/images/python-logo.gif"
return {'actions': ["wget %s" % url],
'targets': ["python-logo.gif"],
'uptodate': [run_once],
}
Note that even with run_once the file will be downloaded again in case the target is removed.
$ doit
. get_pylogo
$ doit
-- get_pylogo
$ rm python-logo.gif
$ doit
. get_pylogo
Note
If you want to use other tasks to calculate uptodate you should combine it with calc_deps.
Note
doit.tools includes four uptodate callables that are ready to use. run_once as described above. timeout that will make your task up-to-date only during a certain time interval. config_changed will check for changes in a “configuration” string or dictionary. check_timestamp_unchanged will check access, status change/create or modify timestamp of a given file/directory.
A task is not up-to-date if any of file_dep, result_dep, or uptodate is not up-to-date or there is a missing target. If a task does not define any of these dependencies it will always be executed.
Apart from these dependencies used to determine if a task is up-to-date or not. doit also include other kind of dependencies to help you combine tasks so they are executed in appropriate order.
Calculation of dependencies might be an expensive operation, so not suitable to be done on task-generators. For this situation is better to delegate the calculation of dependencies to another task. The task calcutating dependencies must have a python-action returning a dictionary with file_dep, task_dep, result_dep, uptodate or another calc_dep.
On the example below mod_deps prints on the screen all direct dependencies from a module. The dependencies itself are calculated on task get_dep (note: get_dep has a fake implementation where the results are taken from a dict).
DOIT_CONFIG = {'verbosity': 2}
MOD_IMPORTS = {'a': ['b','c'],
'b': ['f','g'],
'c': [],
'f': ['a'],
'g': []}
def print_deps(mod, dependencies):
print "%s -> %s" % (mod, dependencies)
def task_mod_deps():
"""task that depends on all direct imports"""
for mod in MOD_IMPORTS.iterkeys():
yield {'name': mod,
'actions': [(print_deps,(mod,))],
'uptodate': [False],
'file_dep': [mod],
'calc_dep': ["get_dep:%s" % mod],
}
def get_dep(mod):
# fake implementation
return {'file_dep': MOD_IMPORTS[mod]}
def task_get_dep():
"""get direct dependencies for each module"""
for mod in MOD_IMPORTS.iterkeys():
yield {'name': mod,
'actions':[(get_dep,[mod])],
'file_dep': [mod],
}
Warning
On initialization doit create implicit task dependencies if a file_dep is a target from another task. file_dep‘s included through calc_dep are not checked against others task targets.
If you are creating the files with another tasks you can achieve it by setting a task_dep explicitily. If you think this is getting on your way get, please complain on discussion group!
It is used to enforce tasks are executed on the desired order. By default tasks are executed on the same order as they were defined in the dodo file. To define a dependency on another task use the task name (whatever comes after task_ on the function name) in the “task_dep” attribute.
Note
A task-dependency only indicates that another task should be “executed” before itself. The task-dependency might not really be executed if it is up-to-date.
Note
task-dependency s are not used to determine if a task is up-to-date or not. If a task defines only task-dependency it will always be executed.
This example we make sure we include a file with the latest revision number of the mercurial repository on the tar file.
def task_tar():
return {'actions': ["tar -cf foo.tar *"],
'task_dep':['version'],
'targets':['foo.tar']}
def task_version():
return {'actions': ["hg tip --template '{rev}' > revision.txt"]}
$ doit
. version
. tar
Note
a task-dependency is executed before checking if the task is up-to-date. If you want to execute another task only if it is not up-to-date you should use a setup task.
You can define group of tasks by adding tasks as dependencies and setting its actions to None.
def task_foo():
return {'actions': ["echo foo"]}
def task_bar():
return {'actions': ["echo bar"]}
def task_mygroup():
return {'actions': None,
'task_dep': ['foo', 'bar']}
Note that tasks are never executed twice in the same “run”.
Some tasks require some kind of environment setup. Tasks may have a list of “setup” tasks.
Task may also define ‘teardown’ actions. These actions are executed after all tasks have finished their execution. They are executed in reverse order their tasks were executed.
Example:
### task setup env. good for functional tests!
DOIT_CONFIG = {'verbosity': 2,
'default_tasks': ['withenvX', 'withenvY']}
def start(name):
print "start %s" % name
def stop(name):
print "stop %s" % name
def task_setup_sample():
for name in ('setupX', 'setupY'):
yield {'name': name,
'actions': [(start, (name,))],
'teardown': [(stop, (name,))],
}
def task_withenvX():
for fin in ('a','b','c'):
yield {'name': fin,
'actions':['echo x %s' % fin],
'setup': ['setup_sample:setupX'],
}
def task_withenvY():
return {'actions':['echo y'],
'setup': ['setup_sample:setupY'],
}
getargs provides a way to use values computed from one task in another task. The values are taken from “saved computed values” (returned dict from a python-action).
For cmd-action use dictionary-based string formatting.
For python-action the action callable parameter names must match with keys from getargs.
getargs is a dictionary where the is the argument name used on actions, and the value is a tuple with 2 strings: task name, “value name”.
DOIT_CONFIG = {'default_tasks': ['use_cmd', 'use_python']}
def task_compute():
def comp():
return {'x':5,'y':10, 'z': 20}
return {'actions': [(comp,)]}
def task_use_cmd():
return {'actions': ['echo x=%(x)s, z=%(z)s'],
'getargs': {'x': ('compute', 'x'),
'z': ('compute', 'z')},
'verbosity': 2,
}
def task_use_python():
return {'actions': [show_getargs],
'getargs': {'x': ('compute', 'x'),
'y': ('compute', 'z')},
'verbosity': 2,
}
def show_getargs(x, y):
print "this is x:%s" % x
print "this is y:%s" % y
The values are being passed on to a python-action you can pass the whole dict by specifying the value name as None.
def task_compute():
def comp():
return {'x':5,'y':10, 'z': 20}
return {'actions': [(comp,)]}
def show_getargs(values):
print values
def task_args_dict():
return {'actions': [show_getargs],
'getargs': {'values': ('compute', None)},
'verbosity': 2,
}
If a group-task is used, the values from all its sub-tasks are passed as a list.
def task_compute():
def comp(x):
return {'x':x}
yield {'name': '5',
'actions': [ (comp, [5]) ]
}
yield {'name': '7',
'actions': [ (comp, [7]) ]
}
def show_getargs(values):
print values
assert sum(v['x'] for v in values) == 12
def task_args_dict():
return {'actions': [show_getargs],
'getargs': {'values': ('compute', None)},
'verbosity': 2,
}
Note
getargs creates an implicit setup-task.
It is common situation to use task information such as targets, dependencies, or changed in its own actions. Note: Dependencies are only file-dependencies.
For cmd-action you can use the python notation for keyword substitution on strings. The string will contain all values separated by a space (” ”).
For python-action create a parameter in the function doit will take care of passing the value when the function is called. The values are passed as list of strings.
def task_hello():
"""hello from shell & python! """
def python_hello(targets):
with open(targets[0], "a") as output:
output.write("Python says Hello World!!!\n")
return {'actions': ['echo Hello World!!! > %(targets)s',
python_hello],
'targets': ["hello.txt"]
}