Metadata-Version: 2.1
Name: cwltool
Version: 1.0.20190915164430
Summary: Common workflow language reference implementation
Home-page: https://github.com/common-workflow-language/cwltool
Author: Common workflow language working group
Author-email: common-workflow-language@googlegroups.com
License: UNKNOWN
Download-URL: https://github.com/common-workflow-language/cwltool
Description: ==================================================================
        Common Workflow Language tool description reference implementation
        ==================================================================
        
        CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status| |Downloads|
        
        |CommandLineTool Support| |DockerRequirement Support| |EnvVarRequirement Support| |ExpressionTool Support| 
        |InitialWorkDirRequirement Support| |InlineJavascriptRequirement Support| |MultipleInputRequirement Support| |Core Support|
        |ResourceRequirement Support| |ScatterRequirement Support| |SchemaDefRequirement Support| |ShellCommandequirement Support|
        |StepInputRequirement Support| |SubWorkflowRequirement Support| |Workflow Support|
        
        .. |Conformance Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |Linux Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=Linux%20builds
           :target: https://travis-ci.org/common-workflow-language/cwltool
        
        .. |Windows Status| image:: https://img.shields.io/appveyor/ci/mr-c/cwltool/master.svg?label=Windows%20builds
           :target: https://ci.appveyor.com/project/mr-c/cwltool
        
        .. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg
           :target: https://codecov.io/gh/common-workflow-language/cwltool
        
        .. |Downloads| image:: https://pepy.tech/badge/cwltool/month
           :target: https://pepy.tech/project/cwltool
        
        .. |CommandLineTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/command_line_tool.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |DockerRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/docker.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |EnvVarRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/env_var.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |ExpressionTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/expression_tool.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |InitialWorkDirRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/initial_work_dir.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |InlineJavascriptRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/inline_javascript.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |MultipleInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/multiple_input.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |Core Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/required.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |ResourceRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/resource.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |ScatterRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/scatter.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |SchemaDefRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/schema_def.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |ShellCommandequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/shell_command.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |StepInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/step_input.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |SubWorkflowRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/subworkflow.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        .. |Workflow Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/workflow.json
           :target: https://ci.commonwl.org/job/cwltool-conformance/
        
        
        This is the reference implementation of the Common Workflow Language.  It is
        intended to be feature complete and provide comprehensive validation of CWL
        files as well as provide other tools related to working with CWL.
        
        This is written and tested for
        `Python <https://www.python.org/>`_ ``2.7 and 3.x {x = 5, 6, 7}``
        
        The reference implementation consists of two packages.  The ``cwltool`` package
        is the primary Python module containing the reference implementation in the
        ``cwltool`` module and console executable by the same name.
        
        The ``cwlref-runner`` package is optional and provides an additional entry point
        under the alias ``cwl-runner``, which is the implementation-agnostic name for the
        default CWL interpreter installed on a host.
        
        Install
        -------
        
        Your operating system may offer cwltool directly. For [Debian](https://tracker.debian.org/pkg/cwltool "Debian cwltool package tracker") or [Ubuntu](https://launchpad.net/ubuntu/+source/cwltool "Ubuntu Launchpad overview for cwltool") try
        
        .. code:: bash
        
          apt-get install cwltool
        
        Otherwise, to
        avoid conflicting versions of the same library,
        it is recommended to do the following:
        
        .. code:: bash
        
          virtualenv -p python2 venv   # Create a virtual environment, can use `python3` as well
          source venv/bin/activate     # Activate environment before installing `cwltool`
        
        Installing the official package from PyPi (will install "cwltool" package as
        well)
        
        .. code:: bash
        
          pip install cwlref-runner
        
        If installing alongside another CWL implementation then
        
        .. code:: bash
        
          pip install cwltool
        
        Or you can install from source:
        
        .. code:: bash
        
          git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
          cd cwltool         # Switch to source directory
          pip install .      # Install `cwltool` from source
          cwltool --version  # Check if the installation works correctly
        
        Remember, if co-installing multiple CWL implementations then you need to
        maintain which implementation ``cwl-runner`` points to via a symbolic file
        system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_.
        
        =====
        Usage
        =====
        
        Run on the command line
        -----------------------
        
        Simple command::
        
          cwl-runner [tool-or-workflow-description] [input-job-settings]
        
        Or if you have multiple CWL implementations installed and you want to override
        the default cwl-runner use::
        
          cwltool [tool-or-workflow-description] [input-job-settings]
        
        Use with boot2docker
        --------------------
        boot2docker runs Docker inside a virtual machine and it only mounts ``Users``
        on it. The default behavior of CWL is to create temporary directories under e.g.
        ``/Var`` which is not accessible to Docker containers.
        
        To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix``
        and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
        
            $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
        
        Using user-space replacements for Docker
        ----------------------------------------
        
        Some shared computing environments don't support Docker software containers for technical or policy reasons.
        As a work around, the CWL reference runner supports using alternative ``docker`` implementations on Linux
        with the ``--user-space-docker-cmd`` option.
        
        One such "user space" friendly docker replacement is ``udocker`` https://github.com/indigo-dc/udocker and another
        is ``dx-docker`` https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images
        
        udocker installation: https://github.com/indigo-dc/udocker/blob/master/doc/installation_manual.md#22-install-from-indigo-datacloud-repositories
        
        dx-docker installation: start with the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
        
        Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
        
        .. code:: bash
        
          cwltool --user-space-docker-cmd=udocker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json
        
        or
        
        .. code:: bash
        
          cwltool --user-space-docker-cmd=dx-docker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json
        
        ``cwltool`` can use `Singularity <http://singularity.lbl.gov/>`_ version 2.6.1
        or later as a Docker container runtime.
        ``cwltool`` with Singularity will run software containers specified in
        ``DockerRequirement`` and therefore works with Docker images only, native
        Singularity images are not supported. To use Singularity as the Docker container
        runtime, provide ``--singularity`` command line option to ``cwltool``.
        With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except
        those involving Docker container ENTRYPOINTs.
        
        
        .. code:: bash
        
          cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/cat-job.json
        
        Running a tool or workflow from remote or local locations
        ---------------------------------------------------------
        
        ``cwltool`` can run tool and workflow descriptions on both local and remote
        systems via its support for HTTP[S] URLs.
        
        Input job files and Workflow steps (via the `run` directive) can reference CWL
        documents using absolute or relative local filesytem paths. If a relative path
        is referenced and that document isn't found in the current directory then the
        following locations will be searched:
        http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem
        
        You can also use `cwldep <https://github.com/common-workflow-language/cwldep>`
        to manage dependencies on external tools and workflows.
        
        Overriding workflow requirements at load time
        ---------------------------------------------
        
        Sometimes a workflow needs additional requirements to run in a particular
        environment or with a particular dataset.  To avoid the need to modify the
        underlying workflow, cwltool supports requirement "overrides".
        
        The format of the "overrides" object is a mapping of item identifier (workflow,
        workflow step, or command line tool) to the process requirements that should be applied.
        
        .. code:: yaml
        
          cwltool:overrides:
            echo.cwl:
              requirements:
                EnvVarRequirement:
                  envDef:
                    MESSAGE: override_value
        
        Overrides can be specified either on the command line, or as part of the job
        input document.  Workflow steps are identified using the name of the workflow
        file followed by the step name as a document fragment identifier "#id".
        Override identifiers are relative to the toplevel workflow document.
        
        .. code:: bash
        
          cwltool --overrides overrides.yml my-tool.cwl my-job.yml
        
        .. code:: yaml
        
          input_parameter1: value1
          input_parameter2: value2
          cwltool:overrides:
            workflow.cwl#step1:
              requirements:
                EnvVarRequirement:
                  envDef:
                    MESSAGE: override_value
        
        .. code:: bash
        
          cwltool my-tool.cwl my-job-with-overrides.yml
        
        
        Combining parts of a workflow into a single document
        ----------------------------------------------------
        
        Use ``--pack`` to combine a workflow made up of multiple files into a
        single compound document.  This operation takes all the CWL files
        referenced by a workflow and builds a new CWL document with all
        Process objects (CommandLineTool and Workflow) in a list in the
        ``$graph`` field.  Cross references (such as ``run:`` and ``source:``
        fields) are updated to internal references within the new packed
        document.  The top level workflow is named ``#main``.
        
        .. code:: bash
        
          cwltool --pack my-wf.cwl > my-packed-wf.cwl
        
        
        Running only part of a workflow
        -------------------------------
        
        You can run a partial workflow with the ``--target`` (``-t``) option.  This
        takes the name of an output parameter, workflow step, or input
        parameter in the top level workflow.  You may provide multiple
        targets.
        
        .. code:: bash
        
          cwltool --target step3 my-wf.cwl
        
        If a target is an output parameter, it will only run only the steps
        that contribute to that output.  If a target is a workflow step, it
        will run the workflow starting from that step.  If a target is an
        input parameter, it will only run only the steps that are connected to
        that input.
        
        Use ``--print-targets`` to get a listing of the targets of a workflow.
        To see exactly which steps will run, use ``--print-subgraph`` with
        ``--target`` to get a printout of the workflow subgraph for the
        selected targets.
        
        .. code:: bash
        
          cwltool --print-targets my-wf.cwl
        
          cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl
        
        
        Visualizing a CWL document
        --------------------------
        
        The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program.  Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file:
        
        .. code:: bash
        
          cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg
        
        Modeling a CWL document as RDF
        ------------------------------
        
        CWL documents can be expressed as RDF triple graphs.
        
        .. code:: bash
        
          cwltool --print-rdf --rdf-serializer=turtle mywf.cwl
        
        
        Leveraging SoftwareRequirements (Beta)
        --------------------------------------
        
        CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
        may in turn use to resolve to packages in various package managers or
        dependency management systems such as `Environment Modules
        <http://modules.sourceforge.net/>`__.
        
        Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional
        dependency, for this reason be sure to use specify the ``deps`` modifier when
        installing cwltool. For instance::
        
          $ pip install 'cwltool[deps]'
        
        Installing cwltool in this fashion enables several new command line options.
        The most general of these options is ``--beta-dependency-resolvers-configuration``.
        This option allows one to specify a dependency resolver's configuration file.
        This file may be specified as either XML or YAML and very simply describes various
        plugins to enable to "resolve" ``SoftwareRequirement`` dependencies.
        
        To discuss some of these plugins and how to configure them, first consider the
        following ``hint`` definition for an example CWL tool.
        
        .. code:: yaml
        
          SoftwareRequirement:
            packages:
            - package: seqtk
              version:
              - r93
        
        Now imagine deploying cwltool on a cluster with Software Modules installed
        and that a ``seqtk`` module is available at version ``r93``. This means cluster
        users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
        sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
        available on the ``PATH``. A simple dependency resolvers configuration file, called
        ``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
        the correct module environment before executing the above tool would simply be:
        
        .. code:: yaml
        
          - type: modules
        
        The outer list indicates that one plugin is being enabled, the plugin parameters are
        defined as a dictionary for this one list item. There is only one required parameter
        for the plugin above, this is ``type`` and defines the plugin type. This parameter
        is required for all plugins. The available plugins and the parameters
        available for each are documented (incompletely) `here
        <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
        Unfortunately, this documentation is in the context of Galaxy tool
        ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
        
        cwltool is distributed with an example of such seqtk tool and sample corresponding
        job. It could executed from the cwltool root using a dependency resolvers
        configuration file such as the above one using the command::
        
          cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
              tests/seqtk_seq.cwl \
              tests/seqtk_seq_job.json
        
        This example demonstrates both that cwltool can leverage
        existing software installations and also handle workflows with dependencies
        on different versions of the same software and libraries. However the above
        example does require an existing module setup so it is impossible to test this example
        "out of the box" with cwltool. For a more isolated test that demonstrates all
        the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
        
        "Galaxy packages" are a lighter weight alternative to Environment Modules that are
        really just defined by a way to lay out directories into packages and versions
        to find little scripts that are sourced to modify the environment. They have
        been used for years in Galaxy community to adapt Galaxy tools to cluster
        environments but require neither knowledge of Galaxy nor any special tools to
        setup. These should work just fine for CWL tools.
        
        The cwltool source code repository's test directory is setup with a very simple
        directory that defines a set of "Galaxy  packages" (but really just defines one
        package named ``random-lines``). The directory layout is simply::
        
          tests/test_deps_env/
            random-lines/
              1.0/
                env.sh
        
        If the ``galaxy_packages`` plugin is enabled and pointed at the
        ``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement``
        such as the following is encountered.
        
        .. code:: yaml
        
          hints:
            SoftwareRequirement:
              packages:
              - package: 'random-lines'
                version:
                - '1.0'
        
        Then cwltool will simply find that ``env.sh`` file and source it before executing
        the corresponding tool. That ``env.sh`` script is only responsible for modifying
        the job's ``PATH`` to add the required binaries.
        
        This is a full example that works since resolving "Galaxy packages" has no
        external requirements. Try it out by executing the following command from cwltool's
        root directory::
        
          cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \
              tests/random_lines.cwl \
              tests/random_lines_job.json
        
        The resolvers configuration file in the above example was simply:
        
        .. code:: yaml
        
          - type: galaxy_packages
            base_path: ./tests/test_deps_env
        
        It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not
        match the module names for a given cluster. Such requirements can be re-mapped
        to specific deployed packages and/or versions using another file specified using
        the resolver plugin parameter `mapping_files`. We will
        demonstrate this using `galaxy_packages` but the concepts apply equally well
        to Environment Modules or Conda packages (described below) for instance.
        
        So consider the resolvers configuration file
        (`tests/test_deps_env_resolvers_conf_rewrite.yml`):
        
        .. code:: yaml
        
          - type: galaxy_packages
            base_path: ./tests/test_deps_env
            mapping_files: ./tests/test_deps_mapping.yml
        
        And the corresponding mapping configuraiton file (`tests/test_deps_mapping.yml`):
        
        .. code:: yaml
        
          - from:
              name: randomLines
              version: 1.0.0-rc1
            to:
              name: random-lines
              version: '1.0'
        
        This is saying if cwltool encounters a requirement of ``randomLines`` at version
        ``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at
        version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl``
        that contains such a source ``SoftwareRequirement``. To try out this example with
        mapping, execute the following command from the cwltool root directory::
        
          cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \
              tests/random_lines_mapping.cwl \
              tests/random_lines_job.json
        
        The previous examples demonstrated leveraging existing infrastructure to
        provide requirements for CWL tools. If instead a real package manager is used
        cwltool has the oppertunity to install requirements as needed. While initial
        support for Homebrew/Linuxbrew plugins is available, the most developed such
        plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties
        of allowing multiple versions of a package to be installed simultaneously,
        not requiring evalated permissions to install Conda itself or packages using
        Conda, and being cross platform. For these reasons, cwltool may run as a normal
        user, install its own Conda environment and manage multiple versions of Conda packages
        on both Linux and Mac OS X.
        
        The Conda plugin can be endlessly configured, but a sensible set of defaults
        that has proven a powerful stack for dependency management within the Galaxy tool
        development ecosystem can be enabled by simply passing cwltool the
        ``--beta-conda-dependencies`` flag.
        
        With this we can use the seqtk example above without Docker and without
        any externally managed services - cwltool should install everything it needs
        and create an environment for the tool. Try it out with the follwing command::
        
          cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json
        
        The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s
        that allow disambiguation of package names. If the mapping files described above
        allow deployers to adapt tools to their infrastructure, this mechanism allows
        tools to adapt their requirements to multiple package managers. To demonstrate
        this within the context of the seqtk, we can simply break the package name we
        use and then specify a specific Conda package as follows:
        
        .. code:: yaml
        
          hints:
            SoftwareRequirement:
              packages:
              - package: seqtk_seq
                version:
                - '1.2'
                specs:
                - https://anaconda.org/bioconda/seqtk
                - https://packages.debian.org/sid/seqtk
        
        The example can be executed using the command::
        
          cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json
        
        The plugin framework for managing resolution of these software requirements
        as maintained as part of `galaxy-lib <https://github.com/galaxyproject/galaxy-lib>`__ - a small, portable subset of the Galaxy
        project. More information on configuration and implementation can be found
        at the following links:
        
        - `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__
        - `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__
        - `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__
        - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
        - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
        
        Use with GA4GH Tool Registry API
        --------------------------------
        
        Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints.
        
        By default, cwltool searches https://dockstore.org/ .  Use ``--add-tool-registry`` to add other registries to the search path.
        
        For example ::
        
          cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json
        
        and (defaults to latest when a version is not specified) ::
        
          cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json
        
        For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats ::
        
          wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json
          wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam
        
        
        .. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas
        
        ===========
        Development
        ===========
        
        Running tests locally
        ---------------------
        
        -  Running basic tests ``(/tests)``:
        
        To run the basic tests after installing `cwltool` execute the following:
        
        .. code:: bash
        
          pip install -rtest-requirements.txt
          py.test --ignore cwltool/schemas/ --pyarg cwltool
        
        To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments
        first downloading the complete code repository (see the ``git clone`` instructions above) and then run
        the following in the terminal:
        ``pip install tox; tox``
        
        List of all environment can be seen using:
        ``tox --listenvs``
        and running a specfic test env using:
        ``tox -e <env name>``
        and additionally run a specific test using this format:
        ``tox -e py36-unit -- tests/test_examples.py::TestParamMatching``
        
        -  Running the entire suite of CWL conformance tests:
        
        The GitHub repository for the CWL specifications contains a script that tests a CWL
        implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_
        program
        
        Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/master/CONFORMANCE_TESTS.md
        
        Import as a module
        ------------------
        
        Add
        
        .. code:: python
        
          import cwltool
        
        to your script.
        
        The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
        
        .. code:: python
        
          import cwltool.factory
          fac = cwltool.factory.Factory()
        
          echo = fac.make("echo.cwl")
          result = echo(inp="foo")
        
          # result["out"] == "foo"
        
        
        CWL Tool Control Flow
        ---------------------
        
        Technical outline of how cwltool works internally, for maintainers.
        
        #. Use CWL ``load_tool()`` to load document.
        
           #. Fetches the document from file or URL
           #. Applies preprocessing (syntax/identifier expansion and normalization)
           #. Validates the document based on cwlVersion
           #. If necessary, updates the document to latest spec
           #. Constructs a Process object using ``make_tool()``` callback.  This yields a
              CommandLineTool, Workflow, or ExpressionTool.  For workflows, this
              recursively constructs each workflow step.
           #. To construct custom types for CommandLineTool, Workflow, or
              ExpressionTool, provide a custom ``make_tool()``
        
        #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
        
           #. ``job()`` is a generator method (uses the Python iterator protocol)
           #. Each time the ``job()`` method is invoked in an iteration, it returns one
              of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
              there is currently no work ready to run) or end of iteration (indicating
              the process is complete.)
           #. Invoke the runnable item by calling ``run()``.  This runs the tool and gets output.
           #. Output of a process is reported by an output callback.
           #. ``job()`` may be iterated over multiple times.  It will yield all the work
              that is currently ready to run and then yield None.
        
        #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
        
           #. The WorkflowJob iterates over each WorkflowJobStep and determines if the
              inputs the step are ready.
           #. When a step is ready, it constructs an input object for that step and
              iterates on the ``job()`` method of the workflow job step.
           #. Each runnable item is yielded back up to top level run loop
           #. When a step job completes and receives an output callback, the
              job outputs are assigned to the output of the workflow step.
           #. When all steps are complete, the intermediate files are moved to a final
              workflow output, intermediate directories are deleted, and the output
              callback for the workflow is called.
        
        #. ``CommandLineTool`` job() objects yield a single runnable object.
        
           #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a
              ``CommandLineJob`` object
           #. The job method configures the CommandLineJob object by setting public
              attributes
           #. The job method iterates over file and directories inputs to the
              CommandLineTool and creates a "path map".
           #. Files are mapped from their "resolved" location to a "target" path where
              they will appear at tool invocation (for example, a location inside a
              Docker container.)  The target paths are used on the command line.
           #. Files are staged to targets paths using either Docker volume binds (when
              using containers) or symlinks (if not).  This staging step enables files
              to be logically rearranged or renamed independent of their source layout.
           #. The ``run()`` method of CommandLineJob executes the command line tool or
              Docker container, waits for it to complete, collects output, and makes
              the output callback.
        
        
        Extension points
        ----------------
        
        The following functions can be passed to main() to override or augment
        the listed behaviors.
        
        executor
          ::
        
            executor(tool, job_order_object, runtimeContext, logger)
              (Process, Dict[Text, Any], RuntimeContext) -> Tuple[Dict[Text, Any], Text]
        
          An implementation of the toplevel workflow execution loop, should
          synchronously run a process object to completion and return the
          output object.
        
        versionfunc
          ::
        
            ()
              () -> Text
        
          Return version string.
        
        logger_handler
          ::
        
            logger_handler
              logging.Handler
        
          Handler object for logging.
        
        The following functions can be set in LoadingContext to override or
        augment the listed behaviors.
        
        fetcher_constructor
          ::
        
            fetcher_constructor(cache, session)
              (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher
        
          Construct a Fetcher object with the supplied cache and HTTP session.
        
        resolver
          ::
        
            resolver(document_loader, document)
              (Loader, Union[Text, dict[Text, Any]]) -> Text
        
          Resolve a relative document identifier to an absolute one which can be fetched.
        
        The following functions can be set in RuntimeContext to override or
        augment the listed behaviors.
        
        construct_tool_object
          ::
        
            construct_tool_object(toolpath_object, loadingContext)
              (MutableMapping[Text, Any], LoadingContext) -> Process
        
          Hook to construct a Process object (eg CommandLineTool) object from a document.
        
        select_resources
          ::
        
            selectResources(request)
              (Dict[str, int], RuntimeContext) -> Dict[Text, int]
        
          Take a resource request and turn it into a concrete resource assignment.
        
        make_fs_access
          ::
        
            make_fs_access(basedir)
              (Text) -> StdFsAccess
        
          Return a file system access object.
        
        In addition, when providing custom subclasses of Process objects, you can override the following methods:
        
        CommandLineTool.make_job_runner
          ::
        
            make_job_runner(RuntimeContext)
              (RuntimeContext) -> Type[JobBase]
        
          Create and return a job runner object (this implements concrete execution of a command line tool).
        
        Workflow.make_workflow_step
          ::
        
            make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv)
              (Dict[Text, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep
        
          Create and return a workflow step object.
        
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Healthcare Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: OS Independent
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: Microsoft :: Windows :: Windows 10
Classifier: Operating System :: Microsoft :: Windows :: Windows 8.1
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: Utilities
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4
Description-Content-Type: text/x-rst
Provides-Extra: deps
