convert to WDL: force co-localization of secondary_files #90

Activity

1. illusional commented on Aug 27, 2021

   

   illusional

   on Aug 27, 2021

   Member

   More actions

   Thanks @mr-c, Janis should already do this on translation to WDL:

   janis-core/janis_core/translations/wdl.py

   Line 2079 in 0b5b79f

   commands.append(wdl.Task.Command(f"cp -f '~{{{sectag}}}' {dest}"))

   (Because I saw the same problems haha)
2. mr-c commented on Aug 29, 2021

   

   mr-c

   on Aug 29, 2021

   ContributorAuthor

   More actions

   @illusional Huh, I'm not seeing that behaviour for either a single File input with a secondary file specifier, nor an array of Files with a secondary file specifier:

   secondary_files.cwl

   cwlVersion: v1.2
   class: CommandLineTool
   
   arguments:
     - {shellQuote: false, valueFrom: "ls | grep -v lsout"}
   inputs:
     input:
       type: File
       secondaryFiles:
         - ^.tar
   stdout: lsout
   outputs:
     output:
       type: File
       outputBinding:
         glob: lsout
   requirements:
     ShellCommandRequirement: {}

   $ janisdk fromcwl secondary_files.cwl -o .
   

   secondary_files_v0_1_0.py

   from datetime import datetime
   from typing import List, Optional, Dict, Any
   
   from janis_core import *
   from janis_core.types.common_data_types import GenericFileWithSecondaries, File
   
   Secondary_Files_V0_1_0 = CommandToolBuilder(tool="secondary_files", base_command=None, inputs=[ToolInput(tag="inp", input_type=GenericFileWithSecondaries(secondaries=["^.tar"]), doc=InputDocumentation(doc=None))], outputs=[ToolOutput(tag="outp", output_type=File(), selector=WildcardSelector(wildcard="lsout"), doc=OutputDocumentation(doc=None))], container="ubuntu:latest", version="v0.1.0", arguments=[ToolArgument(value="ls | grep -v lsout", position=None, doc=InputDocumentation(doc=None), shell_quote=False)])
   
   
   if __name__ == "__main__":
       # or "cwl"
       Secondary_Files_V0_1_0().translate("wdl")
   

   $ janis translate secondary_files_v0_1_0.py wdl -o .
   2021-08-29T11:13:34 [INFO]: The command tool ({'tool': CommandTool<secondary_files>, 'toolid': 'secondary_files'}).outp' used a star-bind (*) glob to find the output, but the return type was not an array. For WDL, the first element will be used, ie: 'glob("lsout")[0]'
   

   secondary_files_v0_1_0.wdl

   version development
   
   task secondary_files {
     input {
       Int? runtime_cpu
       Int? runtime_memory
       Int? runtime_seconds
       Int? runtime_disks
       File inp
       File inp_tar
     }
     command <<<
       set -e
        \
         ls | grep -v lsout
     >>>
     runtime {
       cpu: select_first([runtime_cpu, 1])
       disks: "local-disk ~{select_first([runtime_disks, 20])} SSD"
       docker: "ubuntu@sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9"
       duration: select_first([runtime_seconds, 86400])
       memory: "~{select_first([runtime_memory, 4])}G"
       preemptible: 2
     }
     output {
       File outp = glob("lsout")[0]
     }

   ---

   array_secondary_files.cwl

   cwlVersion: v1.2
   class: CommandLineTool
   
   arguments:
     - {shellQuote: false, valueFrom: "ls | grep -v lsout"}
   inputs:
     input_list:
       type: File[]
       secondaryFiles:
         - ^.tar
   stdout: lsout
   outputs:
     output:
       type: File
       outputBinding:
         glob: lsout
   requirements:
     ShellCommandRequirement: {}

   $ janisdk fromcwl array_secondary_files.cwl -o .
   $ janis translate array_secondary_files_v0_1_0.py wdl -o .
   2021-08-29T11:09:55 [INFO]: The command tool ({'tool': CommandTool<array_secondary_files>, 'toolid': 'array_secondary_files'}).outp' used a star-bind (*) glob to find the output, but the return type was not an array. For WDL, the first element will be used, ie: 'glob("lsout")[0]'
   

   version development
   
   task array_secondary_files {
     input {
       Int? runtime_cpu
       Int? runtime_memory
       Int? runtime_seconds
       Int? runtime_disks
       Array[File] input_list
       Array[File] input_list_tar
     }
     command <<<
       set -e
        \
         ls | grep -v lsout
     >>>
     runtime {
       cpu: select_first([runtime_cpu, 1])
       disks: "local-disk ~{select_first([runtime_disks, 20])} SSD"
       docker: "ubuntu@sha256:1e48201ccc2ab83afc435394b3bf70af0fa0055215c1e26a5da9b50a1ae367c9"
       duration: select_first([runtime_seconds, 86400])
       memory: "~{select_first([runtime_memory, 4])}G"
       preemptible: 2
     }
     output {
       File outp = glob("lsout")[0]
     }
   }