Module Ketrew_yarn (.ml)

module Ketrew_yarn: sig .. end

Implementation of the LONG_RUNNING API asking Aapache Yarn for resources and using Ketrew_daemonize to “keep” the process group together.

This module implements Ketrew_long_running.LONG_RUNNING plugin-API.

include Ketrew_long_running.LONG_RUNNING

The “standard” plugin-API.

type distributed_shell_parameters

val distributed_shell_program : ?hadoop_bin:string ->
       ?distributed_shell_shell_jar:string ->
       container_memory:[ `GB of int | `MB of int | `Raw of string ] ->
       timeout:[ `Raw of string | `Seconds of int ] ->
       application_name:string ->
       Ketrew_program.t ->
       [> `Distributed_shell of
            distributed_shell_parameters * Ketrew_program.t ]

Create a value `Distributed_shell _ to feed to Ketrew_yarn.create, see Ketrew_edsl.yarn_distributed_shell.

val create : ?host:Ketrew_host.t ->
       ?daemonize_using:[ `Nohup_setsid | `Python_daemon ] ->
       ?daemon_start_timeout:float ->
       [ `Distributed_shell of
           distributed_shell_parameters * Ketrew_program.t
       | `Yarn_application of Ketrew_program.t ] ->
       [> `Long_running of string * string ]

Create a “long-running” Ketrew_target.build_process (run parameters are already serialized), see Ketrew_edsl.yarn_application.