Module Ketrew.Yarn

module Yarn: sig .. end

Implementation of the LONG_RUNNING API asking Aapache Yarn for resources and using Ketrew_daemonize to “keep” the process group together.

This module implements Ketrew_long_running.LONG_RUNNING plugin-API.

include Long_running.LONG_RUNNING

The “standard” plugin-API.

type distributed_shell_parameters

val distributed_shell_program : ?hadoop_bin:string ->
       ?distributed_shell_shell_jar:string ->
       ?container_vcores:int ->
       container_memory:[ `GB of int | `MB of int | `Raw of string ] ->
       timeout:[ `Raw of string | `Seconds of int ] ->
       application_name:string ->
       Ketrew_pure.Program.t ->
       [> `Distributed_shell of
            distributed_shell_parameters * Ketrew_pure.Program.t ]

Create a value `Distributed_shell _ to feed to Ketrew.Yarn.create, see Edsl.yarn_distributed_shell.

val create : ?host:Ketrew_pure.Host.t ->
       ?daemonize_using:[ `Nohup_setsid | `Python_daemon ] ->
       ?daemon_start_timeout:float ->
       [ `Distributed_shell of
           distributed_shell_parameters * Ketrew_pure.Program.t
       | `Yarn_application of Ketrew_pure.Program.t ] ->
       [> `Long_running of string * string ]

Create a “long-running” Ketrew_pure.Target.build_process (run parameters are already serialized), see Edsl.yarn_application.

Advanced Global Configuration

val max_name_length : int Pervasives.ref