# API Reference ## Constructs ### ApplicationLoadBalancedNeuronxService #### Initializers ```typescript import { ApplicationLoadBalancedNeuronxService } from 'aws-cdk-neuronx-patterns' new ApplicationLoadBalancedNeuronxService(scope: Construct, id: string, props: ApplicationLoadBalancedNeuronxServiceProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | ApplicationLoadBalancedNeuronxServiceProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* ApplicationLoadBalancedNeuronxServiceProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { ApplicationLoadBalancedNeuronxService } from 'aws-cdk-neuronx-patterns' ApplicationLoadBalancedNeuronxService.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | cluster | aws-cdk-lib.aws_ecs.ICluster | The cluster that hosts the service. | | listener | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener | The listener for the service. | | loadBalancer | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationLoadBalancer | The Application Load Balancer for the service. | | targetGroup | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationTargetGroup | The target group for the service. | | certificate | aws-cdk-lib.aws_certificatemanager.ICertificate | Certificate Manager certificate to associate with the load balancer. | | internalDesiredCount | number | The desired number of instantiations of the task definition to keep running on the service. | | redirectListener | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener | The redirect listener for the service if redirectHTTP is enabled. | | service | aws-cdk-lib.aws_ecs.Ec2Service | The EC2 service in this construct. | | taskDefinition | aws-cdk-lib.aws_ecs.Ec2TaskDefinition | The EC2 Task Definition in this construct. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `cluster`^Required ```typescript public readonly cluster: ICluster; ``` - *Type:* aws-cdk-lib.aws_ecs.ICluster The cluster that hosts the service. --- ##### `listener`^Required ```typescript public readonly listener: ApplicationListener; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener The listener for the service. --- ##### `loadBalancer`^Required ```typescript public readonly loadBalancer: ApplicationLoadBalancer; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationLoadBalancer The Application Load Balancer for the service. --- ##### `targetGroup`^Required ```typescript public readonly targetGroup: ApplicationTargetGroup; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationTargetGroup The target group for the service. --- ##### `certificate`^Optional ```typescript public readonly certificate: ICertificate; ``` - *Type:* aws-cdk-lib.aws_certificatemanager.ICertificate Certificate Manager certificate to associate with the load balancer. --- ##### `internalDesiredCount`^Optional ```typescript public readonly internalDesiredCount: number; ``` - *Type:* number The desired number of instantiations of the task definition to keep running on the service. The default is 1 for all new services and uses the existing services desired count when updating an existing service if one is not provided. --- ##### `redirectListener`^Optional ```typescript public readonly redirectListener: ApplicationListener; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener The redirect listener for the service if redirectHTTP is enabled. --- ##### `service`^Required ```typescript public readonly service: Ec2Service; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2Service The EC2 service in this construct. --- ##### `taskDefinition`^Required ```typescript public readonly taskDefinition: Ec2TaskDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinition The EC2 Task Definition in this construct. --- ### ApplicationLoadBalancedVllmNxDInferenceService ApplicationLoadBalancedVllmNxDInferenceService is a wrapper of ApplicationLoadBalancedNeuronxServiceBase. It provides a simple way to deploy vLLM on NxD Inference. *Example* ```typescript const compiler = new VllmNxdInferenceCompiler(this, "Compiler", { vpc, bucket, model: Model.fromHuggingFace("example/example-7b-chat"), }); const compiledModel = compiler.compile(); const taskDefinition = new VllmNxdInferenceTaskDefinition( this, "TaskDefinition", { vpc, compiledModel, }, ); new ApplicationLoadBalancedVllmNxDInferenceService(this, "Service", { taskDefinition, }); ``` #### Initializers ```typescript import { ApplicationLoadBalancedVllmNxDInferenceService } from 'aws-cdk-neuronx-patterns' new ApplicationLoadBalancedVllmNxDInferenceService(scope: Construct, id: string, props: ApplicationLoadBalancedVllmNxDInferenceServiceProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | ApplicationLoadBalancedVllmNxDInferenceServiceProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* ApplicationLoadBalancedVllmNxDInferenceServiceProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { ApplicationLoadBalancedVllmNxDInferenceService } from 'aws-cdk-neuronx-patterns' ApplicationLoadBalancedVllmNxDInferenceService.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | listener | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener | *No description.* | | loadBalancer | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationLoadBalancer | *No description.* | | service | aws-cdk-lib.aws_ecs.Ec2Service | *No description.* | | targetGroup | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationTargetGroup | *No description.* | | taskDefinition | aws-cdk-lib.aws_ecs.Ec2TaskDefinition | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `listener`^Required ```typescript public readonly listener: ApplicationListener; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationListener --- ##### `loadBalancer`^Required ```typescript public readonly loadBalancer: ApplicationLoadBalancer; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationLoadBalancer --- ##### `service`^Required ```typescript public readonly service: Ec2Service; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2Service --- ##### `targetGroup`^Required ```typescript public readonly targetGroup: ApplicationTargetGroup; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationTargetGroup --- ##### `taskDefinition`^Required ```typescript public readonly taskDefinition: Ec2TaskDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinition --- ### NeuronOptimizedMachineImage - *Implements:* aws-cdk-lib.aws_ec2.IMachineImage #### Initializers ```typescript import { NeuronOptimizedMachineImage } from 'aws-cdk-neuronx-patterns' new NeuronOptimizedMachineImage(scope: Construct, id: string) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | getImage | Return the image to use in the given context. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `getImage` ```typescript public getImage(_scope: Construct): MachineImageConfig ``` Return the image to use in the given context. ###### `_scope`^Required - *Type:* constructs.Construct --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronOptimizedMachineImage } from 'aws-cdk-neuronx-patterns' NeuronOptimizedMachineImage.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | imageId | string | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `imageId`^Required ```typescript public readonly imageId: string; ``` - *Type:* string --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | size | aws-cdk-lib.Size | *No description.* | --- ##### `size`^Required ```typescript public readonly size: Size; ``` - *Type:* aws-cdk-lib.Size --- ### NeuronxBatch Neuronx batch construct. #### Initializers ```typescript import { NeuronxBatch } from 'aws-cdk-neuronx-patterns' new NeuronxBatch(scope: Construct, id: string, props: NeuronxBatchProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxBatchProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxBatchProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxBatch } from 'aws-cdk-neuronx-patterns' NeuronxBatch.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | computeEnvironment | aws-cdk-lib.aws_batch.ManagedEc2EcsComputeEnvironment | *No description.* | | jobDefinition | aws-cdk-lib.aws_batch.EcsJobDefinition | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `computeEnvironment`^Required ```typescript public readonly computeEnvironment: ManagedEc2EcsComputeEnvironment; ``` - *Type:* aws-cdk-lib.aws_batch.ManagedEc2EcsComputeEnvironment --- ##### `jobDefinition`^Required ```typescript public readonly jobDefinition: EcsJobDefinition; ``` - *Type:* aws-cdk-lib.aws_batch.EcsJobDefinition --- ### NeuronxBatchComputeEnvironment - *Implements:* aws-cdk-lib.aws_batch.IComputeEnvironment Neuronx batch construct. #### Initializers ```typescript import { NeuronxBatchComputeEnvironment } from 'aws-cdk-neuronx-patterns' new NeuronxBatchComputeEnvironment(scope: Construct, id: string, props: NeuronxBatchComputeEnvironmentProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxBatchComputeEnvironmentProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxBatchComputeEnvironmentProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | applyRemovalPolicy | Apply the given removal policy to this resource. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxBatchComputeEnvironment } from 'aws-cdk-neuronx-patterns' NeuronxBatchComputeEnvironment.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | computeEnvironmentArn | string | The ARN of this compute environment. | | computeEnvironmentName | string | The name of the ComputeEnvironment. | | computeEnvironmentRef | aws-cdk-lib.interfaces.aws_batch.ComputeEnvironmentReference | A reference to a ComputeEnvironment resource. | | enabled | boolean | Whether or not this ComputeEnvironment can accept jobs from a Queue. | | env | aws-cdk-lib.interfaces.ResourceEnvironment | The environment this resource belongs to. | | instanceRole | aws-cdk-lib.aws_iam.IRole | *No description.* | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | serviceRole | aws-cdk-lib.aws_iam.IRole | The role Batch uses to perform actions on your behalf in your account, such as provision instances to run your jobs. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `computeEnvironmentArn`^Required ```typescript public readonly computeEnvironmentArn: string; ``` - *Type:* string The ARN of this compute environment. --- ##### `computeEnvironmentName`^Required ```typescript public readonly computeEnvironmentName: string; ``` - *Type:* string The name of the ComputeEnvironment. --- ##### `computeEnvironmentRef`^Required ```typescript public readonly computeEnvironmentRef: ComputeEnvironmentReference; ``` - *Type:* aws-cdk-lib.interfaces.aws_batch.ComputeEnvironmentReference A reference to a ComputeEnvironment resource. --- ##### `enabled`^Required ```typescript public readonly enabled: boolean; ``` - *Type:* boolean Whether or not this ComputeEnvironment can accept jobs from a Queue. Enabled ComputeEnvironments can accept jobs from a Queue and can scale instances up or down. Disabled ComputeEnvironments cannot accept jobs from a Queue or scale instances up or down. If you change a ComputeEnvironment from enabled to disabled while it is executing jobs, Jobs in the `STARTED` or `RUNNING` states will not be interrupted. As jobs complete, the ComputeEnvironment will scale instances down to `minvCpus`. To ensure you aren't billed for unused capacity, set `minvCpus` to `0`. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.interfaces.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed in a Stack (those created by creating new class instances like `new Role()`, `new Bucket()`, etc.), this is always the same as the environment of the stack they belong to. For referenced resources (those obtained from referencing methods like `Role.fromRoleArn()`, `Bucket.fromBucketName()`, etc.), they might be different than the stack they were imported into. --- ##### `instanceRole`^Required ```typescript public readonly instanceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `serviceRole`^Optional ```typescript public readonly serviceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The role Batch uses to perform actions on your behalf in your account, such as provision instances to run your jobs. --- ### NeuronxBatchEcsJobDefinition - *Implements:* aws-cdk-lib.aws_batch.IJobDefinition Neuronx batch construct. #### Initializers ```typescript import { NeuronxBatchEcsJobDefinition } from 'aws-cdk-neuronx-patterns' new NeuronxBatchEcsJobDefinition(scope: Construct, id: string, props: NeuronxBatchEcsJobDefinitionProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxBatchEcsJobDefinitionProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxBatchEcsJobDefinitionProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | addRetryStrategy | Add a RetryStrategy to this JobDefinition. | | applyRemovalPolicy | Apply the given removal policy to this resource. | | grantSubmitJob | Grants the `batch:submitJob` permission to the identity on both this job definition and the `queue`. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `addRetryStrategy` ```typescript public addRetryStrategy(strategy: RetryStrategy): void ``` Add a RetryStrategy to this JobDefinition. ###### `strategy`^Required - *Type:* aws-cdk-lib.aws_batch.RetryStrategy --- ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- ##### `grantSubmitJob` ```typescript public grantSubmitJob(identity: IGrantable, queue: IJobQueue): void ``` Grants the `batch:submitJob` permission to the identity on both this job definition and the `queue`. ###### `identity`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable --- ###### `queue`^Required - *Type:* aws-cdk-lib.aws_batch.IJobQueue --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxBatchEcsJobDefinition } from 'aws-cdk-neuronx-patterns' NeuronxBatchEcsJobDefinition.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.interfaces.ResourceEnvironment | The environment this resource belongs to. | | jobDefinitionArn | string | The ARN of this job definition. | | jobDefinitionName | string | The name of this job definition. | | jobDefinitionRef | aws-cdk-lib.interfaces.aws_batch.JobDefinitionReference | A reference to a JobDefinition resource. | | retryStrategies | aws-cdk-lib.aws_batch.RetryStrategy[] | Defines the retry behavior for this job. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | parameters | {[ key: string ]: any} | The default parameters passed to the container These parameters can be referenced in the `command` that you give to the container. | | retryAttempts | number | The number of times to retry a job. | | schedulingPriority | number | The priority of this Job. | | timeout | aws-cdk-lib.Duration | The timeout time for jobs that are submitted with this job definition. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.interfaces.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed in a Stack (those created by creating new class instances like `new Role()`, `new Bucket()`, etc.), this is always the same as the environment of the stack they belong to. For referenced resources (those obtained from referencing methods like `Role.fromRoleArn()`, `Bucket.fromBucketName()`, etc.), they might be different than the stack they were imported into. --- ##### `jobDefinitionArn`^Required ```typescript public readonly jobDefinitionArn: string; ``` - *Type:* string The ARN of this job definition. --- ##### `jobDefinitionName`^Required ```typescript public readonly jobDefinitionName: string; ``` - *Type:* string The name of this job definition. --- ##### `jobDefinitionRef`^Required ```typescript public readonly jobDefinitionRef: JobDefinitionReference; ``` - *Type:* aws-cdk-lib.interfaces.aws_batch.JobDefinitionReference A reference to a JobDefinition resource. --- ##### `retryStrategies`^Required ```typescript public readonly retryStrategies: RetryStrategy[]; ``` - *Type:* aws-cdk-lib.aws_batch.RetryStrategy[] Defines the retry behavior for this job. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `parameters`^Optional ```typescript public readonly parameters: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} The default parameters passed to the container These parameters can be referenced in the `command` that you give to the container. --- ##### `retryAttempts`^Optional ```typescript public readonly retryAttempts: number; ``` - *Type:* number The number of times to retry a job. The job is retried on failure the same number of attempts as the value. --- ##### `schedulingPriority`^Optional ```typescript public readonly schedulingPriority: number; ``` - *Type:* number The priority of this Job. Only used in Fairshare Scheduling to decide which job to run first when there are multiple jobs with the same share identifier. --- ##### `timeout`^Optional ```typescript public readonly timeout: Duration; ``` - *Type:* aws-cdk-lib.Duration The timeout time for jobs that are submitted with this job definition. After the amount of time you specify passes, Batch terminates your jobs if they aren't finished. --- ### NeuronxCompilerBase - *Implements:* INeuronxCompiler Abstract base class for Neuronx compilers. Provides the common orchestration logic (Lambda, CustomResource, WaitCondition) while subclasses define how to create the Batch compute environment and job definition. #### Initializers ```typescript import { NeuronxCompilerBase } from 'aws-cdk-neuronx-patterns' new NeuronxCompilerBase(scope: Construct, id: string, props: NeuronxCompilerBaseProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxCompilerBaseProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxCompilerBaseProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | compile | *No description.* | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `compile` ```typescript public compile(): NeuronxCompiledModel ``` #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxCompilerBase } from 'aws-cdk-neuronx-patterns' NeuronxCompilerBase.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ### NeuronxCrossCompiler Neuronx cross-compiler construct. Compile the model on a non-Neuron instance and upload the artifacts to an S3 bucket. This avoids the need for expensive Neuron instances during the compilation phase. The compilation uses `vllm serve` which performs model tracing and neuronx-cc compilation entirely on CPU. The resulting artifacts are compatible with Neuron instances for inference. #### Initializers ```typescript import { NeuronxCrossCompiler } from 'aws-cdk-neuronx-patterns' new NeuronxCrossCompiler(scope: Construct, id: string, props: NeuronxCrossCompilerProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxCrossCompilerProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxCrossCompilerProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | compile | *No description.* | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `compile` ```typescript public compile(): NeuronxCompiledModel ``` #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxCrossCompiler } from 'aws-cdk-neuronx-patterns' NeuronxCrossCompiler.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ### NeuronxNativeCompiler Neuronx compiler construct. Compile the model to work with Inferentia2 and Trainium1 and upload it to an S3 bucket. #### Initializers ```typescript import { NeuronxNativeCompiler } from 'aws-cdk-neuronx-patterns' new NeuronxNativeCompiler(scope: Construct, id: string, props: NeuronxNativeCompilerProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxNativeCompilerProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxNativeCompilerProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | compile | *No description.* | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `compile` ```typescript public compile(): NeuronxCompiledModel ``` #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { NeuronxNativeCompiler } from 'aws-cdk-neuronx-patterns' NeuronxNativeCompiler.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ### NeuronxTaskDefinition - *Implements:* INeuronxTaskDefinition #### Initializers ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' new NeuronxTaskDefinition(scope: Construct, id: string, props: NeuronxTaskDefinitionProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | NeuronxTaskDefinitionProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* NeuronxTaskDefinitionProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | applyRemovalPolicy | Apply the given removal policy to this resource. | | addContainer | Tasks running in AWSVPC networking mode requires an additional environment variable for the region to be sourced. | | addExtension | Adds the specified extension to the task definition. | | addFirelensLogRouter | Adds a firelens log router to the task definition. | | addInferenceAccelerator | Adds an inference accelerator to the task definition. | | addPlacementConstraint | Adds the specified placement constraint to the task definition. | | addToExecutionRolePolicy | Adds a policy statement to the task execution IAM role. | | addToTaskRolePolicy | Adds a policy statement to the task IAM role. | | addVolume | Adds a volume to the task definition. | | findContainer | Returns the container that match the provided containerName. | | findPortMappingByName | Determine the existing port mapping for the provided name. | | grantRun | Grants permissions to run this task definition. | | obtainExecutionRole | Creates the task execution IAM role if it doesn't already exist. | | addContainerWithDefault | *No description.* | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- ##### `addContainer` ```typescript public addContainer(id: string, props: ContainerDefinitionOptions): ContainerDefinition ``` Tasks running in AWSVPC networking mode requires an additional environment variable for the region to be sourced. This override adds in the additional environment variable as required ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinitionOptions --- ##### `addExtension` ```typescript public addExtension(extension: ITaskDefinitionExtension): void ``` Adds the specified extension to the task definition. Extension can be used to apply a packaged modification to a task definition. ###### `extension`^Required - *Type:* aws-cdk-lib.aws_ecs.ITaskDefinitionExtension --- ##### `addFirelensLogRouter` ```typescript public addFirelensLogRouter(id: string, props: FirelensLogRouterDefinitionOptions): FirelensLogRouter ``` Adds a firelens log router to the task definition. ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.FirelensLogRouterDefinitionOptions --- ##### ~~`addInferenceAccelerator`~~ ```typescript public addInferenceAccelerator(inferenceAccelerator: InferenceAccelerator): void ``` Adds an inference accelerator to the task definition. ###### `inferenceAccelerator`^Required - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator --- ##### `addPlacementConstraint` ```typescript public addPlacementConstraint(constraint: PlacementConstraint): void ``` Adds the specified placement constraint to the task definition. ###### `constraint`^Required - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint --- ##### `addToExecutionRolePolicy` ```typescript public addToExecutionRolePolicy(statement: PolicyStatement): void ``` Adds a policy statement to the task execution IAM role. ###### `statement`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement --- ##### `addToTaskRolePolicy` ```typescript public addToTaskRolePolicy(statement: PolicyStatement): void ``` Adds a policy statement to the task IAM role. ###### `statement`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement --- ##### `addVolume` ```typescript public addVolume(volume: Volume): void ``` Adds a volume to the task definition. ###### `volume`^Required - *Type:* aws-cdk-lib.aws_ecs.Volume --- ##### `findContainer` ```typescript public findContainer(containerName: string): ContainerDefinition ``` Returns the container that match the provided containerName. ###### `containerName`^Required - *Type:* string --- ##### `findPortMappingByName` ```typescript public findPortMappingByName(name: string): PortMapping ``` Determine the existing port mapping for the provided name. ###### `name`^Required - *Type:* string : port mapping name. --- ##### `grantRun` ```typescript public grantRun(grantee: IGrantable): Grant ``` Grants permissions to run this task definition. This will grant the following permissions: - ecs:RunTask - iam:PassRole [disable-awslint:no-grants] ###### `grantee`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable Principal to grant consume rights to. --- ##### `obtainExecutionRole` ```typescript public obtainExecutionRole(): IRole ``` Creates the task execution IAM role if it doesn't already exist. ##### `addContainerWithDefault` ```typescript public addContainerWithDefault(id: string, props: ContainerDefinitionOptions): ContainerDefinition ``` ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinitionOptions --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | isOwnedResource | Returns true if the construct was created by CDK, and false otherwise. | | isResource | Check whether the given construct is a Resource. | | fromTaskDefinitionArn | Imports a task definition from the specified task definition ARN. | | fromTaskDefinitionAttributes | Create a task definition from a task definition reference. | | fromEc2TaskDefinitionArn | Imports a task definition from the specified task definition ARN. | | fromEc2TaskDefinitionAttributes | Imports an existing Ec2 task definition from its attributes. | --- ##### `isConstruct` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `isOwnedResource` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.isOwnedResource(construct: IConstruct) ``` Returns true if the construct was created by CDK, and false otherwise. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `isResource` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.isResource(construct: IConstruct) ``` Check whether the given construct is a Resource. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `fromTaskDefinitionArn` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.fromTaskDefinitionArn(scope: Construct, id: string, taskDefinitionArn: string) ``` Imports a task definition from the specified task definition ARN. The task will have a compatibility of EC2+Fargate. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `taskDefinitionArn`^Required - *Type:* string --- ##### `fromTaskDefinitionAttributes` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.fromTaskDefinitionAttributes(scope: Construct, id: string, attrs: TaskDefinitionAttributes) ``` Create a task definition from a task definition reference. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_ecs.TaskDefinitionAttributes --- ##### `fromEc2TaskDefinitionArn` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.fromEc2TaskDefinitionArn(scope: Construct, id: string, ec2TaskDefinitionArn: string) ``` Imports a task definition from the specified task definition ARN. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `ec2TaskDefinitionArn`^Required - *Type:* string --- ##### `fromEc2TaskDefinitionAttributes` ```typescript import { NeuronxTaskDefinition } from 'aws-cdk-neuronx-patterns' NeuronxTaskDefinition.fromEc2TaskDefinitionAttributes(scope: Construct, id: string, attrs: Ec2TaskDefinitionAttributes) ``` Imports an existing Ec2 task definition from its attributes. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinitionAttributes --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.interfaces.ResourceEnvironment | The environment this resource belongs to. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | compatibility | aws-cdk-lib.aws_ecs.Compatibility | The task launch type compatibility requirement. | | family | string | The name of a family that this task definition is registered to. | | inferenceAccelerators | aws-cdk-lib.aws_ecs.InferenceAccelerator[] | Public getter method to access list of inference accelerators attached to the instance. | | isEc2Compatible | boolean | Return true if the task definition can be run on an EC2 cluster. | | isExternalCompatible | boolean | Return true if the task definition can be run on a ECS anywhere cluster. | | isFargateCompatible | boolean | Return true if the task definition can be run on a Fargate cluster. | | isManagedInstancesCompatible | boolean | Return true if the task definition can be run on Managed Instances. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The networking mode to use for the containers in the task. | | taskDefinitionArn | string | The full Amazon Resource Name (ARN) of the task definition. | | taskDefinitionRef | aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference | A reference to this task definition. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | ephemeralStorageGiB | number | The amount (in GiB) of ephemeral storage to be allocated to the task. | | executionRole | aws-cdk-lib.aws_iam.IRole | Execution role for this task definition. | | pidMode | aws-cdk-lib.aws_ecs.PidMode | The process namespace to use for the containers in the task. | | referencesSecretJsonField | boolean | Whether this task definition has at least a container that references a specific JSON field of a secret stored in Secrets Manager. | | defaultContainer | aws-cdk-lib.aws_ecs.ContainerDefinition | Default container for this task. | | compiledModel | NeuronxCompiledModel | *No description.* | | neuronxInstanceType | INeuronxInstanceType | *No description.* | | tasksPerInstance | number | *No description.* | | tensorParallelSize | number | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.interfaces.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed in a Stack (those created by creating new class instances like `new Role()`, `new Bucket()`, etc.), this is always the same as the environment of the stack they belong to. For referenced resources (those obtained from referencing methods like `Role.fromRoleArn()`, `Bucket.fromBucketName()`, etc.), they might be different than the stack they were imported into. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `compatibility`^Required ```typescript public readonly compatibility: Compatibility; ``` - *Type:* aws-cdk-lib.aws_ecs.Compatibility The task launch type compatibility requirement. --- ##### `family`^Required ```typescript public readonly family: string; ``` - *Type:* string The name of a family that this task definition is registered to. A family groups multiple versions of a task definition. --- ##### `inferenceAccelerators`^Required ```typescript public readonly inferenceAccelerators: InferenceAccelerator[]; ``` - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator[] Public getter method to access list of inference accelerators attached to the instance. --- ##### `isEc2Compatible`^Required ```typescript public readonly isEc2Compatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on an EC2 cluster. --- ##### `isExternalCompatible`^Required ```typescript public readonly isExternalCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a ECS anywhere cluster. --- ##### `isFargateCompatible`^Required ```typescript public readonly isFargateCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a Fargate cluster. --- ##### `isManagedInstancesCompatible`^Required ```typescript public readonly isManagedInstancesCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on Managed Instances. --- ##### `networkMode`^Required ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode The networking mode to use for the containers in the task. --- ##### `taskDefinitionArn`^Required ```typescript public readonly taskDefinitionArn: string; ``` - *Type:* string The full Amazon Resource Name (ARN) of the task definition. --- ##### `taskDefinitionRef`^Required ```typescript public readonly taskDefinitionRef: TaskDefinitionReference; ``` - *Type:* aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference A reference to this task definition. --- ##### `taskRole`^Required ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `ephemeralStorageGiB`^Optional ```typescript public readonly ephemeralStorageGiB: number; ``` - *Type:* number The amount (in GiB) of ephemeral storage to be allocated to the task. Only supported in Fargate platform version 1.4.0 or later. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole Execution role for this task definition. --- ##### `pidMode`^Optional ```typescript public readonly pidMode: PidMode; ``` - *Type:* aws-cdk-lib.aws_ecs.PidMode The process namespace to use for the containers in the task. Only supported for tasks that are hosted on AWS Fargate if the tasks are using platform version 1.4.0 or later (Linux). Not supported in Windows containers. If pidMode is specified for a Fargate task, then runtimePlatform.operatingSystemFamily must also be specified. For more information, see [Task Definition Parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_definition_pidmode). --- ##### `referencesSecretJsonField`^Optional ```typescript public readonly referencesSecretJsonField: boolean; ``` - *Type:* boolean Whether this task definition has at least a container that references a specific JSON field of a secret stored in Secrets Manager. --- ##### `defaultContainer`^Optional ```typescript public readonly defaultContainer: ContainerDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinition Default container for this task. Load balancers will send traffic to this container. The first essential container that is added to this task will become the default container. --- ##### `compiledModel`^Required ```typescript public readonly compiledModel: NeuronxCompiledModel; ``` - *Type:* NeuronxCompiledModel --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType --- ##### `tasksPerInstance`^Required ```typescript public readonly tasksPerInstance: number; ``` - *Type:* number --- ##### `tensorParallelSize`^Required ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | PROPERTY_INJECTION_ID | string | Uniquely identifies this class. | --- ##### `PROPERTY_INJECTION_ID`^Required ```typescript public readonly PROPERTY_INJECTION_ID: string; ``` - *Type:* string Uniquely identifies this class. --- ### VllmNxdInferenceCompiler Neuronx compiler construct for vLLM on NxD Inference. Compile the model to work with Neuronx instance and upload it to an S3 bucket. #### Initializers ```typescript import { VllmNxdInferenceCompiler } from 'aws-cdk-neuronx-patterns' new VllmNxdInferenceCompiler(scope: Construct, id: string, props: VllmNxdInferenceCompileProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | VllmNxdInferenceCompileProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* VllmNxdInferenceCompileProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | compile | Compile the model and return the compiled model. | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `compile` ```typescript public compile(): VllmNxdInferenceCompiledModel ``` Compile the model and return the compiled model. #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | --- ##### `isConstruct` ```typescript import { VllmNxdInferenceCompiler } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceCompiler.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ### VllmNxdInferenceTaskDefinition Task definition for VllmNxdInference. #### Initializers ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' new VllmNxdInferenceTaskDefinition(scope: Construct, id: string, props: VllmNxdInferenceTaskDefinitionProps) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | props | VllmNxdInferenceTaskDefinitionProps | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `props`^Required - *Type:* VllmNxdInferenceTaskDefinitionProps --- #### Methods | **Name** | **Description** | | --- | --- | | toString | Returns a string representation of this construct. | | with | Applies one or more mixins to this construct. | | applyRemovalPolicy | Apply the given removal policy to this resource. | | addContainer | Tasks running in AWSVPC networking mode requires an additional environment variable for the region to be sourced. | | addExtension | Adds the specified extension to the task definition. | | addFirelensLogRouter | Adds a firelens log router to the task definition. | | addInferenceAccelerator | Adds an inference accelerator to the task definition. | | addPlacementConstraint | Adds the specified placement constraint to the task definition. | | addToExecutionRolePolicy | Adds a policy statement to the task execution IAM role. | | addToTaskRolePolicy | Adds a policy statement to the task IAM role. | | addVolume | Adds a volume to the task definition. | | findContainer | Returns the container that match the provided containerName. | | findPortMappingByName | Determine the existing port mapping for the provided name. | | grantRun | Grants permissions to run this task definition. | | obtainExecutionRole | Creates the task execution IAM role if it doesn't already exist. | | addContainerWithDefault | *No description.* | --- ##### `toString` ```typescript public toString(): string ``` Returns a string representation of this construct. ##### `with` ```typescript public with(mixins: ...IMixin[]): IConstruct ``` Applies one or more mixins to this construct. Mixins are applied in order. The list of constructs is captured at the start of the call, so constructs added by a mixin will not be visited. Use multiple `with()` calls if subsequent mixins should apply to added constructs. ###### `mixins`^Required - *Type:* ...constructs.IMixin[] The mixins to apply. --- ##### `applyRemovalPolicy` ```typescript public applyRemovalPolicy(policy: RemovalPolicy): void ``` Apply the given removal policy to this resource. The Removal Policy controls what happens to this resource when it stops being managed by CloudFormation, either because you've removed it from the CDK application or because you've made a change that requires the resource to be replaced. The resource can be deleted (`RemovalPolicy.DESTROY`), or left in your AWS account for data recovery and cleanup later (`RemovalPolicy.RETAIN`). ###### `policy`^Required - *Type:* aws-cdk-lib.RemovalPolicy --- ##### `addContainer` ```typescript public addContainer(id: string, props: ContainerDefinitionOptions): ContainerDefinition ``` Tasks running in AWSVPC networking mode requires an additional environment variable for the region to be sourced. This override adds in the additional environment variable as required ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinitionOptions --- ##### `addExtension` ```typescript public addExtension(extension: ITaskDefinitionExtension): void ``` Adds the specified extension to the task definition. Extension can be used to apply a packaged modification to a task definition. ###### `extension`^Required - *Type:* aws-cdk-lib.aws_ecs.ITaskDefinitionExtension --- ##### `addFirelensLogRouter` ```typescript public addFirelensLogRouter(id: string, props: FirelensLogRouterDefinitionOptions): FirelensLogRouter ``` Adds a firelens log router to the task definition. ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.FirelensLogRouterDefinitionOptions --- ##### ~~`addInferenceAccelerator`~~ ```typescript public addInferenceAccelerator(inferenceAccelerator: InferenceAccelerator): void ``` Adds an inference accelerator to the task definition. ###### `inferenceAccelerator`^Required - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator --- ##### `addPlacementConstraint` ```typescript public addPlacementConstraint(constraint: PlacementConstraint): void ``` Adds the specified placement constraint to the task definition. ###### `constraint`^Required - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint --- ##### `addToExecutionRolePolicy` ```typescript public addToExecutionRolePolicy(statement: PolicyStatement): void ``` Adds a policy statement to the task execution IAM role. ###### `statement`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement --- ##### `addToTaskRolePolicy` ```typescript public addToTaskRolePolicy(statement: PolicyStatement): void ``` Adds a policy statement to the task IAM role. ###### `statement`^Required - *Type:* aws-cdk-lib.aws_iam.PolicyStatement --- ##### `addVolume` ```typescript public addVolume(volume: Volume): void ``` Adds a volume to the task definition. ###### `volume`^Required - *Type:* aws-cdk-lib.aws_ecs.Volume --- ##### `findContainer` ```typescript public findContainer(containerName: string): ContainerDefinition ``` Returns the container that match the provided containerName. ###### `containerName`^Required - *Type:* string --- ##### `findPortMappingByName` ```typescript public findPortMappingByName(name: string): PortMapping ``` Determine the existing port mapping for the provided name. ###### `name`^Required - *Type:* string : port mapping name. --- ##### `grantRun` ```typescript public grantRun(grantee: IGrantable): Grant ``` Grants permissions to run this task definition. This will grant the following permissions: - ecs:RunTask - iam:PassRole [disable-awslint:no-grants] ###### `grantee`^Required - *Type:* aws-cdk-lib.aws_iam.IGrantable Principal to grant consume rights to. --- ##### `obtainExecutionRole` ```typescript public obtainExecutionRole(): IRole ``` Creates the task execution IAM role if it doesn't already exist. ##### `addContainerWithDefault` ```typescript public addContainerWithDefault(id: string, props: ContainerDefinitionOptions): ContainerDefinition ``` ###### `id`^Required - *Type:* string --- ###### `props`^Required - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinitionOptions --- #### Static Functions | **Name** | **Description** | | --- | --- | | isConstruct | Checks if `x` is a construct. | | isOwnedResource | Returns true if the construct was created by CDK, and false otherwise. | | isResource | Check whether the given construct is a Resource. | | fromTaskDefinitionArn | Imports a task definition from the specified task definition ARN. | | fromTaskDefinitionAttributes | Create a task definition from a task definition reference. | | fromEc2TaskDefinitionArn | Imports a task definition from the specified task definition ARN. | | fromEc2TaskDefinitionAttributes | Imports an existing Ec2 task definition from its attributes. | --- ##### `isConstruct` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.isConstruct(x: any) ``` Checks if `x` is a construct. Use this method instead of `instanceof` to properly detect `Construct` instances, even when the construct library is symlinked. Explanation: in JavaScript, multiple copies of the `constructs` library on disk are seen as independent, completely different libraries. As a consequence, the class `Construct` in each copy of the `constructs` library is seen as a different class, and an instance of one class will not test as `instanceof` the other class. `npm install` will not create installations like this, but users may manually symlink construct libraries together or use a monorepo tool: in those cases, multiple copies of the `constructs` library can be accidentally installed, and `instanceof` will behave unpredictably. It is safest to avoid using `instanceof`, and using this type-testing method instead. ###### `x`^Required - *Type:* any Any object. --- ##### `isOwnedResource` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.isOwnedResource(construct: IConstruct) ``` Returns true if the construct was created by CDK, and false otherwise. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `isResource` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.isResource(construct: IConstruct) ``` Check whether the given construct is a Resource. ###### `construct`^Required - *Type:* constructs.IConstruct --- ##### `fromTaskDefinitionArn` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.fromTaskDefinitionArn(scope: Construct, id: string, taskDefinitionArn: string) ``` Imports a task definition from the specified task definition ARN. The task will have a compatibility of EC2+Fargate. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `taskDefinitionArn`^Required - *Type:* string --- ##### `fromTaskDefinitionAttributes` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.fromTaskDefinitionAttributes(scope: Construct, id: string, attrs: TaskDefinitionAttributes) ``` Create a task definition from a task definition reference. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_ecs.TaskDefinitionAttributes --- ##### `fromEc2TaskDefinitionArn` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.fromEc2TaskDefinitionArn(scope: Construct, id: string, ec2TaskDefinitionArn: string) ``` Imports a task definition from the specified task definition ARN. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `ec2TaskDefinitionArn`^Required - *Type:* string --- ##### `fromEc2TaskDefinitionAttributes` ```typescript import { VllmNxdInferenceTaskDefinition } from 'aws-cdk-neuronx-patterns' VllmNxdInferenceTaskDefinition.fromEc2TaskDefinitionAttributes(scope: Construct, id: string, attrs: Ec2TaskDefinitionAttributes) ``` Imports an existing Ec2 task definition from its attributes. ###### `scope`^Required - *Type:* constructs.Construct --- ###### `id`^Required - *Type:* string --- ###### `attrs`^Required - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinitionAttributes --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.interfaces.ResourceEnvironment | The environment this resource belongs to. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | compatibility | aws-cdk-lib.aws_ecs.Compatibility | The task launch type compatibility requirement. | | family | string | The name of a family that this task definition is registered to. | | inferenceAccelerators | aws-cdk-lib.aws_ecs.InferenceAccelerator[] | Public getter method to access list of inference accelerators attached to the instance. | | isEc2Compatible | boolean | Return true if the task definition can be run on an EC2 cluster. | | isExternalCompatible | boolean | Return true if the task definition can be run on a ECS anywhere cluster. | | isFargateCompatible | boolean | Return true if the task definition can be run on a Fargate cluster. | | isManagedInstancesCompatible | boolean | Return true if the task definition can be run on Managed Instances. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The networking mode to use for the containers in the task. | | taskDefinitionArn | string | The full Amazon Resource Name (ARN) of the task definition. | | taskDefinitionRef | aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference | A reference to this task definition. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | ephemeralStorageGiB | number | The amount (in GiB) of ephemeral storage to be allocated to the task. | | executionRole | aws-cdk-lib.aws_iam.IRole | Execution role for this task definition. | | pidMode | aws-cdk-lib.aws_ecs.PidMode | The process namespace to use for the containers in the task. | | referencesSecretJsonField | boolean | Whether this task definition has at least a container that references a specific JSON field of a secret stored in Secrets Manager. | | defaultContainer | aws-cdk-lib.aws_ecs.ContainerDefinition | Default container for this task. | | compiledModel | NeuronxCompiledModel | *No description.* | | neuronxInstanceType | INeuronxInstanceType | *No description.* | | tasksPerInstance | number | *No description.* | | tensorParallelSize | number | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.interfaces.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed in a Stack (those created by creating new class instances like `new Role()`, `new Bucket()`, etc.), this is always the same as the environment of the stack they belong to. For referenced resources (those obtained from referencing methods like `Role.fromRoleArn()`, `Bucket.fromBucketName()`, etc.), they might be different than the stack they were imported into. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `compatibility`^Required ```typescript public readonly compatibility: Compatibility; ``` - *Type:* aws-cdk-lib.aws_ecs.Compatibility The task launch type compatibility requirement. --- ##### `family`^Required ```typescript public readonly family: string; ``` - *Type:* string The name of a family that this task definition is registered to. A family groups multiple versions of a task definition. --- ##### `inferenceAccelerators`^Required ```typescript public readonly inferenceAccelerators: InferenceAccelerator[]; ``` - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator[] Public getter method to access list of inference accelerators attached to the instance. --- ##### `isEc2Compatible`^Required ```typescript public readonly isEc2Compatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on an EC2 cluster. --- ##### `isExternalCompatible`^Required ```typescript public readonly isExternalCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a ECS anywhere cluster. --- ##### `isFargateCompatible`^Required ```typescript public readonly isFargateCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a Fargate cluster. --- ##### `isManagedInstancesCompatible`^Required ```typescript public readonly isManagedInstancesCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on Managed Instances. --- ##### `networkMode`^Required ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode The networking mode to use for the containers in the task. --- ##### `taskDefinitionArn`^Required ```typescript public readonly taskDefinitionArn: string; ``` - *Type:* string The full Amazon Resource Name (ARN) of the task definition. --- ##### `taskDefinitionRef`^Required ```typescript public readonly taskDefinitionRef: TaskDefinitionReference; ``` - *Type:* aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference A reference to this task definition. --- ##### `taskRole`^Required ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `ephemeralStorageGiB`^Optional ```typescript public readonly ephemeralStorageGiB: number; ``` - *Type:* number The amount (in GiB) of ephemeral storage to be allocated to the task. Only supported in Fargate platform version 1.4.0 or later. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole Execution role for this task definition. --- ##### `pidMode`^Optional ```typescript public readonly pidMode: PidMode; ``` - *Type:* aws-cdk-lib.aws_ecs.PidMode The process namespace to use for the containers in the task. Only supported for tasks that are hosted on AWS Fargate if the tasks are using platform version 1.4.0 or later (Linux). Not supported in Windows containers. If pidMode is specified for a Fargate task, then runtimePlatform.operatingSystemFamily must also be specified. For more information, see [Task Definition Parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#task_definition_pidmode). --- ##### `referencesSecretJsonField`^Optional ```typescript public readonly referencesSecretJsonField: boolean; ``` - *Type:* boolean Whether this task definition has at least a container that references a specific JSON field of a secret stored in Secrets Manager. --- ##### `defaultContainer`^Optional ```typescript public readonly defaultContainer: ContainerDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerDefinition Default container for this task. Load balancers will send traffic to this container. The first essential container that is added to this task will become the default container. --- ##### `compiledModel`^Required ```typescript public readonly compiledModel: NeuronxCompiledModel; ``` - *Type:* NeuronxCompiledModel --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType --- ##### `tasksPerInstance`^Required ```typescript public readonly tasksPerInstance: number; ``` - *Type:* number --- ##### `tensorParallelSize`^Required ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | PROPERTY_INJECTION_ID | string | Uniquely identifies this class. | --- ##### `PROPERTY_INJECTION_ID`^Required ```typescript public readonly PROPERTY_INJECTION_ID: string; ``` - *Type:* string Uniquely identifies this class. --- ## Structs ### ApplicationLoadBalancedNeuronxServiceProps #### Initializer ```typescript import { ApplicationLoadBalancedNeuronxServiceProps } from 'aws-cdk-neuronx-patterns' const applicationLoadBalancedNeuronxServiceProps: ApplicationLoadBalancedNeuronxServiceProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | capacityProviderStrategies | aws-cdk-lib.aws_ecs.CapacityProviderStrategy[] | A list of Capacity Provider strategies used to place a service. | | certificate | aws-cdk-lib.aws_certificatemanager.ICertificate | Certificate Manager certificate to associate with the load balancer. | | circuitBreaker | aws-cdk-lib.aws_ecs.DeploymentCircuitBreaker | Whether to enable the deployment circuit breaker. | | cloudMapOptions | aws-cdk-lib.aws_ecs.CloudMapOptions | The options for configuring an Amazon ECS service to use service discovery. | | cluster | aws-cdk-lib.aws_ecs.ICluster | The name of the cluster that hosts the service. | | deploymentController | aws-cdk-lib.aws_ecs.DeploymentController | Specifies which deployment controller to use for the service. | | desiredCount | number | The desired number of instantiations of the task definition to keep running on the service. | | domainName | string | The domain name for the service, e.g. "api.example.com.". | | domainZone | aws-cdk-lib.aws_route53.IHostedZone | The Route53 hosted zone for the domain, e.g. "example.com.". | | enableECSManagedTags | boolean | Specifies whether to enable Amazon ECS managed tags for the tasks within the service. | | enableExecuteCommand | boolean | Whether ECS Exec should be enabled. | | healthCheckGracePeriod | aws-cdk-lib.Duration | The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing target health checks after a task has first started. | | idleTimeout | aws-cdk-lib.Duration | The load balancer idle timeout, in seconds. | | ipAddressType | aws-cdk-lib.aws_elasticloadbalancingv2.IpAddressType | The type of IP address to use. | | listenerPort | number | Listener port of the application load balancer that will serve traffic to the service. | | loadBalancer | aws-cdk-lib.aws_elasticloadbalancingv2.IApplicationLoadBalancer | The application load balancer that will serve traffic to the service. | | loadBalancerName | string | Name of the load balancer. | | maxHealthyPercent | number | The maximum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that can run in a service during a deployment. | | minHealthyPercent | number | The minimum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that must continue to run and remain healthy during a deployment. | | openListener | boolean | Determines whether or not the Security Group for the Load Balancer's Listener will be open to all traffic by default. | | propagateTags | aws-cdk-lib.aws_ecs.PropagatedTagSource | Specifies whether to propagate the tags from the task definition or the service to the tasks in the service. | | protocol | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol | The protocol for connections from clients to the load balancer. | | protocolVersion | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocolVersion | The protocol version to use. | | publicLoadBalancer | boolean | Determines whether the Load Balancer will be internet-facing. | | recordType | aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedServiceRecordType | Specifies whether the Route53 record should be a CNAME, an A record using the Alias feature or no record at all. | | redirectHTTP | boolean | Specifies whether the load balancer should redirect traffic on port 80 to the {@link listenerPort} to support HTTP->HTTPS redirects. | | serviceName | string | The name of the service. | | sslPolicy | aws-cdk-lib.aws_elasticloadbalancingv2.SslPolicy | The security policy that defines which ciphers and protocols are supported by the ALB Listener. | | targetProtocol | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol | The protocol for connections from the load balancer to the ECS tasks. | | taskImageOptions | aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedTaskImageOptions | The properties required to create a new task definition. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC where the container instances will be launched or the elastic network interfaces (ENIs) will be deployed. | | cpu | number | The number of cpu units used by the task. | | memoryLimitMiB | number | The hard limit (in MiB) of memory to present to the container. | | memoryReservationMiB | number | The soft limit (in MiB) of memory to reserve for the container. | | placementConstraints | aws-cdk-lib.aws_ecs.PlacementConstraint[] | The placement constraints to use for tasks in the service. | | placementStrategies | aws-cdk-lib.aws_ecs.PlacementStrategy[] | The placement strategies to use for tasks in the service. | | taskDefinition | aws-cdk-lib.aws_ecs.Ec2TaskDefinition | The task definition to use for tasks in the service. TaskDefinition or TaskImageOptions must be specified, but not both.. | --- ##### `capacityProviderStrategies`^Optional ```typescript public readonly capacityProviderStrategies: CapacityProviderStrategy[]; ``` - *Type:* aws-cdk-lib.aws_ecs.CapacityProviderStrategy[] - *Default:* undefined A list of Capacity Provider strategies used to place a service. --- ##### `certificate`^Optional ```typescript public readonly certificate: ICertificate; ``` - *Type:* aws-cdk-lib.aws_certificatemanager.ICertificate - *Default:* No certificate associated with the load balancer, if using the HTTP protocol. For HTTPS, a DNS-validated certificate will be created for the load balancer's specified domain name if a domain name and domain zone are specified. Certificate Manager certificate to associate with the load balancer. Setting this option will set the load balancer protocol to HTTPS. --- ##### `circuitBreaker`^Optional ```typescript public readonly circuitBreaker: DeploymentCircuitBreaker; ``` - *Type:* aws-cdk-lib.aws_ecs.DeploymentCircuitBreaker - *Default:* disabled Whether to enable the deployment circuit breaker. If this property is defined, circuit breaker will be implicitly enabled. --- ##### `cloudMapOptions`^Optional ```typescript public readonly cloudMapOptions: CloudMapOptions; ``` - *Type:* aws-cdk-lib.aws_ecs.CloudMapOptions - *Default:* AWS Cloud Map service discovery is not enabled. The options for configuring an Amazon ECS service to use service discovery. --- ##### `cluster`^Optional ```typescript public readonly cluster: ICluster; ``` - *Type:* aws-cdk-lib.aws_ecs.ICluster - *Default:* create a new cluster; if both cluster and vpc are omitted, a new VPC will be created for you. The name of the cluster that hosts the service. If a cluster is specified, the vpc construct should be omitted. Alternatively, you can omit both cluster and vpc. --- ##### `deploymentController`^Optional ```typescript public readonly deploymentController: DeploymentController; ``` - *Type:* aws-cdk-lib.aws_ecs.DeploymentController - *Default:* Rolling update (ECS) Specifies which deployment controller to use for the service. For more information, see [Amazon ECS Deployment Types](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-types.html) --- ##### `desiredCount`^Optional ```typescript public readonly desiredCount: number; ``` - *Type:* number - *Default:* The default is 1 for all new services and uses the existing service's desired count when updating an existing service. The desired number of instantiations of the task definition to keep running on the service. The minimum value is 1 --- ##### `domainName`^Optional ```typescript public readonly domainName: string; ``` - *Type:* string - *Default:* No domain name. The domain name for the service, e.g. "api.example.com.". --- ##### `domainZone`^Optional ```typescript public readonly domainZone: IHostedZone; ``` - *Type:* aws-cdk-lib.aws_route53.IHostedZone - *Default:* No Route53 hosted domain zone. The Route53 hosted zone for the domain, e.g. "example.com.". --- ##### `enableECSManagedTags`^Optional ```typescript public readonly enableECSManagedTags: boolean; ``` - *Type:* boolean - *Default:* false Specifies whether to enable Amazon ECS managed tags for the tasks within the service. For more information, see [Tagging Your Amazon ECS Resources](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html) --- ##### `enableExecuteCommand`^Optional ```typescript public readonly enableExecuteCommand: boolean; ``` - *Type:* boolean - *Default:* false Whether ECS Exec should be enabled. --- ##### `healthCheckGracePeriod`^Optional ```typescript public readonly healthCheckGracePeriod: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* defaults to 60 seconds if at least one load balancer is in-use and it is not already set The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing target health checks after a task has first started. --- ##### `idleTimeout`^Optional ```typescript public readonly idleTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* CloudFormation sets idle timeout to 60 seconds The load balancer idle timeout, in seconds. Can be between 1 and 4000 seconds --- ##### `ipAddressType`^Optional ```typescript public readonly ipAddressType: IpAddressType; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.IpAddressType - *Default:* IpAddressType.IPV4 The type of IP address to use. --- ##### `listenerPort`^Optional ```typescript public readonly listenerPort: number; ``` - *Type:* number - *Default:* The default listener port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). A domain name and zone must be also be specified if using HTTPS. Listener port of the application load balancer that will serve traffic to the service. --- ##### `loadBalancer`^Optional ```typescript public readonly loadBalancer: IApplicationLoadBalancer; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.IApplicationLoadBalancer - *Default:* a new load balancer will be created. The application load balancer that will serve traffic to the service. The VPC attribute of a load balancer must be specified for it to be used to create a new service with this pattern. [disable-awslint:ref-via-interface] --- ##### `loadBalancerName`^Optional ```typescript public readonly loadBalancerName: string; ``` - *Type:* string - *Default:* Automatically generated name. Name of the load balancer. --- ##### `maxHealthyPercent`^Optional ```typescript public readonly maxHealthyPercent: number; ``` - *Type:* number - *Default:* 100 if daemon, otherwise 200 The maximum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that can run in a service during a deployment. --- ##### `minHealthyPercent`^Optional ```typescript public readonly minHealthyPercent: number; ``` - *Type:* number - *Default:* 0 if daemon, otherwise 50 The minimum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that must continue to run and remain healthy during a deployment. --- ##### `openListener`^Optional ```typescript public readonly openListener: boolean; ``` - *Type:* boolean - *Default:* true -- The security group allows ingress from all IP addresses. Determines whether or not the Security Group for the Load Balancer's Listener will be open to all traffic by default. --- ##### `propagateTags`^Optional ```typescript public readonly propagateTags: PropagatedTagSource; ``` - *Type:* aws-cdk-lib.aws_ecs.PropagatedTagSource - *Default:* none Specifies whether to propagate the tags from the task definition or the service to the tasks in the service. Tags can only be propagated to the tasks within the service during service creation. --- ##### `protocol`^Optional ```typescript public readonly protocol: ApplicationProtocol; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol - *Default:* HTTP. If a certificate is specified, the protocol will be set by default to HTTPS. The protocol for connections from clients to the load balancer. The load balancer port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). If HTTPS, either a certificate or domain name and domain zone must also be specified. --- ##### `protocolVersion`^Optional ```typescript public readonly protocolVersion: ApplicationProtocolVersion; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocolVersion - *Default:* ApplicationProtocolVersion.HTTP1 The protocol version to use. --- ##### `publicLoadBalancer`^Optional ```typescript public readonly publicLoadBalancer: boolean; ``` - *Type:* boolean - *Default:* true Determines whether the Load Balancer will be internet-facing. --- ##### `recordType`^Optional ```typescript public readonly recordType: ApplicationLoadBalancedServiceRecordType; ``` - *Type:* aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedServiceRecordType - *Default:* ApplicationLoadBalancedServiceRecordType.ALIAS Specifies whether the Route53 record should be a CNAME, an A record using the Alias feature or no record at all. This is useful if you need to work with DNS systems that do not support alias records. --- ##### `redirectHTTP`^Optional ```typescript public readonly redirectHTTP: boolean; ``` - *Type:* boolean - *Default:* false Specifies whether the load balancer should redirect traffic on port 80 to the {@link listenerPort} to support HTTP->HTTPS redirects. This is only valid if the protocol of the ALB is HTTPS. --- ##### `serviceName`^Optional ```typescript public readonly serviceName: string; ``` - *Type:* string - *Default:* CloudFormation-generated name. The name of the service. --- ##### `sslPolicy`^Optional ```typescript public readonly sslPolicy: SslPolicy; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.SslPolicy - *Default:* The recommended elastic load balancing security policy The security policy that defines which ciphers and protocols are supported by the ALB Listener. --- ##### `targetProtocol`^Optional ```typescript public readonly targetProtocol: ApplicationProtocol; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol - *Default:* HTTP. The protocol for connections from the load balancer to the ECS tasks. The default target port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). --- ##### `taskImageOptions`^Optional ```typescript public readonly taskImageOptions: ApplicationLoadBalancedTaskImageOptions; ``` - *Type:* aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedTaskImageOptions - *Default:* none The properties required to create a new task definition. TaskDefinition or TaskImageOptions must be specified, but not both. --- ##### `vpc`^Optional ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc - *Default:* uses the VPC defined in the cluster or creates a new VPC. The VPC where the container instances will be launched or the elastic network interfaces (ENIs) will be deployed. If a vpc is specified, the cluster construct should be omitted. Alternatively, you can omit both vpc and cluster. --- ##### `cpu`^Optional ```typescript public readonly cpu: number; ``` - *Type:* number - *Default:* none The number of cpu units used by the task. Valid values, which determines your range of valid values for the memory parameter: 256 (.25 vCPU) - Available memory values: 0.5GB, 1GB, 2GB 512 (.5 vCPU) - Available memory values: 1GB, 2GB, 3GB, 4GB 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB 2048 (2 vCPU) - Available memory values: Between 4GB and 16GB in 1GB increments 4096 (4 vCPU) - Available memory values: Between 8GB and 30GB in 1GB increments This default is set in the underlying FargateTaskDefinition construct. --- ##### `memoryLimitMiB`^Optional ```typescript public readonly memoryLimitMiB: number; ``` - *Type:* number - *Default:* No memory limit. The hard limit (in MiB) of memory to present to the container. If your container attempts to exceed the allocated memory, the container is terminated. At least one of memoryLimitMiB and memoryReservationMiB is required. --- ##### `memoryReservationMiB`^Optional ```typescript public readonly memoryReservationMiB: number; ``` - *Type:* number - *Default:* No memory reserved. The soft limit (in MiB) of memory to reserve for the container. When system memory is under contention, Docker attempts to keep the container memory within the limit. If the container requires more memory, it can consume up to the value specified by the Memory property or all of the available memory on the container instance—whichever comes first. At least one of memoryLimitMiB and memoryReservationMiB is required. --- ##### `placementConstraints`^Optional ```typescript public readonly placementConstraints: PlacementConstraint[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint[] - *Default:* No constraints. The placement constraints to use for tasks in the service. For more information, see [Amazon ECS Task Placement Constraints](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html). --- ##### `placementStrategies`^Optional ```typescript public readonly placementStrategies: PlacementStrategy[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementStrategy[] - *Default:* No strategies. The placement strategies to use for tasks in the service. For more information, see [Amazon ECS Task Placement Strategies](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html). --- ##### `taskDefinition`^Optional ```typescript public readonly taskDefinition: Ec2TaskDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinition - *Default:* none The task definition to use for tasks in the service. TaskDefinition or TaskImageOptions must be specified, but not both.. [disable-awslint:ref-via-interface] --- ### ApplicationLoadBalancedVllmNxDInferenceServiceProps Props for ApplicationLoadBalancedVllmNxDInferenceService. #### Initializer ```typescript import { ApplicationLoadBalancedVllmNxDInferenceServiceProps } from 'aws-cdk-neuronx-patterns' const applicationLoadBalancedVllmNxDInferenceServiceProps: ApplicationLoadBalancedVllmNxDInferenceServiceProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | capacityProviderStrategies | aws-cdk-lib.aws_ecs.CapacityProviderStrategy[] | A list of Capacity Provider strategies used to place a service. | | certificate | aws-cdk-lib.aws_certificatemanager.ICertificate | Certificate Manager certificate to associate with the load balancer. | | circuitBreaker | aws-cdk-lib.aws_ecs.DeploymentCircuitBreaker | Whether to enable the deployment circuit breaker. | | cloudMapOptions | aws-cdk-lib.aws_ecs.CloudMapOptions | The options for configuring an Amazon ECS service to use service discovery. | | cluster | aws-cdk-lib.aws_ecs.ICluster | The name of the cluster that hosts the service. | | deploymentController | aws-cdk-lib.aws_ecs.DeploymentController | Specifies which deployment controller to use for the service. | | desiredCount | number | The desired number of instantiations of the task definition to keep running on the service. | | domainName | string | The domain name for the service, e.g. "api.example.com.". | | domainZone | aws-cdk-lib.aws_route53.IHostedZone | The Route53 hosted zone for the domain, e.g. "example.com.". | | enableECSManagedTags | boolean | Specifies whether to enable Amazon ECS managed tags for the tasks within the service. | | enableExecuteCommand | boolean | Whether ECS Exec should be enabled. | | healthCheckGracePeriod | aws-cdk-lib.Duration | The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing target health checks after a task has first started. | | idleTimeout | aws-cdk-lib.Duration | The load balancer idle timeout, in seconds. | | ipAddressType | aws-cdk-lib.aws_elasticloadbalancingv2.IpAddressType | The type of IP address to use. | | listenerPort | number | Listener port of the application load balancer that will serve traffic to the service. | | loadBalancer | aws-cdk-lib.aws_elasticloadbalancingv2.IApplicationLoadBalancer | The application load balancer that will serve traffic to the service. | | loadBalancerName | string | Name of the load balancer. | | maxHealthyPercent | number | The maximum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that can run in a service during a deployment. | | minHealthyPercent | number | The minimum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that must continue to run and remain healthy during a deployment. | | openListener | boolean | Determines whether or not the Security Group for the Load Balancer's Listener will be open to all traffic by default. | | propagateTags | aws-cdk-lib.aws_ecs.PropagatedTagSource | Specifies whether to propagate the tags from the task definition or the service to the tasks in the service. | | protocol | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol | The protocol for connections from clients to the load balancer. | | protocolVersion | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocolVersion | The protocol version to use. | | publicLoadBalancer | boolean | Determines whether the Load Balancer will be internet-facing. | | recordType | aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedServiceRecordType | Specifies whether the Route53 record should be a CNAME, an A record using the Alias feature or no record at all. | | redirectHTTP | boolean | Specifies whether the load balancer should redirect traffic on port 80 to the {@link listenerPort} to support HTTP->HTTPS redirects. | | serviceName | string | The name of the service. | | sslPolicy | aws-cdk-lib.aws_elasticloadbalancingv2.SslPolicy | The security policy that defines which ciphers and protocols are supported by the ALB Listener. | | targetProtocol | aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol | The protocol for connections from the load balancer to the ECS tasks. | | taskImageOptions | aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedTaskImageOptions | The properties required to create a new task definition. | | vpc | aws-cdk-lib.aws_ec2.IVpc | The VPC where the container instances will be launched or the elastic network interfaces (ENIs) will be deployed. | | cpu | number | The number of cpu units used by the task. | | memoryLimitMiB | number | The hard limit (in MiB) of memory to present to the container. | | memoryReservationMiB | number | The soft limit (in MiB) of memory to reserve for the container. | | placementConstraints | aws-cdk-lib.aws_ecs.PlacementConstraint[] | The placement constraints to use for tasks in the service. | | placementStrategies | aws-cdk-lib.aws_ecs.PlacementStrategy[] | The placement strategies to use for tasks in the service. | | taskDefinition | aws-cdk-lib.aws_ecs.Ec2TaskDefinition | The task definition to use for tasks in the service. TaskDefinition or TaskImageOptions must be specified, but not both.. | --- ##### `capacityProviderStrategies`^Optional ```typescript public readonly capacityProviderStrategies: CapacityProviderStrategy[]; ``` - *Type:* aws-cdk-lib.aws_ecs.CapacityProviderStrategy[] - *Default:* undefined A list of Capacity Provider strategies used to place a service. --- ##### `certificate`^Optional ```typescript public readonly certificate: ICertificate; ``` - *Type:* aws-cdk-lib.aws_certificatemanager.ICertificate - *Default:* No certificate associated with the load balancer, if using the HTTP protocol. For HTTPS, a DNS-validated certificate will be created for the load balancer's specified domain name if a domain name and domain zone are specified. Certificate Manager certificate to associate with the load balancer. Setting this option will set the load balancer protocol to HTTPS. --- ##### `circuitBreaker`^Optional ```typescript public readonly circuitBreaker: DeploymentCircuitBreaker; ``` - *Type:* aws-cdk-lib.aws_ecs.DeploymentCircuitBreaker - *Default:* disabled Whether to enable the deployment circuit breaker. If this property is defined, circuit breaker will be implicitly enabled. --- ##### `cloudMapOptions`^Optional ```typescript public readonly cloudMapOptions: CloudMapOptions; ``` - *Type:* aws-cdk-lib.aws_ecs.CloudMapOptions - *Default:* AWS Cloud Map service discovery is not enabled. The options for configuring an Amazon ECS service to use service discovery. --- ##### `cluster`^Optional ```typescript public readonly cluster: ICluster; ``` - *Type:* aws-cdk-lib.aws_ecs.ICluster - *Default:* create a new cluster; if both cluster and vpc are omitted, a new VPC will be created for you. The name of the cluster that hosts the service. If a cluster is specified, the vpc construct should be omitted. Alternatively, you can omit both cluster and vpc. --- ##### `deploymentController`^Optional ```typescript public readonly deploymentController: DeploymentController; ``` - *Type:* aws-cdk-lib.aws_ecs.DeploymentController - *Default:* Rolling update (ECS) Specifies which deployment controller to use for the service. For more information, see [Amazon ECS Deployment Types](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-types.html) --- ##### `desiredCount`^Optional ```typescript public readonly desiredCount: number; ``` - *Type:* number - *Default:* The default is 1 for all new services and uses the existing service's desired count when updating an existing service. The desired number of instantiations of the task definition to keep running on the service. The minimum value is 1 --- ##### `domainName`^Optional ```typescript public readonly domainName: string; ``` - *Type:* string - *Default:* No domain name. The domain name for the service, e.g. "api.example.com.". --- ##### `domainZone`^Optional ```typescript public readonly domainZone: IHostedZone; ``` - *Type:* aws-cdk-lib.aws_route53.IHostedZone - *Default:* No Route53 hosted domain zone. The Route53 hosted zone for the domain, e.g. "example.com.". --- ##### `enableECSManagedTags`^Optional ```typescript public readonly enableECSManagedTags: boolean; ``` - *Type:* boolean - *Default:* false Specifies whether to enable Amazon ECS managed tags for the tasks within the service. For more information, see [Tagging Your Amazon ECS Resources](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html) --- ##### `enableExecuteCommand`^Optional ```typescript public readonly enableExecuteCommand: boolean; ``` - *Type:* boolean - *Default:* false Whether ECS Exec should be enabled. --- ##### `healthCheckGracePeriod`^Optional ```typescript public readonly healthCheckGracePeriod: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* defaults to 60 seconds if at least one load balancer is in-use and it is not already set The period of time, in seconds, that the Amazon ECS service scheduler ignores unhealthy Elastic Load Balancing target health checks after a task has first started. --- ##### `idleTimeout`^Optional ```typescript public readonly idleTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* CloudFormation sets idle timeout to 60 seconds The load balancer idle timeout, in seconds. Can be between 1 and 4000 seconds --- ##### `ipAddressType`^Optional ```typescript public readonly ipAddressType: IpAddressType; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.IpAddressType - *Default:* IpAddressType.IPV4 The type of IP address to use. --- ##### `listenerPort`^Optional ```typescript public readonly listenerPort: number; ``` - *Type:* number - *Default:* The default listener port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). A domain name and zone must be also be specified if using HTTPS. Listener port of the application load balancer that will serve traffic to the service. --- ##### `loadBalancer`^Optional ```typescript public readonly loadBalancer: IApplicationLoadBalancer; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.IApplicationLoadBalancer - *Default:* a new load balancer will be created. The application load balancer that will serve traffic to the service. The VPC attribute of a load balancer must be specified for it to be used to create a new service with this pattern. [disable-awslint:ref-via-interface] --- ##### `loadBalancerName`^Optional ```typescript public readonly loadBalancerName: string; ``` - *Type:* string - *Default:* Automatically generated name. Name of the load balancer. --- ##### `maxHealthyPercent`^Optional ```typescript public readonly maxHealthyPercent: number; ``` - *Type:* number - *Default:* 100 if daemon, otherwise 200 The maximum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that can run in a service during a deployment. --- ##### `minHealthyPercent`^Optional ```typescript public readonly minHealthyPercent: number; ``` - *Type:* number - *Default:* 0 if daemon, otherwise 50 The minimum number of tasks, specified as a percentage of the Amazon ECS service's DesiredCount value, that must continue to run and remain healthy during a deployment. --- ##### `openListener`^Optional ```typescript public readonly openListener: boolean; ``` - *Type:* boolean - *Default:* true -- The security group allows ingress from all IP addresses. Determines whether or not the Security Group for the Load Balancer's Listener will be open to all traffic by default. --- ##### `propagateTags`^Optional ```typescript public readonly propagateTags: PropagatedTagSource; ``` - *Type:* aws-cdk-lib.aws_ecs.PropagatedTagSource - *Default:* none Specifies whether to propagate the tags from the task definition or the service to the tasks in the service. Tags can only be propagated to the tasks within the service during service creation. --- ##### `protocol`^Optional ```typescript public readonly protocol: ApplicationProtocol; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol - *Default:* HTTP. If a certificate is specified, the protocol will be set by default to HTTPS. The protocol for connections from clients to the load balancer. The load balancer port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). If HTTPS, either a certificate or domain name and domain zone must also be specified. --- ##### `protocolVersion`^Optional ```typescript public readonly protocolVersion: ApplicationProtocolVersion; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocolVersion - *Default:* ApplicationProtocolVersion.HTTP1 The protocol version to use. --- ##### `publicLoadBalancer`^Optional ```typescript public readonly publicLoadBalancer: boolean; ``` - *Type:* boolean - *Default:* true Determines whether the Load Balancer will be internet-facing. --- ##### `recordType`^Optional ```typescript public readonly recordType: ApplicationLoadBalancedServiceRecordType; ``` - *Type:* aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedServiceRecordType - *Default:* ApplicationLoadBalancedServiceRecordType.ALIAS Specifies whether the Route53 record should be a CNAME, an A record using the Alias feature or no record at all. This is useful if you need to work with DNS systems that do not support alias records. --- ##### `redirectHTTP`^Optional ```typescript public readonly redirectHTTP: boolean; ``` - *Type:* boolean - *Default:* false Specifies whether the load balancer should redirect traffic on port 80 to the {@link listenerPort} to support HTTP->HTTPS redirects. This is only valid if the protocol of the ALB is HTTPS. --- ##### `serviceName`^Optional ```typescript public readonly serviceName: string; ``` - *Type:* string - *Default:* CloudFormation-generated name. The name of the service. --- ##### `sslPolicy`^Optional ```typescript public readonly sslPolicy: SslPolicy; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.SslPolicy - *Default:* The recommended elastic load balancing security policy The security policy that defines which ciphers and protocols are supported by the ALB Listener. --- ##### `targetProtocol`^Optional ```typescript public readonly targetProtocol: ApplicationProtocol; ``` - *Type:* aws-cdk-lib.aws_elasticloadbalancingv2.ApplicationProtocol - *Default:* HTTP. The protocol for connections from the load balancer to the ECS tasks. The default target port is determined from the protocol (port 80 for HTTP, port 443 for HTTPS). --- ##### `taskImageOptions`^Optional ```typescript public readonly taskImageOptions: ApplicationLoadBalancedTaskImageOptions; ``` - *Type:* aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedTaskImageOptions - *Default:* none The properties required to create a new task definition. TaskDefinition or TaskImageOptions must be specified, but not both. --- ##### `vpc`^Optional ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc - *Default:* uses the VPC defined in the cluster or creates a new VPC. The VPC where the container instances will be launched or the elastic network interfaces (ENIs) will be deployed. If a vpc is specified, the cluster construct should be omitted. Alternatively, you can omit both vpc and cluster. --- ##### `cpu`^Optional ```typescript public readonly cpu: number; ``` - *Type:* number - *Default:* none The number of cpu units used by the task. Valid values, which determines your range of valid values for the memory parameter: 256 (.25 vCPU) - Available memory values: 0.5GB, 1GB, 2GB 512 (.5 vCPU) - Available memory values: 1GB, 2GB, 3GB, 4GB 1024 (1 vCPU) - Available memory values: 2GB, 3GB, 4GB, 5GB, 6GB, 7GB, 8GB 2048 (2 vCPU) - Available memory values: Between 4GB and 16GB in 1GB increments 4096 (4 vCPU) - Available memory values: Between 8GB and 30GB in 1GB increments This default is set in the underlying FargateTaskDefinition construct. --- ##### `memoryLimitMiB`^Optional ```typescript public readonly memoryLimitMiB: number; ``` - *Type:* number - *Default:* No memory limit. The hard limit (in MiB) of memory to present to the container. If your container attempts to exceed the allocated memory, the container is terminated. At least one of memoryLimitMiB and memoryReservationMiB is required. --- ##### `memoryReservationMiB`^Optional ```typescript public readonly memoryReservationMiB: number; ``` - *Type:* number - *Default:* No memory reserved. The soft limit (in MiB) of memory to reserve for the container. When system memory is under contention, Docker attempts to keep the container memory within the limit. If the container requires more memory, it can consume up to the value specified by the Memory property or all of the available memory on the container instance—whichever comes first. At least one of memoryLimitMiB and memoryReservationMiB is required. --- ##### `placementConstraints`^Optional ```typescript public readonly placementConstraints: PlacementConstraint[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint[] - *Default:* No constraints. The placement constraints to use for tasks in the service. For more information, see [Amazon ECS Task Placement Constraints](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-constraints.html). --- ##### `placementStrategies`^Optional ```typescript public readonly placementStrategies: PlacementStrategy[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementStrategy[] - *Default:* No strategies. The placement strategies to use for tasks in the service. For more information, see [Amazon ECS Task Placement Strategies](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-placement-strategies.html). --- ##### `taskDefinition`^Optional ```typescript public readonly taskDefinition: Ec2TaskDefinition; ``` - *Type:* aws-cdk-lib.aws_ecs.Ec2TaskDefinition - *Default:* none The task definition to use for tasks in the service. TaskDefinition or TaskImageOptions must be specified, but not both.. [disable-awslint:ref-via-interface] --- ### ComputeEnvironmentResult Result of creating a compute environment. #### Initializer ```typescript import { ComputeEnvironmentResult } from 'aws-cdk-neuronx-patterns' const computeEnvironmentResult: ComputeEnvironmentResult = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | computeEnvironment | aws-cdk-lib.aws_batch.IComputeEnvironment | The compute environment. | | instanceRole | aws-cdk-lib.aws_iam.IRole | The instance role associated with the compute environment. | --- ##### `computeEnvironment`^Required ```typescript public readonly computeEnvironment: IComputeEnvironment; ``` - *Type:* aws-cdk-lib.aws_batch.IComputeEnvironment The compute environment. --- ##### `instanceRole`^Required ```typescript public readonly instanceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The instance role associated with the compute environment. --- ### ModelConfig #### Initializer ```typescript import { ModelConfig } from 'aws-cdk-neuronx-patterns' const modelConfig: ModelConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | attentionHeads | number | *No description.* | | embeddingDimension | number | *No description.* | | layers | number | *No description.* | --- ##### `attentionHeads`^Required ```typescript public readonly attentionHeads: number; ``` - *Type:* number --- ##### `embeddingDimension`^Required ```typescript public readonly embeddingDimension: number; ``` - *Type:* number --- ##### `layers`^Required ```typescript public readonly layers: number; ``` - *Type:* number --- ### ModelOptions Compile target model basic infromation. #### Initializer ```typescript import { ModelOptions } from 'aws-cdk-neuronx-patterns' const modelOptions: ModelOptions = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | parameters | Parameters | *No description.* | | config | ModelConfig | *No description.* | | modelName | string | *No description.* | --- ##### `parameters`^Required ```typescript public readonly parameters: Parameters; ``` - *Type:* Parameters --- ##### `config`^Optional ```typescript public readonly config: ModelConfig; ``` - *Type:* ModelConfig --- ##### `modelName`^Optional ```typescript public readonly modelName: string; ``` - *Type:* string --- ### NeuronxBatchComputeEnvironmentProps Props of NeuronxBatch. #### Initializer ```typescript import { NeuronxBatchComputeEnvironmentProps } from 'aws-cdk-neuronx-patterns' const neuronxBatchComputeEnvironmentProps: NeuronxBatchComputeEnvironmentProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | computeEnvironmentName | string | The name of the ComputeEnvironment. | | enabled | boolean | Whether or not this ComputeEnvironment can accept jobs from a Queue. | | serviceRole | aws-cdk-lib.aws_iam.IRole | The role Batch uses to perform actions on your behalf in your account, such as provision instances to run your jobs. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this Compute Environment will launch Instances. | | maxvCpus | number | The maximum vCpus this `ManagedComputeEnvironment` can scale up to. Each vCPU is equivalent to 1024 CPU shares. | | replaceComputeEnvironment | boolean | Specifies whether this Compute Environment is replaced if an update is made that requires replacing its instances. | | securityGroups | aws-cdk-lib.aws_ec2.ISecurityGroup[] | The security groups this Compute Environment will launch instances in. | | spot | boolean | Whether or not to use spot instances. | | terminateOnUpdate | boolean | Whether or not any running jobs will be immediately terminated when an infrastructure update occurs. | | updateTimeout | aws-cdk-lib.Duration | Only meaningful if `terminateOnUpdate` is `false`. | | updateToLatestImageVersion | boolean | Whether or not the AMI is updated to the latest one supported by Batch when an infrastructure update occurs. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | | allocationStrategy | aws-cdk-lib.aws_batch.AllocationStrategy | The allocation strategy to use if not enough instances of the best fitting instance type can be allocated. | | defaultInstanceClasses | aws-cdk-lib.aws_batch.DefaultInstanceClass[] | Use batch's default instance types. | | images | aws-cdk-lib.aws_batch.EcsMachineImage[] | Configure which AMIs this Compute Environment can launch. | | instanceClasses | aws-cdk-lib.aws_ec2.InstanceClass[] | The instance classes that this Compute Environment can launch. | | instanceRole | aws-cdk-lib.aws_iam.IRole | The execution Role that instances launched by this Compute Environment will use. | | instanceTypes | aws-cdk-lib.aws_ec2.InstanceType[] | The instance types that this Compute Environment can launch. | | launchTemplate | aws-cdk-lib.aws_ec2.ILaunchTemplate | The Launch Template that this Compute Environment will use to provision EC2 Instances. | | minvCpus | number | The minimum vCPUs that an environment should maintain, even if the compute environment is DISABLED. | | placementGroup | aws-cdk-lib.interfaces.aws_ec2.IPlacementGroupRef | The EC2 placement group to associate with your compute resources. | | spotBidPercentage | number | The maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. | | spotFleetRole | aws-cdk-lib.aws_iam.IRole | The service-linked role that Spot Fleet needs to launch instances on your behalf. | | useOptimalInstanceClasses | boolean | Whether or not to use batch's optimal instance type. | --- ##### `computeEnvironmentName`^Optional ```typescript public readonly computeEnvironmentName: string; ``` - *Type:* string - *Default:* generated by CloudFormation The name of the ComputeEnvironment. --- ##### `enabled`^Optional ```typescript public readonly enabled: boolean; ``` - *Type:* boolean - *Default:* true Whether or not this ComputeEnvironment can accept jobs from a Queue. Enabled ComputeEnvironments can accept jobs from a Queue and can scale instances up or down. Disabled ComputeEnvironments cannot accept jobs from a Queue or scale instances up or down. If you change a ComputeEnvironment from enabled to disabled while it is executing jobs, Jobs in the `STARTED` or `RUNNING` states will not be interrupted. As jobs complete, the ComputeEnvironment will scale instances down to `minvCpus`. To ensure you aren't billed for unused capacity, set `minvCpus` to `0`. --- ##### `serviceRole`^Optional ```typescript public readonly serviceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* a serviceRole will be created for managed CEs, none for unmanaged CEs The role Batch uses to perform actions on your behalf in your account, such as provision instances to run your jobs. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this Compute Environment will launch Instances. --- ##### `maxvCpus`^Optional ```typescript public readonly maxvCpus: number; ``` - *Type:* number - *Default:* 256 The maximum vCpus this `ManagedComputeEnvironment` can scale up to. Each vCPU is equivalent to 1024 CPU shares. *Note*: if this Compute Environment uses EC2 resources (not Fargate) with either `AllocationStrategy.BEST_FIT_PROGRESSIVE` or `AllocationStrategy.SPOT_CAPACITY_OPTIMIZED`, or `AllocationStrategy.BEST_FIT` with Spot instances, The scheduler may exceed this number by at most one of the instances specified in `instanceTypes` or `instanceClasses`. --- ##### `replaceComputeEnvironment`^Optional ```typescript public readonly replaceComputeEnvironment: boolean; ``` - *Type:* boolean - *Default:* false Specifies whether this Compute Environment is replaced if an update is made that requires replacing its instances. To enable more properties to be updated, set this property to `false`. When changing the value of this property to false, do not change any other properties at the same time. If other properties are changed at the same time, and the change needs to be rolled back but it can't, it's possible for the stack to go into the UPDATE_ROLLBACK_FAILED state. You can't update a stack that is in the UPDATE_ROLLBACK_FAILED state. However, if you can continue to roll it back, you can return the stack to its original settings and then try to update it again. The properties which require a replacement of the Compute Environment are: > [https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-continueupdaterollback.html](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-continueupdaterollback.html) --- ##### `securityGroups`^Optional ```typescript public readonly securityGroups: ISecurityGroup[]; ``` - *Type:* aws-cdk-lib.aws_ec2.ISecurityGroup[] - *Default:* new security groups will be created The security groups this Compute Environment will launch instances in. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `terminateOnUpdate`^Optional ```typescript public readonly terminateOnUpdate: boolean; ``` - *Type:* boolean - *Default:* false Whether or not any running jobs will be immediately terminated when an infrastructure update occurs. If this is enabled, any terminated jobs may be retried, depending on the job's retry policy. > [https://docs.aws.amazon.com/batch/latest/userguide/updating-compute-environments.html](https://docs.aws.amazon.com/batch/latest/userguide/updating-compute-environments.html) --- ##### `updateTimeout`^Optional ```typescript public readonly updateTimeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* 30 minutes Only meaningful if `terminateOnUpdate` is `false`. If so, when an infrastructure update is triggered, any running jobs will be allowed to run until `updateTimeout` has expired. > [https://docs.aws.amazon.com/batch/latest/userguide/updating-compute-environments.html](https://docs.aws.amazon.com/batch/latest/userguide/updating-compute-environments.html) --- ##### `updateToLatestImageVersion`^Optional ```typescript public readonly updateToLatestImageVersion: boolean; ``` - *Type:* boolean - *Default:* false Whether or not the AMI is updated to the latest one supported by Batch when an infrastructure update occurs. If you specify a specific AMI, this property will be ignored. Note: the CDK will never set this value by default, `false` will set by CFN. This is to avoid a deployment failure that occurs when this value is set. > [https://github.com/aws/aws-cdk/issues/27054](https://github.com/aws/aws-cdk/issues/27054) --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ##### `allocationStrategy`^Optional ```typescript public readonly allocationStrategy: AllocationStrategy; ``` - *Type:* aws-cdk-lib.aws_batch.AllocationStrategy - *Default:* `BEST_FIT_PROGRESSIVE` if not using Spot instances, `SPOT_PRICE_CAPACITY_OPTIMIZED` if using Spot instances. The allocation strategy to use if not enough instances of the best fitting instance type can be allocated. --- ##### `defaultInstanceClasses`^Optional ```typescript public readonly defaultInstanceClasses: DefaultInstanceClass[]; ``` - *Type:* aws-cdk-lib.aws_batch.DefaultInstanceClass[] - *Default:* choose from instanceTypes and instanceClasses Use batch's default instance types. A simpler way to choose up-to-date instance classes based on region instead of specifying exact instance classes. > [https://docs.aws.amazon.com/batch/latest/userguide/instance-type-compute-table.html](https://docs.aws.amazon.com/batch/latest/userguide/instance-type-compute-table.html) --- ##### `images`^Optional ```typescript public readonly images: EcsMachineImage[]; ``` - *Type:* aws-cdk-lib.aws_batch.EcsMachineImage[] - *Default:* ECS_AL2 for non-GPU instances, ECS_AL2_NVIDIA for GPU instances Configure which AMIs this Compute Environment can launch. If you specify this property with only `image` specified, then the `imageType` will default to `ECS_AL2`. *If your image needs GPU resources, specify `ECS_AL2_NVIDIA`; otherwise, the instances will not be able to properly join the ComputeEnvironment*. --- ##### `instanceClasses`^Optional ```typescript public readonly instanceClasses: InstanceClass[]; ``` - *Type:* aws-cdk-lib.aws_ec2.InstanceClass[] - *Default:* the instances Batch considers will be used (currently C4, M4, and R4) The instance classes that this Compute Environment can launch. Which one is chosen depends on the `AllocationStrategy` used. Batch will automatically choose the instance size. --- ##### `instanceRole`^Optional ```typescript public readonly instanceRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* a role will be created The execution Role that instances launched by this Compute Environment will use. --- ##### `instanceTypes`^Optional ```typescript public readonly instanceTypes: InstanceType[]; ``` - *Type:* aws-cdk-lib.aws_ec2.InstanceType[] - *Default:* the instances Batch considers will be used (currently C4, M4, and R4) The instance types that this Compute Environment can launch. Which one is chosen depends on the `AllocationStrategy` used. --- ##### `launchTemplate`^Optional ```typescript public readonly launchTemplate: ILaunchTemplate; ``` - *Type:* aws-cdk-lib.aws_ec2.ILaunchTemplate - *Default:* no launch template The Launch Template that this Compute Environment will use to provision EC2 Instances. *Note*: if `securityGroups` is specified on both your launch template and this Compute Environment, **the `securityGroup`s on the Compute Environment override the ones on the launch template. --- ##### `minvCpus`^Optional ```typescript public readonly minvCpus: number; ``` - *Type:* number - *Default:* 0 The minimum vCPUs that an environment should maintain, even if the compute environment is DISABLED. --- ##### `placementGroup`^Optional ```typescript public readonly placementGroup: IPlacementGroupRef; ``` - *Type:* aws-cdk-lib.interfaces.aws_ec2.IPlacementGroupRef - *Default:* no placement group The EC2 placement group to associate with your compute resources. If you intend to submit multi-node parallel jobs to this Compute Environment, you should consider creating a cluster placement group and associate it with your compute resources. This keeps your multi-node parallel job on a logical grouping of instances within a single Availability Zone with high network flow potential. > [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) --- ##### `spotBidPercentage`^Optional ```typescript public readonly spotBidPercentage: number; ``` - *Type:* number - *Default:* 100% The maximum percentage that a Spot Instance price can be when compared with the On-Demand price for that instance type before instances are launched. For example, if your maximum percentage is 20%, the Spot price must be less than 20% of the current On-Demand price for that Instance. You always pay the lowest market price and never more than your maximum percentage. For most use cases, Batch recommends leaving this field empty. Implies `spot == true` if set --- ##### `spotFleetRole`^Optional ```typescript public readonly spotFleetRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* a new role will be created The service-linked role that Spot Fleet needs to launch instances on your behalf. > [https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html](https://docs.aws.amazon.com/batch/latest/userguide/spot_fleet_IAM_role.html) --- ##### `useOptimalInstanceClasses`^Optional ```typescript public readonly useOptimalInstanceClasses: boolean; ``` - *Type:* boolean - *Default:* true Whether or not to use batch's optimal instance type. The optimal instance type is equivalent to adding the C4, M4, and R4 instance classes. You can specify other instance classes (of the same architecture) in addition to the optimal instance classes. --- ### NeuronxBatchEcsJobDefinitionProps Props of NeuronxBatch. #### Initializer ```typescript import { NeuronxBatchEcsJobDefinitionProps } from 'aws-cdk-neuronx-patterns' const neuronxBatchEcsJobDefinitionProps: NeuronxBatchEcsJobDefinitionProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | cpu | number | The number of vCPUs reserved for the container. | | image | aws-cdk-lib.aws_ecs.ContainerImage | The image that this container will run. | | memory | aws-cdk-lib.Size | The memory hard limit present to the container. | | command | string[] | The command that's passed to the container. | | enableExecuteCommand | boolean | Determines whether execute command functionality is turned on for this task. | | environment | {[ key: string ]: string} | The environment variables to pass to a container. | | executionRole | aws-cdk-lib.aws_iam.IRole | The role used by Amazon ECS container and AWS Fargate agents to make AWS API calls on your behalf. | | jobRole | aws-cdk-lib.aws_iam.IRole | The role that the container can assume. | | linuxParameters | aws-cdk-lib.aws_batch.LinuxParameters | Linux-specific modifications that are applied to the container, such as details for device mappings. | | logging | aws-cdk-lib.aws_ecs.LogDriver | The loging configuration for this Job. | | readonlyRootFilesystem | boolean | Gives the container readonly access to its root filesystem. | | secrets | {[ key: string ]: aws-cdk-lib.aws_batch.Secret} | A map from environment variable names to the secrets for the container. | | user | string | The user name to use inside the container. | | volumes | aws-cdk-lib.aws_batch.EcsVolume[] | The volumes to mount to this container. | | gpu | number | The number of physical GPUs to reserve for the container. | | privileged | boolean | When this parameter is true, the container is given elevated permissions on the host container instance (similar to the root user). | | ulimits | aws-cdk-lib.aws_batch.Ulimit[] | Limits to set for the user this docker container will run as. | | jobDefinitionName | string | The name of this job definition. | | parameters | {[ key: string ]: any} | The default parameters passed to the container These parameters can be referenced in the `command` that you give to the container. | | retryAttempts | number | The number of times to retry a job. | | retryStrategies | aws-cdk-lib.aws_batch.RetryStrategy[] | Defines the retry behavior for this job. | | schedulingPriority | number | The priority of this Job. | | timeout | aws-cdk-lib.Duration | The timeout time for jobs that are submitted with this job definition. | | neuronxInstanceType | INeuronxInstanceType | *No description.* | --- ##### `cpu`^Required ```typescript public readonly cpu: number; ``` - *Type:* number The number of vCPUs reserved for the container. Each vCPU is equivalent to 1,024 CPU shares. For containers running on EC2 resources, you must specify at least one vCPU. --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage The image that this container will run. --- ##### `memory`^Required ```typescript public readonly memory: Size; ``` - *Type:* aws-cdk-lib.Size The memory hard limit present to the container. If your container attempts to exceed the memory specified, the container is terminated. You must specify at least 4 MiB of memory for a job. --- ##### `command`^Optional ```typescript public readonly command: string[]; ``` - *Type:* string[] - *Default:* no command The command that's passed to the container. > [https://docs.docker.com/engine/reference/builder/#cmd](https://docs.docker.com/engine/reference/builder/#cmd) --- ##### `enableExecuteCommand`^Optional ```typescript public readonly enableExecuteCommand: boolean; ``` - *Type:* boolean - *Default:* undefined - AWS Batch default is false Determines whether execute command functionality is turned on for this task. If true, execute command functionality is turned on all the containers in the task. This allows you to use ECS Exec to access containers interactively. When enabled, a job role with required SSM permissions will be created automatically if no job role is provided. If a job role is alreadyprovided, the required permissions will be added to it. > [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* no environment variables The environment variables to pass to a container. Cannot start with `AWS_BATCH`. We don't recommend using plaintext environment variables for sensitive information, such as credential data. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* a Role will be created The role used by Amazon ECS container and AWS Fargate agents to make AWS API calls on your behalf. > [https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html](https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html) --- ##### `jobRole`^Optional ```typescript public readonly jobRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* no job role The role that the container can assume. > [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) --- ##### `linuxParameters`^Optional ```typescript public readonly linuxParameters: LinuxParameters; ``` - *Type:* aws-cdk-lib.aws_batch.LinuxParameters - *Default:* none Linux-specific modifications that are applied to the container, such as details for device mappings. --- ##### `logging`^Optional ```typescript public readonly logging: LogDriver; ``` - *Type:* aws-cdk-lib.aws_ecs.LogDriver - *Default:* the log configuration of the Docker daemon The loging configuration for this Job. --- ##### `readonlyRootFilesystem`^Optional ```typescript public readonly readonlyRootFilesystem: boolean; ``` - *Type:* boolean - *Default:* false Gives the container readonly access to its root filesystem. --- ##### `secrets`^Optional ```typescript public readonly secrets: {[ key: string ]: Secret}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_batch.Secret} - *Default:* no secrets A map from environment variable names to the secrets for the container. Allows your job definitions to reference the secret by the environment variable name defined in this property. > [https://docs.aws.amazon.com/batch/latest/userguide/specifying-sensitive-data.html](https://docs.aws.amazon.com/batch/latest/userguide/specifying-sensitive-data.html) --- ##### `user`^Optional ```typescript public readonly user: string; ``` - *Type:* string - *Default:* no user The user name to use inside the container. --- ##### `volumes`^Optional ```typescript public readonly volumes: EcsVolume[]; ``` - *Type:* aws-cdk-lib.aws_batch.EcsVolume[] - *Default:* no volumes The volumes to mount to this container. Automatically added to the job definition. --- ##### `gpu`^Optional ```typescript public readonly gpu: number; ``` - *Type:* number - *Default:* no gpus The number of physical GPUs to reserve for the container. Make sure that the number of GPUs reserved for all containers in a job doesn't exceed the number of available GPUs on the compute resource that the job is launched on. --- ##### `privileged`^Optional ```typescript public readonly privileged: boolean; ``` - *Type:* boolean - *Default:* false When this parameter is true, the container is given elevated permissions on the host container instance (similar to the root user). --- ##### `ulimits`^Optional ```typescript public readonly ulimits: Ulimit[]; ``` - *Type:* aws-cdk-lib.aws_batch.Ulimit[] - *Default:* no ulimits Limits to set for the user this docker container will run as. --- ##### `jobDefinitionName`^Optional ```typescript public readonly jobDefinitionName: string; ``` - *Type:* string - *Default:* generated by CloudFormation The name of this job definition. --- ##### `parameters`^Optional ```typescript public readonly parameters: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* none The default parameters passed to the container These parameters can be referenced in the `command` that you give to the container. > [https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#parameters](https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#parameters) --- ##### `retryAttempts`^Optional ```typescript public readonly retryAttempts: number; ``` - *Type:* number - *Default:* 1 The number of times to retry a job. The job is retried on failure the same number of attempts as the value. --- ##### `retryStrategies`^Optional ```typescript public readonly retryStrategies: RetryStrategy[]; ``` - *Type:* aws-cdk-lib.aws_batch.RetryStrategy[] - *Default:* no `RetryStrategy` Defines the retry behavior for this job. --- ##### `schedulingPriority`^Optional ```typescript public readonly schedulingPriority: number; ``` - *Type:* number - *Default:* none The priority of this Job. Only used in Fairshare Scheduling to decide which job to run first when there are multiple jobs with the same share identifier. --- ##### `timeout`^Optional ```typescript public readonly timeout: Duration; ``` - *Type:* aws-cdk-lib.Duration - *Default:* no timeout The timeout time for jobs that are submitted with this job definition. After the amount of time you specify passes, Batch terminates your jobs if they aren't finished. --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType --- ### NeuronxBatchProps Props of NeuronxBatch. #### Initializer ```typescript import { NeuronxBatchProps } from 'aws-cdk-neuronx-patterns' const neuronxBatchProps: NeuronxBatchProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | cpu | number | The number of vCPUs reserved for the container. | | image | aws-cdk-lib.aws_ecs.ContainerImage | The image that this container will run. | | memory | aws-cdk-lib.Size | The memory hard limit present to the container. | | command | string[] | The command that's passed to the container. | | enableExecuteCommand | boolean | Determines whether execute command functionality is turned on for this task. | | environment | {[ key: string ]: string} | The environment variables to pass to a container. | | executionRole | aws-cdk-lib.aws_iam.IRole | The role used by Amazon ECS container and AWS Fargate agents to make AWS API calls on your behalf. | | jobRole | aws-cdk-lib.aws_iam.IRole | The role that the container can assume. | | linuxParameters | aws-cdk-lib.aws_batch.LinuxParameters | Linux-specific modifications that are applied to the container, such as details for device mappings. | | logging | aws-cdk-lib.aws_ecs.LogDriver | The loging configuration for this Job. | | readonlyRootFilesystem | boolean | Gives the container readonly access to its root filesystem. | | secrets | {[ key: string ]: aws-cdk-lib.aws_batch.Secret} | A map from environment variable names to the secrets for the container. | | user | string | The user name to use inside the container. | | volumes | aws-cdk-lib.aws_batch.EcsVolume[] | The volumes to mount to this container. | | gpu | number | The number of physical GPUs to reserve for the container. | | privileged | boolean | When this parameter is true, the container is given elevated permissions on the host container instance (similar to the root user). | | ulimits | aws-cdk-lib.aws_batch.Ulimit[] | Limits to set for the user this docker container will run as. | | neuronxInstanceType | INeuronxInstanceType | The instance type of worker instance. | | volumeSize | aws-cdk-lib.Size | The root volume of worker instance. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this will launch worker instance. | | spot | boolean | Whether or not to use spot instances. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | --- ##### `cpu`^Required ```typescript public readonly cpu: number; ``` - *Type:* number The number of vCPUs reserved for the container. Each vCPU is equivalent to 1,024 CPU shares. For containers running on EC2 resources, you must specify at least one vCPU. --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage The image that this container will run. --- ##### `memory`^Required ```typescript public readonly memory: Size; ``` - *Type:* aws-cdk-lib.Size The memory hard limit present to the container. If your container attempts to exceed the memory specified, the container is terminated. You must specify at least 4 MiB of memory for a job. --- ##### `command`^Optional ```typescript public readonly command: string[]; ``` - *Type:* string[] - *Default:* no command The command that's passed to the container. > [https://docs.docker.com/engine/reference/builder/#cmd](https://docs.docker.com/engine/reference/builder/#cmd) --- ##### `enableExecuteCommand`^Optional ```typescript public readonly enableExecuteCommand: boolean; ``` - *Type:* boolean - *Default:* undefined - AWS Batch default is false Determines whether execute command functionality is turned on for this task. If true, execute command functionality is turned on all the containers in the task. This allows you to use ECS Exec to access containers interactively. When enabled, a job role with required SSM permissions will be created automatically if no job role is provided. If a job role is alreadyprovided, the required permissions will be added to it. > [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* no environment variables The environment variables to pass to a container. Cannot start with `AWS_BATCH`. We don't recommend using plaintext environment variables for sensitive information, such as credential data. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* a Role will be created The role used by Amazon ECS container and AWS Fargate agents to make AWS API calls on your behalf. > [https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html](https://docs.aws.amazon.com/batch/latest/userguide/execution-IAM-role.html) --- ##### `jobRole`^Optional ```typescript public readonly jobRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* no job role The role that the container can assume. > [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) --- ##### `linuxParameters`^Optional ```typescript public readonly linuxParameters: LinuxParameters; ``` - *Type:* aws-cdk-lib.aws_batch.LinuxParameters - *Default:* none Linux-specific modifications that are applied to the container, such as details for device mappings. --- ##### `logging`^Optional ```typescript public readonly logging: LogDriver; ``` - *Type:* aws-cdk-lib.aws_ecs.LogDriver - *Default:* the log configuration of the Docker daemon The loging configuration for this Job. --- ##### `readonlyRootFilesystem`^Optional ```typescript public readonly readonlyRootFilesystem: boolean; ``` - *Type:* boolean - *Default:* false Gives the container readonly access to its root filesystem. --- ##### `secrets`^Optional ```typescript public readonly secrets: {[ key: string ]: Secret}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_batch.Secret} - *Default:* no secrets A map from environment variable names to the secrets for the container. Allows your job definitions to reference the secret by the environment variable name defined in this property. > [https://docs.aws.amazon.com/batch/latest/userguide/specifying-sensitive-data.html](https://docs.aws.amazon.com/batch/latest/userguide/specifying-sensitive-data.html) --- ##### `user`^Optional ```typescript public readonly user: string; ``` - *Type:* string - *Default:* no user The user name to use inside the container. --- ##### `volumes`^Optional ```typescript public readonly volumes: EcsVolume[]; ``` - *Type:* aws-cdk-lib.aws_batch.EcsVolume[] - *Default:* no volumes The volumes to mount to this container. Automatically added to the job definition. --- ##### `gpu`^Optional ```typescript public readonly gpu: number; ``` - *Type:* number - *Default:* no gpus The number of physical GPUs to reserve for the container. Make sure that the number of GPUs reserved for all containers in a job doesn't exceed the number of available GPUs on the compute resource that the job is launched on. --- ##### `privileged`^Optional ```typescript public readonly privileged: boolean; ``` - *Type:* boolean - *Default:* false When this parameter is true, the container is given elevated permissions on the host container instance (similar to the root user). --- ##### `ulimits`^Optional ```typescript public readonly ulimits: Ulimit[]; ``` - *Type:* aws-cdk-lib.aws_batch.Ulimit[] - *Default:* no ulimits Limits to set for the user this docker container will run as. --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of worker instance. --- ##### `volumeSize`^Required ```typescript public readonly volumeSize: Size; ``` - *Type:* aws-cdk-lib.Size - *Default:* N bilion parameters * 5GiB EBS The root volume of worker instance. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this will launch worker instance. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ### NeuronxCompiledModel The model compiled by Neuronx compiler. #### Initializer ```typescript import { NeuronxCompiledModel } from 'aws-cdk-neuronx-patterns' const neuronxCompiledModel: NeuronxCompiledModel = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | modelName | string | The model name. | | recommendedInstanceType | INeuronxInstanceType | The recommended Neuron instance type for running inference with this compiled model. | | s3Prefix | string | S3 prefix that compiled artifact uploaded. | | s3Uri | string | S3 URL that compiled artifact uploaded. | | weightSize | aws-cdk-lib.Size | The weight size of the model. | --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `modelName`^Required ```typescript public readonly modelName: string; ``` - *Type:* string The model name. --- ##### `recommendedInstanceType`^Required ```typescript public readonly recommendedInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The recommended Neuron instance type for running inference with this compiled model. --- ##### `s3Prefix`^Required ```typescript public readonly s3Prefix: string; ``` - *Type:* string S3 prefix that compiled artifact uploaded. --- ##### `s3Uri`^Required ```typescript public readonly s3Uri: string; ``` - *Type:* string S3 URL that compiled artifact uploaded. --- ##### `weightSize`^Required ```typescript public readonly weightSize: Size; ``` - *Type:* aws-cdk-lib.Size The weight size of the model. --- ### NeuronxCompilerBaseProps Common props for NeuronxCompilerBase. #### Initializer ```typescript import { NeuronxCompilerBaseProps } from 'aws-cdk-neuronx-patterns' const neuronxCompilerBaseProps: NeuronxCompilerBaseProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | artifactS3Prefix | string | S3 Prefix that compiled artifact uploaded. | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | image | INeuronxContainerImage | An image of the container where the compile job is executed. | | model | Model | The model to be compiled. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this will launch compile worker instance. | | command | string[] | The command to run in the container. | | environment | {[ key: string ]: string} | The environment variables to pass to the container. | | secrets | {[ key: string ]: aws-cdk-lib.aws_batch.Secret} | Secrets to pass to the container. | | spot | boolean | Whether or not to use spot instances. | | volumeSize | aws-cdk-lib.Size | The root volume of worker instance. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | --- ##### `artifactS3Prefix`^Required ```typescript public readonly artifactS3Prefix: string; ``` - *Type:* string S3 Prefix that compiled artifact uploaded. This property is not depends on compile job finish. --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `image`^Required ```typescript public readonly image: INeuronxContainerImage; ``` - *Type:* INeuronxContainerImage An image of the container where the compile job is executed. --- ##### `model`^Required ```typescript public readonly model: Model; ``` - *Type:* Model The model to be compiled. --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this will launch compile worker instance. --- ##### `command`^Optional ```typescript public readonly command: string[]; ``` - *Type:* string[] The command to run in the container. --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables. The environment variables to pass to the container. This is only applicable when using container runtime. --- ##### `secrets`^Optional ```typescript public readonly secrets: {[ key: string ]: Secret}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_batch.Secret} Secrets to pass to the container. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `volumeSize`^Optional ```typescript public readonly volumeSize: Size; ``` - *Type:* aws-cdk-lib.Size - *Default:* N billion parameters * 5GiB EBS The root volume of worker instance. --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ### NeuronxCrossCompilerProps Props of NeuronxCrossCompiler. #### Initializer ```typescript import { NeuronxCrossCompilerProps } from 'aws-cdk-neuronx-patterns' const neuronxCrossCompilerProps: NeuronxCrossCompilerProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | artifactS3Prefix | string | S3 Prefix that compiled artifact uploaded. | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | image | INeuronxContainerImage | An image of the container where the compile job is executed. | | model | Model | The model to be compiled. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this will launch compile worker instance. | | command | string[] | The command to run in the container. | | environment | {[ key: string ]: string} | The environment variables to pass to the container. | | secrets | {[ key: string ]: aws-cdk-lib.aws_batch.Secret} | Secrets to pass to the container. | | spot | boolean | Whether or not to use spot instances. | | volumeSize | aws-cdk-lib.Size | The root volume of worker instance. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | | compileInstanceType | aws-cdk-lib.aws_ec2.InstanceType | The EC2 instance type to use for cross-compilation. | --- ##### `artifactS3Prefix`^Required ```typescript public readonly artifactS3Prefix: string; ``` - *Type:* string S3 Prefix that compiled artifact uploaded. This property is not depends on compile job finish. --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `image`^Required ```typescript public readonly image: INeuronxContainerImage; ``` - *Type:* INeuronxContainerImage An image of the container where the compile job is executed. --- ##### `model`^Required ```typescript public readonly model: Model; ``` - *Type:* Model The model to be compiled. --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this will launch compile worker instance. --- ##### `command`^Optional ```typescript public readonly command: string[]; ``` - *Type:* string[] The command to run in the container. --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables. The environment variables to pass to the container. This is only applicable when using container runtime. --- ##### `secrets`^Optional ```typescript public readonly secrets: {[ key: string ]: Secret}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_batch.Secret} Secrets to pass to the container. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `volumeSize`^Optional ```typescript public readonly volumeSize: Size; ``` - *Type:* aws-cdk-lib.Size - *Default:* N billion parameters * 5GiB EBS The root volume of worker instance. --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ##### `compileInstanceType`^Optional ```typescript public readonly compileInstanceType: InstanceType; ``` - *Type:* aws-cdk-lib.aws_ec2.InstanceType - *Default:* ec2.InstanceType.of(ec2.InstanceClass.C7I, ec2.InstanceSize.XLARGE4) The EC2 instance type to use for cross-compilation. This should be a non-Neuron instance type with sufficient memory and CPU for model compilation. --- ### NeuronxNativeCompilerProps Props of NeuronxNativeCompiler. #### Initializer ```typescript import { NeuronxNativeCompilerProps } from 'aws-cdk-neuronx-patterns' const neuronxNativeCompilerProps: NeuronxNativeCompilerProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | artifactS3Prefix | string | S3 Prefix that compiled artifact uploaded. | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | image | INeuronxContainerImage | An image of the container where the compile job is executed. | | model | Model | The model to be compiled. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this will launch compile worker instance. | | command | string[] | The command to run in the container. | | environment | {[ key: string ]: string} | The environment variables to pass to the container. | | secrets | {[ key: string ]: aws-cdk-lib.aws_batch.Secret} | Secrets to pass to the container. | | spot | boolean | Whether or not to use spot instances. | | volumeSize | aws-cdk-lib.Size | The root volume of worker instance. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | --- ##### `artifactS3Prefix`^Required ```typescript public readonly artifactS3Prefix: string; ``` - *Type:* string S3 Prefix that compiled artifact uploaded. This property is not depends on compile job finish. --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `image`^Required ```typescript public readonly image: INeuronxContainerImage; ``` - *Type:* INeuronxContainerImage An image of the container where the compile job is executed. --- ##### `model`^Required ```typescript public readonly model: Model; ``` - *Type:* Model The model to be compiled. --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this will launch compile worker instance. --- ##### `command`^Optional ```typescript public readonly command: string[]; ``` - *Type:* string[] The command to run in the container. --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables. The environment variables to pass to the container. This is only applicable when using container runtime. --- ##### `secrets`^Optional ```typescript public readonly secrets: {[ key: string ]: Secret}; ``` - *Type:* {[ key: string ]: aws-cdk-lib.aws_batch.Secret} Secrets to pass to the container. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `volumeSize`^Optional ```typescript public readonly volumeSize: Size; ``` - *Type:* aws-cdk-lib.Size - *Default:* N billion parameters * 5GiB EBS The root volume of worker instance. --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ### NeuronxTaskDefinitionProps #### Initializer ```typescript import { NeuronxTaskDefinitionProps } from 'aws-cdk-neuronx-patterns' const neuronxTaskDefinitionProps: NeuronxTaskDefinitionProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | enableFaultInjection | boolean | Enables fault injection and allows for fault injection requests to be accepted from the task's containers. | | executionRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. | | family | string | The name of a family that this task definition is registered to. | | proxyConfiguration | aws-cdk-lib.aws_ecs.ProxyConfiguration | The configuration details for the App Mesh proxy. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | volumes | aws-cdk-lib.aws_ecs.Volume[] | The list of volume definitions for the task. | | inferenceAccelerators | aws-cdk-lib.aws_ecs.InferenceAccelerator[] | The inference accelerators to use for the containers in the task. | | ipcMode | aws-cdk-lib.aws_ecs.IpcMode | The IPC resource namespace to use for the containers in the task. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The Docker networking mode to use for the containers in the task. | | pidMode | aws-cdk-lib.aws_ecs.PidMode | The process namespace to use for the containers in the task. | | placementConstraints | aws-cdk-lib.aws_ecs.PlacementConstraint[] | An array of placement constraint objects to use for the task. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | tensorParallelSize | number | The number of tensor parallel size. | | compiledModel | NeuronxCompiledModel | The model to be compiled. | --- ##### `enableFaultInjection`^Optional ```typescript public readonly enableFaultInjection: boolean; ``` - *Type:* boolean - *Default:* undefined - ECS default setting is false Enables fault injection and allows for fault injection requests to be accepted from the task's containers. Fault injection only works with tasks using the {@link NetworkMode.AWS_VPC} or {@link NetworkMode.HOST} network modes. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* An execution role will be automatically created if you use ECR images in your task definition. The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. The role will be used to retrieve container images from ECR and create CloudWatch log groups. --- ##### `family`^Optional ```typescript public readonly family: string; ``` - *Type:* string - *Default:* Automatically generated name. The name of a family that this task definition is registered to. A family groups multiple versions of a task definition. --- ##### `proxyConfiguration`^Optional ```typescript public readonly proxyConfiguration: ProxyConfiguration; ``` - *Type:* aws-cdk-lib.aws_ecs.ProxyConfiguration - *Default:* No proxy configuration. The configuration details for the App Mesh proxy. --- ##### `taskRole`^Optional ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A task role is automatically created for you. The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `volumes`^Optional ```typescript public readonly volumes: Volume[]; ``` - *Type:* aws-cdk-lib.aws_ecs.Volume[] - *Default:* No volumes are passed to the Docker daemon on a container instance. The list of volume definitions for the task. For more information, see [Task Definition Parameter Volumes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide//task_definition_parameters.html#volumes). --- ##### `inferenceAccelerators`^Optional ```typescript public readonly inferenceAccelerators: InferenceAccelerator[]; ``` - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator[] - *Default:* No inference accelerators. The inference accelerators to use for the containers in the task. Not supported in Fargate. --- ##### `ipcMode`^Optional ```typescript public readonly ipcMode: IpcMode; ``` - *Type:* aws-cdk-lib.aws_ecs.IpcMode - *Default:* IpcMode used by the task is not specified The IPC resource namespace to use for the containers in the task. Not supported in Fargate and Windows containers. --- ##### `networkMode`^Optional ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode - *Default:* NetworkMode.BRIDGE for EC2 tasks, AWS_VPC for Fargate tasks. The Docker networking mode to use for the containers in the task. The valid values are NONE, BRIDGE, AWS_VPC, and HOST. --- ##### `pidMode`^Optional ```typescript public readonly pidMode: PidMode; ``` - *Type:* aws-cdk-lib.aws_ecs.PidMode - *Default:* PidMode used by the task is not specified The process namespace to use for the containers in the task. Not supported in Windows containers. --- ##### `placementConstraints`^Optional ```typescript public readonly placementConstraints: PlacementConstraint[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint[] - *Default:* No placement constraints. An array of placement constraint objects to use for the task. You can specify a maximum of 10 constraints per task (this limit includes constraints in the task definition and those specified at run time). --- ##### `neuronxInstanceType`^Optional ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `tensorParallelSize`^Optional ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number - *Default:* 1 The number of tensor parallel size. --- ##### `compiledModel`^Required ```typescript public readonly compiledModel: NeuronxCompiledModel; ``` - *Type:* NeuronxCompiledModel The model to be compiled. --- ### NeuronxTaskDefinitionPropsBase #### Initializer ```typescript import { NeuronxTaskDefinitionPropsBase } from 'aws-cdk-neuronx-patterns' const neuronxTaskDefinitionPropsBase: NeuronxTaskDefinitionPropsBase = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | enableFaultInjection | boolean | Enables fault injection and allows for fault injection requests to be accepted from the task's containers. | | executionRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. | | family | string | The name of a family that this task definition is registered to. | | proxyConfiguration | aws-cdk-lib.aws_ecs.ProxyConfiguration | The configuration details for the App Mesh proxy. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | volumes | aws-cdk-lib.aws_ecs.Volume[] | The list of volume definitions for the task. | | inferenceAccelerators | aws-cdk-lib.aws_ecs.InferenceAccelerator[] | The inference accelerators to use for the containers in the task. | | ipcMode | aws-cdk-lib.aws_ecs.IpcMode | The IPC resource namespace to use for the containers in the task. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The Docker networking mode to use for the containers in the task. | | pidMode | aws-cdk-lib.aws_ecs.PidMode | The process namespace to use for the containers in the task. | | placementConstraints | aws-cdk-lib.aws_ecs.PlacementConstraint[] | An array of placement constraint objects to use for the task. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | tensorParallelSize | number | The number of tensor parallel size. | --- ##### `enableFaultInjection`^Optional ```typescript public readonly enableFaultInjection: boolean; ``` - *Type:* boolean - *Default:* undefined - ECS default setting is false Enables fault injection and allows for fault injection requests to be accepted from the task's containers. Fault injection only works with tasks using the {@link NetworkMode.AWS_VPC} or {@link NetworkMode.HOST} network modes. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* An execution role will be automatically created if you use ECR images in your task definition. The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. The role will be used to retrieve container images from ECR and create CloudWatch log groups. --- ##### `family`^Optional ```typescript public readonly family: string; ``` - *Type:* string - *Default:* Automatically generated name. The name of a family that this task definition is registered to. A family groups multiple versions of a task definition. --- ##### `proxyConfiguration`^Optional ```typescript public readonly proxyConfiguration: ProxyConfiguration; ``` - *Type:* aws-cdk-lib.aws_ecs.ProxyConfiguration - *Default:* No proxy configuration. The configuration details for the App Mesh proxy. --- ##### `taskRole`^Optional ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A task role is automatically created for you. The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `volumes`^Optional ```typescript public readonly volumes: Volume[]; ``` - *Type:* aws-cdk-lib.aws_ecs.Volume[] - *Default:* No volumes are passed to the Docker daemon on a container instance. The list of volume definitions for the task. For more information, see [Task Definition Parameter Volumes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide//task_definition_parameters.html#volumes). --- ##### `inferenceAccelerators`^Optional ```typescript public readonly inferenceAccelerators: InferenceAccelerator[]; ``` - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator[] - *Default:* No inference accelerators. The inference accelerators to use for the containers in the task. Not supported in Fargate. --- ##### `ipcMode`^Optional ```typescript public readonly ipcMode: IpcMode; ``` - *Type:* aws-cdk-lib.aws_ecs.IpcMode - *Default:* IpcMode used by the task is not specified The IPC resource namespace to use for the containers in the task. Not supported in Fargate and Windows containers. --- ##### `networkMode`^Optional ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode - *Default:* NetworkMode.BRIDGE for EC2 tasks, AWS_VPC for Fargate tasks. The Docker networking mode to use for the containers in the task. The valid values are NONE, BRIDGE, AWS_VPC, and HOST. --- ##### `pidMode`^Optional ```typescript public readonly pidMode: PidMode; ``` - *Type:* aws-cdk-lib.aws_ecs.PidMode - *Default:* PidMode used by the task is not specified The process namespace to use for the containers in the task. Not supported in Windows containers. --- ##### `placementConstraints`^Optional ```typescript public readonly placementConstraints: PlacementConstraint[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint[] - *Default:* No placement constraints. An array of placement constraint objects to use for the task. You can specify a maximum of 10 constraints per task (this limit includes constraints in the task definition and those specified at run time). --- ##### `neuronxInstanceType`^Optional ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `tensorParallelSize`^Optional ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number - *Default:* 1 The number of tensor parallel size. --- ### VllmCacheConfig Configuration for the KV cache. #### Initializer ```typescript import { VllmCacheConfig } from 'aws-cdk-neuronx-patterns' const vllmCacheConfig: VllmCacheConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | blockSize | BlockSize | Size of a contiguous cache block in number of tokens. | | calculateKvScales | boolean | This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. | | cpuOffloadGb | number | The space in GiB to offload to CPU, per GPU. | | enablePrefixCaching | boolean | Whether to enable prefix caching. | | gpuMemoryUtilization | number | The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. | | kvCacheDtype | KvCacheDtype | Data type for kv cache storage. | | numGpuBlocksOverride | number | Number of GPU blocks to use. | | prefixCachingHashAlgo | PrefixCachingHashAlgo | Set the hash algorithm for prefix caching. | | swapSpace | number | Size of the CPU swap space per GPU (in GiB). | --- ##### `blockSize`^Optional ```typescript public readonly blockSize: BlockSize; ``` - *Type:* BlockSize Size of a contiguous cache block in number of tokens. This is ignored on neuron devices and set to –max-model-len. On CUDA devices, only block sizes up to 32 are supported. On HPU devices, block size defaults to 128. --- ##### `calculateKvScales`^Optional ```typescript public readonly calculateKvScales: boolean; ``` - *Type:* boolean - *Default:* false This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. If False, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. --- ##### `cpuOffloadGb`^Optional ```typescript public readonly cpuOffloadGb: number; ``` - *Type:* number - *Default:* 0 The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. --- ##### `enablePrefixCaching`^Optional ```typescript public readonly enablePrefixCaching: boolean; ``` - *Type:* boolean Whether to enable prefix caching. Disabled by default for V0. Enabled by default for V1. --- ##### `gpuMemoryUtilization`^Optional ```typescript public readonly gpuMemoryUtilization: number; ``` - *Type:* number - *Default:* 0.9 The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory utilization. If unspecified, will use the default value of 0.9. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. --- ##### `kvCacheDtype`^Optional ```typescript public readonly kvCacheDtype: KvCacheDtype; ``` - *Type:* KvCacheDtype - *Default:* KvCacheDtype.AUTO Data type for kv cache storage. If “auto”, will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3). --- ##### `numGpuBlocksOverride`^Optional ```typescript public readonly numGpuBlocksOverride: number; ``` - *Type:* number Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None. Used for testing preemption. --- ##### `prefixCachingHashAlgo`^Optional ```typescript public readonly prefixCachingHashAlgo: PrefixCachingHashAlgo; ``` - *Type:* PrefixCachingHashAlgo - *Default:* PrefixCachingHashAlgo.BUILTIN Set the hash algorithm for prefix caching. --- ##### `swapSpace`^Optional ```typescript public readonly swapSpace: number; ``` - *Type:* number - *Default:* 4 Size of the CPU swap space per GPU (in GiB). --- ### VllmDecodingConfig Dataclass which contains the decoding strategy of the engine. #### Initializer ```typescript import { VllmDecodingConfig } from 'aws-cdk-neuronx-patterns' const vllmDecodingConfig: VllmDecodingConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | guidedDecodingBackend | GuidedDecodingBackend | Which engine will be used for guided decoding (JSON schema / regex etc) by default. | | reasoningParser | ReasoningParser | Select the reasoning parser depending on the model that you’re using. | --- ##### `guidedDecodingBackend`^Optional ```typescript public readonly guidedDecodingBackend: GuidedDecodingBackend; ``` - *Type:* GuidedDecodingBackend - *Default:* GuidedDecodingBackend.AUTO Which engine will be used for guided decoding (JSON schema / regex etc) by default. With “auto”, we will make opinionated choices based on request contents and what the backend libraries currently support, so the behavior is subject to change in each release. --- ##### `reasoningParser`^Optional ```typescript public readonly reasoningParser: ReasoningParser; ``` - *Type:* ReasoningParser Select the reasoning parser depending on the model that you’re using. This is used to parse the reasoning content into OpenAI API format. Required for –enable-reasoning. --- ### VllmDeviceConfig #### Initializer ```typescript import { VllmDeviceConfig } from 'aws-cdk-neuronx-patterns' const vllmDeviceConfig: VllmDeviceConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | device | Device | Device type for vLLM execution. | --- ##### `device`^Optional ```typescript public readonly device: Device; ``` - *Type:* Device - *Default:* Device.AUTO Device type for vLLM execution. --- ### VllmEngineArguments Interface for vLLM server command line arguments. #### Initializer ```typescript import { VllmEngineArguments } from 'aws-cdk-neuronx-patterns' const vllmEngineArguments: VllmEngineArguments = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | additionalConfig | {[ key: string ]: any} | Additional config for specified platform. | | allowCredentials | boolean | Allow credentials. | | allowedHeaders | string[] | Allowed headers. | | allowedLocalMediaPath | string | Allowing API requests to read local images or videos from directories specified by the server file system. | | allowedMethods | string[] | Allowed methods. | | allowedOrigins | string[] | Allowed origins. | | apiKey | string | If provided, the server will require this key to be presented in the header. | | chatTemplate | string | The file path to the chat template, or the template in single-line form for the specified model. | | chatTemplateContentFormat | ChatTemplateContentFormat | The format to render message content within a chat template. | | codeRevision | string | The specific revision to use for the model code on Hugging Face Hub. | | collectDetailedTraces | string | Valid choices are model,worker,all. | | compilationConfig | {[ key: string ]: any} | torch.compile configuration for the model. When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. | | configFormat | ConfigFormat | The format of the model config to load. | | disableAsyncOutputProc | boolean | Disable async output processing. | | disableCascadeAttn | boolean | Disable cascade attention for V1. | | disableFastApiDocs | boolean | Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. | | disableFrontendMultiprocessing | boolean | If specified, will run the OpenAI frontend server in the same process as the model serving engine. | | disableLogRequests | boolean | Disable logging requests. | | disableLogStats | boolean | Disable logging statistics. | | disableMmPreprocessorCache | boolean | If true, then disables caching of the multi-modal preprocessor/mapper. | | disableSlidingWindow | boolean | Disables sliding window, capping to sliding window size. | | disableUvicornAccessLog | boolean | Disable uvicorn access log. | | dtype | DataType | Data type for model weights and activations. | | enableAutoToolChoice | boolean | Enable auto tool choice for supported models. | | enablePromptTokensDetails | boolean | Enable prompt_tokens_details in usage. | | enableReasoning | boolean | Enable reasoning_content for the model. | | enableRequestIdHeaders | boolean | If specified, API server will add X-Request-Id header to responses. | | enableServerLoadTracking | boolean | Enable tracking server_load_metrics in the app state. | | enableSleepMode | boolean | Enable sleep mode for the engine. | | enableSslRefresh | boolean | Refresh SSL Context when SSL certificate files change. | | enforceEager | boolean | Always use eager-mode PyTorch. | | generationConfig | string | The folder path to the generation config. | | hfConfigPath | string | Name or path of the huggingface config to use. | | hfOverrides | {[ key: string ]: any} | Extra arguments for the HuggingFace config. | | hfToken | aws-cdk-lib.aws_batch.Secret | The token to use as HTTP bearer authorization for remote files. | | host | string | Host name. | | ignorePatterns | string[] | The pattern(s) to ignore when loading the model.Default to original/**\/* to avoid repeated loading of llama’s checkpoints. | | kvTransferConfig | {[ key: string ]: any} | Configurations for distributed KV cache transfer in object. | | logitsProcessorPattern | string | Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra completion argument. | | loraModules | {[ key: string ]: any} | LoRA module configurations. | | maxLogLen | number | Max number of prompt characters or prompt ID numbers in log. | | maxLogprobs | number | Max number of log probs to return logprobs is specified in SamplingParams. | | maxModelLen | number | Model context length. | | maxSeqLenToCapture | number | Maximum sequence length covered by CUDA graphs. | | middleware | string[] | Additional ASGI middleware to apply to the app. | | mmProcessorKwargs | {[ key: string ]: any} | Overrides for the multimodal input mapping/processing, e.g., image processor. | | model | string | Name or path of the huggingface model to use. | | modelImpl | ModelImpl | Which implementation of the model to use. | | otlpTracesEndpoint | string | Target URL to which OpenTelemetry traces will be sent. | | overrideGenerationConfig | {[ key: string ]: any} | Overrides or sets generation config. | | overrideNeuronConfig | {[ key: string ]: any} | Override or set neuron device configuration. | | overridePoolerConfig | {[ key: string ]: any} | Override or set the pooling method for pooling models. | | port | number | Port number. | | promptAdapters | string[] | Prompt adapter configurations in the format name=path. | | qloraAdapterNameOrPath | string | Name or path of the QLoRA adapter. | | quantization | Quantization | Method used to quantize the weights. | | responseRole | string | The role name to return if `request.add_generation_prompt=true`. | | returnTokensAsTokenIds | boolean | When `--max-logprobs` is specified, represents single tokens as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.. | | revision | string | The specific model version to use. | | rootPath | string | FastAPI root_path when app is behind a path based routing proxy. | | ropeScaling | {[ key: string ]: any} | RoPE scaling configuration in JSON format. | | ropeTheta | number | RoPE theta. | | schedulerCls | string | The scheduler class to use. | | seed | number | Random seed for operations. | | servedModelName | string[] | The model name(s) used in the API. | | showHiddenMetricsForVersion | string | Enable deprecated Prometheus metrics that have been hidden since the specified version. | | skipTokenizerInit | boolean | Skip initialization of tokenizer and detokenizer. | | sslCaCerts | string | The CA certificates file. | | sslCertfile | string | The file path to the SSL cert file. | | sslCertReqs | number | Whether client certificate is required (see stdlib ssl module's). | | sslKeyfile | string | The file path to the SSL key file. | | task | VllmTask | The task to use the model for. | | tokenizer | string | Name or path of the huggingface tokenizer to use. | | tokenizerMode | TokenizerMode | The tokenizer mode. | | tokenizerRevision | string | Revision of the huggingface tokenizer to use. | | toolCallParser | ToolCallParser | Select the tool call parser depending on the model that you’re using. | | toolParserPlugin | string | Specify the tool parser plugin. | | trustRemoteCode | boolean | Trust remote code from huggingface. | | useV2BlockManager | boolean | Block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. | | uvicornLogLevel | UvicornLogLevel | Log level for uvicorn. | | workerCls | string | The worker class to use for distributed execution. | | workerExtensionCls | string | The worker extension class. | | downloadDir | string | Directory to download and load the weights, default to the default cache directory of Hugging Face. | | loadFormat | LoadFormat | The format of the model weights to load: - “auto” will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. | | modelLoaderExtraConfig | {[ key: string ]: any} | Extra config for model loader. | | useTqdmOnLoad | boolean | Whether to enable tqdm for showing progress bar when loading model weights. | | guidedDecodingBackend | GuidedDecodingBackend | Which engine will be used for guided decoding (JSON schema / regex etc) by default. | | reasoningParser | ReasoningParser | Select the reasoning parser depending on the model that you’re using. | | dataParallelSize | number | Number of data parallel groups. | | disableCustomAllReduce | boolean | Disable the custom all-reduce kernel and fall back to NCCL. | | distributedExecutorBackend | DistributedExecutorBackend | Backend to use for distributed model workers, either “ray” or “mp” (multiprocessing). | | enableExpertParallel | boolean | Use expert parallelism instead of tensor parallelism for MoE layers. | | maxParallelLoadingWorkers | number | Maximum number of parallal loading workers when loading model sequentially in multiple batches. | | pipelineParallelSize | number | Number of pipeline parallel groups. | | rayWorkersUseNsight | boolean | Whether to profile Ray workers with nsight. | | tensorParallelSize | number | Number of tensor parallel groups. | | blockSize | BlockSize | Size of a contiguous cache block in number of tokens. | | calculateKvScales | boolean | This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. | | cpuOffloadGb | number | The space in GiB to offload to CPU, per GPU. | | enablePrefixCaching | boolean | Whether to enable prefix caching. | | gpuMemoryUtilization | number | The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. | | kvCacheDtype | KvCacheDtype | Data type for kv cache storage. | | numGpuBlocksOverride | number | Number of GPU blocks to use. | | prefixCachingHashAlgo | PrefixCachingHashAlgo | Set the hash algorithm for prefix caching. | | swapSpace | number | Size of the CPU swap space per GPU (in GiB). | | limitMmPerPrompt | {[ key: string ]: any} | The maximum number of input items allowed per prompt for each modality. | | enableLora | boolean | If True, enable handling of LoRA adapters. | | enableLoraBias | boolean | If True, enable bias for LoRA adapters. | | fullyShardedLoras | boolean | By default, only half of the LoRA computation is sharded with tensor parallelism. | | longLoraScalingFactors | number | Specify multiple scaling factors (which can be different from base model scaling factorsee eg. | | loraDtype | LoraDtype | Data type for LoRA. | | loraExtraVocabSize | number | Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). | | maxCpuLoras | number | Maximum number of LoRAs to store in CPU memory. | | maxLoraRank | number | Max LoRA rank. | | maxLoras | number | Max number of LoRAs in a single batch. | | enablePromptAdapter | boolean | If True, enable handling of PromptAdapters. | | maxPromptAdapters | number | Max number of PromptAdapters in a batch. | | maxPromptAdapterToken | number | Max number of PromptAdapters tokens. | | device | Device | Device type for vLLM execution. | | speculativeConfig | {[ key: string ]: any} | The configurations for speculative decoding. | | disableChunkedMmInput | boolean | If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. | | enableChunkedPrefill | boolean | If True, prefill requests can be chunked based on the remaining max_num_batched_tokens. | | longPrefillTokenThreshold | number | For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. | | maxLongPartialPrefills | number | For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. | | maxNumBatchedTokens | number | Maximum number of tokens to be processed in a single iteration. | | maxNumPartialPrefills | number | For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. | | maxNumSeqs | number | Maximum number of sequences to be processed in a single iteration. | | multiStepStreamOutputs | boolean | If False, then multi-step will stream outputs at the end of all steps. | | numLookaheadSlots | number | The number of slots to allocate per sequence per step, beyond the known token ids. | | numSchedulerSteps | number | Maximum number of forward steps per scheduler call. | | preemptionMode | PreemptionMode | Whether to perform preemption by swapping or recomputation. | | schedulerDelayFactor | number | Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt. | | schedulingPolicy | SchedulingPolicy | The scheduling policy to use: - “fcfs” means first come first served, i.e. requests are handled in order of arrival. - “priority” means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). | --- ##### `additionalConfig`^Optional ```typescript public readonly additionalConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. The input format is like ‘{“config_key”:”config_value”}’ --- ##### `allowCredentials`^Optional ```typescript public readonly allowCredentials: boolean; ``` - *Type:* boolean - *Default:* false Allow credentials. --- ##### `allowedHeaders`^Optional ```typescript public readonly allowedHeaders: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed headers. --- ##### `allowedLocalMediaPath`^Optional ```typescript public readonly allowedLocalMediaPath: string; ``` - *Type:* string Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. --- ##### `allowedMethods`^Optional ```typescript public readonly allowedMethods: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed methods. --- ##### `allowedOrigins`^Optional ```typescript public readonly allowedOrigins: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed origins. --- ##### `apiKey`^Optional ```typescript public readonly apiKey: string; ``` - *Type:* string If provided, the server will require this key to be presented in the header. --- ##### `chatTemplate`^Optional ```typescript public readonly chatTemplate: string; ``` - *Type:* string The file path to the chat template, or the template in single-line form for the specified model. --- ##### `chatTemplateContentFormat`^Optional ```typescript public readonly chatTemplateContentFormat: ChatTemplateContentFormat; ``` - *Type:* ChatTemplateContentFormat - *Default:* ChatTemplateContentFormat.AUTO The format to render message content within a chat template. “string” will render the content as a string. - Example: `"Hello World"` - “openai” will render the content as a list of dictionaries, similar to OpenAI schema. - Example: `[{"type": "text", "text": "Hello world!"}]` --- ##### `codeRevision`^Optional ```typescript public readonly codeRevision: string; ``` - *Type:* string The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `collectDetailedTraces`^Optional ```typescript public readonly collectDetailedTraces: string; ``` - *Type:* string Valid choices are model,worker,all. It makes sense to set this only if --otlp-traces-endpoint is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. --- ##### `compilationConfig`^Optional ```typescript public readonly compilationConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} torch.compile configuration for the model. When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. NOTE: level 0 is the default level without any optimization. level 1 and 2 are for internal testing only. level 3 is the recommended level for production. To specify the full compilation config, use a JSON string, e.g. `{"level": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` Following the convention of traditional compilers, using -O without space is also supported. -O3 is equivalent to -O 3. --- ##### `configFormat`^Optional ```typescript public readonly configFormat: ConfigFormat; ``` - *Type:* ConfigFormat - *Default:* ConfigFormat.AUTO The format of the model config to load. --- ##### `disableAsyncOutputProc`^Optional ```typescript public readonly disableAsyncOutputProc: boolean; ``` - *Type:* boolean - *Default:* false Disable async output processing. This may result in lower performance. --- ##### `disableCascadeAttn`^Optional ```typescript public readonly disableCascadeAttn: boolean; ``` - *Type:* boolean - *Default:* false Disable cascade attention for V1. --- ##### `disableFastApiDocs`^Optional ```typescript public readonly disableFastApiDocs: boolean; ``` - *Type:* boolean - *Default:* false Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. --- ##### `disableFrontendMultiprocessing`^Optional ```typescript public readonly disableFrontendMultiprocessing: boolean; ``` - *Type:* boolean - *Default:* false If specified, will run the OpenAI frontend server in the same process as the model serving engine. --- ##### `disableLogRequests`^Optional ```typescript public readonly disableLogRequests: boolean; ``` - *Type:* boolean - *Default:* false Disable logging requests. --- ##### `disableLogStats`^Optional ```typescript public readonly disableLogStats: boolean; ``` - *Type:* boolean - *Default:* false Disable logging statistics. --- ##### `disableMmPreprocessorCache`^Optional ```typescript public readonly disableMmPreprocessorCache: boolean; ``` - *Type:* boolean - *Default:* false If true, then disables caching of the multi-modal preprocessor/mapper. (not recommended) --- ##### `disableSlidingWindow`^Optional ```typescript public readonly disableSlidingWindow: boolean; ``` - *Type:* boolean - *Default:* false Disables sliding window, capping to sliding window size. --- ##### `disableUvicornAccessLog`^Optional ```typescript public readonly disableUvicornAccessLog: boolean; ``` - *Type:* boolean - *Default:* false Disable uvicorn access log. --- ##### `dtype`^Optional ```typescript public readonly dtype: DataType; ``` - *Type:* DataType - *Default:* DataType.AUTO Data type for model weights and activations. --- ##### `enableAutoToolChoice`^Optional ```typescript public readonly enableAutoToolChoice: boolean; ``` - *Type:* boolean - *Default:* false Enable auto tool choice for supported models. Use `--tool-call-parser` to specify which parser to use. --- ##### `enablePromptTokensDetails`^Optional ```typescript public readonly enablePromptTokensDetails: boolean; ``` - *Type:* boolean - *Default:* false Enable prompt_tokens_details in usage. --- ##### `enableReasoning`^Optional ```typescript public readonly enableReasoning: boolean; ``` - *Type:* boolean - *Default:* false Enable reasoning_content for the model. --- ##### `enableRequestIdHeaders`^Optional ```typescript public readonly enableRequestIdHeaders: boolean; ``` - *Type:* boolean - *Default:* false If specified, API server will add X-Request-Id header to responses. Caution: this hurts performance at high QPS. --- ##### `enableServerLoadTracking`^Optional ```typescript public readonly enableServerLoadTracking: boolean; ``` - *Type:* boolean - *Default:* false Enable tracking server_load_metrics in the app state. --- ##### `enableSleepMode`^Optional ```typescript public readonly enableSleepMode: boolean; ``` - *Type:* boolean - *Default:* false Enable sleep mode for the engine. (only cuda platform is supported) --- ##### `enableSslRefresh`^Optional ```typescript public readonly enableSslRefresh: boolean; ``` - *Type:* boolean - *Default:* false Refresh SSL Context when SSL certificate files change. --- ##### `enforceEager`^Optional ```typescript public readonly enforceEager: boolean; ``` - *Type:* boolean - *Default:* false Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal performance and flexibility. --- ##### `generationConfig`^Optional ```typescript public readonly generationConfig: string; ``` - *Type:* string - *Default:* "auto" The folder path to the generation config. Defaults to ‘auto’, the generation config will be loaded from model path. If set to ‘vllm’, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. --- ##### `hfConfigPath`^Optional ```typescript public readonly hfConfigPath: string; ``` - *Type:* string Name or path of the huggingface config to use. If unspecified, model name or path will be used. --- ##### `hfOverrides`^Optional ```typescript public readonly hfOverrides: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Extra arguments for the HuggingFace config. This should be a object that will be parsed into a dictionary. --- ##### `hfToken`^Optional ```typescript public readonly hfToken: Secret; ``` - *Type:* aws-cdk-lib.aws_batch.Secret The token to use as HTTP bearer authorization for remote files. If provided, the Secret will be passed as HF_TOKEN secret to compile environment. --- ##### `host`^Optional ```typescript public readonly host: string; ``` - *Type:* string Host name. --- ##### `ignorePatterns`^Optional ```typescript public readonly ignorePatterns: string[]; ``` - *Type:* string[] - *Default:* [] The pattern(s) to ignore when loading the model.Default to original/**\/* to avoid repeated loading of llama’s checkpoints. --- ##### `kvTransferConfig`^Optional ```typescript public readonly kvTransferConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Configurations for distributed KV cache transfer in object. --- ##### `logitsProcessorPattern`^Optional ```typescript public readonly logitsProcessorPattern: string; ``` - *Type:* string Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra completion argument. Defaults to None, which allows no processors. --- ##### `loraModules`^Optional ```typescript public readonly loraModules: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} LoRA module configurations. --- *Example* ```typescript {"name": "name", "path": "lora_path", "base_model_name": "id"} ``` ##### `maxLogLen`^Optional ```typescript public readonly maxLogLen: number; ``` - *Type:* number Max number of prompt characters or prompt ID numbers in log. --- ##### `maxLogprobs`^Optional ```typescript public readonly maxLogprobs: number; ``` - *Type:* number - *Default:* 20 Max number of log probs to return logprobs is specified in SamplingParams. --- ##### `maxModelLen`^Optional ```typescript public readonly maxModelLen: number; ``` - *Type:* number Model context length. --- ##### `maxSeqLenToCapture`^Optional ```typescript public readonly maxSeqLenToCapture: number; ``` - *Type:* number - *Default:* 8192 Maximum sequence length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. Additionally for encoder-decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager mode. --- ##### `middleware`^Optional ```typescript public readonly middleware: string[]; ``` - *Type:* string[] - *Default:* [] Additional ASGI middleware to apply to the app. We accept multiple –middleware arguments. The value should be an import path. If a function is provided, vLLM will add it to the server using `@app.middleware('http')`. If a class is provided, vLLM will add it to the server using `app.add_middleware()`. --- ##### `mmProcessorKwargs`^Optional ```typescript public readonly mmProcessorKwargs: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Overrides for the multimodal input mapping/processing, e.g., image processor. --- *Example* ```typescript {"num_crops": 4} ``` ##### `model`^Optional ```typescript public readonly model: string; ``` - *Type:* string - *Default:* "facebook/opt-125m" Name or path of the huggingface model to use. --- ##### `modelImpl`^Optional ```typescript public readonly modelImpl: ModelImpl; ``` - *Type:* ModelImpl - *Default:* ModelImpl.AUTO Which implementation of the model to use. --- ##### `otlpTracesEndpoint`^Optional ```typescript public readonly otlpTracesEndpoint: string; ``` - *Type:* string Target URL to which OpenTelemetry traces will be sent. --- ##### `overrideGenerationConfig`^Optional ```typescript public readonly overrideGenerationConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Overrides or sets generation config. If used with –generation-config=auto, the override parameters will be merged with the default config from the model. If generation-config is None, only the override parameters are used. --- *Example* ```typescript {"temperature": 0.5} ``` ##### `overrideNeuronConfig`^Optional ```typescript public readonly overrideNeuronConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Override or set neuron device configuration. --- *Example* ```typescript {"cast_logits_dtype": "bloat16"} ``` ##### `overridePoolerConfig`^Optional ```typescript public readonly overridePoolerConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Override or set the pooling method for pooling models. --- *Example* ```typescript {"pooling_type": "mean", "normalize": false} ``` ##### `port`^Optional ```typescript public readonly port: number; ``` - *Type:* number - *Default:* 8000 Port number. --- ##### `promptAdapters`^Optional ```typescript public readonly promptAdapters: string[]; ``` - *Type:* string[] Prompt adapter configurations in the format name=path. Multiple adapters can be specified. --- ##### `qloraAdapterNameOrPath`^Optional ```typescript public readonly qloraAdapterNameOrPath: string; ``` - *Type:* string Name or path of the QLoRA adapter. --- ##### `quantization`^Optional ```typescript public readonly quantization: Quantization; ``` - *Type:* Quantization Method used to quantize the weights. If None, we first check the quantization_config attribute in the model config file. If that is None, we assume the model weights are not quantized and use dtype to determine the data type of the weights. --- ##### `responseRole`^Optional ```typescript public readonly responseRole: string; ``` - *Type:* string - *Default:* "assistant" The role name to return if `request.add_generation_prompt=true`. --- ##### `returnTokensAsTokenIds`^Optional ```typescript public readonly returnTokensAsTokenIds: boolean; ``` - *Type:* boolean - *Default:* false When `--max-logprobs` is specified, represents single tokens as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.. --- ##### `revision`^Optional ```typescript public readonly revision: string; ``` - *Type:* string The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `rootPath`^Optional ```typescript public readonly rootPath: string; ``` - *Type:* string FastAPI root_path when app is behind a path based routing proxy. --- ##### `ropeScaling`^Optional ```typescript public readonly ropeScaling: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} RoPE scaling configuration in JSON format. --- *Example* ```typescript {"rope_type":"dynamic","factor":2.0} ``` ##### `ropeTheta`^Optional ```typescript public readonly ropeTheta: number; ``` - *Type:* number RoPE theta. Use with rope_scaling. In some cases, changing the RoPE theta improves the performance of the scaled model. --- ##### `schedulerCls`^Optional ```typescript public readonly schedulerCls: string; ``` - *Type:* string - *Default:* "vllm.core.scheduler.Scheduler" The scheduler class to use. --- ##### `seed`^Optional ```typescript public readonly seed: number; ``` - *Type:* number Random seed for operations. --- ##### `servedModelName`^Optional ```typescript public readonly servedModelName: string[]; ``` - *Type:* string[] The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in model_name tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. --- ##### `showHiddenMetricsForVersion`^Optional ```typescript public readonly showHiddenMetricsForVersion: string; ``` - *Type:* string Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use –show-hidden-metrics-for-version=0.7 as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. --- ##### `skipTokenizerInit`^Optional ```typescript public readonly skipTokenizerInit: boolean; ``` - *Type:* boolean - *Default:* false Skip initialization of tokenizer and detokenizer. Expects valid prompt_token_ids and None for prompt from the input. The generated output will contain token ids. --- ##### `sslCaCerts`^Optional ```typescript public readonly sslCaCerts: string; ``` - *Type:* string The CA certificates file. --- ##### `sslCertfile`^Optional ```typescript public readonly sslCertfile: string; ``` - *Type:* string The file path to the SSL cert file. --- ##### `sslCertReqs`^Optional ```typescript public readonly sslCertReqs: number; ``` - *Type:* number - *Default:* 0 Whether client certificate is required (see stdlib ssl module's). --- ##### `sslKeyfile`^Optional ```typescript public readonly sslKeyfile: string; ``` - *Type:* string The file path to the SSL key file. --- ##### `task`^Optional ```typescript public readonly task: VllmTask; ``` - *Type:* VllmTask - *Default:* VllmTask.AUTO The task to use the model for. Each vLLM instance only supports one task, even if the same model can be used for multiple tasks. When the model only supports one task, "auto" can be used to select it; otherwise, you must specify explicitly which task to use. --- ##### `tokenizer`^Optional ```typescript public readonly tokenizer: string; ``` - *Type:* string Name or path of the huggingface tokenizer to use. If unspecified, model name or path will be used. --- ##### `tokenizerMode`^Optional ```typescript public readonly tokenizerMode: TokenizerMode; ``` - *Type:* TokenizerMode - *Default:* TokenizerMode.AUTO The tokenizer mode. --- ##### `tokenizerRevision`^Optional ```typescript public readonly tokenizerRevision: string; ``` - *Type:* string Revision of the huggingface tokenizer to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `toolCallParser`^Optional ```typescript public readonly toolCallParser: ToolCallParser; ``` - *Type:* ToolCallParser Select the tool call parser depending on the model that you’re using. This is used to parse the model-generated tool call into OpenAI API format. Required for `--enable-auto-tool-choice`. --- ##### `toolParserPlugin`^Optional ```typescript public readonly toolParserPlugin: string; ``` - *Type:* string - *Default:* "" Specify the tool parser plugin. --- ##### `trustRemoteCode`^Optional ```typescript public readonly trustRemoteCode: boolean; ``` - *Type:* boolean - *Default:* false Trust remote code from huggingface. --- ##### ~~`useV2BlockManager`~~^Optional - *Deprecated:* Setting this flag to True or False has no effect on vLLM behavior. ```typescript public readonly useV2BlockManager: boolean; ``` - *Type:* boolean - *Default:* true Block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. --- ##### `uvicornLogLevel`^Optional ```typescript public readonly uvicornLogLevel: UvicornLogLevel; ``` - *Type:* UvicornLogLevel - *Default:* UvicornLogLevel.INFO Log level for uvicorn. --- ##### `workerCls`^Optional ```typescript public readonly workerCls: string; ``` - *Type:* string - *Default:* "auto" The worker class to use for distributed execution. --- ##### `workerExtensionCls`^Optional ```typescript public readonly workerExtensionCls: string; ``` - *Type:* string - *Default:* "" The worker extension class. --- ##### `downloadDir`^Optional ```typescript public readonly downloadDir: string; ``` - *Type:* string Directory to download and load the weights, default to the default cache directory of Hugging Face. --- ##### `loadFormat`^Optional ```typescript public readonly loadFormat: LoadFormat; ``` - *Type:* LoadFormat - *Default:* LoadFormat.AUTO The format of the model weights to load: - “auto” will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. “pt” will load the weights in the pytorch bin format. - “safetensors” will load the weights in the safetensors format. - “npcache” will load the weights in pytorch format and store a numpy cache to speed up the loading. - “dummy” will initialize the weights with random values, which is mainly for profiling. - “tensorizer” will use CoreWeave’s tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - “runai_streamer” will load the Safetensors weights using Run:ai Model Streamer. - “bitsandbytes” will load the weights using bitsandbytes quantization. - “sharded_state” will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - “gguf” will load weights from GGUF format files (details specified in ggml-org/ggml). - “mistral” will load weights from consolidated safetensors files used by Mistral models. --- ##### `modelLoaderExtraConfig`^Optional ```typescript public readonly modelLoaderExtraConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* {} Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. This should be a object that will be parsed into a dictionary. --- ##### `useTqdmOnLoad`^Optional ```typescript public readonly useTqdmOnLoad: boolean; ``` - *Type:* boolean - *Default:* true Whether to enable tqdm for showing progress bar when loading model weights. --- ##### `guidedDecodingBackend`^Optional ```typescript public readonly guidedDecodingBackend: GuidedDecodingBackend; ``` - *Type:* GuidedDecodingBackend - *Default:* GuidedDecodingBackend.AUTO Which engine will be used for guided decoding (JSON schema / regex etc) by default. With “auto”, we will make opinionated choices based on request contents and what the backend libraries currently support, so the behavior is subject to change in each release. --- ##### `reasoningParser`^Optional ```typescript public readonly reasoningParser: ReasoningParser; ``` - *Type:* ReasoningParser Select the reasoning parser depending on the model that you’re using. This is used to parse the reasoning content into OpenAI API format. Required for –enable-reasoning. --- ##### `dataParallelSize`^Optional ```typescript public readonly dataParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. --- ##### `disableCustomAllReduce`^Optional ```typescript public readonly disableCustomAllReduce: boolean; ``` - *Type:* boolean - *Default:* false Disable the custom all-reduce kernel and fall back to NCCL. --- ##### `distributedExecutorBackend`^Optional ```typescript public readonly distributedExecutorBackend: DistributedExecutorBackend; ``` - *Type:* DistributedExecutorBackend Backend to use for distributed model workers, either “ray” or “mp” (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is less than or equal to the number of GPUs available, “mp” will be used to keep processing on a single host. Otherwise, this will default to “ray” if Ray is installed and fail otherwise. Note that tpu and hpu only support Ray for distributed inference. --- ##### `enableExpertParallel`^Optional ```typescript public readonly enableExpertParallel: boolean; ``` - *Type:* boolean - *Default:* false Use expert parallelism instead of tensor parallelism for MoE layers. --- ##### `maxParallelLoadingWorkers`^Optional ```typescript public readonly maxParallelLoadingWorkers: number; ``` - *Type:* number Maximum number of parallal loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. --- ##### `pipelineParallelSize`^Optional ```typescript public readonly pipelineParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of pipeline parallel groups. --- ##### `rayWorkersUseNsight`^Optional ```typescript public readonly rayWorkersUseNsight: boolean; ``` - *Type:* boolean - *Default:* false Whether to profile Ray workers with nsight. > [https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler](https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler) --- ##### `tensorParallelSize`^Optional ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of tensor parallel groups. --- ##### `blockSize`^Optional ```typescript public readonly blockSize: BlockSize; ``` - *Type:* BlockSize Size of a contiguous cache block in number of tokens. This is ignored on neuron devices and set to –max-model-len. On CUDA devices, only block sizes up to 32 are supported. On HPU devices, block size defaults to 128. --- ##### `calculateKvScales`^Optional ```typescript public readonly calculateKvScales: boolean; ``` - *Type:* boolean - *Default:* false This enables dynamic calculation of k_scale and v_scale when kv_cache_dtype is fp8. If False, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. --- ##### `cpuOffloadGb`^Optional ```typescript public readonly cpuOffloadGb: number; ``` - *Type:* number - *Default:* 0 The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. --- ##### `enablePrefixCaching`^Optional ```typescript public readonly enablePrefixCaching: boolean; ``` - *Type:* boolean Whether to enable prefix caching. Disabled by default for V0. Enabled by default for V1. --- ##### `gpuMemoryUtilization`^Optional ```typescript public readonly gpuMemoryUtilization: number; ``` - *Type:* number - *Default:* 0.9 The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50% GPU memory utilization. If unspecified, will use the default value of 0.9. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. --- ##### `kvCacheDtype`^Optional ```typescript public readonly kvCacheDtype: KvCacheDtype; ``` - *Type:* KvCacheDtype - *Default:* KvCacheDtype.AUTO Data type for kv cache storage. If “auto”, will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3). --- ##### `numGpuBlocksOverride`^Optional ```typescript public readonly numGpuBlocksOverride: number; ``` - *Type:* number Number of GPU blocks to use. This overrides the profiled num_gpu_blocks if specified. Does nothing if None. Used for testing preemption. --- ##### `prefixCachingHashAlgo`^Optional ```typescript public readonly prefixCachingHashAlgo: PrefixCachingHashAlgo; ``` - *Type:* PrefixCachingHashAlgo - *Default:* PrefixCachingHashAlgo.BUILTIN Set the hash algorithm for prefix caching. --- ##### `swapSpace`^Optional ```typescript public readonly swapSpace: number; ``` - *Type:* number - *Default:* 4 Size of the CPU swap space per GPU (in GiB). --- ##### `limitMmPerPrompt`^Optional ```typescript public readonly limitMmPerPrompt: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* {} The maximum number of input items allowed per prompt for each modality. This should be a object that will be parsed into a dictionary. Defaults to 1 (V0) or 999 (V1) for each modality. --- ##### `enableLora`^Optional ```typescript public readonly enableLora: boolean; ``` - *Type:* boolean - *Default:* false If True, enable handling of LoRA adapters. --- ##### `enableLoraBias`^Optional ```typescript public readonly enableLoraBias: boolean; ``` - *Type:* boolean - *Default:* false If True, enable bias for LoRA adapters. --- ##### `fullyShardedLoras`^Optional ```typescript public readonly fullyShardedLoras: boolean; ``` - *Type:* boolean - *Default:* false By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. --- ##### `longLoraScalingFactors`^Optional ```typescript public readonly longLoraScalingFactors: number; ``` - *Type:* number Specify multiple scaling factors (which can be different from base model scaling factorsee eg. Long LoRA) to allow for multiple LoRA adapters trained with those scaling factors to be used at the same time. If not specified, only adapters trained with the base model scaling factor are allowed. --- ##### `loraDtype`^Optional ```typescript public readonly loraDtype: LoraDtype; ``` - *Type:* LoraDtype - *Default:* LoraDtype.AUTO Data type for LoRA. If auto, will default to base model dtype. --- ##### `loraExtraVocabSize`^Optional ```typescript public readonly loraExtraVocabSize: number; ``` - *Type:* number - *Default:* 256 Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). --- ##### `maxCpuLoras`^Optional ```typescript public readonly maxCpuLoras: number; ``` - *Type:* number Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras. --- ##### `maxLoraRank`^Optional ```typescript public readonly maxLoraRank: number; ``` - *Type:* number - *Default:* 16 Max LoRA rank. --- ##### `maxLoras`^Optional ```typescript public readonly maxLoras: number; ``` - *Type:* number - *Default:* 1 Max number of LoRAs in a single batch. --- ##### `enablePromptAdapter`^Optional ```typescript public readonly enablePromptAdapter: boolean; ``` - *Type:* boolean - *Default:* false If True, enable handling of PromptAdapters. --- ##### `maxPromptAdapters`^Optional ```typescript public readonly maxPromptAdapters: number; ``` - *Type:* number - *Default:* 1 Max number of PromptAdapters in a batch. --- ##### `maxPromptAdapterToken`^Optional ```typescript public readonly maxPromptAdapterToken: number; ``` - *Type:* number - *Default:* 0 Max number of PromptAdapters tokens. --- ##### `device`^Optional ```typescript public readonly device: Device; ``` - *Type:* Device - *Default:* Device.AUTO Device type for vLLM execution. --- ##### `speculativeConfig`^Optional ```typescript public readonly speculativeConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} The configurations for speculative decoding. Should be a object. --- ##### `disableChunkedMmInput`^Optional ```typescript public readonly disableChunkedMmInput: boolean; ``` - *Type:* boolean - *Default:* false If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. --- ##### `enableChunkedPrefill`^Optional ```typescript public readonly enableChunkedPrefill: boolean; ``` - *Type:* boolean If True, prefill requests can be chunked based on the remaining max_num_batched_tokens. --- ##### `longPrefillTokenThreshold`^Optional ```typescript public readonly longPrefillTokenThreshold: number; ``` - *Type:* number - *Default:* 0 For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. --- ##### `maxLongPartialPrefills`^Optional ```typescript public readonly maxLongPartialPrefills: number; ``` - *Type:* number - *Default:* 1 For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. Setting this less than max_num_partial_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. --- ##### `maxNumBatchedTokens`^Optional ```typescript public readonly maxNumBatchedTokens: number; ``` - *Type:* number Maximum number of tokens to be processed in a single iteration. This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context. --- ##### `maxNumPartialPrefills`^Optional ```typescript public readonly maxNumPartialPrefills: number; ``` - *Type:* number - *Default:* 1 For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. --- ##### `maxNumSeqs`^Optional ```typescript public readonly maxNumSeqs: number; ``` - *Type:* number Maximum number of sequences to be processed in a single iteration. This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context. --- ##### `multiStepStreamOutputs`^Optional ```typescript public readonly multiStepStreamOutputs: boolean; ``` - *Type:* boolean - *Default:* true If False, then multi-step will stream outputs at the end of all steps. --- ##### `numLookaheadSlots`^Optional ```typescript public readonly numLookaheadSlots: number; ``` - *Type:* number - *Default:* 0 The number of slots to allocate per sequence per step, beyond the known token ids. This is used in speculative decoding to store KV activations of tokens which may or may not be accepted. NOTE: This will be replaced by speculative config in the future; it is present to enable correctness tests until then. --- ##### `numSchedulerSteps`^Optional ```typescript public readonly numSchedulerSteps: number; ``` - *Type:* number - *Default:* 1 Maximum number of forward steps per scheduler call. --- ##### `preemptionMode`^Optional ```typescript public readonly preemptionMode: PreemptionMode; ``` - *Type:* PreemptionMode Whether to perform preemption by swapping or recomputation. If not specified, we determine the mode as follows: We use recomputation by default since it incurs lower overhead than swapping. However, when the sequence group has multiple sequences (e.g., beam search), recomputation is not currently supported. In such a case, we use swapping instead. --- ##### `schedulerDelayFactor`^Optional ```typescript public readonly schedulerDelayFactor: number; ``` - *Type:* number - *Default:* 0.0 Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt. --- ##### `schedulingPolicy`^Optional ```typescript public readonly schedulingPolicy: SchedulingPolicy; ``` - *Type:* SchedulingPolicy - *Default:* SchedulingPolicy.FCFS The scheduling policy to use: - “fcfs” means first come first served, i.e. requests are handled in order of arrival. - “priority” means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). --- ### VllmLoadConfig Configuration for loading the model weights. #### Initializer ```typescript import { VllmLoadConfig } from 'aws-cdk-neuronx-patterns' const vllmLoadConfig: VllmLoadConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | downloadDir | string | Directory to download and load the weights, default to the default cache directory of Hugging Face. | | loadFormat | LoadFormat | The format of the model weights to load: - “auto” will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. | | modelLoaderExtraConfig | {[ key: string ]: any} | Extra config for model loader. | | useTqdmOnLoad | boolean | Whether to enable tqdm for showing progress bar when loading model weights. | --- ##### `downloadDir`^Optional ```typescript public readonly downloadDir: string; ``` - *Type:* string Directory to download and load the weights, default to the default cache directory of Hugging Face. --- ##### `loadFormat`^Optional ```typescript public readonly loadFormat: LoadFormat; ``` - *Type:* LoadFormat - *Default:* LoadFormat.AUTO The format of the model weights to load: - “auto” will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. “pt” will load the weights in the pytorch bin format. - “safetensors” will load the weights in the safetensors format. - “npcache” will load the weights in pytorch format and store a numpy cache to speed up the loading. - “dummy” will initialize the weights with random values, which is mainly for profiling. - “tensorizer” will use CoreWeave’s tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - “runai_streamer” will load the Safetensors weights using Run:ai Model Streamer. - “bitsandbytes” will load the weights using bitsandbytes quantization. - “sharded_state” will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - “gguf” will load weights from GGUF format files (details specified in ggml-org/ggml). - “mistral” will load weights from consolidated safetensors files used by Mistral models. --- ##### `modelLoaderExtraConfig`^Optional ```typescript public readonly modelLoaderExtraConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* {} Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. This should be a object that will be parsed into a dictionary. --- ##### `useTqdmOnLoad`^Optional ```typescript public readonly useTqdmOnLoad: boolean; ``` - *Type:* boolean - *Default:* true Whether to enable tqdm for showing progress bar when loading model weights. --- ### VllmLoraConfig Configuration for LoRA. #### Initializer ```typescript import { VllmLoraConfig } from 'aws-cdk-neuronx-patterns' const vllmLoraConfig: VllmLoraConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | enableLora | boolean | If True, enable handling of LoRA adapters. | | enableLoraBias | boolean | If True, enable bias for LoRA adapters. | | fullyShardedLoras | boolean | By default, only half of the LoRA computation is sharded with tensor parallelism. | | longLoraScalingFactors | number | Specify multiple scaling factors (which can be different from base model scaling factorsee eg. | | loraDtype | LoraDtype | Data type for LoRA. | | loraExtraVocabSize | number | Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). | | maxCpuLoras | number | Maximum number of LoRAs to store in CPU memory. | | maxLoraRank | number | Max LoRA rank. | | maxLoras | number | Max number of LoRAs in a single batch. | --- ##### `enableLora`^Optional ```typescript public readonly enableLora: boolean; ``` - *Type:* boolean - *Default:* false If True, enable handling of LoRA adapters. --- ##### `enableLoraBias`^Optional ```typescript public readonly enableLoraBias: boolean; ``` - *Type:* boolean - *Default:* false If True, enable bias for LoRA adapters. --- ##### `fullyShardedLoras`^Optional ```typescript public readonly fullyShardedLoras: boolean; ``` - *Type:* boolean - *Default:* false By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. --- ##### `longLoraScalingFactors`^Optional ```typescript public readonly longLoraScalingFactors: number; ``` - *Type:* number Specify multiple scaling factors (which can be different from base model scaling factorsee eg. Long LoRA) to allow for multiple LoRA adapters trained with those scaling factors to be used at the same time. If not specified, only adapters trained with the base model scaling factor are allowed. --- ##### `loraDtype`^Optional ```typescript public readonly loraDtype: LoraDtype; ``` - *Type:* LoraDtype - *Default:* LoraDtype.AUTO Data type for LoRA. If auto, will default to base model dtype. --- ##### `loraExtraVocabSize`^Optional ```typescript public readonly loraExtraVocabSize: number; ``` - *Type:* number - *Default:* 256 Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). --- ##### `maxCpuLoras`^Optional ```typescript public readonly maxCpuLoras: number; ``` - *Type:* number Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras. --- ##### `maxLoraRank`^Optional ```typescript public readonly maxLoraRank: number; ``` - *Type:* number - *Default:* 16 Max LoRA rank. --- ##### `maxLoras`^Optional ```typescript public readonly maxLoras: number; ``` - *Type:* number - *Default:* 1 Max number of LoRAs in a single batch. --- ### VllmMultiModalConfig Controls the behavior of multimodal models. #### Initializer ```typescript import { VllmMultiModalConfig } from 'aws-cdk-neuronx-patterns' const vllmMultiModalConfig: VllmMultiModalConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | limitMmPerPrompt | {[ key: string ]: any} | The maximum number of input items allowed per prompt for each modality. | --- ##### `limitMmPerPrompt`^Optional ```typescript public readonly limitMmPerPrompt: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} - *Default:* {} The maximum number of input items allowed per prompt for each modality. This should be a object that will be parsed into a dictionary. Defaults to 1 (V0) or 999 (V1) for each modality. --- ### VllmNamedArguments VllmNamedArguments. #### Initializer ```typescript import { VllmNamedArguments } from 'aws-cdk-neuronx-patterns' const vllmNamedArguments: VllmNamedArguments = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | additionalConfig | {[ key: string ]: any} | Additional config for specified platform. | | allowCredentials | boolean | Allow credentials. | | allowedHeaders | string[] | Allowed headers. | | allowedLocalMediaPath | string | Allowing API requests to read local images or videos from directories specified by the server file system. | | allowedMethods | string[] | Allowed methods. | | allowedOrigins | string[] | Allowed origins. | | apiKey | string | If provided, the server will require this key to be presented in the header. | | chatTemplate | string | The file path to the chat template, or the template in single-line form for the specified model. | | chatTemplateContentFormat | ChatTemplateContentFormat | The format to render message content within a chat template. | | codeRevision | string | The specific revision to use for the model code on Hugging Face Hub. | | collectDetailedTraces | string | Valid choices are model,worker,all. | | compilationConfig | {[ key: string ]: any} | torch.compile configuration for the model. When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. | | configFormat | ConfigFormat | The format of the model config to load. | | disableAsyncOutputProc | boolean | Disable async output processing. | | disableCascadeAttn | boolean | Disable cascade attention for V1. | | disableFastApiDocs | boolean | Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. | | disableFrontendMultiprocessing | boolean | If specified, will run the OpenAI frontend server in the same process as the model serving engine. | | disableLogRequests | boolean | Disable logging requests. | | disableLogStats | boolean | Disable logging statistics. | | disableMmPreprocessorCache | boolean | If true, then disables caching of the multi-modal preprocessor/mapper. | | disableSlidingWindow | boolean | Disables sliding window, capping to sliding window size. | | disableUvicornAccessLog | boolean | Disable uvicorn access log. | | dtype | DataType | Data type for model weights and activations. | | enableAutoToolChoice | boolean | Enable auto tool choice for supported models. | | enablePromptTokensDetails | boolean | Enable prompt_tokens_details in usage. | | enableReasoning | boolean | Enable reasoning_content for the model. | | enableRequestIdHeaders | boolean | If specified, API server will add X-Request-Id header to responses. | | enableServerLoadTracking | boolean | Enable tracking server_load_metrics in the app state. | | enableSleepMode | boolean | Enable sleep mode for the engine. | | enableSslRefresh | boolean | Refresh SSL Context when SSL certificate files change. | | enforceEager | boolean | Always use eager-mode PyTorch. | | generationConfig | string | The folder path to the generation config. | | hfConfigPath | string | Name or path of the huggingface config to use. | | hfOverrides | {[ key: string ]: any} | Extra arguments for the HuggingFace config. | | hfToken | aws-cdk-lib.aws_batch.Secret | The token to use as HTTP bearer authorization for remote files. | | host | string | Host name. | | ignorePatterns | string[] | The pattern(s) to ignore when loading the model.Default to original/**\/* to avoid repeated loading of llama’s checkpoints. | | kvTransferConfig | {[ key: string ]: any} | Configurations for distributed KV cache transfer in object. | | logitsProcessorPattern | string | Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra completion argument. | | loraModules | {[ key: string ]: any} | LoRA module configurations. | | maxLogLen | number | Max number of prompt characters or prompt ID numbers in log. | | maxLogprobs | number | Max number of log probs to return logprobs is specified in SamplingParams. | | maxModelLen | number | Model context length. | | maxSeqLenToCapture | number | Maximum sequence length covered by CUDA graphs. | | middleware | string[] | Additional ASGI middleware to apply to the app. | | mmProcessorKwargs | {[ key: string ]: any} | Overrides for the multimodal input mapping/processing, e.g., image processor. | | model | string | Name or path of the huggingface model to use. | | modelImpl | ModelImpl | Which implementation of the model to use. | | otlpTracesEndpoint | string | Target URL to which OpenTelemetry traces will be sent. | | overrideGenerationConfig | {[ key: string ]: any} | Overrides or sets generation config. | | overrideNeuronConfig | {[ key: string ]: any} | Override or set neuron device configuration. | | overridePoolerConfig | {[ key: string ]: any} | Override or set the pooling method for pooling models. | | port | number | Port number. | | promptAdapters | string[] | Prompt adapter configurations in the format name=path. | | qloraAdapterNameOrPath | string | Name or path of the QLoRA adapter. | | quantization | Quantization | Method used to quantize the weights. | | responseRole | string | The role name to return if `request.add_generation_prompt=true`. | | returnTokensAsTokenIds | boolean | When `--max-logprobs` is specified, represents single tokens as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.. | | revision | string | The specific model version to use. | | rootPath | string | FastAPI root_path when app is behind a path based routing proxy. | | ropeScaling | {[ key: string ]: any} | RoPE scaling configuration in JSON format. | | ropeTheta | number | RoPE theta. | | schedulerCls | string | The scheduler class to use. | | seed | number | Random seed for operations. | | servedModelName | string[] | The model name(s) used in the API. | | showHiddenMetricsForVersion | string | Enable deprecated Prometheus metrics that have been hidden since the specified version. | | skipTokenizerInit | boolean | Skip initialization of tokenizer and detokenizer. | | sslCaCerts | string | The CA certificates file. | | sslCertfile | string | The file path to the SSL cert file. | | sslCertReqs | number | Whether client certificate is required (see stdlib ssl module's). | | sslKeyfile | string | The file path to the SSL key file. | | task | VllmTask | The task to use the model for. | | tokenizer | string | Name or path of the huggingface tokenizer to use. | | tokenizerMode | TokenizerMode | The tokenizer mode. | | tokenizerRevision | string | Revision of the huggingface tokenizer to use. | | toolCallParser | ToolCallParser | Select the tool call parser depending on the model that you’re using. | | toolParserPlugin | string | Specify the tool parser plugin. | | trustRemoteCode | boolean | Trust remote code from huggingface. | | useV2BlockManager | boolean | Block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. | | uvicornLogLevel | UvicornLogLevel | Log level for uvicorn. | | workerCls | string | The worker class to use for distributed execution. | | workerExtensionCls | string | The worker extension class. | --- ##### `additionalConfig`^Optional ```typescript public readonly additionalConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. The input format is like ‘{“config_key”:”config_value”}’ --- ##### `allowCredentials`^Optional ```typescript public readonly allowCredentials: boolean; ``` - *Type:* boolean - *Default:* false Allow credentials. --- ##### `allowedHeaders`^Optional ```typescript public readonly allowedHeaders: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed headers. --- ##### `allowedLocalMediaPath`^Optional ```typescript public readonly allowedLocalMediaPath: string; ``` - *Type:* string Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. --- ##### `allowedMethods`^Optional ```typescript public readonly allowedMethods: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed methods. --- ##### `allowedOrigins`^Optional ```typescript public readonly allowedOrigins: string[]; ``` - *Type:* string[] - *Default:* ['*'] Allowed origins. --- ##### `apiKey`^Optional ```typescript public readonly apiKey: string; ``` - *Type:* string If provided, the server will require this key to be presented in the header. --- ##### `chatTemplate`^Optional ```typescript public readonly chatTemplate: string; ``` - *Type:* string The file path to the chat template, or the template in single-line form for the specified model. --- ##### `chatTemplateContentFormat`^Optional ```typescript public readonly chatTemplateContentFormat: ChatTemplateContentFormat; ``` - *Type:* ChatTemplateContentFormat - *Default:* ChatTemplateContentFormat.AUTO The format to render message content within a chat template. “string” will render the content as a string. - Example: `"Hello World"` - “openai” will render the content as a list of dictionaries, similar to OpenAI schema. - Example: `[{"type": "text", "text": "Hello world!"}]` --- ##### `codeRevision`^Optional ```typescript public readonly codeRevision: string; ``` - *Type:* string The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `collectDetailedTraces`^Optional ```typescript public readonly collectDetailedTraces: string; ``` - *Type:* string Valid choices are model,worker,all. It makes sense to set this only if --otlp-traces-endpoint is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. --- ##### `compilationConfig`^Optional ```typescript public readonly compilationConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} torch.compile configuration for the model. When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. NOTE: level 0 is the default level without any optimization. level 1 and 2 are for internal testing only. level 3 is the recommended level for production. To specify the full compilation config, use a JSON string, e.g. `{"level": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` Following the convention of traditional compilers, using -O without space is also supported. -O3 is equivalent to -O 3. --- ##### `configFormat`^Optional ```typescript public readonly configFormat: ConfigFormat; ``` - *Type:* ConfigFormat - *Default:* ConfigFormat.AUTO The format of the model config to load. --- ##### `disableAsyncOutputProc`^Optional ```typescript public readonly disableAsyncOutputProc: boolean; ``` - *Type:* boolean - *Default:* false Disable async output processing. This may result in lower performance. --- ##### `disableCascadeAttn`^Optional ```typescript public readonly disableCascadeAttn: boolean; ``` - *Type:* boolean - *Default:* false Disable cascade attention for V1. --- ##### `disableFastApiDocs`^Optional ```typescript public readonly disableFastApiDocs: boolean; ``` - *Type:* boolean - *Default:* false Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. --- ##### `disableFrontendMultiprocessing`^Optional ```typescript public readonly disableFrontendMultiprocessing: boolean; ``` - *Type:* boolean - *Default:* false If specified, will run the OpenAI frontend server in the same process as the model serving engine. --- ##### `disableLogRequests`^Optional ```typescript public readonly disableLogRequests: boolean; ``` - *Type:* boolean - *Default:* false Disable logging requests. --- ##### `disableLogStats`^Optional ```typescript public readonly disableLogStats: boolean; ``` - *Type:* boolean - *Default:* false Disable logging statistics. --- ##### `disableMmPreprocessorCache`^Optional ```typescript public readonly disableMmPreprocessorCache: boolean; ``` - *Type:* boolean - *Default:* false If true, then disables caching of the multi-modal preprocessor/mapper. (not recommended) --- ##### `disableSlidingWindow`^Optional ```typescript public readonly disableSlidingWindow: boolean; ``` - *Type:* boolean - *Default:* false Disables sliding window, capping to sliding window size. --- ##### `disableUvicornAccessLog`^Optional ```typescript public readonly disableUvicornAccessLog: boolean; ``` - *Type:* boolean - *Default:* false Disable uvicorn access log. --- ##### `dtype`^Optional ```typescript public readonly dtype: DataType; ``` - *Type:* DataType - *Default:* DataType.AUTO Data type for model weights and activations. --- ##### `enableAutoToolChoice`^Optional ```typescript public readonly enableAutoToolChoice: boolean; ``` - *Type:* boolean - *Default:* false Enable auto tool choice for supported models. Use `--tool-call-parser` to specify which parser to use. --- ##### `enablePromptTokensDetails`^Optional ```typescript public readonly enablePromptTokensDetails: boolean; ``` - *Type:* boolean - *Default:* false Enable prompt_tokens_details in usage. --- ##### `enableReasoning`^Optional ```typescript public readonly enableReasoning: boolean; ``` - *Type:* boolean - *Default:* false Enable reasoning_content for the model. --- ##### `enableRequestIdHeaders`^Optional ```typescript public readonly enableRequestIdHeaders: boolean; ``` - *Type:* boolean - *Default:* false If specified, API server will add X-Request-Id header to responses. Caution: this hurts performance at high QPS. --- ##### `enableServerLoadTracking`^Optional ```typescript public readonly enableServerLoadTracking: boolean; ``` - *Type:* boolean - *Default:* false Enable tracking server_load_metrics in the app state. --- ##### `enableSleepMode`^Optional ```typescript public readonly enableSleepMode: boolean; ``` - *Type:* boolean - *Default:* false Enable sleep mode for the engine. (only cuda platform is supported) --- ##### `enableSslRefresh`^Optional ```typescript public readonly enableSslRefresh: boolean; ``` - *Type:* boolean - *Default:* false Refresh SSL Context when SSL certificate files change. --- ##### `enforceEager`^Optional ```typescript public readonly enforceEager: boolean; ``` - *Type:* boolean - *Default:* false Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal performance and flexibility. --- ##### `generationConfig`^Optional ```typescript public readonly generationConfig: string; ``` - *Type:* string - *Default:* "auto" The folder path to the generation config. Defaults to ‘auto’, the generation config will be loaded from model path. If set to ‘vllm’, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. --- ##### `hfConfigPath`^Optional ```typescript public readonly hfConfigPath: string; ``` - *Type:* string Name or path of the huggingface config to use. If unspecified, model name or path will be used. --- ##### `hfOverrides`^Optional ```typescript public readonly hfOverrides: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Extra arguments for the HuggingFace config. This should be a object that will be parsed into a dictionary. --- ##### `hfToken`^Optional ```typescript public readonly hfToken: Secret; ``` - *Type:* aws-cdk-lib.aws_batch.Secret The token to use as HTTP bearer authorization for remote files. If provided, the Secret will be passed as HF_TOKEN secret to compile environment. --- ##### `host`^Optional ```typescript public readonly host: string; ``` - *Type:* string Host name. --- ##### `ignorePatterns`^Optional ```typescript public readonly ignorePatterns: string[]; ``` - *Type:* string[] - *Default:* [] The pattern(s) to ignore when loading the model.Default to original/**\/* to avoid repeated loading of llama’s checkpoints. --- ##### `kvTransferConfig`^Optional ```typescript public readonly kvTransferConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Configurations for distributed KV cache transfer in object. --- ##### `logitsProcessorPattern`^Optional ```typescript public readonly logitsProcessorPattern: string; ``` - *Type:* string Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra completion argument. Defaults to None, which allows no processors. --- ##### `loraModules`^Optional ```typescript public readonly loraModules: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} LoRA module configurations. --- *Example* ```typescript {"name": "name", "path": "lora_path", "base_model_name": "id"} ``` ##### `maxLogLen`^Optional ```typescript public readonly maxLogLen: number; ``` - *Type:* number Max number of prompt characters or prompt ID numbers in log. --- ##### `maxLogprobs`^Optional ```typescript public readonly maxLogprobs: number; ``` - *Type:* number - *Default:* 20 Max number of log probs to return logprobs is specified in SamplingParams. --- ##### `maxModelLen`^Optional ```typescript public readonly maxModelLen: number; ``` - *Type:* number Model context length. --- ##### `maxSeqLenToCapture`^Optional ```typescript public readonly maxSeqLenToCapture: number; ``` - *Type:* number - *Default:* 8192 Maximum sequence length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. Additionally for encoder-decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager mode. --- ##### `middleware`^Optional ```typescript public readonly middleware: string[]; ``` - *Type:* string[] - *Default:* [] Additional ASGI middleware to apply to the app. We accept multiple –middleware arguments. The value should be an import path. If a function is provided, vLLM will add it to the server using `@app.middleware('http')`. If a class is provided, vLLM will add it to the server using `app.add_middleware()`. --- ##### `mmProcessorKwargs`^Optional ```typescript public readonly mmProcessorKwargs: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Overrides for the multimodal input mapping/processing, e.g., image processor. --- *Example* ```typescript {"num_crops": 4} ``` ##### `model`^Optional ```typescript public readonly model: string; ``` - *Type:* string - *Default:* "facebook/opt-125m" Name or path of the huggingface model to use. --- ##### `modelImpl`^Optional ```typescript public readonly modelImpl: ModelImpl; ``` - *Type:* ModelImpl - *Default:* ModelImpl.AUTO Which implementation of the model to use. --- ##### `otlpTracesEndpoint`^Optional ```typescript public readonly otlpTracesEndpoint: string; ``` - *Type:* string Target URL to which OpenTelemetry traces will be sent. --- ##### `overrideGenerationConfig`^Optional ```typescript public readonly overrideGenerationConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Overrides or sets generation config. If used with –generation-config=auto, the override parameters will be merged with the default config from the model. If generation-config is None, only the override parameters are used. --- *Example* ```typescript {"temperature": 0.5} ``` ##### `overrideNeuronConfig`^Optional ```typescript public readonly overrideNeuronConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Override or set neuron device configuration. --- *Example* ```typescript {"cast_logits_dtype": "bloat16"} ``` ##### `overridePoolerConfig`^Optional ```typescript public readonly overridePoolerConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} Override or set the pooling method for pooling models. --- *Example* ```typescript {"pooling_type": "mean", "normalize": false} ``` ##### `port`^Optional ```typescript public readonly port: number; ``` - *Type:* number - *Default:* 8000 Port number. --- ##### `promptAdapters`^Optional ```typescript public readonly promptAdapters: string[]; ``` - *Type:* string[] Prompt adapter configurations in the format name=path. Multiple adapters can be specified. --- ##### `qloraAdapterNameOrPath`^Optional ```typescript public readonly qloraAdapterNameOrPath: string; ``` - *Type:* string Name or path of the QLoRA adapter. --- ##### `quantization`^Optional ```typescript public readonly quantization: Quantization; ``` - *Type:* Quantization Method used to quantize the weights. If None, we first check the quantization_config attribute in the model config file. If that is None, we assume the model weights are not quantized and use dtype to determine the data type of the weights. --- ##### `responseRole`^Optional ```typescript public readonly responseRole: string; ``` - *Type:* string - *Default:* "assistant" The role name to return if `request.add_generation_prompt=true`. --- ##### `returnTokensAsTokenIds`^Optional ```typescript public readonly returnTokensAsTokenIds: boolean; ``` - *Type:* boolean - *Default:* false When `--max-logprobs` is specified, represents single tokens as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.. --- ##### `revision`^Optional ```typescript public readonly revision: string; ``` - *Type:* string The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `rootPath`^Optional ```typescript public readonly rootPath: string; ``` - *Type:* string FastAPI root_path when app is behind a path based routing proxy. --- ##### `ropeScaling`^Optional ```typescript public readonly ropeScaling: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} RoPE scaling configuration in JSON format. --- *Example* ```typescript {"rope_type":"dynamic","factor":2.0} ``` ##### `ropeTheta`^Optional ```typescript public readonly ropeTheta: number; ``` - *Type:* number RoPE theta. Use with rope_scaling. In some cases, changing the RoPE theta improves the performance of the scaled model. --- ##### `schedulerCls`^Optional ```typescript public readonly schedulerCls: string; ``` - *Type:* string - *Default:* "vllm.core.scheduler.Scheduler" The scheduler class to use. --- ##### `seed`^Optional ```typescript public readonly seed: number; ``` - *Type:* number Random seed for operations. --- ##### `servedModelName`^Optional ```typescript public readonly servedModelName: string[]; ``` - *Type:* string[] The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in model_name tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. --- ##### `showHiddenMetricsForVersion`^Optional ```typescript public readonly showHiddenMetricsForVersion: string; ``` - *Type:* string Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use –show-hidden-metrics-for-version=0.7 as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. --- ##### `skipTokenizerInit`^Optional ```typescript public readonly skipTokenizerInit: boolean; ``` - *Type:* boolean - *Default:* false Skip initialization of tokenizer and detokenizer. Expects valid prompt_token_ids and None for prompt from the input. The generated output will contain token ids. --- ##### `sslCaCerts`^Optional ```typescript public readonly sslCaCerts: string; ``` - *Type:* string The CA certificates file. --- ##### `sslCertfile`^Optional ```typescript public readonly sslCertfile: string; ``` - *Type:* string The file path to the SSL cert file. --- ##### `sslCertReqs`^Optional ```typescript public readonly sslCertReqs: number; ``` - *Type:* number - *Default:* 0 Whether client certificate is required (see stdlib ssl module's). --- ##### `sslKeyfile`^Optional ```typescript public readonly sslKeyfile: string; ``` - *Type:* string The file path to the SSL key file. --- ##### `task`^Optional ```typescript public readonly task: VllmTask; ``` - *Type:* VllmTask - *Default:* VllmTask.AUTO The task to use the model for. Each vLLM instance only supports one task, even if the same model can be used for multiple tasks. When the model only supports one task, "auto" can be used to select it; otherwise, you must specify explicitly which task to use. --- ##### `tokenizer`^Optional ```typescript public readonly tokenizer: string; ``` - *Type:* string Name or path of the huggingface tokenizer to use. If unspecified, model name or path will be used. --- ##### `tokenizerMode`^Optional ```typescript public readonly tokenizerMode: TokenizerMode; ``` - *Type:* TokenizerMode - *Default:* TokenizerMode.AUTO The tokenizer mode. --- ##### `tokenizerRevision`^Optional ```typescript public readonly tokenizerRevision: string; ``` - *Type:* string Revision of the huggingface tokenizer to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --- ##### `toolCallParser`^Optional ```typescript public readonly toolCallParser: ToolCallParser; ``` - *Type:* ToolCallParser Select the tool call parser depending on the model that you’re using. This is used to parse the model-generated tool call into OpenAI API format. Required for `--enable-auto-tool-choice`. --- ##### `toolParserPlugin`^Optional ```typescript public readonly toolParserPlugin: string; ``` - *Type:* string - *Default:* "" Specify the tool parser plugin. --- ##### `trustRemoteCode`^Optional ```typescript public readonly trustRemoteCode: boolean; ``` - *Type:* boolean - *Default:* false Trust remote code from huggingface. --- ##### ~~`useV2BlockManager`~~^Optional - *Deprecated:* Setting this flag to True or False has no effect on vLLM behavior. ```typescript public readonly useV2BlockManager: boolean; ``` - *Type:* boolean - *Default:* true Block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. --- ##### `uvicornLogLevel`^Optional ```typescript public readonly uvicornLogLevel: UvicornLogLevel; ``` - *Type:* UvicornLogLevel - *Default:* UvicornLogLevel.INFO Log level for uvicorn. --- ##### `workerCls`^Optional ```typescript public readonly workerCls: string; ``` - *Type:* string - *Default:* "auto" The worker class to use for distributed execution. --- ##### `workerExtensionCls`^Optional ```typescript public readonly workerExtensionCls: string; ``` - *Type:* string - *Default:* "" The worker extension class. --- ### VllmNxdInferenceCompiledModel The model compiled by Neuronx compiler. #### Initializer ```typescript import { VllmNxdInferenceCompiledModel } from 'aws-cdk-neuronx-patterns' const vllmNxdInferenceCompiledModel: VllmNxdInferenceCompiledModel = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | modelName | string | The model name. | | recommendedInstanceType | INeuronxInstanceType | The recommended Neuron instance type for running inference with this compiled model. | | s3Prefix | string | S3 prefix that compiled artifact uploaded. | | s3Uri | string | S3 URL that compiled artifact uploaded. | | weightSize | aws-cdk-lib.Size | The weight size of the model. | | vllmArgs | VllmEngineArguments | Passed to the vllm engine at compile time. | --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `modelName`^Required ```typescript public readonly modelName: string; ``` - *Type:* string The model name. --- ##### `recommendedInstanceType`^Required ```typescript public readonly recommendedInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The recommended Neuron instance type for running inference with this compiled model. --- ##### `s3Prefix`^Required ```typescript public readonly s3Prefix: string; ``` - *Type:* string S3 prefix that compiled artifact uploaded. --- ##### `s3Uri`^Required ```typescript public readonly s3Uri: string; ``` - *Type:* string S3 URL that compiled artifact uploaded. --- ##### `weightSize`^Required ```typescript public readonly weightSize: Size; ``` - *Type:* aws-cdk-lib.Size The weight size of the model. --- ##### `vllmArgs`^Required ```typescript public readonly vllmArgs: VllmEngineArguments; ``` - *Type:* VllmEngineArguments Passed to the vllm engine at compile time. --- ### VllmNxdInferenceCompileProps Props of VllmNxdInferenceCompiler. #### Initializer ```typescript import { VllmNxdInferenceCompileProps } from 'aws-cdk-neuronx-patterns' const vllmNxdInferenceCompileProps: VllmNxdInferenceCompileProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | bucket | aws-cdk-lib.aws_s3.IBucket | The bucket to upload compiled artifacts. | | model | Model | The model to be compiled. | | vpc | aws-cdk-lib.aws_ec2.IVpc | VPC in which this will launch compile worker instance. | | compileInstanceType | aws-cdk-lib.aws_ec2.InstanceType | The EC2 instance type to use for cross-compilation. | | environment | {[ key: string ]: string} | The environment variables to pass to the container. | | image | INeuronxContainerImage | An image of the container where the compile job is executed. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | spot | boolean | Whether or not to use spot instances. | | vllmArgs | VllmEngineArguments | The arguments to pass to the vllm engine. | | volumeSize | aws-cdk-lib.Size | The root volume of worker instance. | | vpcSubnets | aws-cdk-lib.aws_ec2.SubnetSelection | The VPC Subnets this Compute Environment will launch instances in. | --- ##### `bucket`^Required ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket The bucket to upload compiled artifacts. --- ##### `model`^Required ```typescript public readonly model: Model; ``` - *Type:* Model The model to be compiled. --- ##### `vpc`^Required ```typescript public readonly vpc: IVpc; ``` - *Type:* aws-cdk-lib.aws_ec2.IVpc VPC in which this will launch compile worker instance. --- ##### `compileInstanceType`^Optional ```typescript public readonly compileInstanceType: InstanceType; ``` - *Type:* aws-cdk-lib.aws_ec2.InstanceType - *Default:* Automatically selected based on model size The EC2 instance type to use for cross-compilation. This should be a non-Neuron instance type with sufficient memory for model compilation. --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables. The environment variables to pass to the container. This is only applicable when using container runtime. --- ##### `image`^Optional ```typescript public readonly image: INeuronxContainerImage; ``` - *Type:* INeuronxContainerImage - *Default:* latest image An image of the container where the compile job is executed. --- ##### `neuronxInstanceType`^Optional ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `spot`^Optional ```typescript public readonly spot: boolean; ``` - *Type:* boolean - *Default:* false Whether or not to use spot instances. Spot instances are less expensive EC2 instances that can be reclaimed by EC2 at any time; your job will be given two minutes of notice before reclamation. --- ##### `vllmArgs`^Optional ```typescript public readonly vllmArgs: VllmEngineArguments; ``` - *Type:* VllmEngineArguments - *Default:* no specific values. use default values. The arguments to pass to the vllm engine. --- ##### `volumeSize`^Optional ```typescript public readonly volumeSize: Size; ``` - *Type:* aws-cdk-lib.Size - *Default:* N bilion parameters * 5GiB EBS The root volume of worker instance. --- ##### `vpcSubnets`^Optional ```typescript public readonly vpcSubnets: SubnetSelection; ``` - *Type:* aws-cdk-lib.aws_ec2.SubnetSelection - *Default:* new subnets will be created The VPC Subnets this Compute Environment will launch instances in. --- ### VllmNxdInferenceImageOptions #### Initializer ```typescript import { VllmNxdInferenceImageOptions } from 'aws-cdk-neuronx-patterns' const vllmNxdInferenceImageOptions: VllmNxdInferenceImageOptions = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | vllmGitBranch | string | *No description.* | | vllmGitCommitHash | string | *No description.* | --- ##### `vllmGitBranch`^Optional ```typescript public readonly vllmGitBranch: string; ``` - *Type:* string --- ##### `vllmGitCommitHash`^Optional ```typescript public readonly vllmGitCommitHash: string; ``` - *Type:* string --- ### VllmNxdInferenceTaskDefinitionProps Task definition for VllmNxdInference. #### Initializer ```typescript import { VllmNxdInferenceTaskDefinitionProps } from 'aws-cdk-neuronx-patterns' const vllmNxdInferenceTaskDefinitionProps: VllmNxdInferenceTaskDefinitionProps = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | enableFaultInjection | boolean | Enables fault injection and allows for fault injection requests to be accepted from the task's containers. | | executionRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. | | family | string | The name of a family that this task definition is registered to. | | proxyConfiguration | aws-cdk-lib.aws_ecs.ProxyConfiguration | The configuration details for the App Mesh proxy. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | volumes | aws-cdk-lib.aws_ecs.Volume[] | The list of volume definitions for the task. | | inferenceAccelerators | aws-cdk-lib.aws_ecs.InferenceAccelerator[] | The inference accelerators to use for the containers in the task. | | ipcMode | aws-cdk-lib.aws_ecs.IpcMode | The IPC resource namespace to use for the containers in the task. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The Docker networking mode to use for the containers in the task. | | pidMode | aws-cdk-lib.aws_ecs.PidMode | The process namespace to use for the containers in the task. | | placementConstraints | aws-cdk-lib.aws_ecs.PlacementConstraint[] | An array of placement constraint objects to use for the task. | | neuronxInstanceType | INeuronxInstanceType | The instance type of compile worker instance. | | tensorParallelSize | number | The number of tensor parallel size. | | compiledModel | VllmNxdInferenceCompiledModel | The model to be compiled. | | environment | {[ key: string ]: string} | The environment variables to pass to the container. | | image | VllmNxdInferenceEcsImageBase | The image to be used for the container. | --- ##### `enableFaultInjection`^Optional ```typescript public readonly enableFaultInjection: boolean; ``` - *Type:* boolean - *Default:* undefined - ECS default setting is false Enables fault injection and allows for fault injection requests to be accepted from the task's containers. Fault injection only works with tasks using the {@link NetworkMode.AWS_VPC} or {@link NetworkMode.HOST} network modes. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* An execution role will be automatically created if you use ECR images in your task definition. The name of the IAM task execution role that grants the ECS agent permission to call AWS APIs on your behalf. The role will be used to retrieve container images from ECR and create CloudWatch log groups. --- ##### `family`^Optional ```typescript public readonly family: string; ``` - *Type:* string - *Default:* Automatically generated name. The name of a family that this task definition is registered to. A family groups multiple versions of a task definition. --- ##### `proxyConfiguration`^Optional ```typescript public readonly proxyConfiguration: ProxyConfiguration; ``` - *Type:* aws-cdk-lib.aws_ecs.ProxyConfiguration - *Default:* No proxy configuration. The configuration details for the App Mesh proxy. --- ##### `taskRole`^Optional ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole - *Default:* A task role is automatically created for you. The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `volumes`^Optional ```typescript public readonly volumes: Volume[]; ``` - *Type:* aws-cdk-lib.aws_ecs.Volume[] - *Default:* No volumes are passed to the Docker daemon on a container instance. The list of volume definitions for the task. For more information, see [Task Definition Parameter Volumes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide//task_definition_parameters.html#volumes). --- ##### `inferenceAccelerators`^Optional ```typescript public readonly inferenceAccelerators: InferenceAccelerator[]; ``` - *Type:* aws-cdk-lib.aws_ecs.InferenceAccelerator[] - *Default:* No inference accelerators. The inference accelerators to use for the containers in the task. Not supported in Fargate. --- ##### `ipcMode`^Optional ```typescript public readonly ipcMode: IpcMode; ``` - *Type:* aws-cdk-lib.aws_ecs.IpcMode - *Default:* IpcMode used by the task is not specified The IPC resource namespace to use for the containers in the task. Not supported in Fargate and Windows containers. --- ##### `networkMode`^Optional ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode - *Default:* NetworkMode.BRIDGE for EC2 tasks, AWS_VPC for Fargate tasks. The Docker networking mode to use for the containers in the task. The valid values are NONE, BRIDGE, AWS_VPC, and HOST. --- ##### `pidMode`^Optional ```typescript public readonly pidMode: PidMode; ``` - *Type:* aws-cdk-lib.aws_ecs.PidMode - *Default:* PidMode used by the task is not specified The process namespace to use for the containers in the task. Not supported in Windows containers. --- ##### `placementConstraints`^Optional ```typescript public readonly placementConstraints: PlacementConstraint[]; ``` - *Type:* aws-cdk-lib.aws_ecs.PlacementConstraint[] - *Default:* No placement constraints. An array of placement constraint objects to use for the task. You can specify a maximum of 10 constraints per task (this limit includes constraints in the task definition and those specified at run time). --- ##### `neuronxInstanceType`^Optional ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType The instance type of compile worker instance. --- ##### `tensorParallelSize`^Optional ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number - *Default:* 1 The number of tensor parallel size. --- ##### `compiledModel`^Required ```typescript public readonly compiledModel: VllmNxdInferenceCompiledModel; ``` - *Type:* VllmNxdInferenceCompiledModel The model to be compiled. --- ##### `environment`^Optional ```typescript public readonly environment: {[ key: string ]: string}; ``` - *Type:* {[ key: string ]: string} - *Default:* No environment variables. The environment variables to pass to the container. This is only applicable when using container runtime. --- ##### `image`^Optional ```typescript public readonly image: VllmNxdInferenceEcsImageBase; ``` - *Type:* VllmNxdInferenceEcsImageBase - *Default:* latest VllmNxdInferenceImage The image to be used for the container. --- ### VllmParallelConfig Configuration for the distributed execution. #### Initializer ```typescript import { VllmParallelConfig } from 'aws-cdk-neuronx-patterns' const vllmParallelConfig: VllmParallelConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | dataParallelSize | number | Number of data parallel groups. | | disableCustomAllReduce | boolean | Disable the custom all-reduce kernel and fall back to NCCL. | | distributedExecutorBackend | DistributedExecutorBackend | Backend to use for distributed model workers, either “ray” or “mp” (multiprocessing). | | enableExpertParallel | boolean | Use expert parallelism instead of tensor parallelism for MoE layers. | | maxParallelLoadingWorkers | number | Maximum number of parallal loading workers when loading model sequentially in multiple batches. | | pipelineParallelSize | number | Number of pipeline parallel groups. | | rayWorkersUseNsight | boolean | Whether to profile Ray workers with nsight. | | tensorParallelSize | number | Number of tensor parallel groups. | --- ##### `dataParallelSize`^Optional ```typescript public readonly dataParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. --- ##### `disableCustomAllReduce`^Optional ```typescript public readonly disableCustomAllReduce: boolean; ``` - *Type:* boolean - *Default:* false Disable the custom all-reduce kernel and fall back to NCCL. --- ##### `distributedExecutorBackend`^Optional ```typescript public readonly distributedExecutorBackend: DistributedExecutorBackend; ``` - *Type:* DistributedExecutorBackend Backend to use for distributed model workers, either “ray” or “mp” (multiprocessing). If the product of pipeline_parallel_size and tensor_parallel_size is less than or equal to the number of GPUs available, “mp” will be used to keep processing on a single host. Otherwise, this will default to “ray” if Ray is installed and fail otherwise. Note that tpu and hpu only support Ray for distributed inference. --- ##### `enableExpertParallel`^Optional ```typescript public readonly enableExpertParallel: boolean; ``` - *Type:* boolean - *Default:* false Use expert parallelism instead of tensor parallelism for MoE layers. --- ##### `maxParallelLoadingWorkers`^Optional ```typescript public readonly maxParallelLoadingWorkers: number; ``` - *Type:* number Maximum number of parallal loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. --- ##### `pipelineParallelSize`^Optional ```typescript public readonly pipelineParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of pipeline parallel groups. --- ##### `rayWorkersUseNsight`^Optional ```typescript public readonly rayWorkersUseNsight: boolean; ``` - *Type:* boolean - *Default:* false Whether to profile Ray workers with nsight. > [https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler](https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler) --- ##### `tensorParallelSize`^Optional ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number - *Default:* 1 Number of tensor parallel groups. --- ### VllmPromptAdapterConfig Configuration for PromptAdapters. #### Initializer ```typescript import { VllmPromptAdapterConfig } from 'aws-cdk-neuronx-patterns' const vllmPromptAdapterConfig: VllmPromptAdapterConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | enablePromptAdapter | boolean | If True, enable handling of PromptAdapters. | | maxPromptAdapters | number | Max number of PromptAdapters in a batch. | | maxPromptAdapterToken | number | Max number of PromptAdapters tokens. | --- ##### `enablePromptAdapter`^Optional ```typescript public readonly enablePromptAdapter: boolean; ``` - *Type:* boolean - *Default:* false If True, enable handling of PromptAdapters. --- ##### `maxPromptAdapters`^Optional ```typescript public readonly maxPromptAdapters: number; ``` - *Type:* number - *Default:* 1 Max number of PromptAdapters in a batch. --- ##### `maxPromptAdapterToken`^Optional ```typescript public readonly maxPromptAdapterToken: number; ``` - *Type:* number - *Default:* 0 Max number of PromptAdapters tokens. --- ### VllmSchedulerConfig #### Initializer ```typescript import { VllmSchedulerConfig } from 'aws-cdk-neuronx-patterns' const vllmSchedulerConfig: VllmSchedulerConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | disableChunkedMmInput | boolean | If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. | | enableChunkedPrefill | boolean | If True, prefill requests can be chunked based on the remaining max_num_batched_tokens. | | longPrefillTokenThreshold | number | For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. | | maxLongPartialPrefills | number | For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. | | maxNumBatchedTokens | number | Maximum number of tokens to be processed in a single iteration. | | maxNumPartialPrefills | number | For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. | | maxNumSeqs | number | Maximum number of sequences to be processed in a single iteration. | | multiStepStreamOutputs | boolean | If False, then multi-step will stream outputs at the end of all steps. | | numLookaheadSlots | number | The number of slots to allocate per sequence per step, beyond the known token ids. | | numSchedulerSteps | number | Maximum number of forward steps per scheduler call. | | preemptionMode | PreemptionMode | Whether to perform preemption by swapping or recomputation. | | schedulerDelayFactor | number | Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt. | | schedulingPolicy | SchedulingPolicy | The scheduling policy to use: - “fcfs” means first come first served, i.e. requests are handled in order of arrival. - “priority” means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). | --- ##### `disableChunkedMmInput`^Optional ```typescript public readonly disableChunkedMmInput: boolean; ``` - *Type:* boolean - *Default:* false If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. --- ##### `enableChunkedPrefill`^Optional ```typescript public readonly enableChunkedPrefill: boolean; ``` - *Type:* boolean If True, prefill requests can be chunked based on the remaining max_num_batched_tokens. --- ##### `longPrefillTokenThreshold`^Optional ```typescript public readonly longPrefillTokenThreshold: number; ``` - *Type:* number - *Default:* 0 For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. --- ##### `maxLongPartialPrefills`^Optional ```typescript public readonly maxLongPartialPrefills: number; ``` - *Type:* number - *Default:* 1 For chunked prefill, the maximum number of prompts longer than long_prefill_token_threshold that will be prefilled concurrently. Setting this less than max_num_partial_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. --- ##### `maxNumBatchedTokens`^Optional ```typescript public readonly maxNumBatchedTokens: number; ``` - *Type:* number Maximum number of tokens to be processed in a single iteration. This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context. --- ##### `maxNumPartialPrefills`^Optional ```typescript public readonly maxNumPartialPrefills: number; ``` - *Type:* number - *Default:* 1 For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. --- ##### `maxNumSeqs`^Optional ```typescript public readonly maxNumSeqs: number; ``` - *Type:* number Maximum number of sequences to be processed in a single iteration. This config has no static default. If left unspecified by the user, it will be set in EngineArgs.create_engine_config based on the usage context. --- ##### `multiStepStreamOutputs`^Optional ```typescript public readonly multiStepStreamOutputs: boolean; ``` - *Type:* boolean - *Default:* true If False, then multi-step will stream outputs at the end of all steps. --- ##### `numLookaheadSlots`^Optional ```typescript public readonly numLookaheadSlots: number; ``` - *Type:* number - *Default:* 0 The number of slots to allocate per sequence per step, beyond the known token ids. This is used in speculative decoding to store KV activations of tokens which may or may not be accepted. NOTE: This will be replaced by speculative config in the future; it is present to enable correctness tests until then. --- ##### `numSchedulerSteps`^Optional ```typescript public readonly numSchedulerSteps: number; ``` - *Type:* number - *Default:* 1 Maximum number of forward steps per scheduler call. --- ##### `preemptionMode`^Optional ```typescript public readonly preemptionMode: PreemptionMode; ``` - *Type:* PreemptionMode Whether to perform preemption by swapping or recomputation. If not specified, we determine the mode as follows: We use recomputation by default since it incurs lower overhead than swapping. However, when the sequence group has multiple sequences (e.g., beam search), recomputation is not currently supported. In such a case, we use swapping instead. --- ##### `schedulerDelayFactor`^Optional ```typescript public readonly schedulerDelayFactor: number; ``` - *Type:* number - *Default:* 0.0 Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt. --- ##### `schedulingPolicy`^Optional ```typescript public readonly schedulingPolicy: SchedulingPolicy; ``` - *Type:* SchedulingPolicy - *Default:* SchedulingPolicy.FCFS The scheduling policy to use: - “fcfs” means first come first served, i.e. requests are handled in order of arrival. - “priority” means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). --- ### VllmSpeculativeConfig Configuration for speculative decoding. #### Initializer ```typescript import { VllmSpeculativeConfig } from 'aws-cdk-neuronx-patterns' const vllmSpeculativeConfig: VllmSpeculativeConfig = { ... } ``` #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | speculativeConfig | {[ key: string ]: any} | The configurations for speculative decoding. | --- ##### `speculativeConfig`^Optional ```typescript public readonly speculativeConfig: {[ key: string ]: any}; ``` - *Type:* {[ key: string ]: any} The configurations for speculative decoding. Should be a object. --- ## Classes ### Inferentia2Chips - *Implements:* IAcceleratorChips #### Initializers ```typescript import { Inferentia2Chips } from 'aws-cdk-neuronx-patterns' new Inferentia2Chips(chips: number) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | chips | number | *No description.* | --- ##### `chips`^Required - *Type:* number --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | acceleratorMemory | aws-cdk-lib.Size | *No description.* | | chips | number | *No description.* | | neuronxCores | number | *No description.* | --- ##### `acceleratorMemory`^Required ```typescript public readonly acceleratorMemory: Size; ``` - *Type:* aws-cdk-lib.Size --- ##### `chips`^Required ```typescript public readonly chips: number; ``` - *Type:* number --- ##### `neuronxCores`^Required ```typescript public readonly neuronxCores: number; ``` - *Type:* number --- ### Model Compile target model. #### Static Functions | **Name** | **Description** | | --- | --- | | fromBucket | model informations at S3 Bucket. | | fromHuggingFace | model informations at HuggingFace. | --- ##### `fromBucket` ```typescript import { Model } from 'aws-cdk-neuronx-patterns' Model.fromBucket(bucket: IBucket, prefix: string, options?: ModelOptions) ``` model informations at S3 Bucket. ###### `bucket`^Required - *Type:* aws-cdk-lib.aws_s3.IBucket Model stored S3 Bucket. --- ###### `prefix`^Required - *Type:* string Model stored objects prefix. --- ###### `options`^Optional - *Type:* ModelOptions model basic information. --- ##### `fromHuggingFace` ```typescript import { Model } from 'aws-cdk-neuronx-patterns' Model.fromHuggingFace(modelId: string, options?: ModelOptions) ``` model informations at HuggingFace. ###### `modelId`^Required - *Type:* string model id on the HuggingFace. --- ###### `options`^Optional - *Type:* ModelOptions model basic information. --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | modelId | string | *No description.* | | modelName | string | *No description.* | | options | ModelOptions | *No description.* | | bucket | aws-cdk-lib.aws_s3.IBucket | *No description.* | | prefix | string | *No description.* | --- ##### `modelId`^Required ```typescript public readonly modelId: string; ``` - *Type:* string --- ##### `modelName`^Required ```typescript public readonly modelName: string; ``` - *Type:* string --- ##### `options`^Required ```typescript public readonly options: ModelOptions; ``` - *Type:* ModelOptions --- ##### `bucket`^Optional ```typescript public readonly bucket: IBucket; ``` - *Type:* aws-cdk-lib.aws_s3.IBucket --- ##### `prefix`^Optional ```typescript public readonly prefix: string; ``` - *Type:* string --- ### NeuronxInstanceType #### Initializers ```typescript import { NeuronxInstanceType } from 'aws-cdk-neuronx-patterns' new NeuronxInstanceType() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | INF2_24XLARGE | INeuronxInstanceType | inf2.24xlarge. | | INF2_48XLARGE | INeuronxInstanceType | inf2.48xlarge. | | INF2_8XLARGE | INeuronxInstanceType | inf2.8xlarge. | | INF2_XLARGE | INeuronxInstanceType | inf2.xlarge. | | TRN1_2XLARGE | INeuronxInstanceType | trn1.2xlarge. | | TRN1_32XLARGE | INeuronxInstanceType | trn1.32xlarge. | | TRN2_3XLARGE | INeuronxInstanceType | trn2.3xlarge. | | TRN2_48XLARGE | INeuronxInstanceType | trn2.48xlarge. | | TRN2U_48XLARGE | INeuronxInstanceType | trn2u.48xlarge. | --- ##### `INF2_24XLARGE`^Required ```typescript public readonly INF2_24XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType inf2.24xlarge. --- ##### `INF2_48XLARGE`^Required ```typescript public readonly INF2_48XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType inf2.48xlarge. --- ##### `INF2_8XLARGE`^Required ```typescript public readonly INF2_8XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType inf2.8xlarge. --- ##### `INF2_XLARGE`^Required ```typescript public readonly INF2_XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType inf2.xlarge. --- ##### `TRN1_2XLARGE`^Required ```typescript public readonly TRN1_2XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType trn1.2xlarge. --- ##### `TRN1_32XLARGE`^Required ```typescript public readonly TRN1_32XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType trn1.32xlarge. --- ##### `TRN2_3XLARGE`^Required ```typescript public readonly TRN2_3XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType trn2.3xlarge. --- ##### `TRN2_48XLARGE`^Required ```typescript public readonly TRN2_48XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType trn2.48xlarge. --- ##### `TRN2U_48XLARGE`^Required ```typescript public readonly TRN2U_48XLARGE: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType trn2u.48xlarge. --- ### Parameters Represents the amount of parameters. #### Methods | **Name** | **Description** | | --- | --- | | toBillion | Return this number of parameters as billion. | | toMillion | Return this number of parameters as million. | | weightSize | *No description.* | --- ##### `toBillion` ```typescript public toBillion(): number ``` Return this number of parameters as billion. ##### `toMillion` ```typescript public toMillion(): number ``` Return this number of parameters as million. ##### `weightSize` ```typescript public weightSize(): Size ``` #### Static Functions | **Name** | **Description** | | --- | --- | | billion | Create a Parameters representing an amount billion. | | million | Create a Parameters representing an amount million. | --- ##### `billion` ```typescript import { Parameters } from 'aws-cdk-neuronx-patterns' Parameters.billion(parameters: number) ``` Create a Parameters representing an amount billion. ###### `parameters`^Required - *Type:* number number of parameters billionX. --- ##### `million` ```typescript import { Parameters } from 'aws-cdk-neuronx-patterns' Parameters.million(parameters: number) ``` Create a Parameters representing an amount million. ###### `parameters`^Required - *Type:* number number of parameters millionX. --- ### PytorchInferenceNeuronxImage #### Initializers ```typescript import { PytorchInferenceNeuronxImage } from 'aws-cdk-neuronx-patterns' new PytorchInferenceNeuronxImage() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | fromNeuronSdkVersion | *No description.* | --- ##### `fromNeuronSdkVersion` ```typescript import { PytorchInferenceNeuronxImage } from 'aws-cdk-neuronx-patterns' PytorchInferenceNeuronxImage.fromNeuronSdkVersion(neuronSdkVersion: string, pythonVersion: string, pytorchVersion: string, ubuntuVersion: string) ``` ###### `neuronSdkVersion`^Required - *Type:* string --- ###### `pythonVersion`^Required - *Type:* string --- ###### `pytorchVersion`^Required - *Type:* string --- ###### `ubuntuVersion`^Required - *Type:* string --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | imageName | string | *No description.* | | LATEST | INeuronxImage | Latest Neuron SDK. | | SDK_2_18_0 | INeuronxImage | Neuron SDK 2.18.0. | | SDK_2_18_1 | INeuronxImage | Neuron SDK 2.18.1. | | SDK_2_18_2 | INeuronxImage | Neuron SDK 2.18.2. | | SDK_2_19_0 | INeuronxImage | Neuron SDK 2.19.0. | | SDK_2_19_1 | INeuronxImage | Neuron SDK 2.19.1. | | SDK_2_20_0 | INeuronxImage | Neuron SDK 2.20.0. | | SDK_2_20_1 | INeuronxImage | Neuron SDK 2.20.1. | | SDK_2_20_2 | INeuronxImage | Neuron SDK 2.20.2. | | SDK_2_20_2_PYTORCH_1_13_1 | INeuronxImage | Neuron SDK 2.20.2 (PyTorch 1.13.1). | | SDK_2_20_2_PYTORCH_2_1_2 | INeuronxImage | Neuron SDK 2.20.2 (PyTorch 2.1.2). | | SDK_2_21_0 | INeuronxImage | Neuron SDK 2.21.0. | | SDK_2_21_1 | INeuronxImage | Neuron SDK 2.21.1. | | SDK_2_22_0 | INeuronxImage | Neuron SDK 2.22.0. | | SDK_2_23_0 | INeuronxImage | Neuron SDK 2.23.0. | | SDK_2_24_0 | INeuronxImage | Neuron SDK 2.24.0. | | SDK_2_25_0 | INeuronxImage | Neuron SDK 2.25.0. | | SDK_2_26_1 | INeuronxImage | Neuron SDK 2.26.1. | | SDK_2_27_1 | INeuronxImage | Neuron SDK 2.27.1. | | SDK_2_28_0 | INeuronxImage | Neuron SDK 2.28.0. | | size | aws-cdk-lib.Size | *No description.* | --- ##### `imageName`^Required ```typescript public readonly imageName: string; ``` - *Type:* string --- ##### `LATEST`^Required ```typescript public readonly LATEST: INeuronxImage; ``` - *Type:* INeuronxImage Latest Neuron SDK. --- ##### `SDK_2_18_0`^Required ```typescript public readonly SDK_2_18_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.0. --- ##### `SDK_2_18_1`^Required ```typescript public readonly SDK_2_18_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.1. --- ##### `SDK_2_18_2`^Required ```typescript public readonly SDK_2_18_2: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.2. --- ##### `SDK_2_19_0`^Required ```typescript public readonly SDK_2_19_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.19.0. --- ##### `SDK_2_19_1`^Required ```typescript public readonly SDK_2_19_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.19.1. --- ##### `SDK_2_20_0`^Required ```typescript public readonly SDK_2_20_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.0. --- ##### `SDK_2_20_1`^Required ```typescript public readonly SDK_2_20_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.1. --- ##### `SDK_2_20_2`^Required ```typescript public readonly SDK_2_20_2: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.2. --- ##### `SDK_2_20_2_PYTORCH_1_13_1`^Required ```typescript public readonly SDK_2_20_2_PYTORCH_1_13_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.2 (PyTorch 1.13.1). --- ##### `SDK_2_20_2_PYTORCH_2_1_2`^Required ```typescript public readonly SDK_2_20_2_PYTORCH_2_1_2: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.2 (PyTorch 2.1.2). --- ##### `SDK_2_21_0`^Required ```typescript public readonly SDK_2_21_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.21.0. --- ##### `SDK_2_21_1`^Required ```typescript public readonly SDK_2_21_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.21.1. --- ##### `SDK_2_22_0`^Required ```typescript public readonly SDK_2_22_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.22.0. --- ##### `SDK_2_23_0`^Required ```typescript public readonly SDK_2_23_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.23.0. --- ##### `SDK_2_24_0`^Required ```typescript public readonly SDK_2_24_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.24.0. --- ##### `SDK_2_25_0`^Required ```typescript public readonly SDK_2_25_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.25.0. --- ##### `SDK_2_26_1`^Required ```typescript public readonly SDK_2_26_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.26.1. --- ##### `SDK_2_27_1`^Required ```typescript public readonly SDK_2_27_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.27.1. --- ##### `SDK_2_28_0`^Required ```typescript public readonly SDK_2_28_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.28.0. --- ##### `size`^Required ```typescript public readonly size: Size; ``` - *Type:* aws-cdk-lib.Size --- ### PytorchTrainingNeuronxImage #### Initializers ```typescript import { PytorchTrainingNeuronxImage } from 'aws-cdk-neuronx-patterns' new PytorchTrainingNeuronxImage() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | fromNeuronSdkVersion | *No description.* | --- ##### `fromNeuronSdkVersion` ```typescript import { PytorchTrainingNeuronxImage } from 'aws-cdk-neuronx-patterns' PytorchTrainingNeuronxImage.fromNeuronSdkVersion(neuronSdkVersion: string, pythonVersion: string, pytorchVersion: string, ubuntuVersion: string) ``` ###### `neuronSdkVersion`^Required - *Type:* string --- ###### `pythonVersion`^Required - *Type:* string --- ###### `pytorchVersion`^Required - *Type:* string --- ###### `ubuntuVersion`^Required - *Type:* string --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | imageName | string | *No description.* | | LATEST | INeuronxImage | Latest Neuron SDK. | | SDK_2_18_0 | INeuronxImage | Neuron SDK 2.18.0. | | SDK_2_18_1 | INeuronxImage | Neuron SDK 2.18.1. | | SDK_2_18_2 | INeuronxImage | Neuron SDK 2.18.2. | | SDK_2_19_0 | INeuronxImage | Neuron SDK 2.19.0. | | SDK_2_19_1 | INeuronxImage | Neuron SDK 2.19.1. | | SDK_2_20_0 | INeuronxImage | Neuron SDK 2.20.0. | | SDK_2_20_1 | INeuronxImage | Neuron SDK 2.20.1. | | SDK_2_20_2 | INeuronxImage | Neuron SDK 2.20.2. | | SDK_2_21_0 | INeuronxImage | Neuron SDK 2.21.0. | | SDK_2_21_1 | INeuronxImage | Neuron SDK 2.21.1. | | SDK_2_22_0 | INeuronxImage | Neuron SDK 2.22.0. | | SDK_2_23_0 | INeuronxImage | Neuron SDK 2.23.0. | | SDK_2_24_0 | INeuronxImage | Neuron SDK 2.24.0. | | SDK_2_25_0 | INeuronxImage | Neuron SDK 2.25.0. | | SDK_2_26_1 | INeuronxImage | Neuron SDK 2.26.1. | | SDK_2_27_1 | INeuronxImage | Neuron SDK 2.27.1. | | SDK_2_28_0 | INeuronxImage | Neuron SDK 2.28.0. | | size | aws-cdk-lib.Size | *No description.* | --- ##### `imageName`^Required ```typescript public readonly imageName: string; ``` - *Type:* string --- ##### `LATEST`^Required ```typescript public readonly LATEST: INeuronxImage; ``` - *Type:* INeuronxImage Latest Neuron SDK. --- ##### `SDK_2_18_0`^Required ```typescript public readonly SDK_2_18_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.0. --- ##### `SDK_2_18_1`^Required ```typescript public readonly SDK_2_18_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.1. --- ##### `SDK_2_18_2`^Required ```typescript public readonly SDK_2_18_2: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.18.2. --- ##### `SDK_2_19_0`^Required ```typescript public readonly SDK_2_19_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.19.0. --- ##### `SDK_2_19_1`^Required ```typescript public readonly SDK_2_19_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.19.1. --- ##### `SDK_2_20_0`^Required ```typescript public readonly SDK_2_20_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.0. --- ##### `SDK_2_20_1`^Required ```typescript public readonly SDK_2_20_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.1. --- ##### `SDK_2_20_2`^Required ```typescript public readonly SDK_2_20_2: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.20.2. --- ##### `SDK_2_21_0`^Required ```typescript public readonly SDK_2_21_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.21.0. --- ##### `SDK_2_21_1`^Required ```typescript public readonly SDK_2_21_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.21.1. --- ##### `SDK_2_22_0`^Required ```typescript public readonly SDK_2_22_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.22.0. --- ##### `SDK_2_23_0`^Required ```typescript public readonly SDK_2_23_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.23.0. --- ##### `SDK_2_24_0`^Required ```typescript public readonly SDK_2_24_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.24.0. --- ##### `SDK_2_25_0`^Required ```typescript public readonly SDK_2_25_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.25.0. --- ##### `SDK_2_26_1`^Required ```typescript public readonly SDK_2_26_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.26.1. --- ##### `SDK_2_27_1`^Required ```typescript public readonly SDK_2_27_1: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.27.1. --- ##### `SDK_2_28_0`^Required ```typescript public readonly SDK_2_28_0: INeuronxImage; ``` - *Type:* INeuronxImage Neuron SDK 2.28.0. --- ##### `size`^Required ```typescript public readonly size: Size; ``` - *Type:* aws-cdk-lib.Size --- ### Trainium1Chips - *Implements:* IAcceleratorChips #### Initializers ```typescript import { Trainium1Chips } from 'aws-cdk-neuronx-patterns' new Trainium1Chips(chips: number) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | chips | number | *No description.* | --- ##### `chips`^Required - *Type:* number --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | acceleratorMemory | aws-cdk-lib.Size | *No description.* | | chips | number | *No description.* | | neuronxCores | number | *No description.* | --- ##### `acceleratorMemory`^Required ```typescript public readonly acceleratorMemory: Size; ``` - *Type:* aws-cdk-lib.Size --- ##### `chips`^Required ```typescript public readonly chips: number; ``` - *Type:* number --- ##### `neuronxCores`^Required ```typescript public readonly neuronxCores: number; ``` - *Type:* number --- ### Trainium2Chips - *Implements:* IAcceleratorChips #### Initializers ```typescript import { Trainium2Chips } from 'aws-cdk-neuronx-patterns' new Trainium2Chips(chips: number) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | chips | number | *No description.* | --- ##### `chips`^Required - *Type:* number --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | acceleratorMemory | aws-cdk-lib.Size | *No description.* | | chips | number | *No description.* | | neuronxCores | number | *No description.* | --- ##### `acceleratorMemory`^Required ```typescript public readonly acceleratorMemory: Size; ``` - *Type:* aws-cdk-lib.Size --- ##### `chips`^Required ```typescript public readonly chips: number; ``` - *Type:* number --- ##### `neuronxCores`^Required ```typescript public readonly neuronxCores: number; ``` - *Type:* number --- ### VllmEngineArgumentsParser #### Initializers ```typescript import { VllmEngineArgumentsParser } from 'aws-cdk-neuronx-patterns' new VllmEngineArgumentsParser() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | cli | *No description.* | | config | Convert vLLM engine arguments (camel case) to config (kebab case). | --- ##### `cli` ```typescript import { VllmEngineArgumentsParser } from 'aws-cdk-neuronx-patterns' VllmEngineArgumentsParser.cli(args: VllmEngineArguments) ``` ###### `args`^Required - *Type:* VllmEngineArguments --- ##### `config` ```typescript import { VllmEngineArgumentsParser } from 'aws-cdk-neuronx-patterns' VllmEngineArgumentsParser.config(args: VllmEngineArguments) ``` Convert vLLM engine arguments (camel case) to config (kebab case). > [https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#configuration-file](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#configuration-file) ###### `args`^Required - *Type:* VllmEngineArguments vLLM engine arguments. --- ### VllmInferenceNeuronxImage #### Initializers ```typescript import { VllmInferenceNeuronxImage } from 'aws-cdk-neuronx-patterns' new VllmInferenceNeuronxImage() ``` | **Name** | **Type** | **Description** | | --- | --- | --- | --- #### Static Functions | **Name** | **Description** | | --- | --- | | fromNeuronSdkVersion | *No description.* | --- ##### `fromNeuronSdkVersion` ```typescript import { VllmInferenceNeuronxImage } from 'aws-cdk-neuronx-patterns' VllmInferenceNeuronxImage.fromNeuronSdkVersion(neuronSdkVersion: string, vllmVersion: string, pythonVersion: string, ubuntuVersion: string) ``` ###### `neuronSdkVersion`^Required - *Type:* string --- ###### `vllmVersion`^Required - *Type:* string --- ###### `pythonVersion`^Required - *Type:* string --- ###### `ubuntuVersion`^Required - *Type:* string --- #### Constants | **Name** | **Type** | **Description** | | --- | --- | --- | | LATEST | IVllmInferenceNeuronxImage | Latest Neuron SDK. | | SDK_2_24_1 | IVllmInferenceNeuronxImage | Neuron SDK 2.24.1 with vLLM 0.7.2. | | SDK_2_25_0 | IVllmInferenceNeuronxImage | Neuron SDK 2.25.0 with vLLM 0.9.1. | | SDK_2_26_0 | IVllmInferenceNeuronxImage | Neuron SDK 2.26.0 with vLLM 0.9.1. | | SDK_2_26_1 | IVllmInferenceNeuronxImage | Neuron SDK 2.26.1 with vLLM 0.9.1. | | SDK_2_27_1_VLLM_0_11_0 | IVllmInferenceNeuronxImage | Neuron SDK 2.27.1 with vLLM 0.11.0. | | SDK_2_27_1_VLLM_0_13_0 | IVllmInferenceNeuronxImage | Neuron SDK 2.27.1 with vLLM 0.13.0. | | SDK_2_28_0 | IVllmInferenceNeuronxImage | Neuron SDK 2.28.0 with vLLM 0.13.0. | --- ##### `LATEST`^Required ```typescript public readonly LATEST: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Latest Neuron SDK. --- ##### `SDK_2_24_1`^Required ```typescript public readonly SDK_2_24_1: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.24.1 with vLLM 0.7.2. --- ##### `SDK_2_25_0`^Required ```typescript public readonly SDK_2_25_0: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.25.0 with vLLM 0.9.1. --- ##### `SDK_2_26_0`^Required ```typescript public readonly SDK_2_26_0: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.26.0 with vLLM 0.9.1. --- ##### `SDK_2_26_1`^Required ```typescript public readonly SDK_2_26_1: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.26.1 with vLLM 0.9.1. --- ##### `SDK_2_27_1_VLLM_0_11_0`^Required ```typescript public readonly SDK_2_27_1_VLLM_0_11_0: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.27.1 with vLLM 0.11.0. --- ##### `SDK_2_27_1_VLLM_0_13_0`^Required ```typescript public readonly SDK_2_27_1_VLLM_0_13_0: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.27.1 with vLLM 0.13.0. --- ##### `SDK_2_28_0`^Required ```typescript public readonly SDK_2_28_0: IVllmInferenceNeuronxImage; ``` - *Type:* IVllmInferenceNeuronxImage Neuron SDK 2.28.0 with vLLM 0.13.0. --- ### VllmNxdInferenceCompileImage Compile runtime container image for vLLM NxD Inference. #### Initializers ```typescript import { VllmNxdInferenceCompileImage } from 'aws-cdk-neuronx-patterns' new VllmNxdInferenceCompileImage(scope: Construct, id: string, vllmInferenceNeuronxImage?: IVllmInferenceNeuronxImage) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | scope | constructs.Construct | *No description.* | | id | string | *No description.* | | vllmInferenceNeuronxImage | IVllmInferenceNeuronxImage | *No description.* | --- ##### `scope`^Required - *Type:* constructs.Construct --- ##### `id`^Required - *Type:* string --- ##### `vllmInferenceNeuronxImage`^Optional - *Type:* IVllmInferenceNeuronxImage --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | image | aws-cdk-lib.aws_ecs.ContainerImage | The container image. | | neuronSdkVersion | string | The neuronx SDK version. | --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage The container image. --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string The neuronx SDK version. --- ### VllmNxdInferenceEcsImage Inference ECS container image for vLLM on NxD Inference. This image uses the official AWS Neuron Deep Learning Containers which come with vLLM pre-installed. *Example* ```typescript new VllmNxdInferenceEcsImage(VllmInferenceNeuronxImage.LATEST) ``` #### Initializers ```typescript import { VllmNxdInferenceEcsImage } from 'aws-cdk-neuronx-patterns' new VllmNxdInferenceEcsImage(vllmInferenceNeuronxImage?: IVllmInferenceNeuronxImage) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | vllmInferenceNeuronxImage | IVllmInferenceNeuronxImage | *No description.* | --- ##### `vllmInferenceNeuronxImage`^Optional - *Type:* IVllmInferenceNeuronxImage --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | image | aws-cdk-lib.aws_ecs.ContainerImage | The container image. | | neuronSdkVersion | string | The neuronx SDK version. | --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage The container image. --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string The neuronx SDK version. --- ### VllmNxdInferenceEcsImageBase - *Implements:* INeuronxContainerImage Base class for VllmNxdInferenceImage. #### Initializers ```typescript import { VllmNxdInferenceEcsImageBase } from 'aws-cdk-neuronx-patterns' new VllmNxdInferenceEcsImageBase(neuronxImage: IVllmInferenceNeuronxImage) ``` | **Name** | **Type** | **Description** | | --- | --- | --- | | neuronxImage | IVllmInferenceNeuronxImage | *No description.* | --- ##### `neuronxImage`^Required - *Type:* IVllmInferenceNeuronxImage --- #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | image | aws-cdk-lib.aws_ecs.ContainerImage | The container image. | | neuronSdkVersion | string | The neuronx SDK version. | --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage The container image. --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string The neuronx SDK version. --- ## Protocols ### IAcceleratorChips - *Implemented By:* Inferentia2Chips, Trainium1Chips, Trainium2Chips, IAcceleratorChips #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | acceleratorMemory | aws-cdk-lib.Size | *No description.* | | chips | number | *No description.* | | neuronxCores | number | *No description.* | --- ##### `acceleratorMemory`^Required ```typescript public readonly acceleratorMemory: Size; ``` - *Type:* aws-cdk-lib.Size --- ##### `chips`^Required ```typescript public readonly chips: number; ``` - *Type:* number --- ##### `neuronxCores`^Required ```typescript public readonly neuronxCores: number; ``` - *Type:* number --- ### INeuronxCompiler - *Implemented By:* NeuronxCompilerBase, NeuronxCrossCompiler, NeuronxNativeCompiler, INeuronxCompiler Interface for Neuronx compilers. #### Methods | **Name** | **Description** | | --- | --- | | compile | *No description.* | --- ##### `compile` ```typescript public compile(): NeuronxCompiledModel ``` ### INeuronxContainerImage - *Implemented By:* VllmNxdInferenceCompileImage, VllmNxdInferenceEcsImage, VllmNxdInferenceEcsImageBase, INeuronxContainerImage Compile runtime. #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | image | aws-cdk-lib.aws_ecs.ContainerImage | An image of the container where the compile job is executed. | | neuronSdkVersion | string | Neuronx version included in container image. | --- ##### `image`^Required ```typescript public readonly image: ContainerImage; ``` - *Type:* aws-cdk-lib.aws_ecs.ContainerImage An image of the container where the compile job is executed. --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string Neuronx version included in container image. --- ### INeuronxImage - *Implemented By:* INeuronxImage, IVllmInferenceNeuronxImage #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | imageName | string | *No description.* | | imageTag | string | *No description.* | | neuronSdkVersion | string | *No description.* | --- ##### `imageName`^Required ```typescript public readonly imageName: string; ``` - *Type:* string --- ##### `imageTag`^Required ```typescript public readonly imageTag: string; ``` - *Type:* string --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string --- ### INeuronxInstanceType - *Implemented By:* INeuronxInstanceType #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | acceleratorChips | IAcceleratorChips | *No description.* | | instanceType | aws-cdk-lib.aws_ec2.InstanceType | *No description.* | | memory | aws-cdk-lib.Size | *No description.* | | supportedTensorParallelism | number[] | *No description.* | | vCpu | number | *No description.* | --- ##### `acceleratorChips`^Required ```typescript public readonly acceleratorChips: IAcceleratorChips; ``` - *Type:* IAcceleratorChips --- ##### `instanceType`^Required ```typescript public readonly instanceType: InstanceType; ``` - *Type:* aws-cdk-lib.aws_ec2.InstanceType --- ##### `memory`^Required ```typescript public readonly memory: Size; ``` - *Type:* aws-cdk-lib.Size --- ##### `supportedTensorParallelism`^Required ```typescript public readonly supportedTensorParallelism: number[]; ``` - *Type:* number[] --- ##### `vCpu`^Required ```typescript public readonly vCpu: number; ``` - *Type:* number --- ### INeuronxTaskDefinition - *Extends:* aws-cdk-lib.aws_ecs.IEc2TaskDefinition - *Implemented By:* NeuronxTaskDefinition, VllmNxdInferenceTaskDefinition, INeuronxTaskDefinition #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | node | constructs.Node | The tree node. | | env | aws-cdk-lib.interfaces.ResourceEnvironment | The environment this resource belongs to. | | stack | aws-cdk-lib.Stack | The stack in which this resource is defined. | | taskDefinitionRef | aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference | A reference to a TaskDefinition resource. | | compatibility | aws-cdk-lib.aws_ecs.Compatibility | What launch types this task definition should be compatible with. | | isEc2Compatible | boolean | Return true if the task definition can be run on an EC2 cluster. | | isExternalCompatible | boolean | Return true if the task definition can be run on a ECS Anywhere cluster. | | isFargateCompatible | boolean | Return true if the task definition can be run on a Fargate cluster. | | isManagedInstancesCompatible | boolean | Return true if the task definition can be run on Managed Instances. | | networkMode | aws-cdk-lib.aws_ecs.NetworkMode | The networking mode to use for the containers in the task. | | taskDefinitionArn | string | ARN of this task definition. | | taskRole | aws-cdk-lib.aws_iam.IRole | The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. | | executionRole | aws-cdk-lib.aws_iam.IRole | Execution role for this task definition. | | compiledModel | NeuronxCompiledModel | *No description.* | | neuronxInstanceType | INeuronxInstanceType | *No description.* | | tasksPerInstance | number | *No description.* | | tensorParallelSize | number | *No description.* | --- ##### `node`^Required ```typescript public readonly node: Node; ``` - *Type:* constructs.Node The tree node. --- ##### `env`^Required ```typescript public readonly env: ResourceEnvironment; ``` - *Type:* aws-cdk-lib.interfaces.ResourceEnvironment The environment this resource belongs to. For resources that are created and managed in a Stack (those created by creating new class instances like `new Role()`, `new Bucket()`, etc.), this is always the same as the environment of the stack they belong to. For referenced resources (those obtained from referencing methods like `Role.fromRoleArn()`, `Bucket.fromBucketName()`, etc.), they might be different than the stack they were imported into. --- ##### `stack`^Required ```typescript public readonly stack: Stack; ``` - *Type:* aws-cdk-lib.Stack The stack in which this resource is defined. --- ##### `taskDefinitionRef`^Required ```typescript public readonly taskDefinitionRef: TaskDefinitionReference; ``` - *Type:* aws-cdk-lib.interfaces.aws_ecs.TaskDefinitionReference A reference to a TaskDefinition resource. --- ##### `compatibility`^Required ```typescript public readonly compatibility: Compatibility; ``` - *Type:* aws-cdk-lib.aws_ecs.Compatibility What launch types this task definition should be compatible with. --- ##### `isEc2Compatible`^Required ```typescript public readonly isEc2Compatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on an EC2 cluster. --- ##### `isExternalCompatible`^Required ```typescript public readonly isExternalCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a ECS Anywhere cluster. --- ##### `isFargateCompatible`^Required ```typescript public readonly isFargateCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on a Fargate cluster. --- ##### `isManagedInstancesCompatible`^Required ```typescript public readonly isManagedInstancesCompatible: boolean; ``` - *Type:* boolean Return true if the task definition can be run on Managed Instances. --- ##### `networkMode`^Required ```typescript public readonly networkMode: NetworkMode; ``` - *Type:* aws-cdk-lib.aws_ecs.NetworkMode The networking mode to use for the containers in the task. --- ##### `taskDefinitionArn`^Required ```typescript public readonly taskDefinitionArn: string; ``` - *Type:* string ARN of this task definition. --- ##### `taskRole`^Required ```typescript public readonly taskRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole The name of the IAM role that grants containers in the task permission to call AWS APIs on your behalf. --- ##### `executionRole`^Optional ```typescript public readonly executionRole: IRole; ``` - *Type:* aws-cdk-lib.aws_iam.IRole Execution role for this task definition. --- ##### `compiledModel`^Required ```typescript public readonly compiledModel: NeuronxCompiledModel; ``` - *Type:* NeuronxCompiledModel --- ##### `neuronxInstanceType`^Required ```typescript public readonly neuronxInstanceType: INeuronxInstanceType; ``` - *Type:* INeuronxInstanceType --- ##### `tasksPerInstance`^Required ```typescript public readonly tasksPerInstance: number; ``` - *Type:* number --- ##### `tensorParallelSize`^Required ```typescript public readonly tensorParallelSize: number; ``` - *Type:* number --- ### IVllmInferenceNeuronxImage - *Extends:* INeuronxImage - *Implemented By:* IVllmInferenceNeuronxImage #### Properties | **Name** | **Type** | **Description** | | --- | --- | --- | | imageName | string | *No description.* | | imageTag | string | *No description.* | | neuronSdkVersion | string | *No description.* | | vllmVersion | string | *No description.* | --- ##### `imageName`^Required ```typescript public readonly imageName: string; ``` - *Type:* string --- ##### `imageTag`^Required ```typescript public readonly imageTag: string; ``` - *Type:* string --- ##### `neuronSdkVersion`^Required ```typescript public readonly neuronSdkVersion: string; ``` - *Type:* string --- ##### `vllmVersion`^Required ```typescript public readonly vllmVersion: string; ``` - *Type:* string --- ## Enums ### BlockSize Cache block size options in number of tokens. #### Members | **Name** | **Description** | | --- | --- | | SIZE_8 | *No description.* | | SIZE_16 | *No description.* | | SIZE_32 | *No description.* | | SIZE_64 | *No description.* | | SIZE_128 | *No description.* | --- ##### `SIZE_8` --- ##### `SIZE_16` --- ##### `SIZE_32` --- ##### `SIZE_64` --- ##### `SIZE_128` --- ### ChatTemplateContentFormat Format options for rendering message content within a chat template. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | STRING | *No description.* | | OPENAI | *No description.* | --- ##### `AUTO` --- ##### `STRING` --- ##### `OPENAI` --- ### ConfigFormat Model config format options. #### Members | **Name** | **Description** | | --- | --- | | AUTO | “auto” will try to load the config in hf format if available else it will try to load in mistral format. | | HF | *No description.* | | MISTRAL | *No description.* | --- ##### `AUTO` “auto” will try to load the config in hf format if available else it will try to load in mistral format. --- ##### `HF` --- ##### `MISTRAL` --- ### DataType Data types for model weights and activations. #### Members | **Name** | **Description** | | --- | --- | | AUTO | “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. | | HALF | “half” for FP16. | | FLOAT16 | “float16” is the same as “half”. | | BFLOAT16 | “bfloat16” for a balance between precision and range. | | FLOAT | “float” is shorthand for FP32 precision. | | FLOAT32 | “float32” for FP32 precision. | --- ##### `AUTO` “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. --- ##### `HALF` “half” for FP16. Recommended for AWQ quantization. --- ##### `FLOAT16` “float16” is the same as “half”. --- ##### `BFLOAT16` “bfloat16” for a balance between precision and range. --- ##### `FLOAT` “float” is shorthand for FP32 precision. --- ##### `FLOAT32` “float32” for FP32 precision. --- ### DataTypeBits #### Members | **Name** | **Description** | | --- | --- | | BF16_OR_FP16 | *No description.* | | FP8_OR_INT8 | *No description.* | --- ##### `BF16_OR_FP16` --- ##### `FP8_OR_INT8` --- ### Device Device type options for vLLM execution. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | CUDA | *No description.* | | NEURON | *No description.* | | CPU | *No description.* | | TPU | *No description.* | | XPU | *No description.* | | HPU | *No description.* | --- ##### `AUTO` --- ##### `CUDA` --- ##### `NEURON` --- ##### `CPU` --- ##### `TPU` --- ##### `XPU` --- ##### `HPU` --- ### DistributedExecutorBackend Distributed execution backend options. #### Members | **Name** | **Description** | | --- | --- | | RAY | *No description.* | | MP | *No description.* | | UNI | *No description.* | | EXTERNAL_LAUNCHER | *No description.* | --- ##### `RAY` --- ##### `MP` --- ##### `UNI` --- ##### `EXTERNAL_LAUNCHER` --- ### GenerationConfig The folder path to the generation config. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | VLLM | *No description.* | --- ##### `AUTO` --- ##### `VLLM` --- ### GuidedDecodingBackend Available guided decoding backends. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | OUTLINES | *No description.* | | LM_FORMAT_ENFORCER | *No description.* | | XGRAMMAR | *No description.* | --- ##### `AUTO` --- ##### `OUTLINES` --- ##### `LM_FORMAT_ENFORCER` --- ##### `XGRAMMAR` --- ### KvCacheDtype KV cache data type options. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | FP8 | *No description.* | | FP8_E4M3 | *No description.* | | FP8_E5M2 | *No description.* | --- ##### `AUTO` --- ##### `FP8` --- ##### `FP8_E4M3` --- ##### `FP8_E5M2` --- ### LoadFormat Available model weight loading formats. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | PT | *No description.* | | SAFETENSORS | *No description.* | | NPCACHE | *No description.* | | DUMMY | *No description.* | | TENSORIZER | *No description.* | | SHARDED_STATE | *No description.* | | GGUF | *No description.* | | BITSANDBYTES | *No description.* | | MISTRAL | *No description.* | | RUNAI_STREAMER | *No description.* | | FASTSAFETENSORS | *No description.* | --- ##### `AUTO` --- ##### `PT` --- ##### `SAFETENSORS` --- ##### `NPCACHE` --- ##### `DUMMY` --- ##### `TENSORIZER` --- ##### `SHARDED_STATE` --- ##### `GGUF` --- ##### `BITSANDBYTES` --- ##### `MISTRAL` --- ##### `RUNAI_STREAMER` --- ##### `FASTSAFETENSORS` --- ### LoraDtype LoRA data type options. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | FLOAT16 | *No description.* | | BFLOAT16 | *No description.* | --- ##### `AUTO` --- ##### `FLOAT16` --- ##### `BFLOAT16` --- ### ModelImpl Model implementation options. #### Members | **Name** | **Description** | | --- | --- | | AUTO | “auto” will try to use the vLLM implementation if it exists and fall back to the Transformers implementation if no vLLM implementation is available. | | VLLM | “vllm” will use the vLLM model implementation. | | TRANSFORMERS | “transformers” will use the Transformers model implementation. | --- ##### `AUTO` “auto” will try to use the vLLM implementation if it exists and fall back to the Transformers implementation if no vLLM implementation is available. --- ##### `VLLM` “vllm” will use the vLLM model implementation. --- ##### `TRANSFORMERS` “transformers” will use the Transformers model implementation. --- ### OptLevel Optimization level. #### Members | **Name** | **Description** | | --- | --- | | MINIMIZING_COMPILE_TIME | enables the core performance optimizations in the compiler, while also minimizing compile time. | | BEST_BALANCE | provides the best balance between model performance and compile time. | | MODEL_EXECUTION_PERFORMANCE | may provide additional model execution performance but may incur longer compile times and higher host memory usage during model compilation. | --- ##### `MINIMIZING_COMPILE_TIME` enables the core performance optimizations in the compiler, while also minimizing compile time. --- ##### `BEST_BALANCE` provides the best balance between model performance and compile time. --- ##### `MODEL_EXECUTION_PERFORMANCE` may provide additional model execution performance but may incur longer compile times and higher host memory usage during model compilation. --- ### PreemptionMode Preemption mode. #### Members | **Name** | **Description** | | --- | --- | | RECOMPUTE | *No description.* | | SWAP | *No description.* | --- ##### `RECOMPUTE` --- ##### `SWAP` --- ### PrefixCachingHashAlgo Hash algorithm options for prefix caching. #### Members | **Name** | **Description** | | --- | --- | | BUILTIN | “builtin” is Python’s built-in hash. | | SHA256 | “sha256” is collision resistant but with certain overheads. | --- ##### `BUILTIN` “builtin” is Python’s built-in hash. --- ##### `SHA256` “sha256” is collision resistant but with certain overheads. --- ### QuantDtype Quant data type. #### Members | **Name** | **Description** | | --- | --- | | S8 | int8 weight storage. | --- ##### `S8` int8 weight storage. --- ### Quantization Quantization methods. #### Members | **Name** | **Description** | | --- | --- | | AQLM | *No description.* | | AWQ | *No description.* | | DEEPSPEEDFP | *No description.* | | TPU_INT8 | *No description.* | | FP8 | *No description.* | | PTPC_FP8 | *No description.* | | FBGEMM_FP8 | *No description.* | | MODELOPT | *No description.* | | NVFP4 | *No description.* | | MARLIN | *No description.* | | GGUF | *No description.* | | GPTQ_MARLIN_24 | *No description.* | | GPTQ_MARLIN | *No description.* | | AWQ_MARLIN | *No description.* | | GPTQ | *No description.* | | COMPRESSED_TENSORS | *No description.* | | BITSANDBYTES | *No description.* | | QQQ | *No description.* | | HQQ | *No description.* | | EXPERTS_INT8 | *No description.* | | NEURON_QUANT | *No description.* | | IPEX | *No description.* | | QUARK | *No description.* | | MOE_WNA16 | *No description.* | | TORCHAO | *No description.* | | NONE | *No description.* | --- ##### `AQLM` --- ##### `AWQ` --- ##### `DEEPSPEEDFP` --- ##### `TPU_INT8` --- ##### `FP8` --- ##### `PTPC_FP8` --- ##### `FBGEMM_FP8` --- ##### `MODELOPT` --- ##### `NVFP4` --- ##### `MARLIN` --- ##### `GGUF` --- ##### `GPTQ_MARLIN_24` --- ##### `GPTQ_MARLIN` --- ##### `AWQ_MARLIN` --- ##### `GPTQ` --- ##### `COMPRESSED_TENSORS` --- ##### `BITSANDBYTES` --- ##### `QQQ` --- ##### `HQQ` --- ##### `EXPERTS_INT8` --- ##### `NEURON_QUANT` --- ##### `IPEX` --- ##### `QUARK` --- ##### `MOE_WNA16` --- ##### `TORCHAO` --- ##### `NONE` --- ### ReasoningParser Available reasoning parsers. #### Members | **Name** | **Description** | | --- | --- | | DEEPSEEK_R1 | *No description.* | | GRANITE | *No description.* | --- ##### `DEEPSEEK_R1` --- ##### `GRANITE` --- ### SchedulingPolicy Scheduling policy options. #### Members | **Name** | **Description** | | --- | --- | | FCFS | *No description.* | | PRIORITY | *No description.* | --- ##### `FCFS` --- ##### `PRIORITY` --- ### TokenizerMode Tokenizer mode options. #### Members | **Name** | **Description** | | --- | --- | | AUTO | “auto” will use the fast tokenizer if available. | | SLOW | “slow” will always use the slow tokenizer. | | MISTRAL | “mistral” will always use the mistral_common tokenizer. | | CUSTOM | “custom” will use –tokenizer to select the preregistered tokenizer. | --- ##### `AUTO` “auto” will use the fast tokenizer if available. --- ##### `SLOW` “slow” will always use the slow tokenizer. --- ##### `MISTRAL` “mistral” will always use the mistral_common tokenizer. --- ##### `CUSTOM` “custom” will use –tokenizer to select the preregistered tokenizer. --- ### TokenizerPoolType Tokenizer pool type options. #### Members | **Name** | **Description** | | --- | --- | | RAY | *No description.* | --- ##### `RAY` --- ### ToolCallParser Tool call parser options. #### Members | **Name** | **Description** | | --- | --- | | GRANITE_20B_FC | *No description.* | | GRANITE | *No description.* | | HERMES | *No description.* | | INTERNLM | *No description.* | | JAMBA | *No description.* | | LLAMA3_JSON | *No description.* | | MISTRAL | *No description.* | | PHI4_MINI_JSON | *No description.* | | PYTHONIC | *No description.* | --- ##### `GRANITE_20B_FC` --- ##### `GRANITE` --- ##### `HERMES` --- ##### `INTERNLM` --- ##### `JAMBA` --- ##### `LLAMA3_JSON` --- ##### `MISTRAL` --- ##### `PHI4_MINI_JSON` --- ##### `PYTHONIC` --- ### UvicornLogLevel Log level options for Uvicorn. #### Members | **Name** | **Description** | | --- | --- | | DEBUG | *No description.* | | INFO | *No description.* | | WARNING | *No description.* | | ERROR | *No description.* | | CRITICAL | *No description.* | | TRACE | *No description.* | --- ##### `DEBUG` --- ##### `INFO` --- ##### `WARNING` --- ##### `ERROR` --- ##### `CRITICAL` --- ##### `TRACE` --- ### VllmTask Task options for model usage. #### Members | **Name** | **Description** | | --- | --- | | AUTO | *No description.* | | GENERATE | *No description.* | | EMBEDDING | *No description.* | | EMBED | *No description.* | | CLASSIFY | *No description.* | | SCORE | *No description.* | | REWARD | *No description.* | | TRANSCRIPTION | *No description.* | --- ##### `AUTO` --- ##### `GENERATE` --- ##### `EMBEDDING` --- ##### `EMBED` --- ##### `CLASSIFY` --- ##### `SCORE` --- ##### `REWARD` --- ##### `TRANSCRIPTION` ---